Title: PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: implicit neural representation; diffusion; point cloud; volume rendering
Abstract: Recent advances in implicit neural representations have achieved impressive results by sampling and fusing individual points along sampling rays in the sampling space. However, due to the explosively growing sampling space, finely representing and synthesizing detailed textures remains a challenge for unbounded large-scale outdoor scenes. To alleviate the dilemma of using individual points to perceive the entire colossal space, we explore learning the surface distribution of the scene to provide structural priors and reduce the samplable space and propose a Point Diffusion implicit Function, PDF, for large-scale scene neural representation. The core of our method is a large-scale point cloud super-resolution diffusion module that enhances the sparse point cloud reconstructed from several training images into a dense point cloud as an explicit prior. Then in the rendering stage, only sampling points with prior points within the sampling radius are retained. That is, the sampling space is reduced from the unbounded space to the scene surface. Meanwhile, to fill in the background of the scene that cannot be provided by point clouds, the region sampling based on Mip-NeRF 360 is employed to model the background representation. Expensive experiments have demonstrated the effectiveness of our method for large-scale scene novel view synthesis, which outperforms relevant state-of-the-art baselines.

PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation

Abstract

Recent advances in implicit neural representations have achieved impressive results by sampling and fusing individual points along sampling rays in the sampling space. However, accurately representing and synthesizing fine-grained textures in unbounded, large-scale outdoor scenes presents a significant challenge, attributable to the exponentially expanding sampling space. To alleviate the dilemma of using individual sampling points to perceive vast expanses, we explore learning the surface distribution of the scene to provide structural priors, thereby reducing the samplable space, and propose a P oint D iffusion implicit F unction, PDF , for large-scale scene neural representation. The core of our method is a large-scale point cloud super-resolution diffusion module that enhances the sparse point cloud reconstructed from several training images into a dense point cloud as the explicit prior. Then in the rendering stage, only sampling points with prior points within the sampling radius are retained. That is, the sampling space is reduced from the unbounded space to the scene surface. Meanwhile, to fill in the background details not captured by point clouds, we employ region sampling based on Mip-NeRF 360 for modeling comprehensive background representations. Extensive experiments have demonstrated the effectiveness of our method for large-scale scene novel view synthesis, which outperforms relevant state-of-the-art baselines.

1 Introduction

Implicit neural representations have demonstrated proficiency in handling single objects or small scenes and have found extensive applications in the fields of virtual reality [5], 3D reconstruction [21, 31, 36], video generation [7] and computer animation [19, 1, 41] on tasks such as scene representation and new perspective synthesis. Nevertheless, as the scale of the target scene increases, particularly in urban-scale outdoor scenes, traditional implicit neural representation methods encounter significant

performance limitations. This issue primarily arises from the cubic expansion of the sampling space in larger scenes, rendering it challenging for individual sampling points to cover the entire space.

Fortunately, some methods try to solve this problem through two primary strategies, narrowing the sampling space and expanding the sampling area. The first approach represented by Mega-NeRF [28] decomposes the sampling space into multiple subspaces and models each subspace separately to reduce complexity. But with the scene scale growing, such as reaching the city level, the quantity of these subspaces increases cubically, posing scalability concerns. In contrast, the method represented by Mip-NeRF 360 [2] compresses the sampling space or samples an area instead of a single point so that the sampling space can be filled more easily. Nevertheless, this approach may compromise the precision of the representation.

Benefiting from the inspiration of geometric priors aiding vision tasks, which are widely used in the domain of 3D reconstruction and stereo vision [16, 3, 6]. We are curious whether implicit large scene representations could be made easier with explicit representations. Moreover, for outdoor unbounded large scenes, most of the sampling space is filled with air rather than buildings, cars, plants and other objects that we care about. A reasonable solution is to restrict the large-scale sampling space of the implicit neural representation to the object surface, which is provided by the scene geometry prior. That is to say, we compress a 3D sampling space to a 2D surface plane, which will greatly reduce the representation complexity. At the same time, the network will pay more attention to the foreground, which is the same as the human visual perception system. Of course, for the neglected background information that is relatively less important, we can provide a relatively less accurate expression by compressing the scene to sample the area in space.

In this paper, we propose PDF , a P oint D iffusion implicit F unction for large-scale scene neural representation, which learns a dense surface distribution via a diffusion-based point prior generative model to reduce the sampling space. To achieve this, we first explore a large-scale outdoor point cloud augmentation method based on the Point-Voxel Diffusion model [43]. Since point clouds of real outdoor scenes often lack dense ground truth, it is difficult to train a completion module through "sparse-dense" point cloud pairs. Therefore, we initially downsample the point cloud twice, and train a point cloud super-resolution network to generate the denser one from its sparser counterpart. This approach effectively generates dense point clouds in the absence of ground truth data. With the help of the surface point cloud, the sampling points will be retained only if there are reconstruction points within a certain radius, so the space will be greatly reduced to the scene surface. However, the reconstructed point cloud can only model the scene surface and cannot deal with the unbounded background of outdoor scenes. Accordingly, following the concept of NeRF++ [41], we model the foreground and background separately, and use Mip-NeRF 360 [2] to extract background features by sampling regions in the scene space.

Extensive experiments show the effectiveness of our point diffusion implicit function for large-scale scene neural representation, which achieves photo-realistic rendering results and outperforms stateof-the-art methods on OMMO [15] and BlendMVS dataset [35]. We summarize the contributions as follows: 1) Aiming at novel view synthesis for large outdoor scenes, we propose an implicit neural representation framework based on point diffusion models to provide dense surface priors to cope with the exploding sampling space. 2) A novel point cloud super-resolution diffusion module is proposed to generate dense surface points from sparse point clouds without dense annotations. 3) Extensive experiments demonstrate that our PDF network outperforms state-of-the-art methods, including robustness to large-scale outdoor scene representation and the capability to synthesize more photo-realistic novel views. Our code and models will be available.

2 Related Work and Background

2.1 Implicit Neural Representation

In recent years, Implicit Neural Representation (INR) has witnessed significant advancements and provides a versatile framework for representing complex functions and generating high-dimensional data [23, 17, 26, 37]. By implicitly encoding the scene's appearance and geometry, neural radiance fields enable highly realistic rendering and novel view synthesis [25, 20, 8, 18].

Building upon this foundation, subsequent research has focused on addressing the limitations and pushing the boundaries of INR. Efforts have been made to improve the efficiency and scalability

of neural radiance fields. For instance, Hanocka et al. propose DeepSDF[22], which leverages signed distance functions to implicitly represent 3D shapes. This formulation allows for efficient ray-marching and facilitates tasks such as shape manipulation and interpolation. Furthermore, recent advancements in INR have explored differentiable rendering and differentiable volumetric rendering, enabling the incorporation of geometric and physical priors [10, 4, 27] into the representation. These methods leverage the differentiable nature of neural networks to optimize scene parameters, leading to improved realism and control over the generated content [13, 39, 9, 30]. Another significant extension to the field of INR is PixelNeRF [38]. It extends the capabilities of INR to handle images, going beyond the realm of 3D scenes. PixelNeRF introduces a new differentiable sampler to handle image-based representations, enabling efficient and accurate sampling of pixels from the neural radiance field. In addition to PixelNeRF, Semantic Neural Radiance Fields[12] propose a method to learn scene representations that capture geometry, appearance, and semantic information, facilitating interactive virtual scene editing and content creation.

Overall, these advancements have greatly expanded the capabilities of INR. These developments offer promising avenues for realistic image synthesis, shape completion, scene reconstruction, and dynamic content generation. The ongoing research in this field holds great potential for further advancements in computer graphics, computer vision, and virtual reality applications.

2.2 Large-scale Scene Representation

Large-scale scene representation is a crucial aspect of INR research, particularly in the context of computer graphics and computer vision. It involves capturing and modeling complex scenes that encompass extensive spatial extents, such as urban environments, landscapes, or virtual worlds.

One notable work in the domain of large-scale scene representation is Neural Scene Flow Fields[14]. This paper introduces a novel approach to model dynamic scenes at a large scale. The authors propose a scene flow field representation that captures both the geometry and motion of objects in the scene. By leveraging a neural network architecture, they achieve accurate and temporally consistent scene synthesis and reconstruction, even in highly complex and dynamic scenes. The Neural 3D Mesh Renderer[11] is another significant contribution in large-scale scene representation. This work addresses the challenge of representing and rendering detailed 3D meshes of large-scale scenes efficiently. The authors propose a neural network-based renderer that predicts view-dependent textures and geometric details of the scene. This approach enables real-time rendering and interaction with large-scale 3D scenes, opening up possibilities for interactive virtual reality experiences and immersive simulations. In addition to these works, Mega-NeRF [28] and Bungee-NeRF [33] are two other notable approaches based on the neural radiance field for constructing interactive 3D environments from large-scale visual captures. They address the challenges of modeling and rendering large-scale scenes, spanning from buildings to multiple city blocks and utilizing thousands of images captured from drones. They extend the capabilities of NeRF to handle multi-scale rendering, capturing various levels of detail and enabling the interactive exploration of diverse 3D environments.

Overall, the field of large-scale scene representation within INR has witnessed significant progress. These contributions have paved the way for realistic, interactive, and semantically meaningful representations of expansive virtual environments, urban landscapes, and dynamic scenes. The ongoing research in this area holds great potential for further advancements in computer graphics, virtual reality, and immersive simulations.

3 Methodology

In this paper, we aim to develop a novel point diffusion model implicit function to reduce the sampling space and improve the ability to represent large-scale scenes ( c.f. Fig. 1). Our PDF network mainly consists of two modules, a diffusion-based component for point cloud super-resolution and foreground rendering, and a region-sampling module focused on background processing. The former introduces a diffusion model to enhance the sparse point cloud reconstructed from the input images into a dense point cloud, which provides optional points in the rendering stage to reduce the sampling space ( c.f. Sec. 3.1). The latter samples regions rather than individual points from unbounded scenes so that it is easy to fill sampled regions and complement the background for new viewpoint synthesis ( c.f. Sec. 3.2). In the final subsection, implementation details and losses are elaborated ( c.f. Sec. 3.3).

Point Diffusion Render Module

MVS

model

Training Views

𝑋

!

, p(𝑋

!

)

...

𝑋

"#$

𝑋

"

𝑃

!

(𝑋

"

|𝑋

"#$

)

𝑞(𝑋

"#$

|𝑋

"

)

...

𝑋

%

, 𝑞(𝑋

%

)

Dense Point Cloud

Volume Rendering

Multi-View Stereo

reserve:

discard:

𝐹

&

MLP

+

Foreground Feature

Background Feature

𝐹

'

MLP

Synthetic views

Euclidean Gaussians

Contracted Gaussians

Contracted Domain

Unaffected Domain

Sampling

K

q

q

3.1 Point Upsampling Diffusion

reverse process t = 0 t = T In this section, we introduce our large-scale outdoor point cloud super-resolution module based on a denoising diffusion probabilistic model ( c.f. Fig. 2).

𝑝 & 𝑝 & Point Cloud Pair Preparation. Due to the lack of dense large-scale outdoor point cloud ground truth, we need to train a diffusion-based super-resolution network to sample a dense surface, symbolized as x d ∈ R N × 3 , from the point cloud reconstructed by COLMAP [24], denoted as x s ∈ R M × 3 . Concurrently, to mitigate the risk of over-fitting, the point cloud reconstructed from the training views is not utilized as the ground truth for the diffusion model. Instead, training data comprises pairs of the sparse point cloud z 0 ∈ R n × 3 and the sparser point cloud x 0 ∈ R m × 3 , adhering to m < n < M < N . More specifically, we downsample the sparse point cloud x s reconstructed by COLMAP to get an even sparser point cloud z 0 . Then we further downsample z 0 to get the sparsest point cloud x 0 , where x s , z 0 and x 0 have progressively sparser relationships. Our training process recovers z 0 from the sparsest x 0 . During testing, we take x s as input to generate a denser super-resolved point cloud x d .

Point Super-resolution Diffusion. Our point super-resolution denoising diffusion probabilistic model is a generative model, which starts with Gaussian noise and progressively denoises to generate scene structure priors. We record the output containing different levels of noise produced by each step as ˆ x T , ˆ x T -1 ,..., ˆ x 0 , where ˆ x T is sampled from Gaussian noise, and ˆ x 0 represents the generated point cloud with dense surface. Since we already have a sparse point cloud prior z 0 , our target point cloud can be denoted as x 0 = ( z 0 , ˆ x 0 ) and the intermediate point cloud during the denoising process can be denoted as x t = ( z 0 , ˆ x t ) . Subsequently, we define a point super-resolution diffusion process involving a prior shape z 0 , consisting of a forward process and a backward process.

Forward Process. Gaussian noise is repeatedly added to the original point cloud x 0 , resulting in a series of noisy point clouds x 1 , x 2 ,..., x T :

where β t represents a pre-defined increasing sequence of Gaussian noise values, which dictates the magnitude of noise incrementally introduced at each step of the process.

Reverse Process. Given a point cloud with more noise x t , reverse the forward process and find the posterior distribution for a less noisy one x t -1 :

where µ θ ( x t , z 0 , t ) is the predicted shape at t -1 step.

Training Views

forward process

reverse process

t = 0

t = T

q

q

𝑝

&

𝑝

&

Therefore, our point cloud upsampling diffusion model can be regarded as a noise adding and denoising process. The former gradually adds random noise to the initial point cloud x 0 through the forward process; the latter denoises sequentially through the reverse process to obtain a dense point cloud x 0 . Based on Markov transition probabilities, the whole process can be expressed as:

Throughout the optimization process, our prior shape z 0 is fixed, and only the missing surface point cloud is diffused. The network is typically trained with a simplified L 2 denoising loss:

where ϵ is the added random noise and ϵ ∼ N (0 , I ) , and ϵ θ (ˆ x t , z 0 , t ) is the prediction noise output. Since point cloud prior z 0 is fixed, it will be masked when minimizing the loss.

3.2 Volume Rendering and Implicit Function Representation

Foreground Rendering. For the foreground, we sample points along the rays from the dense point cloud x 0 and render features in the neighborhood, following Point-NeRF [34]. The difference is that our point super-resolution diffusion module only generates denser point cloud coordinates without color information, so we redesign a more general point aggregate module. For ray marching through a pixel, we sample M sampling points at { p i | i = 1 , ..., M } , and query K neighboring neural points kp i = { kp 1 i , kp 2 i , ..., kp K i } around p i within a certain euclidean distance radius R . Then we interpret the local geometric structure as a feature f i of each sampling point p i to equip structural information. Therefore, we utilize c i and kc i to represent the coordinates of the sampling point and its neighborhood points, respectively. The geometric structure feature kf i = { kf 1 i , kf 2 i , ..., kf K i } of the neighborhood are encoded as follows:

where d ( , ) is the Euclidean distance between two points, and ⊕ is the concatenation operator. Next, the local geometric features f i of the sampling points p i are obtained by neighborhood points kp i weighted summation:

where softmax operation is performed on each dimension, and ⊙ is the hadamard product. Our point-based radiance field can be abstracted as a neural module that regresses the volume density σ and view-dependent radiance r from coordinates c , local geometric features f , and ray direction d according to Point-NeRF [34]:

Finally, the foreground feature is synthesized by each neural sampling point along the sampling ray.

Background Rendering. In consideration of the limitation that point clouds are confined to representing foreground elements and are incapable of addressing unbounded background contexts, it becomes necessary to procure supplementary background features. Benefiting from Mip-NeRF 360 [2], which contracts the scene to a bounded ball and then samples a region to meet the challenge of large scenes, we employ this method to extract background features as a supplement along the same sampling ray as Point-NeRF [34].

Fore-Background Fusion. Since the detail-preserving foreground features can be obtained from the dense surface points, while the bounded domain can cope with large scenes but loses details during the compression process. So we propose a foreground-background fusion module consisting of several layers of multi-layer perceptrons to preserve their respective advantages.

We adopt L 2 loss to supervise our rendered pixels r p from ray marching with the ground truth r g , to optimize our PDF volume render reconstruction network.

3.3 Implementation Details

Our PDF method is a two-stage neural representation network for outdoor unbounded large-scale scenes. We optimize these two stages separately.

In the first stage, a diffusion-based point cloud super-resolution network is designed to learn a prior distribution to generate a dense point cloud surface. In the point cloud pair preparation process, we employed the random down-sampling method with a retention rate between 0.2 and 1 for both samplings. For point super-resolution diffusion, we set T = 1000, β 0 = 10 -4 , β T = 0.01 and linearly interpolate other β 's for all experiments. We use Adam optimizer with learning rate 2 × 10 -4 and train on 4 A100 GPUs for around one day.

In the second stage, the foreground and background extraction modules plus a feature fusion module are optimized. We find 8 neighbors for each sampling point and expand the dimension of neighborhood geometric features to 8. Both the foreground and the background output a 128-dimensional feature, and then they are concatenated and passed through 4 MLP layers to get the color of the rendered point. We train this stage using Adam optimizer with an initial learning rate 5 × 10 -4 for 2 × 10 6 iterations about 20 hours on a single A100 GPU.

4 Experiments

4.1 Experimental settings

Dataset. We use two outdoor large-scale scene datasets, OMMO [15] and BlendedMVS [35], to evaluate our model. The OMMO dataset is a real fly-view large-scale outdoor multi-modal dataset, containing complex objects and scenes with calibrated images, prompt annotations and point clouds. The number of training point cloud samples in the OMMO dataset varies from 40,000 to 100,000 for different scenes, including abundant real-world urban and natural scenes with various scales, camera trajectories, and lighting conditions. More experimental results can be found in our supplementary material.

Baselines and Evaluation Metrics. We compare our method with the previous state-of-the art methods on novel view synthesis, including NeRF [19], NeRF++ [41], Mip-NeRF [1], Mip-NeRF 360 [2], Mega-NeRF [28], Ref-NeRF [29]. NeRF is the first continuous MLP-based neural network for synthesizing photo-realistic views of a scene through volume rendering. NeRF++ models largescale unbounded scenes by separately modeling foreground and background neural representations.

Mip-NeRF reduces aliasing artifacts and better represents fine details by using anti-aliasing cone sampling. Mip-NeRF 360 models large unbounded scenes using non-linear scene parameterization, online distillation, and distortion-based regularization. Mega-NeRF uses a sparse structure and geometric clustering algorithm to decompose the scenes. Ref-NeRF improves synthesized views by restructuring radiance and regularizing normal vectors. To evaluate the performance of each method for large-scale implicit neural representation, we use three common metrics for novel view synthesis: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM [32]), and Learned Perceptual Image Patch Similarity (LPIPS [42]). Higher PSNR and SSIM indicate better performance, while lower LPIPS indicates better performance.

4.2 Performance Comparison

Quantitative Results. Quantitative comparisons on the OMMO [15] dataset are shown in Tab. 1, including the mean PSNR, SSIM, and LPIPS. We outperform other methods on all average evaluation metrics, especially LPIPS, a perceptual metric close to the human visual system, which is significantly more sensitive to the foreground than the background. So better LPIPS metric indicates that our model can better reconstruct the foreground of the scene, benefiting from sampling the foreground from the reconstructed dense point cloud surface instead of the entire sampling space.

NeRF [19], NeRF++ [41], Mip-NeRF [1], and Ref-NeRF [29] are not specially designed for largescale scenes, so directly applying them to large scenes will lead to performance degradation. MipNeRF 360 [2] and Mega-NeRF [28] have achieved the optimal performance in one or several scenes by sampling regions in the limited sampling space or subdividing the sampling space. But it is still not as good as ours in most scenes due to the loss of detail caused by compression or decomposing the sampling space.

Qualitative Results. Qualitative results on the OMMO [15] dataset are shown in Fig. 3. We can see that the rendering results of NeRF [19] and Mip-NeRF [1] are of the lowest quality, and they use global MLPs for the entire space to reconstruct radiance fields, resulting in a trade-off in the accuracy of sampling the foreground and background which makes them almost impossible to handle large-scale unbounded scenes. NeRF++ [41], Mega-NeRF [28] and Ref-NeRF [29] improve some limitations of NeRF by corresponding techniques, but the rendering results are often missing details, especially when the scene contains a lot of intricate details. The rendering quality of Mip-NeRF 360 [2] is relatively high, but loses some detail and edges due to its down-scaling of the scene into a limited sampling space. Our method uses the dense point cloud up-sampled by the diffusion model as a detailed foreground geometry prior combined with Mip-NeRF 360 background features, so our model can reconstruct the fine foreground texture provided by the generative model. At the same

NeRF

GT

Ours

Ref-NeRF

Mega-NeRF

Mip-NeRF 360

Mip-NeRF

NeRF++

time, compared with Mip-NeRF 360, our method is more robust to scene representation and new view generation without failing scenes ( c.f. Fig. 4).

4.3 Ablation Studies

We perform multiple ablation studies to validate the effectiveness of our proposed modules. Tab. 2 shows the impact of diffusion point cloud super-resolution module and background feature fusion module on the 5-th scene (sydney opera house) from the OMMO dataset [15].

Ours

Mip-NeRF 360

GT

For the ablation experiment on the effectiveness of diffusion, we remove the diffusion-based point cloud up-sampling module and sample directly on the sparse point cloud reconstructed by COLMAP [24] from training views. Since the directly reconstructed point cloud is very sparse and concentrated in the central area, only a very blurry image with large missing blocks can be rendered, as shown in the first column of Fig. 5. At the same time, quantitative indicators also suggest that this method is not suitable for outdoor unbounded large-scale scenes with its PSNR of 9.28.

For the ablation experiment on the effectiveness of background fusion, we remove the background fusion module and render novel view images directly from the diffusion-enhanced point cloud. As shown in the second column of Fig. 5, with the help of the dense point cloud produced by the diffusion module learning the scene distribution, we find that large missing patches have been filled in and produce a more refined foreground. However, limited by the characteristics of point cloud expression, the background points are very sparse, which leads to blurred background rendering results. Quantitative results, while substantially improved, still convey poor image quality.

As shown in the third column of Fig. 5, using the background fusion module alone can also fill in the missing blocks of the background, but due to the sparseness of the point cloud reconstructed by COLMAP [24], it will lead to the loss of detail and blurring of the rendering result. However, our method, which combines a diffusion module and a background fusion module, achieves satisfactory quantitative and qualitative performance and surpasses existing methods.

w/odiffusion

w/

diffusion

w/odiffusion

w/obackground

w/obackground

w/

background

Ours

GT

We also perform ablation experiments to compare our method with other point cloud up-sampling methods. With the same experimental setup, we use a GAN-based method [40] for point cloud up-sampling instead of the diffusion-based up-sampling module. Tab. 3 shows the quantitative results for three scenes (scan5, scan11 and scan12) in the OMMO dataset. Our method exhibits superior performance compared to the GAN-based point cloud up-sampling method, primarily due to its

ability to preserve the structural and topological characteristics of point clouds while effectively handling incomplete or noisy point cloud data. In addition, Fig. 6 shows the visualization results of the diffusion-based point cloud up-sampling module, and our method can not only densify the sparse point cloud reconstructed by the COLMAP, but also fill in the missing regions of the point cloud such as the background and empty space.

(a)                                                    (b)                                                     (c)

(d)

5 Conclusions and Limitations

In this paper, we propose PDF, a point diffusion implicit function for large-scale scene neural representation, and demonstrate its robustness and fidelity on novel view synthesis tasks. The core of our method is to provide dense point cloud surface priors to reduce the huge sampling space of large-scale scenes. Therefore, a point cloud super-resolution module based on diffusion model is proposed to learn from the sparse point cloud surface distribution reconstructed from training views to generate more dense point clouds. However, only constraining the sampling space to the point cloud surface does not fully solve the novel view synthesis problem since point clouds do not have background information. So Mip-NeRF 360 [2] is employed to provide background features and synthesize photo-realistic new perspectives. Extensive experiments demonstrate that our method outperforms current methods in both subjective and objective aspects. At the same time, ablation experiments also prove the effectiveness of our core module, point up-sampling diffusion.

In future work, we will attempt to explore a cross-scene point cloud up-sampling generalization diffusion model instead of training a diffusion model for each scene to improve efficiency. Even more futuristically, it may be possible to extract representative scene representations and inject them into reconstructed point clouds to achieve cross-scene rendering, i.e., generalized point diffusion NeRF.

References

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16123-16133, 2022.

Tianlong Chen, Peihao Wang, Zhiwen Fan, and Zhangyang Wang. Aug-nerf: Training stronger neural radiance fields with triple-level physically-grounded augmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15191-15202, 2022.

Nianchen Deng, Zhenyi He, Jiannan Ye, Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics , 28(11):3854-3864, 2022.

Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18409-18418, 2022.

Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5784-5794, 2021.

Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18398-18408, 2022.

Yi-Hua Huang, Yue He, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao. Stylizednerf: consistent 3d scene stylization as stylized nerf via 2d-3d mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18342-18352, 2022.

Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18365-18375, 2022.

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3907-3916, 2018.

Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems , 35:23311-23330, 2022.

Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4340-4350, 2023.

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6508, 2021.

Chongshan Lu, Fukun Yin, Xin Chen, Wen Liu, Tao Chen, Gang Yu, and Jiayuan Fan. A large-scale outdoor multi-modal dataset and benchmark for novel view synthesis and implicit scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7557-7567, 2023.

Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with onsurface priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6315-6325, 2022.

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7210-7219, 2021.

Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16190-16199, 2022.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Benjamin Joseph Mildenhall. Neural Scene Representations for View Synthesis . University of California, Berkeley, 2020.

Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5589-5599, 2021.

Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 165-174, 2019.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016.

Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. Advances in Neural Information Processing Systems , 34:12278-12291, 2021.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum , volume 41, pages 703-735. Wiley Online Library, 2022.

Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12922-12931, 2022.

Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5481-5490. IEEE, 2022.

Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3835-3844, 2022.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 , 2021.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In European conference on computer vision , pages 106-122. Springer, 2022.

Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-based neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5438-5448, 2022.

Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1790-1799, 2020.

Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems , 34:4805-4815, 2021.

Fukun Yin, Wen Liu, Zilong Huang, Pei Cheng, Tao Chen, and Gang Yu. Coordinates are not lonelycodebook prior helps implicit neural 3d representations. Advances in Neural Information Processing Systems , 35:12705-12717, 2022.

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4578-4587, 2021.

Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18353-18364, 2022.

Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1768-1777, 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5826-5835, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a point cloud-based representation for neural rendering of large-scale scenes.  Point cloud diffusion model is designed to upsample the point cloud for better performance. The proposed method achieves state-of-the-art performance in the tested scenes. However, I still have some concerns about the implementation details and the performance comparison.\n\nSTRENGTHS:\n1. The proposed method achieves state-of-the-art performance in the neural rendering of large-scale scenes.\n2. Using diffusion models for point cloud upsampling is interesting and useful in the tested scenes.\n3. The presentation and writing of the paper is good and the visualization comparison is clear.\n\nWEAKNESSES:\n1. Size of the train dataset for diffusion. Diffusion model typically requires large-scale dataset for better performance. I'm curious about the number of training point cloud samples for diffusion model.\n2. Reconstruction quality. The 3D reconstruction quality of MVS methods like COLMAP is not guaranteed and there can exist large holes in texture-less regions (e.g., white walls in outdoor scenarios.) I wonder the performance of the diffusion model in such situations (as the diffusion model in the manuscript is mainly designed for upsampling, not completion).\n3. Comparison with Point-NeRF. Point-NeRF is an obvious baseline for this method, which should be included in the experiments. The ablations in the experiments can be seen as a variantion of Point-NeRF. However, Point-NeRF include a point grown and pruning strategy, which can also densify the point cloud. It would be better to test the performance of this strategy in large-scale scenes, which has not been explored before.\n4. Inference time. The proposed method exploit an explicit reconstruction of the scene, thus the empty space can be simply skipped for better efficiency. However, the complexity of the proposed method is not reported and some implementation details are missed (e.g., the number of sampled points)\n5. Background features. Mip-NeRF 360 is utilized for background rendering. However, as shown in Fig.5, the background also contributes to fore-ground regions. It would be better to include an explanation for this.\n6. Others. Diffusion models are widely used for generative models and their exsiting randomness in the generated results. However, the 3D reconstruction/the neural rendering should be deterministic. I wonder the influence of the randomness in this method. Besides, I'm also curious about the failure cases of the method.\n\nQUESTIONS:\nPlease refer to Weaknesses.\n\nLIMITATIONS:\nPlease refer to Weaknesses (especially 2. and 6.).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a method for reconstructing large-scale scenes from multi-view images. Given the sparse point cloud computed from an MVS pipeline, the algorithm first employs a point cloud diffusion model to upsample the point cloud. The utilization of the diffusion model could effectively create a dense cloud with missing regions completed. Then, a Point-NeRF-style model is used to reconstruct the scene based on the upsampled point. Foreground and background renderings are fused via an additional fusion network, resulting in the final output. The algorithm is tested on OMMO and BlendMVS datasets. Compared to existing NeRF baselines, the proposed pipeline could limit the sampling region to the surface and achieves better reconstruction and rendering quality.\n\nSTRENGTHS:\n1. Using the diffusion model for point cloud upsampling and completion is novel. While previous methods perform point cloud upsampling or completion using VAE or GAN models, this is the first paper that leverages the power of diffusion models. One related work could be Luo et al. [a], yet the proposed algorithm treats the partial input cloud as a condition and hence is also able to generate an arbitrary number of points.\n2. The results are compelling. Using the upsampled point cloud as the supporting domain of NeRF, the method could reconstruct a much larger scene with finer details. Both the qualitative and quantitative experiments demonstrate the effectiveness of the proposed approach.\n\n[a] Luo, Shitong, and Wei Hu. \"Diffusion probabilistic models for 3d point cloud generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nWEAKNESSES:\n1. While the use of the diffusion model for point cloud upsampling is novel, its combination with the subsequent NeRF model is very straightforward and detached. The NeRF model is only a slight modification from Point-NeRF, with the addition of the background MLP, and such a representation itself is not insightful enough to form an individual contribution.\n2. Some descriptions regarding the method section and experiment section are not very clear. What is the training scheme of the diffusion model? (more details in the 'Question' section). What does '... as a feature $f_i$ of each sampling point $p_i$ to equip structural information' in Line 172 mean? \n3. Ablation study using other point cloud upsampling methods is missing. Given the main contribution of the paper is the diffusion-based upsampling network, it is worth investigating if a diffusion-based backbone could surpass a traditional GAN-based backbone. The study could be conducted on a simple toy dataset.\n\nQUESTIONS:\nThe main question arises from the training strategy as well as the model specifications of the diffusion model:\n1. What is the model architecture of the denoiser? How many parameters are needed?\n2. Which dataset is used to train the diffusion model? How many training samples are there? Is data augmentation applied?\n3. How many points are generated from the diffusion model?\n4. What is the mechanism that allows the diffusion model to complete missing regions (as shown in Fig. 5)? Are regions randomly masked out during the training stage to encourage the model to hallucinate missing contents?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an implicit neural representation for large-scale scenes with two major components, the first one is the point diffusion implicit function (PDF) which adopts diffusion process to generate dense point cloud from point cloud produced by Colmap, the second one is the background rendering which basically borrow the idea from Mip-Nerf360.\n\nSTRENGTHS:\n•\tThe paper is well organized and easy to follow.\n•\tThe qualitative comparison on the fly-view dataset clearly shows the effectiveness of the proposed method on the large-scale scenes, where the rendered images are sharp in both foreground and background.\n•\tMost of nerf paper focus on small or medium scene, this paper instead provides a interesting solution to the large scale scene representation.\n\nWEAKNESSES:\n•\tThe effectiveness of the diffusion process, the motivation behind adding the diffusion process over colmap generated point cloud is not clear to me since the PointNerf method can generate clean and sharp foreground rendering directly with the colmap point cloud. Some explanation or visualization showing the point cloud before and after diffusion will be very helpful. \n•\tOne major advantages of point-cloud based neural field is they can avoid prohibitive reconstruction time of Nerf, so a question what is the training time and rendering speed of this paper. Will the diffusion process significantly slow down the training speed?\n•\tThe overall technical contribution of this paper is weak. The main component PDF shares the major design with PointNerf except applying a diffusion process over the Colmap-generated point cloud. Currently there is no enough discussion over the insight behind this design as I said before. The background rendering module also follows the existing technique from Mip-Nerf 360. Therefore I think the contribution is not enough.\n•\tIn the experiment part the author said they use two dataset for evaluation but I didn’t find the result on BlendMVS\n\nQUESTIONS:\n•\tHow was the diffusion process necessary for large-scale scene representation, I do see an ablation study about that module, but I am not sure why removing the diffusion make the method totally failed but the PointNerf can works well with pointcloud directly from COLMAP. More explanation or visualization will be very helpful. \n•\tWhat’s the runtime of the proposed methods.\n•\tA general (but maybe not that related to the proposed method): Is there any other solutions to the background rendering? Will the environment map work for the background of the large outdoor scene.\n\nLIMITATIONS:\nThe authors have discussed potential negative social impacts of this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new approach to tackle the reconstruction of Neural Radiance Fields from large, unbounded scenes. A major limitation of current neural fields is the lack of scalability due to the number of sampling points in empty space, which is limiting the achievable scale or quality with limited compute resources. In this method, the sampling space is restricted to the surface of the object. To get an approximation of the surface of the scene, they utilize a point cloud, which first is reconstructed through MVS and upscaled to achieve a dense point cloud on the surface using a diffusion probabilistic model. Only sampling points in the near distance of the surface point cloud and the respective feature, which is constructed as a predicted feature from the neighborhood points and sampling points and forms the foreground feature together with the constructed feature. The unbounded background of the scene is modeled on a sphere as features. Foreground and background features are fused in another MLP.\n\nSTRENGTHS:\n- The presented method builds on known techniques for neural rendering and generative super-resolution and combines those to solve a novel task of reconstructing large-scale unbounded scenes. I think this is a big strength of the paper, which is justified by qualitative and quantitative results and ablation studies.\n- The training approach shows a clear explanation of how the missing ground truth data for the point cloud can be optimized with the limited resources \n- The authors present their method in an understandable way, adding the necessary context and references to reproduce the work \n- Presented results show a clear advantage over standard NeRF techniques, and better results in areas of fine textures compared to methods designed for large-scale or unbounded scene\n- Most design choices are ablated and justified in separate studies\n\nWEAKNESSES:\n- One major weakness of the proposed method is that it is required to retrain the whole model per scene. This is also mentioned by the authors in their limitations but leads to really long training times compared to sota methods.\n- While the result section presents all relevant quantitative results in a table, the authors do not provide an explanation for some of the outliers and rather good performances of the baselines in some scenes.  \n- I hope I have not missed something, but there is no explanation of Fig.4. To my understanding, this provides a scene in which Mip-NeRF 360 fails completely to reconstruct the scene, but no explanation is given. Also, I am curious why it results in something like this and not only blurry results as reported in Fig.3 \n- One major limitation or experiment important for the specific task of large-scale, outdoor scene reconstruction is to show the boundaries of the method in terms of scene scale. Mega-NeRF is specially designed to divide a scene in multiple sections and might be able to handle larger scenes, which is not the case for this method. An additional study on the limitation of this method in terms of scene size alongside with training and rendering times would probably just strengthen the claims\n\nQUESTIONS:\n- Can you provide the training and rendering times for a given resolution for your and all reference methods\n- The authors mentioned that the method was only evaluated on a subset provided by the authors of OMMO, but the full dataset results in the supplementary provide similar insights.\n\nLIMITATIONS:\nMost limitations are mentioned at the end of the paper. As discussed in the weakness section, I think an evaluation of limitation wrt scene size and computational times in a similar setting, would provide a better understanding of the capabilities of this methos.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a novel method for the novel view synthesis of large-scale outdoor scenes. The method combines a Point-NeRF-baed rendering stage for the foreground, as well as a background stage based on Mip-NeRF 360. To cope with the issue of overly sparse point cloud from COLMAP MVS, a point could diffusion model is trained on each individual scene from existing MVS point cloud, and used to synthesize more points. The proposed method achieves leading performance on both large and small scenes.\n\nSTRENGTHS:\n* Training a scene-specific diffusion model for point cloud upsampling is a novel approach that seems to work well.\n* The proposed method is able to achieve better quality on a large set of real-world scenes both quantitatively and qualitatively.\n\nWEAKNESSES:\n* The proposed method requires a diffusion model to be trained on each scene, which can require significant computation.\n* In terms of comparison with previous works, it is not sure if the comparison is based on a level ground, as the proposed method might require more computation, more parameter count, or both.\n* It is not sure if the proposed method is able to outperform methods that are based on grid data structure, such as InstantNGP, Plenoxels, etc., which tend to be very fast and scalable to large, sparse and unbounded scenes.\n* It is not sure if other ways of upsampling the point clouds will also work. Although the use of a scene-specific diffusion model to upsample point could is novel, it is not compared against any baselines. It might provide better insight and more helpful for future work if this contribution can be studied independently as a separate work.\n\nQUESTIONS:\n* Could you elaborate more on the reason behind the failure of Mip-NeRF 360 in Figure 4?\n* Why is the background partially missing in Fig. 5 column 1?\n\nLIMITATIONS:\nAlthough there is a Limitations section, it only includes future works and there lacked discussions on the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}