PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation
Authors: Yuhan Ding
Fukun Yin
Jiayuan Fan
Hui Li
Xin Chen
Wen Liu
Chongshan Lu
Gang YU
Tao Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: implicit neural representation; diffusion; point cloud; volume rendering
TL;DR: PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation

Abstract: Recent advances in implicit neural representations have achieved impressive results by sampling and fusing individual points along sampling rays in the sampling space. However, due to the explosively growing sampling space, finely representing and synthesizing detailed textures remains a challenge for unbounded large-scale outdoor scenes. To alleviate the dilemma of using individual points to perceive the entire colossal space, we explore learning the surface distribution of the scene to provide structural priors and reduce the samplable space and propose a Point Diffusion implicit Function, PDF, for large-scale scene neural representation. The core of our method is a large-scale point cloud super-resolution diffusion module that enhances the sparse point cloud reconstructed from several training images into a dense point cloud as an explicit prior. Then in the rendering stage, only sampling points with prior points within the sampling radius are retained. That is, the sampling space is reduced from the unbounded space to the scene surface. Meanwhile, to fill in the background of the scene that cannot be provided by point clouds, the region sampling based on Mip-NeRF 360 is employed to model the background representation. Expensive experiments have demonstrated the effectiveness of our method for large-scale scene novel view synthesis, which outperforms relevant state-of-the-art baselines.

PDF: Point Diffusion Implicit Function for Large-scale Scene Neural Representation
Yuhan Ding
Fukun Yin fkyin21 221mmmmmmmmmmmmmmmmmmmmmm fudan. edu cn
Jiayuan Fan yfan www.ddannedu.....
dingyh220m.c fudan. edu,
Hui Li
Xin Chen
Wen Liu
1ihui210m.. fudan edu. en
chenxth22@hannhaattecc edu. cn
sawvenn@hhanhaittecee edu,
Chongshan Lu
Gang Yu
Tao Chen'
cslui7@fudan edu. en
i iskicy@gmail.com
eetchen&fudan edu.
School of Information Science and Technology, Fudan University, China Academy for Engineering and Technology, Fudan University, China Tencent PCG, Shanghai, China
Abstract
Recent advances in implicit neural representations have achieved impressive results by sampling and fusing individual points along sampling rays in the sampling space. However, accurately representing and synthesizing fine-grained textures unbounded, large-scale outdoor scenes presents significant challenge, attributable to the exponentially expanding sampling space. To alleviate the dilemma of using individual sampling points to perceive vast expanses, explore learning the surface distribution of the scene to provide structural priors, thereby reducing the samplable space, and propose Point Diffusion implicit Function, PDF, for large-scale scene neural representation. The core of our method is a large-scale point cloud super-resolution diffusion module that enhances the sparse point cloud reconstructed from several training images into a dense point cloud as the explicit prior. Then in the rendering stage, only sampling points with prior points within the sampling radius are retained. That is, the sampling space is reduced from the unbounded space to the scene surface. Meanwhile, fill in the background details not captured by point clouds, we employ region sampling based on Mip-NeRF 360 for modeling comprehensive background representations. Extensive experiments have demonstrated the effectiveness of our method for large-scale scene novel view synthesis, which outperforms relevant trt baselines.
Introduction
Implicit neural representations have demonstrated proficiency in handling single objects or small scenes and have found extensive applications in the fields of virtual reality [5). 3D reconstruction [2]] 36), video generation 7 and computer animation [19 [ 41 on tasks such as scene representation and perspective synthesis. Nevertheless, the scale of the target scene increases, particularly in urban-scale outdoor scenes, traditional implicit neural representation methods encounter significant
"Joint first authors. 'Correponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

performance limitations. This issue primarily arises from the cubic expansion of the sampling space larger scenes, rendering it challenging for individual sampling points cover the entire space.
Fortunately, some methods try to solve this problem through two primary strategies, narrowing the sampling space and expanding the sampling area. The first approach represented by Mega-NeRF [28] decomposes the sampling space into multiple subspaces and models each subspace separately to reduce complexity. But with the scene scale growing, such as reaching the city level, the quantity of these subspaces increases cubically, posing scalability concerns. In contrast, the method represented compresses the sampling space or samples an area instead of single point so
by Mip-NeRF 360 2 that the sampling space can be filled more easily. Nevertheless, this approach may compromise the precision of the representation.
Benefiting from the inspiration of geometric priors aiding vision tasks, which are widely used in the domain of 3D reconstruction and stereo vision [16] 3 [ We are curious whether implicit large scene representations could be made easier with explicit representations. Moreover, for outdoor unbounded large scenes, most of the sampling space is filled with air rather than buildings, cars, plants and other objects that we care about. A reasonable solution is to restrict the large-scale sampling space of the implicit neural representation to the object surface, which is provided by the scene geometry prior. That is to say, we compress 3D sampling space to a 2D surface plane, which will greatly reduce the representation complexity. At the same time, the network will pay more attention to the foreground. which is the same as the human visual perception system. Of course, for the neglected background information that is relatively less important, we can provide a relatively less accurate expression by compressing the scene to sample the area in space.
In this paper, we propose PDF, Point Diffusion implicit Function for large-scale scene neural representation, which learns a dense surface distribution via a diffusion-based point prior generative model to reduce the sampling space. To achieve this, we first explore a large-scale outdoor point cloud augmentation method based on the Point- Voxel Diffusion model [3]. Since point clouds of real outdoor scenes often lack dense ground truth, it is difficult to train completion module through uparse-dense point cloud pairs. Therefore, we initially downsample the point cloud twice, and train a point cloud ruper-resolution network to generate the denser one from its sparser counterpart. This approach effectively generates dense point clouds in the absence of ground truth data. With the help of the surface point cloud, the sampling points will be retained only if there are reconstruction points within a certain radius, so the space will be greatly reduced to the scene surface. However, the reconstructed point cloud can only model the scene surface and cannot deal with the unbounded background of outdoor scenes. Accordingly, following the concept of NeRF++ [41], we model the foreground and background separately, and use Mip-NeRF 360 2 to extract background features by sampling regions in the scene space.
Extensive experiments show the effectiveness of our point diffusion implicit function for large-scale scene neural representation. which achieves photo-realistic rendering results and outperforms state- of-the-art methods on OMMO [15] and BlendMVS dataset (35). We summarize the contributions as follows: 1) Aiming at novel view synthesis for large outdoor scenes, we propose an implicit neural representation framework based on point diffusion models to provide dense surface priors to cope with the exploding sampling space. 2) A novel point cloud super-resolution diffusion module proposed to generate dense surface points from sparse point clouds without dense annotations. 3) Extensive experiments demonstrate that our PDF network outperforms state-of-the-art methods, including robustness to large-scale outdoor scene representation and the capability to synthesize more photo-realistic novel views. Our code and models will be available.
2 Related Work and Background
2.1 Implicit Neural Representation
In recent years, Implicit Neural Representation (INR) has witnessed significant advancements and provides a versatile framework for representing complex functions and generating nigh-dinensional data 23 [7] [26] 37]. By implicitly encoding the scene's 's appearance and geometry, neural radiance
fields enable highly realistic rendering and novel view synthesis 25 20 R [8]
Building upon this foundation, subsequent research has focused on addressing the limitations and pushing the boundaries of INR. Efforts have been made to improve the efficiency and scalability

---

of neural radiance fields. For instance, Hanocka et al. propose DeepSDF[2). which leverages signed distance functions to implicitly represent 3D shapes. This formulation allows for efficient ray-marching and facilitates tasks such as shape manipulation and interpolation. Furthermore, recent advancements INR have explored differentiable rendering and differentiable volumetric rendering, enabling the incorporation of geometric and physical priors [TO] 4
[27] into the representation. These methods leverage the differentiable nature neural networks to optimize scene parameters, leading to improved realism and control over the generated content [3 39 30), Another significant extension to the field of INR is PixelNeRF [38]. It extends the capabilities of INR to handle images, going beyond the realm of 3D scenes. PixeINeRF introduces a new differentiable sampler to handle image- image-based representations. enabling efficient and accurate sampling of pixels from the neural radiance field. In addition to PixeINeRF, Semantic Neural Radiance Fields[I2] propose a method to learn scene representations that capture geometry, appearance, and semantic information, facilitating interactive virtual scene editing and content creation.
Overall, these advancements have greatly expanded the capabilities of INR. These developments offer promising avenues for realistic image synthesis, shape completion, scene reconstruction, and dynamic content generation. The ongoing research this field holds great potential for further advancements in computer graphics, computer vision, and virtual reality applications.
2.2 Large-scale Scene Representation
Large-scale scene representation is a crucial aspect of INR research, particularly in the context of computer graphics and computer vision. It involves capturing and modeling complex scenes that encompass extensive spatial extents, such as urban environments, landscapes, or virtual worlds.
One notable work in the domain of large-scale scene representation is Neural Scene Flow Fields [14) This paper introduces novel approach to model dynamic scenes at large scale. The authors propose a scene flow field representation that captures both the geometry and motion of objects in the scene. By leveraging a neural network architecture, they achieve accurate and temporally consistent scene synthesis and reconstruction, even in highly complex and dynamic scenes. The Neural 3D Mesh [enderer([[] another significant contribution large-scale scene representation. This work addresses the challenge of representing and rendering detailed 3D meshes of large-scale scenes efficiently. The authors propose a neural network- based renderer that predicts dieeddependent textures and geometric details of the scene. This approach enables real-time rendering and interaction with large-scale 3D scenes, opening up possibilities for interactive virtual reality experiences and immersive simulations. In addition to these works, Mega-NeRF 2X and Bungee-NeRR [33] are two other notable approaches based on the neural radiance field for constructing interactive 3D environments from large-scale visual captures. They address the challenges of modeling and rendering large-scale scenes, spanning from buildings to multiple city blocks and utilizing thousands of images captured from drones. They extend the capabilities of NeRF to handle multi-scale rendering, capturing various levels of detail and enabling the interactive exploration of diverse 3D environments.
Overall, the field of large-scale scene representation within INR has witnessed significant progress. These contributions have paved the way for realistic, interactive, and semantically meaningful representations of expansive virtual environments, urban landscapes, and dynamic scenes. The ongoing research in this area holds great potential for further advancements computer graphies, virtual reality, and immersive simulations.
3 Methodology
In this paper, we aim to develop novel point diffusion model implicit function to reduce the sampling space and improve the ability to represent large-scale scenes (c.f. Fig. D.. Our PDF network mainly consists of two modules, diffusion-basse based component for point cloud super-resolution and foreground rendering, and sampling module focused on background processing. The former introduces a diffusion model to enhance the sparse point cloud reconstructed from the input images into dense point cloud, which provides optional points in the rendering stage to reduce the sampling space (c. f. Sec. (.T). The latter samples regions rather than individual points from unbounded scenes so that it is easy to fill sampled regions and complement the background for new viewpoint synthesis (c.f. Sec.
3.2 In the final subsection, implementation details and losses elaborated (c.f. Sec. 3.3
3

---

Point Diffusion Render Module
Multi-V Niew
Sparse Point Cloud
Dense Point Cloud
Volume Rendering discard:
MVS
Xess
Xt
model
@ (Ky)
Xa. (Xa)
Background Render Module
Foreground Feature
Synthetic views
Sampling
Pp
MLP
MLP
Background Feature
Cannend Deman
Radideen Ganscies
Training Views
The pipeline of our point diffusion implicit function. Our method consists of two modules, point diffusion rendering module and background rendering module. The former learns the surface distribution of the scene through diffusion- -based point cloud super-resolution model and renders foreground features from the dense point cloud surface. The latter follows Mip-NeRF 360's strategy to render background features. Finally, the foreground and background features are fused to generate photo-realistic novel views for large-scale outdoor scenes.
Figure 1:
3.1 Point Upsampling Diffusion
In this section, we introduce our large-scale outdoor point cloud super-resolution module based on denoising diffusion probabilistic model (c.f. Fig. 2
Point Cloud Pair Preparation. Due to the lack of dense large-scale outdoor point cloud ground truth, need to train liffusion-basse super-resolution network to sample dense surface, symbolized as Id E TR Nx3 from the point cloud reconstructed by COLMAP 24, denoted as Vs E TRMx3 Concurrently,
mitigate the risk of over-fitting, the point cloud reconstructed from the training views is not utilized as the ground truth for the diffusion model. Instead, training data comprises pairs of the sparse point cloud zo Rax3 and the sparser point cloud xnn E I adhering to m < n < M < N. More specifically, we downsample the sparse point cloud x. reconstructed by COLMAP to get an even sparser point cloud 2g. Then we further downsample 2g to get the sparsest point cloud 2o, where I., 2g and zo have progressively sparser relationships. Our training process recovers 20 from the sparsest zg. During testing, we take x, as input generate denser super-resolved point cloud Iz--
Point Super-resolution Diffusion. Our point super-resolution denoising diffusion probabilistic model is a generative model, which starts with Gaussian noise and progressively denoises to generate scene structure priors. We record the output containing different levels of noise produced by each step as ãT, ÆT ..... &o. where र is sampled from Gaussian noise, and do represents the generated point cloud with dense surface. Since we already have sparse point cloud prior zn, our target point cloud can be denoted as xo = (zo, f.)) and the intermediate point cloud during the denoising process can be denoted as 2, (2o, ..). Subsequently, we define a point super-resolution diffusion process involving a prior shape 2p. consisting of forward process and backward process.
Forward Process. Gaussian noise is repeatedly added the original point cloud zo, resulting in series of noisy point clouds Iı, T2.... Dr:
qi../2,11. 20) Ni - 1,8,,),,,8.1))
(I)
where 33 represents pre-defined increasing sequence of Gaussian noise values, which dictates the magnitude of noise incrementally introduced at each step of the process.
Reverse Process. Given a point cloud with more noise F,, reverse the forward process and find the posterior distribution for less noisy one 3:
(a | m0) Nuuo zo, 20,4), 01
where μe (xe, 20, is the predicted shape at t 1 step.

---

forward process
t=0
t=T
reverse process
P8
Po
Our point upsampling diffusion. In the forward process. Gaussian noise is gradually added to the sparse point cloud. In the reverse process, the noise gradually removed to obtain dense point cloud surface.
Figure 2:
Therefore, our point cloud upsampling diffusion model can regarded as noise adding and denoising process. The former gradually adds random noise to the initial point cloud zo through the forward process; the latter denoises sequentially through the reverse process to obtain a dense point cloud Zg. Based on Markov transition probabilities, the whole process can be expressed as:
q 20) q(0, 20) II q 20)
(3)
Po(dur, 20) p(er, 20)
II pe (2.-1 182, (=1
(4)
Throughout the optimization process, prior shape 2o is fixed, and only the missing surface point cloud diffused. The network is typically trained with a simplified L2 denoising loss:
Lo = ||c - (8,, 20, 20. t)11
where is the added random noise and (0, I). and c 20, 1) the prediction noise output. Since point cloud prior zp is fixed, will be masked when minimizing the loss.
3.2 Volume Rendering and Implicit Function Representation
Foreground Rendering. For the foreground, sample points along the rays from the dense point cloud in and render features in the neighborhood, following Point-NeRI (34]. The difference is that our point super-resolution diffusion module only generates denser point cloud coordinates without color information, so we redesign a more general point aggregate module. For ray marching through I....,MI M). and query K neighboring neural points kp; = [kp] kp?.... kp. K) } around p: within a certain euclidean distance radius Then we interpret the local geometric structure as a feature f. of each sampling point p; to equip structural information. Therefore, we utilize e and kc; to represent the coordinates of the sampling point and its neighborhood points, respectively. The geometric structure feature k = [kf]. k ...,,.... kff of the neighborhood are encoded as follows:
pixel, we sample M sampling points {p,
kf. MLP(c @ kc, ① (c: kc.)) @ d(ca, kc,))
(6)
where d(.) ) is the Euclidean distance between two points, and ① is the concatenation operator. Next, the local geometric features f. of the sampling points p. obtained by neighborhood points kp, weighted summation:
f. = SU (00UUMsof kfa)
5

---

where softmax operation is performed on each dimension, and @ is the hadamard product. Our point-based radiance field can be abstracted as a neural module that regresses the volume density and view-dependent radiance r from coordinates c, local geometric features and ray direction d according to Point-NeRF 34
(a.r) Point-NeRF((,d.
(8)
Finally, the foreground feature is synthesized by each neural sampling point along the sampling ray.
Background Rendering. In consideration of the limitation that point clouds are confined rep- resenting foreground elements and are incapable of addressing unbounded background contexts, it becomes necessary procure supplementary background features. Benefiting from Mip-NeRF 360 [2]. which contracts the scene to a bounded ball and then samples a region to meet the challenge of large scenes, employ this method to extract background features supplement along the same sampling ray as Point-NeRF 34
Fore-Background Fusion. Since the detail- preserving foreground features can be obtained from the dense surface points, while the bounded domain can cope with large scenes but loses details during the compression process. So we propose grregroudd-backgrowd fusion module consisting of several layers of multi-layer perceptrons to preserve their respective advantages.
We adopt L2 loss to supervise our rendered pixels Tp from ray marching with the ground truth Tg. to optimize our PDF volume render reconstruction network.
LR 1|r, rroll
3.3 Implementation Details
Our PDF method is a two-stage neural representation network for outdoor unbounded scale scenes. We optimize these two stages separately.
In the first stage, diffusion-based point cloud super-resolution network is designed learn prior distribution to generate a dense point cloud surface. In the point cloud pair preparation process, we employed the random nown-sampling method with retention rate between 0.2 and for both samplings. For point super-resolution diffusion, we set T' = 1000, Bg 10 10-4,
4 BT = 0.01 and linearly
interpolate other 0's for all experiments. We use Adam optimizer with learning rate x 10 10-4 and train on 4 A100 GPUs for around one day.
In the second stage, the foreground and background extraction modules plus a feature fusion module are optimized. We find 8 neighbors for each sampling point and expand the dimension of neighbor- hood geometric features 8. Both the foreground and the background output 1 128-dimensional feature, and then they are concatenated and passed through 4 MLP layers to get the color of the rendered point. We train this stage using Adam optimizer with an initial learning rate x 10-4 for 2 x 10 iterations about 20 hours on a single A100 GPU,
Experiments
4.1 Experimental settings
Dataset. We use two outdoor large-scale scene datasets, OMMO [5] and BlendedMVS [35] to evaluate our model. The OMMO dataset is real fly-view large-scale outdoor multi-modal dataset, containing complex objects and scenes with calibrated images, prompt annotations and point clouds. The number of training point cloud samples in the OMMO dataset varies from 40,000 to 100,000 for different scenes, including abundant real-world urban and natural scenes with various scales, camera trajectories, and lighting conditions. More experimental results can be found in our supplementary material.
Baselines and Evaluation Metrics. We compare our method with the previous state-of-the art methods on novel view synthesis, including NeRF [99, NeRF++ [41), Mip-NeRF M. Mip-NeRF 360 2 Mega-NeRF (28] Ref-NeRF (29]. NeRF is the first continuous ML.P-baeed neural network for synthesizing photo-realistie views of a scene through volume rendering. NeRF++ models large- scale unbounded scenes by separately modeling foreground and background neural representations.

---

Mip-NeRF reduces aliasing artifacts and better represents fine details by using anti-aliasing cone sampling. Mip-NeRF 360 models large unbounded scenes using non-linear scene parameterization. online distillation, and distortion-based regularization. Mega-NeRF uses a sparse structure and geometric clustering algorithm to decompose the scenes. Ref-NeRF improves synthesized views by restructuring radiance and regularizing normal vectors. To evaluate the performance of each method for large-scale implicit neural representation, we use three common metrics for novel view synthesis: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM (321), and Learned Perceptual Image Patch Similarity (LPIPS 42). Higher PSNR and SSIM indicate better performance, while lower LPIPS indicates better performance.
4.2 Performance Comparison
Quantitative comparisons on the OMMO [15] dataset are shown in Tab. I including the mean PSNR, SSIM, and LPIPS. We outperform other methods on all average evaluation metrics, especially LPIPS, a perceptual metric close to the human visual system, which is significantly more sensitive to the foreground than the background. So better LPIPS metric indicates that our model can better reconstruct the foreground of the scene, benefiting from sampling the foreground from the reconstructed dense point cloud surface instead of the entire sampling space.
Quantitative Results.
NeRF 19 NeRF++ [4]] Mip-NeRF [I, and Ref-NeRF [29] are not specially designed for large- scale scenes, directly applying them to large scenes will lead to performance degradation. Mip- NeRF 360 ( and Mega-NeRF (28) have achieved the optimal performance in one or several scenes by sampling regions in the limited sampling space subdividing the sampling space. But it is still not as good as ours in most scenes due to the loss of detail caused by compression or decomposing the sampling space.
Table 1: Quantitative results of our PDF method with the baselines on the OMMO dataset. means the higher, the better.
Scone
PSNR|
NeRE
SSIMI
LPPSS
PSNRI
NeRF. +D
SSIM
TPPSS]
PSNR|
Mip-NeRE D
SSIMI
TIPPS.
PSNR1
Mip-NeRF 360
SSIM! 1PMPS)
PANNR
Mega NeRF
SSIMI
TIPPS|
PSNR
ReL-N-RFE 2V
SSIMI
T.PSS.
PSNR
SSIM
Ours
LPIPS)
16.51
IL337
0,744
15.35
0.35
0.710
16:34
IL.77
01553
13.91
6.31
(177]
15.12
0.34
0.763
15.00
0.54
0.355
[4.00
0.12
(L7S5
15.31
0.44
0.094
14.59
0.47
0.653
15.16
11.00
0.731
15.06
0.44
0.645
15.04
0.47
0.679
15.90
11.99
0.632
19.63
0.63
0,374
14.55 38
0.28
0.556
14.66
0.29
0.347
14.56
0129
0.535
4.25
0.31
0.535
15:21
0.33
0.317
15.44
0.39
0.525
14.24
0.34
0.515
25.30 30
0.56
0.431
27.47
0.90
0.380
21.75
0.76
0.669
27.68
0.54
0.292
23.36
0.85
0.419
20.85
0.91
0.404
31.74
0.54
0,202
23.30
0.67
0.551
34.32
0.73
0.450
14.9%
11.54
0.635
76
0.51
01317
35.7%
0.26
0.436
23.54
1171
01:01
37.88
6.90
8.162
34.09
0.68
0.504
25.50
0.396
23:18
28.86
0.90
0.77
0.72
0.45
23.09
0.57
0.212
21.14
0.534
22.91
0.509
0,101
14.00
0.34
01744
14.57
0.34
0.31
14.58
0.54
0.466
3.40
0.30
0.705
13.32
0.49
0.675
31.34
0.4
0.40
15.77
0.40
0.361
10
22:36
0.5
0.625
24.37
0.0
0,574
15.80
1153
0.65
36.08
0.72
0.430
21.75
0.62
0.55%
24.23
1158
0.507
25.74
0.53
6.13
13
22
36
032
0.53
24.61
0.85
0.312
22
082
0.425
37.06
0.85
0.317
24.37
0.84
0.392
23.0
0.355
36,29
0.95
6.18
12
22
0.59
0.533
24.29
0.65
0.47
22.13
0.00
0.526
25.12
0.65
0.224
21.50
0.62
0.403
23.05
0.99
0.334
27.02
0.56
8,03
13
0,59
01641
31.52
0.2
0.511
1890
0554
0.75
36
0.77
0.403
35,50
0.73
0.517
21.9
800
0.584
0.14
0,315
0.55
0.509
23.89
0.74
0.177
17.05
0.59
24.42
0.75
21.76
8.0
0.508
38.11
0.127
0.631
15.75
0.34
0.53
0.392
0.610
24.20
0.611
17:01
0.55
0.57
1.59
0.555
6.111
36 06
0.75
0.428
25.57
635
0461
24.1
0.75
0.469
28.55
0.86
0.365
24.76
0.71
0.448
22.70
0.6
0.60
58.00
8.91
0.152
19
14.30
0.40
07756
13.86
633
0300
13:84
11.9
0.75
72
0.37
0.676
23.81
0.6
0.465
1434
0.39
00011
33.55
6:54
8.77
20
22 84
0.1
0.490
23:36
0.64
0475
22:41
0.00
0.519
28.33
0.56
0.225
21.11
0.63
0.400
21.54
855
0.324
26.55
0.51
6.197
21
23.50
0.1
21.34
0:47
0.95
32.51
1151
0.37
35 6l
0.75
0.364
31.92
0.51
0.57%
21007
U07
28/0
6:54
61111
16.53
0.4
0.33
20.66
0.56
0.575
13:37
0.77
24.79
0.77
0.60
0.527
20.31
0.3
0.65
26.30
0.55
0.074
016
0,997
0.52
21.42
0615
24.44
0.54
25.91
0.79
0.63
0.457
1.50
0.56
30.5
0,00
26
3.56
0.34
0.564
16.4
0.59
0.313
13.33
0.55
0666
14.50
046
0.666
19.25
0.67
0.467
21.00
062
0.40
13.88
0,83
6,311
27
4.54
0.01
07155
21.25
0.55
0.56
1432
11:5
BG74
21.36
0.0
0.235
30.59
0.61
0.533
3132
1.53
0.50
11.77
0.66
6364
28
24 68
0.66
0.49
23.26
0.64
0,75
24.6
0.65
0.406
19,62
0.67
0.240
25.87
0.72
0.442
22.17
DA5
0868
26.22
8.1
6.53
29
23.40 91
0.1
0.500
21.17
0.2
0.59
2301
116
0.599
35.51
0.74
0.00)
21.57
0.1
0.587
21.11
834
8611
15.86
0.54
6174
20:23
0.53
0,01
31:37
0.64
0.466
1863
0146
0.75
36.54
0.84
0.796
34,04
0.69
0.4%
3161
854
0586
76.00
6:93
8.9%
18
0.37
0.645
19:05
0.64
1891
3.08
0.23
20955
0.60
0.545
19.18
0.3
0.645
6.90
0.308
0.145
20.19
0.597
26
0.64
0.134
Mean
18.72
0.000
21.45
0.558
0.023
33.10
21.65
0.62
0.588
0.55
0.574
35.10
0.305
Qualitative Results. Qualitative results on the OMMO [13] dataset are shown in Fig. 3
We can
see that the rendering results of NeRF [19] and Mip-NeRF [I] are of the lowest quality. and they use global MLPs for the entire space to reconstruct radiance fields, resulting in a trade-off in the accuracy of sampling the foreground and background which makes them almost impossible to handle large-scale unbounded scenes. NeRF++ [1], Mega-NeRF (28) and Ref ReR-NeRF [29 improve some limitations of NeRF by corresponding techniques, but the rendering results are often missing details, especially when the scene contains lot of intricate details. The rendering quality of MiR-NeRF 360 2 is relatively high, but loses some detail and edges due to its down-scaling of the scene into a limited sampling space. Our method uses the dense point cloud up- sampaed by the diffusion model as a detailed foreground geometry prior combined with Mip-NeRF 360 background features, so our model can reconstruct the fine foreground texture provided by the generative model. At the same
7

---

NeRF
NeRF+
Mip-NeRF
Mip-NeRF 360
Mega-NeRF
Ref-NeRF
Ours
GT
Figure Qualitative results our method with the baselines on the OMMO dataset. Our PDF method outperforms baseline methods with reliably constructed details. For Mip-NeRF and Mega- NeRF, which are also aimed at large scenes, we use yellow dashed boxes to mark some areas that are easy to distinguish the performance of details. Please zoom-in for the best of views.
time, compared with Mip-NeRF 360, method is more robust representation and new view generation without failing scenes (c.f. Fig.
4.3 Ablation Studies
We perform multiple ablation studies to validate the effectiveness of our proposed modules. Tab. shows the impact diffusion point cloud super-resolution module and background feature fusion module the 5-th scene (sydney opera house) from the OMMO dataset [15).

---

Mip-NeRF 360
Ours
GT
Figure 4: A failure scene representation of Mip-NeRF 360.
Table Quantitative performance ablation experiments, including removing both the diffusion- based point cloud up-sampling module and the background fusion module, removing only the diffusion-based point cloud up-sampling module, removing only the background fusion module, our PDF method.
Method
PSNR
SSIM
LPIPS.
w/o diffusion, w/o background
9.28
0.51
0.355
w/ diffusion, w/ background
21.05
0.83
0,219
w/ diffusion, w/o background
22.93
0.78
0.235
Ours
27.58
0.90
0.162
For the ablation experiment on the effectiveness of diffusion, we remove the diffusion-based point cloud up-sampling module and sample directly on the sparse point cloud reconstructed by COLMAP [24] from training views. Since the directly reconstructed point cloud is very sparse and concentrated in the central area, only a very blurry image with large missing blocks can rendered, shown in the first column of Fig. 5 At the same time, quantitative indicators also suggest that this method is not suitable for outdoor unbounded large-scale scenes with its PSNR 9.28.
For the ablation experiment on the effectiveness of background fusion, we remove the background fusion module and render novel view images directly from the diffusion-enhanced point cloud. As shown in the second column of Fig. 5 with the help of the dense point cloud produced by the diffusion module learning the scene distribution, we find that large missing patches have been filled in and produce more refined foreground. However, limited by the characteristics of point cloud expression, the background points are very sparse, which leads to blurred background rendering results. Quantitative results, while substantially improved, still convey poor image quality.
As shown in the third column of Fig. [ using the background fusion module alone can also fill in the missing blocks of the background, but due to the sparseness of the point cloud reconstructed by COLMAP 224, it will lead to the loss of detail and blurring of the rendering result. However, our method, which combines a diffusion module and background fusion module, achieves satisfactory quantitative and qualitative performance and surpasses existing methods.
w/o diffusion w/o Buckground
w/ diffission
wio difflasion background
w/o background
Ours
Figure 5: Qualitative performance of ablation experiments. From left to right: removing both the diffusion-based point cloud up-sampling module and the background fusion module, removing only the background fusion module, removing only the diffusion-based point cloud up-sampling module, our PDF method, and the groundtruth.
We also perform ablation experiments to compare our method with other point cloud ap-sampling methods. With the same experimental setup, we use GAN-based method [40 for point cloud up-sampling instead of the diffusion-based up-sampling module. Tab. shows the quantitative results for three scenes (scan5, scan and sean 12) in the OMMO dataset. Our method exhibits superior performance compared to the GAN-based point cloud up-sampling method, primarily due to its

---

Table 3: Quantitative results of ablation experiments, including removing the diffusion-baat point
cloud up-sampling module, using the GAN-based point cloud up-sampling method, our PDF method.
Method
PSNRT
SSIMT
LPIPS
w/o diffusion
21.85
0.84
0.204
GAN-based method 40
24.83
0.86
0.161
Ours
28.60
0.90
0.137
ability to preserve the structural and topological characteristics of point clouds while effectively handling incomplete or noisy point cloud data. In addition, Fig. showw the visualization results of diffusion-basee point cloud up-sampling module, and our method can not only densify the sparse point cloud reconstructed by the COLMAP, but also fill in the missing regions of the point cloud such as the background and empty space.
(a)
(b)
(c)
(d)
Figure 6: Qualitative evaluation of the diffusion based point cloud super-resolution module. From left to right: (a) Point cloud of large-scale of the same large-scale scene enhanced using our method. (c) Point cloud of a small-scale scene reconstructed using COLMAP. (d) Point cloud of the same small-scale scene enhanced using our method (zoom-in for the best view).
reconstructed using COLMAP. (b) Point cloud
Conclusions and Limitations
In this paper, we propose PDF, a point diffusion implicit function for large-scale scene neural representation, and demonstrate its robustness and fidelity on novel view synthesis tasks. The core of our method is to provide dense point cloud surface priors to reduce the huge sampling space of large-scale scenes. Therefore, point cloud super-resolution module based on diffusion model is proposed to learn from the sparse point cloud surface distribution reconstructed from training views to generate more dense point clouds. However, only constraining the sampling space the point cloud surface does not fully solve the novel view synthesis problem since point clouds do not have background information. So Mip-NeRF 360 [2] is employed to provide background features and synthesize photo-realistic new perspectives. Extensive experiments demonstrate that our method outperforms current methods in both subjective and objective aspects. At the same time, ablation experiments also prove the effectiveness our core module, point up sampling diffusion.
In future work, we will attempt to explore cross-scene point cloud up-sampling generalization diffusion model instead of training diffusion model for each scene to improve efficiency. Even more futuristically, it may be possible to extract representative scene representations and inject them into reconstructed point clouds to achieve cross-scene rendering, i.c., generalized point diffusion NeRF,
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 62101 137, 62071 and U1909207), Shanghai Natural Science Foundation (No.23ZR 1402900) and Zhejiang Lab Project KO0.2021K
10

---

References
[1] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3655-5844, 2021.
[2] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[3] Eric Chan, Connor Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient gcometry-aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123 16133, 2022.
[4] Tianlong Chen, Peihao Wang, Zhiwen Fan, and Zhangyang Wang. Aug-nerf: Training stronger neural radiance fields with triple-level physically augmentations. In Proceedings of the IELE/CVF Conference on Computer Vision and Pattern Recognition, pages 15191-15202, 2022.
[5] Nianchen Deng, Zhenyi He, Jiannan Ye. Budmonde Duinkharjav, Praneeth Chakravarthula, Xubo Yang, and Qi Sun. Fov-nerf: Foveated neural radiance fields for virtual reality. IEEE Transactions on Visualization and Computer Graphics, 28(1 1):3854-3864, 2022.
[6] Yuan- Guo, Di Kang, Linchao Bao, Yu He, and Song- Hai Zhang. Nerfren: Neural radiance fields with reflections. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18409-18418, 2022.
[7] Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5784-5794, 2021.
[8] Xin Huang, Qi Zhang, Ying Feng, Hongdong Li, Xuan Wang, and Qing Wang. Hdr-nerf: High dynamic range neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18398-18408, 2022.
[9] Yi-Hua Huang, Yue Hc, Yu-Jie Yuan, Yu-Kun Lai, and Lin Gao. Stylizednerf: consistent 3d scene stylization as stylized nerf via 2d-3d mutual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18342-18352, 2022.
[10] Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, pages 18365-1888 18375, 2022.
[11] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3907-3916 2018.
[12] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via fcature field distillation. Advances in Neural Information Processing Systems, 35:23311-23333 2022.
[13] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 4340-4350, 2023.
[14] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6558,, 2021.
[15] Chongshan Lu, Fukun Yin, Xin Chen, Wen Liu, Tao Chen, Gang Yu, and Jiayuan Fan. A large-scale outdoor modal dataset and benehmark for novel view synthesis and implicit scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7557-7567, 2023.
[16] Baorui Ma, Yu-Shen Liu, and Zhizhong Han. Reconstructing surfaces for sparse point clouds with on- surface priors. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6315-6325, 2022.
[17] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2219, 2021.

---

[18] Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualle Pratul P Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16190-16199, 2022.
[19] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 6561)999-106, 2021.
[20] Benjamin Joseph Mildenhall. Neural Scene Reprexentations for View Synthesis. University of California, Berkeley, 2020.
[21] Michael Occhslc, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589-5558 5599, 2021.
[22] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Pruceedings of the IEEE/CVE conference on compuder vision and pattern recognition, pages 165-174, 2019.
[23] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese -ooeur-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IFEE/CVE Conference on Computer Vision and Pattern Recognition, pages 10315 10318-10327, 2021.
[24] Johannes L. Schonberger and Jan-Michael Frahm. Structureffoom-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4 41044113, 13, 2016.
[25] Bhin Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin, A-nerf: Articulated neural radiance fields for learning human shape, appcarance, and pose. Advances in Neural Information Processing Systems, 34:12278- 112227-12291, 2021.
[26] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8248-8258 2022.
[27] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, Wang Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Bruall Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[28] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 12922 12922-12931, 2022.
[29] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
IEEE, 2022.
[30] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. Clip-nerf: Text-and-image driven manipulation of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 3835-3844, 2022.
[31] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv: 106. 10689, 2021.
[32] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 13(4):600-612, 2004.
[33] Yuanbo Xiangli. Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In European conference on computer vision, pages 106-122. Springer, 2022.
[34] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Point-hased neural radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 5438-5448, 2022.
[35] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1790-1799 2020.
12

---

[36] Lior Yariv. Jiatao Gu. Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 44:4805-4815, 2021.
[37] Fukun Yin, Wen Liu, Zilong Huang, Pei Cheng, Tao Chen, and Gang Yu. Coordinates are not lonely- codebook prior helps implicit neural 3d representations. Advances in Neural Information Processing Systems, 35:12705-12711 2022.
[38] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578 4587, 2021.
[39] Yu-Jie Yuan, Yang- Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedingy of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 98353-18364, 2022.
[40] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai. and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 1768-1777, 2021.
[41] Kai Zhang, Geroot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2018.
[43] Linqi Zhou, Yilun Du and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 3826-5835 2021.
13


Reviews:
[{"review_id": "iMxYPAEqe7", "reviewer": "Reviewer_yNkC", "summary": "The paper proposed a novel method for the novel view synthesis of large-scale outdoor scenes. The method combines a Point-NeRF-baed rendering stage for the foreground, as well as a background stage based on Mip-NeRF 360. To cope with the issue of overly sparse point cloud from COLMAP MVS, a point could diffusion model is trained on each individual scene from existing MVS point cloud, and used to synthesize more points. The proposed method achieves leading performance on both large and small scenes.", "questions": "* Could you elaborate more on the reason behind the failure of Mip-NeRF 360 in Figure 4?\n* Why is the background partially missing in Fig. 5 column 1?", "limitations": "Although there is a Limitations section, it only includes future works and there lacked discussions on the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Training a scene-specific diffusion model for point cloud upsampling is a novel approach that seems to work well.\n* The proposed method is able to achieve better quality on a large set of real-world scenes both quantitatively and qualitatively.", "weaknesses": "* The proposed method requires a diffusion model to be trained on each scene, which can require significant computation.\n* In terms of comparison with previous works, it is not sure if the comparison is based on a level ground, as the proposed method might require more computation, more parameter count, or both.\n* It is not sure if the proposed method is able to outperform methods that are based on grid data structure, such as InstantNGP, Plenoxels, etc., which tend to be very fast and scalable to large, sparse and unbounded scenes.\n* It is not sure if other ways of upsampling the point clouds will also work. Although the use of a scene-specific diffusion model to upsample point could is novel, it is not compared against any baselines. It might provide better insight and more helpful for future work if this contribution can be studied independently as a separate work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the final version of our paper will be updated accordingly. \n\n📝 **Q: The significant computation caused by per-scene optimization.**\n \n💡&#8194;**A:** As your concern, pre-scene optimization is currently a prevalent approach for constructing implicit neural representations. Although there have been some advancements in NeRF generalization methods, they are still limited to representing toy or mini-abstract scenes. This limitation arises from the inherent difficulty in establishing a universal and effective representation for complex scenes or objects, particularly for unbounded outdoor scenes that encompass thousands of objects across numerous categories within each scene. Consequently, this paper adopts the per-scene optimization strategy to construct dense point clouds and neural implicit fields for each scene, thereby facilitating the simultaneous learning of representations for multiple object categories within the scene.\n\n📝 **Q: Performance improvement due to large model size or design.**\n\n💡&#8194;**A:**  To validate the improvement originating from our proposed point cloud upsampling module versus additional parameters, we expanded the MLP layers of PointNeRF and Mip-NeRF360 to roughly match the parameter count of our approach. The following table shows the experimental results. Directly increasing the parameter count does not lead to additional performance improvements, as it can make the network more challenging to optimize and prone to overfitting on the training viewpoints (Due to time constraints, this experiment is exclusively validated on scan5). \n|Method |Params | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|:-:|\n|Enlarge PointNeRF| 38.19M |19.92| 0.77 |0.362|\n|Enlarge Mip-NeRF360| 36.10M |22.01| 0.77 |0.322|\n|**Ours**| 38.68M| **27.58** |**0.90** |**0.162**|\n\n📝 **Q: Additional point cloud upsampling approaches.**\n\n💡&#8194;**A:**  We appreciate your insightful suggestion. In conjunction with the recommendation from Reviewer H9co, we have replaced our diffusion-based method with a Generative Adversarial Network (GAN)-based point cloud up-sampling approach[1].  As evident from the table below, our method exhibits superior performance compared to the GAN-based point cloud upsampling method, primarily due to its ability to preserve the structural and topological characteristics of point clouds while effectively handling incomplete or noisy point cloud data (Due to time constraints, this experiment is exclusively validated on scan5, scan11 and scan12).\n|Method | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|GAN-based method| 24.83| 0.86 |0.161|\n|**Ours**| **28.60** |**0.90** |**0.137**|\n\n\n📝 **Q: An analysis of the causes of the bad case in Mip-NeRF 360.** \n\n💡&#8194;**A:** As illustrated in Figure 4, MipNeRF 360 encountered failure in an outdoor jungle scene characterized by a substantial amount of repetitive textures and similar details. We visualized the camera trajectory provided by the scene and cross-referenced it with the original video, revealing an erroneous calibration of a specific viewpoint (as depicted in Figure 12 in the attached PDF), which likely had a significant impact on the performance of Mip-NeRF 360. In contrast, our approach exhibits enhanced robustness by reconstructing and enhancing the scene point cloud, providing a visual representation that incorporates prior knowledge to counteract partially erroneous data.\n\n📝 **Q: The reasons for the background missing when removing the diffusion-based point cloud up-sampling module.** \n\n💡&#8194;**A:** We apologize for any confusion. Our intention with the ablation study was to create an additive chain, starting from the original multi-view reconstruction point cloud and progressively incorporating our diffusion super-resolution module and background module. Therefore, 'w/o diffusion' actually implies 'w/o diffusion and background'. In our work with city-scale outdoor scenes, the point clouds obtained from multi-view reconstruction methods are extremely sparse, particularly in boundary-less background areas, leading to the absence of background in the rendered images. We will rectify this in the final version and update the results of the 'w/o diffusion' experiment(Due to time constraints, this experiment is exclusively validated on scan5).\n\n|Method | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|w/o diffusion, w/ background| 21.05| 0.83 |0.219|\n|**Ours**| **27.58** |**0.90**|**0.162**|\n\n📝 **Q: Comparison with methods based on grid data.**\n\n💡&#8194;**A:** In contrast to nerf, InstantNGP uses a sparse parameterized voxel grid instead of mlp for scene representation, which compresses the training time from hours to minutes or even seconds. To explore the performance of this method on unbounded large-scale scenes, we trained InstantNGP on three scenes from the OMMO dataset. With a resolution setting of 640*320, after 10 seconds of training, the PSNR of the rendered images averages about 25.69. Qualitative results(as depicted in Figure 11 in the attached PDF) show that although InstantNGP can perform 3D scene modeling very quickly, it is hard to render the details of foreground objects and the background is often completely blurred.\n\n[1] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In CVPR, 2021."}, {"review_id": "CTjfG3RJsP", "reviewer": "Reviewer_DZNn", "summary": "In this paper, the authors propose a new approach to tackle the reconstruction of Neural Radiance Fields from large, unbounded scenes. A major limitation of current neural fields is the lack of scalability due to the number of sampling points in empty space, which is limiting the achievable scale or quality with limited compute resources. In this method, the sampling space is restricted to the surface of the object. To get an approximation of the surface of the scene, they utilize a point cloud, which first is reconstructed through MVS and upscaled to achieve a dense point cloud on the surface using a diffusion probabilistic model. Only sampling points in the near distance of the surface point cloud and the respective feature, which is constructed as a predicted feature from the neighborhood points and sampling points and forms the foreground feature together with the constructed feature. The unbounded background of the scene is modeled on a sphere as features. Foreground and background features are fused in another MLP.", "questions": "- Can you provide the training and rendering times for a given resolution for your and all reference methods\n- The authors mentioned that the method was only evaluated on a subset provided by the authors of OMMO, but the full dataset results in the supplementary provide similar insights. ", "limitations": "Most limitations are mentioned at the end of the paper. As discussed in the weakness section, I think an evaluation of limitation wrt scene size and computational times in a similar setting, would provide a better understanding of the capabilities of this methos.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presented method builds on known techniques for neural rendering and generative super-resolution and combines those to solve a novel task of reconstructing large-scale unbounded scenes. I think this is a big strength of the paper, which is justified by qualitative and quantitative results and ablation studies.\n- The training approach shows a clear explanation of how the missing ground truth data for the point cloud can be optimized with the limited resources \n- The authors present their method in an understandable way, adding the necessary context and references to reproduce the work \n- Presented results show a clear advantage over standard NeRF techniques, and better results in areas of fine textures compared to methods designed for large-scale or unbounded scene\n- Most design choices are ablated and justified in separate studies ", "weaknesses": "- One major weakness of the proposed method is that it is required to retrain the whole model per scene. This is also mentioned by the authors in their limitations but leads to really long training times compared to sota methods.\n- While the result section presents all relevant quantitative results in a table, the authors do not provide an explanation for some of the outliers and rather good performances of the baselines in some scenes.  \n- I hope I have not missed something, but there is no explanation of Fig.4. To my understanding, this provides a scene in which Mip-NeRF 360 fails completely to reconstruct the scene, but no explanation is given. Also, I am curious why it results in something like this and not only blurry results as reported in Fig.3 \n- One major limitation or experiment important for the specific task of large-scale, outdoor scene reconstruction is to show the boundaries of the method in terms of scene scale. Mega-NeRF is specially designed to divide a scene in multiple sections and might be able to handle larger scenes, which is not the case for this method. An additional study on the limitation of this method in terms of scene size alongside with training and rendering times would probably just strengthen the claims", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the final version of our paper will be updated accordingly. \n\n📝 **Q: long training times caused by per-scene optimization.**\n\n💡&#8194;**A:** Due to the challenge of establishing effective and generalizable representations for large-scale scenes containing a multitude of object categories, the common practice is to optimize on a per-scene basis. In comparison to existing state-of-the-art (SOTA) methods, our rendering time is acceptable, thanks to the utilization of a dense point cloud prior that reduces rendering time. A comparison of training times with SOTA methods is presented in the table below and will be included in our final version. Additionally, in future work, we aim to explore the utilization of a cross-scene point cloud upsampling generalization using a diffusion model, instead of training a diffusion model for each scene, to enhance efficiency.\n|Method | Nerf |Nerf++ |Mip-Nerf|Mip-Nerf 360|Point-Nerf|Ours|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Training time(h)| 7.2 | 9.5 |11.2|9.1|8.0|38.0|\n|Tendering time(s)| 3.4 | 5.2 |8.7|10.3|6.3|12.0|\n\n\n📝 **Q: Explanation for some of the outliers and performances.**\n\n💡&#8194;**A:** Thank you for your suggestion. Our proposed method does not have any bad cases, but it may exhibit suboptimal performance in scenes with relatively flat spatial structures, such as grasslands (scan6, scan17), roads (scan1), and plazas (scan9). This is because the reconstructed point cloud is approximately planar and lacks sufficient geometric priors present in other scenes. Similarly, NeRF tends to fail in unbounded scenes (scan1, scan27, scan33) and scenes with abundant reflective surfaces (scan26), which is similar to Mip-NeRF (scan22, scan32, scan33, scan26). NeRF++, Mega-NeRF, and Ref-NeRF are relatively robust methods, but they may produce relatively blurry renderings for large-scale urban scenes. Mip-NeRF 360 requires high data robustness, as less than 10% of abnormal camera poses can lead to the failure of the entire scene (scan16). \n\n📝 **Q: An analysis of the causes of the bad case in Mip-NeRF 360.**\n\n💡&#8194;**A:** As illustrated in Figure 4, mipnerf360 encountered failure in an outdoor jungle scene characterized by a substantial amount of repetitive textures and similar details. We visualized the camera trajectory provided by the scene and cross-referenced it with the original video, revealing an erroneous calibration of a specific viewpoint (as depicted in Figure 12 in the attached PDF), which likely had a significant impact on the performance of Mip-NeRF 360. In contrast, our approach exhibits enhanced robustness by reconstructing and enhancing the scene point cloud, providing a visual representation that incorporates prior knowledge to counteract partially erroneous data.\n\n📝 **Q: The method's upper limit on scene scale.**\n\n💡&#8194;**A:** Thank you for your suggestions. According to the OMMO dataset, their scene areas range from 2km2 to 1000 km2. Mega-NeRF divides a large scene into multiple smaller scenes to accelerate and reduce the fitting difficulty of each NeRF unit. However, for city-scale scenes, each subpart remains too large to generate highly detailed textures. We did not adopt this approach, considering that partitioning would disrupt the integrity of the geometric structure, which is unfavorable for the point cloud super-resolution module to learn the underlying structure of the entire scene.\n\n📝 **Q: dataset issues.**\n\n💡&#8194;**A:** We sincerely apologize for the confusion. Due to the initial release of only a representative subset of the OMMO dataset, we obtained the complete data shortly before the submission deadline. As a result, the additional scene performance was included in the supplementary materials. In the final version, these two parts will be merged to report the overall results on the entire OMMO dataset. \n\n\n"}, {"review_id": "vU1rI9QfQD", "reviewer": "Reviewer_rzgR", "summary": "This paper proposes an implicit neural representation for large-scale scenes with two major components, the first one is the point diffusion implicit function (PDF) which adopts diffusion process to generate dense point cloud from point cloud produced by Colmap, the second one is the background rendering which basically borrow the idea from Mip-Nerf360.", "questions": "•\tHow was the diffusion process necessary for large-scale scene representation, I do see an ablation study about that module, but I am not sure why removing the diffusion make the method totally failed but the PointNerf can works well with pointcloud directly from COLMAP. More explanation or visualization will be very helpful. \n•\tWhat’s the runtime of the proposed methods.\n•\tA general (but maybe not that related to the proposed method): Is there any other solutions to the background rendering? Will the environment map work for the background of the large outdoor scene. \n", "limitations": "The authors have discussed potential negative social impacts of this paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "•\tThe paper is well organized and easy to follow.\n•\tThe qualitative comparison on the fly-view dataset clearly shows the effectiveness of the proposed method on the large-scale scenes, where the rendered images are sharp in both foreground and background.\n•\tMost of nerf paper focus on small or medium scene, this paper instead provides a interesting solution to the large scale scene representation.", "weaknesses": "•\tThe effectiveness of the diffusion process, the motivation behind adding the diffusion process over colmap generated point cloud is not clear to me since the PointNerf method can generate clean and sharp foreground rendering directly with the colmap point cloud. Some explanation or visualization showing the point cloud before and after diffusion will be very helpful. \n•\tOne major advantages of point-cloud based neural field is they can avoid prohibitive reconstruction time of Nerf, so a question what is the training time and rendering speed of this paper. Will the diffusion process significantly slow down the training speed?\n•\tThe overall technical contribution of this paper is weak. The main component PDF shares the major design with PointNerf except applying a diffusion process over the Colmap-generated point cloud. Currently there is no enough discussion over the insight behind this design as I said before. The background rendering module also follows the existing technique from Mip-Nerf 360. Therefore I think the contribution is not enough.\n•\tIn the experiment part the author said they use two dataset for evaluation but I didn’t find the result on BlendMVS\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the final version of our paper will be updated accordingly. \n\n📝 **Q: The combination of novel diffusion-based point cloud upsampling models and Point-NeRF-based NeRF models is relatively weak.** \n\n💡&#8194;**A:** For unbounded, large-scale outdoor scenes, Point-NeRF-based NeRF models are unable to accurately represent and synthesize fine-grained textures due to the vast sampling space and excessively sparse point clouds, as seen in the first column of Figure 5. To overcome this critical issue, we propose a novel diffusion-based point cloud upsampling module that generates dense scene surfaces. By incorporating an explicit surface prior, we reduce the sampling space of the Point-NeRF-based NeRF model from an unbounded 3D urban-level scale to the scene surface. This effective combination of explicit and implicit representations is the key to the success of our method and provides an effective solution for large outdoor scenes. Furthermore, the effectiveness of large-scale point cloud upsampling is difficult to measure through visualization due to the high degree of object occlusion and overlap. Neural rendering provides a solution for evaluating this research.\n\n📝 **Q: The results on the BlendMVS dataset.**\n\n💡&#8194;**A:** We sincerely apologize for the confusion. Due to limitations in space and time, we have included the experimental results of the BlendMVS dataset in the supplementary materials. For further details, please refer to Table 4 and Figure 6 in the supplementary materials, which will be incorporated into the final version of the manuscript.\n\n📝 **Q: The necessity of the diffusion module.**\n\n💡&#8194;**A:** For outdoor urban-level scenes, the point cloud reconstructed using COLMAP tends to be sparse, especially in unbounded background regions, resulting in blurry foreground and missing background in rendered images. To demonstrate this, we provide quantitative results of applying Point-Nerf to the unbounded large-scale OMMO dataset in the following table. It can be observed that Point-Nerf is not directly applicable to large scenes. Additionally, we present visualization results as shown in Figure 9, where severe occlusions in outdoor large-scale scenes cause the loss of most geometric details when projecting the point cloud onto 2D. To facilitate performance observation, we conducted experiments on the hotdog scene in the NeRF-Synthetic dataset. It can be observed that our method not only enhances the geometric structure of the foreground but also fills in the missing parts.\n\n|Method | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|Point-Nerf| 17.85| 0.57 |0.320|\n|**Ours**| **25.10** |**0.79**|**0.205**|\n\n📝 **Q: What's the runtime of the proposed methods.** \n\n💡&#8194;**A:** We evaluate the training and rendering time of our method on the 5th scene (Sydney Opera House) from the OMMO dataset. At a resolution setting of 1280×676, our method takes about 38 hours to train and 12 seconds to render on a single Nvidia A100 GPU device. \n\n📝 **Q: Additional solutions for background rendering.** \n\n💡&#8194;**A:** Foreground-background modeling, originating from NeRF++, is a commonly used method for scene neural implicit representation. This approach includes subsequent works such as Neurs, VolSDF, CoCo-INR, among others. Additionally, environmental texturing is a prevalent technique, often applied to model individual objects. However, for scenes, the background is often an integral component, making the use of foreground-background modeling more prevalent.\n\n\n\n"}, {"review_id": "iVADiPQGPY", "reviewer": "Reviewer_H9co", "summary": "The paper presents a method for reconstructing large-scale scenes from multi-view images. Given the sparse point cloud computed from an MVS pipeline, the algorithm first employs a point cloud diffusion model to upsample the point cloud. The utilization of the diffusion model could effectively create a dense cloud with missing regions completed. Then, a Point-NeRF-style model is used to reconstruct the scene based on the upsampled point. Foreground and background renderings are fused via an additional fusion network, resulting in the final output. The algorithm is tested on OMMO and BlendMVS datasets. Compared to existing NeRF baselines, the proposed pipeline could limit the sampling region to the surface and achieves better reconstruction and rendering quality.", "questions": "The main question arises from the training strategy as well as the model specifications of the diffusion model:\n1. What is the model architecture of the denoiser? How many parameters are needed?\n2. Which dataset is used to train the diffusion model? How many training samples are there? Is data augmentation applied?\n3. How many points are generated from the diffusion model?\n4. What is the mechanism that allows the diffusion model to complete missing regions (as shown in Fig. 5)? Are regions randomly masked out during the training stage to encourage the model to hallucinate missing contents?", "limitations": "Not applicable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Using the diffusion model for point cloud upsampling and completion is novel. While previous methods perform point cloud upsampling or completion using VAE or GAN models, this is the first paper that leverages the power of diffusion models. One related work could be Luo et al. [a], yet the proposed algorithm treats the partial input cloud as a condition and hence is also able to generate an arbitrary number of points.\n2. The results are compelling. Using the upsampled point cloud as the supporting domain of NeRF, the method could reconstruct a much larger scene with finer details. Both the qualitative and quantitative experiments demonstrate the effectiveness of the proposed approach.\n\n[a] Luo, Shitong, and Wei Hu. \"Diffusion probabilistic models for 3d point cloud generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "weaknesses": "1. While the use of the diffusion model for point cloud upsampling is novel, its combination with the subsequent NeRF model is very straightforward and detached. The NeRF model is only a slight modification from Point-NeRF, with the addition of the background MLP, and such a representation itself is not insightful enough to form an individual contribution.\n2. Some descriptions regarding the method section and experiment section are not very clear. What is the training scheme of the diffusion model? (more details in the 'Question' section). What does '... as a feature $f_i$ of each sampling point $p_i$ to equip structural information' in Line 172 mean? \n3. Ablation study using other point cloud upsampling methods is missing. Given the main contribution of the paper is the diffusion-based upsampling network, it is worth investigating if a diffusion-based backbone could surpass a traditional GAN-based backbone. The study could be conducted on a simple toy dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the final version of our paper will be updated accordingly. \n\n📝 **Q: The combination of novel diffusion-based point cloud upsampling models and Point-NeRF-based NeRF models is relatively weak.** \n\n💡&#8194;**A:** For unbounded, large-scale outdoor scenes, Point-NeRF-based NeRF models are unable to accurately represent and synthesize fine-grained textures due to the vast sampling space and excessively sparse point clouds, as seen in the first column of Figure 5. To overcome this critical issue, we propose a novel diffusion-based point cloud upsampling module that generates dense scene surfaces. By incorporating an explicit surface prior, we reduce the sampling space of the Point-NeRF-based NeRF model from an unbounded 3D urban-level scale to the scene surface. This effective combination of explicit and implicit representations is the key to the success of our method and provides an effective solution for large outdoor scenes. Furthermore, the effectiveness of large-scale point cloud upsampling is difficult to measure through visualization due to the high degree of object occlusion and overlap. Neural rendering provides a solution for evaluating this research.\n\n📝 **Q: GAN-based point cloud upsampling approaches.**\n\n💡&#8194;**A:**  We appreciate your insightful suggestion. In conjunction with the recommendation from Reviewer H9co, we have replaced our diffusion-based method with a Generative Adversarial Network (GAN)-based point cloud up-sampling approach[1]. \n\n📝 **Q: What is the training scheme of the diffusion model?**\n\n💡&#8194;**A:**  The structure of our diffusion model is referenced to PVD[2], which has a number of parameters of about 27.6 M. We use the sparse point cloud reconstructed by COLMAP as input to train the diffusion model. Our method requires a diffusion model to be trained on each scene, the number of training point cloud samples varies from 40,000 to 100,000 for different scenes. We didn't use data augmentation. The diffusion model generates 1848 points per round based on 200 prior points, and a total of 200 rounds are performed, i.e., 369600 points are generated. For missing regions completions specifically, the model takes as input 200-point partial points and 1,848 points sampled from noise, totaling 2048 points. At each step, the first 200 of the 2,048 points sampled by the model are replaced with the input partial points. The updated point set is then used as input in the next time step. We didn't use the random masking region strategy.\n\n[1] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In CVPR, 2021.   \n\n[2] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 5826–5835,October 2021.\n"}, {"review_id": "9D1kDXMPQL", "reviewer": "Reviewer_v7wN", "summary": "This paper proposed a point cloud-based representation for neural rendering of large-scale scenes.  Point cloud diffusion model is designed to upsample the point cloud for better performance. The proposed method achieves state-of-the-art performance in the tested scenes. However, I still have some concerns about the implementation details and the performance comparison.", "questions": "Please refer to Weaknesses.", "limitations": "Please refer to Weaknesses (especially 2. and 6.).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves state-of-the-art performance in the neural rendering of large-scale scenes.\n2. Using diffusion models for point cloud upsampling is interesting and useful in the tested scenes.\n3. The presentation and writing of the paper is good and the visualization comparison is clear.", "weaknesses": "1. Size of the train dataset for diffusion. Diffusion model typically requires large-scale dataset for better performance. I'm curious about the number of training point cloud samples for diffusion model.\n2. Reconstruction quality. The 3D reconstruction quality of MVS methods like COLMAP is not guaranteed and there can exist large holes in texture-less regions (e.g., white walls in outdoor scenarios.) I wonder the performance of the diffusion model in such situations (as the diffusion model in the manuscript is mainly designed for upsampling, not completion).\n3. Comparison with Point-NeRF. Point-NeRF is an obvious baseline for this method, which should be included in the experiments. The ablations in the experiments can be seen as a variantion of Point-NeRF. However, Point-NeRF include a point grown and pruning strategy, which can also densify the point cloud. It would be better to test the performance of this strategy in large-scale scenes, which has not been explored before.\n4. Inference time. The proposed method exploit an explicit reconstruction of the scene, thus the empty space can be simply skipped for better efficiency. However, the complexity of the proposed method is not reported and some implementation details are missed (e.g., the number of sampled points)\n5. Background features. Mip-NeRF 360 is utilized for background rendering. However, as shown in Fig.5, the background also contributes to fore-ground regions. It would be better to include an explanation for this.\n6. Others. Diffusion models are widely used for generative models and their exsiting randomness in the generated results. However, the 3D reconstruction/the neural rendering should be deterministic. I wonder the influence of the randomness in this method. Besides, I'm also curious about the failure cases of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your approval of our idea and the detailed and insightful comments. Your concerns will be addressed in the following comments and the final version of our paper will be updated accordingly. \n\n📝 **Q: Size of the train dataset for diffusion.**\n\n💡&#8194;**A:**  Our method requires a diffusion model to be trained on each scene, the number of training point cloud samples varies from 40,000 to 100,000 for different scenes.\n\n📝 **Q: Reconstruction quality.**\n\n💡&#8194;**A:**  As you have expressed concerns, it is evident that there are missing parts in the point cloud reconstructed by COLMAP, as shown in Figure 9. During the diffusion training process, our training data pairs are obtained by subsampling the reconstructed point cloud twice, transforming a sparser point cloud into a denser one. Since the subsampling rate can approach zero, the training process encompasses cases of completing structural gaps. Therefore, our method is capable of addressing such situations, as depicted in Figure 9.  \n\n📝 **Q: Comparison with Point-NeRF.**\n\n💡&#8194;**A:** Thank you for your suggestion. Point-NeRF employs a point growing and pruning strategy to generate a surface point cloud sufficient for rendering in small-scale scenes. However, this approach falls short when dealing with sparse and structurally complex point clouds in large-scale scenes. We present the results of Point-NeRF on the OMMO dataset, as shown in the table below.\n\n|Method | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|Point-Nerf| 17.85| 0.57 |0.320|\n|**Ours**| **25.10** |**0.79**|**0.205**|\n\n📝 **Q: Inference time and implementation details.**\n\n💡&#8194;**A:** Our method avoids ray sampling in the empty scene space by leveraging point clouds, so the inference time will be relatively reduced. We evaluate the training and rendering time of our method on the 5th scene (Sydney Opera House) from the OMMO dataset. At a resolution setting of 1280×676, our method takes about 38 hours to train and 12 seconds to render on a single Nvidia A100 GPU device. Referring to the implementation details of Point-NeRF, we set the number of sampling points on the sampling ray of each pixel to 80, and the features of each sampling point are aggregated from the features of 8 neural points around it.\n\n\n📝 **Q: Reasons for the facilitative impact of the background on the foreground.**\n\n💡&#8194;**A:**  During the training process, the foreground and background are jointly optimized. If the background exhibits poor performance, it can lead to significant losses and adversely affect the optimization of the foreground.\n\n\n📝 **Q: the influence of the randomness.**\n\n💡&#8194;**A:**  In order to explore the influence of this randomness on our method, we conducted experiments on three scenes(scan5, scan11, scan12) from the OMMO dataset. We set three different random number seeds for each scene, and the quantitative results are shown in  the table below. The metrics show that the impact of different random number seeds on rendering quality is almost negligible. In addition, our method has no failure cases on the OMMO dataset,  Figure 10 in the attached PDF shows the rendering results of all scenes.\n\n|Seed | PSNR↑ |SSIM↑ |LPIPS↓|\n|:-:|:-:|:-:|:-:|\n|2| 28.60 | **0.90** |**0.136**|\n|4| **28.63** | **0.90** |**0.136**|\n|8| 28.42 | 0.89 |0.140 |\n|**Ours**| 28.60 |**0.90** |0.137|\n\n"}]

Author rebuttal:
We thank all reviewers for their constructive feedback and recognizing our method as **achieving better qualitative and quantitative performance** (yNkC, DZNn, rzgR, H9co, v7wN); **introducing a novel and effective diffusion-based up-sampling module for large-scale point clouds** (yNkC, DZNn, H9co, v7wN); **capable of generating detailed texture areas** (DZNn, rzgR, H9co, v7wN); and **being well-organized and easy to understand** (DZNn, rzgR, v7wN). We will address their concerns below.


Meta-review: NVS method. Innovation is per-scene diffusion trained on points used for inpainting.

Downside is, that the comparison is not as substantiated as several R would like it to be. Something much simpler for upsampling might work, too. A revised paper will have to admit this. R criticize also long per-scene train, but AC sees this only as a practical issue.

Rebuttal gave some answers to specific questions and some new experimental evidence. Could not sway ``yNkC``. One R was happy with the authors comments and increased their score. Some negative R's did not take part in the decision. One R was so kind and honest to admit that they only understand the method after the author response and changed to 'accept'.

The AC suggests the paper admits the long training time as a limitation and that it is more about the idea than about state-of-the-art performance.

Overall, vibes are positive and on a positive trajectory after the author response: Accept.
Decision: Accept (poster)