SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Quantitative Results. Quantitative comparisons on the OMMO [15] dataset are shown in Tab. 1, including the mean PSNR, SSIM, and LPIPS. We outperform other methods on all average evaluation metrics, especially LPIPS, a perceptual metric close to the human visual system, which is significantly more sensitive to the foreground than the background. So better LPIPS metric indicates that our model can better reconstruct the foreground of the scene, benefiting from sampling the foreground from the reconstructed dense point cloud surface instead of the entire sampling space.
NeRF [19], NeRF++ [41], Mip-NeRF [1], and Ref-NeRF [29] are not specially designed for large-scale scenes, so directly applying them to large scenes will lead to performance degradation. Mip-NeRF 360 [2] and Mega-NeRF [28] have achieved the optimal performance in one or several scenes by sampling regions in the limited sampling space or subdividing the sampling space. But it is still not as good as ours in most scenes due to the loss of detail caused by compression or decomposing the sampling space.
Qualitative Results. Qualitative results on the OMMO [15] dataset are shown in Fig. 3. We can see that the rendering results of NeRF [19] and Mip-NeRF [1] are of the lowest quality, and they use global MLPs for the entire space to reconstruct radiance fields, resulting in a trade-off in the accuracy of sampling the foreground and background which makes them almost impossible to handle large-scale unbounded scenes.  $\mathrm{NeRF} + +$  [41], Mega-NeRF [28] and Ref-NeRF [29] improve some limitations of NeRF by corresponding techniques, but the rendering results are often missing details, especially when the scene contains a lot of intricate details. The rendering quality of Mip-NeRF 360 [2] is relatively high, but loses some detail and edges due to its down-scaling of the scene into a limited sampling space. Our method uses the dense point cloud up-sampled by the diffusion model as a detailed foreground geometry prior combined with Mip-NeRF 360 background features, so our model can reconstruct the fine foreground texture provided by the generative model. At the same
NeRF
NeRF++
Mip-NeRF
Mip-NeRF 360
Mega-NeRF
Ref-NeRF
Ours
GT
Figure 3: Qualitative results of our method with the baselines on the OMMO dataset. Our PDF method outperforms baseline methods with reliably constructed details. For Mip-NeRF and MegaNeRF, which are also aimed at large scenes, we use yellow dashed boxes to mark some areas that are easy to distinguish the performance of details. Please zoom-in for the best of views.
time, compared with Mip-NeRF 360, our method is more robust to scene representation and new view generation without failing scenes (c.f. Fig. 4).
We perform multiple ablation studies to validate the effectiveness of our proposed modules. Tab. 2 shows the impact of diffusion point cloud super-resolution module and background feature fusion module on the 5-th scene (sydney opera house) from the OMMO dataset [15].
Figure 4: A failure scene representation of Mip-NeRF 360.
Table 2: Quantitative performance of ablation experiments, including removing both the diffusion-based point cloud up-sampling module and the background fusion module, removing only the diffusion-based point cloud up-sampling module, removing only the background fusion module, our PDF method.
Method	PSNR↑	SSIM↑	LPIPS↓
w/o diffusion, w/o background	9.28	0.51	0.355
w/o diffusion, w/ background	21.05	0.83	0.219
w/ diffusion, w/o background	22.93	0.78	0.235
Ours	27.58	0.90	0.162
For the ablation experiment on the effectiveness of diffusion, we remove the diffusion-based point cloud up-sampling module and sample directly on the sparse point cloud reconstructed by COLMAP [24] from training views. Since the directly reconstructed point cloud is very sparse and concentrated in the central area, only a very blurry image with large missing blocks can be rendered, as shown in the first column of Fig. 5. At the same time, quantitative indicators also suggest that this method is not suitable for outdoor unbounded large-scale scenes with its PSNR of 9.28.
For the ablation experiment on the effectiveness of background fusion, we remove the background fusion module and render novel view images directly from the diffusion-enhanced point cloud. As shown in the second column of Fig. 5, with the help of the dense point cloud produced by the diffusion module learning the scene distribution, we find that large missing patches have been filled in and produce a more refined foreground. However, limited by the characteristics of point cloud expression, the background points are very sparse, which leads to blurred background rendering results. Quantitative results, while substantially improved, still convey poor image quality.
As shown in the third column of Fig. 5, using the background fusion module alone can also fill in the missing blocks of the background, but due to the sparseness of the point cloud reconstructed by COLMAP [24], it will lead to the loss of detail and blurring of the rendering result. However, our method, which combines a diffusion module and a background fusion module, achieves satisfactory quantitative and qualitative performance and surpasses existing methods.
Figure 5: Qualitative performance of ablation experiments. From left to right: removing both the diffusion-based point cloud up-sampling module and the background fusion module, removing only the background fusion module, removing only the diffusion-based point cloud up-sampling module, our PDF method, and the groundtruth.
We also perform ablation experiments to compare our method with other point cloud up-sampling methods. With the same experimental setup, we use a GAN-based method [40] for point cloud up-sampling instead of the diffusion-based up-sampling module. Tab. 3 shows the quantitative results for three scenes (scan5, scan11 and scan12) in the OMMO dataset. Our method exhibits superior performance compared to the GAN-based point cloud up-sampling method, primarily due to its
Table 3: Quantitative results of ablation experiments, including removing the diffusion-based point cloud up-sampling module, using the GAN-based point cloud up-sampling method, our PDF method.
Method	PSNR↑	SSIM↑	LPIPS↓
w/o diffusion	21.85	0.84	0.204
GAN-based method [40]	24.83	0.86	0.161
Ours	28.60	0.90	0.137
ability to preserve the structural and topological characteristics of point clouds while effectively handling incomplete or noisy point cloud data. In addition, Fig. 6 shows the visualization results of the diffusion-based point cloud up-sampling module, and our method can not only densify the sparse point cloud reconstructed by the COLMAP, but also fill in the missing regions of the point cloud such as the background and empty space.
(b)
Figure 6: Qualitative evaluation of the diffusion-based point cloud super-resolution module. From left to right: (a) Point cloud of a large-scale scene reconstructed using COLMAP. (b) Point cloud of the same large-scale scene enhanced using our method. (c) Point cloud of a small-scale scene reconstructed using COLMAP. (d) Point cloud of the same small-scale scene enhanced using our method (zoom-in for the best view).
In this paper, we propose PDF, a point diffusion implicit function for large-scale scene neural representation, and demonstrate its robustness and fidelity on novel view synthesis tasks. The core of our method is to provide dense point cloud surface priors to reduce the huge sampling space of large-scale scenes. Therefore, a point cloud super-resolution module based on diffusion model is proposed to learn from the sparse point cloud surface distribution reconstructed from training views to generate more dense point clouds. However, only constraining the sampling space to the point cloud surface does not fully solve the novel view synthesis problem since point clouds do not have background information. So Mip-NeRF 360 [2] is employed to provide background features and synthesize photo-realistic new perspectives. Extensive experiments demonstrate that our method outperforms current methods in both subjective and objective aspects. At the same time, ablation experiments also prove the effectiveness of our core module, point up-sampling diffusion.
In future work, we will attempt to explore a cross-scene point cloud up-sampling generalization diffusion model instead of training a diffusion model for each scene to improve efficiency. Even more futuristically, it may be possible to extract representative scene representations and inject them into reconstructed point clouds to achieve cross-scene rendering, i.e., generalized point diffusion NeRF.
This work is supported by National Natural Science Foundation of China (No. 62101137, 62071127, and U1909207), Shanghai Natural Science Foundation (No.23ZR1402900), and Zhejiang Lab Project (No.2021KH0AB05).
[36] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
[37] Fukun Yin, Wen Liu, Zilong Huang, Pei Cheng, Tao Chen, and Gang Yu. Coordinates are not lonelycodebook prior helps implicit neural 3d representations. Advances in Neural Information Processing Systems, 35:12705-12717, 2022.
[38] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[39] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: geometry editing of neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18353-18364, 2022.
[40] Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, and Chen Change Loy. Unsupervised 3d shape completion through gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1768-1777, 2021.
[41] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[42] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[43] Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.