SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuhan Ding $^{1*}$
dingyh22@m.fudan.edu.cn
Fukun Yin $^{1*}$
fkyin21@m.fudan.edu.cn
Jiayuan Fan $^{2\dagger}$
yfan@fudan.edu.cn
Hui Li2
lihui21@m.fudan.edu.cn
Xin Chen3
chenxin2@shanghaitech.edu.cn
Wen Liu<sup>3</sup>
liuwen@shanghaitech.edu.cn
Chongshan Lu<sup>1</sup>
cslu17@fudan.edu.cn
Gang  $\mathbf{Y}\mathbf{u}^3$
iskicy@gmail.com
Tao Chen
eetchen@fudan.edu.cn
$^{1}$  School of Information Science and Technology, Fudan University, China
$^{2}$  Academy for Engineering and Technology, Fudan University, China
3 Tencent PCG, Shanghai, China
Recent advances in implicit neural representations have achieved impressive results by sampling and fusing individual points along sampling rays in the sampling space. However, accurately representing and synthesizing fine-grained textures in unbounded, large-scale outdoor scenes presents a significant challenge, attributable to the exponentially expanding sampling space. To alleviate the dilemma of using individual sampling points to perceive vast expanses, we explore learning the surface distribution of the scene to provide structural priors, thereby reducing the samplable space, and propose a Point Diffusion implicit Function, PDF, for large-scale scene neural representation. The core of our method is a large-scale point cloud super-resolution diffusion module that enhances the sparse point cloud reconstructed from several training images into a dense point cloud as the explicit prior. Then in the rendering stage, only sampling points with prior points within the sampling radius are retained. That is, the sampling space is reduced from the unbounded space to the scene surface. Meanwhile, to fill in the background details not captured by point clouds, we employ region sampling based on Mip-NeRF 360 for modeling comprehensive background representations. Extensive experiments have demonstrated the effectiveness of our method for large-scale scene novel view synthesis, which outperforms relevant state-of-the-art baselines.
Implicit neural representations have demonstrated proficiency in handling single objects or small scenes and have found extensive applications in the fields of virtual reality [5], 3D reconstruction [21, 31, 36], video generation [7] and computer animation [19, 1, 41] on tasks such as scene representation and new perspective synthesis. Nevertheless, as the scale of the target scene increases, particularly in urban-scale outdoor scenes, traditional implicit neural representation methods encounter significant
*Joint first authors.
performance limitations. This issue primarily arises from the cubic expansion of the sampling space in larger scenes, rendering it challenging for individual sampling points to cover the entire space.
Fortunately, some methods try to solve this problem through two primary strategies, narrowing the sampling space and expanding the sampling area. The first approach represented by Mega-NeRF [28] decomposes the sampling space into multiple subspaces and models each subspace separately to reduce complexity. But with the scene scale growing, such as reaching the city level, the quantity of these subspaces increases cubically, posing scalability concerns. In contrast, the method represented by Mip-NeRF 360 [2] compresses the sampling space or samples an area instead of a single point so that the sampling space can be filled more easily. Nevertheless, this approach may compromise the precision of the representation.
Benefiting from the inspiration of geometric priors aiding vision tasks, which are widely used in the domain of 3D reconstruction and stereo vision [16, 3, 6]. We are curious whether implicit large scene representations could be made easier with explicit representations. Moreover, for outdoor unbounded large scenes, most of the sampling space is filled with air rather than buildings, cars, plants and other objects that we care about. A reasonable solution is to restrict the large-scale sampling space of the implicit neural representation to the object surface, which is provided by the scene geometry prior. That is to say, we compress a 3D sampling space to a 2D surface plane, which will greatly reduce the representation complexity. At the same time, the network will pay more attention to the foreground, which is the same as the human visual perception system. Of course, for the neglected background information that is relatively less important, we can provide a relatively less accurate expression by compressing the scene to sample the area in space.
In this paper, we propose PDF, a Point Diffusion implicit Function for large-scale scene neural representation, which learns a dense surface distribution via a diffusion-based point prior generative model to reduce the sampling space. To achieve this, we first explore a large-scale outdoor point cloud augmentation method based on the Point-Voxel Diffusion model [43]. Since point clouds of real outdoor scenes often lack dense ground truth, it is difficult to train a completion module through "sparse-dense" point cloud pairs. Therefore, we initially downsample the point cloud twice, and train a point cloud super-resolution network to generate the denser one from its sparser counterpart. This approach effectively generates dense point clouds in the absence of ground truth data. With the help of the surface point cloud, the sampling points will be retained only if there are reconstruction points within a certain radius, so the space will be greatly reduced to the scene surface. However, the reconstructed point cloud can only model the scene surface and cannot deal with the unbounded background of outdoor scenes. Accordingly, following the concept of NeRF++ [41], we model the foreground and background separately, and use Mip-NeRF 360 [2] to extract background features by sampling regions in the scene space.
Extensive experiments show the effectiveness of our point diffusion implicit function for large-scale scene neural representation, which achieves photo-realistic rendering results and outperforms state-of-the-art methods on OMMO [15] and BlendMVS dataset [35]. We summarize the contributions as follows: 1) Aiming at novel view synthesis for large outdoor scenes, we propose an implicit neural representation framework based on point diffusion models to provide dense surface priors to cope with the exploding sampling space. 2) A novel point cloud super-resolution diffusion module is proposed to generate dense surface points from sparse point clouds without dense annotations. 3) Extensive experiments demonstrate that our PDF network outperforms state-of-the-art methods, including robustness to large-scale outdoor scene representation and the capability to synthesize more photo-realistic novel views. Our code and models will be available.
2 Related Work and Background
2.1 Implicit Neural Representation
In recent years, Implicit Neural Representation (INR) has witnessed significant advancements and provides a versatile framework for representing complex functions and generating high-dimensional data [23, 17, 26, 37]. By implicitly encoding the scene's appearance and geometry, neural radiance fields enable highly realistic rendering and novel view synthesis [25, 20, 8, 18].
Building upon this foundation, subsequent research has focused on addressing the limitations and pushing the boundaries of INR. Efforts have been made to improve the efficiency and scalability
of neural radiance fields. For instance, Hanocka et al. propose DeepSDF[22], which leverages signed distance functions to implicitly represent 3D shapes. This formulation allows for efficient ray-marching and facilitates tasks such as shape manipulation and interpolation. Furthermore, recent advancements in INR have explored differentiable rendering and differentiable volumetric rendering, enabling the incorporation of geometric and physical priors [10, 4, 27] into the representation. These methods leverage the differentiable nature of neural networks to optimize scene parameters, leading to improved realism and control over the generated content [13, 39, 9, 30]. Another significant extension to the field of INR is PixelNeRF [38]. It extends the capabilities of INR to handle images, going beyond the realm of 3D scenes. PixelNeRF introduces a new differentiable sampler to handle image-based representations, enabling efficient and accurate sampling of pixels from the neural radiance field. In addition to PixelNeRF, Semantic Neural Radiance Fields[12] propose a method to learn scene representations that capture geometry, appearance, and semantic information, facilitating interactive virtual scene editing and content creation.
Overall, these advancements have greatly expanded the capabilities of INR. These developments offer promising avenues for realistic image synthesis, shape completion, scene reconstruction, and dynamic content generation. The ongoing research in this field holds great potential for further advancements in computer graphics, computer vision, and virtual reality applications.
2.2 Large-scale Scene Representation
Large-scale scene representation is a crucial aspect of INR research, particularly in the context of computer graphics and computer vision. It involves capturing and modeling complex scenes that encompass extensive spatial extents, such as urban environments, landscapes, or virtual worlds.
One notable work in the domain of large-scale scene representation is Neural Scene Flow Fields[14]. This paper introduces a novel approach to model dynamic scenes at a large scale. The authors propose a scene flow field representation that captures both the geometry and motion of objects in the scene. By leveraging a neural network architecture, they achieve accurate and temporally consistent scene synthesis and reconstruction, even in highly complex and dynamic scenes. The Neural 3D Mesh Renderer[11] is another significant contribution in large-scale scene representation. This work addresses the challenge of representing and rendering detailed 3D meshes of large-scale scenes efficiently. The authors propose a neural network-based renderer that predicts view-dependent textures and geometric details of the scene. This approach enables real-time rendering and interaction with large-scale 3D scenes, opening up possibilities for interactive virtual reality experiences and immersive simulations. In addition to these works, Mega-NeRF [28] and Bungee-NeRF [33] are two other notable approaches based on the neural radiance field for constructing interactive 3D environments from large-scale visual captures. They address the challenges of modeling and rendering large-scale scenes, spanning from buildings to multiple city blocks and utilizing thousands of images captured from drones. They extend the capabilities of NeRF to handle multi-scale rendering, capturing various levels of detail and enabling the interactive exploration of diverse 3D environments.
Overall, the field of large-scale scene representation within INR has witnessed significant progress. These contributions have paved the way for realistic, interactive, and semantically meaningful representations of expansive virtual environments, urban landscapes, and dynamic scenes. The ongoing research in this area holds great potential for further advancements in computer graphics, virtual reality, and immersive simulations.
In this paper, we aim to develop a novel point diffusion model implicit function to reduce the sampling space and improve the ability to represent large-scale scenes (c.f. Fig. 1). Our PDF network mainly consists of two modules, a diffusion-based component for point cloud super-resolution and foreground rendering, and a region-sampling module focused on background processing. The former introduces a diffusion model to enhance the sparse point cloud reconstructed from the input images into a dense point cloud, which provides optional points in the rendering stage to reduce the sampling space (c.f. Sec. 3.1). The latter samples regions rather than individual points from unbounded scenes so that it is easy to fill sampled regions and complement the background for new viewpoint synthesis (c.f. Sec. 3.2). In the final subsection, implementation details and losses are elaborated (c.f. Sec. 3.3).
Figure 1: The pipeline of our point diffusion implicit function. Our method consists of two modules, a point diffusion rendering module and a background rendering module. The former learns the surface distribution of the scene through a diffusion-based point cloud super-resolution model and renders foreground features from the dense point cloud surface. The latter follows Mip-NeRF 360's strategy to render background features. Finally, the foreground and background features are fused to generate photo-realistic novel views for large-scale outdoor scenes.
3.1 Point Upsampling Diffusion
In this section, we introduce our large-scale outdoor point cloud super-resolution module based on a denoising diffusion probabilistic model (c.f. Fig. 2).
Point Cloud Pair Preparation. Due to the lack of dense large-scale outdoor point cloud ground truth, we need to train a diffusion-based super-resolution network to sample a dense surface, symbolized as  $x_{d} \in \mathbb{R}^{N \times 3}$ , from the point cloud reconstructed by COLMAP [24], denoted as  $x_{s} \in \mathbb{R}^{M \times 3}$ . Concurrently, to mitigate the risk of over-fitting, the point cloud reconstructed from the training views is not utilized as the ground truth for the diffusion model. Instead, training data comprises pairs of the sparse point cloud  $z_{0} \in \mathbb{R}^{n \times 3}$  and the sparser point cloud  $x_{0} \in \mathbb{R}^{m \times 3}$ , adhering to  $m < n < M < N$ . More specifically, we downsample the sparse point cloud  $x_{s}$  reconstructed by COLMAP to get an even sparser point cloud  $z_{0}$ . Then we further downsample  $z_{0}$  to get the sparser point cloud  $x_{0}$ , where  $x_{s}, z_{0}$  and  $x_{0}$  have progressively sparser relationships. Our training process recovers  $z_{0}$  from the sparser  $x_{0}$ . During testing, we take  $x_{s}$  as input to generate a denser super-resolved point cloud  $x_{d}$ .
Point Super-resolution Diffusion. Our point super-resolution denoising diffusion probabilistic model is a generative model, which starts with Gaussian noise and progressively denoises to generate scene structure priors. We record the output containing different levels of noise produced by each step as  $\hat{x}_T$ ,  $\hat{x}_{T-1}, \dots, \hat{x}_0$ , where  $\hat{x}_T$  is sampled from Gaussian noise, and  $\hat{x}_0$  represents the generated point cloud with dense surface. Since we already have a sparse point cloud prior  $z_0$ , our target point cloud can be denoted as  $x_0 = (z_0, \hat{x}_0)$  and the intermediate point cloud during the denoising process can be denoted as  $x_t = (z_0, \hat{x}_t)$ . Subsequently, we define a point super-resolution diffusion process involving a prior shape  $z_0$ , consisting of a forward process and a backward process.
Forward Process. Gaussian noise is repeatedly added to the original point cloud  $x_0$ , resulting in a series of noisy point clouds  $x_1, x_2, \ldots, x_T$ :
$$
q \left(\hat {x} _ {t} \mid \hat {x} _ {t - 1}, z _ {0}\right) \sim \mathcal {N} \left(\hat {x} _ {t}; \sqrt {1 - \beta_ {t}} \hat {x} _ {t - 1}, \beta_ {t} I\right) \tag {1}
$$
where  $\beta_{t}$  represents a pre-defined increasing sequence of Gaussian noise values, which dictates the magnitude of noise incrementally introduced at each step of the process.
Reverse Process. Given a point cloud with more noise  $x_{t}$ , reverse the forward process and find the posterior distribution for a less noisy one  $x_{t-1}$ :
$$
p _ {\theta} \left(\hat {x} _ {t - 1} \mid \hat {x} _ {t}, z _ {0}\right) \sim \mathcal {N} \left(\mu_ {\theta} \left(x _ {t}, z _ {0}, t\right), \sigma_ {t} ^ {2} I\right) \tag {2}
$$
where  $\mu_{\theta}(x_t,z_0,t)$  is the predicted shape at  $t - 1$  step.
Figure 2: Our point upsampling diffusion. In the forward process, Gaussian noise is gradually added to the sparse point cloud. In the reverse process, the noise is gradually removed to obtain a dense point cloud surface.
Therefore, our point cloud upsampling diffusion model can be regarded as a noise adding and denoising process. The former gradually adds random noise to the initial point cloud  $x_0$  through the forward process; the latter denoises sequentially through the reverse process to obtain a dense point cloud  $x_0$ . Based on Markov transition probabilities, the whole process can be expressed as:
$$
q \left(\hat {x} _ {0: T}, z _ {0}\right) = q \left(\hat {x} _ {0}, z _ {0}\right) \prod_ {t = 1} ^ {T} q \left(\hat {x} _ {t} \mid \hat {x} _ {t - 1}, z _ {0}\right) \tag {3}
$$
$$
p _ {\theta} \left(\hat {x} _ {0: T}, z _ {0}\right) = p \left(\hat {x} _ {T}, z _ {0}\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(\hat {x} _ {t - 1} \mid \hat {x} _ {t}, z _ {0}\right) \tag {4}
$$
Throughout the optimization process, our prior shape  $z_0$  is fixed, and only the missing surface point cloud is diffused. The network is typically trained with a simplified  $L2$  denoising loss:
$$
\mathcal {L} _ {\mathcal {D}} = \left| \left| \epsilon - \epsilon_ {\theta} \left(\hat {x} _ {t}, z _ {0}, t\right) \right| \right| ^ {2} \tag {5}
$$
where  $\epsilon$  is the added random noise and  $\epsilon \sim \mathcal{N}(0, I)$ , and  $\epsilon_{\theta}(\hat{x}_t, z_0, t)$  is the prediction noise output. Since point cloud prior  $z_0$  is fixed, it will be masked when minimizing the loss.
3.2 Volume Rendering and Implicit Function Representation
Foreground Rendering. For the foreground, we sample points along the rays from the dense point cloud  $x_0$  and render features in the neighborhood, following Point-NeRF [34]. The difference is that our point super-resolution diffusion module only generates denser point cloud coordinates without color information, so we redesign a more general point aggregate module. For ray marching through a pixel, we sample  $M$  sampling points at  $\{p_i \mid i = 1, \dots, M\}$ , and query  $K$  neighboring neural points  $kp_i = \{kp_i^1, kp_i^2, \dots, kp_i^K\}$  around  $p_i$  within a certain euclidean distance radius  $R$ . Then we interpret the local geometric structure as a feature  $f_i$  of each sampling point  $p_i$  to equip structural information. Therefore, we utilize  $c_i$  and  $k c_i$  to represent the coordinates of the sampling point and its neighborhood points, respectively. The geometric structure feature  $kf_i = \{kf_i^1, kf_i^2, \dots, kf_i^K\}$  of the neighborhood are encoded as follows:
$$
k f _ {i} = M L P \left(c _ {i} \oplus k c _ {i} \oplus \left(c _ {i} - k c _ {i}\right) \oplus d \left(c _ {i}, k c _ {i}\right)\right) \tag {6}
$$
where  $d(,)$  is the Euclidean distance between two points, and  $\oplus$  is the concatenation operator. Next, the local geometric features  $f_{i}$  of the sampling points  $p_i$  are obtained by neighborhood points  $kp_{i}$  weighted summation:
$$
f _ {i} = \operatorname {S U M} \left(\operatorname {s o f t m a x} \left(M L P \left(k f _ {i}\right)\right) \odot k f _ {i}\right) \tag {7}
$$
where softmax operation is performed on each dimension, and  $\odot$  is the hadamard product. Our point-based radiance field can be abstracted as a neural module that regresses the volume density  $\sigma$  and view-dependent radiance  $r$  from coordinates  $c$ , local geometric features  $f$ , and ray direction  $d$  according to Point-NeRF [34]:
$$
(\sigma , r) = \text {P o i n t - N e R F} (c, d, f) \tag {8}
$$
Finally, the foreground feature is synthesized by each neural sampling point along the sampling ray.
Background Rendering. In consideration of the limitation that point clouds are confined to representing foreground elements and are incapable of addressing unbounded background contexts, it becomes necessary to procure supplementary background features. Benefiting from Mip-NeRF 360 [2], which contracts the scene to a bounded ball and then samples a region to meet the challenge of large scenes, we employ this method to extract background features as a supplement along the same sampling ray as Point-NeRF [34].
Fore-Background Fusion. Since the detail-preserving foreground features can be obtained from the dense surface points, while the bounded domain can cope with large scenes but loses details during the compression process. So we propose a foreground-background fusion module consisting of several layers of multi-layer perceptrons to preserve their respective advantages.
We adopt  $L2$  loss to supervise our rendered pixels  $r_p$  from ray marching with the ground truth  $r_g$ , to optimize our PDF volume render reconstruction network.
$$
\mathcal {L} _ {\mathcal {R}} = \left\| r _ {p} - r _ {g} \right\| ^ {2} \tag {9}
$$
Our PDF method is a two-stage neural representation network for outdoor unbounded large-scale scenes. We optimize these two stages separately.
In the first stage, a diffusion-based point cloud super-resolution network is designed to learn a prior distribution to generate a dense point cloud surface. In the point cloud pair preparation process, we employed the random down-sampling method with a retention rate between 0.2 and 1 for both samplings. For point super-resolution diffusion, we set  $T = 1000$ ,  $\beta_0 = 10^{-4}$ ,  $\beta_T = 0.01$  and linearly interpolate other  $\beta$ 's for all experiments. We use Adam optimizer with learning rate  $2 \times 10^{-4}$  and train on 4 A100 GPUs for around one day.
In the second stage, the foreground and background extraction modules plus a feature fusion module are optimized. We find 8 neighbors for each sampling point and expand the dimension of neighborhood geometric features to 8. Both the foreground and the background output a 128-dimensional feature, and then they are concatenated and passed through 4 MLP layers to get the color of the rendered point. We train this stage using Adam optimizer with an initial learning rate  $5 \times 10^{-4}$  for  $2 \times 10^{6}$  iterations about 20 hours on a single A100 GPU.
Dataset. We use two outdoor large-scale scene datasets, OMMO [15] and BlendedMVS [35], to evaluate our model. The OMMO dataset is a real fly-view large-scale outdoor multi-modal dataset, containing complex objects and scenes with calibrated images, prompt annotations and point clouds. The number of training point cloud samples in the OMMO dataset varies from 40,000 to 100,000 for different scenes, including abundant real-world urban and natural scenes with various scales, camera trajectories, and lighting conditions. More experimental results can be found in our supplementary material.
Baselines and Evaluation Metrics. We compare our method with the previous state-of-the-art methods on novel view synthesis, including NeRF [19], NeRF++ [41], Mip-NeRF [1], Mip-NeRF 360 [2], Mega-NeRF [28], Ref-NeRF [29]. NeRF is the first continuous MLP-based neural network for synthesizing photo-realistic views of a scene through volume rendering. NeRF++ models large-scale unbounded scenes by separately modeling foreground and background neural representations.
Mip-NeRF reduces aliasing artifacts and better represents fine details by using anti-aliasing cone sampling. Mip-NeRF 360 models large unbounded scenes using non-linear scene parameterization, online distillation, and distortion-based regularization. Mega-NeRF uses a sparse structure and geometric clustering algorithm to decompose the scenes. Ref-NeRF improves synthesized views by restructuring radiance and regularizing normal vectors. To evaluate the performance of each method for large-scale implicit neural representation, we use three common metrics for novel view synthesis: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM [32]), and Learned Perceptual Image Patch Similarity (LPIPS [42]). Higher PSNR and SSIM indicate better performance, while lower LPIPS indicates better performance.