SlowGuess/NeurIPS_2023_OCR huggingface dataset:
UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene
Jiaming Gu $^{1,2*}$  Minchao Jiang $^{1*}$  Hongsheng Li $^{1}$  Xiaoyuan Lu $^{2}$  Guangming Zhu $^{1}$  Syed Afaq Ali Shah $^{3}$  Liang Zhang $^{1\dagger}$  Mohammed Bennamoun $^{4}$
$^{1}$ School of Computer Science and Technology, Xidian University
$^{2}$ Algorithm R&D Center, Qing Yi (Shanghai)
<sup>3</sup>Edith Cowan University
4 The University of Western Australia
jiaming_gu_xidian@outlook.com, {jamchaos, hsli}@stu.xidian.edu.cn
xylu@bnc.org.cn, {gmzhu, liangzhang}@xidian.edu.cn
afaq.shah@ecu.edu.au, mohammed.bennamoun@uwa.edu.au
Neural Radiance Field (NeRF) is an implicit 3D reconstruction method that has shown immense potential and has gained significant attention for its ability to reconstruct 3D scenes solely from a set of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, has significant limitations. To address this challenge, we propose a novel neural rendering system called UE4-NeRF that is designed for real-time rendering of large-scale scenes. Our proposed approach partitions large scenes into sub-NeRFs, and uses polygonal meshes to represent them. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from the Level of Detail (LOD) techniques, we train meshes with varying levels of detail for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Our experimental results demonstrate that our method attains rendering quality on par with state-of-the-art approaches, while additionally offering the advantage of real-time performance. Furthermore, rendering within UE4 facilitates scene editing in subsequent stages. Project page: https://jamchaos.github.io/UE4-NeRF/.
Real-time interactive rendering capability for 3D large-scale scenes is a crucial tool in creating digital worlds for applications such as virtual reality (VR), computer games, and the Metaverse. Neural Radiance Fields (NeRF) [23] is a pioneering method for 3D reconstruction and synthesis of novel views, which is different from traditional 3D reconstruction techniques that represent scenes using explicit expressions such as point clouds, grids, and voxels [5, 4, 14, 1]. NeRF samples each ray and retrieves the 3D location of each sampling point and the 2D viewing direction of the ray. Subsequently, these 5D vector values are fed into a neural network to determine the color and volume density of each sampling point. NeRF constructs a field parameterized by an MLP neural network [16, 34] to continuously optimize parameters and reconstruct the scene. NeRF has demonstrated remarkable
*Contribute equally
(a) captured by UAV
(b) real-time rendering of large-scale scene in UE4
(c) edition
Figure 1: (a) Large-scale scenes characterized by complex textures and significant height differences are captured with monocular or multi-camera drone footage. We divide scene into multiple individual sub-NeRFs.(b) Real-time interactive rendering in UE4 with distinct levels of detail, along with frame rate disparities among them. (c) Defferent editing capabilities on the scenes.
performance for 3D reconstruction when provided with a set of posed camera images [29, 2, 23], but early NeRF works [2, 23] had only focused on small-scale scenes and object-centric reconstruction due to the model computational complexity. Although Block-NeRF [37] and BungeeNeRF [43] are capable of reconstructing large-scale environments, they are still unable to achieve real-time rendering. Traditional NeRF obtains the color of each pixel through volume rendering, but this rendering process is far too slow for interactive visualization and compromises high fidelity. Recently, Mobile-NeRF [6] combined with classic polygon rasterization pipeline has shown to achieve real-time rendering on various devices. However, training just a small-scale scene incurs significant overheads, such as training time and GPU resources. It requires 8 v100 GPUs to train the model successfully over several days. Additionally, due to opacity binarization, Mobile-NeRF cannot handle outdoor scenes with a variety of materials, including transparent or semi-transparent objects such as water, glass and plastic shed. Performing subsequent editing on the scenes generated by NeRF is another meaningful but challenging task, and previous works [13, 44, 19] required training complex neural networks to achieve this capability. These methods are time-consuming and are limited in their editing capabilities.
This paper introduces UE4-NeRF to address the aforementioned issues regarding large-scale scene rendering and editing. UE4-NeRF combines NeRF with the Unreal Engine 4 (UE4) [17, 30, 17] to achieve real-time interactive rendering of large-scale scenes with the scene editing ability. As shown in Figure 1(a), to achieve this, large-scale scene such as industrial parks with large height differences, roads and farmlands are partitioned into separately trained blocks. The size of individual scenes captured by drones is in the range of several square kilometers. The multi-resolution hash encoding proposed in Instant-NGP [25] is employed for training acceleration. In order to efficiently utilize the parallel computing capabilities of GPUs, the computationally intensive portions are implemented using CUDA. Through these efforts, the training speed of UE4-NeRF is  $1000 \times$  faster than Mobile-NeRF. For each block, we represent the scene using textured polygonal meshes obtained from regular octahedrons. Similar to Mobile-NeRF, the texture atlas stores feature vectors and opacity information. We utilize an MLP as the High-Level Shading Language(HLSL) fragment Shader in UE4, which takes in the feature vectors and outputs colors. Furthermore, to ensure real-time rendering performance for different observation heights viewpoints, we adopted a method akin to the Level of Detail (LOD) [8, 21] approach commonly used in computer graphics, where the complexity of 3D model representation is quantified in terms of metrics such as geometric detail and texture resolution. Figure 1(b) demonstrates that UE4-NeRF is capable of achieving real-time rendering at different relative observation heights by training different levels of polygon meshes, where coarser meshes are used for relatively higher observation heights. One of the other advantages of combining NeRF with UE4 is the powerful development capabilities of the rendering engine, allowing further processing of the final scene rendered in UE4. Users can freely combine different sub-NeRFs and add objects to the scene as needed. Moreover, object manipulation, as illustrated in Figure 1(c), is also supported.
In summary, our paper presents the following contributions:(1) We propose a novel UE4-NeRF technique which has 1000 times faster training times than MobileNeRF, while maintaining higher
quality and smaller storage overhead;(2) The demonstration of the capability of UE4-NeRF to reconstruct large-scale scenes in real-time and showcase the practical viability of NeRF;(3) The integration of NeRF and UE4 facilitates editing and manipulation of 3D scenes.
Large Scale 3D Reconstruction. The original NeRF algorithm was limited to object-centric scenes or scenes with boundaries.  $\mathrm{NeRF}++$  [46] expanded upon NeRF by introducing an inverted sphere parameterization to handle unbounded scenarios. Mip-NeRF 360 [3] addressed the challenges associated with unbounded scenes through the use of non-linear scene parameterization, online distillation and a novel distortion-based regularizer. However, it was observed that simply expanding the scene without adjusting the model capacity, i.e., using a fixed-size MLP, can lead to unstable training and loss of high-fidelity. Meanwhile, naively increasing the model capacity can significantly increase the training cost. BungeeNeRF [43] proposed a progressive approach to build and train large-scale scene models, which promoted layer specialization and unleashed the power of positional encoding over the full frequency range. Block-NeRF [37] and Mega-NeRF [40] spatially partitioned a large scene into several individually trained sub-NeRFs. They also incorporated appearance embeddings like NeRF-W [22] to handle appearance variations in the scene. Similar to Block-NeRF, our approach partitions the large-scale scene into sub-NeRFs, and trains a coarse scene model to assist in accurately delineating each sub-NeRF. However, our approach is not only applicable to the grid structure like Block-NeRF, but is also generally suitable for various types of scenes, e.g. mountainous terrain. Additionally, in contrast to the aforementioned NeRFs designed for large-scale scenes, our method accomplishes real-time rendering.
Rendering of Neural Fields. NeRF has tremendous potential for rendering and has gained attention for 3D reconstruction from a set of posed camera images [29, 2, 23]. However, the computational overhead of NeRF limits its practical applications. Some prior works [25, 11] have addressed training costs, while others have focused on rendering [9, 36, 27, 31]. SNeRG [20] and PlenOctrees [45] improved performance by baking the components of NeRF. However, these methods do not take sufficient advantage of the texture-based rasterization pipeline and require significant GPU memory to store volumetric textures. MobileNeRF [6] is a variation of NeRF that can run in real-time on a variety of common mobile devices. To render an image, Mobile-NeRF utilized the classic polygon rasterization pipeline with Z-buffering to produce a feature vector for each pixel and then passed it to a lightweight MLP running in a GLSL fragment Shader to produce the output color. In this work, we accelerate the rendering of NeRF by converting the learned knowledge into a mesh representation for efficient mesh rasterization pipelines. Moreover, we achieve a significant milestone by successfully implementing NeRF for real-time interactive rendering of large-scale scenes, which was previously unexplored.
Manipulation and Composition of NeRF. Practical applications of 3D representations require effective manipulation and composition capabilities. While, explicit 3D representations, such as meshes and voxels, inherently support editing and composition, it becomes challenging for neural network-based implicit representations such as vanilla NeRF [23]. NSVF [20] uses a sparse voxel octree for modeling, employing voxel embeddings for each scene while sharing the same MLP for predicting density and color. However, the flexibility of this approach is somewhat constrained. GIRAFFE [28], combined with GAN [49, 18], enabled the addition of multiple objects in a scene, expanding from single-object to multi-object generation, even in the absence of such objects in the training data. Plenoxels [11] utilizes explicit sparse voxel representation for direct object composition of different objects, but suffers from large storage requirements of the dense index matrix. None of these methods support object manipulation in a 3D scene. In contrast, our approach leverages UE4 to achieve large-scale scene rendering, enabling users to freely add and manipulate objects within the UE4-NeRF environment. For instance, users can use a broomstick to fly or drive a car within the rendered environment powered by UE4-NeRF.
Our novel method involves representing large scenes as polygon meshes, where the positions of mesh vertices are iteratively optimized during the training phase. The encoding network generates texture
Figure 2: Overview of UE4-NeRF. UE4-NeRF consists of three modules: (1) In the training module, we partition the sub-NeRFs and initialize the grids in each partitioned scene. During the training process, we continuously optimize the parameters of the Encoder-Decoder network and the vertex positions of the meshes; (2) Through the pre-rendering module, we extract the polygonal meshes at different levels of detail that will be used for the final rendering; (3) The rendering module consists of an inference submodule and an UE4 submodule, which communicate with each other to achieve the final real-time rendering.
maps that store opacity and feature vectors, essential for rendering. To accomplish real-time rendering of large scenes, we combine the pre-rendered polygon meshes with the rasterization pipeline available in Unreal Engine 4 (UE4). In the subsequent subsections, we present a detailed and comprehensive description of entire procedure.
3.1.1 Block Partition
As depicted in Figure 2, a large scene is partitioned into multiple blocks. To ensure smooth transitions at the boundaries of the model, each training area, indicated by dashed lines, is slightly larger than its corresponding scene area, with overlapping borders with neighboring blocks. We train a coarse model for sub-region segmentation, focusing on capturing the overall terrain and building outlines. We then filter the images associated with the sub-regions and use a mask to exclude pixels outside the scene area.
3.1.2 Optimization
We begin by initializing a  $128 \times 128 \times 128$  grid, and selecting the center point of each grid along with its six neighboring grids (front, back, left, right, top, and bottom) to create polygonal meshes. To address the slow and unstable convergence issue encountered by Mobile-NeRF when dealing with tilted surfaces, we utilize a regular octahedron with 20 faces, including the 8 exterior faces and 12 interior faces. This is illustrated in the "Initial Mesh" section of Figure 2. The ray emitted from the camera origin to the pixel is defined as  $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$  and  $\mathbf{d} = (\theta, \phi)$  is the viewing direction of the ray. For each ray emitted from the camera origin to the pixel, we calculate its intersection with the polygonal meshes. The point of intersection serves as the sampling point for that ray. For each block, we define an Encoder-Decoder network:
$$
\mathcal {E} (\operatorname {h a s h} \left(\mathbf {p} _ {\mathbf {i}}\right); \theta_ {\mathcal {E}}) \rightarrow \mathbf {M} _ {\mathbf {i}}, \alpha_ {i} \tag {1}
$$
$$
\mathcal {D} \left(\mathbf {M} _ {\mathrm {i}}, \mathcal {S H} \left(\mathbf {d} _ {\mathrm {i}}\right); \theta_ {\mathcal {D}}\right)\rightarrow \mathbf {c} _ {\mathrm {i}} \tag {2}
$$
The encoder network, denoted as MLP  $\mathcal{E}$ , takes positional information  $(\mathbf{p_i})$  encoded by multi-resolution hash functions[25] as input. It generates an 8D feature vector  $\mathbf{M_i}$ , which incorporates opacity information. On the other hand, the decoder network  $\mathcal{D}$  utilizes  $\mathbf{M_i}$  and the ray direction as inputs and predicts the color of the sampling point. The direction of the ray is encoded using Spherical Harmonics.
We use the traditional NeRF volume rendering method (Eq. 3) to obtain the predicted color  $\hat{\mathbf{C}}(\mathbf{r})$ . Due to the complexity of calculating the distance between sampling points, we simplify the process by fixing the distance value. The opacity  $\alpha$  is then directly predicted instead of the volume density.
$$
\hat {\mathbf {C}} (\mathbf {r}) = \sum_ {i = 1} ^ {N} T _ {i} \alpha_ {i} \mathbf {c} _ {\mathbf {i}}, \quad T _ {i} = \prod_ {j = 0} ^ {i - 1} (1 - \alpha_ {j}) \tag {3}
$$
Unlike the traditional photometric loss, our proposed photometric loss consists of two components. The first component referred to as level-1 photometric loss is calculated using Eq. 4. It quantifies the mean squared error between the predicted colors and the ground truth colors of the corresponding pixel.
$$
\mathcal {L} _ {r g b} ^ {\text {p a r t 1}} (\theta , V _ {p}) = \sum_ {r \in \mathcal {R}} \left\| \hat {\mathbf {C}} (\mathbf {r}) - \mathbf {C} (\mathbf {r}) \right\| _ {2} ^ {2} \tag {4}
$$
where  $V_{p}$  is the position of the vertices of the polygonal mesh under the corresponding level and  $\mathcal{R}$  is the set of rays in each batch.  $\mathbf{C}(\mathbf{r})$
represents the ground truth RGB colors for ray  $r$ . In addition, to improve the proximity of the triangle meshes to the object surface, we design a second component of the photometric loss, denoted as  $\mathcal{L}_{rgb}^{part2}$ . As depicted in Figure 3, in contrast to the previous sampling rule, we only select sampling points with opacity greater than a threshold  $f$ . Furthermore, the volumetric rendering process is halted when the accumulated opacity exceeds 0.8, indicating that the remaining opacity  $T'$  is less than 0.2. In the initial 10,000 epochs, the threshold  $f$  is maintained at 0, and afterward, as shown in Eq. 5, the  $f$  increases as the number of epochs increases, but does not exceed 0.3.
Figure 3: Illustration of sampling details. Our approach involves selectively sampling intersection points that are located above a threshold value  $f$  and terminating the volume rendering process when the remaining opacity drops below 0.2. This strategy effectively concentrates opacity on a limited number of specific meshes and compresses it towards the object, leading to improved rendering results.
$$
f = \min  \{0. 3, \left\lfloor \frac {\text {e p o c h} - 1 0 0 0 0}{1 0 0 0 0} \right\rfloor^ {2} \times 0. 0 1 2 \} \tag {5}
$$
The photometric loss for the second part of the level-1 can be obtained using the following formula:
$$
\mathcal {L} _ {r g b} ^ {p a r t 2} (\theta , V _ {p}) = \sum_ {r \in \mathcal {R}} \left\| \hat {\mathbf {C}} ^ {\prime} (\mathbf {r}) - \mathbf {C} (\mathbf {r}) \right\| _ {2} ^ {2}, \quad \hat {\mathbf {C}} ^ {\prime} (\mathbf {r}) = \frac {\sum_ {\phi = 1} ^ {N} T _ {\phi} ^ {\prime} \alpha_ {\phi} \mathbf {c} _ {\phi}}{1 - T _ {\phi} ^ {\prime}} \tag {6}
$$
Where  $\hat{\mathbf{C}}^{\prime}(\mathbf{r})$  is obtained through volume rendering under the aforementioned sampling scheme. To cover the entire color range, we divide the expression by  $1 - T_{\phi}^{\prime}$  as shown in Eq. 6, since our volume rendering stops when the remaining opacity is less than 0.2. We obtain the final photometric loss of the level-1 as follows:
$$
\beta \mathcal {L} _ {r g b} ^ {\text {p a r t 1}} (\theta , V _ {p}) + (1 - \beta) \mathcal {L} _ {r g b} ^ {\text {p a r t 2}} (\theta , V _ {p}), \quad \beta = 0. 8 \left\lfloor \frac {\text {e p o c h}}{1 0 0 0 0} \right\rfloor \tag {7}
$$
During the initial 10,000 epochs, the second component of the photometric loss does not have any impact. This is done to align with the traditional volume rendering approach and adhere to the modeling principles of NeRF. However, as the training progresses, the weight for the second component gradually increases. This enables the preservation of opacity only on specific meshes and ensures consistency with the later stages, where only meshes with opacity greater than 0.3 are exported for rasterization.
After 10000 epochs, a process is initiated where we randomly select sampling points within each grid, calculate the opacity of each grid, and select the vertex of the grid with the highest opacity among the eight grids as the vertex of the merged grid. This allows us to generate grids of  $64 \times 64 \times 64$ ,  $32 \times 32 \times 32$ ,  $16 \times 16 \times 16$ , and  $8 \times 8 \times 8$  sequentially. Each grid level is tailored for specific observation heights, enabling us to minimize the rendering cost of the scene. It is important to note that for the same block, different levels always use the same network, but the gradient in the rough levels are
only used to update mesh vertex coordinates. This measure can prevent coarse-scale meshes from providing undue supervision to shared networks when they cannot represent fine object structures. The final loss function of the photometric part is calculated as follows:
$$
\mathcal {L} _ {r g b} = \sum_ {l = 1} ^ {5} \beta \mathcal {L} _ {r g b} ^ {\text {p a r t 1}} (\theta , V _ {p}) + (1 - \beta) \mathcal {L} _ {r g b} ^ {\text {p a r t 2}} (\theta , V _ {p}) \tag {8}
$$
Pseudo-depth. Utilizing depth information [10, 33] as a supervisory signal has demonstrated the ability to enhance the convergence rate and improve the quality of reconstructions. However, when operating under the influence of ambient light in outdoor environments, RGB-D cameras are susceptible to produce inaccurate depth measurements and have limited measurement ranges. In our approach, we propose to use the pseudo-depth as the supervision information for NeRF. In conventional NeRF, structure-from-motion (SFM) [35] solvers such as COLMAP [15, 39, 12] are typically utilized to estimate the camera poses. As part of this process, a sparse point cloud can be obtained. By aligning the camera and point cloud in the same coordinate system, we calculate the pseudo-depth by measuring the distance between the sparse point cloud and the camera position. For a detailed explanation of the pseudo-depth loss  $(\mathcal{L}_D)$  calculation, please refer to the supplementary material. The final loss  $\mathcal{L}$  of the model is calculated as follows:
$$
\mathcal {L} = \mathcal {L} _ {r g b} + \mathcal {L} _ {D} \tag {9}
$$
Transient Objects. Moving/dynamic objects, such as pedestrians and cars, have the potential to cause inconsistency in the observed scene across different viewpoints and can also create gaps or "holes" in the depth map. To address this challenge, we integrate a semantic segmentation model [7, 48] into our training process. By leveraging this model, we generate masks that identify the regions occupied by the moving objects, effectively excluding them from the training of the scene. This strategy mitigates the impact of moving objects on the scene's depth map, and thus improves the overall fidelity and generalization performance of the model.
3.2 Pre-rendering
To optimize computational efficiency, we leverage the predicted values from the acceleration grid to identify and remove meshes that do not possess significant geometric surfaces. Once the training phase is complete, we obtain the network weights as well as the meshes with optimized vertex positions. Subsequently, we perform the pre-rendering process. The overall workflow is illustrated in Figure 4. First, for each block, in addition to the camera view used for training, we incorporate parallel lights from various angles above. We compute the intersection of each ray with the polygon mesh, and the traversal of light is halted when the cumulative
Opacity surpasses 0.8. A triangular face will be clipped if the opacity of all intersection points on that face is below 0.3, resulting in the retention of only those meshes that meet this criterion. From this, we obtain the final polygon mesh representations of the scene. Based on our statistical analysis, the final extracted meshes account for only  $5\%$  of the original mesh count. This significant reduction effectively improves mesh utilization, inference speed during rendering, and reduces storage costs. Second, we perform coordinate transformation to ensure the obtained mesh aligns with real-world scale. Third, we map the vertex coordinates of the triangles to their corresponding positions in UV coordinates.
Figure 4: Structure of the pre-rendering. We employ a multi-viewpoint approach to extract polygonal faces, generating multiple rays from different perspectives. Subsequently, we apply scaling transformations and calculate UV coordinates to further process the extracted data.
3.3 Rendering
After training, we obtain the weights of the Encoder-Decoder network, and refined polygonal meshes at different levels through vertex optimization for each individual block. Subsequently, we perform pre-rendering and extract the final meshes. For each polygonal mesh, we perform rasterization using a