SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Details of the real-world scene Table 2: Quantitative Comparisons with Existing captured by UAV. The memory requirement Methods. Our method achieves remarkable results in increases with the size of the scene, while terms of both rendering quality and speed. Among the the variation in FPS is primarily influenced compared methods, UE4-NeRF requires the least GPU by the materials present in the scene itself. memory. The training costs are also provided.
Memory(4k)	Area(m2)	FPS	CA	Method	PSNR↑	SSIM↑	LPIPS↓	FPS(2K)	Memory(GB)	time(mins)
Host	GPU	2k	4k	Host	GPU
FL	~12GB	~5GB	180×240	55	36	120m	NeRFacto [38]	20.99	0.663	0.389	0.5	~14	~3	~45
CS	~25GB	~11GB	420×240	66	43	70m	Instant-NGP [25]	22.00	0.631	0.426	11	~10	~32	~15
IP	~40GB	~14GB	600×600	58	35	120m	Mega-NeRF [40]	17.37	0.23	0.546	x	-	-	~2160
Mobile-NeRF [6]	16.92	0.23	0.419	x	-	-	~2880
UE4-NeRF	25.03	0.704	0.287	55	~19	~3	~40
texture map of size  $32 \times 32$ . However, storing texture maps for numerous polygonal meshes in a large-scale scene can lead to significant storage overhead. To address this, we utilize dynamic inference instead of directly storing all texture maps, as done in Mobile-NeRF [6]. Real-time rendering is achieved through continuous and efficient communication between the inference submodule and the UE4 submodule. During the block composition process, since the large-scale scene is accurately divided into multiple blocks and scene fusion is also considered during training, transitions between different blocks in a large-scale scene are smooth and interpolation is not required at the junction.
Texture map. Prior to rasterizing the faces of the polygonal mesh, we perform sampling on each mesh and use the encoder network to obtain 8 channels feature vectors. These 8 channels along with the opacity  $\alpha$  obtained are processed to generate nine single-channel texture maps using BC4 compression [26]. Compared to the original texture map, this reduces 2/3 of the GPU memory consumption during rendering without significantly compromising the image quality. Please note that texture maps do not store colors, but rather spatial features, which also contain transparency information. Taking into account the principle of locality, we employ a strategy where we collectively infer and store information from nearby regions into texture maps. This enables us to directly access the texture maps without the need for re-inference when traversing these regions again.
Rasterization. We utilize the traditional rasterization pipeline in UE4 to rasterize each face of the polygonal mesh. Subsequently, the decoder network is applied to convert the 17D features of each pixel into RGB colors (see UE4 submodule in Figure 2). The 17D features comprise 8 channels of learned features and a 9D view direction, which is encoded using spherical harmonic functions [11, 24]. The decoder network is implemented as a HLSL fragment Shader of the UE4, which is called Neural Shader. To achieve translucent effects without the need for volume rendering, we employ alpha-dithered [42] and temporal anti-aliasing, which offers a relatively low-overhead approach.
Dynamic inference. The interaction between the UE4 submodule and the inference submodule is depicted in Figure 2. When a block requiring rendering is encountered, the UE4 submodule supplies the block and level index information, while the inference submodule provides the corresponding regional meshes and feature maps. In cases where feature maps are not readily for a specific region, the inference operation is carried out to generate them. This communication mechanism ensures efficient and prompt processing between these two submodules.
Similar to the Instant-NGP approach, we employ multi-resolution hash encoding and construct an Encoder network comprising a 4-layer MLP  $(32\times 64,64\times 64,64\times 64,64\times 8)$  to estimate the opacity and an 8-dimensional feature vector. Additionally, we develop a Decoder network with 3-layer MLP  $(17\times 16,16\times 16,16\times 3)$  to predict the final color of the sampled points. For each block, we train the model using one Nvidia RTX 3090 GPU for 80,000 epochs to achieve convergence, with an approximate duration of 40 minutes.
Datasets and Metric. Our objective is to enable real-time rendering of large-scale scenes with complex textures and materials. Table 1 presents three distinct scenarios, namely Farmland(FL),
Figure 5: Qualitative Comparisons with Existing Methods. We visually compare our proposed technique with recently implemented real-time rendering NeRF models [25, 38] and Mip-NeRF [3]. All these methods are implemented within nerfstudio [38]. It is evident that UE4-NeRF yields the best reconstruction for all the three scenes.
Ground Truth
Mobile-NeRF
Ground Truth
Mega-NeRF
UE4-NeRF
Figure 6: Qualitative Comparisons with Mega-NeRF and Mobile-NeRF. Detailed comparison with Mobile-NeRF and Mega-NeRF. Since these methods are very time-consuming, we compared them in a block or scene.
Construction site(CS) and Industrial parks(IP) which were captured using drones to address the challenges that UE4-NeRF was designed to tackle. Each captured image has an approximate resolution of  $6000 \times 4000$  pixels and contains GPS information. One of our goals is to establish a measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. We performed a coordinate system transformation prior to modeling. In the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. We also performed comparative experiments on the dataset introduced in Mega-NeRF[40]. At level-1, UE4-NeRF can achieve an maximum frame rate of around 43 FPS at 4K resolution. Due to the limited battery life of the drone, the captured areas were constrained in size. Nevertheless, subsequent experiments have revealed that limitation does not represent the upper limit of UE4-NeRF. In fact, UE4-NeRF can render even larger-scale scenes in real-time. To evaluate the reconstruction quality and fidelity of our model, we use evaluation metrics such as PSNR/SSIM [41] and LPIPS [47]. We also compare the real-time rendering speed of UE4-NeRF with other models when rendering vast scenes.
4.2 Comparisons with Existing Methods
We performed qualitative comparisons between UE4-NeRF and other NeRF implementations on four scenes captured by UAV, and the results are depicted in Figure 5. Existing tools like NeRF Facto [38] and Instant-NGP [25] aim to achieve real-time rendering, but tend to procure subpar
Table 3: Quantitative Comparison with Mega-NeRF and Mobile-NeRF. Mobile-NeRF can not support complete scene. The maximal resolution supported by Mega-NeRF is  $800^{*}800$  on RTX 3090.
Methods	Training time	FPS(1×3090)
block	scene(n blocks,4×3090)
Mobile-NeRF	48h(4×3090)	n×48h	35(block,2K)
Mega-NeRF	36h(1×3090)	12h+36h×n/4	60(scene,800×800)
UE4-NeRF	40mins(1×3090)	1h+40mins×n/4	50(scene,2k), ≥70(block,2K)
results when applied to large-scale scenes, as evident in Figure 5. Mip-NeRF, despite increasing the number of training iterations, still exhibits noticeable blurriness when rendering natural scenes. In contrast, UE4-NeRF showcases remarkable accuracy in rendering small objects, such as power lines, in the "farmland" scene. It also excels in handling dense textures, as demonstrated in the "construction site" scene. The "construction site" scene also showcases the rendering effects of translucent objects. In Table 2, we provide a quantitative comparison of our proposed technique with other existing methods using different metrics. It is important to note that the actual rendering quality of UE4-NeRF is expected to surpass the metric's performance, as achieving consistent exposure matching with the origi
nal image is challenging within the Unreal Engine environment. Despite this limitation, UE4-NeRF still outperforms other methods by a significant margin and its real-time rendering capability surpasses that of the current state-ofthe-art real-time rendering models. In Table 3, we see that Mobile-NeRF takes 2 days to train just one block and requires 4x3090ti GPUs. If it trains the whole scene, it takes two months. For each block, Mega-NeRF needs 36 hours to train for 500,000 epochs, while we only need 40 minutes to train for 80,000 epochs to reach convergence. When training the entire scene, Mega-NeRF requires 12 hours of additional time, but UE4-NeRF only needs 1 hour. It is worth noting that Mega-NeRF generates hundreds of GBs of temporary files during training. And as can be seen from Figure 6 and Figure 7, compared with Ground truth, UE4-NeRF shows amazing rendering results, but Mobile-NeRF and Mega-NeRF produce significant blur.
Figure 7: Comparison with Mega-NeRF on the public dataset. Due to the large exposure gap between images in the Mega-NeRF dataset, different shadows appear at different viewing angles.
4.3 LOD render results
Figure 8 showcases the qualitative rendering results at different levels of detail, along with the corresponding real-time frame rates, at a resolution of  $2\mathrm{K}$ . As the level of hierarchy decreases, the rendering quality and clarity progressively improve. UE4-NeRF demonstrates a high level of fidelity when compared to the ground truth. In addition, it can be observed that the LOD approach effectively
Figure 8: Rendering result at different LOD. The red boxes indicate the presentation of the same content at different levels. In addition to achieving high FPS rendering at different LOD in 2K resolution, UE4-NeRF also demonstrates a remarkable photo-realistic.
Figure 9: Shading mesh- not textured. Polygonal meshes at different levels are displayed, revealing the varying complexity of the scene.
Figure 10: Scene edition. The results of multi-view scene editing are presented. One can carefully observe the occlusion relationship between objects and the scene.
addresses the challenge of real-time rendering at higher levels although it may sacrifice some rendering quality at higher levels for the sake of real-time performance. When comparing materials and frame rates across different scenes, it is worth noting that the presence of semi-transparent objects, such as vegetation(as seen in the last scene of Figure 8), can potentially lead to a decrease in frame rate due to increased computational overhead.
Shading mesh. Figure 9 showcases the shading meshes extracted without textures. Despite the coarse appearance of the meshes and the limited level of detail even in the closest level-1, such as excavators, the final rendering quality remains impressive. This can be attributed to the vertex optimization carried out during the training process, the texture maps encoded in the network outputs, and the deployment of the final neural shader during rendering.
4.4 Scene edition
Editing scenes in UE4-NeRF, especially the composition of different scenes, is a seamless process. UE4 supports multiple 3D model file formats, allowing us to import and edit the rendered scenes. The Unreal Engine automatically handles occlusion between the original scene and added objects, making the editing process effortless. We have complete freedom to manipulate the newly added objects according to our needs. In Figure 10, when an object obstructs the NeRF-rendered scene, the occluded parts do not require shading by the neural shader. Interestingly, we observed that after adding and manipulating objects, the frame rate of the scene actually slightly improves. This is attributed to the reduction in NeRF computation, resulting in improved frame rates.
In this research paper, we introduce UE4-NeRF, a real-time interactive rendering system designed for large-scale scenes. UE4-NeRF partitions the scene into blocks and trains a separate NeRF model for each block. Inspired by LOD techniques, different levels of mesh precision are used for different layers, and the NeRF model is integrated with the rasterization pipeline in UE4. Currently, UE4-NeRF stands as the only NeRF model capable of achieving real-time interactive rendering of large-scale realistic scenes while maintaining high fidelity. Moreover, its integration with UE4 opens up possibilities of scene editing and manipulation, allowing for the flexible addition and manipulation of traditional 3D models such as .obj and .fbx within the scene. Overall, UE4-NeRF presents a comprehensive neural rendering system that combines large-scale, real-time interactive rendering with photo-realism and editability. However, there are still some issues that we aim to address in the future: 1) UE4-NeRF requires the direct use of CUDA and therefore relies on NVIDIA GPUs. Rendering large scenes, such as a few square kilometers, incurs significant memory overhead when without reducing rendering accuracy, and increasing rendering accuracy further amplifies memory consumption. 2) During the pre-rendering process, it is difficult to capture rays from any viewing angles and extract meshes for each viewpoint, resulting in gaps or holes in the final rendered scene. The strategy of pre-rendering can be optimized.
This work is supported by National Natural Science Foundation of China (No.62072358,62073252). The authors would like to express gratitude to the anonymous reviewers who greatly help improve the manuscript. Besides, we sincerely thank Hongli Wang, Ke Zhang and Mengqi Shen for their technical support in drone shooting.