Title: UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene
Venue: NeurIPS 2023
Accepted: yes
Keywords: NeRF

UE4

Large-scale scenes

Real-time rendering

Rasterization
Abstract: Neural Radiance Fields (NeRF) is a novel implicit 3D reconstruction method that shows immense potential and has been gaining increasing attention. It enables the reconstruction of 3D scenes solely from a set of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, still has significant limitations. To address these challenges, in this paper, we propose a novel neural rendering system called UE4-NeRF, specifically designed for real-time rendering of large-scale scenes. We partitioned each large scene into different sub-NeRFs. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and  the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from Level of Detail (LOD) techniques, we trained meshes of varying levels of detail for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Rendering within UE4 also facilitates  scene editing in subsequent stages. Furthermore, through experiments, we have demonstrated that our method achieves rendering quality comparable to state-of-the-art approaches. Project page: https://jamchaos.github.io/UE4-NeRF/.

UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene

Abstract

Neural Radiance Field (NeRF) is an implicit 3D reconstruction method that has shown immense potential and has gained significant attention for its ability to reconstruct 3D scenes solely from a set of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, has significant limitations. To address this challenge, we propose a novel neural rendering system called UE4-NeRF that is designed for real-time rendering of large-scale scenes. Our proposed approach partitions large scenes into subNeRFs, and uses polygonal meshes to represent them. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from the Level of Detail (LOD) techniques, we train meshes with varying levels of detail for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Our experimental results demonstrate that our method attains rendering quality on par with state-of-the-art approaches, while additionally offering the advantage of real-time performance. Furthermore, rendering within UE4 facilitates scene editing in subsequent stages. Project page: https://jamchaos.github.io/UE4-NeRF/.

1 Introduction

Real-time interactive rendering capability for 3D large-scale scenes is a crucial tool in creating digital worlds for applications such as virtual reality (VR), computer games, and the Metaverse. Neural Radiance Fields (NeRF) [23] is a pioneering method for 3D reconstruction and synthesis of novel views, which is different from traditional 3D reconstruction techniques that represent scenes using explicit expressions such as point clouds, grids, and voxels [5, 4, 14, 1]. NeRF samples each ray and retrieves the 3D location of each sampling point and the 2D viewing direction of the ray. Subsequently, these 5D vector values are fed into a neural network to determine the color and volume density of each sampling point. NeRF constructs a field parameterized by an MLP neural network [16, 34] to continuously optimize parameters and reconstruct the scene. NeRF has demonstrated remarkable

Converting

Manoeuvringthe

Removethecar

buildingstograss

flyingsaucer

GT

(b) real-time rendering of large-scale scene in UE4

(a)captured byUAV

(c) edition

performance for 3D reconstruction when provided with a set of posed camera images [29, 2, 23], but early NeRF works [2, 23] had only focused on small-scale scenes and object-centric reconstruction due to the model computational complexity. Although Block-NeRF [37] and BungeeNeRF [43] are capable of reconstructing large-scale environments, they are still unable to achieve real-time rendering. Traditional NeRF obtains the color of each pixel through volume rendering, but this rendering process is far too slow for interactive visualization and compromises high fidelity. Recently, Mobile-NeRF [6] combined with classic polygon rasterization pipeline has shown to achieve real-time rendering on various devices. However, training just a small-scale scene incurs significant overheads, such as training time and GPU resources. It requires 8 v100 GPUs to train the model successfully over several days. Additionally, due to opacity binarization, Mobile-NeRF cannot handle outdoor scenes with a variety of materials, including transparent or semi-transparent objects such as water, glass and plastic shed. Performing subsequent editing on the scenes generated by NeRF is another meaningful but challenging task, and previous works [13, 44, 19] required training complex neural networks to achieve this capability. These methods are time-consuming and are limited in their editing capabilities.

This paper introduces UE4-NeRF to address the aforementioned issues regarding large-scale scene rendering and editing. UE4-NeRF combines NeRF with the Unreal Engine 4 (UE4) [17, 30, 17] to achieve real-time interactive rendering of large-scale scenes with the scene editing ability. As shown in Figure 1(a) , to achieve this, large-scale scene such as industrial parks with large height differences, roads and farmlands are partitioned into separately trained blocks. The size of individual scenes captured by drones is in the range of several square kilometers. The multi-resolution hash encoding proposed in Instant-NGP [25] is employed for training acceleration. In order to efficiently utilize the parallel computing capabilities of GPUs, the computationally intensive portions are implemented using CUDA. Through these efforts, the training speed of UE4-NeRF is 1000 × faster than MobileNeRF. For each block, we represent the scene using textured polygonal meshes obtained from regular octahedrons. Similar to Mobile-NeRF, the texture atlas stores feature vectors and opacity information. We utilize an MLP as the High-Level Shading Language(HLSL) fragment shader in UE4, which takes in the feature vectors and outputs colors. Furthermore, to ensure real-time rendering performance for different observation heights viewpoints, we adopted a method akin to the Level of Detail (LOD) [8, 21] approach commonly used in computer graphics, where the complexity of 3D model representation is quantified in terms of metrics such as geometric detail and texture resolution. Figure 1(b) demonstrates that UE4-NeRF is capable of achieving real-time rendering at different relative observation heights by training different levels of polygon meshes, where coarser meshes are used for relatively higher observation heights. One of the other advantages of combining NeRF with UE4 is the powerful development capabilities of the rendering engine, allowing further processing of the final scene rendered in UE4. Users can freely combine different sub-NeRFs and add objects to the scene as needed. Moreover, object manipulation, as illustrated in Figure 1(c) , is also supported.

In summary, our paper presents the following contributions:(1) We propose a novel UE4-NeRF technique which has 1000 times faster training times than MobileNeRF, while maintaining higher

quality and smaller storage overhead;(2) The demonstration of the capability of UE4-NeRF to reconstruct large-scale scenes in real-time and showcase the practical viability of NeRF;(3) The integration of NeRF and UE4 facilitates editing and manipulation of 3D scenes.

2 Related work

Large Scale 3D Reconstruction. The original NeRF algorithm was limited to object-centric scenes or scenes with boundaries. NeRF++ [46] expanded upon NeRF by introducing an inverted sphere parameterization to handle unbounded scenarios. Mip-NeRF 360 [3] addressed the challenges associated with unbounded scenes through the use of non-linear scene parameterization, online distillation and a novel distortion-based regularizer. However, it was observed that simply expanding the scene without adjusting the model capacity, i.e., using a fixed-size MLP, can lead to unstable training and loss of high-fidelity. Meanwhile, naively increasing the model capacity can significantly increase the training cost. BungeeNeRF [43] proposed a progressive approach to build and train large-scale scene models, which promoted layer specialization and unleashed the power of positional encoding over the full frequency range. Block-NeRF [37] and Mega-NeRF [40] spatially partitioned a large scene into several individually trained sub-NeRFs. They also incorporated appearance embeddings like NeRF-W [22] to handle appearance variations in the scene. Similar to Block-NeRF, our approach partitions the large-scale scene into sub-NeRFs, and trains a coarse scene model to assist in accurately delineating each sub-NeRF. However, our approach is not only applicable to the grid structure like Block-NeRF, but is also generally suitable for various types of scenes, e.g. mountainous terrain. Additionally, in contrast to the aforementioned NeRFs designed for large-scale scenes, our method accomplishes real-time rendering.

Rendering of Neural Fields. NeRF has tremendous potential for rendering and has gained attention for 3D reconstruction from a set of posed camera images [29, 2, 23]. However, the computational overhead of NeRF limits its practical applications. Some prior works [25, 11] have addressed training costs, while others have focused on rendering [9, 36, 27, 31]. SNeRG [20] and PlenOctrees [45] improved performance by baking the components of NeRF. However, these methods do not take sufficient advantage of the texture-based rasterization pipeline and require significant GPU memory to store volumetric textures. MobileNeRF [6] is a variation of NeRF that can run in real-time on a variety of common mobile devices. To render an image, Mobile-NeRF utilized the classic polygon rasterization pipeline with Z-buffering to produce a feature vector for each pixel and then passed it to a lightweight MLP running in a GLSL fragment shader to produce the output color. In this work, we accelerate the rendering of NeRF by converting the learned knowledge into a mesh representation for efficient mesh rasterization pipelines. Moreover, we achieve a significant milestone by successfully implementing NeRF for real-time interactive rendering of large-scale scenes, which was previously unexplored.

Manipulation and Composition of NeRF. Practical applications of 3D representations require effective manipulation and composition capabilities. While, explicit 3D representations, such as meshes and voxels, inherently support editing and composition, it becomes challenging for neural network-based implicit representations such as vanilla NeRF [23]. NSVF [20] uses a sparse voxel octree for modeling, employing voxel embeddings for each scene while sharing the same MLP for predicting density and color. However, the flexibility of this approach is somewhat constrained. GIRAFFE [28], combined with GAN [49, 18], enabled the addition of multiple objects in a scene, expanding from single-object to multi-object generation, even in the absence of such objects in the training data. Plenoxels [11] utlizes explicit sparse voxel representation for direct object composition of different objects, but suffers from large storage requirements of the dense index matrix. None of these methods support object manipulation in a 3D scene. In contrast, our approach leverages UE4 to achieve large-scale scene rendering, enabling users to freely add and manipulate objects within the UE4-NeRF environment. For instance, users can use a broomstick to fly or drive a car within the rendered environment powered by UE4-NeRF.

3 Proposed Method

Our novel method involves representing large scenes as polygon meshes, where the positions of mesh vertices are iteratively optimized during the training phase. The encoding network generates texture

BlockPartition

InitialMesh

Multi-resolution

hashencoding

Network

Loss

VolumetricRendering

Opacity

40.

onnecy

-□

-~

7Spherical

Harmonics

Photometric Loss Depth Loss

Pre-Rendering

Mesh

Infer

NeuralShader

BlockFuse

DynamicLODblock

Encoder

Render

Skip

Mesh

FeatureMap

maps that store opacity and feature vectors, essential for rendering. To accomplish real-time rendering of large scenes, we combine the pre-rendered polygon meshes with the rasterization pipeline available in Unreal Engine 4 (UE4). In the subsequent subsections, we present a detailed and comprehensive description of entire procedure.

3.1 Training

3.1.1 Block Partition

As depicted in Figure 2, a large scene is partitioned into multiple blocks. To ensure smooth transitions at the boundaries of the model, each training area, indicated by dashed lines, is slightly larger than its corresponding scene area, with overlapping borders with neighboring blocks. We train a coarse model for sub-region segmentation, focusing on capturing the overall terrain and building outlines. We then filter the images associated with the sub-regions and use a mask to exclude pixels outside the scene area.

3.1.2 Optimization

We begin by initializing a 128 × 128 × 128 grid, and selecting the center point of each grid along with its six neighboring grids (front, back, left, right, top, and bottom) to create polygonal meshes. To address the slow and unstable convergence issue encountered by Mobile-NeRF when dealing with tilted surfaces, we utilize a regular octahedron with 20 faces, including the 8 exterior faces and 12 interior faces. This is illustrated in the "Initial Mesh" section of Figure 2. The ray emitted from the camera origin to the pixel is defined as r ( t ) = o + t d and d = ( θ, ϕ ) is the viewing direction of the ray. For each ray emitted from the camera origin to the pixel, we calculate its intersection with the polygonal meshes. The point of intersection serves as the sampling point for that ray. For each block, we define an Encoder-Decoder network:

The encoder network, denoted as MLP E , takes positional information( p i ) encoded by multiresolution hash functions[25] as input. It generates an 8D feature vector M i , which incorporates opacity information. On the other hand, the decoder network D utilizes M i and the ray direction as inputs and predicts the color of the sampling point. The direction of the ray is encoded using Spherical Harmonics.

We use the traditional NeRF volume rendering method (Eq. 3) to obtain the predicted color ˆ C ( r ) . Due to the complexity of calculating the distance between sampling points, we simplify the process by fixing the distance value. The opacity α is then directly predicted instead of the volume density.

Unlike the traditional photometric loss, our proposed photometric loss consists of two components. The first component referred to as level -1 photometric loss is calculated using Eq. 4. It quantifies the mean squared error between the predicted colors and the ground truth colors of the corresponding pixel.

where V p is the position of the vertices of the polygonal mesh under the corresponding level R C ( r )

and is the set of rays in each batch. represents the ground truth RGB colors for ray r . In addition, to improve the proximity of the triangle meshes to the object surface, we design a second component of the photometric loss, denoted as L part 2 rgb . As depicted in Figure 3, in contrast to the previous sampling rule, we only select sampling points with opacity greater than a threshold f . Furthermore, the volumetric rendering process is halted when the accumulated opacity exceeds 0.8, indicating that the remaining opacity T ′ is less than 0.2. In the initial 10,000 epochs, the threshold f is maintained at 0, and afterward, as shown in Eq. 5, the f increases as the number of epochs increases, but does not exceed 0.3.

The photometric loss for the second part of the level -1 can be obtained using the following formula:

Where ˆ C ′ ( r ) is obtained through volume rendering under the aforementioned sampling scheme. To cover the entire color range, we divide the expression by 1 -T ′ ϕ as shown in Eq. 6, since our volume rendering stops when the remaining opacity is less than 0 . 2 . We obtain the final photometric loss of the level -1 as follows:

During the initial 10,000 epochs, the second component of the photometric loss does not have any impact. This is done to align with the traditional volume rendering approach and adhere to the modeling principles of NeRF. However, as the training progresses, the weight for the second component gradually increases. This enables the preservation of opacity only on specific meshes and ensures consistency with the later stages, where only meshes with opacity greater than 0.3 are exported for rasterization.

After 10000 epochs, a process is initiated where we randomly select sampling points within each grid, calculate the opacity of each grid, and select the vertex of the grid with the highest opacity among the eight grids as the vertex of the merged grid. This allows us to generate grids of 64 × 64 × 64, 32 × 32 × 32, 16 × 16 × 16, and 8 × 8 × 8 sequentially. Each grid level is tailored for specific observation heights, enabling us to minimize the rendering cost of the scene. It is important to note that for the same block, different levels always use the same network, but the gradient in the rough levels are

only used to update mesh vertex coordinates. This measure can prevent coarse-scale meshes from providing undue supervision to shared networks when they cannot represent fine object structures. The final loss function of the photometric part is calculated as follows:

Pseudo-depth. Utilizing depth information [10, 33] as a supervisory signal has demonstrated the ability to enhance the convergence rate and improve the quality of reconstructions. However, when operating under the influence of ambient light in outdoor environments, RGB-D cameras are susceptible to produce inaccurate depth measurements and have limited measurement ranges. In our approach, we propose to use the pseudo-depth as the supervision information for NeRF. In conventional NeRF, structure-from-motion (SFM) [35] solvers such as COLMAP [15, 39, 12] are typically utilized to estimate the camera poses. As part of this process, a sparse point cloud can be obtained. By aligning the camera and point cloud in the same coordinate system, we calculate the pseudo-depth by measuring the distance between the sparse point cloud and the camera position. For a detailed explanation of the pseudo-depth loss ( L D ) calculation, please refer to the supplementary material. The final loss L of the model is calculated as follows:

Transient Objects. Moving/dynamic objects, such as pedestrians and cars, have the potential to cause inconsistency in the observed scene across different viewpoints and can also create gaps or "holes" in the depth map. To address this challenge, we integrate a semantic segmentation model [7, 48] into our training process. By leveraging this model, we generate masks that identify the regions occupied by the moving objects, effectively excluding them from the training of the scene. This strategy mitigates the impact of moving objects on the scene's depth map, and thus improves the overall fidelity and generalization performance of the model.

3.2 Pre-rendering

To optimize computational efficiency, we leverage the predicted values from the acceleration grid to identify and remove meshes that do not possess significant geometric surfaces. Once the training phase is complete, we obtain the network weights as well as the meshes with optimized vertex positions. Subsequently, we perform the pre-rendering process. The overall workflow is illustrated in Figure 4. First , for each block, in addition to the camera view used for training, we incorporate parallel lights from various angles above. We compute the intersection of each ray with the polygon mesh, and the traversal of light is halted when the cumulative

opacity surpasses 0.8. A triangular face will be clipped if the opacity of all intersection points on that face is below 0.3, resulting in the retention of only those meshes that meet this criterion. From this, we obtain the final polygon mesh representations of the scene. Based on our statistical analysis, the final extracted meshes account for only 5% of the original mesh count. This significant reduction effectively improves mesh utilization, inference speed during rendering, and reduces storage costs. Secend , we perform coordinate transformation to ensure the obtained mesh aligns with real-world scale. Third , we map the vertex coordinates of the triangles to their corresponding positions in UV coordinates.

3.3 Rendering

After training, we obtain the weights of the Encoder-Decoder network, and refined polygonal meshes at different levels through vertex optimization for each indivicual block. Subsequently, we perform pre-rendering and extract the final meshes. For each polygonal mesh, we perform rasterization using a

texture map of size 32 × 32. However, storing texture maps for numerous polygonal meshes in a largescale scene can lead to significant storage overhead. To address this, we utilize dynamic inference instead of directly storing all texture maps, as done in Mobile-NeRF [6]. Real-time rendering is achieved through continuous and efficient communication between the inference submodule and the UE4 submodule. During the block composition process, since the large-scale scene is accurately divided into multiple blocks and scene fusion is also considered during training, transitions between different blocks in a large-scale scene are smooth and interpolation is not required at the junction.

Texture map. Prior to rasterizing the faces of the polygonal mesh, we perform sampling on each mesh and use the encoder network to obtain 8 channels feature vectors. These 8 channels along with the opacity α obtained are processed to generate nine single-channel texture maps using BC4 compression [26]. Compared to the original texture map, this reduces 2/3 of the GPU memory consumption during rendering without significantly compromising the image quality. Please note that texture maps do not store colors, but rather spatial features, which also contain transparency information. Taking into account the principle of locality, we employ a strategy where we collectively infer and store information from nearby regions into texture maps. This enables us to directly access the texture maps without the need for re-inference when traversing these regions again.

Rasterization. We utilize the traditional rasterization pipeline in UE4 to rasterize each face of the polygonal mesh. Subsequently, the decoder network is applied to convert the 17D features of each pixel into RGB colors (see UE4 submodule in Figure 2). The 17D features comprise 8 channels of learned features and a 9D view direction, which is encoded using spherical harmonic functions [11, 24]. The decoder network is implemented as a HLSL fragment shader of the UE4, which is called Neural Shader. To achieve translucent effects without the need for volume rendering, we employ alpha-dithered [42] and temporal anti-aliasing, which offers a relatively low-overhead approach.

Dynamic inference. The interaction between the UE4 submodule and the inference submodule is depicated in Figure 2. When a block requiring rendering is encountered, the UE4 submodule supplies the block and level index information, while the inference submodule provides the corresponding regional meshes and feature maps. In cases where feature maps are not readibly for a specific region, the inference operation is carried out to generate them. This communication mechanism ensures efficient and prompt processing between these two submodules.

4 Experiments

4.1 Implementation Details

Similar to the Instant-NGP approach, we employ multi-resolution hash encoding and construct an Encoder network comprising a 4-layer MLP (32 × 64,64 × 64,64 × 64,64 × 8) to estimate the opacity and an 8-dimensional feature vector. Additionally, we develop a Decoder network with 3-layer MLP (17 × 16,16 × 16,16 × 3) to predict the final color of the sampled points. For each block, we train the model using one Nvidia RTX 3090 GPU for 80,000 epochs to achieve convergence, with an approximate duration of 40 minutes.

Datasets and Metric. Our objective is to enable real-time rendering of large-scale scenes with complex textures and materials. Table 1 presents three distinct scenarios, namely Farmland(FL),

Farmland

Construction site

Industrial parks

NeRFacto

Mip-NeRF

Instant-NGP

UE4-NeRF

Ground Truth

Mobile-NeRF

UE4-NeRF

Ground Truth

Mega-NeRF

UE4-NeRF

Construction site(CS) and Industrial parks(IP) which were captured using drones to address the challenges that UE4-NeRF was designed to tackle. Each captured image has an approximate resolution of 6000 × 4000 pixels and contains GPS information. One of our goals is to establish a measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. We performed a coordinate system transformation prior to modeling. In the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. We also performed comparative experiments on the dataset introduced in Mega-NeRF[40]. At level -1, UE4-NeRF can achieve an maximum frame rate of around 43 FPS at 4K resolution. Due to the limited battery life of the drone, the captured areas were constrained in size. Nevertheless, subsequent experiments have revealed that limitation does not represent the upper limit of UE4-NeRF. In fact, UE4-NeRF can render even larger-scale scenes in real-time. To evaluate the reconstruction quality and fidelity of our model, we use evaluation metrics such as PSNR/SSIM [41] and LPIPS [47]. We also compare the real-time rendering speed of UE4-NeRF with other models when rendering vast scenes.

4.2 Comparisons with Existing Methods

We performed qualitative comparisons between UE4-NeRF and other NeRF implementations on four scenes captured by UAV, and the results are depicted in Figure 5. Existing tools like NeRFacto [38] and Instant-NGP [25] aim to achieve real-time rendering, but tend to procude subpar

results when applied to large-scale scenes, as evident in Figure 5. Mip-NeRF, despite increasing the number of training iterations, still exhibits noticeable blurriness when rendering natural scenes. In contrast, UE4-NeRF showcases remarkable accuracy in rendering small objects, such as power lines, in the "farmland" scene. It also excels in handling dense textures, as demonstrated in the "construction site" scene. The "construction site" scene also showcases the rendering effects of translucent objects. In Table 2, we provide a quantitative comparison of our proposed technique with other existing methods using different metrics. It is important to note that the actual rendering quality of UE4-NeRF is expected to surpass the metric's performance, as achieving consistent exposure matching with the origi- nal image is challenging within the Unreal Engine environment. Despite this limitation, UE4-NeRF still outperforms other methods by a significant margin and its real-time rendering capability surpasses that of the current state-ofthe-art real-time rendering models. In Table 3, we see that Mobile-NeRF takes 2 days to train just one block and requires 4x3090ti GPUs. If it trains the whole scene, it takes two months. For each block, Mega-NeRF needs 36 hours to train for 500,000 epochs, while we only need 40 minutes to train for 80,000 epochs to reach convergence. When training the entire scene, Mega-NeRF requires 12 hours of additional time, but UE4-NeRF only needs 1 hour. It is worth noting that Mega-NeRF generates hundreds of GBs of temporary files during training. And as can be seen from Figure 6 and Figure 7, compared with Ground truth, UE4-NeRF shows amazing rendering results, but Mobile-NeRF and Mega-NeRF produce significant blur.

4.3 LOD render results

Figure 8 showcases the qualitative rendering results at different levels of detail, along with the corresponding real-time frame rates, at a resolution of 2K. As the level of hierarchy decreases, the rendering quality and clarity progressively improve. UE4-NeRF demonstrates a high level of fidelity when compared to the ground truth. In addition, it can be observed that the LOD approach effectively

FPS:60

FPS:52

FPS:55

FPS:65

FPS:60

FPS:70

Ground Truth

L3

L2

L1

Ours

FPS:41

FPS:46

FPS:58

FPS:50

FPS:35

FPS:38

Ground Truth

L3

L2

L1

Ours

Mega-NeRF

UE4-NeRF

L1

addresses the challenge of real-time rendering at higher levels although it may sacrifice some rendering quality at higher levels for the sake of real-time performance. When comparing materials and frame rates across different scenes, it is worth noting that the presence of semi-transparent objects, such as vegetation(as seen in the last scene of Figure 8), can potentially lead to a decrease in frame rate due to increased computational overhead.

Shading mesh. Figure 9 showcases the shading meshes extracted without textures. Despite the coarse appearance of the meshes and the limited level of detail even in the closest level -1, such as excavators, the final rendering quality remains impressive. This can be attributed to the vertex optimization carried out during the training process, the texture maps encoded in the network outputs, and the deployment of the final neural shader during rendering.

4.4 Scene edition

Editing scenes in UE4-NeRF, especially the composition of different scenes, is a seamless process. UE4 supports multiple 3D model file formats, allowing us to import and edit the rendered scenes. The Unreal Engine automatically handles occlusion between the original scene and added objects, making the editing process effortless. We have complete freedom to manipulate the newly added objects according to our needs. In Figure 10, when an object obstructs the NeRF-rendered scene, the occluded parts do not require shading by the neural shader. Interestingly, we observed that after adding and manipulating objects, the frame rate of the scene actually slightly improves. This is attributed to the reduction in NeRF computation, resulting in improved frame rates.

5 Conclusion and Limitations

In this research paper, we introduce UE4-NeRF, a real-time interactive rendering system designed for large-scale scenes. UE4-NeRF partitions the scene into blocks and trains a separate NeRF model for each block. Inspired by LOD techniques, different levels of mesh precision are used for different layers, and the NeRF model is integrated with the rasterization pipeline in UE4. Currently, UE4-NeRF stands as the only NeRF model capable of achieving real-time interactive rendering of large-scale realistic scenes while maintaining high fidelity. Moreover, its integration with UE4 opens up possibilities of scene editing and manipulation, allowing for the flexible addition and manipulation of traditional 3D models such as .obj and .fbx within the scene. Overall, UE4-NeRF presents a comprehensive neural rendering system that combines large-scale, real-time interactive rendering with photo-realism and editability. However , there are still some issues that we aim to address in the future: 1) UE4-NeRF requires the direct use of CUDA and therefore relies on NVIDIA GPUs. Rendering large scenes, such as a few square kilometers, incurs significant memory overhead when without reducing rendering accuracy, and increasing rendering accuracy further amplifies memory consumption. 2) During the pre-rendering process, it is difficult to capture rays from any viewing angles and extract meshes for each viewpoint, resulting in gaps or holes in the final rendered scene. The strategy of pre-rendering can be optimized.

Acknowledgments and Disclosure of Funding

This work is supported by National Natural Science Foundation of China (No.62072358,62073252). The authors would like to express gratitude to the anonymous reviewers who greatly help improve the manuscript. Besides, we sincerely thank Hongli Wang, Ke Zhang and Mengqi Shen for their technical support in drone shooting.

References

John Ashburner and Karl J Friston. Voxel-based morphometry-the methods. Neuroimage , 11(6):805-821, 2000.

Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

Matthew Berger, Andrea Tagliasacchi, Lee M Seversky, Pierre Alliez, Gael Guennebaud, Joshua A Levine, Andrei Sharf, and Claudio T Silva. A survey of surface reconstruction from point clouds. In Computer graphics forum , volume 36, pages 301-329. Wiley Online Library, 2017.

Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon mesh processing . CRC press, 2010.

Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.00277 , 2022.

Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and LiangChieh Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12475-12485, 2020.

James H Clark. Hierarchical geometric models for visible surface algorithms. Communications of the ACM , 19(10):547-554, 1976.

Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics (ToG) , 34(4):1-13, 2015.

Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12882-12891, 2022.

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5501-5510, 2022.

Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multi-view stereopsis (pmvs). In IEEE Computer Society Conference on Computer Vision and Pattern Recognition , volume 2, 2007.

Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser. Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503 , 2020.

Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence , 43(12): 4338-4364, 2020.

Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision . Cambridge university press, 2003.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks , 2(5):359-366, 1989.

Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice , 4(3):1, 2013.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4340-4350, 2023.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020.

David Luebke, Martin Reddy, Jonathan D Cohen, Amitabh Varshney, Benjamin Watson, and Robert Huebner. Level of detail for 3D graphics . Morgan Kaufmann, 2003.

Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7210-7219, 2021.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65 (1):99-106, 2021.

Claus Müller. Spherical harmonics , volume 17. Springer, 2006.

Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) , 41(4):1-15, 2022.

Jae-Ho Nah. Quicketc2: Fast etc2 texture compression using luma differences. ACM Transactions on Graphics (TOG) , 39(6):1-10, 2020.

Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H Mueller, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of compact neural radiance fields using depth oracle networks. In Computer Graphics Forum , volume 40, pages 45-59. Wiley Online Library, 2021.

Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11453-11464, 2021.

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

Weichao Qiu and Alan Yuille. Unrealcv: Connecting computer vision to unreal engine. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14 , pages 909-916. Springer, 2016.

Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12932-12942, 2022.

Antoni Rosinol, John J Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.13641 , 2022.

Franco Scarselli and Ah Chung Tsoi. Universal approximation using feedforward neural networks: A survey of some existing methods, and some new results. Neural networks , 11(1):15-37, 1998.

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4104-4113, 2016.

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems , 34:19313-19325, 2021.

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04264 , 2023.

Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment-a modern synthesis. In Vision Algorithms: Theory and Practice: International Workshop on Vision Algorithms Corfu, Greece, September 21-22, 1999 Proceedings , pages 298-372. Springer, 2000.

Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12922-12931, 2022.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Peter L Williams, Randall J Frank, and Eric C LaMar. Alpha dithering to correct low-opacity 8 bit compositing errors. Lawrence Livermore National Laboratory Technical Report UCRL-ID-153185 , 2003.

Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII , pages 106-122. Springer, 2022.

Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13779-13788, 2021.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5752-5761, 2021.

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision , pages 2223-2232, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a method for large-scale scene reconstruction and rendering.\nIt utilizes NeRF to learn a mesh representation for the scene by optimizing the \nvertex positions and neural features/MLPs using standard volume rendering. \nTo handle large-scale scenes, the method divides the scene into multiple blocks \nand trains a NeRF for each block. The initial mesh within each grid is got \nby creating an octahedron at the center of each voxel grid. The paper also learns \na hierarchical representation with Level-of-Details to support efficient \nrendering viewed from different distances. The paper shows that such a representation \ncan be integrated with Unreal Engine for interactive rendering and scene composition. \nThe paper shows results on outdoor scenes captured by drones and makes comparisons \nagainst baselines such as instant-ngp, urban-NeRF, and NeRF-w. The experiment results \nshow that the proposed method achieves higher accuracy in rendering and faster rendering\nspeed.\n\nSTRENGTHS:\n1. The pipeline of optimizing a mesh representation with volume rendering for large-scale \nscene reconstruction is technically sound, and the paper shows that the proposed pipeline \nachieves better accuracy than the baseline method in terms of both accuracy and rendering speed. \n\n2. The paper introduces multiple techniques to increase the rendering efficiency and quality\nof the reconstructions, such as using an alpha threshold to make the mesh close to the real \nsurface, using pseudo-depth to remove floaters, and using pre-rendering to filter out \ntriangles that have small contributions.\n\nWEAKNESSES:\nSee Questions.\n\nQUESTIONS:\n1. The overall pipeline of the paper is similar to that of MobileNeRF. The paper says in Line 100\nthat \"in this work, we accelerate the rendering by ... into a mesh representation for efficient \nmesh rasterization pipeline\".  How is this different from MobileNeRF? \nWhile I agree that the proposed method is faster than MobileNeRF in terms of training, I think \nit's the main benefit of using hash grid instead of MLP. In terms of representations, I think \nthe proposed method is similar to MobileNeRF. \nIn terms of supporting large-scale inputs, it seems that the major change is separating the scene into blocks and doing blockwise\ntraining. Can MobileNeRF be adopted for this task if we train it for each block? I think the paper \nshould make it more clear the difference of the proposed representation. \n\n2. In Line 137, I think it should be \"a regular octahedron with **8** faces\". The paper says the \nMobileNeRF has slow and unstable convergence issues. It's not clear to me why using octahedron \ncan help resolve this issue. What are the benefits of using it? It's needed to add a validation \nstudy and comparisons to MobileNeRF. It will give the readers more insight if the paper \ninvolves a comparison to MobileNeRF on standard benchmarks such as NeRF synthetic and LLFF. \n\n3. The paper mentions having overlaps between blocks. How are the reconstructions handled at \noverlapped regions? More details should be provided in this part.   \n\n4. In Equation 5, how is the threshold 0.3 chosen? How does the performance of the proposed method \nchange with different values of this threshold?\n\n5. In Line 222, the paper says \"filter out meshes that have an intersection opacity less 0.3\". Is \nthis opacity only calculated at a single intersection point? I am wondering whether this will \ncause problems in regions where there are sharp edges with abrupt opacity changes.\n\n6. I think the title of the paper is kind of misleading. The reconstructed models are compatible \nwith standard rasterization engines using customized shaders, and it's not specific to UE4. Integrating \ninto UE4 is also something trivial given the underlying polygonal meshes. I would suggest the authors\nmodify the short name to more focus on large-scale reconstructions.\n\n\nOverall, I think the proposed method is technically sound, and the presented results are convincing. \nMy major concern is that the backbone used in the paper is similar to MobileNeRF and the paper is combining \nit with blockwise reconstructions (which is explored in previous large-scale NeRF works), \nwhich prevents me from giving a higher rating.\n\nLIMITATIONS:\nThe limitations look good to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a multi-scale surface based representation for radiance field reconstruction and real-time rendering of large-scale scenes. The method involves subdividing the scene into partitioned scenes initialized with a regular octahedron mesh. Through joint optimization of the vertices' positions and multi-resolution hash grids, the proposed method incorporates two rendering loss functions and depth supervision. Experimental results demonstrate that the proposed method achieves improved rendering quality while maintaining real-time rendering capabilities (>30 FPS) for large-scale scenes at 4K resolution.\n\nSTRENGTHS:\n1. The manuscript successfully tackles the crucial challenge of fast training and rendering for large-scale scenes. Excellent work!\n\n2. The multi-scale meshes representation seems fresh and interesting.\n\nWEAKNESSES:\n1. Presentation. The \"transient object\" section distracted me a lot, I don't this manuscript work on the dynamic scenario, correct me if I am wrong. \n\n2. Evaluation. The proposed method is only evaluated on the UAV dataset, leading it hard to judge its general performance, would be great to have some results on standard dataset such as mipnerf 360 and Tanks and Temple.\n\n3. Comparison. I could not find any comparison or discussion regarding Mobile-NeRF,  even though it is the closest method to the approach presented in this manuscript.\n\n4. Fair comparison. Table 1 shows the proposed method provides clearly better performance than previous Mip-NeRF, NeRF-W NeRFacto and Instant-NGP, however, these methods don't use depth supervision by default, did you plug in the additional loss for these methods or use their official setting?\n\n5. Anonymous. I don't think putting the project link associated with your personal GitHub account to the manuscript is a good idea. I'm not sure if it violates the anonymity rule, but I don't take that into account when scoring now and bring it up in hopes of hearing the authors' and another review's perspectives.\n\nMinor:\nThe caption of Fig. 2: we partition the ...\n\nQUESTIONS:\n1. L137, why tilted surfaces can result in unstable convergence while regular octahedrons wouldn't?\n\n2. The usage of \"epochs\" is confusing and not clear to me what is it? Do you want to refer it to \"iterations\"?\n\n3. L196, how dense point clouds are used in the manuscript?\n\nLIMITATIONS:\nYes, the authors provide a limitation section and make sense to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a system to represent a large scene using NeRF given drone-captured photos. It partitions a large scene into overlapping smaller tiles, and represent each tile with a sub-NeRF. To enable real-time rendering, it represents each sub-NeRF using meshes and neural textures (represented as a encoder-decoder network) during training; after training, meshes are further simplified, and the encoder part of neural textures are converted to texture image on request, while the decoder part is implemented as a GLSL shader. The whole real-time rendering system for the proposed NeRF representation is built on top of Unreal Engine 4, and allows for scene editing.\n\nSTRENGTHS:\nThe proposed method not just trains fast, but also renders fast. It's cool to see the whole NeRF real-time rendering system integrated into the existing renderer, Unreal engine 4.\n\nWEAKNESSES:\n1. Writing could use some improvements; many key technical details seem missing, making the paper a bit hard to understand.\n* 1.1 The frame rate of real time rendering is reported on what device?\n* 1.2 Line 129-132 says that \"we train a coarse model for sub-region segmentation...\"; what model was trained here? Is it counted in the total training time of the proposed method?\n* 1.3 Line 143 says \"it generates 8D feature vector, which incorporates opacity information\"; this seems very confusing to me, as in Eq. 1, the opacity is predicted alongside the feature vector.\n* 1.4 Line 167-169 seems to suggest that max should be used in Eq. 5, as opposed to min?\n* 1.5 Octahedron with 20 faces. Might be better to explain why there're 20 faces instead of regular 8 faces.\n* 1.6 Fig 5 village scene, the lower-right inset of ours does not seem to align with those of baselines.\n* 1.7 How is the comparison with baselines with MipNeRF and InstantNGP performed? Were the baselines trained on each individual blocks or they were trained on the whole scene?\n* 1.8 Line 248-254: alpha values seem to be dropped during real-time rendering. If so, please make it explicit, and include examples how such dropping affects the final rendering quality.\n* 1.9 Fig. 1c seems confusing without labels \"before editing\" and \"after editing\". In addition, how was the inpainting performed when the building/car were removed? \n* 1.10 I find it hard to understand what's going on in line 181-185. The authors might want to add some illustrations. \n* 1.11 Line 52-54 seems to suggest that the speed up of training, compared with MobileNeRF, is due to implement some compute-heavy components using customized cuda functions. What are these components?\n\n2. Ablations and comparisons can be improved; it's hard to judge if the proposed components are useful or not, and how it compares against baselines. \n* 2.1 The BungeeNeRF work seems to a better baseline than MipNeRF; but for some reason, comparison with that work is missing.\n* 2.2 Why not compare with MobileNeRF on each tile? Line 135-137 claims that the proposed mesh initialization addresses the unstable convergence issue of MobileNeRF; is there any intuitive explanation or empirical evidence to support this claim?\n* 2.3 Ablations are needed to justify the effectiveness of the loss function in Eq. 6.\n\nQUESTIONS:\n1. It seems that the proposed method is a bit complicated during training phase. Will the training code be released for reproducibility upon acceptance?\n\n2. Would the proposed method also work for blender, besides the unreal engine 4?\n\nLIMITATIONS:\nLimitations are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method that combines NeRF and the Unreal Engine for real-time rendering of large-scale scenes. The method first partitions large scenes into sub-blocks, and represent NeRF via polygonal meshes initialized from regular octahedron. The opacity and feature vector are represented via hash-encoding. The mesh, opacity, and features are optimized during training. Inspired by LOD, the method trains meshes with different levels of detail to improve rendering efficiency at different scales. The optimized mesh can be integrated into the Unreal Engine 4 to achieve real-time rendering of large-scale scenes. Experiments demonstrate that the proposed method achieves better rendering quality than existing approaches and supports real-time rendering.\n\nSTRENGTHS:\n- How to achieve real-time rendering of large-scale scenes under the NeRF setting is an important problem. This paper presents, to the best of my knowledge, the first solution to this problem. The main idea is to represent the NeRF via polygonal meshes so that they can be combined with the rasterization pipeline in Unreal Engine after optimization. Several other designs are introduced such as block partition and LOD representations. The overall design is reasonable.\n\n- The experimental results are impressive. UE4-NeRF shows clearly better rendering quality than other baselines such as Mip-NeRF and Instant-NGP. It is also the only method that supports real-time rendering of 2k large-scale scenes.\n\nWEAKNESSES:\n- Some baselines are lacking.\n  - The closest method to the proposed UE4-NeRF is Mobile-NeRF, as it also represents NeRF via polygonal mesh. However, the comparison to Mobile-NeRF is missing. According to line 53, UE4-NeRF is faster than Mobile-NeRF because it implements computationally intensive portions with CUDA. However, this advantage comes from different implementations rather than the method itself. Authors should compare the two methods in a fair setting to demonstrate the benefits of the techniques proposed in this paper.\n  - I suggest authors to also compare with Mega-NeRF (Mega-nerf: Scalable construction of largescale nerfs for virtual fly-throughs), which also studies building NeRF for large-scale scenes.  \n\n- The writing lacks clarity. For example:\n  - For the initialized mesh, line 10 mentions \"octahedra\" (i.e., 8 faces), line 55 mentions \"tetrahedra\" (i.e., 4 faces), while line 137 mentions \"octahedron with 20 faces\". These terms are inconsistent and make it very confusing which one is correct.\n  - At Eq.(1), it is not explained what is $P_i$. Is it the 3D coordinate of the point of intersection? Please clarify.\n  - At line 142 \"encoded by multi-resolution hash functions\", authors should mention that this follows Instant-NGP and cite it.\n  - At line 208, what is \"acceleration grid\"?\n\n- In Fig.5, not all baselines are included. Authors should provide the qualitative results of other baselines in the supplementary material. The ground truth is also missing.  \nBesides, I noticed that the camera poses for different methods are not exactly the same. The camera poses should keep consistent in qualitative comparison.\n\n- Grammar:\n  - Line 217: \"we incorporate parallel lights from various angles are added above.\" There are two verbs in this sentence.\n\nQUESTIONS:\n- Authors may respond to my concerns in the weaknesses.\n\n- At line 290, it is mentioned that \"the actual rendering quality of UE4-NeRF is expected to surpass the metric’s performance, as achieving consistent exposure matching with the original image is challenging within the Unreal Engine environment.\" Then I suggest authors also report the performance using original volume rendering to remove the exposure gap.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UE4-NeRF, a system that combines Neural Radiance Field (NeRF) with the Unreal Engine 4 (UE4) for real-time rendering and editing of large-scale 3D scenes. To achieve this, the system partitions scenes into sub-NeRFs and represents them using optimized polygonal meshes based on regular octahedra and tetrahedrons. Leveraging a Level of Detail (LOD) approach and the powerful development capabilities of UE4, the system enables high-performance rendering at different observation levels and seamless scene editing. Experimental results show that UE4-NeRF achieves rendering quality comparable to state-of-the-art methods. and also  accelerates the training speed.\n\nSTRENGTHS:\n- This paper describes a method that addresses the scalability limitations of NeRF by dividing the scene into smaller chunks and enabling parallel processing. This allows for the construction of large-scale NeRF models with reduced memory requirements and increased computational efficiency.\n\nWEAKNESSES:\n- Overall results: The results present an aerial view of an urban scene with good resolution. However, it is not clear how the authors leverage NeRF's capabilities in this scenario. Specifically, a comparison with traditional Multi-View Stereo (MVS) should demonstrate view-dependent effects, subtle details, transparent objects, and how it performs compared to multi-view methods. Additionally, it would be beneficial to include a first-person wandering case similar to Urban NeRF[32].\n\n- Contribution: From my understanding, this method combines previous work on Urban NeRF[32] scene partitioning with Instant-NGP. While it has its merits, compared to Instant-NGP, the qualitative improvement in the results is marginal overall.\n\n- Experiments: The authors have compared their method with their own dataset, which is commendable. However, it is encouraged to include comparisons with established datasets such as the one used in Mega-NeRF[40]. Additionally, qualitative and quantitative comparisons with Mega-NeRF[40] would provide further insights.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nBased on the concerns raised, the marginal qualitative improvement compared to Instant-NGP, lack of comprehensive comparisons with established datasets, and unclear demonstration of NeRF's capabilities in the urban scene, it is recommended to borderline reject the paper unless relatively significant revisions are made.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-14"}