UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene
Authors: Jiaming Gu
Minchao Jiang
Hongsheng Li
Xiaoyuan Lu
Guangming Zhu
Syed Afaq Ali Shah
Liang Zhang
Mohammed Bennamoun
Venue: NeurIPS 2023 poster (2023)
Keywords: NeRF
UE4
Large-scale scenes
Real-time rendering
Rasterization

Abstract: Neural Radiance Fields (NeRF) is a novel implicit 3D reconstruction method that shows immense potential and has been gaining increasing attention. It enables the reconstruction of 3D scenes solely from a set of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, still has significant limitations. To address these challenges, in this paper, we propose a novel neural rendering system called UE4-NeRF, specifically designed for real-time rendering of large-scale scenes. We partitioned each large scene into different sub-NeRFs. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and  the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from Level of Detail (LOD) techniques, we trained meshes of varying levels of detail for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Rendering within UE4 also facilitates  scene editing in subsequent stages. Furthermore, through experiments, we have demonstrated that our method achieves rendering quality comparable to state-of-the-art approaches. Project page: https://jamchaos.github.io/UE4-NeRF/.

UE4-NeRF:Neural Radiance Field for Real-Time Rendering of Large-Scale Scene
Jiaming Gu Minchao Jiang
Hongsheng Li! Xiaoyuan Lu² Guangming Zhu
Syed Afaq Ali Shah Liang Zhang Mohammed Bennamoun"
School of Computer Science and Technology, Xidian University
Algorithm R&D Center, Qing Yi (Shanghai)
Edith Cowan University
The University of Western Australia
j xylu@bnc. org. cn, afaq. shwh@ecu. edu. mohammed bennamoun@uwa,ccmmmm edw.
com, {jamchaos, hsli)@stu. xidian. edu. on {gmzhu, liangzhang) exidian.c. edu. cn
Abstract
Neural Radiance Field (NeRF) an implicit 3D reconstruction method that has shown immense potential and has gained sigmificant attention for its ability to reconstruct 3D scenes solely from a of photographs. However, its real-time rendering capability, especially for interactive real-time rendering of large-scale scenes, has sigmificant limitations. To address this challenge, we propose a novel neural rendering system called UE4-NeRF that is designed for real-time rendering large-scale scenes. Our proposed approach partitions large scenes into sub- NeRFs, and uses polygonal meshes to represent them. In order to represent the partitioned independent scene, we initialize polygonal meshes by constructing multiple regular octahedra within the scene and the vertices of the polygonal faces are continuously optimized during the training process. Drawing inspiration from the Level of Detail (LOD) techniques, for different observation levels. Our approach combines with the rasterization pipeline in Unreal Engine 4 (UE4), achieving real-time rendering of large-scale scenes at 4K resolution with a frame rate of up to 43 FPS. Our experimental results demonstrate that method attains rendering quality par with fate-offthh--ee approaches, while additionally offering the advantage of real-time performance. Furthermore, rendering within UE4 facilitates scene editing in subsequent stages. Project page: https://jamchaos wwwphhhh.s.gggtbiioooooooo gwtthh.io/UE4--N.
train meshes with varying levels of detail
Introduction
Real-time interactive rendering capability for 3D large-scale scenes is crucial tool in creating digital worlds for applications such as virtual reality (VR), computer games, and the Metaverse. Neural Radiance Fields (NeRF) [23] is pioneering method for 3D reconstruction and synthesis of novel views, which different from traditional 3D reconstruction techniques that represent scenes using explicit expressions such as point clouds, grids, and voxels [ [4 1 NeRF samples each ray and retrieves the 3D location of each sampling point and the 2D viewing direction of the ray. Subsequently, these 5D vector values are fed into neural network to determine the color and volume density of each sampling point. NeRF constructs field parameterized by an MLP neural network [16 34 to continuously optimize parameters and reconstruct the scene. NeRF has demonstrated remarkable
"Contribute equally
'Correponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Misscwsing Ouine saacer
Anaere Ale car
(b) real-time rendering of large-scale scene in UE4
(n) captured by UAV
(c) edition
Figure 1: (a) Large-scale scenes characterized by complex textures and significant height differences are captured with monocular or multi nulti-camera drone footage. We divide scene into multiple individual (ub-NeRFs.(b) Real-time interactive rendering in UE4 with distinct levels of detail, along with frame rate disparities among them. (c) Defferent editing capabilities on the scenes.
performance for 3D reconstruction when provided with a set of posed camera images 29 2 23 but early NeRF works 2 (23 had only focused on small-scale scenes and object-centric reconstruction due to the model computational complexity. Although Block-NeRF [37] and BungeeNeRF [43 capable of reconstructing large-scale environments, they still unable to achieve real-time rendering. Traditional NeRF obtains the color of each pixel through volume rendering, but this rendering process is far too slow for interactive visualization and compromises high fidelity. Recently, Mobile-NeRF [6] combined with classic polygon rasterization pipeline has shown to achieve real-time rendering on various devices. However, training just small-scale scene incurs significant overheads, such as training time and GPU resources. It requires 8 v100 GPUs to train the model successfully over several days. Additionally, due to opacity binarization, Mobile-NeRF cannot handle outdoor scenes with a variety of materials, including transparent or semi-transpper glass and plastic shed. Performing subsequent editing on the scenes generated by NeRF is another meaningful but challenging task, and previous works [13 44 [9] required training complex neural networks to achieve this capability. These methods are time-consuming and are limited in their editing capabilities.
objects such as water,
This paper introduces NE4-NeRF to address the aforementioned issues regarding large-scale scene rendering and editing. UE4-NeRF combines NeRF with the Unreal Engine 4 (UE4) [17 30 17 to achieve real-time interactive rendering of large-scale scenes with the scene editing ability. As shown in Figure Ta) to achieve this, large-scale scene such as industrial parks with large height differences, roads and farmlands are partitioned into separately trained blocks. The size of individual scenes captured by drones is in the range of several square kilometers. The multi-resolution hash encoding proposed in Instant-NGP (25) is employed for training acceleration. In order to efficiently utilize the parallel computing capabilities of GPUs, the computationally intensive portions are implemented using CUDA. Through these efforts, the training speed of UE4-NeRF is 1000 faster than Mobile- NeRF. For each block, we represent the scene using textured polygonal meshes obtained from regular octahedrons. Similar Mobile-NeRF, the texture atlas stores feature vectors and opacity information. We utilize an MIP as the High-Level Shading Language(HI fragment shader in UE4, which takes the feature vectors and outputs colors. Furthermore, to ensure real-time rendering performance for different observation heights viewpoints, we adopted method akin to the Level of Detail (LOD) 8 21 approach commonly used in computer graphics, where the complexity of 3D model representation is quantified in terms of metrics such as geometric detail and texture resolution. Figure demonstrates that UE4-NeRF is capable of achieving real-time rendering at different relative observation heights by training different levels of polygon meshes, where coarser meshes are used for relatively higher observation heights. One of the other advantages of combining NeRF with UE4 is the powerful development capabilities of the rendering engine, allowing further processing of the final scene rendered in UE4. Users can freely combine different sub-NeRFs and add objects to the scene needed. Moreover, object manipulation, as illustrated in Figure [e) is also supported.
In summary, our paper presents the following contributions:()) We propose novel UER-NeRF technique which has 1000 times faster training times than MobileNeRF, while maintaining higher

---

quality and smaller storage overhead:(2) The demonstration of the capability of UE4-NeRF to reconstruct large-scale scenes in real-time and showcase the practical viability of (eRF:(3) The integration of NeRF and UE4 facilitates editing and manipulation of 3D scenes.
2 Related work
Large Scale 3D Reconstruction. The original NeRF algorithm was limited to object-centric scenes or scenes with boundaries. NeRF++ [46] expanded upon NeRF by introducing an inverted sphere parameterization to handle unbounded scenarios. Mip-NeRF 360 [ addressed the challenges associated with unbounded scenes through the use of non-linear scene parameterization, online distillation and a novel distortion-han haased regularizer. However, was observed that simply expanding the scene without adjusting the model capacity, i.e., using fixed-size MLP, can lead to unstable training and loss of high-fidelity. Meanwhile, naively increasing the model capacity can significantly increase the training cost. BungeeNeRF [K] proposed a progressive approach to build and train large-scale scene models, which promoted layer specialization and unleashed the power of positional encoding over the full frequency range. Block-NeRF (37) and Mega-NeRF [40] spatially partitioned a large scene into several individually trained sub-NeRFs. They also incorporated appearance embeddings like NeRF-W 22 to handle appearance variations in the scene. Similar to Block-NeRF, our approach partitions the large-scale scene into sub-NeRFs, and trains a coarse scene model to assist in accurately delineating each sub-NeRF. However, approach is only applicable to the grid structure like Block-NeRF, but is also generally suitable for various rypes of scenes, e-g. mountainous terrain. Additionally, contrast to the aforementioned NeRFss designed for large-scale scenes, our method accomplishes real-time rendering.
Rendering of Neural Fields. NeRF has tremendous potential for rendering and has gained attention for 3D reconstruction from set of posed camera images [29 2 23]. However, the computational
overhead of NeRF limits its practical applications. Some prior works 25
11 have addressed training
costs, while others have focused on rendering ( 36 27 31. SNeRG 20 and PlenOctrees [45]
improved performance by baking the components of NeRF. However, these methods do not take sufficient advantage of the texture- to store volumetric textures. MobileNeRR (6) is variation of NeRF that can run in real-time variety of common mobile devices. To render an image, Mobile-NeRF utilized the classic polygon rasterization pipeline with Z-buffering to produce feature vector for each pixel and then passed it to a lightweight MLP running in a GLSL fragment shader to produce the output color. In this work, we accelerate the rendering of NeRF by converting the learned knowledge into a mesh representation for efficient mesh rasterization pipelines. Moreover, we achieve a significant milestone by successfully implementing NeRF for real-time interactive rendering of large-scale scenes, which was previously unexplored.
rasterization pipeline and require significant GPU memory
Manipulation and Composition of NeRF Practical applications of 3D representations require effective manipulation and composition capabilities. While, explicit 3D representations, such as meshes and voxels, inherently support editing and composition. becomes challenging for neural network- implicit representations such as vanilla NeRF [23). NSVF (20) uses a sparse voxel octree for modeling, employing voxel embeddings for each scene while sharing the same MLP for predicting density and color. However, the flexibility of this approach is somewhat constrained. GIRAFFE (28), combined with GAN [49 18, enabled the addition of multiple objects in a scene, expanding from single-object to multi-objed generation, even in the absence of such objects in the training data. Plenoxels [II] utlizes explicit sparse voxel representation for direct object composition of different objects, but suffers from large storage requirements of the dense index matrix. None of these methods support object manipulation to achieve large-scale scene rendering, enabling users to freely add and manipulate objects within the UE4-NeRF environment. For instance, users can use a broomstick to fly or drive a car within the rendered environment powered by UE4-NeRF.
3D scene. In contrast, our approach leverages UE4
3 Proposed Method
Our novel method involves representing large scenes as polygon meshes, where the positions of mesh vertices are iteratively optimized during the training phase. The encoding network generates texture
3

---

Train
Block Partttion
Initaal Mesh
Multi-rcsllation egooding
Netwonk
Loss
Operio
Rundering
Spaciece Hormaniiz
Photomotrcc Lun Depek
Pre-Rendering
Mesh
Infor
Netural Shader
Block Fiste
Dyaanie bluck
Encoder
|Render
Founur Map
Figure 2: Overview of UE4.NeRE. UE4-NeRF consists of three modules: (1) In the training module, we partition the sub-NeRFss and initialize the grids each partitioned scene. During the training process, we continuously optimize the parameters of the Incoder-Decodde network and the vertex positions of meshes; (2) Through the pre-rendering module, we extract the polygonal meshes at different levels of detail that will be used for the final rendering: (3) The rendering module consists of an inference submodule and an UE4 submodule, which communicate with each other to achieve the final real-time rendering.
maps that store opacity and feature vectors, essential for rendering. To accomplish real-time rendering of large scenes, we combine the pre-rendered polygon meshes with the rasterization pipeline available in Unreal Engine 4 (UE4). In the subsequent subsections, we present a detailed and comprehensive description of entire procedure.
3.1 Training
3.1.1 Block Partition
As depicted in Figure 2 large scene is partitioned into multiple blocks. To ensure smooth transitions at the boundaries of the model, each training area, indicated by dashed lines, is slightly larger than its corresponding scene area, with overlapping borders with neighboring blocks. We train coarse model for sub-region segmentation, focusing on capturing the overall terrain and building outlines. We then filter the images associated with the sub-regions and mask exclude pixels outside the scene area.
3.1.2 Optimization
We begin by initializing 128 128x 128 grid, and selecting the center point of each grid along with its six neighboring grids (front, back, left, right, top, and bottom) to create polygonal meshes. To address the slow and unstable convergence issue encountered by Mobile-NeRF when dealing with tilted surfaces, we utilize a regular octahedron with 20 faces, including the 8 exterior faces and 12 interior faces. This illustrated in the "Initial Mesh" section Figure 2 The ray emitted from the camera origin to the pixel is defined r(t) + (d and d = (0, .) is the viewing direction of the ray. For each ray emitted from the camera origin to the pixel, we calculate its intersection with the polygonal meshes. The point of intersection serves as the sampling point for that ray. For each block, we define an
network:
((hash(p):: Mj
(1)
(MM,,H((d))
(2)
The encoder network, denoted MLP E, takes positional information(p:) encoded by multi- resolution hash functions 25) as input. generates an 8D feature vector M,, which incorporates opacity information. On the other hand, the decoder network D utilizes M; and the ray direction as inputs and predicts the color of the sampling point. The direction of the ray encoded using Spherical Harmonics.

---

We use the traditional NeRF volume rendering method (Eq. to obtain the predicted color C(r). Due to the complexity of calculating the distance between sampling points, we simplify the process by fixing the distance value. The opacity is then directly predicted instead of the volume density.
N
C(r) = Σ Taaci T (1-
J=0
Unlike the traditional photometric loss, our proposed photometric loss consists of two components.
The first component referred to as level-1 photometric loss is calculated using Eq. 4 It quantifies the mean squared error between the predicted colors and the ground truth colors of the corresponding pixel.
Crge vp) = Σ C(r)
- -(D)
PER
where V is the position of the vertices of the polygonal mesh under the corresponding level and R is the set of rays in each batch. C(r) represents the ground truth RGB colors for ray
r. In addition, to improve the proximity of the triangle meshes to the object surface, we design a second component the photometric loss, de- noted as cpart? As depicted in Figure 3, in contrast to the previous sampling rule, we only select sampling points with opacity greater than a threshold f. Furthermore, the volumetric ren- dering process is halted when the accumulated opacity exceeds 0.8, indicating that the remain- ing opacity T is less than 0.2. In the initial 10,000 epochs, the threshold f is maintained at O, and afterward,
T982
Figure 3: Illustration of sampling details. Our approach involves selectively sampling intersec- tion points that are located above threshold value f and terminating the volume rendering process when the remaining opacity drops below 0.2. This strategy effectively concentrates opacity on a lim- ited number of specific meshes and compresses towards the object, leading to improved rendering results.
shown in Eq. S the f in- creases as the number of epochs increases, but does not exceed 0.3.
f = m0.3 epoch 10000 2
x 0.012]
(5)
10000
The photometric loss for the second part of the level-1 can be obtained using the following formula:
mb= Ty Cs
Cpart2 Vp)
Σ C - C(r) C'(r) FR
(6)
T
Where (r) is obtained through volume rendering under the aforementioned sampling scheme. To cover the entire color range, we divide the expression by - T as shown in Eq. since our volume
rendering stops when the remaining opacity less than 0.2. We obtain the final photometric loss of the level-1 as follows:
C (0, Vip) + (1 100--) C) (0, Vp), = 0.8
spack
During the initial 10,000 epochs, the second component of the photometric loss does not have any impact. This is done to align with the traditional volume rendering approach and adhere to the modeling principles of NeRF. However, as the training progresses, the weight for the second component gradually increases. This enables the preservation of opacity only specific meshes and ensures consistency with the later stages, where only meshes with opacity greater than 0.3 are exported for rasterization.
After 10000 epochs, a process is initiated where we randomly select sampling points within each grid, calculate the opacity of each grid, and select the vertex of the grid with the highest opacity among the eight grids as the vertex of the merged grid. This allows us to generate grids 64x64x 64, 32x32x32, 16x 16x 16, and 8x8x8 x8 sequentially. Each grid level is tailored for specific observation heights, enabling us minimize the rendering cost of the scene. It is important to note that for the same block, different levels always use the same network, but the gradient in the rough levels are
5

---

only used to update mesh vertex coordinates. This measure can prevent coarse-scale meshes from providing undue supervision to shared networks when they cannot represent fine object structures. The final loss function of the photometric part is calculated as follows:
Lrgb Σ 3 h v.) + (1 3) Ligh (0, Vp))
(8)
1=1
as a supervisory signal has demonstrated the ability to enhance the convergence and improve the quality of reconstructions. However, when operating under the influence of ambient light in outdoor environments, RGB-D cameras are susceptible produce inaccurate depth measurements and have limited measurement ranges. In our approach, we propose to use the pseudo-depth as the supervision information for NeRF, In conventional NeRF, cenetturr-fro-motiiee typically utilized to estimate the camera poses. As part of this process, a sparse point cloud can be obtained. By aligning the camera and point cloud in the same coordinate system, we calculate the seeudo-depth by measuring the distance between the sparse point cloud and the camera position. For detailed explanation of the pseudo-depth loss (LD) calculation, please refer the supplementary material. The final loss L of the model is calculated as follows:
Pseudo-depth. Utilizing depth information [O 33
(SFM) [35] solvers such COLMAP [
39 12
are
C = Ergb + LD
Transient Objects. Moving/dynamic objects, such pedestrians and cars, have the potential to cause inconsistency in the observed scene across different viewpoints and can also create gaps or "holes" in the depth map. To address this challenge, we integrate semantic segmentation model 7 48) into our training process. By leveraging this model, we generate masks that identify the regions occupied by the moving objects, effectively excluding them from the training of the scene. This strategy mitigates the impact of moving objects on the scene's 's depth map, and thus improves the overall fidelity and generalization performance of the model.
3.2 Pre-rendering
To optimize computational efficiency, we lever- age the predicted values from the acceleration grid to identify and remove meshes that do not possess significant geometric surfaces. Once the training phase is complete, we obtain the network weights well the meshes with optimized vertex positions. Subsequently, we perform the pre-rendering process. The overall workflow is illustrated in Figure First. for each block, in addition the camera view used for training. we incorporate parallel lights from various angles above. We compute the intersec- tion of each ray with the polygon mesh, and the traversal of light halted when the cumulative opacity surpasses 0.8. A triangular face will be clipped if the opacity of all intersection points on that face below 0.3, resulting in the retention of only those meshes that meet this criterion. From this, we obtain the final polygon mesh representations of the scene. Based on our statistical analysis, the final extracted meshes account for only 5% of the original mesh count. This significant reduction effectively improves mesh utilization, inference speed during rendering, and reduces storage costs. Secend. we perform coordinate transformation to ensure the obtained mesh aligns with real-world scale. Third, we map the vertex coordinates of the triangles to their corresponding positions in UV coordinates.
Raysy
Transfore
canrdbaam
Calosae
Pre-rondering
Figure 4: Structure of the pre-rendering, We em- ploy multi-viewpoint approach extract polygo- nal faces, generating multiple rays from different perspectives. Subsequently, we apply scaling trans- formations and calculate UV coordinates to further process the extracted data.
3.3 Rendering
After training, we obtain the weights of the Encoder-Decode network, and refined polygonal meshes different levels through vertex optimization for each indivicual block. Subsequently, we perform pre-rendering and extract the final meshes. For each polygonal mesh perform rasterization using

---

Table 1: Details of the real-world scene Table 2: Quantitative Comparisons with Existing captured by UAV. The memory requirement Methods. Our method achieves remarkable results in increases with the size of the scene, while terms of both rendering quality and speed. Among the the variation in FPS is primarily influenced compared methods, UF4-NeRF requires the least GPU by the materials present in the scene itself. memory. The training costs are also provided.
Host
(emory(4k)
GPU
)
2k
FPS
4k
CA
Methaal
PSNR
SSIM
LPIPS.
FPS(2K)
Menorr (GB)
Host
GPU
iine/miss)
12GB
240
120mm
NaRPaato 33
Ine 23
22.00
20.99
0.663
0.631
0.380
0.426
0.5
II
~e 10
~ 14
~ 32
3 65
15
CS
25GB
11GB
420x240
66
43
70m
Mega NeRT 50
17.57
0:23
0.546
X
-160
MIGB
14GB
600x600
120m
Mobile Noki A
16.92
0:23
0.419
X
-2500
UE4-NeRF
25.03
0,764
0.267
85
~ 19
00
texture map of size x2. However, storing texture maps for numerous polygonal meshes in a large- scale scene can lead to significant storage overhead. To address this, we utilize dynamic inference instead of directly storing all texture maps, as done in Mobile-NeRF (6. Real-time rendering is achieved through continuous and efficient communication between the inference submodule and the UE4 submodule. During the block composition process, since the large-scale scene is accurately divided into multiple blocks and scene fusion is also considered during training, transitions between different blocks in large-scale scene are smooth and interpolation not required at the junction.
Texture map. Prior rasterizing the faces of the polygonal mesh, we perform sampling on each mesh and use the encoder network to obtain 8 channels feature vectors. These 8 channels along with the opacity a obtained are processed to generate nine single-channel texture maps using BC4 compression [26). Compared to the original texture map, this reduces 2/3 of the GPU memory consumption during rendering without significantly compromising the image quality. Please note that texture maps do not store colors, but rather spatial features, which also contain transparency information. Taking into account the principle of locality, we employ a strategy where we collectively infer and store information from nearby regions into texture maps. This enables us to directly access the texture maps without the need for re-inference when traversing these regions again.
Rasterization. We utilize the traditional rasterization pipeline in UE4 to rasterize each face of the polygonal mesh. Subsequently, the decoder network is applied to convert the 17D features of each pixel into RGB colors (see UE4 submodule in Figure 2 The 1 17D features comprise 8 channels of learned features and a 9D view direction, which is encoded using spherical harmonic functions 24]. The decoder network implemented as HLSL fragment shader of the UE4, which is called Neural Shader. To achieve translucent effects without the need for volume rendering, we employ alpha-dithered
42 and temporal anti-aliasing, which offers a relatively low-overhead approach.
Dynamic inference. The interaction between the UE4 submodule and the inference submodule is depicated in Figure When a block requiring rendering is encountered, the UE4 submodule supplies the block and level index information, while the inference submodule provides the corresponding regional meshes and feature maps. In cases where feature maps are not readibly for a specific region, the inference operation is carried out to generate them. This communication mechanism ensures efficient and prompt processing between these two submodules.
4 Experiments
4.1
Implementation Details
Similar to the Instant-NGr approach, we employ multi-resolution hash encoding and construct an Encoder network comprising a 4-layer MIP (32x64,66 (2,664444x4 66,,64x64,66 and an 8-dimensional feature vector. Additionally, we develop Decoder network with 3-layer MLP (17 x 16,16x 16 16, 16x3) to predict the final color of the sampled points. For each block, we train the model using one Nvidia RTX 3090 GPU for 80,000 epochs to achieve convergence, with an approximate duration of 40 minutes.
to estimate the opacity
Datasets and Metric. Our objective is to enable real-time rendering of large-scale scenes with complex textures and materials. Table I presents three distinct scenarios, namely (FLm)
7

---

Famitan
Corastacioon
BYcAAA pants
NeRFacto
Mip-NeRF
Instant-NGP
UE4-NeRF
Figure 5: Qualitative Comparisons with Existing Methods. We visually compare our proposed technique with recently implemented real-time rendering NeRF models [2 38 and Mip-NeRF [3] All these methods are implemented within nerfstudio [38]. It is evident that UE4-NeRF yields the best reconstruction for all the three scenes.
Trush
Mohile-NaRF
Tnut
Mage-NoRF
LIF-NNRRF
Figure 6: Qualitative Comparisons with Mega-NeRF and Mobile-NeRF Detailed comparison with Mobile-NeRE and Mega-NeRF. Since these methods are very tonecconum,ng, we compared them in block or scene.
Construction site(CS) and Industrial parks(IP) which were captured using drones to address the challenges that UE4-NeRF was designed to tackle. Each captured image has an approximate resolution of 6000x4000 pixels and contains GPS information. One of goals is to establish measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. We performed coordinate system transformation prior modeling. In the UF4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. We also performed comparative experiments on the dataset introduced in Mega-NeRF(4O)) At level-1, UE4-NeRF can achieve an maximum frame rate of around 43 FPS 4K resolution. Due the limited battery life of the drone, the captured areas were constrained in size. Nevertheless, subsequent experiments have revealed that limitation does not represent the upper limit of UE4-NeRF. In fact, UE4-NeRF can render even larger-scale scenes in val-time. To evaluate the reconstruction quality and fidelity of our model, we use evaluation metrics such as PSNR/SSIM [41] and LPIPS [47]. We also compare the real-time rendering speed of UE4-NeRF with other models when rendering vast scenes.
4.2 Comparisons with Existing Methods
We performed qualitative comparisons between UE4-NeRF and other NeRF implementations on four scenes captured by UAV, and the results are depicted in Figure 5 Existing tools like NeR- Facto (88 and Instant-NGP (25) aim to achieve real-time rendering, but tend to procude subpar

---

Table Quantitative Comparison with Mega-NeRF and Mobile-NeRE. Mobile-NeRF can not
support complete scene. The maximal resolution supported by Mega-NeRF is 800*800 on RTX 3090.
Training time
Methods
FPS(1x300))
block
scene(n blocks,4x3090)
Mobile-NeRF
48h(4x3090)
nx48h
35(block,2K)
Mega-NeRF
(6h(1x3090)
12h+36hxn/4
60(scene,800x800) 50(scene,2k), 270(block,2K)
UE4-NeRF
40m30s()
1h+44mmnssxn/4
results when applied to large-scale scenes, as evident in Figure 5 Mip-NeRF, despite increas- ing the number of training iterations, still exhibits noticeable blurriness when rendering natural scenes. contrast, UE4-NeRF showcases remarkable accuracy in rendering small objects, such as power lines, in the "farmland" scene. It also excels in handling dense textures, as demon- strated in the "construction site" scene. The "construction site" scene also showcases the ren- dering effects of translucent objects. In Table 2] we provide quantitative comparison of our proposed technique with other existing methods using different metrics. It is important to note that the actual rendering quality of UE4-NeRF is expected to surpass the metric's performance, as achieving consistent exposure matching with the origi- nal image challenging within the Unreal Engine environ- ment. Despite this limitation, UE4-NeRF still outperforms other methods by significant margin and its real-time rendering capability surpasses that of the current state-of- the-art real-time rendering models. In Table 3 we see that Mobile-NeRF takes days to train just one block and requires 4x 3x90iiti GPUs. If it trains the whole scene, it takes two months. For each block, Mega-NeRF needs 36 hours train for 500,000 epochs, while we only need 40 minutes to train for 80,000 epochs to reach convergence. When training the entire scene, Mega-NeRF requires hours of additional time, but UE4-NeRF only needs hour. worth noting that Mega-NeRF generates hundreds of GBs of temporary files during training. And as can be seen from Figure(6aadd Figure compared with Ground truth, UE4-NeRF shows amazing rendering results, but Mobile-NeRF and Mega-NeRF produce significant blur.
Mepa-NeRF
UE4-NoRF
Comparison with Mega- NeRF on the public dataset. Due to the large exposure gap between images in the Mega-NeRE dataset, different shad- ows appear at different viewing angles.
Figure 7:
4.3
LOD render results
Figure g showcases the qualitative rendering results at different levels of detail, along with the corresponding real-time frame rates, resolution of 2K. As the level of hierarchy decreases, the rendering quality and clarity progressively improve. UE4-NeRF demonstrates a high level of fidelity when compared the ground truth. In addition, it be observed that the LOD approach effectively
F05000
5P552
Ground Truth
L5
1.2
Ground Truth
E.2
L.1
Ourr
Ours
Figure 8: Rendering result at different LOD. The red boxes indicate the presentation of the same content different levels. addition to achieving high FPS rendering different LOD in 2K resolution, UE4-NeRF also demonstrates a remarkable photo-realistic.

---

Figure 9: Shading mesh- not textured. Polygonal meshes at different levels are displayed, revealing the varying com- plexity of the scene.
Figure 10: Scene edition. The results of multi-view scene editing are presented. One can carefully ob- serve the occlusion relationship between objects and the scene.
addresses the challenge of real-time rendering at higher levels although it may sacrifice some rendering quality at higher levels for the sake of real-time performance. When comparing materials and frame rates across different scenes, it is worth noting that the presence of semi-transparent objects, such as vegetation(as seen in the last scene of Figure 8), can potentially lead to a decrease frame rate due to increased computational overhead.
Shading mesh. Figure 2 showcases the shading meshes extracted without textures. Despite the coarse appearance of the meshes and the limited level of detail even in the closest level-1, such excavators, the final rendering quality remains impressive. This be attributed to the vertex optimization carried out during the training process. the texture maps encoded in the network outputs, and the deployment of the final neural shader during rendering.
4.4
Scene edition
Editing scenes in UE4-NeRF, especially the composition of different scenes, is scamless process. UE4 supports multiple 3D model file formats, allowing us to import and edit the rendered scenes. The Unreal Engine automatically handles occlusion between the original scene and added objects, making the editing process effortless. We have complete freedom to manipulate the newly added objects according to our needs. In Figure when object obstructs the NeRF-rendered scene, the occluded parts do not require shading by the neural shader. Interestingly, we observed that after adding and manipulating objects, the frame rate of the scene actually slightly improves. This is attributed to the reduction in NeRF computation, resulting in improved frame rates.
Conclusion and Limitations
In this research paper, we introduce UE4-NeRF, real-time interactive rendering system designed for large-scale scenes. UE4-NeRF partitions the scene into blocks and trains a separate NeRF model for each block. Inspired by LOD techniques, different levels of mesh precision are used for different layers, and the NeRF model is integrated with the rasterization pipeline in UE4. Currently, UE4-NeRF stands the only NeRF model capable of achieving real-time interactive rendering of large-scale realistic scenes while maintaining high fidelity. Moreover, its integration with UF4 opens up possibilities of scene editing and manipulation, allowing for the flexible addition and manipulation of traditional 3D models such as .ob and fbr within the scene. Overall, UE4-NeRF presents a comprehensive neural rendering system that combines large-scale, real-time interactive rendering with photo-realism and editability. However, there are still some issues that we aim to address in the future: UE4-Ne TEE4-NeRF requires the direct use CUDA and therefore relies on NVIDIA GPUs. Rendering large scenes, such as a few square kilometers, incurs significant memory overhead when without reducing rendering accuracy, and increasing rendering accuracy further amplifies memory consumption. 2) During the pre-rendering process, it is difficult to capture rays from any viewing angles and extract meshes for each viewpoint, resulting in gaps or holes the final rendered scene. The strategy of pre-rendering can be optimized.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by National Natural Science Foundation of China (N0.62072333.660772255)) The authors would like to express gratitude to the anonymous reviewers who greatly help improve the manuscript. Besides, we sincerely thank Hongli Wang, Ke Zhang and Mengqi Shen for their technical support in drone shooting.
References
John Ashhurner and Karl Friston. Voxel-hased morphometry methods. Neuroimage, 1166)800--21, 2000.
[2] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- Brualla, and Pratul Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5470-5479, 2022.
[4] Matthew Berger, Andrea Tagliasacchi,
Seversky, Pierre Alliez, Gael Guennebaud, Joshua Levine, Andrei Sharf, and Claudio Silva. A survey of surface reconstruction from point clouds. In Computer graphics forum, volume 36, pages 1019229, Wiley Online Library, 2017.
[5] Mario Botsch, Leif Kobbelt, Mark Pauly, Pierre Alliez, and Bruno Lévy. Polygon mesh processing. CRC press, 2010.
[6] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208. 00277, 2022.
[7] Bowen Cheng, Maxwell D Collins, Yukun Zhu, Ting Liu, Thomas S Huang, Hartwig Adam, and Liang- Chich Chen. denoptic-deplab: A simple, strong, and fast baseline for hottom-up panoptic segmentation. In Proceedings of the IELE/CVF conference on computer vision and puttern recognition, pages 12475-12485. 2020.
James H Clark. Hierarchical geometric models for visible surface algorithms, Communications of the ACM, 19(1)5547 554, 1976.
[9] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillet, Dennis Evseev. David Calabrese, Hugues Hoppe. Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoinl video. ACM Transactions on Graphics (ToG), 34(4):1-13, 2015.
[10] Kangle Deng, Andrew Liu, Jun Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12882-12891, 2022.
[11] Sara Fridovich-Kan Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxcls: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5501-5510, 2022.
Computer Society Conference Computer Vision and Pattern Recognition, volume 2, 2007. [12] Yasutaka Furukawa and Jean Ponce. Accurate, dense, and robust multi-view stereopsis (pmvs). In IEEE
[13] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser: Object-centric neural scene rendering. arXiv preprint arXiv:2012.08503. 2020.
[14] Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence, 43(22): 4338 4364, 2020.
[15] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
[16] Kurt Hornik, Maxwell Stincheombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-36 1989.

---

[17] Brian Karis and Epic Games. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, 4(3):1, 2013.
[18] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[19] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editable feature volumes for scene rendering and manipulation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4340-4350, 2023.
[20] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 33: 13:11515--16665 2020.
[21] David Luebke, Martin Reddy, Jonathan D Cohen, Amitabh Varshney, Benjamin Watson, and Robert Huebner. Level of detail for 3D graphics. Morgan Kaufmann, 2003.
[22] Ricardo Martin-Brualle Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210-7219, 2021.
[23] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM. 65 (1):99-106, 106, 2021.
[24] Claus Müller. Spherical harmonics, volume 17. Springer, 2006.
[25] Thomas Müller, Alex Evans, Christoph Sehied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41(:):1-1, 2022.
[26] Jac-Ho Nah. Quickete2: Fast etc2 texture compression using luma differences. ACM Transactions on Graphics (TOG), 39(6):1-10, 2020.
[27] Thomas Neff, Pascal Stadlbauer. Mathias Parger, Andreas Kurz, Joerg H Mueller. Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards sal-time rendering of compact neural radiance fields using depth oracle networks. Computer Grapkics Forum, volume 40, pages 45-59. Wiley Online Library, 2021.
[28] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453 1464, 2021.
[29] Keunhong Park, Utkarsh Sinha. Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[30] Weichao Qiu and Alan Yuille. Unrealcv: Connecting computer vision to unreal engine. In Computer Vision ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 16, 2016, Proceedings, Part III 14, pages 909 916. Springer, 2016.
[31] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14335-14345, 2021.
[32] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 12932-12942, 2022.
[33] Antoni Rosinol, John Leonard, and Luca Carlone. Nerf-slam: Real-time dense monocular slam with neural radiance fields. arXiv preprint arXiv:2210.16441, 2022.
[34] Franco Scarselli and Ah Chung Tsoi. Universal approximation using foedforward neural networks: A survey of some existing methods, and some new results. Neural networks, 11(1):15-37, 1998.
[35] Johannes L Schonberger and Jan- Michael Frahm. Structure-frone motion revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1104-4113 2016.
12

---

[36] Vincent Sitzmann, Semon Rezchikov, Bill Freeman. Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. Advances in Neural Information Processing Systems, 34:19313-19325, 2021.
[37] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nert. Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5248-8258, 2022.
[38] Matthew Tancik, Ethan Weber, Evonne Ng. Ruilong Li, Brent Yi, Justin Kerr, Terrance Wang. Alexander Kristoffersen, Jake Austin, Kamyar Salahi, et al. Nerfstudio: A modular framework for neural radiance field development. arXiv preprint arXiv:2302.04266 2023.
[39] Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment modemn synthesis. In Vision Algorithms: Theory and Practice: International Workshop on Vision Algo- rithms Corfu, Greece, September 21-22 22, 1999 Proceedings, pages 298 372. Springer, 2000.
[40] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerts for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12922-12931, 2022.
[41] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visihility to structural similarity. IEEE transactions on image processing, 13(4)x600-612, 2004.
[42] Peter L Williams, Randall J Frank, and Eric C LaMar. Alpha dithering to correct low-opacity bit compositing errors. Lawrence Livermore National Laboratory Technical Report (CCRL-ID-133185 2003.
[43] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, and Dahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In Computer Vision-ECCCY 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 106-122. Springer, 2022.
[44] Bangbang Yang. Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In Proceedings of the JEEF/CVF International Conference on Computer Vision, pages 13779-13788, 2021.
[45] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[46] Kai Zhang, Germot Riegler, Noah Snavely, and Vladien Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
[47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as a perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[48] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[49] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
13


Reviews:
[{"review_id": "z0KFaG9fpo", "reviewer": "Reviewer_x4aP", "summary": "This paper introduces UE4-NeRF, a system that combines Neural Radiance Field (NeRF) with the Unreal Engine 4 (UE4) for real-time rendering and editing of large-scale 3D scenes. To achieve this, the system partitions scenes into sub-NeRFs and represents them using optimized polygonal meshes based on regular octahedra and tetrahedrons. Leveraging a Level of Detail (LOD) approach and the powerful development capabilities of UE4, the system enables high-performance rendering at different observation levels and seamless scene editing. Experimental results show that UE4-NeRF achieves rendering quality comparable to state-of-the-art methods. and also  accelerates the training speed.", "questions": "see above", "limitations": "Based on the concerns raised, the marginal qualitative improvement compared to Instant-NGP, lack of comprehensive comparisons with established datasets, and unclear demonstration of NeRF's capabilities in the urban scene, it is recommended to borderline reject the paper unless relatively significant revisions are made.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- This paper describes a method that addresses the scalability limitations of NeRF by dividing the scene into smaller chunks and enabling parallel processing. This allows for the construction of large-scale NeRF models with reduced memory requirements and increased computational efficiency.", "weaknesses": "- Overall results: The results present an aerial view of an urban scene with good resolution. However, it is not clear how the authors leverage NeRF's capabilities in this scenario. Specifically, a comparison with traditional Multi-View Stereo (MVS) should demonstrate view-dependent effects, subtle details, transparent objects, and how it performs compared to multi-view methods. Additionally, it would be beneficial to include a first-person wandering case similar to Urban NeRF[32].\n\n- Contribution: From my understanding, this method combines previous work on Urban NeRF[32] scene partitioning with Instant-NGP. While it has its merits, compared to Instant-NGP, the qualitative improvement in the results is marginal overall.\n\n- Experiments: The authors have compared their method with their own dataset, which is commendable. However, it is encouraged to include comparisons with established datasets such as the one used in Mega-NeRF[40]. Additionally, qualitative and quantitative comparisons with Mega-NeRF[40] would provide further insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- **In the submitted PDF (Figure 4), we have provided additional qualitative comparisons with MVS.** Our experiments offer comparisons in transparent objects as well as subtle details. MVS utilizes sparse reconstruction to extract feature points, which are then expanded based on morphological and color differences to generate a dense point cloud. This dense point cloud is further used for surface reconstruction, resulting in triangulated meshes. However, due to the dynamic nature of water, the feature points extracted from images taken at different moments and perspectives often lack consistent and mutual matches. Consequently, when using MVS for reconstruction, water bodies may exhibit a substantial number of gaps or holes. Additionally, surface reconstruction methods are not well-suited for handling multiple surfaces, particularly situations involving multi-layered object surfaces due to semi-transparency.\n\n  Within the principles of MVS, there exists a parameter controlling the neighborhood range. Generally, the default neighborhood value prioritizes hole avoidance, which can result in suboptimal modeling effects for object detail structures and smaller objects. In MVS modeling, textured models are created by selecting an appropriate patch from all images to serve as the texture for a triangular face. Consequently, the color observed for this triangular face remains consistent from any angle during rendering. However, this approach works well only for objects that ideally adhere to the diffuse reflection model, particularly those that are opaque. In reality, some objects exhibit highlights and semi-transparency, and MVS-generated textures cannot accurately reflect these characteristics. As a result, rendering visual effects can be subpar. Our approach does not encounter the issues mentioned above. \n\n   Urban-NeRF is trained on a dataset of horizontally captured street views, making it most suitable for simulating first-person wandering along trekker motion routes. In contrast, our training images are taken from a top-down drone perspective, representing an aerial viewpoint. This aerial perspective aligns with the first-person wandering view in our application. Our project's demonstration video showcases first-person aerial wandering. Additionally, we have included our rendering plugin code and model, allowing users to control camera poses and perform real-time rendering independently.\n- Thanks for your comments. Actually, our proposed method is quite different work as combination of Urban NeRF and NGP. The main differences are as followings:\n1. Distinct methods of scene representation. We trained meshes of multiple scales to represent a scene. And effectively combined with the LOD system in UE4.\n2. The rendering principle is different. In UE4, the high-quality feature texture map on the mesh surface is asynchronously inferred and the MLP inference in the real-time rendering pipeline is combined to achieve high-resolution real-time rendering. However, NGP cannot be integrated into rendering engines such as UE4 for use, nor can it be conveniently added to the modeling scene for mixed rendering.\n\n   **Please refer to \"Supplementary Explanation of NGP and Block Strategy.\" in \"Author Rebuttal\" for further discussion**. Through our experiments, simple combination of NGP and block strategy cannot achieve high quality and high frame rate rendering. We just combine the hash grides to minimize the training time for a single epoch. And in large scenes, if real-time rendering is considered, partitioning strategy is basically the only option given limited hardware storage and computing resources. The focus of our paper is not the combination of the choice of block strategy and a certain nerf training method, but the rapid construction of each block area on a single RTX3090-level graphics card in less than 1 hour, and the ability to achieve high resolution Rendering in real time.\n\n    Our method uses multi-scale meshes to represent the scene and greatly optimizes the rendering process, achieving high rendering accuracy, fast speed, and high frame rate. We managed to attain 50 fps at 2K resolution using a single 3090Ti. Moreover, as we further increased the screen resolution to 4K, the main impact was just on the frame rate, with a relatively minor increase in VRAM usage. In summary, our approach extends beyond a mere amalgamation of NGP and block-wise strategies. We exhibit discernible advantages in rendering scene scale, quality, speed, memory utilization, and other relevant aspects.\n\n\n\n\n- Thank you for your encouragement and advice. **In submitted PDF and \"Author Rebuttal\", we conducted a comprehensive comparison with Mega-NeRF**. The focus of our work is on images captured by drones in large-scale scenes with GPS information. One of our goals is to establish a measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. We performed a coordinate system transformation prior to modeling. In the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. Compared with other NeRF models, it is also very meaningful and novel to be able to measure the actual scale in the scene modeled by NeRF. Additionally, the current version of our method relies on GPS data for various processing steps such as block division, point cloud generation from feature points, optimal modeling height calculation, and ground estimation. However, existing public datasets lack GPS information within their images.  In principle, if we remove the reliance on GPS information, our approach is feasible for a variety of datasets. We are in the process of exploring the generalizability of the proposed approach tailored for aerial data, to be applicable across diverse scenarios.\n\n\n"}, {"review_id": "kLReDaHuXI", "reviewer": "Reviewer_JaPB", "summary": "This paper presents a method that combines NeRF and the Unreal Engine for real-time rendering of large-scale scenes. The method first partitions large scenes into sub-blocks, and represent NeRF via polygonal meshes initialized from regular octahedron. The opacity and feature vector are represented via hash-encoding. The mesh, opacity, and features are optimized during training. Inspired by LOD, the method trains meshes with different levels of detail to improve rendering efficiency at different scales. The optimized mesh can be integrated into the Unreal Engine 4 to achieve real-time rendering of large-scale scenes. Experiments demonstrate that the proposed method achieves better rendering quality than existing approaches and supports real-time rendering.", "questions": "- Authors may respond to my concerns in the weaknesses.\n\n- At line 290, it is mentioned that \"the actual rendering quality of UE4-NeRF is expected to surpass the metric’s performance, as achieving consistent exposure matching with the original image is challenging within the Unreal Engine environment.\" Then I suggest authors also report the performance using original volume rendering to remove the exposure gap.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- How to achieve real-time rendering of large-scale scenes under the NeRF setting is an important problem. This paper presents, to the best of my knowledge, the first solution to this problem. The main idea is to represent the NeRF via polygonal meshes so that they can be combined with the rasterization pipeline in Unreal Engine after optimization. Several other designs are introduced such as block partition and LOD representations. The overall design is reasonable.\n\n- The experimental results are impressive. UE4-NeRF shows clearly better rendering quality than other baselines such as Mip-NeRF and Instant-NGP. It is also the only method that supports real-time rendering of 2k large-scale scenes.", "weaknesses": "- Some baselines are lacking.\n  - The closest method to the proposed UE4-NeRF is Mobile-NeRF, as it also represents NeRF via polygonal mesh. However, the comparison to Mobile-NeRF is missing. According to line 53, UE4-NeRF is faster than Mobile-NeRF because it implements computationally intensive portions with CUDA. However, this advantage comes from different implementations rather than the method itself. Authors should compare the two methods in a fair setting to demonstrate the benefits of the techniques proposed in this paper.\n  - I suggest authors to also compare with Mega-NeRF (Mega-nerf: Scalable construction of largescale nerfs for virtual fly-throughs), which also studies building NeRF for large-scale scenes.  \n\n- The writing lacks clarity. For example:\n  - For the initialized mesh, line 10 mentions \"octahedra\" (i.e., 8 faces), line 55 mentions \"tetrahedra\" (i.e., 4 faces), while line 137 mentions \"octahedron with 20 faces\". These terms are inconsistent and make it very confusing which one is correct.\n  - At Eq.(1), it is not explained what is $P_i$. Is it the 3D coordinate of the point of intersection? Please clarify.\n  - At line 142 \"encoded by multi-resolution hash functions\", authors should mention that this follows Instant-NGP and cite it.\n  - At line 208, what is \"acceleration grid\"?\n\n- In Fig.5, not all baselines are included. Authors should provide the qualitative results of other baselines in the supplementary material. The ground truth is also missing.  \nBesides, I noticed that the camera poses for different methods are not exactly the same. The camera poses should keep consistent in qualitative comparison.\n\n- Grammar:\n  - Line 217: \"we incorporate parallel lights from various angles are added above.\" There are two verbs in this sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1: Thank you for your encouragement and advice. The description of \"therefore\" in line 53 of the manuscript is not entirely accurate. The impact is not solely due to the implementation in CUDA; this is merely one contributing factor, albeit not the determining one. Our approach, even without utilizing CUDA implementation, achieves training times of less than 2 hours for a region on a single 3090Ti. With the integration of CUDA implementation, this time is further reduced to less than 1 hour.\nOur training strategy diverges from Mobile-Nerf, and we optimize the method for generating polygonal meshes. **In  our submitted PDF and  Author Rebuttal, we provided both qualitative and quantitative comparisons with Mobile-NeRF.** There are several main reasons for the slow training speed and poor rendering quality of Mobile-NeRF:\n1. The model backbone's inference is slow, resulting in slow training for individual batches. Additionally, each batch demands a substantial amount of GPU memory. With a constrained GPU memory size, further expanding the batch size is not feasible. Consequently, training necessitates a greater number of epochs to complete; \n2. Slow convergence of the grid-like structure. We attempted Mobile-NeRF's grid layout (initially without tilted surfaces), but found that it required more training epochs to achieve a fitting result on tilted roofs (reaching a visual quality similar to our current approach required an additional 20,000 epochs of training, whereas our method only needed 80,000 epochs). Moreover, the rasterization artifacts were noticeable in certain shrub areas (although they improved with further training, it took an additional 50,000 epochs to significantly reduce the visible artifacts). On the other hand, our octahedral grid (including 12 interior triangular faces) handles more complex face-to-face intersections. The average distance from any point in space to the nearest triangular face is smaller than in Mobile-NeRF's grid layout. This results in faster convergence in the vertex optimization part of our approach. \n3. Time-consuming separation of sampling points for triangular faces into strictly transparent and opaque regions. This process contributes significantly to training time.\n4. The absence of depth information supervision also leads to slow convergence. In contrast, our approach benefits from sparse depth supervision, reducing the number of epochs required for training.\n\nA2: **In our submitted PDF and \"Author Rebuttal\", we have conducted a comprehensive comparison with Mega-NeRF.** \n\nA3: The description on line 55 is inaccurate and should be \"octahedron\" instead of \" tetrahedra.\" A regular octahedron has 20 faces in total, **including the 8 exterior faces and 12 interior faces( In PDF Figure 3(c)).**\n\nA4:Yes!\n\nA5:We will further explain the hash function and cite instant-NGP in the revised version.\n\nA6:In the training architecture of \"Instant-NGP\", there exists a 128x128x128 binary dense grid structure designed to efficiently skip empty regions during rendering. We have leveraged this dense grid while calculating the visible grid, allowing for rapid culling of empty regions within the triangular mesh. **In the submitted PDF(Figure 3(b)), we illustrate this method.**\n\nA7:Thank you for your attention to this detail. In the revised version, we will provide qualitative results for other baselines in the supplementary material.\nTo facilitate accurate measurements with our established model, we converted coordinates into a physically meaningful framework, establishing a transformation from GPS coordinates to UE4's coordinates. In the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. Compared with other NeRF models, it is also very meaningful to be able to measure the actual scale in the scene modeled by NeRF. However, There exists a slight discrepancy between coordinates in UE4 and those estimated by COLMAP, but this error is minor and has negligible impact on the overall visual comparison. Currently, we have rectified this discrepancy and assure that in subsequent material revisions, adjustments will be made for consistency, enhancing the visual prominence.\nTo aptly showcase more intricate details from diverse perspectives, we have opted to compare images captured from free viewpoints rather than images from the training viewpoint. Consequently, the ground truth is missing.\n\nA8:Thanks for your suggestion, we will modify it to \"we incorporate parallel lights from various angles above\".\n\nA9:We responded to all weaknesses. Sincere thanks for all your suggestions!\n\nA10:Due to the significant presence of intersecting faces in our octahedral mesh, while it aids model convergence during training, it introduces complications in volume rendering. In our method, during step-wise rendering, a substantial number of triangle intersections and depth sorting are required before rendering can proceed according to the transparency blending formula. This results in slow rendering speeds when adopting a volume rendering approach. Additionally, the challenge of blending multiple models from various regions further adds complexity. Addressing this issue demands efforts akin to developing a dedicated rendering engine, which contradicts our initial intention of integrating rendering seamlessly into the UE rendering engine. One positive development is that we've successfully resolved the camera exposure issue in UE4 by replacing the tone mapper with a custom tone mapper during the post-processing phase of rendering. In the revised version, we will update our rendering results. **In the submitted PDF, the presented results for UE4-NeRF depict the outcomes achieved after resolving the exposure gap, revealing an observable improvement in rendering quality.**\n"}, {"review_id": "KpDoH0hUoa", "reviewer": "Reviewer_PLf2", "summary": "This work presents a system to represent a large scene using NeRF given drone-captured photos. It partitions a large scene into overlapping smaller tiles, and represent each tile with a sub-NeRF. To enable real-time rendering, it represents each sub-NeRF using meshes and neural textures (represented as a encoder-decoder network) during training; after training, meshes are further simplified, and the encoder part of neural textures are converted to texture image on request, while the decoder part is implemented as a GLSL shader. The whole real-time rendering system for the proposed NeRF representation is built on top of Unreal Engine 4, and allows for scene editing. ", "questions": "1. It seems that the proposed method is a bit complicated during training phase. Will the training code be released for reproducibility upon acceptance?\n\n2. Would the proposed method also work for blender, besides the unreal engine 4?", "limitations": "Limitations are adequately addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The proposed method not just trains fast, but also renders fast. It's cool to see the whole NeRF real-time rendering system integrated into the existing renderer, Unreal engine 4.", "weaknesses": "1. Writing could use some improvements; many key technical details seem missing, making the paper a bit hard to understand.\n* 1.1 The frame rate of real time rendering is reported on what device?\n* 1.2 Line 129-132 says that \"we train a coarse model for sub-region segmentation...\"; what model was trained here? Is it counted in the total training time of the proposed method?\n* 1.3 Line 143 says \"it generates 8D feature vector, which incorporates opacity information\"; this seems very confusing to me, as in Eq. 1, the opacity is predicted alongside the feature vector.\n* 1.4 Line 167-169 seems to suggest that max should be used in Eq. 5, as opposed to min?\n* 1.5 Octahedron with 20 faces. Might be better to explain why there're 20 faces instead of regular 8 faces.\n* 1.6 Fig 5 village scene, the lower-right inset of ours does not seem to align with those of baselines.\n* 1.7 How is the comparison with baselines with MipNeRF and InstantNGP performed? Were the baselines trained on each individual blocks or they were trained on the whole scene?\n* 1.8 Line 248-254: alpha values seem to be dropped during real-time rendering. If so, please make it explicit, and include examples how such dropping affects the final rendering quality.\n* 1.9 Fig. 1c seems confusing without labels \"before editing\" and \"after editing\". In addition, how was the inpainting performed when the building/car were removed? \n* 1.10 I find it hard to understand what's going on in line 181-185. The authors might want to add some illustrations. \n* 1.11 Line 52-54 seems to suggest that the speed up of training, compared with MobileNeRF, is due to implement some compute-heavy components using customized cuda functions. What are these components?\n\n2. Ablations and comparisons can be improved; it's hard to judge if the proposed components are useful or not, and how it compares against baselines. \n* 2.1 The BungeeNeRF work seems to a better baseline than MipNeRF; but for some reason, comparison with that work is missing.\n* 2.2 Why not compare with MobileNeRF on each tile? Line 135-137 claims that the proposed mesh initialization addresses the unstable convergence issue of MobileNeRF; is there any intuitive explanation or empirical evidence to support this claim?\n* 2.3 Ablations are needed to justify the effectiveness of the loss function in Eq. 6.\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1.1 In UE4, you can input the command \"stat fps\" to show frame rate.\n\nA1.2 Training a NGP model on low-resolution images of the entire scene aids us in better segmenting the scene. This step takes only a matter of minutes.\n\nA1.3 Opacity is solely dependent on position and not influenced by direction. As a result, we can directly obtain opacity through the encoder. Utilizing the encoder, we derive an 8-dimensional feature vector, where one dimension's feature can be transformed into opacity via a nonlinear transformation. This is why we assert that the 8-dimensional feature vector encompasses opacity information.\n\nA1.4 Eq 5 is correct. The lines 170-173 should be revised to read: \"In the initial 10,000 epochs, the f is maintained at 0., and afterward, the f increases as the number of epochs increases, but does not exceed 0.3.\"\n\nA1.5 A regular octahedron has 20 faces in total, including the 8 exterior faces and 12 interior faces. **We have included illustrative diagrams of three representative interior faces in the submitted PDF(Figure 3(c)) to aid your understanding.** \n\nA1.6 To facilitate accurate measurements with our established model, we performed a coordinate system transformation to make it have physical meaning. There exists a slight discrepancy between coordinates in UE4 and those estimated by COLMAP, but this error is minor and has negligible impact on the overall visual comparison. Currently, we have rectified this discrepancy and assure that in subsequent material revisions, adjustments will be made for consistency, enhancing the visual prominence. \n\nA1.7 They were trained on the whole scene.\n\nA1.8 We do not discard opacity values. The Alpha information is preserved within the 8 channels, and it is calculated non-linearly from one of those channels. \n\nA1.9 The first line is before editing, and the second line is after editing. We directly remove the triangular mesh from the area and copy a portion of triangular mesh from surrounding similar regions.\n\nA1.10 **We illustrate the generation of different levels of detail (LOD) in the submitted PDF (Figure 3(a)).** We choose the point in the grid with the largest opacity as the point of the synthesized grid to generate the octahedron, and the point on the edge is not used as the center point of the octahedron.\n\nA1.11 Component: UV sampling based on vertices, generating feature maps from UV-sampled points, followed by BC4 compression. The description of \"therefore\" in line 53 of the manuscript is not entirely accurate. The impact is not solely due to the implementation in CUDA; this is merely one contributing factor, albeit not the determining one. Our approach, even without utilizing CUDA implementation, achieves training times of less than 2 hours for a region on a single 3090Ti. With CUDA , this time is further reduced to less than 1 hour.\n\nA2.1 Thanks for your suggestion, and we also tried. But it's highly impractical to obtain a wide range of multiscale images, from low to high, for training BungeeNeRF. The data used in BungeeNeRF is derived from various scales of rendered images of models in Google Earth, rather than being captured from real-world imagery. **In the \"Author Rebuttal\", we extensively compared our approach with Mega-NeRF and Mobile-NeRF.**\n\nA2.2 Based on our previous experiments, training a block with Mobile-NeRF takes around two days on 4x3090Ti GPUs. Our focus is on modeling large scenes, and if we were to model a scene as described in the manuscript, it would require around two months. Additionally, Mobile-NeRF demands significant GPU memory during training, and the generated faces and features necessitate substantial storage. It's not feasible to load such a large number of models for real-time rendering. Considering the rationale for comparison, we did not provide a complete comparison in the manuscript. **In our \"Author Rebuttal\" and submitted PDF, we discussed the comparison between our approach and Mobile-NeRF.** \n\tThe grid layout we use differs from that of Mobile-NeRF. We attempted Mobile-NeRF's grid layout (initially without tilted surfaces), but found that it required more training epochs to achieve a fitting result on tilted roofs (reaching a visual quality similar to our current approach required an additional 20,000 epochs of training, whereas our method only needed 80,000 epochs). Moreover, the rasterization artifacts were noticeable in certain shrub areas (although they improved with further training, it took an additional 50,000 epochs to significantly reduce the visible artifacts). On the other hand, our octahedral grid (including 12 interior triangular faces) handles more complex face-to-face intersections. The average distance from any point in space to the nearest triangular face is smaller than in Mobile-NeRF's grid layout. This results in faster convergence in the vertex optimization part of our approach.\n\nA2.3 **In the submitted PDF (Figure 5), we provide rendered results without the second part of the loss.** When not utilizing the second part of the loss, the model tends to utilize a multitude of low-opacity faces to fit in regions of low color variation and semitransparency. This leads to accidental deletion of faces that should be part of the model's surface when exporting triangles. In scenes with glass and shadows, more triangles may be unexpectedly deleted. This effect is especially prominent on glass surfaces in the picture, where numerous triangles may be unintentionally removed.\n\n\nA3. If our manuscript is fortunate enough to be accepted, we plan to make the training code publicly available!\n\nA4. It seems you also have a keen interest in computer graphics! Currently, much of our related work is based on the Unreal Engine. While our method is theoretically applicable to other rendering engines like Unity or Blender, we haven't conducted experimental validation in those environments. Once again, thank you for the questions and suggestions you've raised!\n\n"}, {"review_id": "7RoxwyqZm8", "reviewer": "Reviewer_eVeu", "summary": "The authors introduce a multi-scale surface based representation for radiance field reconstruction and real-time rendering of large-scale scenes. The method involves subdividing the scene into partitioned scenes initialized with a regular octahedron mesh. Through joint optimization of the vertices' positions and multi-resolution hash grids, the proposed method incorporates two rendering loss functions and depth supervision. Experimental results demonstrate that the proposed method achieves improved rendering quality while maintaining real-time rendering capabilities (>30 FPS) for large-scale scenes at 4K resolution.", "questions": "1. L137, why tilted surfaces can result in unstable convergence while regular octahedrons wouldn't?\n\n2. The usage of \"epochs\" is confusing and not clear to me what is it? Do you want to refer it to \"iterations\"?\n\n3. L196, how dense point clouds are used in the manuscript? ", "limitations": "Yes, the authors provide a limitation section and make sense to me.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The manuscript successfully tackles the crucial challenge of fast training and rendering for large-scale scenes. Excellent work!\n\n2. The multi-scale meshes representation seems fresh and interesting.", "weaknesses": "1. Presentation. The \"transient object\" section distracted me a lot, I don't this manuscript work on the dynamic scenario, correct me if I am wrong. \n\n2. Evaluation. The proposed method is only evaluated on the UAV dataset, leading it hard to judge its general performance, would be great to have some results on standard dataset such as mipnerf 360 and Tanks and Temple.\n\n3. Comparison. I could not find any comparison or discussion regarding Mobile-NeRF,  even though it is the closest method to the approach presented in this manuscript.\n\n4. Fair comparison. Table 1 shows the proposed method provides clearly better performance than previous Mip-NeRF, NeRF-W NeRFacto and Instant-NGP, however, these methods don't use depth supervision by default, did you plug in the additional loss for these methods or use their official setting?\n\n5. Anonymous. I don't think putting the project link associated with your personal GitHub account to the manuscript is a good idea. I'm not sure if it violates the anonymity rule, but I don't take that into account when scoring now and bring it up in hopes of hearing the authors' and another review's perspectives.\n\nMinor:\nThe caption of Fig. 2: we partition the ...\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1. We can use masks to exclude dynamic objects (such as people or vehicles) and prevent them from affecting the rendering results.\n\nA2. Sincerely thank you for your suggestion. \nThe focus of our work is on images captured by drones in large-scale scenes with GPS information. One of our goals is to establish a measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. \nWe performed a coordinate system transformation prior to modeling. We converted coordinates into a physically meaningful framework, establishing two transformations from NeRF coordinates to UE4's coordinates and GPS coordinates to UE4's coordinates. \nIn the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. Compared with other NeRF models, it is also very meaningful and novel to be able to measure the actual scale in the scene modeled by NeRF.\nAdditionally, the current version of our method relies on GPS data for various processing steps such as block division, point cloud generation from feature points, optimal modeling height calculation, and ground estimation. However, existing public datasets lack GPS information within their images.  In principle, if we remove the reliance on GPS information, our approach is feasible for a variety of datasets. We are in the process of exploring the generalizability of the proposed approach tailored for aerial data, to be applicable across diverse scenarios.\n\nA3. Based on our previous experiments, training a block with Mobile-NeRF takes around two days on 4 * 3090Ti GPUs. Our focus is on modeling large scenes, and if we were to model a scene as described in the manuscript, it would require around two months. Additionally, Mobile-NeRF demands significant GPU memory during training, and the generated faces and features necessitate substantial storage. It's not feasible to load such a large number of models for real-time rendering. Considering the rationale for comparison, we did not provide a complete comparison in the manuscript. **In our \"Author Rebuttal\", we discussed the comparison between our approach and Mobile-NeRF. Moreover, in the submitted PDF, we provided both qualitative and quantitative comparisons with Mobile-NeRF.** \n\nA4. In order to further speed up the convergence and reduce the abnormal phantom-like floating objects in the air, we propose pseudo-depth to optimize the training, which itself is an innovative method proposed for our scenario. Compared with others, their official code is used, but the pseudo-depth method we proposed is suitable for traditional NeRF methods, and we can try to apply pseudo-depth to the mentioned NeRF framework later. Your invaluable review and insights are greatly appreciated.\n\nA5. Thank you. While it is a private GitHub account, we removed all personal information before manuscript submission. This maintains anonymity and adheres to the principle of anonymity.\n\nA. 5.5 Thanks for your suggestion, we will modify it in the revised version.\n\nA6. The grid layout we use differs from that of Mobile-NeRF. A regular octahedron consists of a total of 20 faces, which includes the 8 tilted exterior faces and the 12 interior faces. We attempted Mobile-NeRF's grid layout (initially without tilted surfaces), but found that it required more training epochs to achieve a fitting result on tilted roofs (reaching a visual quality similar to our current approach required an additional 20,000 epochs of training, whereas our method only needed 80,000 epochs). Moreover, the rasterization artifacts were noticeable in certain shrub areas (although they improved with further training, it took an additional 50,000 epochs to significantly reduce the visible artifacts). On the other hand, our octahedral grid (including 12 interior triangular faces) handles more complex face-to-face intersections. The average distance from any point in space to the nearest triangular face is smaller than in Mobile-NeRF's grid layout. This results in faster convergence in the vertex optimization part of our approach.\n\nA7. I'm sorry for confusing you with my unclear expression. Yes, epochs refers to iterations.\n\nA8. The point cloud points here are derived from feature points with minimal matching loss during camera pose estimation. We employ them as sparse depth supervision targets. Given that this point cloud is extremely sparse, with a ratio of about 1/10000 compared to pixels, we use a certain neighborhood around the projected coordinates of the point cloud on the image as supervision targets and confidence diminishes with increasing projection distance, resulting in lower supervision strength. Specific implementation details can be found in the attached document.\n\n"}, {"review_id": "B3Xgp1HrHX", "reviewer": "Reviewer_3kud", "summary": "The paper proposed a method for large-scale scene reconstruction and rendering.\nIt utilizes NeRF to learn a mesh representation for the scene by optimizing the \nvertex positions and neural features/MLPs using standard volume rendering. \nTo handle large-scale scenes, the method divides the scene into multiple blocks \nand trains a NeRF for each block. The initial mesh within each grid is got \nby creating an octahedron at the center of each voxel grid. The paper also learns \na hierarchical representation with Level-of-Details to support efficient \nrendering viewed from different distances. The paper shows that such a representation \ncan be integrated with Unreal Engine for interactive rendering and scene composition. \nThe paper shows results on outdoor scenes captured by drones and makes comparisons \nagainst baselines such as instant-ngp, urban-NeRF, and NeRF-w. The experiment results \nshow that the proposed method achieves higher accuracy in rendering and faster rendering\nspeed.  ", "questions": "\n1. The overall pipeline of the paper is similar to that of MobileNeRF. The paper says in Line 100\nthat \"in this work, we accelerate the rendering by ... into a mesh representation for efficient \nmesh rasterization pipeline\".  How is this different from MobileNeRF? \nWhile I agree that the proposed method is faster than MobileNeRF in terms of training, I think \nit's the main benefit of using hash grid instead of MLP. In terms of representations, I think \nthe proposed method is similar to MobileNeRF. \nIn terms of supporting large-scale inputs, it seems that the major change is separating the scene into blocks and doing blockwise\ntraining. Can MobileNeRF be adopted for this task if we train it for each block? I think the paper \nshould make it more clear the difference of the proposed representation. \n\n2. In Line 137, I think it should be \"a regular octahedron with **8** faces\". The paper says the \nMobileNeRF has slow and unstable convergence issues. It's not clear to me why using octahedron \ncan help resolve this issue. What are the benefits of using it? It's needed to add a validation \nstudy and comparisons to MobileNeRF. It will give the readers more insight if the paper \ninvolves a comparison to MobileNeRF on standard benchmarks such as NeRF synthetic and LLFF. \n\n3. The paper mentions having overlaps between blocks. How are the reconstructions handled at \noverlapped regions? More details should be provided in this part.   \n\n4. In Equation 5, how is the threshold 0.3 chosen? How does the performance of the proposed method \nchange with different values of this threshold?\n\n5. In Line 222, the paper says \"filter out meshes that have an intersection opacity less 0.3\". Is \nthis opacity only calculated at a single intersection point? I am wondering whether this will \ncause problems in regions where there are sharp edges with abrupt opacity changes.\n\n6. I think the title of the paper is kind of misleading. The reconstructed models are compatible \nwith standard rasterization engines using customized shaders, and it's not specific to UE4. Integrating \ninto UE4 is also something trivial given the underlying polygonal meshes. I would suggest the authors\nmodify the short name to more focus on large-scale reconstructions.\n\n\nOverall, I think the proposed method is technically sound, and the presented results are convincing. \nMy major concern is that the backbone used in the paper is similar to MobileNeRF and the paper is combining \nit with blockwise reconstructions (which is explored in previous large-scale NeRF works), \nwhich prevents me from giving a higher rating.  \n", "limitations": "The limitations look good to me.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The pipeline of optimizing a mesh representation with volume rendering for large-scale \nscene reconstruction is technically sound, and the paper shows that the proposed pipeline \nachieves better accuracy than the baseline method in terms of both accuracy and rendering speed. \n\n2. The paper introduces multiple techniques to increase the rendering efficiency and quality\nof the reconstructions, such as using an alpha threshold to make the mesh close to the real \nsurface, using pseudo-depth to remove floaters, and using pre-rendering to filter out \ntriangles that have small contributions.\n", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "A1: Thanks for your comments. Actually, our proposed method is different work from Mobile-NeRF. **For a detailed comparison, please refer to Author Rebuttal.** In our test, using NGP in the same block requires 70,000 epochs to achieve results close to our method. Mobile-Nerf training is divided into three phases, the first two of which involve the training of the model, which requires 300,000 epochs for the first phase and 500,000 epochs for the second phase. Assuming we use a hash mesh instead of mobile-nerf's backbone, consider that neural network fitting opacity (high frequency information) requires more training than fitting volume density and requires additional mesh optimization. Then we can roughly estimate that the first stage needs at least 70,000 epochs to achieve the effect close to our method. Based on the highest speedup achieved in the first phase of NGP, the replaced mobile nerf backbone network needs at least 120,000 epochs to complete the second phase of training, and then in the third phase, Mobile nerf exports precomputed feature maps and grids. Our method only needs a total of 80,000 epochs to complete the training and export the UV coordinates of the vertices and mesh.\n  **In the Author Rebuttal, we provided a comprehensive comparison between UE4-NeRF and Mobile-Nerf.** Based on our previous experiments, training a block with Mobile-NeRF takes around two days on 4 * 3090Ti GPUs. Our focus is on modeling large scenes, and if we were to model a scene as described in the manuscript, it would require around two months. Additionally, Mobile-NeRF demands significant GPU memory during training, and the generated faces and features necessitate substantial storage. It's not feasible to load such a large number of models for real-time rendering. \n\nA2: A regular octahedron consists of a total of 20 faces, which includes the 12 interior faces. We attempted Mobile-NeRF's grid layout (initially without tilted surfaces), but found that it required more training epochs to achieve a fitting result on tilted roofs (An additional 20,000 epochs are needed compared to 80,000 epochs). Moreover, the rasterization artifacts were noticeable in certain shrub areas (it took an additional 50,000 epochs to significantly reduce the visible artifacts). On the other hand, our octahedral grid handles more complex face-to-face intersections. The average distance from any point in space to the nearest triangular face is smaller than in Mobile-NeRF's grid layout. This results in faster convergence in the vertex optimization part of our approach.\nThe focus of our work is on images captured by drones in large-scale scenes with GPS information. One of our goals is to establish a measurable NeRF model for these scenarios, which requires GPS data for real-world scale conversion. We performed a coordinate system transformation prior to modeling. \nIn the UE4 coordinate system, the positive direction of the X-axis points to the east, the positive direction of the Y-axis points to the south, and the positive direction of the Z-axis points up. The scale is 100:1. Compared with other NeRF models, it is also very meaningful and novel to be able to measure the actual scale in the scene modeled by NeRF.\nAdditionally, the current version of our method relies on GPS data for various processing steps such as block division, optimal modeling height calculation, and ground estimation. However, existing public datasets lack GPS information within their images.  In principle, if we remove the reliance on GPS information, our approach is feasible for a variety of datasets. We are in the process of exploring the generalizability of the proposed approach tailored for aerial data, to be applicable across diverse scenarios.\n\nA3: When dividing the area, we accurately divide the scene into multiple areas, and there is no intersection between the target modeling areas of each area, but when modeling, in order to avoid the influence of the area mask error, the actual modeling the area is expanded outward by 1.333 times, but when exporting, we do not export the mesh of the expanded area.\n\nA4: The higher the threshold, the more the final model tends to capture opaque objects, while a lower threshold performs better in modeling translucent objects, but it results in more triangles during export, leading to increased rendering and storage costs. After comparing visualization effects and triangle counts across various threshold values, we found that a threshold of 0.3 strikes a good balance between effective modeling of translucent objects and reasonable triangle count.\n\nA5: Thanks for your reminder. A triangular face will only be clipped if the opacity of all intersection points on that face is below 0.3. In our rendering process, we haven't defined a minimum opacity threshold, and any valid opacity on effective triangular faces is preserved during asynchronous feature map computation. There's no threshold truncation. Hence, in practical rendering scenarios, this issue is scarcely encountered.\n\nA6: Currently, a significant portion of our related work is built upon the UE engine, encompassing features such as dynamic model loading, asynchronous feature map inference, and LOD layer calculations. Our rendering process is not a static loading process similar to mobile-nerf's static inference of feature maps and saving Mesh grids, and then the rendering engine loads for rendering, but real-time on-demand inference of feature maps and switching of LOD layers in rendering. The model is a dynamic process of predicting the model area that needs to be loaded and unloaded, and releasing data such as textures in the memory and cache when necessary. A large number of operations involve the implementation of specific APIs that depend on the UE4 engine. While theoretically adaptable to rendering engines like Unity, we have not been experimentally validated in other engines. Hence, in the interest of rigor, we refer to our approach as \"UE4-Nerf.\" \n\n\n"}]

Author rebuttal:
## Comparison with Mobile-NeRF.
**Dataset** We test the performance of Mobile-NeRF in a block. It contains 239 pictures, each with a resolution of 6000x4000. 

  In Table 1, we see that Mobile-NeRF takes 2 days to train just one block and requires 4x3090ti GPUs. If it trains the whole scene, it takes two months. And our method only need 40 minutes to train a block (1x3090ti), when training multiple blocks, we can perform multi-GPUs parallel training. In Figure 2, the left side is comparison of the rendered image under the training perspective, and the right side is comparison of the final result under free perspective in the renderer. Compared with Ground truth, UE4-NeRF shows amazing rendering results, and Mobile-NeRF produces obvious blurring. The results in the final renderer on the right show that our real-time rendering results are also much better than Mobile-NeRF. Mobile-NeRF finally uses webgl for real-time rendering. We speculate that the poor rendering ability of webgl has a certain impact on the rendering quality. WebGL sorts objects according to the coordinates of the center of the object instead of sorting according to the surface, and there will be various strange interlacing.
## Differences between Mobile-NeRF and UE4-NeRF.
1.The training strategy is different. Instead of Mobile-NeRF's three-stage training, we adjust weights of each part of the loss function in stages during training to control the tendency of the training process.

2.We train 5 levels of meshes at the same time, and the overhead of 5 levels of mesh training is only increased by 1/7 compared to training only the highest-precision mesh. And UE4-NeRF can control the rendering overhead by dynamically switching between different levels of precision of the model during rendering.

3.We use a different grid layout than Mobile-Nerf. In our mesh layout, the vertices converge faster.

4.The export strategy is different. Our method of calculating visibility is different from that of Mobile-NeRF, and we only save the structural relationship between vertices and faces and the UV coordinates of vertices, and do not directly save the features on the faces as texture maps.

5.The rendering strategy is different. Our approach is a dynamic loading and rendering strategy. The asynchronous process only takes less than 1.5 seconds. For comparison, Mobile-NeRF is an offline precomputation and real-time rendering strategy, which requires all the feature maps on all surfaces to be computed when the model is exported, resulting in exponentially higher hardware resource requirements in large scenes. We can infer higher-precision feature maps in the rendering stage without requiring additional storage space. We use higher precision 32x32 feature maps, while Mobile-NeRF uses 20x20 feature maps and requires more storage space.

6.The Mobile-NeRF is unable to model any translucent objects, but our method can approximate modeling and rendering translucent objects in UE4. We achieve our goal of rendering translucent objects by alpha dithering and temporal fusion.

7.Measuable.
## Comparison with Mega-NeRF.
**Dataset** We measure the performance of Mega-NeRF in a scene. The scene contains 2000 images, each with a resolution of 6000x4000.

For each block, Mega-NeRF needs 36 hours to train for 500,000 epochs, while we only need 40 minutes to train for 80,000 epochs to reach convergence. When training the entire scene, Mega-NeRF requires 12 hours of additional time, but UE4-NeRF only needs 1 hour. It is worth noting that Mega-Nerf generates a large number of temporary files during training. In Figure 1, The left side is comparison of the rendered image under the training perspective, and the right side is comparison of the final result under the free perspective in the renderer. Compared with Ground truth, UE4-NeRF shows amazing rendering results, and Mega-NeRF produces obvious blurring. The results in the final renderer on the right show that our real-time rendering results are also much better than Mega-NeRF. Real-time rendering results in Mega-nerf-viewer are weird. Through our experiments, the maximum resolution supported by Mega-nerf-viewer is only 800x800 (1x RTX3090ti), when we enlarge its resolution, the program crashes.
## Supplementary Explanation of NGP and Block Strategy.
The NGP approach employs dynamic resolution and multi-frame fusion to achieve high frame rates for real-time rendering. There is a noticeable presence of large pixel artifacts when the window size is larger. In NGP experiments, we assessed the rendering speed and computational cost at a fixed 2K rendering resolution. Even with a RTX 3090ti , it can only support rendering of a single high-complexity nerf model, yielding a frame rate of under 30fps. Further elevating the rendering resolution would lead to insufficient VRAM capacity, causing crash. 

In the past, we also attempted to directly combine NGP with block to model large scenes. For a large scene, we use 32 lower-complexity NGP models to model each area, minimizing the memory usage of a single model's hash grid. When rendering, the method of dynamically loading and selecting models is combined to further reduce the occupation of GPU memory and computing resources.  The modeling accuracy was improved when using NGP with block strategy compared to using a single NGP model for the entire scene. However, despite these efforts, the rendering speed remained relatively low in practical. We could achieve approximately 10 fps at 720P resolution using two 3090Ti. Furthermore, doubling the resolution led to a twofold increase in VRAM usage and halved the frame rate. Therefore, only the method of combining the block strategy with NGP cannot realize high-resolution real-time rendering of the nerf model modeled in a large scene. Even if our method loads 32 blocks, it can achieve a rendering frame rate of 50fps at 2K resolution on a 3090ti, and can also achieve a relatively high rendering frame rate at 4K resolution.


Meta-review: Reviewers and the AC read the rebuttal and took that into consideration for their final recommendation. The AC believes that the answers to the unresponsive reviewer in the rebuttal reasonably addressed their concerns. Many suggestions for improving the exposition, additional baseline comparisons, among other suggestions by reviewers should be included in the camera-ready paper.
Decision: Accept (poster)