HS-Surf: A Novel High-Frequency Surface Shell Radiance Field to Improve Large-Scale Scene Rendering

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 06:59:48

Abstract: Prior neural radiance fields often struggle to preserve high-frequency textures in urban and aerial large-scale scenes due to insufficient model capacity on the scene surface. This is attributed to their sampling locations or grid vertices falling in empty areas. Additionally, most models do not consider the drastic changes in distances. To address these issues, we propose a novel high-frequency surface shell radiance field, which uses depth-guided information to create a shell enveloping the scene surface under the current view, and then samples conic frustums on this shell to render high-frequency textures. Specifically, our method comprises three parts. Initially, we propose a strategy to fuse voxel grids and information of distance scales to generate a coarse scene at different distance scales. Subsequently, we construct a shell based on the depth information to carry out compensation to incorporate texture details not captured by voxels. Finally, the smooth and denoise post-processing further improves the rendering quality. Substantial scene experiments and ablation experiments demonstrate that our method achieves the obvious improvement of high-frequency textures at different distance scales and outperforms the state-of-the-art methods.

Main paper: 
 introduction: 
 Rendering urban and aerial large-scale scenes has many applications like AR/VR and digital navigation. Prior neural radiance fields [1] [2] [3] (NeRFs) have tried improving the rendering quality, which can be categorized into two streams. The first [4] [5] [6] divides the scene or camera poses into multiple sub-regions or groups, and each unit is represented by a NeRF. This increases the number of NeRF modules, indirectly enhancing model capacity on the surface. However, those NeRFs sample along the entire ray, including empty spaces. The second [7, 8] reconstructs a coarse scene or density field to guide subsequent sampling, concentrating in high-density areas near the surface. Nevertheless, some samples inevitably fall into empty regions, such as the sampling interval ends.Multi-layer perceptron (MLP) based NeRFs [1, 2] frequently sample along rays, resulting in many points falling into empty regions. Therefore, much model capacity for storing geometry and appearance is used to represent these meaningless spaces rather than the scene surface. Additionally, voxel [9] and grid-based [10] [11] [12] NeRFs only have a few vertices to be sampled near the target surface, leading to an upper bound on the model capacity allocated to the surface. These inefficient samplings result in a significant waste of model capacity, lacking enough model capacity on the scene surface to render high-frequency textures. Moreover, most NeRFs [4, 5, 7] have not considered the drastic changes in distances between the camera and the scene surface, which is prone to generate blurry rendering results at various distances.To overcome the inefficient sampling and enhance the quality of high-frequency textures, we propose a novel high-frequency surface shell radiance field (HS-Surf) to efficiently increase model capacity on scene surfaces. It constructs a shell enveloping the scene surface based on the current view's scene depth. As shown in Figure 1 (a), the shell's width increases with the depth, and conic frustums sampled on the shell are used to render high-frequency textures at different distance scales. We call this shell as High-Frequency Shell (HS). HS confines the rendering to scene surfaces, greatly enhancing the utilization of model capacity. Additionally, to model geometry and appearance at different distances, we propose a feature fusion strategy to embed conic frustums representing distances into voxel grids.Our HS-Surf consists of three stages: initialization, compensation, and post-processing. The initialization uses hash-based voxel grids to generate coarse geometry and appearance. To model distances with drastic changes in large-scale scenes, the proposed feature fusion strategy embeds conic frustums into voxel grids. The compensation generates high-frequency textures at different distance scales. It first augments the coarse scene depth under the current view and constructs an HS based on the augmented depth. Conic frustums are then exclusively sampled on the shell to generate high-frequency textures lost in the coarse appearance. The postprocessing uses a convolutional neural network (CNN) to smooth and denoise the rendering results to achieve a better visual effect.The experimental results indicate that HS-Surf greatly improves high-frequency textures (see Figure 1(b) ) and achieves state-of-theart rendering quality. Additionally, we observe that our rendering speed improves 2× to 4× faster than previous NeRFs, achieving double improvement of the rendering effect and computation efficiency. Our contributions can be summarized as follows:• Our proposed high-frequency shell overcomes the sampling inefficiency of previous methods, efficiently increasing model capacity on scene surfaces to render high-frequency textures.• The proposed feature fusion strategy embeds conic frustums into voxels to represent the distance scales, enabling the voxel to model the scene at various distances. 
related work 2.1 neural radiance fields: 
 NeRF [1, 3] employs MLPs to model volume density and color of spatial points. A lot of NeRF variants [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] render different size objects from small goods to large-scale scenes, aiming to enhance fidelity, rectify camera poses, and accelerate rendering. There are also models [24, 25] designed for unbounded scenes. To speed up rendering, some methods replace MLPs in NeRF with voxel [9, 26] or plane grids [10] [11] [12] , but these increase GPU memory consumption. Recent methods [27] [28] [29] map the voxel vertices into smaller hash tables, which compress 3D spaces and achieve more compact representations.MipNeRF [2] samples conic frustums along rays, and uses integrated positional encoding (IPE) of the frustums to represent the distance scales. However, the local continuous space of the frustum is incompatible with interpolation operations in grids. ZipNeRF [8] simulates the local space by sampling six points within a conic frustum. These points are then fed into InstantNGP [27] . 3D Gaussian [30] is another recent method with different mechanisms for scene representation, which involves fitting a large number of ellipsoids to approximate the target scene and render novel views.Some traditional methods [31] [32] [33] [34] [35] have been proposed to reconstruct the large-scale scenes. Their working pipeline usually needs three stages: keypoint detection, feature matching, and bundle adjustment. Keypoint detection [36] [37] [38] looks for unique and easily identifiable regions in images and constructs corresponding feature descriptions. Then, the features of key points are matched to compute camera poses and locations of 3D points. Finally, the camera poses and 3D points are jointly optimized by bundle adjustment [39, 40] . These methods can roughly reconstruct the target scene and synthesize novel views [41, 42] , but the results often contain artifacts and holes.The NeRFs-based methods are also introduced into large-scale scene rendering, including BungeeNeRF [6] , BlockNeRF [4] , MegaN-eRF [5] , URF [43] , and GridNeRF [7] . BungeeNeRF divides camera poses into four groups based on their heights. A single NeRF is used in the highest group. For each subsequent group with lower heights, the model adds a NeRF module to capture the finer texture details. BlockNeRF and MegaNeRF divide the target scene into multiple sub-regions, with each region represented by a separate NeRF. By partitioning the camera poses or the scene, BungeeNeRF, BlockN-eRF, and MegaNeRF reduce the target regions for each sub-NeRF, increasing the model capacity on scene surfaces. However, their performance improvements are limited because NeRF still needs to sample the entire ray, including empty regions. The key issue of low utilization and allocation of model capacity on the scene surface remains unresolved. URF [43] leverages depth data of radar as auxiliary information to reconstruct street-level scenes.GridNeRF [7] comprises a grid branch and a NeRF branch. The grid branch compresses the scene onto a ground plane to reconstruct a coarse radiance field, guiding a second sampling operation to add points in high-density regions near the scene surface. All the sampling points are then fed into the NeRF branch for rendering novel views. However, adding points near the surface still can have some samples falling into empty regions, particularly at the ends of the sampling interval. Moreover, the points in the first sampling operation are distributed across the entire ray, which increases the consumption of model capacity by empty regions. As a result, GridNeRF still lacks sufficient model capacity on the scene surface to render high-frequency texture details.Both our HS-Surf and GridNeRF include rendering on the scene surface based on a coarse reconstruction. However, our method has different motivation and working mechanism. HS-Surf constructs high-frequency shells on the scene surface. These shells confine the computation of MLPs to the surface while excluding the surrounding empty regions, which greatly enhances the utilization of model capacity. Thus, our method has more power to render high-frequency textures. 
method: 
 The overview of HS-Surf is illustrated in Figure 2 , consisting of three stages: initialization, compensation, and post-processing. The initialization reconstructs the coarse geometry and appearance using hash-based voxel grids. IPE encoding of conic frustums and grid features are fused to model the target scene at different distance scales. The compensation first augments the coarse depth map under the current view to ensure a more accurate surface geometry. Subsequently, it constructs an HS based on the depth map, and samples conic frustums on HS to compensate for the lost high-frequency textures in the coarse appearance. This step is very important because the texture is attached to the geometry. Thus, good depth and its subsequent product of HS can confine sampling to effective texture areas. In the post-processing, a lightweight CNN is employed to smooth and denoise the rendering results of the compensation.Hash-based voxel grid is suitable for large-scale scenes as uniformly distributed vertices ensure that model capacity is reasonably allocated across the entire scene to generate a coarse radiance field. Additionally, hash tables are very useful for reducing GPU occupancy for high-resolution voxel grids.Due to the drastic changes of distance scales in large-scale scenes, sampling points along rays can easily result in blurry rendering results at different distances. Inspired by MipNeRF [2] , we sample conic frustums within the target scene and use IPE encoding to model distance scales. Specifically, a frustum is approximated by mean and covariance, which are then fed into IPE to generate the corresponding encoding [2] . Like MipNeRF, the radius of conic containing the frustums at image plane 𝑜 + 𝑑𝑖𝑟 is set to 𝑟 , and 𝑟 is the width of the pixel in world coordinates scaled by 2/ √ 12. To leverage the advantages of both the hash-based voxel grids and conic frustums, we propose a feature fusion strategy to reconstruct the coarse scene at different distance scales. As shown in Figure 2 , the center coordinates 𝑥 of the frustums are used to query features in the density grids 𝐷𝐺 (•) and the color grids 𝐶𝐺 (•). The IPE encoding 𝐸 𝑖 of the frustums is fed into a two-layer MLP 𝑀 𝑖 (•), which is then fused with density features and color features to compute density 𝜎 and color 𝑟𝑔𝑏 as follows:EQUATIONwhere 𝑀 𝜎 (•) and 𝑀 𝑐 (•) are small MLPs for generating density and color, and 𝑑𝑖𝑟 is the ray direction. Volume rendering subsequently generates scene depth and rendering results.The initialization stage can be described in two steps. The first step samples conic frustums along rays to render pixel colors 𝐶 1 . The second step conducts finer sampling based on existing sample densities to obtain pixel depths 𝑑 𝑐 and colors 𝐶 2 . Both steps utilize the same voxel grids and hash tables. To ensure that the initialization results contain less artifacts such as "floaters" and "background collapse", we add an interval-based regularization loss 𝐿 𝑑𝑖𝑠𝑡 in the fine step, which is proposed in MipNeRF360 [24] .EQUATIONwhere 𝑠 and 𝑤 represent the (normalized) ray distances and weights of conic frustums in volume rendering, respectively. The role of 𝐿 𝑑𝑖𝑠𝑡 is to concentrate the frustums with high density into a narrower region. Then, the loss of the initialization stage is as follows:EQUATIONwhere 𝐶 𝑔𝑡 is the real pixel color. 𝜆 1 and 𝜆 2 are set to 0.1 and 0.001 in all experiments.The initialization includes a large number of voxel vertices far from the scene surface, leading to the insufficient model capacity on the surface and the loss of high-frequency textures. To recover the lost texture details, we construct a high-frequency shell for each view to efficiently increase the capacity on the surface. The details are as follows:Depth Augmentation. The limited model capacity on the surface leads to coarse geometry with noises and holes. Therefore, before constructing the high-frequency shell, we propose a depth augmentation module to improve the depth map under the current view. As shown in Figures 2(a ) and 2(c), the coarse depth 𝑑 𝑐 is utilized to compute the coordinate 𝑝 𝑐 of the scene surface. The depth augmentation then employs a four-layer MLP 𝐷 (•) to predict the distance from 𝑝 𝑐 to surface along the ray direction 𝑑𝑖𝑟 . The output of 𝐷 (•) is added with 𝑑 𝑐 to obtain a more accurate surface depth 𝑑 𝑓 as follows: where 𝛾 represents the Fourier encoding [1] . More accurate depths and positions of the scene surface guarantee fewer errors in the inputs of subsequent modules. We optimize the parameters of depth augmentation using both depth loss and subsequent rendering loss 𝐿 𝑟𝑒𝑛𝑑𝑒𝑟 . The depth loss ensures that the module preserves the basic geometry and structure of the scene, and the rendering loss 𝐿 𝑟𝑒𝑛𝑑𝑒𝑟 is used to refine the scene depth for more accurate surface representation. Without depth ground truth, we take an augmentation manner to improve depth. Like the pyramid image processing, we take a down-sample on the coarse depth 𝑑 𝑐 to get a low-resolution depth map, which could filter out certain noises while preserving the major depth information. The low-resolution depth map is determined by 𝑊 and is then compared with the output 𝑑 𝑓 of depth augmentation to compute the depth loss 𝐿 𝑑𝑒𝑝𝑡ℎ . Specifically, we perform downsampling in 𝑑 𝑐 by a factor of 3:EQUATIONEQUATIONwhere 𝑥 and 𝑦 are the pixel coordinates.High-frequency Shell. After obtaining accurate scene depth 𝑑 𝑓 under the current view, we need to construct a high-frequency shell on the surface based on 𝑑 𝑓 . As shown in Figure 1 (a), the shell's width 𝑡 𝑟𝑎𝑛𝑔𝑒 along the ray determines the enclosed space on the ray {𝑥 |𝑥 = 𝑜 +𝑡 ×𝑑𝑖𝑟, 𝑡 ∈ [𝑑 𝑓 -0.5 ×𝑡 𝑟𝑎𝑛𝑔𝑒 , 𝑑 𝑓 + 0.5 ×𝑡 𝑟𝑎𝑛𝑔𝑒 ]}. An infinite cone is constructed based on camera position, ray direction and pixel.Then, we truncate the interval [𝑑 𝑓 -0.5 × 𝑡 𝑟𝑎𝑛𝑔𝑒 , 𝑑 𝑓 + 0.5 × 𝑡 𝑟𝑎𝑛𝑔𝑒 ] on conic axis to obtain a conic frustum Δ.As the depth increases, the high-frequency textures on the surface should become blurred or even disappear. Therefore, the width of Δ needs to increase with depth to suppress the high-frequency components in IPE encoding. The relationship between shell's width 𝑡 𝑟𝑎𝑛𝑔𝑒 and depth 𝑑 𝑓 follows a linear function. First, we calculate the normalized width 𝑟 :EQUATIONwhere 𝜆 𝑟 = 0.1 is a scaling factor to stabilize the model. 𝑁 𝑒𝑎𝑟 and 𝑓 𝑎𝑟 are the distances traveled along the ray from camera to enter and exit the target scene. 𝑘 ≥ 0 and 𝑏 ≥ 0 are estimated by a fourlayer MLP, and the input of MLP is the concatenation of 𝑛𝑒𝑎𝑟 and 𝑓 𝑎𝑟 . The length 𝑡 𝑟𝑎𝑛𝑔𝑒 of the shell in the world coordinate system is as follows:EQUATIONEQUATIONThe normalized length 𝑟 needs to be clipped to [1/2000, 1/50] for more stable results. Texture Compensation. After sampling a conic frustum Δ for each ray on the high-frequency shell, we utilize continuous MLPs to complete the high-frequency textures lost in the voxel grids during initialization. As illustrated in Figures 2(a 𝑑𝑖𝑟 are then fed into an eight-layer MLP 𝐹 (•) to compute the highfrequency texture details missing in the voxel grids. The generated details are then added with the coarse rendering 𝐶 2 to obtain the complete appearance 𝐶 𝑓 as follows:EQUATION)where 𝛾 is the Fourier encoding [1] . 𝜆 𝑓 = 0.2 is a scale factor to stabilize the model. The output of 𝐹 (•) is the texture residual instead of the complete rendering result. The reason is that this method can reduce the learning burden of MLPs and focus the attention of model on the generation of texture details.Since the scope of inputs in the texture compensation is confined to the scene surface, the model capacity of MLP is dedicated to rendering textures on the target surface. This significantly improves both the capacity utilization on surface and the ability to render high-frequency details. The rendering loss 𝐿 𝑟𝑒𝑛𝑑𝑒𝑟 is as follows:EQUATIONwhere 𝐶 𝑔𝑡 is the real pixel color, and 𝜆 3 is set to 0.1. The loss function 𝐿 𝑡𝑒𝑥𝑡𝑢𝑟𝑒 of the compensation stage includes rendering loss 𝐿 𝑟𝑒𝑛𝑑𝑒𝑟 and depth loss 𝐿 𝑑𝑒𝑝𝑡ℎ :EQUATION)As the initialization and compensation stages calculate each pixel individually, the generated results may contain noises and are not continuously smooth. Therefore, a lightweight CNN-based postprocessing is constructed to deal with them. The network contains two residual blocks to adjust the features of the original image 𝐶 𝑓 . Sigmoid activation is used in the final layer of convolution, which limits the output range to [0, 1], whose details are provided in the supplementary material. We opt for a CNN due to the necessity of incorporating correlations between neighboring pixels in the smoothing and denoising processes. Convolution kernels offer a natural way to introduce such information. The loss function 𝐿 𝑖𝑚𝑔 of the post-processing stage is as follows:EQUATIONwhere 𝐶 and 𝐶 𝑔𝑡 are the predicted and real colors, and 𝜆 4 = 0.1.The compensation stage can further enhance the quality of highfrequency textures by partitioning the target scene. Assuming the scene is uniformly divided into 𝑁 sub-regions, and each contains a depth augmentation and a texture compensation, with corresponding MLPs represented as 𝐷 𝑖 (•) and 𝐹 𝑖 (•). The outputs of all subregions are combined into the final results. Therefore, the output 𝑑 𝑓 of depth augmentation in Equation ( 4) is modified as follows:EQUATION)where 𝑀 𝑖 = 1 indicates that 𝑝 𝑐 is located in the 𝑖-th region, otherwise 𝑀 𝑖 = 0. The output 𝐶 𝑓 of texture compensation in Equation ( 8) is modified as follows:EQUATION)where 𝑀 𝑖 = 1 indicates that 𝑝 𝑓 is located in the 𝑖-th region, otherwise 𝑀 𝑖 = 0. The increment in the number of MLPs leads to an augmentation in model capacity on the scene surface.In Figure 2 , the depth 𝑑 𝑐 generated by the initialization needs to go through a gradient stop. Otherwise, the depth augmentation performance may decrease. In the initialization, the first coarse and second fine step samples 64 and 128 inters along a ray, respectively. The minimum resolution of the voxel grids is 256 3 , and the maximum resolution is 8192 3 after 15 increments. The size of hash table is 2 21 × 4 or 2 22 × 4, and the hidden nodes of MLPs are 128. For the compensation stage, the hidden nodes of MLPs in the depth augmentation, estimation of high-frequency shell, and texture compensation are set to 256, 64, and 512. The channel of CNN in the post-processing is set to 32.The training of HS-Surf consists of two stages. The first stage involves joint training of the initialization and compensation. The second stage only trains the post-processing. Their losses are as follows:EQUATION)EQUATION)The learning rate is 1𝑒 -4 for both stages and decays exponentially to 1𝑒 -5 during training. More details can be found in the supplementary material. 
experiment setup: 
 Our experiments are conducted on six scenes, including𝑇 𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎, 56𝐿𝑒𝑜𝑛𝑎𝑟𝑑, 𝐵𝑢𝑖𝑙𝑑𝑖𝑛𝑔, 𝑟𝑢𝑏𝑏𝑙𝑒, 𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑐𝑒, and 𝑐𝑎𝑚𝑝𝑢𝑠. All models are implemented in environments of Python and PyTorch on a single RTX 3090 24G GPU. 𝑇𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎 and 56𝐿𝑒𝑜𝑛𝑎𝑟𝑑 are two synthetic scenes from the satellite level to the ground level, provided in BungeeNeRF [6] . All images are collected from Google Earth Studio [44] , where the camera rotates around the central object of the scene, and the distance to the ground gradually decreases. 𝐵𝑢𝑖𝑙𝑑𝑖𝑛𝑔, 𝑟𝑢𝑏𝑏𝑙𝑒, 𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑐𝑒, and 𝑐𝑎𝑚𝑝𝑢𝑠 are four real aerial data. The first two are from Mill 19 [5] , and the remaining two are from UrbanScene3D [45] . In these four datasets, the drone always keeps a stable flying height and shoots to the ground along parallel lines or grid tracks. Therefore, the data distribution is uniform.HS-Surf is compared with the previous state-of-the-art NeRFs, including BungeeNeRF [6] , MegaNeRF [5] , and GridNeRF [7] . Mip-NeRF [2] and ZipNeRF [8] are also used in the experiments because they are basic neural rendering methods and can model different distance scales in large-scale scenes. Since each image in the four aerial photography datasets has a different exposure and white balance, we refer to NeRF-in-the-wild [46] to assign a 48-dimensional appearance embedding for each image to model the lighting information. The highest frequencies of positional and directional encodings are set to 12 and 4 for all models. The details of training and testing sets, and more model configurations can be found in the supplementary material.HS-Surf is also compared to 3D Gaussian [30] with different mechanisms, and the experimental results are presented in the supplementary material. The ability of 3D Gaussian to render highfrequency textures is not as good as HS-Surf because ellipsoids may In Table 1 , we use PSNR, SSIM, LPIPS (VGG), and the time of rendering a frame to compare the rendering performance. The data distribution in 𝑡𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎 and 56𝐿𝑒𝑜𝑛𝑎𝑟𝑑 [6] is uneven because the cameras make loop shoot for the centers of scenes. Therefore, 𝑡𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎 and 56𝐿𝑒𝑜𝑛𝑎𝑟𝑑 are not divided. MegaNeRF [5] is not suitable for this mode, so it has no corresponding results. For the four aerial photography data, MegaNeRF and HS-Surf divide each scene into 8 sub-regions evenly. Since the drone always keeps a stable flight height, BungeeNeRF [6] cannot divide the camera poses according to the height from the camera to the ground, so it has no corresponding results on the later four datasets.In 𝑡𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎 and 56𝐿𝑒𝑜𝑛𝑎𝑟𝑑 [6] , the distance scales undergo drastic changes, HS-Surf achieves noticeable improvements in all metrics compared to models designed for variable distance scales (MipNeRF [2] , ZipNeRF [8] and BungeeNeRF [6] ), as shown in Table 1 . The LPIPS errors of our method are reduced by 30%-40%. Figure 3 demonstrates a visual comparison example in the 𝑡𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎 from the satellite level to the ground level. HS-Surf renders more high-frequency texture details for objects with different distances and shapes, which benefits from that HS-Surf embeds conic frustums into voxels to model distances and uses compensation to render high-frequency details at different distance scales.In aerial photography scenes (𝑏𝑢𝑖𝑙𝑑𝑖𝑛𝑔, 𝑟𝑢𝑏𝑏𝑙𝑒, 𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑐𝑒, 𝑐𝑎𝑚𝑝𝑢𝑠) with stable heights, HS-Surf also demonstrates better performance, with LPIPS error decreasing by 10%-20% compared to MegaNeRF [5] and GridNeRF [7] . Figure 4 presents visual comparison examples in 𝑏𝑢𝑖𝑙𝑑𝑖𝑛𝑔 (top) and 𝑟𝑒𝑠𝑖𝑑𝑒𝑛𝑐𝑒 (bottom), HS-Surf synthesizes the most accurate views, especially regarding high-frequency textures for buildings with complex structures, wall or roof planes with different directions, and the vehicles with tiny height difference. Compared to the other models, these cases validate the advantage of our high-frequency shells in confining the rendering to the surface and improving the utilization of model capacity. GridNeRF reconstructs a coarse scene using ground plane grids, then guides the sampling of NeRF branch in the high-density regions rather than on the surface. Its shortcomings are similar to MipNeRF, BungeeN-eRF, and MegaNeRF. The inefficiency of sampling results in low capacity utilization and loss of high-frequency textures, as shown in Figure 4 . We find that the rendering of HS-Surf is 2× to 4× faster than other NeRFs in Table 1 . Except for benefiting from the voxel grids, there are two other contributing reasons: 1) The compensation stage only samples a conic frustum for each ray. Thus, each ray only needs to be calculated once. 2) The hidden channel of shallow CNN in the post-processing is only 32. MipNeRF, BungeeNeRF, MegaNeRF and GridNeRF query multiple samples along a ray in each sampling stage, resulting in multiple computations for rendering a pixel. ZipNeRF samples six points in a conic frustum, bringing it a huge cost and a slow speed.The model sizes on the dataset of 𝑏𝑢𝑖𝑙𝑑𝑖𝑛𝑔 are presented in Table 2, where the result of BungeeNeRF are from 𝑡𝑟𝑎𝑛𝑠𝑎𝑚𝑒𝑟𝑖𝑐𝑎. The compensation and post-processing are denoted as 𝑐𝑜𝑚𝑝 and 𝑝𝑝. The compensation stage of HS-Surf introduces extra parameters on the scene surface, resulting in a parameter increase of 3.5%.In Table 3 , we present an ablation study of HS-Surf. The postprocessing stage is represented by PP. We summarize the findings as follows. F) The removal of feature fusion strategy in the initialization stage results in a decreased modeling ability for distance scales. In supplementary material, we demonstrate the impacts of feature fusion and compensation on rendering results. G) The removal of compensation leads to a significant decrease in model performance.In Figure 5 , the results without compensation lose a lot of highfrequency textures at the top of the building. For more results of depth and rendering, please refer to the supplementary material. H) Removing depth augmentation (dea) in compensation leads to poor geometry and holes, and it also affects rendering quality, as shown in Figure 6 . I) Replacing the high-frequency shell with a single spatial point in compensation results in the framework's inability to model distance scales, as shown in Figure 7 . A)-E) remove the post-processing based on F)-J). Therefore, the rendering results contain noise and have relatively low quality, as shown in Table 3 and Figure 5 .  
discussion and conclusion: 
 In this work, we aim to improve the quality of high-frequency textures in urban and aerial large-scale scenes by dealing with issues of inefficient sampling and various distances. We have presented HS-Surf, a novel high-frequency surface shell radiance field method to improve large-scale scene rendering. We create a high-frequency shell on the scene surface under the current view, and sample conic frustums on this shell to overcome the sampling inefficiency in previous methods. As a result, model capacity is efficiently utilized to render high-frequency textures. Additionally, to model the distances with drastic changes in large-scale scenes, we embed frustums representing distance into voxel grids to construct the scene at different distance scales. Our HS-Surf achieves better rendering results in large-scale scenes, particularly concerning high-frequency textures. The solid ablation study experiments validate the effectiveness of each component in our model. Meanwhile, as we optimize the sampling and make better use of the model capacity, our implementation speed is also faster. Furthermore, our method can easily generalize to NeRFs variants and new rendering techniques. In the future, we will explore integrating the high-frequency surface shell into other rendering techniques, and consider rendering texture details at different distance scales in dynamic scenes.  
 
 Appendix:  


Decision: This paper proposes a high-frequency surface shell to focus model capacity on texture-rich areas, which is interesting and effective. All the reviewers agree to accept this paper. The authors should carefully address the concerns in the final version, such as the clear explanations of the working principle, experimental evaluations and discussions of relevant works.

Reviews: review: paper_summary: paper_summary: The paper introduces a method aimed at enhancing the rendering quality of high-frequency textures in large-scale urban and aerial scenes. The method leverages depth-guided sampling to construct a high-frequency surface shell (HS-Surf) that envelops the scene surface, thereby improving the model's capacity to capture detailed textures.
The introduction of a high-frequency surface shell to focus model capacity on texture-rich areas is a significant contribution. This addresses the problem of inefficient sampling in empty spaces seen in prior methods.
strength_weakness: strength_weakness: The introduction of a high-frequency surface shell to focus model capacity on texture-rich areas is a significant contribution. The use of depth information to guide the construction of the surface shell is well-motivated and effectively reduces the sampling in empty spaces. This addresses the problem of inefficient sampling in empty spaces seen in prior methods. Then it includes substantial experiments and ablation studies demonstrating the effectiveness of HS-Surf over existing techniques. The proposed method not only improves rendering quality but also increases rendering speed by 2× to 4×, which is a notable improvement in computational efficiency.
limitations: limitations: In my opinion, this article proposes more detailed scene solutions while ensuring that rendering speed does not decrease, which is a good work. However, the paper mainly focuses on urban and aviation scenarios, but the generalization ability of this method may be challenged in more scenarios. The method used in this article seems to be a universal solution. Can it be applied in general scenarios?
paper_summary: paper_summary: This paper presents a hybrid radiance field representation that integrates a high-frequency surface shell with volume density for large-scale scene reconstruction and rendering. The proposed high-frequency surface shell has an adaptive width proportional to depth, enabling detailed levels in rendered images. It is also more efficient in sampling compared to the standard volume rendering method for large-scale scene rendering. Experiments indicate that the proposed method achieves superior rendering quality compared to existing methods on large-scale scenes. An extensive ablation study is included to analyze the effectiveness of each module.
strength_weakness: strength_weakness: The proposed adaptive width of the surface shell is an effective approach for achieving various levels of detail in rendering, particularly beneficial for the large-scale scenes examined in this paper. It allows for rough sampling near the surface to provide fewer high-frequency details when depth is large, and more focused sampling for details when depth is small. 

The rendering quality improvement of the proposed method is substantial compared to existing methods based on NeRF. The visual comparisons in Figures 3 and 4 further demonstrate the superiority of the proposed method in rendering quality. 

Additionally, the paper is well-organized and clearly written, making it easy to follow the structure and understand the content presented.
limitations: limitations: The functions of integrated positional encoding (IPE) and the proposed adaptive width of the surface shell are similar  (depth increases lead to a larger frustum in IPE for less details). Since the paper employs both approaches to model different levels of rendering details, it is unclear which method contributes more to the final rendering quality. 

Furthermore, the concept of the surface shell is similar to the adaptive shell in [R1]. The authors should include this reference and discuss the differences to the proposed method. 

Additionally, the method of VastGaussian [R2], which addresses the same problem of large-scale scene reconstruction, should be mentioned and discussed, even though its underlying representation, based on 3D Gaussian Splatting, differs from the volume density representation in this paper.

[R1] Wang, Z., Shen, T., Nimier-David, M., Sharp, N., Gao, J., Keller, A., ... & Gojcic, Z. (2023). Adaptive Shells for Efficient Neural Radiance Field Rendering. ACM Transactions on Graphics (TOG), 42(6), 1-15.

[R2] Lin, J., Li, Z., Tang, X., Liu, J., Liu, S., Liu, J., ... & Yang, W. (2024). VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction. arXiv preprint arXiv:2402.17427.
paper_summary: paper_summary: This research aims to improve the rendering quality of existing NeRF methods in large-scale scenes. 
First, it retains high-frequency textures and thus proposes a high-frequency surface shell created based on depth information and samples it on the shell to improve the capacity utilization of the model in the surface area.
The second is the rationality of sampling, so a method of combining voxel grid and MLP features is proposed to combine the advantages of the two. 
Finally, the CNN was used for post-processing to ensure the high quality of the result
strength_weakness: strength_weakness: Whether it is visual quality or indicator quality, this method has certain advantages over previous methods. At the same time, the surface shell sampling method proposed by this method is very novel.
limitations: limitations: Although this method can achieve very good results, its model parameters are too large, and the overall architecture is very complex compared to other methods.
paper_summary: paper_summary: This paper proposes a method called HS-Surf for the task of large-scale scene reconstruction. It first uses a hybrid representation from instant-NGP and Mip-NeRF to build a coarse scene. Then refines the surface depth and appearance details by training two additional MLPs. Substantial experiments and ablation experiments are conducted.
strength_weakness: strength_weakness: - According to the reported results, the novel idea of building compensation modules is effective in refining high-frequency details for large-scene reconstruction.
- Substantial experiments and ablation experiments are conducted, with extensive quantitative and qualitative results reported.
limitations: limitations: - **Concept.** The method is called a "Surface Shell Radiance Field" that "constructs high-frequency shells on the scene surface", which seems to be hugely different from previous NeRF-based methods in representation. But actually, it heavily relies on a traditional neural field to shrink the sampling range and offer a basic appearance, and then train new NeRFs for compensation. I do not think this representation is significantly different in technique from GridNeRF [7] and instant-ngp [27] (similarly, it uses an occupancy grid to limit the sampling range, which the authors have not discussed), as the "surface shell" does not dominate the representation but is just for compensation. Meanwhile, the proposed sampling strategy is also somehow similar to some existing methods that only sample points near the surface. Overall, it's unsuitable and unnecessary to raise such a new and confusing concept to describe the proposed method. Please consider simplifying it. 

- **Contribution.** The second contribution declared in this paper is to fuse the features from conical frustum with grid features at the initialization stage. However, it seems to not contain a special design for large-scale scenarios. While this part shares a similar idea with ZipNeRF [8], the authors didn't provide discussions about the differences between them. Meanwhile, as this point can not provide sufficient novel insights, it is likely not worth being individually listed in the introduction.

- **Problems in evaluation.** In Table 1,  the reported performances of the baselines, especially GridNeRF, drop tremendously from its original papers (e.g. for GridNeRF, 25.505->20.00 in PSNR of *Campus*). And I couldn't find any explanation for this in the paper. The results are less convincing in my view. More explanations are necessary about it.

- **Doubtful declaration.** In the abstract, the authors claimed existing problems are "due to insufficient model capacity on the scene surface. This is attributed to their sampling locations or grid vertices falling in empty areas". This causal relationship needs to be proven or explained. Also, the proposed method has not been verified to exceed other methods in avoiding sampling empty areas.

- **Missing work.** The paper has ignored a famous related work Switch-NeRF. Nevertheless, it may not influence the conclusions. 

- **Descriptions.** 1) Are 1/50 and 1/200 generative for all scenes? Otherwise, I think the specific value should be mentioned only in the implementation details. 2) If the "Hash-based voxel grid" in Section 3 refers to instant-ngp or some other works, it should be clearly clarified by a citation.

Will consider raising the rating if the first four concerns can be well explained.