Superpixel-based Efficient Sampling for Learning Neural Fields from Large Input

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 13:29:02

Abstract: In recent years, novel view synthesis methods using neural implicit fields have gained popularity due to their exceptional rendering quality and rapid training speed. However, the computational cost of volumetric rendering has increased significantly with the advancement of camera technology and the consequent rise in average camera resolution. Despite extensive efforts to accelerate the training process, the training duration remains unacceptable for high-resolution inputs. Therefore, the development of efficient sampling methods is crucial for optimizing the learning process of neural fields from a large volume of inputs. In this paper, we introduce a novel method named Superpixel Efficient Sampling (SES), aimed at enhancing the learning efficiency of neural implicit fields. Our approach optimizes pixel-level ray sampling by segmenting the error map into multiple superpixels using the slic algorithm and dynamically updating their errors during training to increase ray sampling in areas with higher rendering errors. Compared to other methods, our approach leverages the flexibility of superpixels, effectively reducing redundant sampling while considering local information. Our method not only accelerates the learning process but also improves the rendering quality obtained from a vast array of inputs. We conduct extensive experiments to evaluate the effectiveness of our method across several baselines and datasets. The code will be released.

Main paper: 
 introduction: 
 Novel view synthesis(NVS) has always been a hot research topic in computer vision, with wide-ranging applications in virtual reality, medical imaging, and augmented reality. In recent years, NVS methods that represent scenes using neural fields have gradually become prevailing due to their promising results [16-18, 21, 24, 41, 42, 46] . These methods represent the scenes with Deep Neural Networks (DNNs) and render the scenes through volume rendering. Compared with traditional Multi-View Stereo (MVS) methods [3, 5, 9, 11, 12, 40] , they can easily handle textureless surfaces and interpolate unseen areas via regularization techniques [13] . However, the volume rendering of huge sampled points brings substantial The top right corner displays PSNR curves for both methods on the test set, while the visualization below showcases their rendering results. Incorporating our method enhances the convergence speed and rendering quality of Instant NGP noticeably.computational overhead, which becomes even more severe with increasing image resolution.For novel view synthesis, the resolution of input images directly determines the upper limit of reconstruction quality. With the advancement of camera technology, the resolution of captured images continues to increase. However, high-resolution images not only capture more details but also bring a substantial amount of redundant data. Therefore, traditional random pixel sampling methods [20, 34] become overwhelmed by the huge number of pixels in high-resolution inputs. [44] employs a patch sampling strategy combined with an encoder-decoder structure. They use the encoder to encode 3D geometric information and the decoder to render high-resolution outputs. [35] introduces color-guided ray sampling. They guide the sampling of rays by color and depth, allowing the rays to pass through regions with richer spatial information. However, their method does not simultaneously consider both color and rendering errors, and the redundant sampling of rays results in slower convergence on high-resolution images.To address the challenge of large input, we introduce a Superpixelbased Efficient Sampling (SES) method for learning neural fields. Our core idea is to increase the number of light paths to acquire more information, especially in regions with significant rendering errors, thereby reducing redundancy in sampling. Simultaneously, we aim to optimize the sampling of light rays to consider local image information. We first employ the SLIC algorithm to segment the input image into different superpixels, where each superpixel represents distinct local information. Compared to other segmentation methods like quadtree, superpixels can significantly reduce redundancy. In subsequent steps, we treat superpixels as the smallest processing unit. We set different sampling probabilities for different superpixels based on their rendering errors, continuously update them during training, and incorporate the rendering errors into the loss function. We apply our method to several popular NeRF algorithms and test it on multiple common datasets. Our image rendering quality reached the state of the art at different resolutions, and integrating our method did not incur additional computational overhead. In summary, our main contributions can be summarized as follows:• We propose an efficient and effective sampling method for learning neural fields from large input. By segmenting the image into superpixels, we reduce light ray redundancy, improve sampling efficiency, and take into account local sampling information.  
related work 2.1 novel view synthesis: 
 In graphics research, the pursuit of Novel View Synthesis (NVS) hinges on adeptly handling intermediate 3D scene representations. Previous endeavors have focused on exploring suitable 3D representations: Mesh-based or pointcloud-based methods [2, 4, 6, 27, 39] often rely on depth information or structure from motion(SFM) [28] techniques for geometric shape restoration, presenting challenges in rendering realistic images. While approaches utilizing multiplane images(MPIs) [7, 19, 32, 33, 38, 51] to express scenes as feature maps or stacked images can yield high-quality renderings, they constrain the variability of perspectives. Voxel-based representations [14, 17, 25, 29, 30, 36] offer swift image rendering but face limitations in resolution. By modeling scenes as neural implicit fields [20, 22, 31, 47] , greater flexibility is attained alongside high-quality image generation: Differentiable Volume Rendering (DVR) [23] and Implicitly Differentiable Renderers (IDR) [48] have demonstrated promising results, albeit still requiring precise object mask inputs during training. The year 2020 saw the groundbreaking introduction of NeRF [20] , revolutionizing NVS through volume rendering.NeRF employs neural networks to model spatial 3D coordinates and view directions as volume density and color, employing alpha blending for pixel color rendering, resulting in remarkable outcomes. NeRF's advent has significantly advanced the domains of new view synthesis, 3D reconstruction, point cloud rendering, and beyond.NeRF employs volume rendering to render pixel colors, and to bring sampling points closer to the surface, it introduces a twostage sampling method along the rays. However, such an approach typically demands 1-2 days or even several days for training completion. To expedite NeRF training, ENeRF [10] utilizes the geometric features of the scene to guide surface sampling. Some methods leverage explicit modeling techniques to accelerate scene training: kilo NeRF [26] decomposes NeRF into multiple small MLPs for faster training, while DVGO [34] , Plenoxels [8] , and similar works employ learnable explicit meshes to expedite training. Instant ngp [21] utilizes multi-resolution hash encoding with a hash table and jump sampling to swiftly approach the scene surface during training. The aforementioned works focus on accelerating sampling strategies in space; Wang et al. [50] use a quadtree to guide sampling, reducing the number of sampled rays to accelerate training, while Sun et al [35] . efficiently sample rays by filtering them using images and depth information. These methods greatly accelerate NeRF training, although they may encounter limitations when high-resolution images are input.To handle high-resolution image inputs and generate high-resolution images, 4k-NeRF [44] encodes high-resolution images into feature maps. After training, a decoder is used to decode these feature maps into high-resolution images. This method reduces the computational burden during NeRF training. However, joint training of the encoder-decoder with NeRF still requires a considerable amount of time. UHD-NeRF [15] reconstructs higher-resolution images by using a combination of neural networks and point cloud representations. It intends to reconstruct the low-frequency components of the scene using MLP and the high-frequency components using point clouds. It is worth noting that both methods mentioned above use patch sampling to avoid excessive memory consumption. While patches can capture local information in the scene, their fixed shape may lead to fragmented image information. 
method: 
 Our primary objective is to address the slow convergence and scattered sampling issues brought about by high-resolution inputs, ensuring superior rendering results within the same training timeframe. Firstly, in Section 3.1, we introduce the NeRF [20] and SLIC [1] segmentation algorithm, forming our work's foundation. Subsequently, in Section 3.2, we present our two-stage segmentation method and how we utilize superpixels to guide sampling. Finally, in Section 3.3, we demonstrate how our method integrates into existing baselines. Figure 2 illustrates our entire pipeline.NeRF: In NeRF [20] , a 3D scene as a continuous function which takes as input 3D position 𝑥 = (𝑥, 𝑦, 𝑧) and viewing direction 𝑑 = (𝜃, 𝜙) and predicts the radiance color 𝑐 and volume density 𝜎. The architecture of 𝑓 𝜃 is chosen such that only the color 𝑐 depends on the viewing direction 𝑑. This allows the modeling of view-dependent effects like specularities and reflections while also encouraging a consistent geometry to be learned. In a deterministic pre-processing step 𝑥 and 𝑑 are transformed by a positional encoding 𝛾 which promotes learning of high-frequency details. NeRF is typically parametrized by a multilayer perceptron (MLP) 𝑓 : (𝛾 (𝑥), 𝑑) = (𝑐, 𝜎). To render a pixel by a given camera pose, the expected color Ĉ (𝑟 ) of a camera ray 𝑟 = 𝑜 + 𝑡𝑑 through the pixel that shots from the camera 𝑜 in direction 𝑑 can be calculated asEQUATIONwhereEQUATIONthe accumulated transmittance indicates the probability that a ray travels from 𝑡 𝑛 to 𝑡 without hitting any particle. NeRF is trained to minimize the mean-squared error(MSE) between the predicted renderings and the corresponding ground-truth color:EQUATION)where P denotes all pixels of training set images, Ĉ (𝑟 𝑝 ) and 𝐶 (𝑟 𝑝 ) are the ground truth and output color of 𝑝. SLIC: Simple Linear Iterative Clustering(SLIC) [1] is a fast and effective algorithm for image segmentation. As shown in algorithm1 combines superpixel methods with K-means clustering to cluster pixels in the color space, thus achieving image segmentation. SLIC first grids pixels into compact superpixels and then groups these superpixels into segments based on pixel similarity and distance measurements. By controlling superpixels' compacity and distance parameters, SLIC can significantly reduce computational complexity while preserving image details. Furthermore, SLIC excels in preserving boundary information of the image, allowing the segmentation results better to retain the sharpness and continuity of Algorithm 1 SLIC superpixel segmentation Compute the distance 𝐷 between 𝐶 𝑘 and 𝑖. 𝐷 = √︄ (𝐿 -𝐿 𝑖 ) 2 + (𝑎 -𝑎 𝑖 ) 2 + (𝑏 -𝑏 𝑖 ) 2 + 𝑆 -𝑆 𝑖 𝑆 𝑁 2 (4)where 𝐿, 𝑎, 𝑏 are the values in the LAB color space of the pixel, 𝑆 is the position of the pixel,𝐿 𝑖 , 𝑎 𝑖 , 𝑏 𝑖 are the values in the LAB color space of the center of the superpixel block, 𝑆 𝑖 is the position of the center of the superpixel block, 𝑆 𝑁 is the normalization factor, equal to image size divided by the number of superpixel numbers.NeRF [20] perceives space by learning a color-constrained network to output color and density information of three-dimensional spatial points. Compared to simple regions in space, complex regions require more iterations for the network to better learn spatial information. Previous methods [35, 50] have attempted multiple sampling in regions with large rendering errors or drastic color changes while reducing sampling in areas with small rendering errors to achieve better results. Redundant computations resulting from insufficient partitioning by quadtrees and other segmentation methods increase continuously with input scale and are not feasible. Methods guiding computation through color gradients also struggle with handling large-scale image blocks. To address these issues, we use the SLIC[? ] algorithm to segment images into different superpixels, which serve as the smallest processing units for guiding sampling. Due to the characteristics of superpixels, the problem of insufficient partitioning caused by structures like quadtrees is resolved, reducing computational overhead in single processing instances.Image to superpixel During the training process, we will update the SLIC segmentation results twice, which enables the segmentation to focus on rendering errors while conforming to geometric constraints. At the beginning of training, we will employ the SLIC algorithm to partition the input RGB image into superpixels.EQUATIONwhere ℓ 𝑖 represents the 𝑖th superpixel in the image, 𝐿 denotes the collection of all superpixels segmented in the image, 𝐹 represents the SLIC algorithm1, with its process outlined in Algorithm 1, and 𝑇 signifies the transformation of the image from the RGB color space to the ICE color space. Then, during the training process, we will update the superpixel segmentation results for the second time to make the segmentation more focused on areas with larger rendering errors. However, it is not necessary to update all superpixels to avoid disrupting the original segmentation structure. Refer to Figure 3 , we will concentrate the re-segmentation effort on the areas with the largest rendering errors and their adjacent areas, using the masked slic algorithm to re-segment these regions. In the re-segmentation process, both RGB color and rendering error are considered simultaneously. First, the rendering error is mapped to the RGB color space, and then it is converted to the CIE color space. During the distance calculation with the SLIC algorithm, the distances in both the RGB and CIE color spaces are considered.EQUATIONWhere 𝐼 represents the portion of the input RGB image that needs to be re-segmented, 𝑅 represents the portion of the render error that needs to be recalculated. This portion is replicated three times to meet the dimensional requirements for the transformation to the ICE space. 𝛽 0 is a regularization term used to adjust the weights between the RGB image and the render error. Sample with superpixel We employ superpixels to guide the pixel-level sampling during the training process. Initially, we compute the average sampling error within each superpixel. Each superpixel's error ℓ 𝑖 𝑒𝑟𝑟𝑜𝑟 is computed as follows:EQUATIONwhere 𝑁 represents the number of pixels sampled in this segment, and 𝑟 𝑗 denotes the rendering error of the current pixel. We will calculate the sampling probability for each superpixel based on its rendering error and RGB color. Initially, we will exclude superpixels with negligible errors, corresponding to either blank areas in the scene or well-learned regions. Rendering error serves as the primary indicator for sampling probability calculation. However, due to significant variations in rendering error scales within the scene, sampling tends to concentrate on a few superpixels. To ensure comprehensive scene coverage, we employ the square root of the error as a representation of sampling probability. Despite the stochastic nature of sampling, sampling error may not fully capture sampling probability. Thus, we supplement this with RGB information. By computing the variance of colors within each superpixel, we collectively determine the sampling probability for each superpixel. Therefore, the sampling probability 𝑔(ℓ 𝑖 𝑒𝑟𝑟𝑜𝑟 ) for each superpixel can be calculated as follows:EQUATIONEQUATION)EQUATION)where 𝑡 denotes the minimum error threshold we set. We normalize the sampling probability of the rendering error component to lie between 0 and 1, 𝑚 represents the number of pixels in the current superpixel, and c denotes the average color within the superpixel. 𝜆 𝑒𝑟𝑟𝑜𝑟 and 𝜆 𝑠𝑡𝑑 are the coefficients for the sampling probabilities.We adjust these coefficients to balance the weights of the two probabilities.We integrate superpixel sampling into NeRF training and employ a series of strategies to adapt to high-resolution inputs, thereby enhancing the rendering quality of NeRF.Training Progress. Our training process is illustrated in Figure 2 . Initially, we employ a random sampling method for training while updating the errors of superpixels in each training iteration. After the secondary segmentation computation, we utilize superpixels to guide sampling and ray generation, concurrently accumulating rendering errors. Upon completion of training for all rays generated in a single sampling, we update the superpixel errors and regenerate rays.Loss Design. We optimize the parameters of the MLP using pixel-level Mean Squared Error (MSE) loss L 𝑚𝑠𝑒 and superpixel loss L 𝑠𝑝 . Our total loss is computed as follows:L = 𝜆 𝑚𝑠𝑒 • L 𝑚𝑠𝑒 + 𝜆 𝑠𝑝 • L 𝑠𝑝 (11)The calculation method of L 𝑠𝑝 is as follows:EQUATION)Due to the strong bias of L 𝑠𝑝 , 𝜆 𝑠𝑝 is typically set to be three to four orders of magnitude smaller than the 𝜆 𝑚𝑠𝑒 during the training process.Progressive Training. In NeRF [20] and its related works, position encoding is utilized to enable MLPs to learn more highfrequency information. However, in practical tests, we found that while higher frequency position encoding captures more detailed information, it leads to slower convergence. In our work, we focus on enabling the training to rapidly learn the scene's low-frequency information in the initial, low-frequency stages. Therefore, we adopt a strategy similar to that used in NeuS2 [42] and FreeNerf [45] , gradually increasing the bandwidth of the position encoding during the training process. In NeRF, the representation of position encoding is as follows.𝛾 𝐿 (x) = sin(x), cos(x), ..., sin(2 𝐿-1 x), cos(2 𝐿-1 x) (13)EQUATIONwhere, 𝐿 represents the maximum frequency in position encoding, and 𝛾 𝐿 (x) defines the specific formulation of position encoding.The position encoding used in our work is 𝛾 ′ 𝐿 (x) which is formed by multiplying 𝛾 𝐿 (x) with 𝑀 which is a mask with the same dimension as 𝛾 ′ 𝐿 (x), designed to mask the high-frequency components, with its specific parameters controlled by the number of input iterations 𝑖. In practical training, we set the maximum frequency of the position encoding to a quarter of its original value, which is gradually increased as the training progresses and the resolution enhances. 
experiments 4.1 implementation details: 
 In comparison to structures like quad-trees, superpixels do not possess similar tree-like structures to minimize sampling redundancy. To fully leverage hardware performance, we provide our method's implementation on CUDA. In our implementation, for synthetic datasets, we set 𝛽 0 to 0.8 as per Equation 6, 𝜆 𝑚𝑠𝑒 as 1, 𝜆 𝑠𝑝 as 1e-6, whereas for real datasets, we choose 𝛽 0 as 0.65, 𝜆 𝑚𝑠𝑒 as 1, 𝜆 𝑠𝑝 as 1e-8. Our method can be seamlessly integrated into the popular NeRF framework. To demonstrate the efficiency of our approach, we selected Instant-NGP [21] and DVGO [34] as our baselines, both known for their fast training speeds within the NeRF domain. It is worth noting that, in our experiments, we utilized an open-source implementation of Instant NGP, namely the nerf-template, which is a clean version derived from the open-source project torch-ngp [37] . We employed this implementation in our experiments as a replacement for the original Instant NGP. We conducted experiments using a single NVIDIA RTX 4090 GPU.We tested our method on the currently popular NeRF dataset to demonstrate its superiority. We conducted experiments on three datasets: eight synthetic 360°scenes provided by NeRF, four scenes from Tank and Template, and eight real scenes from MipNeRF 360 dataset. We also provided ablation experiments to validate our method. We evaluate the quality of view synthesis for ground truth from the same pose using three metrics: PSNR, SSIM [43] , and LPIPS[49].The Realistic Synthetic 360 evaluation dataset [20] is a commonly used benchmark in the NeRF field. Using Blender, it renders eight scenes, each with distinct characteristics, and generates 100 training images per scene, making it a popular choice for multi-view reconstruction benchmarks. While the native renderings are at a resolution of 800x800, based on the provided Blender files, we generated images at higher resolution(1600x1600, 3200x3200). These newly rendered high-resolution images maintain the original viewing angles but offer enhanced clarity. Due to memory constraints, we conducted experiments only at resolutions of 800 and 1600 on DVGO.The quantitative results of our method are presented in Table 1 . We compare our approach with the baseline at different scales, revealing superior performance across all resolutions, notably in Hotdog and Lego scenes. Visualizations of our results are illustrated in Figure 4 , demonstrating enhanced scene details and reducedInstant ngp Ours artifacts, particularly evident in scenarios with large-scale inputs, upon integration of our method.MipNeRF 360 is a commonly used real-world unbounded scene dataset, comprising intricate scenes without boundaries. We conducted experiments on seven distinct scenes within this dataset. Each scene dataset includes original 4K images alongside their corresponding downsampled versions. We evaluated the 2x and 4x downsampled input images. Results demonstrate that our proposed approach outperforms baseline models, particularly excelling in high-resolution scenes.In Table 2 , we present the quantitative results of our experiments, demonstrating that our method adapts well to both resolutions and consistently improves performance over the selected baselines. Our visual results are showcased in Figure 5 , where it can be observed that our method reconstructs more details compared to the baselines, particularly evident with high-resolution inputs. We also monitored the impact of incorporating our method on the training time of the baseline. The addition of our method results in only a Instant ngp(d4) ours(d4) Instant ngp(d2) ours(d2)Figure 5 : The experimental results graph for MipNeRF. We compared our visualization results with Instant NGP at two downsampling rates: 2x and 4x. Our method consistently produces higher-quality images at both resolutions, with a more pronounced effect at higher resolutions."slight increase in training time, posing no significant computational burden.The Tankandtemple dataset provides eight scenes at a resolution of 1080p. We tested our method on four bounded scenes within the dataset. As shown in Table 3 , our method achieves state-of-the-art performance in several scenes provided by the dataset. Figure1 in the paper demonstrates the convergence speed and final rendering results of our method compared to the baseline. We observe faster convergence and better image quality with our method. Figure 6 illustrates the rendering results of our method at the edges, where we capture high-frequency information more effectively, resulting in clearer rendered images.Our method employs a two-stage segmentation approach, aiming to balance the influence of color and error on segmentation. To validate the effectiveness of our method, we separately test the results of segmentation based solely on RGB color and solely on Table 3 : These are the quantitative results for the Tankand-Tamplate dataset. As demonstrated, our method shows performance improvements in the majority of the scenes.rendering error. We use Instant NGP as the baseline and test on the NeRF synthetic dataset. As shown in the table 4, segmentation based solely on RGB color leads to unstable results. While it achieves good performance in some scenes, the results in certain scenes become unacceptable. On the other hand, using error-based segmentation alone results in relatively stable improvements, but areas with large errors may overlook structural information in the scene. Therefore, our two-stage sampling strategy strikes a good balance between stability and rendering quality. 
conclusion: 
 In this paper, we introduce a method called SPS which employs superpixels to guide the pixel-level ray sampling process during NeRF training, allowing for the processing of large-scale image inputs and the generation of high-quality rendered images. Leveraging the characteristics of superpixels, we devise a novel segmentation, updating, and sampling strategy tailored to them. Our strategy effectively balances the relationship between sampling errors and color information, enabling more efficient information extraction from large-scale inputs. Additionally, we propose a new loss function that better utilizes the characteristics of superpixels. In experimental evaluations, our approach achieves state-of-the-art performance compared to existing methods.  
 
 Appendix:  


Decision: Authors have proposed novel method for enhancing the efficiency and accuracy of learning neural fields. The use of superpixel segmentation via SLIC algorithm to partition the image into distinct local information units is a novel approach, supported by a robust theoretical framework including color space conversion. The SES method demonstrates superior performance in reconstructing high-frequency details as the resolution of synthesized view increases. This evidenced both quantitatively and qualitatively through experiments.

The shortcoming of this paper is in its ablation studies. There are lack of result shown in ablation study to show effectiveness of its method compared to other possibilities. Because the limitation of this paper is merely in its ablation study, I am lean toward accepting this paper for a poster session

Reviews: review: paper_summary: paper_summary: This paper introduces a novel method called Superpixel-based Efficient Sampling(SES) to enhance the efficiency and accuracy of learning neural fields, particularly for novel view synthesis tasks. SES effectively addresses challenges such as redundant sampling of rays and slower convergence on high-resolution images by optimizing light ray sampling. It leverages superpixels to reduce redundancy and concentrate on areas with significant rendering errors.
strength_weakness: strength_weakness: The use of superpixel segmentation via SLIC algorithm to partition the image into distinct local information units is a novel approach, supported by a robust theoretical framework including color space conversion. The SES method demonstrates superior performance in reconstructing high-frequency details as the resolution of synthesized view increases. This evidenced both quantitatively and qualitatively through experiments.
limitations: limitations: Despite the focus on efficient and effective sampling for learning neural fields, the paper lacks sufficient quantitative evaluation regarding efficiency metrics, such as training time. Additionally, it does not analyze the increase in training time when applied to the MipNeRF dataset.
paper_summary: paper_summary: This work proposes an efficient and effective sampling method for learning neural fields from large inputs. It utilizes superpixels to guide the pixel-level ray sampling process during NeRF training.
strength_weakness: strength_weakness: The visualization results seem impressive, and the motivation for employing a superpixel-based approach is reasonable.

The writing is clear and well-articulated.
limitations: limitations: 1. Could you provide comparisons of efficiency changes between your method and quadtree sampling as the resolution increases? This would further highlight the effectiveness of your proposed method.

2. In Table 4, why does instant ngp(rgb) outperform your proposed method by approximately 1dB?

3. There are many hyperparameters involved; are there any ablation studies concerning them?

4. There are typographical errors. For example, "33 neighborhood" (Line 324-325) and "SLIC[?]" algorithm (Line 372-373).

5. I suggest including more citations and discussions of relevant MM conference papers on 3D to enhance the article's relevance for this conference. For example, "Geometry-Aware Reference Synthesis for Multi-View Image Super-Resolution" and "Space-Angle Super-Resolution for Multi-View Images."
paper_summary: paper_summary: A super-pixel efficient sampling method (SES) is introduced to sample large input data.
strength_weakness: strength_weakness: The SES method optimizes pixel-level ray sampling by segmenting the error map into multiple superpixels using the slic algorithm, and dynamically updating their errors during training to increase ray sampling in areas with higher rendering errors. 
The authors conduct some experiments to evaluate the effectiveness of their method.
limitations: limitations: However, there are still some shortcomings in the paper.

1. In the case that there is still space left in the paper, the visual comparative experiment is still insufficient, and there is no supplementary material to show more experimental effects. 
2. There is no ablation experiment of the proposed loss function and segmenting the image into superpixels.