Volume Feature Rendering for Fast Neural Radiance Field Reconstruction
Authors: Kang Han
Wei Xiang
Lu Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: neural rendering
volume rendering
view synthesis
3D reconstruction

Abstract: Neural radiance fields (NeRFs) are able to synthesize realistic novel views from multi-view images captured from distinct positions and perspectives. In NeRF's rendering pipeline, neural networks are used to represent a scene independently or transform queried learnable feature vector of a point to the expected color or density. With the aid of geometry guides either in the form of occupancy grids or proposal networks, the number of color neural network evaluations can be reduced from hundreds to dozens in the standard volume rendering framework. However, many evaluations of the color neural network are still a bottleneck for fast NeRF reconstruction. This paper revisits volume feature rendering (VFR) for the purpose of fast NeRF reconstruction. The VFR integrates the queried feature vectors of a ray into one feature vector, which is then transformed to the final pixel color by a color neural network. This fundamental change to the standard volume rendering framework requires only one single color neural network evaluation to render a pixel, which substantially lowers the high computational complexity of the rendering framework attributed to a large number of color neural network evaluations. Consequently, we can use a comparably larger color neural network to achieve a better rendering quality while maintaining the same training and rendering time costs. This approach achieves the state-of-the-art rendering quality on both synthetic and real-world datasets while requiring less training time compared with existing methods.

Volume Feature Rendering for Fast Neural Radiance Field Reconstruction
Kang Han School of Computing, Engineering and Mathematical Sciences La Trobe University (k. han, w.xiang, 1. 1.yu)@latrobe edw.obbbeeedd.....
Wei Xiang" Lu Yu
Abstract
Neural radiance fields (NeRFs) are able to synthesize realistic novel views from multi-view images captured from distinct positions and perspectives. NeRF's rendering pipeline, neural networks are used to represent scene independently or transform queried learnable feature vector density. With the aid of geometry guides either in the form of occupancy grids or proposal networks, the number of color neural network evaluations can be reduced from hundreds to dozens in the standard volume rendering framework. However, many evaluations the color neural network are still a bottleneck for fast NeRF reconstruction. This paper revisits volume feature rendering (VFR) for the purpose of fast NeRF reconstruction. The VFR integrates the queried feature vectors of a ray into feature vector, which is then transformed the final pixel color by a color neural network. This fundamental change to the standard volume rendering framework requires only one single color neural network evaluation to render a pixel. which substantially lowers the high computational complexity the rendering framework attributed evaluations. Consequently, we can use a comparably larger color neural network to achieve a better rendering quality while maintaining the same training and rendering time costs. This approach achieves the state-of-the-ee rendering quality on both synthetic and real-worle datasets while requiring less training time compared with existing methods.
point to the expected color or
large number of color neural network
Introduction
For the task of view synthesis, unobserved views of a scene are synthesized from captured multi-view images. This long-standing problem that has been studied for several decades in computer vision and computer graphics. The most popular solution to this problem at present is the neural radiance field (NeRF) [8] since it can render views with high fidelity. NeRF represents the density and color of spatial point in a given direction using neural network (NN), typically the multilayer perceptron (MLP). With the predicted densities and colors of sampled points along a ray, the NeRF and its variants 2 3 28
use the volume rendering technique to aggregate colors obtain the final rendered pixel color. As many samples are required render one pixel, the underlying MLP needs to run many times, leading to high computational complexities for both training and rendering. Alternatively, researchers introduced extra learnable features in the form of 3D grids [25 [7], hash tables [19] and decomposed tensors [6] 12 7 23) in addition to MLP's parameters represent scene's density and color fields. As querying the feature vector of sampled point by interpolation is much faster than one MLP evaluation, only small MLP is used transform the queried feature vector the density or color, leading to substantial speed acceleration compared with pure MLP representations.
"Correspodiing author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Furthermore, modeling the density field independently by computational efficient representations can greatly reduce the number of samples in the color field. The density representations could be occupancy grids [9], hash grids with less parameters [16], and small MLPs 33. These coarse density fields enable importance sampling to make the model focus on non-empty points near the surface. As result, only dozens of samples in the color field (typically represented by both learnable features and NNs) are needed to render a pixel. For example, Zip-NeRF uses 32 samples after importance sampling through efficient density representations.
Nevertheless, standard volume rendering techniques still need rum color NN dozens of times to render single pixel. This is why Instant-NGP [99] uses a small color MLP for fast training and rendering, and why the latest Zip-NeRF 4 is comparably slow when using large color MLPs to achieve a better rendering quality. Although a radiance field can be implemented even without NNs 19), the importance of large color NNs for high-quality rendering is undoubted, considering that the state-of-the-art rendering quality can be achieved by pure MLP representations. The authors of Zip-NeRF 4 and NRFF 12 also highlight the importance of large color NNs in modeling view-dependent effects. Unfortunately, the standard volume rendering technique limits the size of color NNs for the sake of fast training and rendering.
One the other hand, research in generative NeRF models [5] shows that the feature vectors of samples along a ray can be rendered or integrated first to enable one single evaluation of following NNs. However, it is unclear whether this volume feature rendering (VFR) technique can also be used for fast NeRF reconstruction. Towards this end, this paper revisits the VFR method for the purpose of fast NeRF reconstruction. Our investigation reveals that the VFR can be integrated into the training phase of NeRF models, thereby enabling the utilization of large color neural networks to either enhance rendering quality while maintaining the same training time, or to achieve a similar rendering quality with much faster training.
21 and fast NeRF rendering 13
2 Related work
The volume rendering technique integrates the colors of samples along ray according to their densities [ A global density field that is differentiable is very effective in finding the underlying geometry of scene from multi-view 2D color observations using volume rendering. This capability is one of the essential reasons to NeRF's success. As every point in a scene has a density value, the density field is capable of modeling complex geometry. However, standard volume rendering employed by NeRF and its successors [8 3 26 35 27
6 34] suffer from high computational complexity caused by many NN calls even with importance sampling guided by coarse density fields, as discussed in the preceding section.
Representing the underlying geometry by the signed distance function (SDF) leads to a well-defined surface to to enable one sample-based fast rendering R3 g 29
IC However, the SDF struggles with modeling complex geometry in the context of neural inverse rendering, e.g., trees and leaves, and this is why the best rendering quality is still achieved by volume representations. In this paper, we revisit the VFR framework that uses density the geometry representation but shares similar behavior as the SDF that processes a feature vector by color NNs only once. Thus, the VFR inherits the advantage of volume representation in modeling complex geometry and also has the strength of the SDF in single NN evaluation.
Generative NeRF models such as EG3D (5). StyleNeRF [1] and StyleSDF (1 focus on 3D consistency and fidelity of generated content using generalizable neural networks with NeRF's structure. In comparison, 3D reconstruction from multi-view images using NeRF is different task from 3D generation. As per-scene optimization is required for NeRF reconstruction, the reconstruction speed is critical issue in this field, but this problem has not been investigated in the existing generative NeRF works. This paper revisits feature integration in volume rendering for fast reconstruction and demonstrate its effectiveness in the NeRF reconstruction task.
is noted that feature accumulation is also discussed in the Sparse Neural Radiance Grid (SNeRG) 1 mainly for the purpose of modeling specular effects. However, the diffuse color is still obtained using the standard volume rendering method in [33, which requires many times of MLP evaluations. Besides, the SNeRG needs to train a NeRF first (require days to train) and then bakes the trained NeRF to a sparse grid with specular features. this work, we demonstrate that this pre-training is not necessary, and the accumulated feature vector can be used to predict the final view-dependent

---

color instead of only the specular color. Our VFR method can be directly optimized from scratch instead of baking the optimized features in the SNeRG. As result, our method requires significantly less training time but achieves a much better rendering quality compared with the SNeRG.
Integrating feature vectors of samples along the ray is natural in transformer models
24 30 For
example, Suhail et al. proposed an epipolar transformer to aggregate features extracted from reference views on epipolar lines (24). However, the computational cost of transformer is much higher than MLP and that transformer-based method is significantly (8 times) slower than MLP-based Mip-NeRF 2 on the same hardware. In this work, we demonstrate that the VFR's aggregation only involves a weighted combination of feature vectors based on densities such that the VFR is significantly faster than transformer-basee feature aggregation.
3 Model architecture
3.1 Standard volume rendering
Standard volume rendering integrates color along cast ray according to density [15]. A high density of a point on the ray indicates high probability of hitting the surface. The NeRF and its various variants adopt this standard volume rendering technique, where the color and density of point are typically predicted by a neural network. For a cast ray r(() = o + Id. where o is the camera origin and d is the view direction, the rendered color C(r) using standard volume rendering is:
C(r) - (et)=)((ttarriiiii) (()))(r(()))((((())) d) dt, where T(t) = exp (-/ a(((s) ds
(1)
where (x) and c(x, d) are the density and color at position x, repsectively. In practice, the above integral is solved by sampling and integration along the cast ray. After transforming the densities to weights by ur(() = (r)))r(()), the color will be rendered as follows:
C(r) Σ wr NeuralNee (F(x;),
(2)
where x is the position of a sample point on the ray and F is a function to query the corresponding feature vector of x. Pure MLP-based methods including the NeRF [8), Mip-NeRF (2], Ref-NeRt
(8) and Mip-NeRF 360 [3 represent F by using MLPs. However, this representation imposes computational burden as N MLP evaluations
required to render ray. Alternatively, recent shows that modeling F by extra learnable features is significantly faster than pure-MLP based representations, because linear interpolation of learnable features is much more efficient than MLP evaluations. These learnable features are typically organized in the form of the 3D grids, hash tables, and decomposed tensors. With the learnable features, a small NN can achieve the state-of-the-art rendering quality but accelerate both the training and rendering time significantly.
research [99 25 6
Furthermore, hicrarchical importance sampling guided by coarse geometries is employed greatly reduce the number of samples in (2). This importance sampling can be achieved by modeling the
density fields independently 125 6 12). occupancy grid [19], and proposal networks 3 4. As shown in Fig. [ importance sampling makes the model focus on valuable samples on the surface. For instance, by using two levels of importance sampling, Zip-NeRF [4] reduces the final number of valuable samples to 32. However, the color NN still needs to run 32 times to deliver the best rendering quality. Although integrating colors predicted by the color NN is in line with the concept in classical volume rendering, the many color NN calls are the main obstacle for fast training and rendering. On the other hand, the importance of large color NNs for realistic rendering especially in modeling the view-dependent effect is highlighted in recent works [2] Considering the fact that valuable samples on the surface are close in terms of spatial position (see red sampling points in Fig.[]. the queried feature vectors are also very similar as they are interpolated according to their position. As such, evaluating the color NN with similar input feature vectors is not necessary. In the next subsection, we will introduce the volume feature rendering method that integrates these feature vectors to enable a single evaluation of the color NN.
3.2 Volume feature rendering
As shown in Fig. [] the volume feature rendering method integrates queried feature vectors instead of predicted colors to form a rendered feature vector. A subsequent color NN is then applied to
3

---

Density
Sample on surface
Empty sample
Cast ray
Feature Querying
-V
Feature Querying
NeuralNet
Feature Integration
Color Integration
NeuralNet
Rendered color
Rendered color
Standord Volume Rendering
Volume Feature Rendering
Figure 1: Standard volume rendering needs to run NeuralNet many times for all the samples on the surface, while the volume feature rendering only needs evaluate NeuralNet once.
transform the feature vector to the final rendered color. By doing this, the color NN only needs to run once. Specifically, we take the NN out of the integral as follows:
Σ wi F(xi)
(rr) NeuralNet
This fundamental change to the standard volume rendering framework relieves the rendering frame- work from the high computational cost caused by many NN evaluations. Consequently, we can use larger network achieve better rendering quality while maintaining a similar training time.
3.3 Pilot network
is reasonable to integrate feature vectors of non-empty samples in the VFR. However, at the beginning of training, the model does not have coarse geometry focus on samples the surface. The VFR will integrate feature vectors of all samples during the early training stage. We found that
32.5
Marhhhhl Molaaaon AW / M/
With pilot network pilot network
30,0
Trining PSNR (dB)
27.5
25.0
22.5
20.0
www.
2.5K
5k
7.5K
10K
12.5K
15K 17.5K
10K
Training stop
Figure 2: The VFR fails to converge the scene of ship from the dataset in 18] A small pilot network functioning as standard volume rendering by few training steps in the early training stage resolves this problem.

---

Bottleneck
Concal
Rendered
Spatial
feature encoding
Directional MLP
Color
feature
MLP
View direction
SH basis
SH feature
Figure 3: Architecture of the designed neural network for modeling view-dependent effects. Each spherical harmonics basis function is multiplied by a feature vector to encode the input view direction.
this feature integration works well for most scenes but fails to converge for some scenes. Such failure is more likely to occur when employing comparably large NN. The VFR normally converges using the NN with two hidden layers each of 256 neurons, but it does not converge when using four hidden layers. Fig. 2 Pillustrates failure case on the scene of ship from the NeRF synthetic dataset [18].
As increasing the size of the NN only has slight implication reconstruction and rendering speeds but can lead to a better rendering quality thanks to the VFR framework, we are motivated to use large NN. We hypothesize that the reason for the aforementioned convergence failure is that the model cannot find the correct geometry due to the feature integration of all samples in the early training stage. Motivated by this insight, design a pilot network that functions as standard volume rendering to aid the model in finding a coarse geometry in the early training stage. A small pilot network and few training steps are sufficient achieve this goal. In this work, we use the pilot network with two hidden layers, each with 64 neurons. The pilot network is only used in the first 300 training steps. After this training, the pilot network will be discarded, and we can use the VFR to train the model normally. The number of pilot training steps is small compared with the total training steps, e.g., 20K steps. Thus, a negligible effect on the overall training time, As shown Fig. the model converges normally with the aid of the pilot network.
3.4 View-dependent effect
We the spherical harmonics (SH) feature encoding method encode view direction for modeling view- viewddependent effects. This approach can seen as a variant of the rendering equation encoding in [12]. Different from the vanilla SH encoding [19] employed in the literature, multiply each encoded SH coefficient by a feature vector as shown in Fig- 3 Specifically, the rendered feature vector by the VFR is fed into a spatial MLP produce
feature vectors: SH feature vector and bottleneck feature vector. The SH feature vector will be split into small feature vectors f for SH coefficients, i.e., ( = 0). | f 1). etc. In the SH feature encoding block, the SH feature vector f will be multiplied with SH coefficient Y,m (d) with view direction form one SH feature encoding vector e/,, written as
e f Y (d)
(4)
A comphrehensive SH encoding derived by concatenating all encoding vectors as {cf. ej cf....) which will be further concatenated with the bottleneck feature vector (view di- rection independent). This concatenated vector is used to predict the final rendered color by directional MLP. The NN 33 thus includes the spatial and directional MLPs.
Implemention details
We implement the proposed VFR using the NerfAce library [6]. which is based on the deep learning framework PyTorch (22). The learnable features are organized by a multiresolution hash grid (MHG) 19]. This feature representation models the feature function F in (3) that accepts the input of the position and outputs feature vector. The color NN contains a spatial MLP and directional MLP. The spatial MI.P has two layers and the directional MLP has four layers, all with 256 hidden neurons. The size of the bottleneck feature set to 256. We use the GELU [44 instead the commonly used
ReLU I activation function, found the GELU results in a slightly better quality.
5

---

The density of a sample x; is derived from the queried feature vector F(x;) by tiny density mapping layer. On the real-world 360 dataset [3], the mapping layer linear transform from the feature vector to the density value. On the NeRF synthetic [8] dataset, we found a tiny network with one hidden layer and 64 neurons yields a slightly better rendering quality. The density mapping layers have negligible impact on the overall training time for both datasets.
We employ a hierarchical importance sampling strategy on both the NeRF synthetic [8] and real- world 360 datasets [3]. The occupancy grid is used for efficient sampling on the NeRF synthetic dataset. For the 360 dataset, we follow the most recent Zip-NeRF importance sampling. These two density fields are implemented by two small MHGs, where the number of hashing levels ten and the feature channel in each level is one. The final number of samplings on the radiance field is 32. The models are trained using the Adam optimizer with a learning rate of 0.01 and the default learning rate scheduler NerfAce [6]. We use PSNR, SSIM 31 and LPIPS (36] to evaluate the rendered image quality. The running times are measured on one RTX 3090 GPU. Table T summarizes other experimental settings in this work.
that uses two levels of
Table 1: Experimental settings on the NeRF synthetic and real-world 360 datasets. #Features represents the number of learnable features in the underlying hash grids, which is estimated by hash grid hyperparameters (HGH), i.e., the number of levelsx the number of voxels each level the number of feature channels.
Dataset
Model
#MHG
HGH
#Batch
SH degree
#SH feature
NeRF synthetic [18]
VFR-small
13M
16x219 x2
218 points
4
Real-world 360
VFR-base
34M
32x2
x2
2 rays
8
Real-world 360
VFR-
134M
32x221 x2 213 rays
7
8
5 Experimental results
5.1 NeRF synthetic dataset
The VFR achieves the best rendering quality but uses the minimum training time compared with state-of-the-are fast methods on the NeRF synthetic dataset. Since the original Instant-NGP [19] uses the alpha channel in the training images to perform background augmentation, we report the results of the Instant-NGP from NerfAcc [16). where such augmentation is not employed for fair comparison. As shown in Table 2 the VFR outperforms Instant-NGP by 2.27 dB and NerfAce by 1.51 dB, but only uses 3.3 minutes for training. Compared with TensoRF (6). our method only uses 33% of its training time while surpassing TensoRF by 1.48 dB in PSNR. In addition, the VFR achieves the best SSIM and LPIPS on this dataset, which is consistent with the PSNR metric.
This significant improvement rendering quality stems from the increased size of the NN. The MLP in iistant-NGP and NerfAcc has 4 layers and 64 neurons in each layer. By comparison, we use six-layer MI.P with 256 neurons. The computational cost of our MI.P is roughly 24 times higher than MLPs in Instant-NGP and NerfAcc. However, thanks to the VFR framework, our MLP only needs to run one time instead of many times as in the standard volume rendering framework employed in the
Table 2: Rendering quality and training time on the NeRF synthetic dataset. The VFR trained with 6K steps in 0.97 minutes achieves a similar quality compared with existing methods.
PSNRT
SSIM
LPIPSJ.
Training time..
NeRF 18
31.69
0.953
0.050
hours
Mip-NeRF
2
33.09
0.961
0.043
hours
TensoRF
33.14
0.963
0.047
10 mins
Instant-NGr
19
32.35
0.960
0.042
4.2 mins
NerfAce [6
33.11
0,961
0.053
4.3 mins
VFR-small: 6K
33.02
0.960
0.055
0.97 mins
VFR-small: 20K
34.62
0.971
0.038
3.3 mins

---

23.4 dB
28.0 dB
31.8 dB
33.6
Ground truth
NerfAce
Ours
NerfAcc error map
Ours map
Figure 4: Visual comparison of the synthesized novel views on the scenes of (from top to bottom) drums and hotdog the NeRF synthetic dataset.
compared methods. As can be observed from Table the single color NN evaluation property in the VFR leads to a minimum training time even when we use a much larger color NN. Noticeably, the VFR only needs 6K training steps to reach a similar rendering quality compared with the existing algorithms. Such 6K training steps can be completed within 1 minute on one RTX 3090 GPU, which is less than the 25% of training times of Instant-NGP and NerfAcc, and only 10% of the training time TensoRF.
Fig. 4 provides a visual comparison of the rendered novel views by NerfAce (based on Instant- NGPP and the VFR. On the compared scenes, both NerfAcc and the VFR can recover good geometries. However, for the scene of drums, obvious visual artefacts appear in the rendered views from NerfAce while our results do not have such visual artefacts. Besides, the VFR produces more realistic visual effects such shadow and reflection, as observed on the scenes of hotdog. The presented error maps and objective PSNRs in Fig. 4 also demonstrate the advantage of the VFR terms of rendering quality. While NerfAce and our method use the same feature representation, i.e., hash grids, we believe the quality advantage of our method stems from the large color NN enabled by the VFR.
5.2 Real-world 360 dataset
We also experiment seven real-world unbounded scenes from the 360 dataset [3] The scenes of flowers and treehill in this dataset are not included as they are not publicly available. We use the unbounded ray parameterization method proposed in Mip-NeRF 360 [3, where points mapped to a sphere if their radius to the scene center is larger than one.
The visual comparison in Fig. 5 shows that the VFR is able to render views in similar quality with Zip-NeRF but render more realistic views than NerfAce (based on Instant-NGP), The VFR provides more accurate geometry reconstruction than NerfAcc on the scene of bicycle. As can be observed from Fig.51 the rendering results NerfAcc contain obvious visual artefacts on the scenes of bonsai and kitchen, while the VFR's 's results are more realistic and closer to the ground truth. The reflective effect in the highlighted patches on the scene of kitchen is also well modeled by our model thanks to new SH feature encoding method.
As shown in Table 3 the VFR is able achieve similar rendering quality the state-of-the-ar Zip- NeRF 4 but using the minimum training time. Although Mip-NeRF 360 is able to synthesize novel high-fidelity views, requires days to train on high-end GPUs because is based on pure MLP representation. Using learned features organized in hash grids. Instant-NOR achieves a comparable rendering quality but reduces the training time from days to hours. The most recent Zip-NeRF uses multisampling for high-quality anti-aliasing view rendering at the expense of training time relative to Instant-NGP, The VFR achieves slightly better rendering quality than Zip-NeRF terms of the PSNR. However, we achieve this ofte-of-thh-ann art rendering quality with significantly less training
7

---

Table 3: Rendering quality and training time on the real-world 360 dataset. The scale factor for NerfAce and the VFR-base models remains constant four for both outdoor and indoor scenes. For the VFR-large model, the scale factor is adjusted to four for outdoor scenes (bicycle, garden, stump) and two for indoor scenes (bonsai, counter, kitchen, room) in order to maintain conformity with Zip-NeRF 4
PSNRT
SSIM
LPIPS
#eeatures.
Training time.
NeRF
18
24.85
0.659
0.426
N/A
days
Mip-NeRF 2
25.12
0.672
0,414
N/A
days
NeRF++ 35
26.21
0.729
0.348
N/A
days
Mip-NeRI 360
29.11
0.846
0.203
N/A
days
Instant-NGP 19
27.06
0.796
0.265
84M
0.81 hrs
Zip-NeRF 4
29.82
0.874
0.170
84M
5.20 hrs
NerfAce [6
28.69
0.834
0.221
34M
11 mins
VFR-base: 20K
29.48
0.830
0.233
34M
5.7 mins
VFR-base: 40K
29.92
0.850
0.208
34M
11 mins
VFR-large: 20K
29.51
0,846
0.252
134M
11 mins
VaR-large: 40K
29.90
0.862
0.231
134M
22 mins
Ground truth
NerfAce
Zip-NeRF
VFR
Figure 5: Visual comparison of synthesized novel views the scenes of (from top bottom) bicycle,
bonsai, and kitchen on the real-world 360 dataset 3
time. As a comparison, the VlR-large model only uses 22 minutes to train on one RTX 3090 GPU, while Zip-NeRF's training time is 5.2 hours on the same GPU.
should be noted that we achieve this high-quality view rendering using an entirely different approach than Zip-NeRF. As Zip-NeRF focuses on anti-aliasing, multisampling reasonable solution but leads to a higher computational cost. Besides, the authors of Zip-NeRF found a larger color NN helps in improving the rendering quality. Since Zip-NeRF still employs the standard volume rendering method, a larger color NN will undoubtedly increase the training time. The VFR, however, relieves the rendering framework from the high computational cost caused by many color network evaluations. As only one color network evaluation is required in the VFR, NN evaluation is no longer the computing bottleneck in the rendering framework, As result, the VFR enables one achieve a similar rendering quality as ZiR-NeRF but with significantly less training time.
5.3 Ablation study
Table shows the ablation study of the proposed methods. Compared with the commonly used ReLU, we found that GELU activation improves the rendering quality for both the standard volume rendering and the VFR. Larger MIP does increase the quality using the standard volume rendering but the expense of more training time. For the VFR, when increasing the size of the NN by using more hidden neurons, the rendering quality constantly improves but the training time only slightly
8

---

Table 4: Ablation study. SH means spherical harmonics for view direction encoding (VDE) while SHFE stands for SH feature encoding. We use the NerfAce's implementation of Instant-NGP as a baseline. MLP's size is represented by layersx Times are measured in minutes by 20K training steps.
NeRF synthetic (VFR-small)
Real-world 360
(VFR-base)
Activation
MLP
VDE
Time
PSNRT
SSIM LPIPS, Time, PSNRT SSIM LPIPS
Baseline
ReLU
4x64
SH
4.4
33.11
0.9614
0.053
5.6
28.39
0.8147
0.245
+GELU
GELU
4x64
SH
4.4
33,43
0,9634
0,049
5.6
28.43
0,8152
0.245
+Medium MLP
GELU
4x128
SH
5.6
33.92
0.9663
0.045
6.8
28.70
0.8205
0.239
VER
ReI.U
4x64
SH
2.9
33.17
0.9619
0.053
5.0
28.20
0.8013
0.262
+GELU
GELU
4x64
SH
3.0
33.61
0.9641
0.048
5.0
28.26
0,8064
0.254
+Medium MI.P
GELU
6x64
SHFE
3.0
34.05
0.9678
0.043
5.2
29.06
0.8215
0.240
+Large MLP
GELU
6x128
SHFE
3.0
34.33
0.9697
0.041
5.4
29.28
0,8266
0.236
W/O SHIF
GELU 6x256
SH
3.2
34,46
0.9701
0.040
5.5
28.91
0,8192
0.245
Full model
GELU 6x256
SHFE
3.3
34.62
0.9713
0.038
5.7
29.48
0.8301
0.233
increases. Replacing the SH encoding with the SH feature encoding (W/O SHFE vs Full model) also leads to better quality with negligible extra training time.
6 Limitations
Similar to many other neural rendering methods, the VFR requires per-scene optimization deliver high-fidelity view rendering. Although the VFR can be optimized in several minutes for one scene, we are still far from realizing real-time reconstruction for 3D video applications. As for the rendering time, our method can render 4-5 frames per second at the resolution of 800x800 on the NeRF synthetic dataset, similar to the rendering speed of NerfAce. On the real-world 360 dataset, the rendering speed is about 1~2 frames per second at the resolution of x300x840.
Elaborate implementation optimization can help in achieving high-quality real-time rendering using this VFR. However, compared with the recent work that specifically focuses on real-time rendering such as BakedSDF (33] and MobileNeRF (8. the rendering speed of the VFR needs be further improved. BakedSDF and MobileNeRF represent a scene's 's surface to enable one feature querying and one neural network evaluation, the expense of some rendering quality decline compared with volume representation. The VFR uses the volume representation with one color NN evaluation to deliver the sfate-of-the times for cast ray. Considering the fact that the queried feature vectors of non-empty samples are finally integrated into one feature vector using the VFR, we believe the number of feature querying can also be reduced to small value, e.g., from 32 to 5, to achieve high-quality and real-time rendering simultancously. An additional prospective constraint of the VFR lies in its potential inefficacy when confronted with semitransparent objects it integrates feature vectors first and then predicts single final color. This property is not well reflected in the NeRF synthetic and world 360 datasets and requires further investigation in future work.
rendering quality, but the feature querying still needs to be conducted many
7 Conclusion
We revisited the volume feature rendering method in the context of fast NeRF reconstruction. The VFR integrates the queried feature vectors of non-empty samples on a ray and then transforms the integrated feature vector the final rendered color using one single color neural network evaluation. We found the VFR cannot guarantee the convergence on some scenes and thus introduced the pilot network guide the early-stage training to tackle this problem. The VFR able to achieve the state-of-the-an rendering quality while using significantly less training time. It has great potential to replace the standard volume rendering in many neural rendering methods based on volume representation, with the benefits of either better rendering quality by employing a larger color neural network or a significantly reduced training time thanks to one single color neural network evaluation.

---

References
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803008375 2018.
[2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- BauainB and Pratul P Srinivasan. Mip-NeRF: multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[3] Jonathan Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip- NeRF 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470-5479, 2022.
[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-NeRF: Anti-aliased grid-based neural radiance fields. arXiv preprint wrXiv:2304006700 2023.
[5] Eric R Chan, Connor Z. Lin, Matthew Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et al. Efficient geometry- aware 3D generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16123-16133, 2022.
[6] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial radiance fields. In Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 333-350. Springer, 2022.
[7] Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: unified framework for neural fields and beyond. arXiv preprint arXiv:2302.01226. 2023.
[8] Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. MobileNeRF: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. arXiv preprint arXiv:2208.00277 2022.
[9] Sara Fridovich-Keil Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5501-5510, 2022.
[10] Qiancheng Fu, Qingshan Xu, Yew Soon Ong, and Wenbing Tao. Geo-NeuS: Geometry- consistent neural implicit surfaces learning for multi-viww reconstruction. Advances in Neural Information Processing Systems, 35:3403-3416, 2022.
[11] Jiatao Gu, Lingjie Liu, Peng Wang, and Christian Theobalt. StyleNeRF: A style-based 3D aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2021.
[12] Kang Han and Wei Xiang. Multiscale tensor decomposition and rendering equation encoding for view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4232-4241, 2023.
[13] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec. Baking neural radiance fields for real-time view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2021.
[14] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint wwXiv:1606.084 2016.
[15] James T Kajiya and Brian Von Herzen. Ray tracing volume densities. ACM SIGGRAPH, 18 (3):165-174, 1984.
[16] Ruilong Li, Matthew Tancik, and Angjoo Kanazawa. Nerface: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.0488 04847, 2022.
[17] Lingji Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 1..11611--16633 2020.
10

---

[18] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
[19] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics (ToG), 41()):
1-15, 2022.
[20] Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5589 5099,55992200 2021.
[21] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and Ira Kemelmacher- Shlizerman. StyleSDF: High-resolution 3d-consistent image and geometry generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13503-13513, 2022.
[22] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Proceedings of Thirty-first Conference on Neural Information Processing Systems Workshop, 2017.
[23] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4D: Efficient neural 4D decomposition for high-fidelity dynamic reconstruction and rendering, arXiv preprint arXiv:2211.11610 2022.
[24] Mohammed Suhail, Carlos Esteves, Leonid Sigal, and Ameesh Makadia. Light field neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8269-8279, 2022.
[25] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IÉEE/CVF Conference Computer Vision and Pattern Recognition, pages 5459- -4699, 2022.
[26] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keit Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
[27] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan Barron, and Henrik Kretzschmar. Block-NeRF: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pages 8248-822 8258, 2022.
[28] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan Barron, and Pratul Srinivasan. Ref-NeRF: Structured view- new-deepndent appearance for neural radiance fields. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481-5490. IEEE, 2022.
[29] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi view reconstruction. In Advances in Neural Information Processing Systems, 2021.
[30] Yunxiao Wang, Yanjie Li, Peidong Liu, Tao Dai, and Shu-Tao Xia. NeXT: Towards high quality neural radiance fields via multi-skip transformer. In Computer EisvonEECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 69-86. Springer, 2022.
[31] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro Simoncelli. Image quality assessment: from eиror visibility to structural similarity. IEEE transactions image processing, 13(4): 600-612, 2004.
[32] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
11

---

[33] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan Barron, and Ben Mildenhall. BakedSDF: Meshing neural SDFs for real-time view synthesis. arXiv preprin! arXiv: 14859, 2023.
[34] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa, Plenoctrees for real-time rendering of neural radiance fields. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5752-5761. 2021.
[35] Kai Zhang, Gemoot Riegler, Noah Snavely, and Vladlen Koltun. NeRF++: Analyzing and improving neural radiance fields. arXiv preprint 4rXi::2010.0749, 2020.
[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
12


Reviews:
[{"review_id": "1RHgGhZKt6", "reviewer": "Reviewer_4LZz", "summary": "In the paper, a novel method for view synthesis is proposed. More specifically, for given posed RGB input images, multi-resolution hash grid features are trained which are rendered to the image plane via volume rendering. The rendered feature is that passed through an MLP, processed with Spherical Harmonics (SH) feature encoding schemes, and then the final color is predicted. The main difference of the proposed system to previous NeRF models is that in 3D space, only feature grids without any MLP layers are optimized, and the view-dependent color prediction is only performed on the rendered features, i.e. \"deferred shading\" is applied. ", "questions": "- Are the time comparisons performed with respect to different hardware, e.g. RTX 3090 vs V100 GPUs?\n- Could the authors expand on the \"3D consistency\" / \"smoothness\" of in-between frames when rendering a fly-through, compared to non-feature-based methods such as ZipNeRF?\n- How exactly is the pilot network used? Is this predicting density and color per 3D sample point, or is this a smaller network but also operating on the rendered feature?\n- Could the authors explain the SH-based deferred rendering technique more precise?\n- Is the density predicted in 3D and stored in 3D hash grids, similar to the features?", "limitations": "- The authors discuss limitations of their work in the main manuscript. They mainly discuss the limitation of per-scene optimization and comparably slow / not yet real-time rendering times. I believe the limitation discussion is OK like it is; if some interesting failure mode could be illustrated with a figure, this could be interesting to add to the manuscript. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Results: I believe the main strengths of the paper is that the proposed system leads to good results, i.e. the proposed system outperforms SOTA methods such as Zip-NeRF or Instant-NGP on the two dataset types in both, the reported training times as well as the reported view synthesis quality.\n\n- Ablation Study: A rather extensive ablation study is performed which I believe makes the manuscript stronger. It highlights the importance of the various components, and helps the reader to understand where the speed / quality improvements are coming from.\n\n- Limitation Section: The manuscript contains a rather extensive limitation section which is beneficial as it portrays a more complete impression of the method. \n\n- Manuscript organization: The manuscript is organized coherently and the information flow is rather easy to understand. ", "weaknesses": "- Time comparison (L. 216, L. 229): The time comparison is, if I understood this correctly, not reported with running the methods on the same hardware, e.g. different GPUs (RTX 3090 vs V100) are used, and also times are simply adopted from other papers. As the time complexity is the main selling point of this work, I believe this should be reported more accurately by running the methods on the exact same hardware / setup. \n\n- Qualitative Results - Animations: While the reported method achieves high-quality view synthesis performance measured by per-image metrics, the \"smoothness\" / \"3d consistency\" between frames is not shown. As the proposed system uses features instead of directly color predictions, the consistency between frames might be lower than for previous methods while not having a significant influence on the per-image metrics. It would there be very helpful to also show animations, e.g. in the form of fly-through videos, to compare methods with this focus.\n\n- Unclear how the “pilot network” is used: Is simply the MLP size changed, or is this operating as the “default NeRF” models, i.e. per-point MLP calls are performed, and then color/density is aggregated via volume rendering? It would be helpful if the description of the pilot network would be made more precise. \n\n- Figure 3: Unclear. What exactly is happening in the individual blocks? Is a per SH feature predicted, and then they are summed together? What is the “bottleneck arrow” indicating? Why is this skip connection actually necessary? The exact math formulations is also not contained in the respective Section 3.4. It would be helpful if the description of the deferred rendering part is more precise and clear formulas are used so that the reader can better understand the exact technique.\n\n- Density prediction: How is the density obtained? Is this stored, similarly to the features, in a 3D hash grid?\n\n- Qualitative Comparison - Figure 5: I think in Figure 5, also results from Zip-NeRF should be shown.\n\nTypos / Unclear passages: \n\n- L. 5: either in occupancy grids -> either in the form of occupancy grids\n- L. 8: after neural network evaluation -> after neural network evaluations\n- L. 19: View synthesis is a task that synthesizes -> For the task of view synthesis, unobserved views are synthesized …\n- L. 36: near surface -> near the surface\n- L. 39: standard volume rendering technique -> standard volume rendering techniques\n- L. 58: NN runnings -> NN calls\n- L. 61: to enable one sample on the surface -> to enable one sample-based rendering\n- L. 69: modeling the specular effect -> modeling specular effects\n- L. 96: represent F by use of MLPs. -> represent F by using MLPs\n- L. 97: as N times MLP evaluations are required -> as N MLP evaluations are required\n- L. 97/98: Alternatively, recent research shows that -> please add respective citation\n- L. 100: organized in the forms of -> organized in the form of\n- L. 110: the many times NN running is … -> the many NN calls are …\n- L. 111: the importance of a large NN in realistic rendering -> the importance of a large NNs for realistic rendering\n- L. 112 - 114: “Considering the fact that valuable samples on the surface are close in terms of spatial position, the queried feature vectors are also very similar as they are interpolated according to their position” -> unclear meaning\n- L. 213: reports on the rendering -> reports the rendering\n- L. 223: Using learning features organized in hash grids -> Using learned features organized in hash grids\n- L. 249: Larger MLP (B, C) -> B does not indicate a “larger MLP” if I understood these symbols correctly. I would also suggest to structure the names and ablation study slightly different, e.g. with the typical “Baseline”, “+ GeLU”, “+larger MLP”, etc naming convention; currently, it is very cryptic and hard to understand.\n\nMissing Citations:\n- PlenOctrees For Real-time Rendering of Neural Radiance Fields, Yu et al., ICCV 2021\n- Plenoxels Radiance Fields without Neural Networks, Yu et al., CVPR 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments on our manuscript. We address your concerns as follows.\n\n* __Time comparison:__ In light of your suggestion, we report on the training times of various comparison methods measured on the same GPU RTX 3090. As shown in Tables 2 and 3, we can draw the same conclusion as in the original manuscript. Our method achieves a better rendering quality while taking less training time compared with the comparison methods. On the NeRF synthetic dataset, our VFR outperforms the SOTA NerfAcc by 1.51 dB in PSNR but using less training time, or achieves a similar PSNR as NerfAcc but 4+ times faster. On the real-world 360 dataset, our method offers a better rendering quality (29.48 dB) than NerfAcc (28.69 dB) while using 52 % of NerfAcc’s training time. Compared with the concurrent Zip-NeRF that achieves the SOTA PSNR results using 5.2 hours, our method deliveries a slightly better PSNR using only 11 minutes, which is 28 times faster than Zip-NeRF.\n\nTable 2. PSNR and training time on the NeRF synthetic dataset on one RTX 3090.\n|       | PSNR | Time\n| ------ | ------ | --- |\n| NeRF          | 31.69   | hours \n| Mip-NeRF   | 33.09   | hours \n| TensoRF      |  33.14  | 10 mins\n| Instant-NGP | 32.35  | 4.2 mins\n| NerfAcc       |  33.11  | 4.2 mins\n| Our VFR: 6K | 33.02 | 0.97 mins\n| Our VFR: 20K | __34.62__ | __3.3 mins__\n\nTable 3. PSNR and training time on the real-world 360 dataset on one RTX 3090.\n|       | PSNR | #Feature | Time\n| ------ | ------ | --- | --- |\n| NeRF          | 24.85   | N/A | days \n| Mip-NeRF   | 25.12   | N/A  |days \n| NeRF++      |  26.21  | N/A |days\n| Mip-NeRF360 | 29.11  | N/A |days\n| Instant-NGP       |  27.06  | 84M |0.81 hrs\n| Zip-NeRF | 29.82 | 84M | 5.2 hrs\n|NerfAcc | 28.69 | 34M | 11 mins\n|Our VFR: 20K| 29.48 | __34M__ | __5.7 mins__\n|Our VFR: 40K| __29.92__ | __34M__ | __11 mins__\n\n* __Qualitative Results - Animations:__ We do not observe the consistency difference between our rendered videos and those from the standard volume rendering. Both our VFR and the standard volume rendering method are based on global representations (representing a scene using a MLP and a multiresolution hash grid), which are recognized as the source of consistency between frames. Therefore, the rendered frames using our VFR are as smooth as those from the standard volume rendering. We will upload the rendered videos to demonstrate the frame consistency in the final version.\n\n* __Unclear how the “pilot network” is used:__ The pilot network is a small MLP operating as the “default NeRF” models, i.e., per-point MLP calls are performed, and then color/density is aggregated via volume rendering.\n\n* __Figure 3: Unclear:__ Thanks for pointing this out. For an integrated feature vector, the spatial MLP yields two feature vectors: SH feature vector and bottleneck feature vector. The SH feature vector will be spitted into small feature vectors for each SH coefficient, i.e.,  {$\\mathbf{f}_0^0 | l=0$}, {$\\mathbf{f}_1^{-1},  \\mathbf{f}_1^{0}, \\mathbf{f}_1^{1} | l=1$}, etc. In the SH feature encoding block, the SH feature vector $\\mathbf{f}_m^l$ will be multiplied with SH coefficient (basis function) $Y_l^m(\\mathbf{d})$ with view direction $\\mathbf{d}$ to form one SH feature encoding vector $\\mathbf{e}_m^l$, written as $\\mathbf{e}_m^l = \\mathbf{f}_m^l Y_l^m(\\mathbf{d})$. A comprehensive SH encoding is derived by concatenating all encoding vectors as $\\mathbf{E}$ = {$\\mathbf{e}_0^0, \\mathbf{e}_1^{-1}, \\mathbf{e}_1^{0} ...$}, which will be further concatenated with the bottleneck feature vector (view direction independent). This concatenated vector is used to predict the final rendered color by the directional MLP. Based on the above description, we summarize answers to your questions as follows:\n\n    * For each SH coefficient $Y_l^m(\\mathbf{d})$, the spatial MLP will predict a small SH feature vector $\\mathbf{f}_m^l$ for it. The SH feature encodings $\\mathbf{e}_m^l$ will be concatenated to form a comprehensive encoding. \n    * The bottleneck arrow indicates a bottleneck feature vector predicted by the spatial MLP.\n    * The skip connection is necessary because we want the bottleneck feature vector to be independent to view direction for the prediction of diffuse color.\n\n    We will add these details in the final version as suggested by the reviewer.\n\n* __Density prediction:__ The density is obtained by applying a linear layer to the queried feature vector from the underlying multi-resolution hash encoding. As this linear layer only has one output neuron, it has a negligible effect on the overall training and rendering times.\n\n* __Qualitative Comparison - Figure 5:__ The results from Zip-NeRF are similar to ours in visual. We will add the results from Zip-NeRF in Figure 5 as per your suggestion.\n\n* __Typos / Unclear passages:__ Thanks for pointing out these typos / unclear passages. We will implement these corrections in our final paper. For line 249, we apologize for the confusion caused by this small mistake. The correct description should be “Larger MLP (C) does increase … ”. We will revise the description of ablation study by following your suggestion in structuring the names and ablation study using the typical “Baseline”, “+ GeLU”, “+larger MLP”.\n\n* __Missing Citations:__ We will include these citations in our final version of the paper.\n\n* __Limitations--interesting failure mode:__ We have not found other failure modes related to our method in addition to the convergence issue (solved by our proposed pilot network). One potential limitation of our method is that it may not work well for semitransparent objects as our method integrates feature vectors first and then predicts a single final color. This property is not well reflected in the NeRF synthetic and real-world 360 datasets and requires further investigation in our future work. We will add this discussion to our final version."}, {"review_id": "31GWa5oJVi", "reviewer": "Reviewer_WTsx", "summary": "The authors propose to perform feature accumulation first at each pixel location, and then pass the accumulated feature to a MLP to get the rendered color in NeRF. They show improved rendering quality over baseline methods on NeRF synthetic dataset and object-centric 360-degree captures. ", "questions": "See bullet point 1, 2, 3 in the weakness section. ", "limitations": "The limitations seem to be discussed in detail in the paper. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Idea seems simple and easy to reproduce. \n2. Writing is good and paper is easy to follow overall.\n3. Evaluation seems extensive, though some important aspects might be missing (see bullet point 1, 3 in the weakness section). ", "weaknesses": "1. Will jittering happen when rendering videos using the proposed method? I feel the proposed method might be more likely to suffer from jittering issues than standard volume rendering, as the large MLP network might introduce view-consistencies. It would be great to include some video results in the demo. \n\n2. Training time improvement doesn't seem to be that big, compared with NeRFACC that the proposed method is built upon. Is it due that volume-rendering of a relatively high-dimensional features is slower than volume-rendering of RGB colors, hence cutting down the performance gain from reducing MLP evaluations?\n\n3. Table 2 and Table 3 only contains PSNR comparisons against baselines. I don't think this is indicative enough of image sharpness; I would like to see SSIM and LPIPS comparisons to make a better judgement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments on our manuscript. We address your concerns as follows.\n\n1. __Jittering problem:__ We understand jittering is the problem of 3D inconsistency when rendering video with changing viewpoints. We do not observe the consistency difference between our rendered videos and those obtained by using standard volume rendering methods. As MLPs are conditioning on the queried feature vectors for both ours and standard volume rendering methods, a larger MLP does not introduce extra 3D inconsistency. In addition, considering that pure large MLP-based representations (e.g., the original NeRF and mip-NeRF) are able to render videos with high 3D consistency, we believe large MLPs might not be a source of jittering issue.  We will add rendered videos in the final version to demonstrate 3D consistency.\n\n2. __Training time improvement:__ Volume rendering of relatively high-dimensional features (e.g., 64 channels) instead of RGB colors has a negligible effect on the overall training time. Two main computations in the rendering process are feature querying and MLP evaluation, so the overall training time is not only dependent on the number of MLP evaluations. As shown in Table 2, to reach a similar quality, our method is 4+ times faster than the NerfAcc based on the standard volume rendering on the NeRF synthetic dataset. On the real-world 360 dataset, our method achieves a 0.79 dB improvement in PSNR while only taking 52% of its training time, compared with NerfAcc. As NerfAcc based on Instant-NGP is the most state-of-the-art method in fast training, we believe our training time improvement is significant and our method is valuable to the community.\n\n3. __SSIM and LPIPS comparisons:__ Thanks for your suggestion. We add SSIM and LPIPS results in Tables 2 and 3 as follows. On the NeRF synthetic dataset in Table 2, our method outperforms the compared methods in all metrics including PSNR, SSIM and LPIPS while using the minimal training time. In Table 3, it is observed that our method achieves similar rendering quality (slightly better PSNR but slightly worse SSIM and LPIPS) as the concurrent Zip-NeRF on the real-world 360 dataset. However, our method requires significantly less parameters and less training time (11 minutes vs 5.2 hours) compared with Zip-NeRF. In addition, our method only uses 52% of NerfAcc’s training time to reach better PSNR and similar SSIM and LPIPS compared with NerfAcc.\n\nTable 2. Rendering quality and training time on the NeRF synthetic dataset on one RTX 3090.\n|       | PSNR | SSIM | LPIPS | Time\n| --- | --- | --- | --- | --- |\n| NeRF          | 31.69   | 0.953 | 0.050 | hours \n| Mip-NeRF   | 33.09   | 0.961 | 0.043 | hours \n| TensoRF      |  33.14  | 0.963 | 0.047 | 10 mins\n| Instant-NGP | 32.35  | 0.960 | 0.042 | 4.2 mins\n| NerfAcc       |  33.11  | 0.961 | 0.053 | 4.2 mins\n| Our VFR: 6K | 33.02 | 0.960 | 0.055 | 0.97 mins\n| Our VFR: 20K | __34.62__ |  __0.971__ | __0.038__ | __3.3 mins__\n\n\nTable 3. Rendering quality and training time on the real-world 360 dataset on one RTX 3090.\n|       | PSNR | SSIM | LPIPS | #Feature | Time\n| --- | --- | --- | --- | --- | --- |\n| NeRF          | 24.85   | 0.659 | 0.426 | N/A | days \n| Mip-NeRF   | 25.12   | 0.672 | 0.414 | N/A  |days \n| NeRF++      |  26.21  | 0.729 | 0.348 | N/A |days\n| Mip-NeRF 360 | 29.11  | 0.846 | 0.203 | N/A |days\n| Instant-NGP       |  27.06  | 0.796 | 0.265 | 84M |0.81 hrs\n| Zip-NeRF | 29.82 | __0.874__ | __0.170__ | 84M | 5.2 hrs\n|NerfAcc | 28.69 | 0.834 | 0.221 | 34M |11 mins\n|Our VFR: 20K| 29.48 | 0.830 | 0.233 | __34M__  | __5.7 mins__\n|Our VFR: 40K| __29.92__ | 0.850 | 0.208 | __34M__ | __11 mins__\n"}, {"review_id": "Vpjnxpyfnm", "reviewer": "Reviewer_y3xd", "summary": "The paper proposes a method for fast NeRF reconstruction. The main contribution of the paper \nis that instead of integrating radiance along the camera ray which requires evaluation of the \ncolor MLP at each point, the paper proposes to integrate features along the ray and only \nevaluate a single MLP on the integrated features to get the final pixel color. In this way,\nthe proposed method can reduce the number of MLP evaluations, thus allowing the method to \nuse achieve faster reconstruction even with a larger MLP. The larger MLP also allows the method \nto achieve higher-quality reconstructions. The paper does experiments on NeRF synthetic dataset\nand real-world 360 dataset and demonstrates higher accuracy and fast reconstruction speed \nthan baseline methods such as instant-nap, nerfacc, and tensorf. \n\n", "questions": "1. It's not clear to me how the method integrates the features along the ray. In Equation 3, \nthe paper says it performs a weighted combination of the features. I would assume that the \nweight is calculated in a similar way to the standard volume rendering based on the density \nfield. Then it's not clear to me how the density field is predicted. The paper is based on \nthe NGP representation, and it in fact uses a MLP to predict the density of each point. Is \nthe paper doing in the same way? If that's the case, I think the paper should make it clear.\nAlso in this case, the paper should not claim that it is performing a single evaluation NN\nfor each pixel (or make it more clear, it's a single evaluation of color MLP). \nSimilarly, in the case of real-world scenes, the paper uses proposal networks for sampling,\nwhich also requires additional MLP evaluations. Overall, I think the paper should make it \nmore adequate when it says it's performing a single neural network evaluation.\n\n2. In Line 70, the paper says SNeRG still requires many times of MLP evaluations to get \nthe diffuse color. This is not true. SNeRG stores the diffuse colors at each voxel, and \ndirectly applies alpha compositing to get the diffuse color for the ray. Therefore, no \nMLP evaluation is needed.\n\n3. The motivation behind the pilot network is kind of ad-hoc to me. First, the paper \nsays VFR works well for most scenes but fails to converge for some scenes. If it's a scene-specific\nproblem, It's good to know the characteristics of the scenes where it will fail.  In addition,\nthe paper says the reason for VFR not converging is that it does not have a coarse geometry to focus on samples on \nthe surface. The first question is that why VFR needs to focus on samples on the surface. Does it\nmean that VFR will not work well on objects that don't have an opaque surface (like furry objects)? \nWhy standard volume rendering does not this problem? Moreover, the paper also says larger networks\ntend to have more convergence issues than smaller networks, how does this relate to the lack of \ncoarse geometry? I think the paper should have a better explanation on the convergence issues \nwith VFR and make a more convincing argument behind the motivation of the pilot network.\n\n4. In Figure 4, the results on the Ficus scene of Nerfacc is suspicious to me. What's the reason\nfor the weird specularity on the pot?\n\n5. I do have concerns over the technical contributions of the paper. As the paper said in Line 68,\nSNeRG has been using feature integrations for faster NeRF rendering. In addition, previous works such as StyleNeRF, EG3D, and StyleSDF have also been using feature integration for NeRF. While\nthey don't do a systematic evaluation on the per-scene reconstruction task, I think generally \nfeature integration has been a standard technique in the NeRF community, and has been explored \nin similar tasks as mentioned above.\n\nTo summarize, while I appreciate the thorough experiments and evaluations of the paper, my major\nconcern over the paper is that the proposed method has been explored in similar tasks in \nprevious works and there is a lack of technical contributions from the paper.\n", "limitations": "The limitations look good to me.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. The paper shows that integrating the features along the ray and decoding them to the \npixel color with a large MLP can achieve faster and higher-quality reconstructions.\n\n2. The paper proposes to use a small pilot network at the beginning of the optimization \nthat applies standard volume rendering, which helps stabilize the training of the proposed method. \n\n3. The paper did a thorough evaluation of the proposed method against different baseline methods \non both synthetic and real-world datasets. \n", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments for our manuscript. We respond to your concerns one-by-one as follows.\n\n1. __How the method integrates the features along the ray:__ We apologize for the missed details of how to obtain the integration weights. The weights are calculated from densities of sampled points, where density of a point is predicted by a linear layer from the queried feature vector before integration. As this linear layer only has one output neuron to predict the density value, it has a negligible effect on the overall training time. On the real-world 360 scenes, the proposal networks do require additional MLP evaluations for density prediction as mentioned by the reviewer. However, these MLPs for the density field are usually very small and MLPs are even not necessary like that in TensoRF. Thus, MLP evaluations for the density field have a very small effect on the overall training time. The reduction in the number of MLP evaluations for color prediction is more valuable as the color MLP evaluation is typically one of the bottlenecks for the overall training time. Nevertheless, we appreciate the reviewer’s suggestion in making the argument clear and will adopt the suggestion by revising the argument on a single evaluation of color MLP in our final version.\n\n2. __The SNeRG still requires many times of MLP evaluations:__ This statement is true because the SNeRG needs to obtain the diffuse color through many MLP evaluations first and then storing the obtained diffuse color for fast rendering. In inverse rendering, the diffuse color of each voxel is unknown and needs to be estimated in NeRF. Thus, the training of the SNeRG is identical to the original NeRF using the standard volume rendering. We agree with the reviewer that no MLP evaluation is needed for SNeRG in the rendering stage. But the SNeRG needs to train a standard NeRF first and then bakes the trained NeRF to a sparse grid. This training and baking in the SNeRG need many MLP evaluations. In our work, we show that the many times of MLP evaluations in the training stage can be avoided by our VFR, leading to a significant reduction in training time for our method (several minutes) and a better rendering quality than the SNeRG (days to train).\n\n3. __The motivation behind the pilot network:__\n    * We found that our VFR without the pilot network is more likely to fail when the scene is of complex geometry.\n    * Our VFR needs to focus on samples on the surface because the feature vectors of these samples will be integrated into one feature vector to predict the color of that surface point. If the feature vectors from the foreground and background are fused together (could happen when there is no coarse geometry in the early training stage), the following MLP may have difficulties in predicting the correct color. For objects that do not have an opaque surface, our VFR may not work well, and this potential limitation needs to be further investigated in our future work. The standard volume rendering does not have the convergence issue because the color prediction of each sample by the MLP is independent.\n    * We believe larger networks tend to have more convergence issues in our model because larger networks have stronger learning ability, and they try to directly predict the correct color from the integrated feature vector of all samples on a ray and completely ignore the scene geometry. As the training of NeRF models is to find the correct geometry and the color field simultaneously, the above ignorance of geometry leads to failed convergence.\n    * We will add the above explanations to our final version.\n\n4. __Weird specularity on the pot in Fig 4:__ We believe that specularity on the pot is caused by insufficient modeling of the view-dependent effect using a comparably small MLP in the NerfAcc (based on Instand-NGP).\n\n5. __Technical contributions of the paper:__ As aforementioned in our response to your question on SNeRG, the SNeRG aims at faster NeRF rendering without considering the training time. The SNeRG still needs to evaluate the MLP for many times in the training stage, while our VFR can directly integrate queried feature vectors to enable one large MLP evaluation in the training stage, leading to a significant faster training speed as well as a better rendering quality than the SNeRG. We agree with the reviewer that some works also employ a similar idea from different perspectives. But none of existing works investigate the direct integration of queried feature vectors as ours to accelerate the training of NeRF-like models and to improve the rendering quality for the per-scene reconstruction task. The mentioned works (i.e., StyleNeRF, EG3D, and StyleSDF) are in the field of 3D generation instead of neural inverse rendering from multiple views. Although these works employ feature integration, they all require one MLP (large or small in size) evaluation first for each sampled point and then integrate the yielded feature vectors from the MLP evaluations.\n\n    We respectfully disagree with the reviewer that feature integration has been a standard technique in the NeRF community. We believe that it is not a common view among the NeRF community that queried feature vectors from feature representations like multi-resolution hash encoding can be directly integrated to enable one single-time MLP evaluation. This point is evidenced by the fact that many of the latest representative NeRF works are still based on the standard volume rendering (such as Instant-NGP, TensoRF, and the concurrent Zip-NeRF), although feature integration offers significant benefits as identified by our work. This paper formally introduces the volume feature rendering for per-scene reconstruction task, identifies the potential problem and proposes the solution, conducts extensive experiments and provides a detailed analysis. As such, we believe the contributions of this work are solid and the proposed methods are valuable to the NeRF community."}, {"review_id": "lNKiASgIjA", "reviewer": "Reviewer_q1tF", "summary": "This work improves grid-based NeRF on both quality and training speed. To predict view-dependent color, they proposes to condition MLP on the volume rendered voxel features instead of each original point features. As a result, the MLP only run once for each pixel instead of the original dozen of MLP evaluations. To improve robustness, they find using the old pipeline (called pilot network) to warmup the scene for few hundreds iteration is enough. A new spherical-harmonic-based view-direction feature encoding is proposed, which combined with larger MLP achieve state-of-the-art quality. Experiments results show the superior quality and training speed from previous arts.", "questions": "1. **Eq.4.** Is the final view-direction feature summed toghether ($\\sum_{l,m} e_l^m$) like what we conventionally do for rgb or concatenated ($\\mathrm{concant}(\\{e_l^m | \\forall l,m\\})$)?\n\n2. **Table 3.** It's better to put a table footnote that some baselines' training times are rescaled by 8 (the number of GPUs they used). As discussed in L216-219, this is not rigorous so I think it's better to inform readers who only skims the table.\n\n3. **Table 4's H row.** It seems that the 56 should have be 256.\n", "limitations": "I think the limitation is well discussed.", "rating": 9, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The improvement is really solid and promising. The proposed method is not hard to adapt to other method. The paper is easy to follow as well. I believe this work can benefit many future researches.", "weaknesses": "It's a pity that videos are not provided to showcase the improvement on view-dependent effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your encouraging feedback. We respond to your concerns one-by-one as follows.\n\n**Weaknesses:**\n\n__Videos are not provided:__ Thanks for your suggestion. We will upload the rendered videos to showcase the improvement on view-dependent effect in our final version of this paper.\n\n**Questions:**\n1. __Eq. 4:__ The encoded view direction feature vectors are concatenated to form one comprehensive view direction encoding. We will provide more detailed description for SH feature encoding to highlight this feature vector concatenation in our final version.\n\n2. __Table 3:__ Thanks for your valuable suggestion. We provide the running time comparison of various methods measured on the same RTX 3090 for fair and rigorous comparison. As shown in the updated Tables 1 and 2, we can draw the same conclusion as in our first submission.\n\n3. __Table 4’s H row:__ Thanks for pointing this out. We will correct this error in our final version.\n\nTable 2. PSNR and training time on the NeRF synthetic dataset on one RTX 3090.\n|       | PSNR | Time\n| ------ | ------ | --- |\n| NeRF          | 31.69   | hours \n| Mip-NeRF   | 33.09   | hours \n| TensoRF      |  33.14  | 10 mins\n| Instant-NGP | 32.35  | 4.2 mins\n| NerfAcc       |  33.11  | 4.2 mins\n| Our VFR: 6K | 33.02 | 0.97 mins\n| Our VFR: 20K | __34.62__ | __3.3 mins__\n\n\nTable 3. PSNR and training time on the real-world 360 dataset on one RTX 3090.\n|       | PSNR | #Feature | Time\n| ------ | ------ | --- | --- |\n| NeRF          | 24.85   | N/A | days \n| Mip-NeRF   | 25.12   | N/A  |days \n| NeRF++      |  26.21  | N/A |days\n| Mip-NeRF360 | 29.11  | N/A |days\n| Instant-NGP       |  27.06  | 84M |0.81 hrs\n| Zip-NeRF | 29.82 | 84M | 5.2 hrs\n|NerfAcc | 28.69 | 34M | 11 mins\n|Our VFR: 20K| 29.48 | __34M__ | __5.7 mins__\n|Our VFR: 40K| __29.92__ | __34M__ | __11 mins__\n\n\n"}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their precious time and thoughtful review of this manuscript. The comments and suggestions raised are extremely valuable and constructive, and very helpful for improving the quality of the manuscript. 

Please see the detailed response to each reviewer in the separate response.

Meta-review: As discussed in the rebuttal and post-rebuttal discussion, feature integration is not an entirely new idea but has been explored to some extent in generative 3D methods, such as StyleNeRF and EG3D. The authors should edit their discussion regarding novelty and positioning to prior work. Many other suggestions were made by the reviewers to improve the manuscript - please incorporate these in your camera-ready paper.
Decision: Accept (poster)