SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As shown in Table 3, the VFR is able to achieve similar rendering quality to the state-of-the-art Zip-NeRF [4] but using the minimum training time. Although Mip-NeRF 360 is able to synthesize novel high-fidelity views, it requires days to train on high-end GPUs because it is based on a pure MLP representation. Using learned features organized in hash grids, Instant-NGP achieves a comparable rendering quality but reduces the training time from days to hours. The most recent Zip-NeRF uses multisampling for high-quality anti-aliasing view rendering at the expense of training time relative to Instant-NGP. The VFR achieves a slightly better rendering quality than Zip-NeRF in terms of the PSNR. However, we achieve this state-of-the-art rendering quality with a significantly less training
Table 3: Rendering quality and training time on the real-world 360 dataset. The scale factor for NerfAcc and the VFR-base models remains constant at four for both outdoor and indoor scenes. For the VFR-large model, the scale factor is adjusted to four for outdoor scenes (bicycle, garden, stump) and two for indoor scenes (bonsai, counter, kitchen, room) in order to maintain conformity with Zip-NeRF [4].
PSNR↑	SSIM↑	LPIPS↓	#Features↓	Training time↓
NeRF [18]	24.85	0.659	0.426	N/A	days
Mip-NeRF [2]	25.12	0.672	0.414	N/A	days
NeRF++ [35]	26.21	0.729	0.348	N/A	days
Mip-NeRF 360 [3]	29.11	0.846	0.203	N/A	days
Instant-NGP [19]	27.06	0.796	0.265	84M	0.81 hrs
Zip-NeRF [4]	29.82	0.874	0.170	84M	5.20 hrs
NerfAcc [16]	28.69	0.834	0.221	34M	11 mins
VFR-base: 20K	29.48	0.830	0.233	34M	5.7 mins
VFR-base: 40K	29.92	0.850	0.208	34M	11 mins
VFR-large: 20K	29.51	0.846	0.252	134M	11 mins
VFR-large: 40K	29.90	0.862	0.231	134M	22 mins
Figure 5: Visual comparison of synthesized novel views on the scenes of (from top to bottom) bicycle, bonsai, and kitchen on the real-world 360 dataset [3].
time. As a comparison, the VFR-large model only uses 22 minutes to train on one RTX 3090 GPU, while Zip-NeRF's training time is 5.2 hours on the same GPU.
It should be noted that we achieve this high-quality view rendering using an entirely different approach than Zip-NeRF. As Zip-NeRF focuses on anti-aliasing, multisampling is a reasonable solution but leads to a higher computational cost. Besides, the authors of Zip-NeRF found a larger color NN helps in improving the rendering quality. Since Zip-NeRF still employs the standard volume rendering method, a larger color NN will undoubtedly increase the training time. The VFR, however, relieves the rendering framework from the high computational cost caused by many color network evaluations. As only one color network evaluation is required in the VFR, NN evaluation is no longer the computing bottleneck in the rendering framework. As a result, the VFR enables one to achieve a similar rendering quality as Zip-NeRF but with a significantly less training time.
Table 4 shows the ablation study of the proposed methods. Compared with the commonly used ReLU, we found that GELU activation improves the rendering quality for both the standard volume rendering and the VFR. Larger MLP does increase the quality using the standard volume rendering but at the expense of more training time. For the VFR, when increasing the size of the NN by using more hidden neurons, the rendering quality constantly improves but the training time only slightly
Table 4: Ablation study. SH means spherical harmonics for view direction encoding (VDE) while SHFE stands for SH feature encoding. We use the NerfAcc's implementation of Instant-NGP as a baseline. MLP's size is represented by layers  $\times$  neurons. Times are measured in minutes by 20K training steps.
Activation	MLP	VDE	NeRF synthetic (VFR-small)	Real-world 360 (VFR-base)
Time↓	PSNR↑	SSIM↑	LPIPS↓	Time↓	PSNR↑	SSIM↑	LPIPS↓
Baseline	ReLU	4×64	SH	4.4	33.11	0.9614	0.053	5.6	28.39	0.8147	0.245
+GELU	GELU	4×64	SH	4.4	33.43	0.9634	0.049	5.6	28.43	0.8152	0.245
+Medium MLP	GELU	4×128	SH	5.6	33.92	0.9663	0.045	6.8	28.70	0.8205	0.239
VFR	ReLU	4×64	SH	2.9	33.17	0.9619	0.053	5.0	28.20	0.8013	0.262
+GELU	GELU	4×64	SH	3.0	33.61	0.9641	0.048	5.0	28.26	0.8064	0.254
+Medium MLP	GELU	6×64	SHFE	3.0	34.05	0.9678	0.043	5.2	29.06	0.8215	0.240
+Large MLP	GELU	6×128	SHFE	3.0	34.33	0.9697	0.041	5.4	29.28	0.8266	0.236
W/O SHFE	GELU	6×256	SH	3.2	34.46	0.9701	0.040	5.5	28.91	0.8192	0.245
Full model	GELU	6×256	SHFE	3.3	34.62	0.9713	0.038	5.7	29.48	0.8301	0.233
increases. Replacing the SH encoding with the SH feature encoding (W/O SHFE vs Full model) also leads to a better quality with negligible extra training time.
Similar to many other neural rendering methods, the VFR requires per-scene optimization to deliver high-fidelity view rendering. Although the VFR can be optimized in several minutes for one scene, we are still far from realizing real-time reconstruction for 3D video applications. As for the rendering time, our method can render  $4\sim 5$  frames per second at the resolution of  $800\times 800$  on the NeRF synthetic dataset, similar to the rendering speed of NerfAcc. On the real-world 360 dataset, the rendering speed is about  $1\sim 2$  frames per second at the resolution of  $1300\times 840$ .
Elaborate implementation optimization can help in achieving high-quality real-time rendering using this VFR. However, compared with the recent work that specifically focuses on real-time rendering such as BakedSDF [33] and MobileNeRF [8], the rendering speed of the VFR needs to be further improved. BakedSDF and MobileNeRF represent a scene's surface to enable one feature querying and one neural network evaluation, at the expense of some rendering quality decline compared with volume representation. The VFR uses the volume representation with one color NN evaluation to deliver the state-of-the-art rendering quality, but the feature querying still needs to be conducted many times for a cast ray. Considering the fact that the queried feature vectors of non-empty samples are finally integrated into one feature vector using the VFR, we believe the number of feature querying can also be reduced to a small value, e.g., from 32 to 5, to achieve high-quality and real-time rendering simultaneously. An additional prospective constraint of the VFR lies in its potential inefficacy when confronted with semitransparent objects as it integrates feature vectors first and then predicts a single final color. This property is not well reflected in the NeRF synthetic and real-world 360 datasets and requires further investigation in future work.
We revisited the volume feature rendering method in the context of fast NeRF reconstruction. The VFR integrates the queried feature vectors of non-empty samples on a ray and then transforms the integrated feature vector to the final rendered color using one single color neural network evaluation. We found the VFR cannot guarantee the convergence on some scenes and thus introduced the pilot network to guide the early-stage training to tackle this problem. The VFR is able to achieve the state-of-the-art rendering quality while using significantly less training time. It has a great potential to replace the standard volume rendering in many neural rendering methods based on volume representation, with the benefits of either a better rendering quality by employing a larger color neural network or a significantly reduced training time thanks to one single color neural network evaluation.
[33] Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P Srinivasan, Richard Szeliski, Jonathan T Barron, and Ben Mildenhall. BakedSDF: Meshing neural SDFs for real-time view synthesis. arXiv preprint arXiv:2302.14859, 2023.
[34] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoc trees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761, 2021.
[35] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun.  $\mathrm{NeRF}++$ : Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[36] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.