SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To evaluate the performance of our approach, we conduct experiments on NHR [48] and DyNeRF [17] datasets. NHR contains four videos recorded at 30fps that capture an athlete performing large
C-NeRF (1019MB)
Neural Volumes (658MB)
DyNeRF (12MB)
DyMap (239MB)
Figure 2: Qualitative results on NHR dataset. Our method achieves comparable rendering quality to state-of-the-art methods while requiring significantly less storage. Note that although DyNeRF also has a small storage size, it requires days for training.
Ours (16.6MB)
Table 1: Quantitative results on NHR dataset. Metrics are averaged over all scenes. Note that D-NeRF and DyNeRF are both single MLP based methods, which is of low model size but extremely slow training speed.
NV[22]	C-NeRF[47]	D-NeRF[31]	DyNeRF[17]	DyMap[30]	K-Planes[35]	Ours
PSNR↑	30.86	31.32	29.25	30.87	32.30	31.08	32.10
SSIM↑	0.941	0.949	0.920	0.943	0.953	0.946	0.947
Size (MB)↓	658	1019	4	12	239	103	16.6
movements in a bounded scene with simple backgrounds. Following the setting of DyMap [30], we select 100 frames from each video and use 90 percent of camera views for training and the rest for testing. DyNeRF contains six 10 videos recorded at 30fps that capture a person cooking in a kitchen by 15-20 cameras from a range of forward-facing view directions. We use all 300 frames and follow the same training and testing split as in [17].
5.2 Comparison with the state-of-the-art methods
Baseline methods. For experiments on NHR dataset, we compare with (1) Neural Volumes (NV) [22] (2) C-NeRF [47] (3) D-NeRF [31] (4) DyMap [30] (5) K-Planess [35]. For experiments
Mixvoxels (500MB)
K-Planes (103MB)
Ours (27.4MB)
Figure 3: Qualitative results on DyNeRF dataset. Our method achieves comparable rendering quality to previous methods while requiring significantly less storage. Please zoom in for details.
Table 2: Quantitative results on DyNeRF dataset. Metrics are averaged over all scenes. DyNeRF $^1$  and LLFF $^1$  only report metrics on the flame salmon scene. We provide more detailed results on each scene in the supplementary materials.
LLFF1 [23]	DyNeRF1 [17]	Mixvoxels [45]	K-Planes [35]	Ours
PSNR↑	23.24	29.58	30.81	30.54	30.58
SSIM↑	0.848	0.961	0.960	0.927	0.923
Size (MB)↓	-	28	500	103	27
on DyNeRF dataset, we compare with (1) LLFF (2) The original method proposed in DyNeRF [17] (3) Mixvoxels [45] (4) K-Planes [35]. For K-Planes, we run their official implementation and found that the results on DyNeRF are slightly different from their paper. For other methods, we directly adopt the results from [30] on NHR and [35, 45] on DyNeRF dataset.
Tabs. 1 and 2 list the comparison of our method with the state-of-the-art methods on NHR and DyNeRF datasets in terms of rendering quality and model size. We report PSNR and SSIM as the metrics for rendering quality, and the size of the model is measured in megabytes (MB). We also provide qualitative results in Figs. 2 and 3. The results show that our method can achieve rendering quality comparable to the state-of-the-art while offering significant advantages in storage efficiency. Single MLP-based methods, such as D-NeRF and DyNeRF, have similar storage sizes to ours, but their training process is extremely slow, requiring more than days.
5.3 Ablation studies
To analyze the effectiveness of our proposed dynamic codebook strategy, we conduct ablation studies on NHR and DyNeRF datasets. We first train the feature plane based model without compression, then we apply compression without dynamic codebook, and finally, we utilize dynamic codebook to incrementally refine the model. We report quantitative results in Tab. 3 and provide qualitative results in Fig. 4. Results in Tab. 3 show that the compression step can drastically reduce storage requirements, but causes a noticeable decline in rendering quality. And the proposed dynamic codebook strategy can compensate for this by introducing a small amount of additional storage while
compression w/o dynamic codebook
compression w/ dynamic codebook
Figure 4: Qualitative results of ablation studies on NHR dataset. Compression without dynamic codebook will result in a non-negligible loss of rendering quality in facial regions, while the usage of dynamic codebook can effectively compensate for this.
Table 3: Quantitative results of ablation studies. We report the rendering quality and model before compression and after compression with and without DC (dynamic codebook).
Method	NHR	DyNeRF
PSNR↑	SSIM↑	Size (MB)↓	PSNR↑	SSIM↑	Size (MB)↓
baseline	32.16	0.947	91	30.56	0.923	523
compression w/o DC	31.52	0.943	5.7	30.37	0.920	22
compression w/ DC	32.10	0.947	16.6	30.58	0.923	27
achieving rendering quality almost equivalent to the level before compression. Results in Fig. 4 demonstrate that compression particularly degrades the rendering quality in detailed regions, such as the facial regions. The use of a dynamic codebook can effectively compensate for the loss induced by compression.
We presented a novel neural representation, named dynamic codebook, for learning neural volumetric videos with low storage costs while enabling high-quality view synthesis. Our approach models the volumetric video using a set of feature planes and compresses the video by reducing the spatial and temporal redundancy of features. We calculate the importance score of features and merge similar features that are of small importance scores. To compensate for the artifacts caused by the compression, our approach pinpoints the regions with lower rendering quality and appends dynamic codes for them. Experiments demonstrated that our approach achieves competitive rendering quality while being able to take up much fewer storage cost compared to state-of-the-art methods.
Discussion. We currently update the code independently for each time segment, which takes more than one hour for each dynamic scene, thereby slowing down the reconstruction. A solution is to increase the parallelism of optimizing codes. We leave it to future works.
Acknowledgement. The authors would like to acknowledge support from NSFC (No.62172364), Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
[45] F. Wang, S. Tan, X. Li, Z. Tian, and H. Liu. Mixed neural voxels for fast multi-view video synthesis. arXiv preprint arXiv:2212.00190, 2022.
[46] L. Wang, Q. Hu, Q. He, Z. Wang, J. Yu, T. Tuytelaars, L. Xu, and M. Wu. Neural residual radiance fields for streamably free-viewpoint videos. arXiv preprint arXiv:2304.04452, 2023.
[47] Z. Wang, T. Bagautdinov, S. Lombardi, T. Simon, J. Saragih, J. Hodgins, and M. Zollhofer. Learning compositional radiance fields of dynamic human heads. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5704-5713, 2021.
[48] M. Wu, Y. Wang, Q. Hu, and J. Yu. Multi-view neural human rendering. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1682-1691, 2020.
[49] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[50] S. Zhang, S. Guo, W. Huang, M. R. Scott, and L. Wang. V4d: 4d convolutional neural networks for video-level representation learning. arXiv preprint arXiv:2002.07442, 2020.
[51] J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on information theory, 23(3):337-343, 1977.