Compact Neural Volumetric Video Representations with Dynamic Codebooks
Authors: Haoyu Guo
Sida Peng
Yunzhi Yan
Linzhan Mou
Yujun Shen
Hujun Bao
Xiaowei Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Computer Vision
3D Vision
Volumetric Video
TL;DR: We propose Dynamic Codebook, which efficiently compresses volumetric videos with minimal impact on rendering quality.

Abstract: This paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. Some recent feature grid-based methods have shown superior performance of fast learning implicit neural representations from input 2D images. However, such explicit representations easily lead to large model sizes when modeling dynamic scenes. To solve this problem, our key idea is reducing the spatial and temporal redundancy of feature grids, which intrinsically exist due to the self-similarity of scenes. To this end, we propose a novel neural representation, named dynamic codebook, which first merges similar features for the model compression and then compensates for the potential decline in rendering quality by a set of dynamic codes. Experiments on the NHR and DyNeRF datasets demonstrate that the proposed approach achieves state-of-the-art rendering quality, while being able to achieve more storage efficiency. The source code is available at https://github.com/zju3dv/compact_vv.

Compact Neural Volumetric Video Representations with Dynamic Codebooks
Haoyu Guo
Sida Penglt
Yunzhi Yan
Linzhan Mou
Yujun Shen²
Hujun Bao
Xiaowei Zhou
Zhejiang University
2 Ant Group
Abstract
This paper addresses the challenge of representing high-fide figh-fideitty volumetric videos with low storage cost. Some recent feature grid-based methods have shown superior performance of fast learning implicit neural representations from input 2D images. However, such explicit representations easily lead to large model sizes when modeling dynamic scenes. To solve this problem, our key idea is reducing the spatial and temporal redundancy of feature grids, which intrinsically exist due the self-similarity of scenes. To this end, we propose novel neural representation, named dynamic codebook, which first merges similar features for the model compression and then compensates for the potential decline in rendering quality by set of dynamic codes. Experiments on the NHR and DyNeRF datasets demonstrate that the proposed approach achieves state-of sate--o-theeart art rendering quality, while being able to achieve more storage efficiency. The source code available https:/ / thub. cwm/ haamm//juddv//v cttrddd/omppcttvv
Introduction
Volumetric videos record the content of dynamic 3D scenes and allow users to watch captured scenes from arbitrary viewpoints, which has a wide range of applications, such virtual reality, augmented reality, telepresence, and so on. Traditional methods typically represent volumetric videos as sequence of textured meshes, which are reconstructed by the multi-view stereo [36, 37, 12, 15] or depth fusion [10, 27J. Such techniques require sophisticated hardware, e.g., multi-view RGB-D sensors, achieve high reconstruction accuracy, which expensive and limits their application scenarios.
Recently, implicit neural representations [24, 1, 49, 2] have emerged as promising strategy to recover 3D scenes from only 2D images. As a representative work, neural radiance field (NeRF) [24] employs an MLP network predict the density and color for any 3D point, and effectively learns the model parameters from 2D images through the volumetric rendering technique. DyNeRF [17] extends NeRF to dynamic scenes by introducing time-varying latent codes as additional input of the MLP network, thereby enabling the model to encode the per-frame content of the scene. Although these methods achieve impressive rendering quality, they are extremely slow train, especially dynamic scenes, For example, DyNeRF requires over 1000 GPU hours to learn a 10-second volumetric video.
To overcome this problem, [8, 50, 20] or feature planes [35, 4, 38), to accelerate the training process. They typically store learnable feature vectors at explicit structures and use the interpolation technique assign a feature
methods exploit explicit representations, such feature volumes
Coorespponiigg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

vector to arbitrary scene coordinate, which is then fed into a lightweight MLP network to predict the density and color. By decreasing the number of parameters the MLP network, these methods achieve a significant speedup in training. However, the explicit representations can easily take up large storage, which is hard to scale dynamic 3D scenes.
Some methods [42, 16] reduce storage redundancy based on the internal similarity of scenes, achieving satisfactory compression results on static scenes. However, they do not take into account the characteristics of dynamic scenes. Experiments show that directly applying these methods to dynamic scenes results in negligiie performance degradation. For dynamic scenes, we observe that the redundancy of features learned by existing methods [8, 50, 20, 35, 4, 38] is not only reflected in spatial correlations, but more importantly, dynamic scenes are essentially produced by the movement and changes of static scenes, thereby intrinsically having a strong temporal correlation. If this prior knowledge is not adequately utilized, the model will inevitably contain a considerable amount of unnecessary repetitive storage.
Motivated by these observations, we propose a novel representation, named dynamic codebook, for efficient and compact modeling of dynamic 3D scenes. We first employ a compact codebook to accomplish compression, then incrementally update the codebook to compensate for the potential decline in rendering quality of dynamic detailed regions caused by compression. The representation of codebook can effectively mitigate the issue of code redundancy in time and space, thereby significantly reducing unnecessary storage. However, this representation might cause the codes of some regions to be less precise, which will decrease the rendering quality, especially in dynamic detailed regions. The spatial distribution of these regions may vary significantly over time. Therefore, we separately identify the most necessary parts of the codes to optimize for each small time fragment and incrementally add them to the codebook to improve the rendering quality of these regions. Since these regions typically account for a small proportion of the scene, the new codes will not occupy much storage. Therefore, our proposed dynamic codebook based strategy can achieve high compression rate while maintaining high rendering quality.
We evaluate our approach on the NHR [48] and DyNeRF [17] datasets, which are widely-used benchmarks for dynamic view synthesis. Experimental results demonstrate that the proposed approach could achieve rendering quality comparable to the state-of-the-a methods while achieving much higher storage efficiency.
In summary, our contributions are as follows: (1) We propose a novel representation called dynamic codebook, which effectively decreases the model size by reducing the feature redundancy in time and space while maintaining the rendering quality of dynamic detailed regions. (2) Experiments demonstrate that our approach achieves rendering quality on par with state-of-the-an methods, while significantly improving storage efficiency.
2 Related works
View synthesis of static scenes. Traditional methods use mesh represent scene geometry and texture mapping to depict the scene's appearance, which can be rendered from arbitrary viewpoints. However, the reconstruction process based on this representation is typically not end-to-end, leading to lack of photo-realism of the rendering results. [24] proposes a continuous representation that models the density and color of static scenes with MLP networks, which can be used for high-quality free view synthesis using volume rendering technique. [1] utilizes multi-scale representation to address the inherent aliasing of NeRF, and improve the ability to represent fine details. [49, 2] enable modeling of unbounded scenes with complex backgrounds by the design of coordinate mapping.
The pure MLP-based representation much more compact compared traditional representations. however, a large amount of points sampling is required during rendering, and each point will be fed into the MLP, resulting in very low efficiency. [21] utilizes a sparse occupancy grid to avoid sampling points in empty space to accelerate the training and rendering speed. [32] replaces the large MLP in NeRF with thousands of tiny MLPs. [9, 41] directly adopt the explicit volumetric representation during training and utilize coarse to fine strategy to achieve high resolution. [25] employs multiresolution hash table for encoding and lightweight MLPs as decoders, achieving unprecedentedly fast training. [3] further extends anti-aliasing to grid-based representation by contraction function to model large unbounded scenes with feature grids, and enables real-time view synthesis.
multisampling and prefiltering. [33] introduces novel
2

---

Volumetric video. Following NeRF's considerable success in modeling static scenes, some studies [19, 30. 17, 31, 35. 45] have sought extend it to dynamic scenes. There are primarily two approaches to this. The first approach is to directly model temporal variations by directly modifying the input format of MLP to enable view synthesis not only at any viewpoint but also at any given time. [18] directly input the timestamp to MLP for space-time view synthesis. [17] utilizes time-varying latent codes as the additional input to MLP, the latent codes can be learned to model the per-frame content of dynamic scenes. The training of this approach is typically challenging, either requiring some additional supervision such as monocular depth or optical flow, or taking much time to converge. The second approach involves using NeRF to model canonical space and then learning the deformation from any moment to this canonical space. [31] employs another MLP to model the deformation at each timestamp to this canonical space. [28] utilizes latent codes to model deformation. However, deformation struggles to model complex temporal changes, such as large movements, or the emergence of new content, such fluid mixing Moreover, these two kinds of approaches both encounter issues similar to those faced by NeRF in static scenes, namely, low training and rendering efficiency.
Methods based on explicit or hybrid representations have also been studied and explored for modeling volumetric videos. [22] encodes multi-view information at arbitrary timestamp and uses 3D deconvolutional network to produce volumes of density and color. [47] uses 3D CNN to predict feature volumes instead of the explicit volumes, and additionally adopts an MLP to predict the radiance fields. [8, 20, 50] employ a feature grid to model the canonical space, which enables efficient training and rendering. [35, 4, 38] represent volumetric videos using multiple feature planes, offering a more compact representation compared to the feature grids. [45] explicitly partitions the scene into dynamic and static regions using proposed variation field, and models them separately using TensoRF [5], allowing for remarkably fast training. [40] divides scenes into static, deforming and new regions, and utilizes a sliding window-based hybrid representation for efficient scene modeling. [46] explicitly models the residual information between adjacent timestamps in the spatial-temporal feature space and employs a compact motion grid along with residual feature grid to exploit
inter-frame feature similarities.
Compression of implicit neural representations. Methods based feature grids and feature planes can provide significant efficiency improvements, but they also introduce substantial storage requirements. To alleviate this issue, [7] proposes to use pruning strategy eliminate unnecessary parameters in the feature grid. [5] decomposes the 3D grids into lower dimension. It utilizes VM decomposition decompose the 3D grid into multiple 2D matrices and even employs CP decomposition to decompose it into multiple ID vectors. While CP decomposition significantly improves storage efficiency, it results in neg-neglig performance loss. [43] implements compression that can be dynamically adjusted by the proposed rank-residual learning strategy. Vector quantization [1 1, 13] is classic compression technique that is widely applied in 2D image and video compression tasks [26, 6, 39]. The main idea of which is to merge similar codes into one through clustering, thereby reducing storage redundancy. Recently, VQAD [42] and VQRF [16] use vector quantization to achieve compression of static 3D scenes. VQAD learns a codebook while training the model, while VQRF implements compression in post post-processing manner. [14] utilizes Fourier transform for compression, which is able to capture high-frequency details while remaining compact. [34] proposes to use wavelet coefficients to improve parameter sparsity.
3 Method
Given multi-view videos captured by synchronized and calibrated cameras, our goal is to generate volumetric video that requires low disk storage while retaining the capability of high-fidelity rendering. Our approach is illustrated in Fig. 1. We represent the volumetric video by multiple feature planes, which can be learned from multi- videos using the volume rendering technique (Sec. 3.1). To enhance the storage efficiency of the model, we propose a codebook- based representation, which aims to reduce the redundancy of the feature plane, thereby achieving model compression (Sec. 3.2). To compensate for the loss of rendering quality in certain areas due to compression, we extend the codebook to dynamic codebook. This significant enhancement in rendering quality introduces only minimal amount of additional storage (Sec. 3.3).
3

---

2
3
2
2
3
4
5
6
4
5
6
7
8
9
7
9
9
plane
codebook
index plane
codebook
(a) Original feature plane
(b) Feature plane reformulation
(c) Codebook compression
2
2
3
2
2
n-1
n
3
10
11
4
10
11
12
5
6
4
5
6
...
7
9
9
13
9
9
7
9
9
updated index planes for all time fragments
dynamic codebook
(d) Construction of dynamic codebook with incrementally added codes
Figure 1: Overview of our method. (a) We model volumetric videos with multiple feature planes, which can be learned with the supervision of multi-view 2D videos. (b) To facilitate the subsequent process, we simply convert the feature planes an index plane as well as codebook, the codebook is constructed by directly flattening the feature plane. (c) To reduce the model redundancy. we use
clustering method to merge similar codes, such as code {1, 2} and (8, 9} (b). (d) To compensate for the decline in rendering quality caused by the compression, dynamic codes are appended to the trimmed codebook. First, we maintain an index plane for each time fragment, where most indices on these planes remain the same. Then, optimize portion of the codes (masked as green in this figure) for each time fragment. These codes are incrementally added to the codebook, and the corresponding positions in the index plane are updated, such {10, 11} } in the index plane of the first time fragment this figure.
3.1 Learning volumetric videos with feature planes
Inspired by [35, 4, 38], we model the dynamic scene by six feature planes, including three spatial planes denoted as Pzy. Pzz. and P y= and three spatial-temporal planes denoted as Par,, P Pyt: and P.t. Given 3D coordinate x (x. y. 2) and time we obtain feature vectors [f(x, L); by
normalizing (x, t) according to the resolution of feature planes and projecting onto six planes:
f(x ()2 = (P (x,()))
where denotes projection function to i-th feature plane, and & denotes bilinear interpolation. We then calculate the feature vector of by:
f(x, () = f(x, Qaa @ fxx, 1) f(x, l)s @ [xx, Oue + ((x, s(x, ()x,,
(2)
where @ is the Hadamard product. Then f(x, t) is fed into an MLP network to predict the density (x, t) and color c(x,
Following [24], we adopt volume rendering to optimize the feature planes as well as the MLP parameters. Specifically, render pixel time t, first sample set of 3D coordinates {x:) along the ray passing through p. and then calculate the accumulated color along the ray:
K
K
Č(r, = Σ Tarc = Σ - exx((-xxx0
111
where 6, = ||xi4| x:l12 is the distance between adjacent sampled points, and T exp(- Σ;=1 o,6,) denotes the accumulated transmittance along the ray.

---

Then we optimize the model with L2 rendering loss with the supervision of ground truth pixel colors. Moreover, we use the standard regularization terms, including L1 norm loss and TV (total variation) loss following [5]. Please refer to supplementary materials for more details.
3.2 Compression of model
Though volumetric videos based on feature plane representations allow for efficient training and rendering, they suffer from a substantial model size, which in practice can reach approximately 30- 50MB per second. This poses significant challenge for the storage and dissemination of volumetric videos. In this section, we demonstrate how to achieve model compression using a codebook-based representation.
Previous works in static scene reconstruction and view synthesis have demonstrated that the usage of feature grids or feature planes introduces significant spatial redundancy. They achieve compression by reducing spatial redundancy through CP decomposition [5, 43]. vector quantization [42, 16], Fourier / Wavelet transformations/ only spatially, but also temporally. The usage of multiple 2D planes has to some extent mitigated the spatial redundancy. However, spatial redundancy still exists, and more importantly, there is non-negligible temporal redundancy.
34], and so on. For volumetric videos, redundancy exists not
As stated by [16], when representing 3D scenes using feature grids, 99.9% of the importance is contributed by merely 10% of the voxels. While the situation is not as extreme when we use 2D planes to represent volumetric videos, similar pattern can still be observed. Hence, we can apply this principle to reduce the redundancy the feature planes and achieve compression.
To facilitate the subsequent process, we initially convert the model into a codebook format, as illustrated in Fig. lb. Specifically, we flatten all features on feature planes into codebook, after which we convert the feature planes into index planes. In other words, we store the index mapping to the codebook at each grid point of the planes, instead of directly storing features.
In order to perform model compression based on the principle discussed above, we first calculate the importance score for each code in the codebook, according to its contribution the rendering weight in the volume rendering process. Specifically, we randomly sample rays among all camera views and timestamps to perform volume rendering. For each sampled point p, on each ray, we record the volume rendering weights T,a; of p:, we also record the corresponding codes as well as their weights when performing trilinear interpolation at point p,. We calculate the importance score of each code by accumulating the rendering weights of all points that are mapped to it:
Le Σ Woi Taa P.EN.
(4)
where, N. denotes the set of points that are mapped to code and w denotes the trilinear interpolation weight that code c contribute to point p,. To reduce the storage, we first discard the less important codes by merging portion of the codes with the lowest importance scores into single one. For codes with medium importance scores, we cluster them into a smaller number of codes using a clustering method, as illustrated in Fig. Ic. The clustering is performed in optimization manner, following VQ-VAE VAE [44]. Specifically,, we randomly initialize n codes, where n is much smaller than the number of the codes to be clustered. Then the codes iteratively, please refer to the supplementary materials for more details. We retain the codes with the highest importance scores as they are. After completing the merging and clustering operations, we update the indices on the index planes for each time fragment accordingly.
exponential moving averages (EMA) to update
3.3 Dynamic codebook
The badebook-based compression method can significantly reduce model storage but inevitably leads to decrement in rendering quality. We observe that the rendering of detailed regions will degrade after compression. The reason lies in the fact that these regions require particularly high precision for codes. The previously mentioned clustering strategy may reduce the accuracy some of these codes. Although the importance score can measure the contribution of each code during the rendering process, it does not distinguish well between its contribution to detailed and non-detailed regions. Although these detailed regions typically occupy small proportion of the scenes, their
5

---

spatial distribution may vary over time, making it difficult to enhance the rendering quality by directly optimizing a small subset of codes.
To solve the problem, we first divide the temporal dimension into numerous fragments. Our goal is to adaptively identify and optimize the parts most in need of enhancement within each time fragment. These are the portions where the shared spatial feature plane provides relatively poorer representation during that time fragment. Since these parts constitute a smaller proportion, they do not lead to substantial storage increase.
To this end, leverage the backpropagation gradient to pinpoint the top codes demanding the most optimization within each time fragment. Specifically, for each time fragment, we run the forward and backpropagation steps in the training process without optimization and accumulate the gradients of each code for several iterations. Then we simply select the k codes with the highest accumulated gradients and optimize them for this time fragment. We repeat this process for each time fragment, and the optimized codes are independent of all time fragments.
Thanks to the codebook representation, we can append the optimized codes for each time fragment into the codebook. Additionally, we store an independent index plane for this time fragment and update the corresponding indices within it, as shown in Fig. ld. We term the final codebook as dynamic codebook because it constructed incrementally by optimizing for each time fragment.
The index planes store integer indices, thus occupying minimal storage space. Furthermore, as each time fragment only requires optimization of small number of codes, the dynamic codebook does not introduce large additional storage. However, the design of dynamic codebook could effectively enhance the rendering quality of detailed regions.
Post-processing For further compression, we apply post-processing steps to the codebook. We use uniform weight quantization to quantize the codebook. Then we encode all components, including the quantized dynamic codebook, index planes as well as MLP parameters with entropy encoding [5 I].
4 Implementation details
Network architecture. We set the resolution of the feature planes in the spatial dimension as 256 on NHR and 640 on DyNeRF dataset, and the one in the temporal dimension is set as 100 on NHR and 150 on DyNeRF dataset. We model density and appearance with two individual feature planes with the number of feature channels as 16 and 48 respectively. For appearance feature planes, our model also includes multi-scale planes in addition to the six feature planes the highest resolution to encourage spatial smoothness and coherence, following [35]. For neward-ffcing scenes in the DyNeRF dataset, we apply NDC transformation that bounds the scene in a perspective frustum.
Construction of dynamic codebook. We construct separate codebook for both density and appearance. During the compression step, we discard the portion of codes that only accumulate an importance score contribution of 0.001 %, by merging them into simple zero code. We then retain the top 30% of codes with the highest importance score contributions and cluster the remaining codes into k codes. We set l 4096 for NHR dataset and 16384 for DyNeRF dataset. We divide each scene on NHR / DyNeRF into 100 60 time fragments, and optimize 1000 appearance codes and 5000 density codes for each time fragment.
Training. We implement our method with PyTorch [29]. We set the learning rate as 2e-3 for MLP parameters and 0.03 for feature planes, and use Adam optimizer to train the network with batches of 4096 rays. We train the model on one single NVIDIA A100 GPU, which takes about 1.5 / 4.3 hours for training and 1.0/ 1.7 hours for the construction of dynamic codebook on NHR / DyNeRF datasets.
Experiments
5.1 Datasets
To evaluate the performance of our approach, we conduct experiments on NHR [48] and DyNeRF [17] datascts. NHR contains four videos recorded at 30fps that capture athlete performing large

---

8
Ground truth
C-NeRF (1019MB)
Neural Volumes (658MB)
DyNeRF (12MB)
DyMap (239MB)
Ours (16.6MB)
Figure 2: Qualitative results on NHR dataset. Our method achieves comparable rendering quality state-of-the art methods while requiring significantly less storage. Note that although DyNeRF also has small storage size, it requires days for training.
Table 1: Quantitative results NHR dataset. Metrics are averaged over all scenes. Note that D-NeRF and DyNeRF are both single MLP based methods, which is of low model size but extremely slow training speed.
NV[22]
C-NeRF[47] D-NeRF[31] DyNeRF[17] DyMap[30] K-Planes[35]
Ours
PSNR1 30.86
31.32
29.25
30.87
32.30
31.08
32.10
SSIM
0.941
0.949
0.920
0.943
0.953
0.946
0.947
Size (MB)
658
1019
4
12
239
103
16.6
movements in a bounded scene with simple backgrounds. Following the setting of DyMap [30], we select 100 frames from each video and use percent camera views for training and the rest for testing. DyNeRF contains six 10 videos recorded at 30fps that capture person cooking in kitchen by 15-20 cameras from range of forward-facid view directions. We use all 300 frames and follow the same training and testing split as in [17].
5.2 Comparison with the state-of-the-an methods
Baseline methods. For experiments on NHR dataset, we compare with (1) Neural Volumes
(NV) [22] (2) C-NeRF [47] (3) D-NeRF [31] (4) DyMap [30] (5) K-Planes [35]. For experiments
7

---

Ground truth
Mixvoxels (500MB)
K-Planes (103MB)
Ours (27.4MB)
Figure 3: Qualitative results on DyNeRF dataset. Our method achieves comparable rendering
quality to previous methods while requiring significantly less storage. Please zoom in for details.
Table 2:
Quantitative results on DyNeRF dataset. Metrics are averaged over all scenes. DyNeRFi and LLFF only report metries on the flame salmon scene. We provide more detailed results on each scene in the supplementary materials.
LLFF [23]
DyNeRF! [17]
Mixvoxels [45]
K-Planes [35]
Ours
PSNR SSIM
23.24
29.58
30.81
30.54
30.58
0.848
0.961
0.960
0.927
0.923
Size (MB)
28
500
103
27
on DyNeRF dataset, we compare with (1) LLFF (2) The original method proposed in DyNeRF [17] (3) Mixvoxels [45] (4) K-Planes [35]. For K-Planes, we run their official implementation and found that the results on DyNeRF slightly different from their paper. For other methods, directly adopt the results from [30] on NHR and [35, 45] on DyNeRF dataset.
Tabs. and 2 list the comparison of our method with the state-of-the-an methods on NHR and DyNeRF datasets terms of rendering quality and model size. We report PSNR and SSIM as the metrics for rendering quality, and the size of the model is measured in megabytes (MB). We also provide qualitative results in Figs. 2 and 3. The results show that our method can achieve rendering quality comparable to the state-of-the-an while offering significant advantages in storage efficiency. Single MLP-based methods, such as D-NeRF and DyNeRF, have similar storage sizes to ours, but their training process is extremely slow, requiring more than days.
5.3 Ablation studies
To analyze the effectiveness of our proposed dynamic codebook strategy, we conduct ablation studies on NHR and DyNeRF datasets. We first train the feature plane based model without compression, then we apply compression without dynamic codebook, and finally, we utilize dynamic codebook to incrementally refine the model. results in Fig. 4. Results in Tab. 3 show that the compression step can drastically reduce storage requirements, but causes noticeable decline in rendering quality. And the proposed dynamic codebook strategy compensate for this by introducing small amount of additional storage while
report quantitative results in Tab. and provide qualitative

---

SIS
EIS
SIS
compression w/o dynamic codebook
compression w/ dynamic codebook
baseline
Figure 4: Qualitative results ablation studies on NHR dataset. Compression without dynamic codebook will result in non-negligible loss of rendering quality in facial regions, while the usage of dynamic codebook can effectively compensate for this.
Table 3: Quantitative results of ablation studies. We report the rendering quality and model before compression and after compression with and without DC (dynamic codebook).
NHR
DyNeRF
Method
PSNRT
SSIM
Size (MB)
PSNR
SSIM
Size (MB).
basline
32.16
0.947
91
30.56
0.923
523
compression w/o DC
31.52
0.943
5.7
30.37
0.920
22
compression w/ DC
32.10
0.947
16.6
30.58
0.923
27
achieving rendering quality almost equivalent to the level before compression. Results in Fig. demonstrate that compression particularly degrades the rendering quality in detailed regions, such as the facial regions. The use of a dynamic codebook can effectively compensate for the loss induced by compression.
Conclusion
We presented novel neural representation, named dynamic codebook, for learning neural volumetric videos with low storage costs while enabling high-quality view synthesis. Our approach models the volumetric video using set of feature planes and compresses the video by reducing the spatial and temporal redundancy of features. We calculate the importance score features and merge similar features that of small importance scores. To compensate for the artifacts caused by the compression, our approach pinpoints the regions with lower rendering quality and appends dynamic codes for them. Experiments demonstrated that our approach achieves competitive rendering quality while being able to take much fewer storage cost compared to state-of-the-an methods.
Discussion. We currently update the code independently for each time segment, which takes more than one hour for each dynamic scene, thereby slowing down the reconstruction. A solution is to increase the parallelism of optimizing codes. We leave to future works.
The authors would like to acknowledge support from NSFC (No.62172364), Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.
Acknowledgement.

---

References
[1] J. T. Barron, B. Mildenhall, M. Tancik, Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: multiscale representation anti-aliasing neural radiance fields. In Int. Conf. Comput. Vis., pages 5855-5864, 2021.
[2] T. Barron, Mildenhall, D. Verbin, P. P. Srinivasan, and Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In JEEE Conf. Comput. Vis. Pattern Recog., pages 5470-5479, 2022.
[3] Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.067 06706, 2023.
[4] A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. In IEEE Conf. Comput. Vis. Puttern Recog., 2023.
[5] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In Eur. Conf. Comput. Vis.. 2022.
[6] P. C. Cosman, K. L. Oehler, E. A. Riskin, and R. M. Gray. Using vector quantization for image processing. Proceedings of the IEEE, 81(9):1326-133: 1341, 1993.
[7] C. L.. Deng and E. Tartaglione. Compressing explicit voxel grid representations: fast nerfs become also small. In IEEE Winter Conf. Appl. Comput. Vis., pages 1236-1245, 2023.
[8] J. Fang. T. Yi, Wang, Xie, X. Zhang, W. Liu, M. Nießner, and Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH, pages 1-9, 2022.
[9] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3501-5510, 2022.
[10] P. Gao, Jiang, H. You, P. La, S. C. Hoi, X. Wang, and Li. Dynamic fusion with intra-and inter- modality attention flow for visual question answering. In JEEE Conf. Comput. Vis. Pattern Recog,, pages 6639-6648, 2019.
[11] A. Gersho and R. M. Gray. Vector quantization and signal compression, volume 159. Springer Science Business Media, 2012.
[12] M. Goesele, B. Curless, and S. M. Scitz. Multi-view stereo revisited. In IEEE Conf. Comput. Vis. Pattern Recog,, pages 2402-2409, 2006.
[13] R. Gray. Vector quantization. IEEE Assp Magazine, 1(2):4 190,4-29,110 1984.
[14] B. Huang, X. Yan, A. Chen, S. Gao, and J. Yu. Pref: Phasorial embodding fields for compact neural representations. arXiv preprint arXiv:2205. 13524, 2022.
[15] M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Trans. Graph., pages 1-13, 2013.
[16] L. Li, Z. Shen, Z. Wang, L. Shen, and L. Bo. Compressing volumetric radiance fields to I mb. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
[17] T. Li. M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, Neural 3d video synthesis from multi-view video. In IEEE Conf. Comput. Vis. Pattern
R. Newcombe, Recog,, pages 5521-5531, 2022.
[18] Z. Li, Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6498 6508, 2021.
[19] H. Lin, S. Peng, Z. Xu, Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Efficient neural radiance fields with learned depth-guided sampling. In SIGGRAPH Asia, 2022.
[20] J. Liu, Y.P. Cao, W. Mao, W. Zhang, D. J. Zhang. J. Keppo, Y. Shan, X. Qie, and M. Z. Shou. Devif: Fast deformable voxel radiance fields for dynamic scenes. arXiv preprint arXiv:2205.15725 2022.
[21] L. Liu, J. Gu, K. Zaw Lin, T.-S, Chua, and C. Theobalt. Ncural sparse voxel fields. In Adv. Neural Inform. Process. Syst., pages 15651-15663, 2020.
[22] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph., 2019.
10

---

[23] B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng. and A. Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. TOG, 38(4):1-14, 2019.
[24] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Fur. Conf. Comput. Vis., 2020.
[25] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphies primitives with a multiresolution hash encoding. ACM Trans. Graph., pages 1-15, 2022.
[26] N. M. Nasrabadi and R. A. King. Image coding using vector quantization: A review. IEEE Transactions communications, 36(8):957-99) 971, 1988.
[27] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking, In ISMAR. pages 127-136, 2011.
[28] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin- Brualla. Nerfies: Deformable neural radiance fields. In Int. Conf. Comput. Vis., pages 3865-5874, 2021.
[29] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Adv. Neural Inform. Process. Syst., pages 8026-8037, 2019.
[30] S. Peng. Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Representing volumetric videos as dynamic mlp maps. In IEEE Conf. Comput. Vis. Pottern Recog., 2023,
[31] A. Pumarola, F. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic 2021.
scenes. In IEEE Conf. Comput. Vis. Pattern Recog., pages
[32] C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Int. Conf. Comput. Vis., pages 14335 14345, 2021.
[33] C. Reiser, R. Szeliski, D. Verbin, P. P. Srinivasan, B. Mildenhall, A. Geiger, J. T. Barron, and Hedman. Mcrf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. arXiv preprint arXiv:2302. 12249, 2023.
[34] D. Rho, B. Lee, S. Nam, J. C. Lee, H. Ko, and E. Park. Masked wavelet representation for compact neural radiance fields. In IEEE Conf. Comput. Vis. Puttern Recog., 2023.
[35] Sara Fridovich- Keil and Giacomo Mcanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In IEEF Conf. Comput. Vis. Pattern Recog., 2023.
[36] L. Schonberger and J.-M. Frahm. Structure-from frommmmoonn revisited. IEEE Conf. Comput. Vis. Pattern Recog., pages 4104-4113, 2016.
[37] L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Eur. Conf. Compur. Vis., pages 501-518, 2016.
[38] R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In IEEE Conf. Comput. Vis. Pattern Recog., 2023.
[39] T. Sikora. The mpeg-4 video standard verification model. IEEE Transactions on circuits and systems for video technology, 7(1):19-1, 1997.
[40] L. Song, A. Chen, 7. Li, Z. Chen, I.. Chen, J. Yuan, Y. Xu, and A. Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. ACM Trans. Graph, pages 2732- 2742, 2023.
[41] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. IEEE Conf. Comput. Vis. Pattern Recog.. pages 5459-5469, 2022.
[42] T. Takikawa, A. Evans, J. Tremblay, T. Müller, M. McGuire, A. Jacobson, and S. Fidler. Variable bitrate neural fields. In SIGGRAPH, pages 1-9, 2022.
[43] J. Tang, X. Chen, J. Wang, and G. Zeng. Compressible-composabe nerf via rank-residual decomposition. arXiv preprint 2rXiv:2205.48877 2022.
[44] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In Adv. Neural Inform. Process. Syst. 2017.

---

[45] F. Wang, S. Tan, X. Li, Z. Tian. and H. Liu. Mixed neural voxels for fast mulli-view video synthesis. arXiv preprint 20X10:2222.00990, 2022.
[46] L. Wang. Q. Hu, Q. He, Z. Wang. J. Yu, T. Tuytelaars, L. Xu, and M. Wu. Neural residual radiance fields for streamably viee-vie videos. arXiv preprint S.K3::2304.045522. 2023.
[47] Z. Wang, T. Bagautdinov, S. Lombardi, T. Simon, J. Saragih, J. Hodgins, and M. Zollhofer. Learning compositional radiance fields of dynamic human heads. In IEEE Conf. Comput. Vis, Pattern Recog,, pages 5704 5713, 2021.
[48] M. Wu, Y. Wang. Q. Hu, and J. Yu. Multi-view neural human rendering. In IEEE Conf. Comput. Vis. Pattern Recog, pages 1682-1691, 2020.
[49] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++; Analyzing and improving neural radiance fields. arXiv preprint arXiv2011007499, 2020.
[50] S. Zhang, S. Guo, W. Huang, M. R. Scott, and L. Wang. V4d: 4d convolutional neural networks for video-level representation learning. arXiv preprint urX1v:2002.07442, 2020.
[51] J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on information theory, 23(3)333-343, 1977.
12


Reviews:
[{"review_id": "szhhBHjqDO", "reviewer": "Reviewer_uPwZ", "summary": "This paper presents a novel approach for representing volumetric video using a dynamic codebook that incorporates the temporal correlation of features. This addresses the drawback of existing feature grid-based methods, which overlook this correlation.", "questions": "1. What specific improvements have been made in Sections 3.1 and 3.2 compared to existing works, like [16]?\n2. Why were certain methods not tested on the DyNeRF dataset?", "limitations": "Further research is needed to develop effective methodologies for dynamic codebooks. Although the paper enhances the expressive capabilities of the dynamic neural representation model, there is room for future investigation into optimizing training and rendering speed.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method uses a multidimensional feature space and a dynamic codebook to model the changing scene.\n2. The paper presents model compression techniques, including pruning and weight clustering, which reduce the size of the model.", "weaknesses": "1. The article's novelty is limited as the introduction of the dynamic codebook approach appears to be primarily focused on engineering aspects. Additionally, the results from ablation experiments suggest that this approach is not particularly effective, and although a large number of model parameters are introduced, there is only a marginal improvement in video quality.\n\n2. It is impractical to manually adjust dynamic codebook parameters (such as k) for different videos in real-world applications.\n\n3. The paper lacks comparison results regarding training and rendering time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions. We address the major concerns below:\n\n**Q1: Limited novelty and no apparent improvements compared to existing works.**\n\n**A1:** We would like to emphasize that we have two core contributions, which make our method distinct from previous works:\n- **Technical contributions.** We present a carefully-designed method for volumetric video compression. We build our method on feature plane instead of 3D feature grid, which is a more suitable representation for volumetric videos. We claim that directly applying compression methods for static scenes to dynamic scenes will result in considerable information loss. This is because they do not take into account the temporal variability characteristics of dynamic scenes, which is fundamentally different from static scenes. To this end, we designed a dynamic codebook compression method tailored to the characteristics of dynamic scenes. Our approach identifies areas that most require enhancement in each time fragment and incrementally supplements codes into dynamic codebook.\n- **Experimental contributions.** Empirically, we discovered that simply applying the codebook compression methods from static scenes to dynamic scenes results in a noticeable decline in quality in detailed regions, such as facial areas. To overcome this problem, we implemented the proposed dynamic codebook with thoughtful method design and engineering efforts, which achieved a high compression rate on two representative and challenging dynamic scene datasets (NHR and DyNeRF) while ensuring rendering quality comparable to state-of-the-art (SOTA) methods.\n\nWe believe that the contributions mentioned above will bring new insights to this field and benefit the community.\n\n**Q2: The results from ablation experiments suggest that this approach is not particularly effective.**\n\n**A2:** The dynamic codebook can compensate for the quality loss resulting from compression while requiring less storage. We have provided corresponding qualitative and quantitative analyses in Figure 4 and Table 3. The qualitative results clearly show that compression leads to significant quality loss in detailed areas such as the facial regions, while the dynamic codebook can improve the quality in these areas. From a quantitative perspective, the PSNR improvement brought by the dynamic codebook is not particularly large. We claim there are two reasons for this: 1. The purpose of our method is to compress while maintaining rendering quality, so the rendering quality of the baseline before compression can be considered an upper limit that we can hardly exceed. 2. Detailed areas occupy a small proportion of the image, while PSNR is averaged over all pixels, so the improvement in detailed areas is not very pronounced in terms of PSNR enhancement.\n\n**Q3: It is impractical to manually adjust the hyperparameters in real-world applications.**\n\n**A3:** Our method indeed introduces some new hyperparameters. However, there are some principles to follow. Here, we outline some guidelines for setting these hyperparameters:\n\n- **Size of the codebook ('k'):** The setting of the codebook size relates to the trade-off between storage size and rendering quality. For more content-rich scenarios like DyNeRF, a larger codebook is needed. On the other hand, for relatively simpler scenarios like NHR, a smaller codebook suffices.\n- **The ratio of codes to discard or retain during codebook compression:** We retain the top 30% of codes with the highest importance score contributions in all datasets. These contribute approximately 80% of the total importance score. Setting the threshold based on this ratio allows us to substantially reduce storage while trying to maintain the original rendering quality.\n\nWe designed corresponding ablation studies on NHR, and the results are as follows:\n\n|percent of retained code   |10   |20   |30   |\n|---------------------------|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.39|32.57|32.85|\n|PSNR (w/ dynamic codebook) |33.40|33.46|33.51|\n|final model size (MB)      |16.3 |16.5 |16.6 |\n\n|k                          |1024 |2048 |4096 |8192 |16384|\n|---------------------------|-----|-----|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.09|32.47|32.85|32.89|33.01|\n|PSNR (w/ dynamic codebook) |33.30|33.43|33.51|33.54|33.58|\n|final model size (MB)      |16.3 |16.4 |16.6 |17.1 |18.0 |\n\nThe results evidents that both codebook size and the retention ratio influence the storage size and rendering quality of our method. However, the impact is not substantial, indicating that our method is fairly robust to these hyperparameters.\n\nWe will design and conduct more comprehensive ablation studies, which will be incorporated into the revised paper.\n\n**Q4: The paper lacks comparison results regarding training and rendering time.**\n\n**A4:** In Section 4, we mentioned the details of training time: \"We train the model on a single NVIDIA A100 GPU, which takes about 1.5 / 4.3 hours for training and 1.0 / 1.7 hours for the construction of a dynamic codebook on NHR / DyNeRF datasets.\" The rendering time depends on factors such as the complexity of the scene and the resolution of the image. We tested the rendering speed of our method and other methods on NHR dataset at resolution of 512*384. The full comparison results are as follows:\n\n|Method             |NV |C-NeRF|D-NeRF|DyNeRF|DyMap|K-Planes|Ours|\n|-------------------|---|------|------|------|-----|--------|----|\n|Training time (hrs)|>20|>20   |>20   |>100  |16   |2       |2.5 |\n|Rendering time (ms)|73 |1969  |2303  |5195  |33   |384     |61  |\n\n**Q5: Why were certain methods not tested on the DyNeRF dataset?**\n\n**A5:** Based on the results reported in DyNeRF, the performance of Neural Volumes is quite poor. C-NeRF, as a follow-up to Neural Volumes, is also expected to perform poorly, and they both require substantial storage. As for DyMAP, we ran it ourselves and found that it does not work on the DyNeRF dataset."}, {"review_id": "XfLzjgNz5h", "reviewer": "Reviewer_bWLa", "summary": "The authors of this paper propose  a dynamic codebook, which optimizes away codes of low importance to rendering the scene, clusters 70% of the least importance codes and optimizes the remaining codes for every time fragment. The process the authors introduce are intuitive and each step furthers compression or works towards reducing the distortion introduced in the compression process.\n\nI have read and acknowledged the author's rebuttal and will remain with my generally positive assessment.", "questions": "1. In L169, the authors state that 99.9% of the importance is contributed by merely 10% of the voxels in the 2D case and that it is not as bad in the 3D case. Is there a citation or an empirical result the authors could share on this?\n\n2. How large are the codebooks for each of the time fragments? In L237, the authors state that the clustered codes are set to 4096 (NHR) and 16,384 for DyNeRF, but not how many codes are being optimized by the dynamic codebook for each fragment.\n\n3. How much experimentation did it take to discover the different hyperparameters for the dynamic codebook? Is it easy to extrapolate to other datasets? Do much more could be gained by even more optimal hyperparameters on a nerf by nerf basis?\n\n4. Any qualitative or user study on different nerfs? The quality difference seems minor, so it's not clear if it's visually preferred over other methods.\n\nTypo in Table 3:\nbasline => baseline", "limitations": "No concerns", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The introduction of a dynamic codebook is intuitive and handles a number of problems in the codebook creation process. First keeping highly important codes, clustering low important codes to a zero token, clustering remaining codes and then using gradient aware code optimization to further increase details for specific time fragments. \n\nThe authors demonstrate near state of the art PSNR on NHR and DyNeRF datasets ( < 0.25 dB PSNR difference) with substantially (more than 14x) smaller models.", "weaknesses": "As discussed by the authors, the dynamic codebook construction takes at least an hour for NHR and almost two hours for DyNeRF and substantially increases the total end to end time and rendering time.\n\nThe dynamic codebook introduces more hyperparameters to tune, what percentage of features to keep, drop, cluster and how many dynamic codewords get added. NHR and DyNeRF seem to differ substantially in these and it's unclear how much of this is based off of framerate, resolution, or complexity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions. We address the major concerns below:\n\n**Q1: The dynamic codebook construction increases the total end to end time and rendering time.**\n\n**A1:** Firstly, the dynamic codebook has a slight influence on the rendering time as the indexing process of the codebook is nearly instantaneous. Secondly, although the training time increases by 1-2 hours, the overall training time remains comparable to other methods based on feature grids, and is at least an order of magnitude faster than methods like DyNeRF. Moreover, the benefits are significant, especially in terms of storage efficiency improvement. This presents substantial application value in scenarios where high storage efficiency is required but sensitivity to training time is not a concern.\n\nWe also provide comparison results of training and rendering time with SOTA methods, where rendering speed is tested on NHR dataset at resolution of 512*384:\n\n|Method             |NV |C-NeRF|D-NeRF|DyNeRF|DyMap|K-Planes|Ours|\n|-------------------|---|------|------|------|-----|--------|----|\n|Training time (hrs)|>20|>20   |>20   |>100  |16   |2       |2.5 |\n|Rendering time (ms)|73 |1969  |2303  |5195  |33   |384     |61  |\n\n**Q2: The dynamic codebook introduces more hyperparameters, and it's unclear how to adjust them.**\n\n**A2:** Our method indeed introduces some new hyperparameters. However, there are some principles to follow. Here, we outline some guidelines for setting these hyperparameters:\n\n- **Size of the codebook ('k'):** The setting of the codebook size relates to the trade-off between storage size and rendering quality. For more content-rich scenarios like DyNeRF, a larger codebook is needed. On the other hand, for relatively simpler scenarios like NHR, a smaller codebook suffices.\n- **The ratio of codes to discard or retain during codebook compression:** We retain the top 30% of codes with the highest importance score contributions in all datasets. These contribute approximately 80% of the total importance score. Setting the threshold based on this ratio allows us to substantially reduce storage while trying to maintain the original rendering quality.\n\nWe designed corresponding ablation studies on NHR, and the results are as follows:\n\n|percent of retained code   |10   |20   |30   |\n|---------------------------|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.39|32.57|32.85|\n|PSNR (w/ dynamic codebook) |33.40|33.46|33.51|\n|final model size (MB)      |16.3 |16.5 |16.6 |\n\n|k                          |1024 |2048 |4096 |8192 |16384|\n|---------------------------|-----|-----|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.09|32.47|32.85|32.89|33.01|\n|PSNR (w/ dynamic codebook) |33.30|33.43|33.51|33.54|33.58|\n|final model size (MB)      |16.3 |16.4 |16.6 |17.1 |18.0 |\n\nThe results evidents that both codebook size and the retention ratio influence the storage size and rendering quality of our method. However, the impact is not substantial, indicating that our method is fairly robust to these hyperparameters.\n\nWe will design and conduct more comprehensive ablation studies, which will be incorporated into the revised paper.\n\n**Q3: The basis of the distribution of importance score.**\n\n**A3:** The statement \"when representing 3D scenes using feature grids, 99.9% of the importance is contributed by merely 10% of the voxels\" originates from reference [16], which focused on compressing static scenes. In our paper, we claim that \"the situation is not as extreme when we use 2D planes to represent volumetric videos\" primarily for two reasons: 1. Volumetric videos have much richer content compared to static scenes. 2. The representation using feature planes is somewhat more compact than using a 3D feature grid. Hence, when we use feature planes to represent volumetric videos, the situation isn't as extreme as when using feature grids for static scenes. Moreover, our empirical findings support this claim--we discovered that 99.9% of the importance is contributed by approximately 75% of the voxels and 80% of the importance is contributed by approximately 30% of the voxels.\n\n**Q4: How many codes are being optimized by the dynamic codebook for each fragment?**\n\n**A4:** In section 4, we mentioned that \"we optimize 1000 appearance codes and 5000 density codes for each time fragment.\"\n\n**Q5: Any qualitative or user study on different nerfs? The quality difference seems minor, so it's not clear if it's visually preferred over other methods.**\n\n**A5:** We provide qualitative comparisons with other methods in Figure 2 and Figure 3. The quality difference between our approach and the state-of-the-art methods is indeed minor. However, this is not conflict with our claimed contribution: we can achieve rendering quality comparable to state-of-the-art methods, but with a significant boost in storage efficiency. The improvement of storage efficiency can be clearly seen from the quantitative results in Table 1 and Table 2."}, {"review_id": "sMaDNxCDBG", "reviewer": "Reviewer_ZNi1", "summary": "The proposed method applies the codebook technique to explicit feature plane representation and a dynamic codebook is further proposed for dynamic scenes. Experimental results demonstrate the good performance of the proposed methods.", "questions": "1. The proposed dynamic codebook should be applicable to other explicit representations, e.g., feature grids. The authors may explain the reason why they did not conduct related experiments.\n", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method significantly reduces the model size while keeping a similar render quality with K-Planes [35] from both qualitative metrics and visualization.\n2. The proposed dynamic codebook is easy to follow and the problem addressed is important and interesting.\n", "weaknesses": "1. Considering that previous work VQRF has introduced codebook to feature grid representation for reducing the model size with an excellent compression ratio, the proposed method here that introduces codebook to feature plane representation is reasonable but not of sufficient novelty.  \n2. The main contribution of model size reduction comes from the codebook technique. The efficiency of the proposed dynamic codebook is somewhat limited.\n3. There is no experimental comparison of the time cost. The influence of the optimization process caused by the dynamic codebook is unclear.\n4. There lack some explanation and evaluation about some hyperparameters and settings. For example, In Sec 4, Line 235: \"We then retain the top 30% of codes with the highest importance score contributions and cluster the remaining codes into k codes\". The reason for the setting 30% and the influence of k can be provided for better demonstrating the efficiency of the proposed strategy.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions. We address the major concerns below:\n\n**Q1: Introducing codebook to feature plane representation is reasonable but not of sufficient novelty.**\n\n**A1:** We would like to emphasize that we have two core contributions, which make our method distinct from previous works:\n\n- **Technical contributions.** We present a carefully-designed method for volumetric video compression. We claim that directly applying compression methods for static scenes to dynamic scenes will result in considerable information loss. This is because they do not take into account the temporal variability characteristics of dynamic scenes, which is fundamentally different from static scenes. To this end, we designed a dynamic codebook compression method tailored to the characteristics of dynamic scenes. Our approach identifies areas that most require enhancement in each time fragment and incrementally supplements codes into dynamic codebook.\n- **Experimental contributions.** Empirically, we discovered that simply applying the codebook compression methods from static scenes to dynamic scenes results in a noticeable decline in quality in detailed regions, such as facial areas. To overcome this problem, we implemented the proposed dynamic codebook with thoughtful method design and engineering efforts, which achieved a high compression rate on two representative and challenging dynamic scene datasets (NHR and DyNeRF) while ensuring rendering quality comparable to state-of-the-art (SOTA) methods.\n\nWe believe that the contributions mentioned above will bring new insights to this field and benefit the community.\n\n**Q2: The effectiveness of the proposed dynamic codebook is somewhat limited.**\n\n**A2:** The dynamic codebook can compensate for the quality loss resulting from compression while requiring less storage. We have provided corresponding qualitative and quantitative analyses in Figure 4 and Table 3. The qualitative results clearly show that compression leads to significant quality loss in detailed areas such as the face, while the dynamic codebook can improve the quality in these areas. From a quantitative perspective, the PSNR improvement brought by the dynamic codebook is not particularly large. We claim there are two reasons for this: 1. The purpose of our method is to compress while maintaining rendering quality, so the rendering quality of the baseline before compression can be considered an upper limit that we can hardly exceed. 2. Detailed areas occupy a smaller proportion of the image, while PSNR is averaged over all pixels when calculated, so the improvement in the quality of detailed areas is not very pronounced in terms of PSNR enhancement.\n\n**Q3: The influence of the optimization process caused by the dynamic codebook is unclear.**\n\n**A3:** In Section 4, we mentioned the details of training time: \"We train the model on a single NVIDIA A100 GPU, which takes about 1.5 / 4.3 hours for training and 1.0 / 1.7 hours for the construction of a dynamic codebook on NHR / DyNeRF datasets.\"\n\n**Q4: There lack some explanation and evaluation about some hyperparameters and settings.**\n\n**A4:** Our method indeed introduces some new hyperparameters. However, there are some principles to follow. Here, we outline some guidelines for setting these hyperparameters:\n\n- **Size of the codebook ('k'):** The setting of the codebook size relates to the trade-off between storage size and rendering quality. For more content-rich scenarios like DyNeRF, a larger codebook is needed. On the other hand, for relatively simpler scenarios like NHR, a smaller codebook suffices.\n- **The ratio of codes to discard or retain during codebook compression:** We retain the top 30% of codes with the highest importance score contributions in all datasets. These contribute approximately 80% of the total importance score. Setting the threshold based on this ratio allows us to substantially reduce storage while trying to maintain the original rendering quality.\n\nWe designed corresponding ablation studies on NHR, and the results are as follows:\n\n|percent of retained code   |10   |20   |30   |\n|---------------------------|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.39|32.57|32.85|\n|PSNR (w/ dynamic codebook) |33.40|33.46|33.51|\n|final model size (MB)      |16.3 |16.5 |16.6 |\n\n|k                          |1024 |2048 |4096 |8192 |16384|\n|---------------------------|-----|-----|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.09|32.47|32.85|32.89|33.01|\n|PSNR (w/ dynamic codebook) |33.30|33.43|33.51|33.54|33.58|\n|final model size (MB)      |16.3 |16.4 |16.6 |17.1 |18.0 |\n\nThe results evidents that both codebook size and the retention ratio influence the storage size and rendering quality of our method. However, the impact is not substantial, indicating that our method is fairly robust to these hyperparameters.\n\nWe will design and conduct more comprehensive ablation studies, which will be incorporated into the revised paper.\n\n**Q5: Why not conduct experiments to other explicit representations, e.g., feature grids?**\n\n**A5:** 4D feature grid requires too large memory, which is infeasible, and there is currently no such work. At present, there are only two types of dynamic scene reconstruction/view synthesis work based on feature grids: 1. those based on feature planes (2D grid) 2. those using a canonical 3D feature grid in combination with deformation field.\n\nOur work focuses on compression based on the first type of method. The second type of method, is actually a technique for compressing temporal information with deformation. However, it is not capable of dealing with some complex dynamic scenes (some complex changes or movements cannot be well modeled using deformation)."}, {"review_id": "7A7pwd8O0I", "reviewer": "Reviewer_Jvh8", "summary": "The paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. The authors proposed a novel neural representation called the dynamic codebook, which aims to reduce the spatial and temporal redundancy of feature grids inherent to scenes due to self-similarity. It achieves this by merging similar features for model compression and compensating for potential declines in rendering quality through a set of dynamic codes. The experimental results demonstrate that the dynamic codebook method achieves state-of-the-art rendering quality while achieving higher storage efficiency.\n\nThe contributions of the paper are twofold: \n1. This paper proposes the dynamic codebook representation, which effectively reduces model size by minimizing feature redundancy in space and time while maintaining rendering quality in dynamic detailed regions. \n2. Experimental results show that the proposed approach achieves rendering quality comparable to state-of-the-art methods while significantly improving storage efficiency.\n", "questions": "1. This paper claims that “…which is of low model size but extremely slow training speed.” In Table 1, but there is no mention of the training time for the models. It would be helpful to provide insights into the time required for training the model and how it compares to other models.\n2. Regarding the comparison of model size, I noticed that this paper uses “post-processing” including quantization and entropy coding techniques. According to my understanding, this techniques can affect the model size, whether other comparison models use this techniques? If not, it would be beneficial to discuss the potential outcomes if these post-processing steps were not applied. \n3. In line 237, this paper uses different k for different dataset. Is there any particular consideration followed while setting the value of k?\nIn reference to the dynamic codebook's size, it would be useful to explore what constitutes an appropriate size. The codebook-based compression method mentioned can significantly reduce model storage but may result in a decline in rendering quality. Thus, it would be meaningful to investigate how variations in k affect the same dataset.\n4. The model size is composed of the quantized dynamic codebook, index planes as well as MLP parameters with entropy encoding, so the bit allocation of different components of the model should be provided, which can help the understanding of the proposed methodology.\n\n", "limitations": "The authors have partially addressed the limitations of existing work, though there is space for improvement (see the section Weaknesses and Questions).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel approach that use the dynamic codebook to reduce the spatial and temporal redundancy.\n\nThe paper is well-structured, with clear section headings, and visual aids such as figures are effectively utilized to illustrate key points or enhance comprehension.\n", "weaknesses": "Experimental details and results seem incompletely reported, so I have some confusion about this work. I defer some of my issues in \"Questions\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions. We address the major concerns below:\n\n**Q1: It would be helpful to provide insights into the time required for training the model and how it compares to other models.**\n\n**A1:** In Section 4, we mentioned the details of training time: \"We train the model on a single NVIDIA A100 GPU, which takes about 1.5 / 4.3 hours for training and 1.0 / 1.7 hours for the construction of a dynamic codebook on NHR / DyNeRF datasets.\" And we provide comparison to other models here.\n\n|Method             |NV |C-NeRF|D-NeRF|DyNeRF|DyMap|K-Planes|Ours|\n|-------------------|---|------|------|------|-----|--------|----|\n|Training time (hrs)|>20|>20   |>20   |>100  |16   |2       |2.5 |\n\n**Q2: Analysis of post-processing and allocation of model storage.**\n\n**A2:** To more clearly analyse the model storage in our method, we examined the allocation of model storage size at each step on the NHR dataset, in accordance with the sequence of method implementation. Given that our baseline has fewer MLP parameters, we did not further compress the MLP parameters. The primary storage of the initial model (1) lies in the feature planes. We compressed these using a representation of codebook + index planes (2), which significantly reduces storage. We then utilized quantization (3) and entropy encoding (4) techniques for post-processing, achieving further reductions. The results at this stage correspond to the \"compression w/o DC\" in Table 3 of our paper. Finally, we employed a dynamic codebook (5) to introduce additional storage for codebook and index planes, enhancing the rendering quality. We will further refine these experiments and analysis and incorporate them into the revised paper.\n\n|   |feature planes|codebook|index planes|MLP |Sum   |\n|---|--------------|--------|------------|----|------|\n|1  |119.92        |0       |0           |0.10|120.02|\n|2  |0             |29.04   |0.08        |0.10|29.22 |\n|3  |0             |7.52    |0.08        |0.10|7.70  |\n|4  |0             |5.51    |0.08        |0.10|5.69  |\n|5  |0             |15.50   |0.98        |0.10|16.58 |\n\n**Q3: Analysis of hyperparameters such as k (codebook size).**\n\n**A3:** Our method introduces some new hyperparameters such as k. However, there are some principles to follow. Here, we outline some guidelines for setting these hyperparameters:\n\n- **Size of the codebook ('k'):** The setting of the codebook size relates to the trade-off between storage size and rendering quality. For more content-rich scenarios like DyNeRF, a larger codebook is needed. On the other hand, for relatively simpler scenarios like NHR, a smaller codebook suffices.\n- **The ratio of codes to discard or retain during codebook compression:** We retain the top 30% of codes with the highest importance score contributions in all datasets. These contribute approximately 80% of the total importance score. Setting the threshold based on this ratio allows us to substantially reduce storage while trying to maintain the original rendering quality.\n\nWe designed corresponding ablation studies on NHR, and the results are as follows:\n\n|percent of retained code   |10   |20   |30   |\n|---------------------------|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.39|32.57|32.85|\n|PSNR (w/ dynamic codebook) |33.40|33.46|33.51|\n|final model size (MB)      |16.3 |16.5 |16.6 |\n\n|k                          |1024 |2048 |4096 |8192 |16384|\n|---------------------------|-----|-----|-----|-----|-----|\n|PSNR (w/o dynamic codebook)|32.09|32.47|32.85|32.89|33.01|\n|PSNR (w/ dynamic codebook) |33.30|33.43|33.51|33.54|33.58|\n|final model size (MB)      |16.3 |16.4 |16.6 |17.1 |18.0 |\n\nThe results evidents that both codebook size and the retention ratio influence the storage size and rendering quality of our method. However, the impact is not substantial, indicating that our method is fairly robust to these hyperparameters.\n\nWe will design and conduct more comprehensive ablation studies, which will be incorporated into the revised paper."}, {"review_id": "YYUt0IDGt1", "reviewer": "Reviewer_u1Fc", "summary": "A method for compressing Volumetric Videos is presented in this work. It is based on NeRF with a factored multi feature plane representation. The features of the model are compressed in two stages. In the first, a codebook for features is constructed based on the average contribution a feature has to the total reconstruction. Features are weighted based on their score contribution to the NeRF integral. Ones with low score contributions are merged into a zero code, the top 30% of codes are retained, and the rest are clustered using an exponential-moving average algorithm. In the second compression stage,  a small number of features per temporal segment are selected based on the accumulated gradients of back propagation on the codebook compressed model. These features are then optimized and added to the codebook, dynamically growing the codebook with time. The resulting method produces reconstruction quality and compression ratios comparable to SOTA while requiring only hours instead of days to train. ", "questions": "- Adding some implementation details where the method is described would help clarify thing sooner. For example, how accumulated score contributions are used to select the three categories, and how a temporal fragment is defined. \n- The clustering algorithm in Sec B or supplementary could be clearer and added to the main text. ie. is this performed while training the features as in VQ-VAE, or is performed as a post-processing?\n", "limitations": "Authors do not address limitations.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A simple method that achieves SOTA accuracy and compression performance but is much more efficient. ", "weaknesses": "- Writing could be a bit clearer\n- Only evaluated on two scenes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful suggestions. We address the major concerns below:\n\n**Q1: Writing could be a bit clearer.**\n\n**A1:** Thanks for your suggestion. We will improve our writing and include more detailed descriptions of our methods in the revised paper.\n\n**Q2: Only evaluated on two scenes.**\n\n**A2:**: We conducted experiments on two datasets, totaling 10 scenes (4 from the NHR dataset and 6 from the DyNeRF dataset). We have provided the results for each scene in the supplementary materials.\n\n**Q3: Adding some implementation details where the method is described would help clarify thing sooner.**\n\n**A3:** We appreciate your suggestion. In the revised paper, we will include some of the crucial details in the method section, making the paper easier to follow.\n\n**Q4: The clustering algorithm in Sec B or supplementary could be clearer and added to the main text. ie. is this performed while training the features as in VQ-VAE, or is performed as a post-processing?**\n\n**A4:** The clustering algorithm is performed as a post-processing. We will make this part clearer and add them to the main text in the revised paper."}]

Author rebuttal:

Meta-review: The manuscript is focused on minimizing the size of a NeRF model represented by k-planes and has garnered mixed reviews. Three out of five reviewers are supportive, but two have raised concerns about the originality of using codebooks for compression, the necessity for manual hyperparameter tuning, and the absence of a comparison between training and rendering times. Notably, one reviewer found the experimental results ambiguous even after the authors' rebuttal. The AC concurs that the manuscript necessitates significant revisions, especially in terms of training and rendering times (since trading time for space is typically deemed suboptimal, making the inclusion of a time comparison crucial). After consultation with the SAC, the decision is to recommend the manuscript for acceptance. Nevertheless, the AC strongly advises the authors to undertake the necessary revisions for the camera-ready version of the paper.
Decision: Accept (poster)