Title: Compact Neural Volumetric Video Representations with Dynamic Codebooks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computer Vision

3D Vision

Volumetric Video
Abstract: This paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. Some recent feature grid-based methods have shown superior performance of fast learning implicit neural representations from input 2D images. However, such explicit representations easily lead to large model sizes when modeling dynamic scenes. To solve this problem, our key idea is reducing the spatial and temporal redundancy of feature grids, which intrinsically exist due to the self-similarity of scenes. To this end, we propose a novel neural representation, named dynamic codebook, which first merges similar features for the model compression and then compensates for the potential decline in rendering quality by a set of dynamic codes. Experiments on the NHR and DyNeRF datasets demonstrate that the proposed approach achieves state-of-the-art rendering quality, while being able to achieve more storage efficiency. The source code is available at https://github.com/zju3dv/compact_vv.

Compact Neural Volumetric Video Representations with Dynamic Codebooks

Abstract

This paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. Some recent feature grid-based methods have shown superior performance of fast learning implicit neural representations from input 2D images. However, such explicit representations easily lead to large model sizes when modeling dynamic scenes. To solve this problem, our key idea is reducing the spatial and temporal redundancy of feature grids, which intrinsically exist due to the self-similarity of scenes. To this end, we propose a novel neural representation, named dynamic codebook, which first merges similar features for the model compression and then compensates for the potential decline in rendering quality by a set of dynamic codes. Experiments on the NHR and DyNeRF datasets demonstrate that the proposed approach achieves state-of-the-art rendering quality, while being able to achieve more storage efficiency. The source code is available at https://github.com/zju3dv/compact_vv .

1 Introduction

Volumetric videos record the content of dynamic 3D scenes and allow users to watch captured scenes from arbitrary viewpoints, which has a wide range of applications, such as virtual reality, augmented reality, telepresence, and so on. Traditional methods typically represent volumetric videos as a sequence of textured meshes, which are reconstructed by the multi-view stereo [36, 37, 12, 15] or depth fusion [10, 27]. Such techniques require sophisticated hardware, e.g. , multi-view RGB-D sensors, to achieve high reconstruction accuracy, which is expensive and limits their application scenarios.

Recently, implicit neural representations [24, 1, 49, 2] have emerged as a promising strategy to recover 3D scenes from only 2D images. As a representative work, neural radiance field (NeRF) [24] employs an MLP network to predict the density and color for any 3D point, and effectively learns the model parameters from 2D images through the volumetric rendering technique. DyNeRF [17] extends NeRF to dynamic scenes by introducing time-varying latent codes as an additional input of the MLP network, thereby enabling the model to encode the per-frame content of the scene. Although these methods achieve impressive rendering quality, they are extremely slow to train, especially in dynamic scenes. For example, DyNeRF requires over 1000 GPU hours to learn a 10-second volumetric video.

To overcome this problem, some methods exploit explicit representations, such as feature volumes [8, 50, 20] or feature planes [35, 4, 38], to accelerate the training process. They typically store learnable feature vectors at explicit structures and use the interpolation technique to assign a feature

vector to arbitrary scene coordinate, which is then fed into a lightweight MLP network to predict the density and color. By decreasing the number of parameters in the MLP network, these methods achieve a significant speedup in training. However, the explicit representations can easily take up large storage, which is hard to scale to dynamic 3D scenes.

Some methods [42, 16] reduce storage redundancy based on the internal similarity of scenes, achieving satisfactory compression results on static scenes. However, they do not take into account the characteristics of dynamic scenes. Experiments show that directly applying these methods to dynamic scenes results in non-negligible performance degradation. For dynamic scenes, we observe that the redundancy of features learned by existing methods [8, 50, 20, 35, 4, 38] is not only reflected in spatial correlations, but more importantly, dynamic scenes are essentially produced by the movement and changes of static scenes, thereby intrinsically having a strong temporal correlation. If this prior knowledge is not adequately utilized, the model will inevitably contain a considerable amount of unnecessary repetitive storage.

Motivated by these observations, we propose a novel representation, named dynamic codebook , for efficient and compact modeling of dynamic 3D scenes. We first employ a compact codebook to accomplish compression, then incrementally update the codebook to compensate for the potential decline in rendering quality of dynamic detailed regions caused by compression. The representation of codebook can effectively mitigate the issue of code redundancy in time and space, thereby significantly reducing unnecessary storage. However, this representation might cause the codes of some regions to be less precise, which will decrease the rendering quality, especially in dynamic detailed regions. The spatial distribution of these regions may vary significantly over time. Therefore, we separately identify the most necessary parts of the codes to optimize for each small time fragment and incrementally add them to the codebook to improve the rendering quality of these regions. Since these regions typically account for a small proportion of the scene, the new codes will not occupy much storage. Therefore, our proposed dynamic codebook based strategy can achieve a high compression rate while maintaining high rendering quality.

We evaluate our approach on the NHR [48] and DyNeRF [17] datasets, which are widely-used benchmarks for dynamic view synthesis. Experimental results demonstrate that the proposed approach could achieve rendering quality comparable to the state-of-the-art methods while achieving much higher storage efficiency.

In summary, our contributions are as follows: (1) We propose a novel representation called dynamic codebook, which effectively decreases the model size by reducing the feature redundancy in time and space while maintaining the rendering quality of dynamic detailed regions. (2) Experiments demonstrate that our approach achieves rendering quality on par with state-of-the-art methods, while significantly improving storage efficiency.

2 Related works

View synthesis of static scenes. Traditional methods use mesh to represent scene geometry and texture mapping to depict the scene's appearance, which can be rendered from arbitrary viewpoints. However, the reconstruction process based on this representation is typically not end-to-end, leading to a lack of photo-realism of the rendering results. [24] proposes a continuous representation that models the density and color of static scenes with MLP networks, which can be used for high-quality free view synthesis using volume rendering technique. [1] utilizes multi-scale representation to address the inherent aliasing of NeRF, and improve the ability to represent fine details. [49, 2] enable modeling of unbounded scenes with complex backgrounds by the design of coordinate mapping.

The pure MLP-based representation is much more compact compared to traditional representations, however, a large amount of points sampling is required during rendering, and each point will be fed into the MLP, resulting in very low efficiency. [21] utilizes a sparse occupancy grid to avoid sampling points in empty space to accelerate the training and rendering speed. [32] replaces the large MLP in NeRF with thousands of tiny MLPs. [9, 41] directly adopt the explicit volumetric representation during training and utilize coarse to fine strategy to achieve high resolution. [25] employs multiresolution hash table for encoding and lightweight MLPs as decoders, achieving unprecedentedly fast training. [3] further extends anti-aliasing to grid-based representation by multisampling and prefiltering. [33] introduces a novel piecewise-projective contraction function to model large unbounded scenes with feature grids, and enables real-time view synthesis.

Volumetric video. Following NeRF's considerable success in modeling static scenes, some studies [19, 30, 17, 31, 35, 45] have sought to extend it to dynamic scenes. There are primarily two approaches to this. The first approach is to directly model temporal variations by directly modifying the input format of MLP to enable view synthesis not only at any viewpoint but also at any given time. [18] directly input the timestamp to MLP for space-time view synthesis. [17] utilizes time-varying latent codes as the additional input to MLP, the latent codes can be learned to model the per-frame content of dynamic scenes. The training of this approach is typically challenging, either requiring some additional supervision such as monocular depth or optical flow, or taking much time to converge. The second approach involves using NeRF to model a canonical space and then learning the deformation from any moment to this canonical space. [31] employs another MLP to model the deformation at each timestamp to this canonical space. [28] utilizes latent codes to model deformation. However, deformation struggles to model complex temporal changes, such as large movements, or the emergence of new content, such as fluid mixing Moreover, these two kinds of approaches both encounter issues similar to those faced by NeRF in static scenes, namely, low training and rendering efficiency.

Methods based on explicit or hybrid representations have also been studied and explored for modeling volumetric videos. [22] encodes multi-view information at arbitrary timestamp and uses a 3D deconvolutional network to produce volumes of density and color. [47] uses 3D CNN to predict feature volumes instead of the explicit volumes, and additionally adopts an MLP to predict the radiance fields. [8, 20, 50] employ a feature grid to model the canonical space, which enables efficient training and rendering. [35, 4, 38] represent volumetric videos using multiple feature planes, offering a more compact representation compared to the feature grids. [45] explicitly partitions the scene into dynamic and static regions using proposed variation field, and models them separately using TensoRF [5], allowing for remarkably fast training. [40] divides scenes into static, deforming and new regions, and utilizes a sliding window-based hybrid representation for efficient scene modeling. [46] explicitly models the residual information between adjacent timestamps in the spatial-temporal feature space and employs a compact motion grid along with a residual feature grid to exploit inter-frame feature similarities.

Compression of implicit neural representations. Methods based on feature grids and feature planes can provide significant efficiency improvements, but they also introduce substantial storage requirements. To alleviate this issue, [7] proposes to use pruning strategy to eliminate unnecessary parameters in the feature grid. [5] decomposes the 3D grids into lower dimension. It utilizes VM decomposition to decompose the 3D grid into multiple 2D matrices and even employs CP decomposition to decompose it into multiple 1D vectors. While CP decomposition significantly improves storage efficiency, it results in a non-negligible performance loss. [43] implements compression that can be dynamically adjusted by the proposed rank-residual learning strategy. Vector quantization [11, 13] is a classic compression technique that is widely applied in 2D image and video compression tasks [26, 6, 39]. The main idea of which is to merge similar codes into one through clustering, thereby reducing storage redundancy. Recently, VQAD [42] and VQRF [16] use vector quantization to achieve compression of static 3D scenes. VQAD learns a codebook while training the model, while VQRF implements compression in a post-processing manner. [14] utilizes Fourier transform for compression, which is able to capture high-frequency details while remaining compact. [34] proposes to use wavelet coefficients to improve parameter sparsity.

3 Method

Given multi-view videos captured by synchronized and calibrated cameras, our goal is to generate a volumetric video that requires low disk storage while retaining the capability of high-fidelity rendering. Our approach is illustrated in Fig. 1. We represent the volumetric video by multiple feature planes, which can be learned from multi-view videos using the volume rendering technique (Sec. 3.1). To enhance the storage efficiency of the model, we propose a codebook-based representation, which aims to reduce the redundancy of the feature plane, thereby achieving model compression (Sec. 3.2). To compensate for the loss of rendering quality in certain areas due to compression, we extend the codebook to a dynamic codebook. This significant enhancement in rendering quality introduces only a minimal amount of additional storage (Sec. 3.3).

1

1

1

2

8

8

9

2

3

4

5

6

7

8

9

. . .

index plane

codebook

(a) Original feature plane

(b) Feature plane reformulation

2

2

3

4

5

6

7

9

9

index plane

(c) Codebook compression

2

2

3

4

10

7

9

9

2

2

3

5

6

9

9

3

4

5

6

7

9

9

11

12

13

n-1

. . .

n

. . .

dynamic codebook

updated index planes for all time fragments

0

t

10

11

n-1

n

(d) Construction of dynamic codebook with incrementally added codes

. . .

2

9

codebook

. . .

X

3

7

. . .

3

7

. . .

X

1

8

2

9

X

3

7

. . .

X

12

. . .

3.1 Learning volumetric videos with feature planes

Inspired by [35, 4, 38], we model the dynamic scene by six feature planes, including three spatial planes denoted as P xy , P xz , and P yz and three spatial-temporal planes denoted as P xt , P yt , and P zt . Given a 3D coordinate x = ( x, y, z ) and time t , we obtain six feature vectors { f ( x , t ) i } by normalizing ( x , t ) according to the resolution of feature planes and projecting onto six planes:

where π i denotes projection function to i -th feature plane, and ψ denotes bilinear interpolation. We then calculate the feature vector of x by:

where ⊙ is the Hadamard product. Then f ( x , t ) is fed into an MLP network to predict the density σ ( x , t ) and color c ( x , t ) .

Following [24], we adopt volume rendering to optimize the feature planes as well as the MLP parameters. Specifically, to render a pixel p at time t , we first sample a set of 3D coordinates { x i } along the ray r passing through p , and then calculate the accumulated color along the ray:

where δ i = || x i +1 -x i || 2 is the distance between adjacent sampled points, and T i = exp( -∑ i -1 j =1 σ j δ j ) denotes the accumulated transmittance along the ray.

Then we optimize the model with L2 rendering loss with the supervision of ground truth pixel colors. Moreover, we use the standard regularization terms, including L1 norm loss and TV (total variation) loss following [5]. Please refer to supplementary materials for more details.

3.2 Compression of model

Though volumetric videos based on feature plane representations allow for efficient training and rendering, they suffer from a substantial model size, which in practice can reach approximately 30 -50 MBper second. This poses a significant challenge for the storage and dissemination of volumetric videos. In this section, we demonstrate how to achieve model compression using a codebook-based representation.

Previous works in static scene reconstruction and view synthesis have demonstrated that the usage of feature grids or feature planes introduces significant spatial redundancy. They achieve compression by reducing spatial redundancy through CP decomposition [5, 43], vector quantization [42, 16], Fourier / Wavelet transformations[14, 34], and so on. For volumetric videos, redundancy exists not only spatially, but also temporally. The usage of multiple 2D planes has to some extent mitigated the spatial redundancy. However, spatial redundancy still exists, and more importantly, there is non-negligible temporal redundancy.

As stated by [16], when representing 3D scenes using feature grids, 99 . 9% of the importance is contributed by merely 10% of the voxels. While the situation is not as extreme when we use 2D planes to represent volumetric videos, a similar pattern can still be observed. Hence, we can apply this principle to reduce the redundancy in the feature planes and achieve compression.

To facilitate the subsequent process, we initially convert the model into a codebook format, as illustrated in Fig. 1b. Specifically, we flatten all features on feature planes into a codebook, after which we convert the feature planes into index planes. In other words, we store the index mapping to the codebook at each grid point of the planes, instead of directly storing features.

In order to perform model compression based on the principle discussed above, we first calculate the importance score for each code in the codebook, according to its contribution to the rendering weight in the volume rendering process. Specifically, we randomly sample rays among all camera views and timestamps to perform volume rendering. For each sampled point p i on each ray, we record the volume rendering weights T i α i of p i , we also record the corresponding codes as well as their weights when performing trilinear interpolation at point p i . We calculate the importance score of each code by accumulating the rendering weights of all points that are mapped to it:

where N c denotes the set of points that are mapped to code c , and w ci denotes the trilinear interpolation weight that code c contribute to point p i . To reduce the storage, we first discard the less important codes by merging a portion of the codes with the lowest importance scores into a single one. For codes with medium importance scores, we cluster them into a smaller number of codes using a clustering method, as illustrated in Fig. 1c. The clustering is performed in an optimization manner, following VQ-VAE [44]. Specifically, we randomly initialize n codes, where n is much smaller than the number of the codes to be clustered. Then we use exponential moving averages (EMA) to update the codes iteratively, please refer to the supplementary materials for more details. We retain the codes with the highest importance scores as they are. After completing the merging and clustering operations, we update the indices on the index planes for each time fragment accordingly.

3.3 Dynamic codebook

The codebook-based compression method can significantly reduce model storage but inevitably leads to a decrement in rendering quality. We observe that the rendering of detailed regions will degrade after compression. The reason lies in the fact that these regions require particularly high precision for codes. The previously mentioned clustering strategy may reduce the accuracy of some of these codes. Although the importance score can measure the contribution of each code during the rendering process, it does not distinguish well between its contribution to detailed and non-detailed regions. Although these detailed regions typically occupy a small proportion of the scenes, their

spatial distribution may vary over time, making it difficult to enhance the rendering quality by directly optimizing a small subset of codes.

To solve the problem, we first divide the temporal dimension into numerous fragments. Our goal is to adaptively identify and optimize the parts most in need of enhancement within each time fragment. These are the portions where the shared spatial feature plane provides relatively poorer representation during that time fragment. Since these parts constitute a smaller proportion, they do not lead to a substantial storage increase.

To this end, we leverage the backpropagation gradient to pinpoint the top k codes demanding the most optimization within each time fragment. Specifically, for each time fragment, we run the forward and backpropagation steps in the training process without optimization and accumulate the gradients of each code for several iterations. Then we simply select the k codes with the highest accumulated gradients and optimize them for this time fragment. We repeat this process for each time fragment, and the optimized codes are independent of all time fragments.

Thanks to the codebook representation, we can append the optimized codes for each time fragment into the codebook. Additionally, we store an independent index plane for this time fragment and update the corresponding indices within it, as shown in Fig. 1d. We term the final codebook as dynamic codebook because it is constructed incrementally by optimizing for each time fragment.

The index planes store integer indices, thus occupying minimal storage space. Furthermore, as each time fragment only requires optimization of a small number of codes, the dynamic codebook does not introduce large additional storage. However, the design of dynamic codebook could effectively enhance the rendering quality of detailed regions.

Post-processing. For further compression, we apply post-processing steps to the codebook. We use uniform weight quantization to quantize the codebook. Then we encode all components, including the quantized dynamic codebook, index planes as well as MLP parameters with entropy encoding [51].

4 Implementation details

Network architecture. We set the resolution of the feature planes in the spatial dimension as 256 on NHR and 640 on DyNeRF dataset, and the one in the temporal dimension is set as 100 on NHR and 150 on DyNeRF dataset. We model density and appearance with two individual feature planes with the number of feature channels as 16 and 48 respectively. For appearance feature planes, our model also includes multi-scale planes in addition to the six feature planes at the highest resolution to encourage spatial smoothness and coherence, following [35]. For forward-facing scenes in the DyNeRF dataset, we apply NDC transformation that bounds the scene in a perspective frustum.

Construction of dynamic codebook. We construct a separate codebook for both density and appearance. During the compression step, we discard the portion of codes that only accumulate an importance score contribution of 0 . 001% , by merging them into a simple zero code. We then retain the top 30% of codes with the highest importance score contributions and cluster the remaining codes into k codes. We set k as 4096 for NHR dataset and 16384 for DyNeRF dataset. We divide each scene on NHR / DyNeRF into 100 / 60 time fragments, and optimize 1000 appearance codes and 5000 density codes for each time fragment.

Training. We implement our method with PyTorch [29]. We set the learning rate as 2e-3 for MLP parameters and 0.03 for feature planes, and use Adam optimizer to train the network with batches of 4096 rays. We train the model on one single NVIDIA A100 GPU, which takes about 1.5 / 4.3 hours for training and 1.0 / 1.7 hours for the construction of dynamic codebook on NHR / DyNeRF datasets.

5 Experiments

5.1 Datasets

To evaluate the performance of our approach, we conduct experiments on NHR [48] and DyNeRF [17] datasets. NHR contains four videos recorded at 30fps that capture an athlete performing large

S.S.

S.S.

S.S.

B

8

B

movements in a bounded scene with simple backgrounds. Following the setting of DyMap [30], we select 100 frames from each video and use 90 percent of camera views for training and the rest for testing. DyNeRF contains six 10 videos recorded at 30fps that capture a person cooking in a kitchen by 15-20 cameras from a range of forward-facing view directions. We use all 300 frames and follow the same training and testing split as in [17].

5.2 Comparison with the state-of-the-art methods

Baseline methods. For experiments on NHR dataset, we compare with (1) Neural Volumes (NV) [22] (2) C-NeRF [47] (3) D-NeRF [31] (4) DyMap [30] (5) K-Planess [35]. For experiments

Ground truth

Mixvoxels

(500MB)

K-Planes

(103MB)

Ours

(27.4MB)

on DyNeRF dataset, we compare with (1) LLFF (2) The original method proposed in DyNeRF [17] (3) Mixvoxels [45] (4) K-Planes [35]. For K-Planes, we run their official implementation and found that the results on DyNeRF are slightly different from their paper. For other methods, we directly adopt the results from [30] on NHR and [35, 45] on DyNeRF dataset.

Tabs. 1 and 2 list the comparison of our method with the state-of-the-art methods on NHR and DyNeRF datasets in terms of rendering quality and model size. We report PSNR and SSIM as the metrics for rendering quality, and the size of the model is measured in megabytes (MB). We also provide qualitative results in Figs. 2 and 3. The results show that our method can achieve rendering quality comparable to the state-of-the-art while offering significant advantages in storage efficiency. Single MLP-based methods, such as D-NeRF and DyNeRF, have similar storage sizes to ours, but their training process is extremely slow, requiring more than days.

5.3 Ablation studies

To analyze the effectiveness of our proposed dynamic codebook strategy, we conduct ablation studies on NHR and DyNeRF datasets. We first train the feature plane based model without compression, then we apply compression without dynamic codebook, and finally, we utilize dynamic codebook to incrementally refine the model. We report quantitative results in Tab. 3 and provide qualitative results in Fig. 4. Results in Tab. 3 show that the compression step can drastically reduce storage requirements, but causes a noticeable decline in rendering quality. And the proposed dynamic codebook strategy can compensate for this by introducing a small amount of additional storage while

$.S.!

8

$.1S.

B

80

$.1S.

8

achieving rendering quality almost equivalent to the level before compression. Results in Fig. 4 demonstrate that compression particularly degrades the rendering quality in detailed regions, such as the facial regions. The use of a dynamic codebook can effectively compensate for the loss induced by compression.

6 Conclusion

We presented a novel neural representation, named dynamic codebook, for learning neural volumetric videos with low storage costs while enabling high-quality view synthesis. Our approach models the volumetric video using a set of feature planes and compresses the video by reducing the spatial and temporal redundancy of features. We calculate the importance score of features and merge similar features that are of small importance scores. To compensate for the artifacts caused by the compression, our approach pinpoints the regions with lower rendering quality and appends dynamic codes for them. Experiments demonstrated that our approach achieves competitive rendering quality while being able to take up much fewer storage cost compared to state-of-the-art methods.

Discussion. We currently update the code independently for each time segment, which takes more than one hour for each dynamic scene, thereby slowing down the reconstruction. A solution is to increase the parallelism of optimizing codes. We leave it to future works.

Acknowledgement. The authors would like to acknowledge support from NSFC (No.62172364), Information Technology Center and State Key Lab of CAD&CG, Zhejiang University.

References

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Int. Conf. Comput. Vis. , pages 5855-5864, 2021.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5470-5479, 2022.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 , 2023.

A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In Eur. Conf. Comput. Vis. , 2022.

P. C. Cosman, K. L. Oehler, E. A. Riskin, and R. M. Gray. Using vector quantization for image processing. Proceedings of the IEEE , 81(9):1326-1341, 1993.

C. L. Deng and E. Tartaglione. Compressing explicit voxel grid representations: fast nerfs become also small. In IEEE Winter Conf. Appl. Comput. Vis. , pages 1236-1245, 2023.

J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH , pages 1-9, 2022.

S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5501-5510, 2022.

P. Gao, Z. Jiang, H. You, P. Lu, S. C. Hoi, X. Wang, and H. Li. Dynamic fusion with intra-and intermodality attention flow for visual question answering. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 6639-6648, 2019.

A. Gersho and R. M. Gray. Vector quantization and signal compression , volume 159. Springer Science & Business Media, 2012.

M. Goesele, B. Curless, and S. M. Seitz. Multi-view stereo revisited. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 2402-2409, 2006.

R. Gray. Vector quantization. IEEE Assp Magazine , 1(2):4-29, 1984.

B. Huang, X. Yan, A. Chen, S. Gao, and J. Yu. Pref: Phasorial embedding fields for compact neural representations. arXiv preprint arXiv:2205.13524 , 2022.

M. Kazhdan and H. Hoppe. Screened poisson surface reconstruction. ACM Trans. Graph. , pages 1-13, 2013.

L. Li, Z. Shen, Z. Wang, L. Shen, and L. Bo. Compressing volumetric radiance fields to 1 mb. In IEEE Conf. Comput. Vis. Pattern Recog. , 2022.

T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al. Neural 3d video synthesis from multi-view video. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5521-5531, 2022.

Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 6498-6508, 2021.

H. Lin, S. Peng, Z. Xu, Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Efficient neural radiance fields with learned depth-guided sampling. In SIGGRAPH Asia , 2022.

J.-W. Liu, Y.-P. Cao, W. Mao, W. Zhang, D. J. Zhang, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. arXiv preprint arXiv:2205.15723 , 2022.

L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. In Adv. Neural Inform. Process. Syst. , pages 15651-15663, 2020.

S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph. , 2019.

B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. TOG , 38(4):1-14, 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Eur. Conf. Comput. Vis. , 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph. , pages 1-15, 2022.

N. M. Nasrabadi and R. A. King. Image coding using vector quantization: A review. IEEE Transactions on communications , 36(8):957-971, 1988.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon. KinectFusion: Real-time dense surface mapping and tracking. In ISMAR , pages 127-136, 2011.

K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. In Int. Conf. Comput. Vis. , pages 5865-5874, 2021.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. PyTorch: An imperative style, high-performance deep learning library. In Adv. Neural Inform. Process. Syst. , pages 8026-8037, 2019.

S. Peng, Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Representing volumetric videos as dynamic mlp maps. In IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 10318-10327, 2021.

C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Int. Conf. Comput. Vis. , pages 14335-14345, 2021.

C. Reiser, R. Szeliski, D. Verbin, P. P. Srinivasan, B. Mildenhall, A. Geiger, J. T. Barron, and P. Hedman. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. arXiv preprint arXiv:2302.12249 , 2023.

D. Rho, B. Lee, S. Nam, J. C. Lee, J. H. Ko, and E. Park. Masked wavelet representation for compact neural radiance fields. In IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

Sara Fridovich-Keil and Giacomo Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

J. L. Schonberger and J.-M. Frahm. Structure-from-motion revisited. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 4104-4113, 2016.

J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Eur. Conf. Comput. Vis. , pages 501-518, 2016.

R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In IEEE Conf. Comput. Vis. Pattern Recog. , 2023.

T. Sikora. The mpeg-4 video standard verification model. IEEE Transactions on circuits and systems for video technology , 7(1):19-31, 1997.

L. Song, A. Chen, Z. Li, Z. Chen, L. Chen, J. Yuan, Y. Xu, and A. Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. ACM Trans. Graph. , pages 27322742, 2023.

C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5459-5469, 2022.

T. Takikawa, A. Evans, J. Tremblay, T. Müller, M. McGuire, A. Jacobson, and S. Fidler. Variable bitrate neural fields. In SIGGRAPH , pages 1-9, 2022.

J. Tang, X. Chen, J. Wang, and G. Zeng. Compressible-composable nerf via rank-residual decomposition. arXiv preprint arXiv:2205.14870 , 2022.

A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In Adv. Neural Inform. Process. Syst. , 2017.

F. Wang, S. Tan, X. Li, Z. Tian, and H. Liu. Mixed neural voxels for fast multi-view video synthesis. arXiv preprint arXiv:2212.00190 , 2022.

L. Wang, Q. Hu, Q. He, Z. Wang, J. Yu, T. Tuytelaars, L. Xu, and M. Wu. Neural residual radiance fields for streamably free-viewpoint videos. arXiv preprint arXiv:2304.04452 , 2023.

Z. Wang, T. Bagautdinov, S. Lombardi, T. Simon, J. Saragih, J. Hodgins, and M. Zollhofer. Learning compositional radiance fields of dynamic human heads. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 5704-5713, 2021.

M. Wu, Y. Wang, Q. Hu, and J. Yu. Multi-view neural human rendering. In IEEE Conf. Comput. Vis. Pattern Recog. , pages 1682-1691, 2020.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

S. Zhang, S. Guo, W. Huang, M. R. Scott, and L. Wang. V4d: 4d convolutional neural networks for video-level representation learning. arXiv preprint arXiv:2002.07442 , 2020.

J. Ziv and A. Lempel. A universal algorithm for sequential data compression. IEEE Transactions on information theory , 23(3):337-343, 1977.

Peer reviews:
comments: SUMMARY:
A method for compressing Volumetric Videos is presented in this work. It is based on NeRF with a factored multi feature plane representation. The features of the model are compressed in two stages. In the first, a codebook for features is constructed based on the average contribution a feature has to the total reconstruction. Features are weighted based on their score contribution to the NeRF integral. Ones with low score contributions are merged into a zero code, the top 30% of codes are retained, and the rest are clustered using an exponential-moving average algorithm. In the second compression stage,  a small number of features per temporal segment are selected based on the accumulated gradients of back propagation on the codebook compressed model. These features are then optimized and added to the codebook, dynamically growing the codebook with time. The resulting method produces reconstruction quality and compression ratios comparable to SOTA while requiring only hours instead of days to train.

STRENGTHS:
A simple method that achieves SOTA accuracy and compression performance but is much more efficient.

WEAKNESSES:
- Writing could be a bit clearer
- Only evaluated on two scenes

QUESTIONS:
- Adding some implementation details where the method is described would help clarify thing sooner. For example, how accumulated score contributions are used to select the three categories, and how a temporal fragment is defined. 
- The clustering algorithm in Sec B or supplementary could be clearer and added to the main text. ie. is this performed while training the features as in VQ-VAE, or is performed as a post-processing?

LIMITATIONS:
Authors do not address limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of representing high-fidelity volumetric videos with low storage cost. The authors proposed a novel neural representation called the dynamic codebook, which aims to reduce the spatial and temporal redundancy of feature grids inherent to scenes due to self-similarity. It achieves this by merging similar features for model compression and compensating for potential declines in rendering quality through a set of dynamic codes. The experimental results demonstrate that the dynamic codebook method achieves state-of-the-art rendering quality while achieving higher storage efficiency.\n\nThe contributions of the paper are twofold: \n1. This paper proposes the dynamic codebook representation, which effectively reduces model size by minimizing feature redundancy in space and time while maintaining rendering quality in dynamic detailed regions. \n2. Experimental results show that the proposed approach achieves rendering quality comparable to state-of-the-art methods while significantly improving storage efficiency.\n\nSTRENGTHS:\nThe paper introduces a novel approach that use the dynamic codebook to reduce the spatial and temporal redundancy.\n\nThe paper is well-structured, with clear section headings, and visual aids such as figures are effectively utilized to illustrate key points or enhance comprehension.\n\nWEAKNESSES:\nExperimental details and results seem incompletely reported, so I have some confusion about this work. I defer some of my issues in \"Questions\".\n\nQUESTIONS:\n1. This paper claims that “…which is of low model size but extremely slow training speed.” In Table 1, but there is no mention of the training time for the models. It would be helpful to provide insights into the time required for training the model and how it compares to other models.\n2. Regarding the comparison of model size, I noticed that this paper uses “post-processing” including quantization and entropy coding techniques. According to my understanding, this techniques can affect the model size, whether other comparison models use this techniques? If not, it would be beneficial to discuss the potential outcomes if these post-processing steps were not applied. \n3. In line 237, this paper uses different k for different dataset. Is there any particular consideration followed while setting the value of k?\nIn reference to the dynamic codebook's size, it would be useful to explore what constitutes an appropriate size. The codebook-based compression method mentioned can significantly reduce model storage but may result in a decline in rendering quality. Thus, it would be meaningful to investigate how variations in k affect the same dataset.\n4. The model size is composed of the quantized dynamic codebook, index planes as well as MLP parameters with entropy encoding, so the bit allocation of different components of the model should be provided, which can help the understanding of the proposed methodology.\n\nLIMITATIONS:\nThe authors have partially addressed the limitations of existing work, though there is space for improvement (see the section Weaknesses and Questions).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The proposed method applies the codebook technique to explicit feature plane representation and a dynamic codebook is further proposed for dynamic scenes. Experimental results demonstrate the good performance of the proposed methods.

STRENGTHS:
1. The proposed method significantly reduces the model size while keeping a similar render quality with K-Planes [35] from both qualitative metrics and visualization.
2. The proposed dynamic codebook is easy to follow and the problem addressed is important and interesting.

WEAKNESSES:
1. Considering that previous work VQRF has introduced codebook to feature grid representation for reducing the model size with an excellent compression ratio, the proposed method here that introduces codebook to feature plane representation is reasonable but not of sufficient novelty.  
2. The main contribution of model size reduction comes from the codebook technique. The efficiency of the proposed dynamic codebook is somewhat limited.
3. There is no experimental comparison of the time cost. The influence of the optimization process caused by the dynamic codebook is unclear.
4. There lack some explanation and evaluation about some hyperparameters and settings. For example, In Sec 4, Line 235: "We then retain the top 30% of codes with the highest importance score contributions and cluster the remaining codes into k codes". The reason for the setting 30% and the influence of k can be provided for better demonstrating the efficiency of the proposed strategy.

QUESTIONS:
1. The proposed dynamic codebook should be applicable to other explicit representations, e.g., feature grids. The authors may explain the reason why they did not conduct related experiments.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this paper propose  a dynamic codebook, which optimizes away codes of low importance to rendering the scene, clusters 70% of the least importance codes and optimizes the remaining codes for every time fragment. The process the authors introduce are intuitive and each step furthers compression or works towards reducing the distortion introduced in the compression process.\n\nI have read and acknowledged the author's rebuttal and will remain with my generally positive assessment.\n\nSTRENGTHS:\nThe introduction of a dynamic codebook is intuitive and handles a number of problems in the codebook creation process. First keeping highly important codes, clustering low important codes to a zero token, clustering remaining codes and then using gradient aware code optimization to further increase details for specific time fragments. \n\nThe authors demonstrate near state of the art PSNR on NHR and DyNeRF datasets ( < 0.25 dB PSNR difference) with substantially (more than 14x) smaller models.\n\nWEAKNESSES:\nAs discussed by the authors, the dynamic codebook construction takes at least an hour for NHR and almost two hours for DyNeRF and substantially increases the total end to end time and rendering time.\n\nThe dynamic codebook introduces more hyperparameters to tune, what percentage of features to keep, drop, cluster and how many dynamic codewords get added. NHR and DyNeRF seem to differ substantially in these and it's unclear how much of this is based off of framerate, resolution, or complexity.\n\nQUESTIONS:\n1. In L169, the authors state that 99.9% of the importance is contributed by merely 10% of the voxels in the 2D case and that it is not as bad in the 3D case. Is there a citation or an empirical result the authors could share on this?\n\n2. How large are the codebooks for each of the time fragments? In L237, the authors state that the clustered codes are set to 4096 (NHR) and 16,384 for DyNeRF, but not how many codes are being optimized by the dynamic codebook for each fragment.\n\n3. How much experimentation did it take to discover the different hyperparameters for the dynamic codebook? Is it easy to extrapolate to other datasets? Do much more could be gained by even more optimal hyperparameters on a nerf by nerf basis?\n\n4. Any qualitative or user study on different nerfs? The quality difference seems minor, so it's not clear if it's visually preferred over other methods.\n\nTypo in Table 3:\nbasline => baseline\n\nLIMITATIONS:\nNo concerns", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presents a novel approach for representing volumetric video using a dynamic codebook that incorporates the temporal correlation of features. This addresses the drawback of existing feature grid-based methods, which overlook this correlation.

STRENGTHS:
1. The proposed method uses a multidimensional feature space and a dynamic codebook to model the changing scene.
2. The paper presents model compression techniques, including pruning and weight clustering, which reduce the size of the model.

WEAKNESSES:
1. The article's novelty is limited as the introduction of the dynamic codebook approach appears to be primarily focused on engineering aspects. Additionally, the results from ablation experiments suggest that this approach is not particularly effective, and although a large number of model parameters are introduced, there is only a marginal improvement in video quality.

2. It is impractical to manually adjust dynamic codebook parameters (such as k) for different videos in real-world applications.

3. The paper lacks comparison results regarding training and rendering time.

QUESTIONS:
1. What specific improvements have been made in Sections 3.1 and 3.2 compared to existing works, like [16]?
2. Why were certain methods not tested on the DyNeRF dataset?

LIMITATIONS:
Further research is needed to develop effective methodologies for dynamic codebooks. Although the paper enhances the expressive capabilities of the dynamic neural representation model, there is room for future investigation into optimizing training and rendering speed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09