Rate-aware Compression for NeRF-based Volumetric Video

Authors: zhiyu  zhang
guo  lu
huanxiong  liang
zhengxue  cheng
anni  tang
li  song
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 04:59:35

Abstract: The neural radiance fields (NeRF) have advanced the development of 3D volumetric video technology, but the large data volumes they involve pose significant challenges for storage and transmission. To address these problems, the existing solutions typically compress these NeRF representations after the training stage, leading to a separation between representation training and compression. 
In this paper, we try to directly learn a compact NeRF representation for volumetric video in the training stage based on the proposed rate-aware compression framework. 
Specifically, for volumetric video,  we use a simple yet effective modeling strategy to reduce temporal redundancy for the NeRF representation.
Then, during the training phase, an implicit entropy model is utilized to estimate the bitrate of the NeRF representation. This entropy model is then encoded into the bitstream to assist in the decoding of the NeRF representation. This approach enables precise bitrate estimation, thereby leading to a compact NeRF representation.
Furthermore, we propose an adaptive quantization strategy and learn the optimal quantization step for the NeRF representations. 
Finally, the NeRF representation can be optimized by using the rate-distortion trade-off. 
Our proposed compression framework can be used for different representations and experimental results demonstrate that our approach significantly reduces the storage size with marginal distortion and achieves state-of-the-art rate-distortion performance for volumetric video on the HumanRF and ReRF datasets. 
Compared to the previous state-of-the-art method TeTriRF, we achieved an approximately -80\% BD-rate on the HumanRF dataset and -60\% BD-rate on the ReRF dataset.

Main paper: 
 introduction: 
 3D volumetric video can provide immersive experience for viewers and exhibits a potential trend to be the next-generation video format. Previous works for volumetric videos reconstruction includes point cloud-based approaches [15] and depth-based approaches [3] . Recently, the popularized 3D representation known as Neural Radiance Fields (NeRF) [32] , acclaiming for its photorealistic rendering capabilities from novel viewpoints, have attracted tremendous attention.NeRF [32] utilizes Multilayer Perceptron (MLP) to model 3D scenes and employ volume rendering techniques to generate images from novel perspectives. To increase the training and rendering speeds, the mainstream works [6, 34, 44] incorporate explicit NeRF representations, such as grids [44] , planes [6] , and hash tables [34] . Furthermore, various methods [5, 10, 11, 28, 35, 36, 50] have been proposed to extend NeRF of static scenes to dynamic scenes, i.e., volumetric videos. However, the majority of existing research primarily emphasizes enhancing the reconstruction quality of NeRF representations, while often overlooks the critical need for reducing the storage size and transmission bandwidth. This oversight presents significant challenges for practical applications, particularly in dynamic volumetric video scenarios.To address these problems, several methods [14, 24, 25, 30, 41, 46 ] have been proposed to compress explicit NeRF representations for static scenes and dynamic scenes [48, 49, 52] . These methods utilize prediction, transformation, quantization, and entropy coding techniques, inspired from traditional video compression algorithms to compress the dynamic NeRF representations after the training stage, achieving significant compression rates. Among these approaches, ReRF [48] utilizes grid-based explicit representations to model dynamic scenes and devises a compression method akin to JPEG [47] to compress the representations. VideoRF [49] builds upon ReRF [48] by integrating traditional 2D video codec into the compression process. TeTriRF [52] employs a more compact triplane representation to model dynamic NeRF and utilizes an HEVC codec to compress the tri-plane representations. However, these methods concentrate exclusively on compressing dynamic NeRF representations post-training, neglecting the rate-distortion tradeoff during the training phase. Therefore, they fail to adequately reduce spatial and temporal redundancy within the NeRF representations, thus their compression performance is far from optimal.In this paper, we propose a rate-aware compression method tailored for NeRF-based volumetric video. Our approach estimates the bitrate of NeRF representations during its training stage and incorporates rate and distortion terms into the loss function, enabling end-to-end training. Therefore, we can obtain a compact NeRF representation with optimal rate-distortion performance. Specifically, first of all, targeting explicit NeRF grid representations, we propose inter prediction-based dynamic modeling technique by learning residual information based upon the previous frame, which effectively reduces the entropy of the NeRF representation. Second, we propose an adaptive quantization strategy with learnable quantization step to preserve more detailed information which contribute better reconstruction quality at different locations and scales. Third, in order to incorporate bitrate constraints into the training process and leverage rate-distortion loss, we introduce a tiny MLP-based implicit entropy model to estimate the rate. Given the potentially complex distribution of the explicit NeRF representation, we use temporal and spatial context for a more accurate rate estimation. Experimental results demonstrate that by applying our methodology to grid-based explicit representation [7] , we can achieve state-of-the-art rate-distortion compression performance on the representative HumanRF [17] and ReRF [48] datasets. Compared with SOTA method TeTriRF [52] , we achieve an approximate -80% BD-rate on HumanRF dataset and -60% BD-rate on ReRF dataset.Our main contributions can be summarized as follows:• We proposed a rate-aware compression framework for NeRF grid representations. Our pipeline introduces adaptive quantization strategy and spatial-temporal implicit entropy model to achieve joint rate-distortion optimization during training, which greatly enhances the compression performance than post-training methods. • Extensive experimental results on the benchmark datasets demonstrate the effectiveness of our approach. We can save more than 80% bitrate when compared with state-of-the-art dynamic NeRF compression approach at the same reconstruction quality.2 Related Work 2.1 NeRF for Scene RepresentationNeRF [32] enables the generation of images from arbitrary viewpoints through volume rendering, by employing a large-scale Multilayer Perceptron (MLP) to fit the colors and densities of sampled points in a scene. However, utilizing a large-scale MLP to represent the scene for real-time rendering is infeasible, prompting several works to propose combining explicit representations with smaller MLPs as a substitute for the pure implicit representation of the scene. This approach aims to reduce the computational complexity associated with large-scale MLPs. Some existing works have explored various approaches in this regard. For instance, [12] utilizes an octree, [44] employs a voxel grid, [34] implements hash tables, and [6] utilizes tensors.Expanding NeRF [32] to dynamic scenes is not a trivial task, necessitating consideration of object movements within dynamic scenes. [9, 13, 53 ] model dynamic scenes by using time as an additional condition for the implicit MLP, yet these purely implicit modeling methods are not only inefficient but also perform poorly in motion-intensive settings. Alternatively, [26, 27, 35] incorporate motion modeling of the scene using a deformation field to predict displacements in dynamic scenes between each frame and a canonical frame. To expedite training and rendering, [10, 11] introduce explicit representations to accelerate the reconstruction of dynamic scenes. Essentially, these approaches utilize a single NeRF model to fit all frames within a dynamic scene, achieving commendable reconstruction results for short sequences. However, the reconstruction quality dramatically declines with longer sequences. Consequently, to ensure high-quality reconstructions in long-sequence settings, we adopt a frame-by-frame modeling approach for dynamic scenes in this paper. 
nerf representations compression: 
 Although NeRF has achieved superior rendering quality and fast training speed via the explicit representations, it is at the cost of additional model size and storage cost. Recently, numerous works [14, 20, 24, 25, 30, 38, 41, 46, 57] have focused on compressing the explicit representation of NeRF in static scenes. [24] employs post-processing techniques to compress trained voxel grids through voxel pruning and vector quantization. [46] and [41] introduce compression-related operations during training, such as vector quantization [46] and binarization [41] to compress neural radiance fields, yet they do not consider rate-distortion optimization during training. [14, 20, 30] reduce the entropy of the NeRF representation by introducing rate loss during training, but their methods of rate estimation are relatively simplistic, and the compression performance is not optimal. NeRFCodec [25] compresses NeRF representations by adjusting encoder and decoder heads, building on the reuse of 2D neural image codec, but this method is only applicable to plane-based NeRF representations. Moreover, these methods solely address compression in static scenes and do not account for temporal redundancy in dynamic scenes.For dynamic NeRF compression, the recent works [16, 48, 49, 52, 56, 58] adopt frame-by-frame modeling, which enables high-quality reconstruction for long-duration sequences. These approaches also incorporate compression algorithms to ensure efficient storage utilization. ReRF [48] models the radiance field as a combination of motion fields and residual fields and employs traditional video compression pipelines to compress the representation of the radiance field. Guo et al. [16] adopts a vector quantization approach to eliminate the spatial-temporal redundancy of the radiance field. TeTriRF [52] directly models the radiance field as a tri-plane representation and applies a 2D video encoder to compress the representation. However, these methods do not consider the optimization of radiance field representation guided by rate-distortion loss functions. Our proposed method not only eliminates temporal redundancy but also leverages rate-distortion functions to optimize the representation of the radiance field, achieving optimal rate-distortion performance.To achieve efficient transmission, data compression has been investigated for other multimedia data, including image, video and 3D contents. This work also inspires from these previous approaches, thus we introduce them briefly.In the field of image compression, various methods have been developed to formulates the rate-distortion optimization problem as an end-to-end training process to enhance compression efficiency utilizing neural networks [1, 8, 18] . Balle [1] proposed a hyperpriorbased autoencoder to achieve better compression performance than traditional codecs, such as JPEG [47] and JPEG2000 [42] . Cheng [8] have employed discrete Gaussian mixture models to estimate the distribution of latents, achieving compression performance that exceeds the VVC-intra [4] . For video compression algorithms, traditional video coding standards [4, 45] utilize a hybrid coding framework that compresses video through intra prediction, inter prediction, transformation, quantization, and entropy coding. Neural network-based end-to-end video compression methods [19, 21-23, 29, 40] employ neural networks to replace modules within the hybrid coding framework. By using rate-distortion optimization to tune the entire network, the SOTA method have achieved compression performance surpassing the latest VVC standard [4] .3D content comes in various forms, such as NeRF, point clouds, and multi-view videos. For point cloud compression, MPEG has introduced G-PCC and V-PCC [39] to compress different types of point clouds. With the advancement of neural networks in image and video compression, numerous studies [37, 43] have explored neural network-based point cloud compression. [54] introduced 3D wavelet transform to compress volumetric images. [37] suggested enhancing the utilization of spatio-temporal information by leveraging voxelized information from neighboring nodes within an octree architecture, thereby further improving the efficiency of point cloud compression. For the compression of multi-view videos, MPEG has introduced MIV [3] standard, which encodes multi-view videos using 2D encoders, eliminates inter-view redundancies, and utilizes novel view synthesis algorithms to synthesize new viewpoints. However, spatio-temporal redundancy reduction has not yet fully investigated for NeRF representations. 
preliminaries: 
 NeRF [32] is a continuous 3D scene representation technique that learns a mapping fuction 𝑔 𝜙 (x, d) : R 𝑑 → R 𝑐 , transforming the coordinates x = (𝑥, 𝑦, 𝑧) of sampled points along a ray and the viewing direction d = (𝜃, 𝜙) into color c and density 𝜎:(c, 𝜎) = 𝑔 𝜙 (x, d).(EQUATIONWhen rendering images from novel viewpoints, given a target camera extrinsic, the color Ĉ (r) of corresponding pixel can be obtained through volume rendering:EQUATION)where 𝑇 𝑖 and 𝛼 represents the transmittance and alpha value of 𝑖-th sampled point and 𝛿 𝑖 denotes the distance between adjacent sampled points.The original NeRF [32] employs a purely implicit MLP to approximate mapping function 𝑔. Subsequent works have introduced explicit representations (such as grids [7] , planes [6] , etc.) combined with a tiny rendering MLP to accelerate the training and rendering. It should be noted that our proposed compression method can be used for different explicit representations and we provide further analysis in our experimental part. Here, we adopt the DiF [7] as our default static representation in this paper and we extend it for the dynamic scenes.  
methodology: 
 Fig. 3 illustrates the training pipeline of our method. This includes a dynamic modeling method based on inter prediction, an adaptive quantization strategy, and bitrate estimation for the radiance field representation through a spatio-temporal implicit entropy model, combined with distortion loss for rate-distortion optimization.Given that directly modeling entire dynamic scenes using NeRF representation like [5, 11] may result in poor performance for long sequences [48] , we extend the current static NeRF representation DiF to dynamic scenes through a frame-by-frame inter prediction based modeling approach in the time dimension. Specifically, the components required for volume rendering of a static scene include the explicit representation, occupancy grid, and the rendering MLP. These are also the components that need to be compressed and transmit. Tab 1 demonstrates the size of different components in the grid-based representation [7] . It is evident that independently modeling a NeRF for each frame without compression would require about 600 MBps (20MBx30fps) bandwidth for volumetric video, which is unacceptable. Given that the explicit representation occupies the majority of the model size, eliminating the inter-frame redundancy of the explicit representation could make dynamic NeRF more compact. To simplify the description, we collectively refer to the coefficient grid and the basis grid as the feature grid 𝐺. To model dynamic scenes, we use a simple yet effective dynamic frame-by-frame modeling strategy. Here, we first divide the entire sequence into equally long groups, where the first frame of each group is an I-frame, modeled independently, and subsequent frames are P-frames, modeled only in terms of the residual relative to the previous frame. Specifically, when modeling a P-frame, the reconstruction grid of the previous frame Ĝ𝑡 -1 is retrieved from the decoded buffer, and we can learn the residual grid 𝑅 𝑡 for the current frame. Then the representation for current frame is expressed as:EQUATIONFinally, Ĝ𝑡 is stored into decode buffer for the reconstruction of the next frame. Furthermore, to ensure the temporal continuity and facilitate compression, we have applied an L1 regularization to the magnitude of the residual grid, i.e., L 𝑟𝑒𝑔 = ∥𝑅 𝑡 ∥ 1 .In addition, given the feature-level similarity of frames within the same group, we allow frames within a group to share the rendering MLP. This strategy significantly reduces bitrate consumption at a lower bitrate range without incurring substantial performance degradation.In our proposed framework, the learned representation 𝐺 𝑡 (𝐺 𝑡 for I-frame and 𝑅 𝑡 for P-frame) for the current frame should be quantized before the actual entropy coding. One straightforward approach is to use a uniform quantization step for all the items in 𝐺 𝑡 . However, the importance of different regions/scales in the learned representation 𝐺 𝑡 may vary and uniform quantization may be not the optimal solution.In this paper, we have adopted an adaptive quantization training strategy to identify the optimal quantization step for NeRF representations. Taking DiF representation as an example, their explicit representations encompass grids at different scales. During training, we assign different quantization step parameters to grids of different scales. These parameters are set as trainable and are continuously optimized throughout the training process. Then quantization is defined as Ĝ𝑡 = 𝐺 𝑡 𝑞 𝑡 , where 𝑞 𝑖 is trainable quantization step. Fig. 4 illustrates the distribution histograms of quantized explicit NeRF representations obtained through adaptive quantization training versus fixed quantization training. Fig. 4 demonstrates that the representations obtained from adaptive quantization training have a wider distribution range of [-150, 150] and a larger variance. On the other hand, the representations obtained from fixed quantization training are concentrated within the range of [-20, 20] and have a smaller variance. Additionally, the bitrates of the two cases are similar, indicating that adaptive quantization allows for more finegrained quantization step adjustments. It does not indiscriminately remove high-frequency information but rather preserves useful high-frequency information. As a result, it can improve reconstruction quality while maintaining the same bitrate. We also present subjective result in the supplementary materials demonstrating the effects of adaptive quantization. The results using adaptive quantization are able to reconstruct more details. Considering that the NeRF representation is learned during training stage, it is plausible to incorporate a rate loss term into the loss function to guide the NeRF representation towards a lower compression bitrate direction. However, it is not feasible to obtain the actual bitrate of the NeRF representation during training stage because entropy coding is non-differentiable. Hence, we propose a spatial-temporal implicit entropy model for accurate rate estimation. Furthermore, unlike the learned entropy model in image compression [33] , which is learned from large-scale training datasets, our proposed implicit entropy model is learned on-the-fly with the corresponding NeRF representation in the training stage. Consequently, for each frame of the NeRF model, we opt to train a specific implicit entropy model, enabling us to accurately estimate the bitrate of the NeRF representation. Furthermore, we encode the implicit entropy model into the bitstream and the decoded entropy model will be used to decode NeRF representation at the decoder side. In particular, we employ the autoregressive entropy model to estimate the bitrate of the explicit NeRF representation. As shown in Fig. 5 , the implicit entropy model consists of a two-layer shallow MLP network and incorporates both spatial and temporal context information to estimate the distribution of the NeRF representation. (4)EQUATION)) ( V𝑖,𝑗,𝑡 ). (5) where N (𝜇 , 𝜎) denotes the Laplace distribution.Then we can calculate the distribution's cumulative function 𝑃 𝑐𝑑 𝑓 (•) using the predicted parameters 𝜇 and 𝜎 and estimate the bitrate of the current voxel:EQUATION.(6) where 𝑁 denotes the total number of voxels in the grids, Ṽ𝑖,𝑗,𝑡 denote the simulated quantized 𝑉 𝑖,𝑗,𝑡 .Rate-Distortion Loss Function. Striking a balance between bitrate and distortion is pivotal to efficient NeRF representation compression. Thus, we estimate the bitrate of the NeRF representation and optimize the NeRF representation by using the rate-distortion loss as follows,EQUATION)where 𝐶 (r) is ground truth color, Ĉ (r) is the rendered color as shown in Equ. 2, and we use the L2 loss as the distortion loss. L 𝑟𝑒𝑔 essentially serves as a rate constraint. By assigning a larger weight to L 𝑟𝑒𝑔 , the magnitude of the residual grid will be smaller, resulting in a lower bitrate. Therefore, we categorized L 𝑟𝑒𝑔 and L 𝑟𝑎𝑡𝑒 as bitrate-related loss. 𝜆 is a trade-off parameter for distortion and rate. To ensure stable training and control the trade-off between quality and bitrate solely through the 𝜆 parameter, we designed the weights of L 𝑟𝑎𝑡𝑒 and L 𝑟𝑒𝑔 to have a fixed linear relationship, specifically 𝜆 𝑟𝑒𝑔 = 𝛼 * 𝜆 𝑟𝑎𝑡𝑒 .In summary, within the NeRF model, the components that need to be compressed and transmitted include the explicit representation, occupancy grid, rendering MLP, and the implicit entropy model. The encoding and decoding of the explicit representation rely on the implicit entropy model, while the other parts can be encoded and decoded independently.Occupancy grid. The characteristic of the occupancy grid is that its elements are either 0 or 1. Thus, for compression, it is first flattened into a vector, grouped in sets of 8 to be packed into uint8 data, and then these uint8 data are compressed. Since a significant portion of the packed uint8 data consists of zeros, the LZMA dictionary encoding algorithm is employed for compression.Rendering MLP and Implicit entropy model. Both the components are MLPs, with the parameters needing compression being the network's weights and biases. Given the small size of these parameters, we simply quantize them and then apply entropy encoding using a range coder [31] .Explicit representation. The encoding and decoding of the explicit representation rely on the distribution parameters estimated by the implicit entropy model. Hence, before encoding and decoding the explicit representation, the implicit entropy model must be encoded and decoded first. Based on the distribution parameters estimated by the implicit entropy model, we employ a range coder to encode and decode the explicit representation, following a spatio-temporal causal order.Decoding pipeline. Fig. 6 illustrates the decoding pipeline of our compression framework. Initially, the occupancy grid, rendering MLP, and implicit entropy model are decoded from the bitstream. Subsequently, voxels in the explicit representation are decoded from the bitstream in causal order. Using the already decoded voxels as context, the distribution of undecoded voxels is predicted through an implicit entropy model. Based on the parameters of the distribution, voxels are decoded from the bitstream. 
experiments 5.1 experimental setup: 
 Datasets. We utilized two datasets: HumanRF [17] and ReRF [48] , to validate the effectiveness of our method. For the HumanRF dataset [17] , we selected 100 viewpoints with the same camera intrinsics. The image resolution was approximately 1020x750. Among these viewpoints, we designated viewpoints 15 and 25 as the test viewpoints, while the remaining 98 viewpoints were used for training. Regarding the ReRF dataset [17] , it comprised approximately 75 viewpoints with an image resolution of 1920x1080. We chose viewpoints 6 and 39 as the test viewpoints following [52] , while the rest of the viewpoints were employed for training.Evaluation Metrics. To assess the compression performance of our method on the experimental datasets, we employ Peak Signalto-Noise Ratio (PSNR), Structural Similarity Index (SSIM) [51] , and Learned Perceptual Image Patch Similarity (LPIPS) [55] as quality evaluation metrics. Additionally, we utilize KB per frame as the bitrate evaluation metric. Furthermore, for an overall comparison of compression performance, we utilize the BD-rate metric [2] .Implementation Details. In the experiment, we set the group size to 20, which means that an I-frame is inserted every 20 frames within the dynamic scene. For the rate-distortion loss, we defined four different 𝜆 values(7𝑒 -4, 1𝑒 -3, 2𝑒 -3, and 5𝑒 -3) to achieve different compression ratios. We set 𝛼 to 10 for the regularization loss. For the training details of NeRF, we refer to the default configuration of DiF [7] .In our experiments, our primary benchmarks are the state-of-theart methods TeTriRF [52] and ReRF [48] . Fig. 7 and Fig. 8 illustrate the rate-distortion curves on the HumanRF [17] and ReRF [48] datasets respectively. The figures clearly show that our method achieves the best rate-distortion performance on all metrics-PSNR, SSIM, and LPIPS-both in the training and testing viewpoints. Due to the substantial disparity in the bitrate range between our method and ReRF [48] , we only calculate the BD-rate in comparison with TeTriRF [52] . When computing the BD-rate, PSNR is selected as the quality metric. On the HumanRF dataset, our method achieves a BD-rate reduction of -84.52% for training viewpoints and -85.49% for testing viewpoints. On the ReRF dataset, our method attains a BD-rate reduction of -60.58% for training viewpoints and -76.60% for testing viewpoints. Tab. 2 shows the detailed quantitative results on HumanRF dataset and ReRF dataset, where Ours(Low) and Ours(High) respectively represent the results of our method under different 𝜆 configurations. The table indicates that both our method and TeTriRF can achieve good reconstruction quality at lower bitrate range, while ReRF requires significantly higher bitrate for similar quality level. Moreover, Ours(Low) achieves a bitrate substantially lower than TeTriRF with comparable reconstruction quality. And Ours(High) offers much higher reconstruction quality at a bitrate similar to TeTriRF, further demonstrating the rate-distortion performance advantage of our method.Fig. 9 displays qualitative comparison results on the HumanRF's actor07 sequence and the ReRF's kpop sequence. The figure illustrates that our method can reconstruct more details at a lower bitrate, such as the beard in actor07 and the clothing logo in kpop, which are lost in the reconstructions by TeTriRF and ReRF. This demonstrates that our method also provides a superior subjective experience.Our method is universally applicable to a large amount of explicit representation. To verify the effectiveness of our method, we further evaluated its performance on the plane-based representation TensoRF [6] . The comparison baseline is: modeling each frame in dynamic scenes with an independent TensoRF model, followed by quantization and entropy encoding of TensoRF after modeling is complete.Tab. 3 shows the comparison results between our method and the baseline method at various compression ratios. It is evident that our method achieves significantly lower bitrates than the baseline method, without a noticeable decline in reconstruction quality. This demonstrates that our method is not only suitable for DiF [7] but also applicable to TensoRF [6] , proving its effectiveness across different explicit representations. In the ablation studies, we aim to verify the efficacy of inter predictionbased dynamic modeling, adaptive quantization, and rate-distortion joint optimization we proposed. We conduct ablation studies on the actor02 sequence from the HumanRF [17] dataset. Based on the full method, we disable dynamic modeling and adaptive quantization separately during training. Tab. 4 shows the bitrates for different ablation experiments at the same quality level (PSNR = 34 dB). The definition of "Baseline" is similar to Sec. 5.3, where each frame of the dynamic scene is modeled using an independent DiF model, followed by quantization and entropy encoding of the representation after modeling is complete. From Tab. 4, it is apparent that, based on the "Full" method, either disabling dynamic modeling or adaptive quantization results in an increase in bitrate, indicating the effectiveness of these two modules. Comparing the "Baseline" with the other three experiments, it is evident that the bitrates of the other three experiments are significantly lower than that of the "Baseline", demonstrating the efficacy of our proposed rate-distortion joint optimization.Moreover, we analyze the average bit consumption for I-frame and P-frame under different 𝜆 configurations, and the result is illustrated in Tab. 5. It is evident that the bit consumption of Pframes is significantly lower than that of I-frames, which validates  
conclusion: 
 In this paper, we present a dynamic modeling and rate-distortion optimization framework tailored for explicit NeRF representation. By incorporating a rate-aware training strategy into the NeRF training process, our approach demonstrates enhanced compression efficiency. Utilizing an implicit entropy model for accurate rate estimation of NeRF representations, coupled with adaptive quantization, we further amplify the compression efficacy. Experimental results reveal that our method secures the highest compression efficiency across two benchmark datasets. The NeRF-based modeling and compression methodology we propose is capable of reducing the data volume of a single frame of radiance field to below 100KB, markedly advancing the transmission of volumetric videos. 
: 
 Figure 1 : Comparison of compression performance with the state-of-the-art method, TeTriRF [52] . Compared to TeTriRF, our method achieves approximately 1 dB higher PSNR at a similar bitrate, and the BD-rate is -83%. 
 
 Appendix:  


Decision: The paper proposes a rate-aware compression model for Nerfs.  The proposed approach includes a rate objective during the training stage, which allows it to be optimized using a rate-distortion trade-off.  The reviewers agree that the paper should be accepted.  Among other things, the reviewers mention that the motivation is clear, the proposed method is novel, the paper is well-written, and it performs well.  In the rebuttal, the authors also answered the main concerns raised by the reviewers, including an analysis of the complexity of the model, a better explanation of the loss function, and a supplement experiment on the DyNeRF dataset.

 After a thorough evaluation, including the reviewers' feedback and the authors' rebuttal, I recommend the paper be accepted.

Reviews: review: paper_summary: paper_summary: This paper proposes a compression model for NeRF representation, which combines the compression of NeRF and the NeRF training together. Experimental results verify the effectiveness of the proposed method, in terms of various quality metrics. This paper is a pioneer work considering the NeRF data compression in training phase, which is an interesting idea.
strength_weakness: strength_weakness: Compared with existing NeRF compression methods, the proposed method could achieve significant performance improvement, indicating the efficiency of the proposed method. 

There are some highlights for the proposed NeRF compression model:
1. The temporal correlation is well modeled with a inter prediction based dynamic modeling
2. Adaptive quantization, which could bear more high-frequency information under the similar bitrate, compared with fixed quantization. 
3. A joint loss function designed for NeRF compact representation.
limitations: limitations: The complexity of the proposed algorithm should be investigated, and compare the complexity of the proposed method with other popular methods.

generally speaking, in signal compression models, the loss function are usually formulated with two parts, the bitrate one and the distortion one. However, regarding the compression distortions, the color distortion is an individual part in loss function. But, the grid distortion is combined with the bitrate component. Please give some explanation about the reason of this formulation.

In equation 7, the \alpha is fixed 10, and \lambda is varied to achieved various compression ratios. Why \alpha is fixed?
paper_summary: paper_summary: In this paper, the authors propose a rate-aware compression framework for NeRF-based volumetric video compression, and propose to introduce the entropy model module and adaptive quantization strategy to achieve higher compression efficiency. The proposed method can be used for compression of different representations, and achieves state-of-the-art rate-distortion performance for volumetric video.
strength_weakness: strength_weakness: ++ Clear Motivation. The entropy coding and adaptive quantization are introduced into volumetric video compression task, it’s a milestone work.

++ Good Performance. The proposed method achieves the SOTA performance, and significantly improves the compression efficiency.

++ Well Written. The paper is easy to understand.

++ Well Design. The effectiveness of proposed modules is verified in ablation experiments.
limitations: limitations: -- Some Advices. 

1) In this paper, author follows the reference structure of I-P-P, how about the reference structure I-B-B in this task? I think the efficient reference structure can further improve this work.

2) In the design of spatial-temporal implicit entropy model, the autoregressive entropy model is used, how about the cooperation of hyperprior and autoregressive entropy model in this work? In learned video compression, many works are follow this manner [1].

Ref: [1] Li J, Li B, Lu Y. Deep contextual video compression [J]. Advances in Neural Information Processing Systems, 2021, 34: 18114-18125.

-- Lack of Reference.

1) Some volumetric image\video compression references are missing.

[1] Xue D, Ma H, Li L, et al. aiWave: Volumetric image compression with 3-D trained affine wavelet-like transform [J]. IEEE Transactions on Medical Imaging, 2022, 42(3): 606-618.

2) Some milestone end-to-end video compression references are missing in related work.

[1] Li J, Li B, Lu Y. Neural video compression with diverse contexts[C]. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 22616-22626.

[2] Sheng X, Li L, Liu D, et al. VNVC: A Versatile Neural Video Coding Framework for Efficient Human-Machine Vision[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
paper_summary: paper_summary: Inspired by traditional compression approaches, the authors creatively introduced the spatial-temporal implicit entropy model to reduce redundant information. An adaptive quantization strategy further enhances the performance of the framework. Additionally, due to the well-designed framework, it can be used for other explicit NeRF-based models.
strength_weakness: strength_weakness: 1. Authors have successfully identified and distilled the key ideas of the paper and systematically validated them through experiments. In particular, categorizing the main ideas has been very effective and added clarity to the overall paper.
2. The method is novel and experiments demonstrate that the proposed framework achieves superior performance while also exhibiting a degree of applicability to other explicit NeRF-based models.
limitations: limitations: 1. Missing ", "or ". "at the end of all formulas in the paper.
2. The description of Figure 4 is not clear enough.
3. It is regrettable that validation has been only limited to performance aspects. In this regard, more diverse experimental validations are needed, such as comparing training time with baseline.
4. It is regrettable that the framework has not been validated on other types of datasets like DyNeRF.