NVRC: Neural Video Representation Compression
Authors: Ho Man Kwan
Ge Gao
Fan Zhang
Andrew Peter Gower
David Bull
Venue: NeurIPS 2024 poster (2024)
Keywords: Video compression
Implicit neural representations
Primary area: machine_vision

Abstract: Recent advances in implicit neural representation (INR)-based video coding have
demonstrated its potential to compete with both conventional and other learning-
based approaches. With INR methods, a neural network is trained to overfit a
video sequence, with its parameters compressed to obtain a compact representation
of the video content. However, although promising results have been achieved,
the best INR-based methods are still out-performed by the latest standard codecs,
such as VVC VTM, partially due to the simple model compression techniques
employed. In this paper, rather than focusing on representation architectures, which
is a common focus in many existing works, we propose a novel INR-based video
compression framework, Neural Video Representation Compression (NVRC),
targeting compression of the representation. Based on its novel quantization and
entropy coding approaches, NVRC is the first framework capable of optimizing an
INR-based video representation in a fully end-to-end manner for the rate-distortion
trade-off. To further minimize the additional bitrate overhead introduced by the
entropy models, NVRC also compresses all the network, quantization and entropy
model parameters hierarchically. Our experiments show that NVRC outperforms
many conventional and learning-based benchmark codecs, with a 23% average
coding gain over VVC VTM (Random Access) on the UVG dataset, measured
in PSNR. As far as we are aware, this is the first time an INR-based video codec
achieving such performance.

NVRC: Neural Video Representation Compression
Ho Man Kwanf, Ge Gaot, Fan Zhangi, Andrew Goweri, David Bullf Visual Information Lab, University of Bristol, UK
Immersive Content & Comms Research, BT, UK
{mm. kwan, gaa, fan zhang, dave. buvl. bull))bristool.. ac ac.uk, andrew, p. gowerbbt.com
Abstract
Recent advances implicit neural representation (INR)-based video coding have demonstrated its potential to compete with both conventional and other learning- based approaches. With INR methods, a neural network is trained to overfit a video sequence, with its parameters compressed obtain compact representation the video content. However, although promising results have been achieved, the best INR- methods are still out-performed by the latest standard codecs, such as VVC VTM, partially due to the simple model compression techniques employed. In this paper, rather than focusing representation architectures, which is a common focus in many existing works, we propose a novel INR-hased video compression framework, Neural Video Representation Compression (NVRC)) targeting compression of the representation. Based on its novel quantization and entropy coding approaches, NVRC the first framework capable of optimizing an INR-based video representation in a fully end-to-end manner for the rate-distortion trade-off. To further minimize the additional bitrate overhead introduced by the entropy models, NVRC also compresses all the network, quantization and entropy model parameters hierarchically. Our experiments show that NVRC outperforms many conventional and learning-basi based benchmark codecs, with a 23% average coding gain over VVC VTM (Random Access) on the UVG dataset, measured in PSNR. As far are aware, this is the first time an -Ns-b video codec achieving such performance.
Introduction
30 has demonstrated its significant potential to compete with conventional video coding standards, with some recent contributions (e.g.. DCVC-DC (36)) reported to outperform the latest MPEG standard codec, VVC VTM [10]. However, learning-based codecs are typically associated with high computational complexity, particular at
In recent years, learning-basse based video compression [4 34
36
13
the decoder, which therefore limits their practical deployment. To address this, new type of learned video codec has been proposed, based on implicit neural representation (INR) models [13 30), where each INR instance is overfitted and compressed to represent video sequence (or video dataset). INR-based codecs enable much faster decoding speed compared to most non-INR learning based coding methods, and do not require offline optimization due to its overfitting nature. Although they have shown promise, based codecs are yet to compete with state-of of-tee---the-rn art conventional and learned video coding methods in terms of rate-distortion performance.
To enhance coding performance, it noted that most recent -bR-based video coding methods [3 30 focus on improving network architectures but still perform simply model pruning, quantization and entropy coding to obtain compact representations. Morcover, these methods are not fully end-to- end optimized; for example, NeRV [3) and HiNeRV 30 are not trained with a rate-distortion
Project page: https://hmka. hwwpinnk....g.uu.......e htt.ae/..iittbbbbooree.ee.e io/nvrc/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CI
Hi PSWR 322
POVR, 0.2333 Bop
HiN682- 32.75 PSMA, 0132 apo
MWC PSWR, C300 bap
Figure 1: Comparison between the output from HiNeRV 30 and the proposed NVRC. The image is from UVG dataset (Jackey/ReadySetGo sequence)
45
objective but only perform fine-tuning with pruning and quantization applied. Although COOL-CHIC [31 and C3 [29] are almost end-to-end optimized, the rate consumed by their entropy model and decoder/synthesis networks does not contribute to the training process. In contrast, ootte-of-the-ar art
non-INR learning- based codecs [34] (35] are typically end-to-end trained with advanced entropy models, and this contributes to their improved coding performance compared INR-based methods.
To address this issue, this paper proposes a new framework, referred to as Neural Video Representation Compression (NVRC). Unlike other INR-based video codecs, NVRC is enhanced framework for compressing neural representations, which, for the first time, enables INR-based coding methods to be fully end-to-end optimized with advanced entropy models. particular, NVRC groups network parameters and quantizes them with per-group learned quantization parameters. The feature grids are then encoded by a context-based entropy model, where the network layer parameters are compressed by dual-axis conditional Gaussian model. The quantization and entropy model parameters are further compressed by lightweight entropy model to reduce their bit rate consumption. The overall rate of the parameters from the INR, quantization, and entropy models is optimized together with the representation quality. NVRC also utilizes a refined training procedure, where the rate and distortion objectives are optimized alternatively to reduce the computational cost. The primary contributions of this work are summarized below.
The proposed NVRC is the first fully end-to-end optimized INR-based framework for video compression. In NVRC, neural representations, well as quantization and entropy models, are optimized simultaneously based on a rate-distortion objective.
Enhanced quantization and entropy models have been applied encode neural representation parameters, where the context and side information have been utilized to achieve higher coding efficiency.
new parameter coding method based on a hierarchical structure has been introduced which allows NVRC to minimize the rate overhead. The parameters from quantization and entropy models for encoding the neural representation, are all quantized and coded with learnable parameters.
4. NVRC features an enhanced training pipeline, where the rate and distortion losses are optimized alternatively, to reduce the computational cost of advanced entropy models.
We conducted experiments to compare the proposed approach with state-of- of-e--of-he-arr conventional and learning- based video codecs on the UVG (5)). MCL-JCV [59 and JVET-CTC Class B [9] datasets. To enable a fair comparison, we use both the RGB444 (like most learned video codecs) and YUV420 (like standard video coding methods) configurations. The results demonstrate the effectiveness of NVRC, which achieved up to 23% and 50% BD-rate savings when compared the latest MPEG standard codec, H.266/VVCC VTM-20.0 (Random Access) (II), and the state-of-ther of-tthe-rt INR- based codec, HiNeRV [30], respectively. To our best knowledge, NVRC is the first INR-based video codec outperforming VVC VTM with such significant coding gains.
2 Related work
2.1 Learning-based video compression
Video compression an important research topic that underpins the development of many video- related applications, such video streaming, video conferencing, surveillance, and gaming. In

---

the past three decades, multiple generations of video coding standards [60, [6, 10 have evolved by integrating advanced coding techniques. Recently, learning-basee video compression emerged as an popular alternative due to its strong expressive power and the ability to be optimized in
data- and metric-driven manner. Neural networks can be combined with conventional codecs for performance enhancement 42 [4] or used to build end-to-end optimized frameworks. DVC [40] first
proposed to replace all modules in conventional codecs using neural networks. uollow-up innovations
include those improving motion estimation [38 2 26 [6), applying feature space conditional coding
[27 [4]] optimizing context modeling in terms of performance and efficiency [23] [2 68], and
adopting novel architectures such as normalizing flows [25], transformers [61
441 etc. In addition
to these architecture modifications, improvements to quantization-ina -nvollvvd optimization have also
been achieved to handle the non-iifferentiability caused by hard thresholding operations [29] 3 19].
41 39 have validated the effectiveness of adapting a model to individual image of video sequence via iterative refinement to reduce the amortization gap
Moreover, several studies 17 51 28
63 588 and optimize bit allocation over a sequence of frames [62]. Despite demonstrating impressive distoiitootion performance, with some recent advancements reporting outperformance of VVC [10), neural video codecs are generally too computationally intensive 48 thus limiting their adoption in practical applications.
2.2 INR-based video compression
Implicit neural representation (INR) 52 is an emerging paradigm for representing multimedia data, such as audios [55], images [4 54, videos 13 and 3D scenes [46 49). This type method exploits the mapping from the coordinate inputs to high-dimensional feature space and aims to output the corresponding target data value that location. Neural representation for videos (NeRV) [3] has been proposed to model the mapping from frame indices to video frames, showing competitive reconstruction performance with very high decoding speed. When applied to video compression, the network parameters of these models are compressed through pruning, quantization, and entropy- penalization 220
16 I8 65 to achieve high coding efficiency. The following contributions further investigated patch- patch-wise (5], volume-wis wise [43]. or spatial-tempon disentangled representations [37] to improve representational flexibility. There are also methods that explicitly model the volume- wise residual [43] frame-wise residual [67], or flow-based motion compensation [66, 32 21 33)
to enable scalable encoding and representation of longer and more diverse videos. In addition to these index-based approaches, other work has exploited content-specific embeddings/feature grids to provide visual prior for the network. The embeddings may be associated with single [12] or multiple resolutions 32
30 29 33). Although they hold promise in terms of low decoding complexity and
competitive performance, all these aforementioned IbR-based video codecs are still outperformed by
state-of-the-ar conventional (e.g., VVC VTM (T0) and autoencoder-based
35 36 video codecs.
Method
Figure 2 shows the proposed NVRC framework, which follows workflow similar existing b video compression methods, such as 13 30 29]. but with a more advanced model compression pipeline. It trains neural representation for given video(s) and utilizes model compression techniques to obtain the compact representation (with compressed network parameters) of the video(s). Specifically, in NVRC, for target video sequence V with T frames, height H, width W. and C' channels, i.e., Val e xx X xC a neural representation F' parameterized by 0 is trained to map coordinates pixel intensities such as RGB colors, in a patch-wise manner. This can be represented by:
Vpatch = k)),
(I)
where j, k is the patch coordinates. Vpatch E Trrrr Hporch x xW, patch X is the corresponding W
H. and 0 S k < Tratch. This formulation video patch, in which 0- < Wpatch 0 Hpatch
T_
(the same as in (00) generalizes different frameworks: when (Tpatch: Hpatch, Wpatch) = (1, 1), the neural network maps coordinates individual pixels [52) when (Tpatchs Hpatchs W patchh) (1, H. W), the network maps coordinates to video frames [3]
As mentioned in Section I existing INR-based video codecs either split the training of the INR model and model compression 13
30 or train the model with compression techniques applied, but
not entirely in an end-to-end manner 29 33 Unlike these works, NVRC employs more advances
3

---

(f,2.8)
Scaling
Scaling
Scaling
INR Grids
Coar
NK Layers
AE
Contestbant Gauppeen
AE
AE
Dusionst
AD
Frwes
Unscaliin
Apc
Aquar
Equand ( Bugge (apereeee.)
Super
Oiuee Maggeeeeeee.
Saper
Scaling for 0 Ogrids
Scaling for Ouager
Mayer
duc (or.eeet
dves Hagger Happerani Disperant Plager
8yal (ooe
Plysh Fyri
Plaeer Plager
Higgeryn
Playmaatt
Context-based Gaussian model
Dual-axis Conditional Gaussian model
Figure 2: In NVRC. the parameters encoded in hierarchical structure, where (Middle-left) per-block quantization scales and (bottom-left) context-based model are utilized for encoding feature grids, and (Middle-right and bottom-right) per-axis quantization scales and dual-axis Gaussian model are applied for encoding network layer parameters.
entropy models and allows fully end-to-end optimization. Specifically, neural representation F contains a set of learnable parameters 0 including feature grids parameters (fgrid) and network layer parameters (Otayer). To quantize and encode these parameters, quantization and entropy models are employed with learnable compression parameters Φ {é }.m J. o is determined by the distribution of the representation parameters scales, and can be considered as side information in compression [7]. While these parameters do improve overall coding efficiency, the introduced overhead is not negligible, particularly when the compression ratio of the representation parameters is high. Therefore is also quantized in this work, denoted as & { Pemb) }, and entropy coded. Here the quantization and entropy coding are performed based on another set of learnable parameters /i = {Viquant vem.). which can be simply quantized into full/half precision as = f&gqunee emm J. This forms a hicrarchical coding strategy for encoding these model and compression parameters
fine-grained manner, e.g., per-group quantization
and v) as illustrated in Figure 2
All these parameters are optimized in a fully end-to-end manner based on a rate-distortion objective. Here the distortion metric D, e.g., mean-square-erro (MSE), is calculated between reconstructed video patch, Vpatchh Fg [i,j. k). and the corresponding target video patch Vg ypatch. The rate R is based on the number of bits consumed by the three levels of quantized parameters 0, G, &.
3.1 Feature grid coding
Although employing feature grids convergence rate and reconstruction quality, these typically rely a large number of parameters,
30 29 3
for neural representations improves both

---

which could potentially challenge model compression techniques. To address this issue, related works utilize multi-resolution grids to improve parameter efficiency and/or perform entropy coding for the
feature grid compression 12 32
30
29
33
31
To improve feature grid encoding, in NVRC the parameters first partitioned into small blocks, for which different quantization parameters are applied to improve the encoding efficiency. Context-bascl entropy models are utilized with parallel coding
3D block partitioning. For a feature grid = in grrid, with 2 E I Hgrid ii x W Werid Cgrsa, it is divided into blocks with size of Tulk x Hork x Woik X Cgrid (padding is applied if the grid size is
not divisible) and produced Terid x Marid W Warid blocks, For multi-scale grids [32 30 33
Thie HHik x
Wazk
the
same block size is applied to partition the grids at each scale.
Quantization. With the partitioned blocks, a transformation is then applied before quantization.
Here, the corresponding per-block quantization scales ógrid,blk from Oquant are utilized, where Farid SgridMi E R Pak
W Wordd XCyria is learnable, and all the scales in ògrid,M are shared by features in the same channel and in the same block in z. To achieve this, Sgrid,blk is first expanded to garid, which has the same shape as and such that:
x Marid
h
8gria (L, h, w, cl = Sgrid,bx Tann
Task" J. Haak WAEK
J.c. J.c.
(2)
where 0 < Tgrid. 0 S h < Hg Horid. 0 S < Worid and O S < Cgrid- In practice, the logarithm of Ogrid, MR learned, instead of 6 ggiid,bik which ensures that the scales are non-negative.
Following 16
29
65
31
the scaling and quantization can then be computed by:
(3)
Orrid
in which 2" represent the scaled parameters, 8 denote the quantized 2".
The corresponding unscaling operation defined by:
2 = 2² x ggrid,
(4)
to obtain the final quantized parameters 8.
Context-based entropy model. Although entropy coding has been used for reducing the bit rate consumed by feature grids, many implementations are only based on simple entropy models and treat the grids as ordinary network parameters [12
30 31. A better solution is to exploit the spatial-temporal redundancy within the feature grids, due to the inter dependent nature of the features. COOL-CHIC [3 and C3 (29) utilize context-basee based model and achieve efficient feature grid encoding; however, these methods are not associated with optimal rate distortion performance due to their low complexity constraints, and the use of grid entropy models for INR-based video compression has not been fully explored the literature.
32
To enhance the efficiency of feature grid coding, NVRC employs context- based Gaussian models with nuto-regressive style encoding and decoding processes (47) exploit spatial-temporal redundancy within feature grids. While auto-regressive style coding is sequential, in NVRC, the feature grids at different resolutions are coded independently. Moreover, as mentioned above, each grid is partitioned into many small 3D blocks, which are also coded in parallel. Thus, the context model in NVRC has high degree of parallelism, which enables fast coding despite of reduced amount of available context. The context-based model employs 3D masked convolution 47), with parameters Woontest from dem. which are applied to all blocks from the same feature grid, but are not shared between grids at different scales. The context model estimates the means Agrid and scales Gorid for the Gaussian distribution in per-feature manner and uses coder such as the arithmetic coder to code the parameters, i.e., g²_ Here, the estimated means Pyrid and scales o grid will also be scaled by the corresponding quantization scale dgrid before applying for encoding and decoding.
3.2 Network layer parameter coding
Unlike the feature grids in INR models, the network parameters, such as the weights in linear and convolutional layers, are difficult to compress there is spatial temporal correlation between
5

---

parameters.
Existing works typically use simple entropy models for encoding these parameters
13
30
16
33
65].
2D block partitioning. In the proposed NVRC framework, similar to feature grids, network layer parameters are also partitioned into groups prior quantization and coding. Here assume that the encoding of the parameters from the same input/output fcatures/channels could benefit from sharing quantization and entropy coding parameters. For example, an input feature/channel zero, then the corresponding group of parameters are likely to be zeros as well. In NVRC, the quantization and entropy coding models for network parameters are designed based on this assumption, and aim to share the quantization and entropy models between parameters in the same row/column. Since there are parameters with different numbers of dimensions, e.g., 2D for linear layer weights and 4/5D for convolution weights, all layer parameters in NVRC are first reshaped into 2D tensors, where the parameters in a row correspond to the weight from the same output feature/channel, and the parameters in a column are the weights for the same input feature/channel. While existing works can be directly employed on partitioned parameter tensors by rows or columns, and applied per-row per-column entropy parameters for coding. this may not be the best solution, because (i) the partitioning axis needs to be decided, (ii) the coding could benefit from sharing quantization and entropy parameters across both rows and columns. Therefore, in NVRC, the tensors are partitioned according to both axes at the same time, and the quantization and entropy parameters are learned in both axes and mixed during coding. We noticed that existing work has utilized quantization parameters on both input and output channels in different contexts [1]]. Here, entropy parameters are also utilized, and both the quantization and entropy parameters are further compressed.
13 30
Quantization. For the weights of a layer, Wlayers from Olayers Wlayer E C the quantization scales Stayer IR CoCo combined by two vectors of scales òtager,out E RCoue and òiager,in RCi.. such that:
ōtayer [i, Stager,outli [i] x ōtayer, Li],
(5)
where 0 S i < Cout and 0 S j < Cin. In practice, only the logarithms of ótayer.out and Ótayer, in stored, and quantization is performed similarly to the grid parameters (Section 3.1.
Dual-axis conditional Gaussian model. In NVRC, a dual-axis conditional Gaussian model is used for coding the network layer parameters. Similar to the quantization parameters mentioned above, the means Plager and the scales Olayers are represented in two per-axis parameter vectors, i.e. Maayer,outs
Mlyyer,in ouy and Olayer,nns and they are both from dem-
The combined means Player and scales layer are obtained by MMgger (i = Miager,ont [i] x Olayer,in + Maayeran [
and
Ttgger [1 = Ilayeroout [i]
ongyer,inlll
where 0 S Sic Cout, 0 S < Cin- Like the quantization parameters diagers only the per-axis means ouy Mlayeriins and the logarithms of the per-axis scales Mlyeeroous Oayerrin are stored. Finally, the means and scales here will also (as for feature grids) be scaled by dayer, before being utilized for coding Wieyer.
3.3
Coding of entropy model parameters
Since our use of more advanced quantization and entropy models will introduce additional bit rate overhead, the quantization parameters Oquant and entropy model parameters Pem are also quantized into Φquant and &em and entropy coded. Here the same (as for feature grids in Section 3.1 scaling and quantization operation applied, in which a conditional Gaussian model used, except that the quantization scales and the means/scales for the Gaussian distribution are learned in a per-tensor manner.
3.4 Rate-distortion optimization
Combined loss for NVRC. In NVRC, the overall loss function is given below:
L = R + XD
(8)

---

Here D stands for the distortion calculated between the reconstructed content and the original input. R is the total bitrate (bits/pixel) consumed by the quantized representation parameters 9, quantized compression parameters é. Specifically
8-1 Σ loga (0g((()))))
161 loga (p ( (n]))
R= Ranr + Rem TX X
Σ
By jointly optimizing different parameters with the combined rate-distortion loss, the trade-off' between the rate and the reconstruction quality can be achieved.
Alternating optimization. In existing INR-based video representations and compression methods 13 [30], the distortion loss minimized iteratively with sampling batches of frames, patches or pixels. To introduce the entropy regularization, this process has been extended [2 65 where the rate loss is also calculated each training step, similar to other learning- video compression methods 40, 34). However, in the INR-based video compression, the training is the process for over-fitting the network to a video sequence, in which the samples of each steps are from the same sequence, and the code, i.e., the INR model parameters, is the same set of parameters for all steps. Thus, it is not necessary to update the rate term in every step, especially when a significant amount of computation memory is needed for this due to the use of entropy models. In NVRC, more efficient training process is used, where the rate R and distortion D are optimized alternately. In every K + steps, the D is minimized in the first K steps, and where R is minimized at the K 1-th step. Empirically, the rate loss is also scaled by K to keep the rate roughly the same. Note that, the quantization step is still applied each step, and skipping the entropy model is only possible when the quantization parameters are separated from the entropy model.
Two-stage training. Similar to some existing works [3 30 29]. NVRC is also trained in two stages.
In Stage 1, to optimize L, the non-differentiable quantization operation needs be emulated through a differentiable approximation during training. Recent work 22) has shown that a soft-rounding operation with an additive Kumaraswamy noise can be used to replace quantization for neural representation training. While in [29], this is applied only to feature grids, we extend this idea and apply it to both feature grids and network parameters in the first stage of training. Compared (29]. soft-rounding with higher temperature (0.5 to 0.3) is used in NVRC, as the original, low temperature (e.g. 0.3 to 0.1 in (29)) for both feature grids and network parameters will lead to training difficulty due to the large variance of the gradients.
In the second stage, instead of using soft-rounding, following [3 31 Quant-Noise (53) is used to fine-tune the neural representation, as we empirically found that Quant-Noise remains stable with different hyper-parameter settings and is suitable for high quantization levels.
Experiment
4.1 Experiment Configuration
Evaluation database. To evaluate the performance of the proposed NVRC framework, we conducted experiments on the UVG (45) and MCL-JCV [59] dataset. The UVG dataset includes 7 video sequences with 300/600 frames, while the MCL-JCV dataset consists of 30 video clips with 120-150 frames. All sequences are compressed at their original resolution this experiment. We also provide the result of JVET-CTC dataset Class B 19] in the Appendix.
Implementation details. NVRC a new framework focusing on INR model compression, which can be integrated with any typical INR-based models. To test its effectiveness, we employed one of the latest INR network architectures, HiNeRV [30], and integrated into our NVRC framework. This INR model has been reported to provide competitive performance compared to many standard and end-to-end codecs for the video compression task. Minor adjustments have been made on top of HiNeRV in terms of the network structure and the training configuration (see Appendix for details)- rate points are now obtained by both turning the scale of the neural representation and the value. The model is trained for 360 or 720 epochs in the first stage and 30 or 60 epochs in the second stage, depending on the UVG 45 and MCL-ICV (59) datasets, due to the differing lengths of the sequences.
7

---

Table BD-rate results on the UVG dataset 445
Color Space
(veryslow)
(RA))
(RM(
DCVC-HEM
DCVC-DC
HiNeRV
HNckY-Boost
RGB 4:4:4
MS-SSIM
PSNR
-73.74%
-80.65%
46738%
-0.38%
-49.75%
-23.42%
-00.7%%
-3..20%
11.75%
50.16% 44.27%
66.86% -76.99%
-78.01%
-66.4%%
YUV
PSNK
66.89%
42 50%50%
-12.96%
-3.988
MS-SSIM
-59.38%
-15.04%
-39.12
Table 2: BD-rate results on the MCL-JCV dataset [39].
Color Space
Metric
x265 (veryulow)
HM (RA)
(RA)
DCVC-HEM
DCVC-DC
HiNeRV
C3
HNNeRV-oool
PSNR
-1.61%
-13.88% -11001
36.91% -6.39%
-2.9% 21.6%%
13.40%
-31. 6909
42.23% 49.239
-59,80%
MS- SSIM
66.83%
33,07%
-4162%
-83.35%
YUV 4:2.0
PSNR MS- SSIM
-43.00%
49.022%
-13. 1996
-12.61%8
40,80%
33.09%
-9.89%
3.75%
UVS 108466
UVS RC8446
VUM420
(MS V1M420
25.6
23.8
360
Eduper plrd Dey!
846 2001
pinc 2001
Eats por clee Stad
Figure 3: Average rate quality curves of various tested codecs on the UVG dataset 45
MCL-EV
MCL
MCL 9LV420
NCL-CCV-
12.00
C530
8330
1000 E.75 [th eiaim floe
Shoorppne Reel
14o mon-oommeeeoo 30me1
Bh-ampphe Shee
Figure 4: Average rate quality curves of various tested codecs on the MCL-JCV dataset 59
Benchmark methods. Conventional codees, x265 [ with the veryslow configuration, HM-18.0 (50 and VTM-20.0 [1]] with the Random Access configuration, are used for benchmarking. together with two recent learned video codecs, DCVC-HEM (35], DCVC-DC [36]. Three state-of-the-art
INR-based codecs, including the original HiNeRV (30, C3 [29] and HNeRV-Boost [65] have also
been included in this experiment. All results are produced by the open source implementations.
Evaluation methods. The evaluation was performed in the RGB color space (for comparing both conventional codecs and learning-based methods) with the BT.601 color conversion, and in the original YUV420 color space (for comparing both conventional methods and the learning-baa based methods that support this feature in their public implementations). PSNR (RGB/YUV 6:1:1) and MS-SSIM (RGB/Y) are used here to assess video quality, based on which Bjøntegaard Delta Rate figures are calculated against each benchmark codec.
4.2 Results and discussion
Figure 3 and Table 12 provide the results for the proposed NVRC model and the benchmark methods. It can be observed that when tested in the RGB 4:4:4 color space (as in many learning- based works), NVRC significantly outperforms the original HiNeRV model (30, with an average coding gain of 50.16% measured by PSNR. Similar improvement has also been achieved against other INR-based methods including HNeRV-Boost [65] and C3 29). Moreover, NVRC also offers better performance compared to latest MPEG standard codec VVC VTM (Random-
[]] on the UVG dataset 455. with a 23.4% average coding gain based on PSNR. To the best of our knowledge, is the first based video codec outperforming VTM. Compared to state-of-the-an learned video coding methods, NVRC also exhibits superior comparable performance to DCVC-HEM (35 and
8

---

Table Complexity results of NVRC with the UVG dataset 445
Encoding and decoding FPS are measured by the number of training steps/evaluation steps per second performed by the INR model with 1080p inputs/outputs. The model compression MACs and encoding/decoding time are measured by the steps for performing quantization and entropy coding. The complexity figures are calculated based on NVIDIA RTX 4090 GPU with FP16.
Rate point
Frame MACs/Enc FPS/Dec FPS
Model compression MACs/Enc time/Dec time
1-2
359.6G/6.4/21.0 842.8G/3.6/15.1
25.2G//2.9w/37..0 .s 50.4G/29 69/G//99..s/4.88.
3-4
5-6
1929.0G/2.2/9.77
100.8G/43.44/53.75
Table 4: Ablation studies on the UVG dataset 45
Results are BD-rates.
Metrie
NVRC
(V1)
(V2)
(V3)
(V4)
(V5)
PSNR
0.00%
13.04% 13.84%
11.06%
23.37%
30.84%
14.42%
MS-SSIM
0.00%
10.24%
23.88%
30.07%
8.54%
DCVC-DC 336 on the UVG 445 and MCL-JCV [59] datasets, respectively.. When evaluated in the YUV 4:2:0 color space. NVRC still offers superior performance as for RGB 4:4:4 color space, outperforming most benchmarked methods based on PSNR and MS-SSIM. It should be also noted that INR- based video codecs do not require offline training large-scale datasets, whereas other learning-baad based methods do. Qualitative results are provided Figure T in terms of visual comparison between the content reconstructed by NVRC and HiNeRV.
4.3 Computational complexity
The complexity figures of NVRC with the UVG dataset [5] are provided in Table When compared the original HiNeRV [30 the proposed method (with HiNeRV as its INR network) is associated with increased computational complexity. However, the MACs figure is still significantly lower than that of other learning-ban video codecs (e.g.. DCVC-DC (36)), which allows faster decoding. should be noted that the complexity figures shown here are obtained based on research source code that has not been optimized for latency. The actual latency of INR and entropy coding can further reduced by (1) optimizing the implementation of the INR and entropy models, (2) performing lower precision computation, and (3) implementing parallel decoding between different resolution feature grids.
4.4 Ablation study
To evaluate the contribution of the main components in NVRC, ablation study performed based on the UVG dataset [45), using the configurations Section 4.1 but four rate points for each variant.
Alternative entropy model settings. We compared different combinations of entropy models for encoding feature grids 0 grid and network parameters Otayer: Context model dual-axis conditional Gaussian model (Default setting in NRVC), (V1) Context model per-tensor conditional Gaussian model, (V2) per-tensor conditional Gaussian model dual-axis conditional Gaussian model, (V3) per-tensor conditional Gaussian model + per-tensor conditional Gaussian model.
Hierarchical parameters coding. In NVRC, the quantization parameters @quont and the entropy model parameters dem are also entropy coded. To verify this, we created another variant (V4) with Quuntt and 6em not coded but stored in half-precision
Learned quantization steps. We also compared the of learned quantization steps Qquant and Vquant (Default setting in NRVC) and (V5), a new variant with fixed quantization steps for grids, where the log-step size is -4.
Table 4 shows the ablation study results, in terms of BD-rate values against the original NVRC. These figures confirmed the contribution of the tested components in the NVRC framework.

---

In addition, we conducted experiments to evaluate the effects of fully end-to-end optimization and alternating optimization on selected challenging sequences from the UVG dataset (Jockey and ReadySerGio) [45] When removing fully end-to-end optimization, the variant without rate loss in the first stage exhibits up to 35% BD-rate increase compared to the proposed model. However, this loss diminishes if the number of epochs in the second stage increases. With the proposed alternating optimization, we did not observe any noticeable difference in performance. Nevertheless, without alternating optimization, the training step time can increase by up to 40% under our experimental settings.
Conclusion
In this paper, present NVRC, new INR-based video compression framework with focus representation compression. By employing novel entropy coding and quantization models, NVRC significantly improved coding efficiency and allows real end-to-end optimization for the INR model. The experimental results show that NVRC outperforms all the benchmarked conventional and learning-based video codecs, in particular with 23% bitrate saving against VVC VTM (Random Access) [11] on the UVG database 45 this achievement.
This is the first time an INR-based video codec has obtained
Acknowledgments and Disclosure of Funding
This work was supported by UK EPSRC (iCASE Awards), BT. the UKRI MyWorld Strength in Places Programme and the University of Bristol. Part of the computational work was also supported by the facilities provided by the Advanced Computing Research Centre at the University of Bristol.
References
[1] x265. httph::/ wwww:///c/wvvieclnn http://www..........rrrreeeerppp 2266 html
[2] E. Agustsson, Minnen, Johnston. J. Balle, Hwang, and G. Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8503-8512, 2020.
[3] E. Agustsson and Theis. Universally quantized neural compression. Advances in neural information processing systems, 33:1 33:12367-12376, 2020.
[4] L. J. Ba, J. R. Kiros, and E. Hinton. Layer normalization. CoRR, abs/1607, 100.0600706450, 2016.
[5] Y. Bai, C. Dong, C. Wang, and C. Yuan. Ps-nerv: Patch-wise stylized neural representations for videos. In
2023 IEEE International Conference Image Processing (ICIP), pages 41-45. IEEE, 2023.
[6] J. Ballé, N. Johnston, and D. Minnen. Integer networks for data compression with latent-variable models.
In ICLR (Poster). OpenReview.net, 2019.
[7] J. Ballé, D. Minnen, S. Singh, S. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In ICLR. OpenRevicw.net, 2018.
[8] Y. Bengio, N. Léonard, and A. C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308 3308.343,, 2013.
[9] J. Boyce, K. Suchring, X. Li, and V. Seregin. Jvet-j1010: Jvet common test conditions and software reference configurations. In 10th Meeting of the Joint Video Experts Team, pages JVET J1010, 2018.
[10] B. Bross, Y. Wang, Ye, S. Liu, Chen, G. J. Sullivan, and Ohm. Overview of the Versatile Vidco Coding (VVC) Standard and its Applications. IEEE Trans. Circuits Syst. Video Technol., 31(10):33363374, 2021.
[11] A. Browne, Y. Yc, and S. H. Kim, Algorithm description for Versatile Vidco Coding and Test Model 19 (VTM 19). In the JVET meeting, number JVET-AC2002. ITU-T and ISO/IEC, 2023.
[12] H. Chen, M. Gwilliam, S.-N. Lim, and A. Shrivastava. Hnerv: A hybrid neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10270-10277 2023.
10

---

[13] H. Chen, B. He, H. Wang. Y. Ren, S. Lim, and A. Shrivastava. NeRV: Neural Representations for Videos. In NeurlPS. pages 21557-21568, 2021.
[14] E. Dupont, A. Goliáski, M. Alizadeh, Y. W. Tch, and A. Doucet. Coin: Compression with implicit neural representations. arXiv preprint arXiv:2103.0322 2021.
[15] A. Finkelstein, Fuchs, Tal, M. Grobman, N. Vosco, and E. Meller. QFT: post-training quantization via fast joint finetuning of degrees of freedom. ECCV Workshops (7), volume 13807 of Lecture Notes in Computer Science, pages 115-129. Springer, 2022.
[16] C. Gomes, R. Azevedo, and C. Schroers. Video compression with tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18497-18506, 2023.
neural representa-
[17] T. Guo, J. Wang. Z. Cui, Y. Feng, Y. Ge, and B. Bai. Variable rate image compression with content adaptive optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 122-123, 2020.
[18] Z. Guo, G. Flamich, J. He, Z. Chen, and M. Hernández-Lobato. Compression with Bayesian Implicit Neural Representations. NIS,, 36: 19:1998-1956, 2023.
[19] Z. Guo, Z. Zhang, R. Feng, and Z. Chen. Soft then hard: Rethinking the quantization in neural image compression. In International Conference on Machine Learning, pages 3920-3929. PM....,99....RR 2021.
[20] S. Han, H. Mao, and W. J. Dally. Deep compression: Compressing deep neural networks with pruning. trained quantization and huffman coding. arXiv preprini arXi::1570.00149 2015.
[21] B. He, X. Yang. H. Wang, Z. Wu, H. Chen, S. Huang, Y. Ren, S.-N. Lim, and A. Shrivastava, Towards scalable neural representation for diverse videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6132-6142, 2023.
[22] D. He Z. Yang. W. Peng, R. Ma, H. Qin, and Y. Wang. ELIC: Efficient Learned Image Compression with Unevenly Grouped Space-Channed Channel Contextual Adaptive Coding. In CVPR, pages 5718-5727, 2022.
[23] D. He, Zheng. Sun, Wang, and H. Qin. Checkerboard Context Model for Efficient Learned Image Compression. In CVPR, pages
2021.
[24] D. Hendrycks and K. Gimpel. Bridging nonlincaritics and stochastic regularizers with gaussian error linear units. CoRR, abs/ 60..18115...... 2016.
[25] Y.H. Ho, C.-P. Chang, P.-Y. Chen, A. Gnutti, and W.-H. Peng. Canf-ve: Conditional augmented normaliz- ing flows for video compression. In European Conference on Computer Vision, pages 207 -233. Springer, 2022.
[26] Z. Hu, G. Lu, J. Guo, S. Liu, W. Jiang, and D. Xu. Coarse-to-fine deep video coding with hyperprior- guided mode prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5921-5930, 2022.
[27] Z. Hu. Lu, and D. Fve; A new framework towards deep video compression feature space. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1502-1511, 2021.
[28] M. Khani, V. Sivaraman, and M. Alizadeh, Efficient video compression via content-adaptive super- resolution. In Proceedings of the IEEE/CVF international conference computer vision, pages 4521-4530. 2021.
[29] H. Kim, M. Bauer, L. Theis. J. R. Schwarz, and E. Dupont. C3: Hiigh-performancce and low-complexity neural compression from a single image or video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[30] H. M. Kwan, G. Gao, F. Zhang, A. Gower, and D. Bull, HiNeRV: Video compression with hierarchical encoding-basel neural representation. In NeurlPS, 2023.
[31] H. M. Kwan, F. Zhang, A. Gower, and D. Bull. Immersive video compression using implicit neural representations. CoRR, abs/2402.01596, 2024.
[32] J. Lee, D. Rho, J. H. Ko, and E. Park. Finerv: Floweguided wise neural representations for videos. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7859-7870, 2023.
11

---

[33] T. Leguay, T. Ladune. P. Philippe, and O. Déforges. Cool-chic video: Learned video coding with 800 parameters. arXiv preprint urXiv:2402.0317, 2024.
[34] J. Li, Li, and Y. Lu. Deep contextual video compression. In NeurlPS, pages 18114- 14-18125, 2021.
[35] Li, B. Li, and Lu. Hybrid Spatial- Temporal Entropy Modelling for Neural Video Compression. In ACM Multimedia, pages 1503-1511. ACM, 2022.
[36] Li, B. Li, and Y. Lu. Neural video compression with diverse contexts, In CVPR, pages 22616-22626. IEEE, 2023.
[37] Z. Li, M. Wang, H. Pi. K. Xu, J. Mei, and Y. Liu. E-nerv: Expedite neural video representation with disentangled saatial-temporal context. European Conference on Computer Vision, pages 267-284. Springer, 2022.
[38] J. Lin, D. Liu, H. Hi,, and F. Wu. M-LVC: Multiple frames prediction for learned video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3546-3554, 2020.
[39] G. Lu, C. Cai, X. Zhang. L. Chen, W. Ouyang, D. Xu, and Z. Gao. Content adaptive and error propagation aware doep vidco compression. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 456-472. Springer, 2020.
[40] G. Lu. W. Ouyang. D. Xu, X. Zhang, Cai, and Gao. DVC: an end-to-end deep video compression framework. CVPR, pages 11006-11000 11015. Computer Vision Foundation IEEE, 2019.
[41] Y. Lx, J. Xiang, Zhang, W. Yang, X. Han, and W. Yang. Dynamic low-rank instance adaptation for universal neural image compression. In Proceedings of the 31st ACM International Conference on Multimedia, pages 632-642. 2023.
[42] D. Ma. Zhang, and D. R. Bull. MFRNet: a new CNN architecture for post pooteescesssnng and in-loop filtering. IEEE Journal of Selected Topics in Signal Processing, 15(2)3378-387 2020.
[43] S. R. Maiya, Girish, M. Ehrlich, H. Wang, K.S. Lee, P. Poirson, P. Wu, C. Wang, and A. Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14378-14387,, 2023.
[44] F. Mentzer, G. Toderici, D. Minnen, S.JJ. Hwang, S. Caelles, Lucic, and E, Agustsson. Vet: A video compression transformer. arXiv preprint Sikiv:2206.073007, 2022.
[45] A. Mercat, M. Viitanen, and J. Vanne. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In MMSys, pages 297-302, ACM, 2020.
[46] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorti,, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99 100.999969991111111 2021.
[47] D. Minnen, J. Ballé, and G. Toderici. Joint autoregressive and hicrarchical priors for learned image compression. NeurlPS, pages 10794-10803, 2018
[48] D. Minnen and N. Johnston. Advancing the rate- deee-ittttton-ooppaatiii computation frontier for neural image com- pression. In 2023 IEEE International Conference on Image Processing (ICIP), pages 2940-2944, IEEE, 2023.
[49] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions graphics (TOG), 41(4):1-15, 2022.
[50] C. R. K. Sharman, R. Sjoberg, and G. Sullivan. High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Improved Encoder Description Update 14. In the JVET meeting, number JVET-AN5002. ITU-T and ISO/TEC, 2020.
[51] S. Shen, H. Yue, and J. Yang. Dec-adapter: Exploring efficient decoder-side adapter for bridging screen content and natural image compression. In Proceedingy of the IEEE/CVF International Conference on Computer Vision, pages 12887-12896 2023.
[52] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representa- tions with Periodic Activation Functions. In NeurlPS, 2020.
[53] P. Stock, A. Fan. B. Graham, E. Grave, R. Gribonval. H. Jégou, and A. Joulin. Training with Quantization Noise for Extreme Model Compression. In ICLR. OpenReview.net, 2021.
12

---

[54] Y. Strümpler. J. Postels, R. Yang, L. V. Gool, and F. Tombari. Implicit neural representations for image compression. In European Conference on Computer Vision, pages 74-91. Springer, 2022.
[55] K. Su, M. Chen, and E. Shlizerman. Inras: Implicit neural representation for audio scenes, Advances in Neural Information Processing Systems, 35:8144 8158, 2022.
[56] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Trans. Circuits Syst. Video Technol., 22( 1264111669-1666 2012.
[57] A. van den Oord, N. Kalchbrenner, I.. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves. Conditional image generation with pixelenn decoders. In NIPS, pages 4790-4798, 2016.
[58] T. van Rozendaal, I. A. Huijben, and T. S. Cohen, Overfitting for fun and profit: Instance-adaptive data compression. arXiv preprint arXiv:2101.08683 2021.
[59] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang. I. Katsavounidis, A. Aaron, and C. J. Kuo. MCL-JCV: JND-based H.264/AVC video quality assessment dataset. In ICIP, pages 1509-1513, IEEE, 2016.
[60] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and Luthra. Overview of the H.264/AVC Video Coding Standard. IEEE Trans. Circuits Syst. Video Technol. 13(7):560-577 0766600--,,277 2003.
[61] J. Xiang, K. Tian, and J. Zhang. Mimt: Masked image modeling transformer for video compression. In The Eleventh International Conference on Learning Representations, 2022.
[62] T. Xu, H. Gao, C. Gao, Y. Wang, D. He, J. Pi, Luo, Z. Zhu, M. Ye, H. Qin, ct al. Bit Allocation using Optimization. In ICML, pages 38377-38399, PMLR, 2023.
[63] Y. Yang, Bamler, and S. Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33:573-584, 2020.
[64] F. Zhang, C. Feng, and D. Bull. Enhancing VVC through enn enn-based post-prose processing. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2020.
[65] X. Zhang, Yang, D. Hc, X. Ge, T. Xu, Y. Wang, H. Qin, and J. Zhang. Boosting neural representations for vidcos with a conditional decoder. arXiv preprint Xrx::22002 18/52, 2024.
[66] Y. Zhang, van Rozendaal, Brehmer, M. Nagel, and Cohen. Implicit neural video compression. CoRR, abs/2112.11312, 2021.
[67] Q. Zhao, M. S. Asif, and Z. Ma. Dnerv: Modeling inherent dynamics via difference neural representation for vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2031-2040, 2023.
[68] X. Zhou, C. R. Qi, Y. Zhou, and Anguelov. RIDDLE: Lidar Data Compression with Range Image Deep Delta Encoding. In CVPR, pages 17212-17221, 2022.
13

---

A Appendix
A.1 Additional experiments
Table 5: BD-rate results on the JVET-CTC Class dataset 19).
Color Space
Metric
x265 (veryslow)
HM (RA)
VTM (RA)
DCVC-DC
PSNR
-68.70%
-3.533%
7.48%
-1385%%
RGB 4:4:4
ISIM
-84.69%
-65.51%
-2.60%
-18.81%
PSNR
-66.33%
-34.49%
9.57%
-24.25%
YUV 4:2:0
MS- MS-SSIM
-66.02%
-39.99%
-6.79%
-36.94%
[AlT ETC Class D- SGH44
NET Cleo B-RGD44
NET CTC Clasa YUW470
NET CITCIC Cloa B-YL9420
28
0.50
40.5
0.993
0.955
0393
6375
30
0.5
1022 E32 5033
837 $T60
0663 5325 320 0330
8:30 13005 D231 airpoet 2401
DID' 120 5.25 m188
Recperpo Rei
Figure 5: Average rate quality curves various tested codecs the JVET-CTC Class B datasets
In addition to the main paper experiments, here we provide the results of NVRC with the JVET-CTC test sequences for VTM 99. Similar experiments configurations as in the main paper have been used, where x265 [1] with the configuration, HM-18.0 (50) and VTM-20.0 [1] with the Random Access configuration, and DCVC-DC (6) have been used as the baseline models. Similar to the primary experiment results, our proposed NVRC consistently outperforms the baseline models in most cases.
A.2 NVRC configurations at different scales.
Neural representation. In NVRC, the INR architecture is based on HiNeRV 30. with minor modifications. The differences include: (1) due to the enhanced capability of feature grids coding, feature grids in NVRC are larger for better capability on capturing dynamic contents. (2) The bilinear interpolation of the imput encoding is performed after the stem convolution layer, while in the original HiNeRV was before applying convolution. This improves the performance marginally. (3.) The hyper-paramend selection between different scales has been simplified, where only the number of channels of the network layers and the feature grids change with scales.
The hyper-parameter of the neural representation in NVRC are provided in Table and g It noted that in NVRC, only the input feature grids in HiNeRV are coded the context model, while the local grids are simply considered as general layer parameters, as they only account for small number of parameters.
Table 6: Comparison between NVRC to existing works with entropy regularization.
Method
Stage
Stage
Grid EM
Layer FM
Quan//MM parameter sharing
Multi-level coding
Zhang
66
R+D
N/A
Uniform
per-channel per weight per-weight
No
Gomes et al.
16
R+D
N/A
Neural Network
No
Maiya et
43
R+D
R-D
N/A
Neural Network
No
Kim et
20)
R+D
R+D
Context
Laplace
No
No
Leguay et al.
13
R+D
Context
Laplace
No
No
Zhang et
65
R+D
Gaussian
Gaussian
wer-we
No
R+D
R+D
Context
Gaussian
per nea-bbocc/per-eee axis
Yes
Applied only after training
Context-based entropy model. The context-based entropy model in NVRC has autoregressive style coding process. It is based on masked convolution [57 47 contains blocks, where each
block contains a Layer Normalization layer [4, 3D convolution and GeLU activation [24] (except
14

---

Table 7: NVRC configurations.
SI
82
S3
54
Namber of parameters
2.14M
6.35M
14,
31.4IM
Channels
(224, 112, 56, 28)
(336, 168, 84, 42)
(512, 256, 128, 64)
(768, 384, 192, 96)
kemnel size
3 x 3
Expansion ratios
(4, 4, 4, 4)
Depths
3,3,,3,
Strides
(3,
kernel
Grid sizes
Tyria 15 80
Tyeia 45 80 x Tyria 45 x 80 x
Tyeia 45 80
Grid levels
(4)
Grid scaling ratios
2,
Local grid sizes
T x
TX8
x 16
T x 32
Local grid levels
(3)
Local grid scaling catios
(2, 0.5)
T: the nomber video
T Tyri:: 200 for UVG AET Class
( 50 for MCL-JCV 50
for the final output). The context-based model is independent between channels, and has been implemented with depth-wise convolution. The kernel size is 5 and the width is 8. The output of the context- based model is the means and log scales of the Gaussian distribution.
Feature grids/layer parameters partitioning. In NVRC, the feature grid parameters are partitioned into blocks, and the quantization parameters are shared within each block, while the layer parameters are partitioned into rows and columns, with both of the quantization and entropy parameters shared. For feature grid parameters, the block size used 16 x 8, which is relative small size but can provide high degree of parallelism for the auto-regressive coding process. For the layer parameters, there are different size of parameters in the neural representation, where some of them are very small and could be too costly to include the per-column/row parameters. Thus, for those light weight parameters, either single-axis or per-tensor quantization/entropy parameters are used. particular, use the per-column/row quantization and entropy parameters if the number of parameters on the column/row is at least 128.
A.3 Experiment configurations
In the experiments, the configurations of NVRC are based on [30 The training is performed by sampling patches from the target video, where the patch size is 120 120, and the batch size of each training step is 144 patches (equal to frame). For learning RGB output, the distortion loss is:
(10) where the MS-SSIM has reduced window size (5 x 5) due to the training in small patches. For YUV output, NVRC is trained the YUV444 setting, to avoid changing the model architecture, but evaluation is in YUV420. In related works, different loss functions for YUV outputs have not been studied. Here we use the loss
D = 0.7 I.1 nGB + 0.3 x (1 MS-SSIMRGB)
0.99 X (MSE X MSE!/ x MSE /") + 0.01 X (1 MS-SSIMy),
(1)) which align with both the commonly used PSNR-YU YUV (6:1:1) and MS-SSIN (Y) metrics. While we did not thoroughly study the weighting between two terms, found that this ratio offers both good PSNR and MS-SSIM performance.
The learning rates in Stage and Stage 2 are 2e-3 (or 1e-3 for rare case which the training is less stable) and le-4, where the cosine decay is applied for scaling the learning rate with a minimum learning rate of 1e-4 and le-5, respectively. The norm clipping with 1.0 is applied. 1.2 regularization of Ic-6 is applied to improve the numerical stability, as we observe that the norm of weight could grow too large in some cases. The magnitude of L2 regularization linearly decays in the first stage and is not applied in the second stage, to avoid under-fitting. For the soft- roundinn and the Kumaraswamy noise [29] in Stage I, the temperatures and the noise scale ratio scale from 0.5 to 0.3 and 2.0 to 1.0, respectively. For Quant-Noise (53] in Stage 2, the noise ratio scales from 0.5 to 1.0. Note that we follow [30]. where rounding, instead of the straight-thoough estimator (STE) [ is applied to the quantized variables. is optimized once for every 8 steps of D.
For the conventional codecs, we used QP range of 16-44 for x265 1 and 16-36 for HM 50 and VTM [I], with QP values selected at intervals of The results for HiNeRV 30) and HNeRV-Boost 65 are obtained by training with their provided implementations. with the original configurations.
15

---

Table NVRC representation scales and regularization (A) configurations for the UVG 45), MCL. JCV 59 and JVET-CTC Class B 9 datasets (RGB/YUV).
UVG
MCL.JCV
JVET-CTC Class
point
RGB
YUV
YUV
RGB
YUV
1
S2, = 1.0
52, À 32.0
S1. = 0.25
S1. 8.0
52, A 1.0
S2, 32.0
2
$2, 2.0
S2, 64.0
51, =0===00 0.5
51, = 16.0
$2,
2.0
52, 64.0
3
$3, = 4.0
83, A = 128.0
82, À 1.0
82, À 32.0
83, A 4.0
S3, À 128.0
256.0
S2, =
S2.
S3, 8.0
256.0
5
$4, X = 16.0
54, = 512.0
83, X = 4.0
83, A = 128.0
84. A=
= 16.0
S4, 512.0
54, 1024.0
32.0
Rate distribution of different parameter types
Bits per parameter of different parameter types
Gidd
Layer
0.0
Grld Quant + EX
Loyer Quant: EM
0.
Bapphons
Hemeydee
Jackey
Resdylattia
Shaletoy
Markifude
Beauty
HemeyBap
Rasepptetto
Shaletty
Figure 6: (Left) Rate distribution and (Right) bits-perr of different parameter types.
A.3.1 Rate distribution
(left,, we provide the rate distribution of different types of parameters (feature grids, network layer parameters. quantization and entropy coding parameters for feature grids/layer parameters). The data is collected from the lowest rate point models with the UVG dataset [45). can be observed that, in general, the quantization and entropy coding parameters contribute to very small amount of the total bitrate, where the ratio between the feature grids and the layer parameters vary between different video sequences. In Figure (right), further provide the bits-per-pan pee---eeepaartee data for different video sequences. In this very low bitrate rate point, NVRC is capable learning parameter distributions with very low pee--eeaaaeeerr.....eee and it also varies between sequences. For example, on some sequences with larger motion (Jockey and YachtRide), the tits-per-parameer of the feature grids can be doubled, but for some relatively static sequences, the bits tts-per-parrmeten of the feature grids is nearly zero.
A.4 Positive Impacts
NVRC is the first INR-based codec which outperforms VVC VTM [ with a 23% coding gain the UVG database [45). will potentially contribute to the next generation of video coding standards, and improve current video streaming services if deployed in practice.
A.5 Limitations
Encoder complexity. As the implementation NVRC is based on sophisticated neural networks, requires substantial computational resources, in particular the encoder. This is common issuc with many INR-based approaches that require content overfitting during encoding. It makes this type of approaches unsuitable for real-time encoding scenarios like video conferencing, This also results in increased energy consumption and a negative environmental impact. Future work should focus on reducing the complexity of this model.
Latency, As INR-based video codecs require overseeing all frames of a video sequence at the same time during encoding, the system latency become more longer compared to conventional and some end-to-end learned video codecs which perform per-frame encoding. This prevents these INR models from adoption in practical application scenarios.
16

---

Reproducibility. In this paper, we have not studied reproducibility, which is a critical issue for the practical upplication of deep video compression with entropy coding. While our experiments focus on floating-point operations, the operations in the proposed method can also be implemented, for example, using integer operations 66. which can ensure reproducibility.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction can accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have mentioned the limitation of this work in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
This work does not contain any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We did our best to disclose all the information required to reproduce the main experimental results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer:
[Yes]
Justification: We will release the code and data publicly available to support the reproducibil- ity of the results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
//nips //nips.cc/pil cccpublic/gut
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We clearly specify all the training and test details in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[NA]
Justification:
This does not apply to the results in this work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We did provide the information related to the hardware resources used for generating the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research reported in this paper does conform with the NeurIPS Code of Ethics, in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We did discuss the impact of this work in Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We did not identify such risks with this work.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have mentioned and cited the code/data/models used in this paper, and respected the license and terms while using them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We describe the NVRC model, and will release the implementation alongside the paper after acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did perform experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did perform experiments involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "y6PVNEIDKp", "reviewer": "Reviewer_csHr", "summary": "- This paper proposes an INR-based video codec, NVRC, which aims to improve the rate efficiency by encoding parameters hierarchically. Experimental results of the proposed method have been shown in RD performance on the UVG dataset.", "questions": "- Have you compared each of the changes in the proposed method (1-4 on page. 2) with other methods ([24], [23], [57])? For example, V4 in Table 3 is intended to show the usefulness of hierarchical parameter coding, but have you compared it with the parameter coding method used in INR-based codecs ([24], [23], [57])?", "limitations": "- Section A.4 describes the limitations of the proposed method.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Experiments show good RD performance compared to recent INR-based codecs ([24], [23], [57]) (Figure 3), supporting this as an effective approach.", "weaknesses": "- I thought that the usefulness of the method could be better demonstrated by showing how much the RD performance was improved for each of the changes in the proposed method (1-4 on page. 2).\n- The fact that the comparison was made with only one UVG dataset is also a weak point in demonstrating the general effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q: Showing coding gain for each contribution on Page 2.***\n\nA: Thank you for your suggestion. In the original paper, we presented these figures in the ablation study, including (contribution 2) the use of different quantization setting and entropy models (V1/V2/V3 for entropy models and V5 for quantization) and (contribution 3) the hierarchical parameter coding (V4).\n\nWe are conducting additional ablation studies for V6 (pretraining + fine-tuning [13, 37, 57], i.e., without fully end-to-end optimized settings) and V7 (without alternating optimization) to confirm our contributions 1 and 4, respectively. Preliminary results for V6 have been included in the rebuttal pdf file.\n\n***Q: More databases for evaluation***\n\nA: We agree with the reviewer that performing evaluation on multiple datasets is important. Additional (preliminary) experiment results on the MCL-JCV and JVET CTC Class B datasets with the RGB settings are provided in the rebuttal (in the pdf). Here we used the JVET CTC Class B rather then JCT-VC CTC Class B (HEVC-B), as the former is the latest HD testset for VVC. It can be observed that promising results have been observed in both dataset. The performance of NVRC is significantly better than HiNeRV, the previous best INR-based method. It is noted that the performance of NVRC with the MCL-JCV dataset is worse compare with the other datasets. We believe that this is mainly due to the shorter length of the sequences, and the above observation is consistent with previous work [24] We will include full results in the camera-ready version when the paper is accepted.\n\n***Q: Compare each change with [24], [23], and [57]. ***\n\nA: Due to the large amount of experimentation required, we were unable at this stage to directly compare our approach with the parameter coding methods used in other INR-based codecs. However, the variants in our ablation study (including the additional results provided in the rebuttal) share some common and important features with these methods; the results indicate the effectiveness of our compression model. Specifically:\n\n- V6 (in the rebuttal results) employs the pre-training + fine-tuning settings used in [24, 57]. The results of this experiment show that the fully end-to-end optimized pipeline (in NVRC) does offer improved performance.\n\n- V5 uses a fixed quantization step size for feature grids, which is the same as that in [23]. The results demonstrate its inferiority compared to the learned step size. In addition, as mentioned in the paper (Line 332), we have applied the fixed quantization step size for neural network parameters, but the networks do not converge well.\n\nFurthermore, our proposed NVRC framework employs HiNeRV [24] as the neural representation model with minimal changes; we demonstrate significant performance gain compared to HiNeRV with the original compression pipeline (Table 1 in the paper). We believe that this is mainly due to our improved encoding and compression pipeline. We are happy to provide the results of the NVRC pipeline with the original HiNeRV in the final paper if needed.\n\nCompared to C3 [23], our model type is very different: in NVRC, HiNeRV is used as the neural representation model, where NeRV-style networks are much larger than the network used in C3. In C3, the neural network only contains a very small number of parameters, and the rate of the network is excluded in the training loop and the network is simply compressed with searched quantization parameters. This method is therefore inefficient in our setting.\n\nWe will modify the text and include the justification text/additional results mentioned above in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCPS3GmncC", "reviewer": "Reviewer_YLxP", "summary": "This paper describes the INR-based video codec NVRC. NVRC is optimized E2E and includes a quantized model which is critical for device reproducibility (though this is not discussed in the paper). The performance on UVG is good and significantly better than VVC VTM 20.0. Benchmarks are given for an NVIDIA 4090 and shows near real-time for decode but 100X slower than real-time for decode. Source code will be released when accepted.", "questions": "Is NVRC device reproducible, i.e., can the same stream be decoded accross any device? This would be a huge advantage. Floating point neural codecs are generally not device reproducible. \n\nIn order for NVRC to be practical for video streaming the decode needs to run on NPUs. What is needed to run on an NPU and what is the benchmark for that (say an Apple M3).", "limitations": "Yes, though because the evaluation is pretty weak (only UVG) there may be video scenarios that perform much worse than UVG.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "NVRC provides SOTA performance on the UVG dataset, provides a quantized model which is necessary for device reproducibility and efficient inference (both speed and power), and will release source code. Decoding time is near real-time on a NVIDIA 4090. It is a good solution for streaming scenarios.", "weaknesses": "The encoding time is 100X off from real-time, making it not approprate for any real-time scenarios (e.g., video conferencing, surveillance, etc). There is no discussion of device reproducibility. Is the entire system quantized or is there still some floating point operations that could break device reproducibility? The evaluation is only done on UVG which is insufficient. Others to include (see DCVC-DC paper) are: MCL-JCV, HEVC B, HEVC C, HEVC D, HEVC E, HEVC RGB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q: For real-time scenarios***\n\nA: We agree with the reviewer that the proposed method is not yet appropriate for real-time applications. The relatively long encoding time is one of the main limitations of this AND other INR-based compression methods. We will mention this in the paper as an important topic for future work.\n\n***Q: The reproducibility issue***\n\nA: Regarding reproducibility, we have not verified it for our work. As our model only utilizes a small convolutional network for autoregressive coding, and simple element-wise operations for network parameter coding, we believe that our network is compatible with integer operations and lookup table techniques, which enables reproducibility [Balle2018]. We will include a discussion of reproducibility when describing the limitations in our paper.\n\n***Q: More databases for evaluation***\n\nA: We agree with the reviewer that performing evaluation on multiple datasets is important. Additional (preliminary) experiment results on the MCL-JCV and JVET CTC Class B datasets with the RGB settings are provided in the rebuttal (in the results pdf). Here we used the JVET CTC Class B rather then JCT-VC CTC Class B (HEVC-B) as the former is the latest HD testset for VVC. It can be observed that promising results have been observed in both dataset. The performance of NVRC is significantly better than HiNeRV, the previous best INR-based method in the MCL-JCV dataset. It is noted that the performance of NVRC with the MCL-JCV dataset is worse compare with the other datasets. We believe that this is mainly due to the shorter length of the sequences, and the above observation is consistent with previous work [24]. We will include full results in the camera-ready version if the paper is accepted.\n\n***Refs***\n\n[Balle2018] Ballé, Johannes, Nick Johnston, and David Minnen. \"Integer networks for data compression with latent-variable models.\", ICLR. 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lBpZG1eR92", "reviewer": "Reviewer_BGkd", "summary": "This paper focuses on the implicit neural representation for video compression, where several modifications are applied for the coding of feature grid and network layer. An enhanced training pipeline is also applied.", "questions": "1.  In Table 1, what is the anchor method? Where is the result of the proposed NVRC in Table 1?\n\n2.  In Table 2, the Frame Enc FPS can be as high as 6.4 FPS, so can the NVRC encode a 600-frame UVG video within 100 seconds? From my understanding, the INR-based solution is quite slow for encoding. The author should give the encoding time from RGB video to binary bit-stream, and the decoding time from binary bit-stream to RGB video, where the actual arithmetic coding should be performed, especially when considering NVRC has auto-regressive model.", "limitations": "no negative societal impact", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The claimed performance over traditional codec and VAE-based codec is impressive.", "weaknesses": "1. This paper claims the first fully end-to-end optimized INR-based framework. However, the problem of joint rate-distortion optimization has been addressed for INR-based compression in several existing works, including [13] and [Ref1] (missing in this paper). As these two papers have already been dedicated to enabling joint rate-distortion optimization in INR-based image compression, it is important to tell the difference in this paper when compared with these two previous works. I know these two previous works are for image, while this paper focuses on video. If the author think the difference between image and video in this part is large, the author should highlight the challenge and the corresponding solution of joint rate-distortion optimization for video when compared with image.\n\n[Ref1] Compression with Bayesian Implicit Neural Representations. NeurIPS 2023.\n\n2. In Fig. 3 UVG-YUV 420, the PSNR of baseline DCVC-DC is about 40 dB at 0.03 bpp. However, in the Fig. 7 of DCVC-FM paper [https://github.com/microsoft/DCVC/tree/main/DCVC-FM], the PSNR of DCVC-DC is obviously larger than 41 dB at 0.03 bpp. So, for the same model, what is the reason for this large RD difference? \n\n3. The results on MCL-JCV and HEVC B are missing. The ablation study is only based on two videos, which is not reliable. \n\nCurrently I think this paper has very good performance. If the author can address my concerns and questions, I will consider increasing my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q: The difference between NVRC and [13]/[Guo2023]****\n\nA: We thank the review for highlighting this point. We agree that there are multiple INR-based codecs that focus on joint rate-distortion optimization, and will describe these in the revised paper.\n\nWhen we refer to a \"fully end-to-end optimized framework\" in the paper, we imply that i) the approaches should be optimized using a rate-distortion loss during the whole encoding process, and ii) all components with parameters that will be entropy coded, and their related quantization/entropy parameters should be optimized with according to the rate-distortion objective. Based on this definition, we believe that [13] is not fully optimized. Specifically, in [13], the rate term only contributes to the second stage optimization. In contrast, our approach is fully end-to-end optimized, because:\n\n1) The RD loss is used throughout the encoding process, including both Stage 1 and 2. This is important because i) it provides improved performance (as shown in the additional ablation study in the rebuttal, V6); ii) it could also provide a more streamlined solution, as pointed out in a recent work [23], when incorporating rate distortion optimization in the first stage of training, the second stage is not mandatory as it only contributes to very little gain (we have observed a similar characteristic in our experiment). \n\n2) The total rate is also jointly optimized in our work, which includes all the parameters that are entropy coded and have non-negligible overhead. In contrast, previous works [23, 27, Leguay2023] only include some of them. In [Leguay2023], only the latent grids are optimized with RD optimization, while the network parameters are entropy coded without optimizing the rates. In their limitation section (Sect V.), they pointed out that the network parameters actually cost a high overhead (up to >30\\% bit-rate) especially at low rates.\n\nIn [Guo2023], the focus was solely on image compression. We also observed that the framework in [Guo2023] differs significantly from those used in INR-based video compression [10, 13, 17, 23, 24, 26, 27]. While the former encodes the parameters without quantization, the latter performs quantization followed by entropy coding. Our approach follows the second method, which is mainly adopted in INR-based video coding tasks.\n\nRegarding the difference between INR-based image and video compression: for video compression, INR-based approaches usually utilize a larger network, due to both the larger size of the signal instance and the higher compression ratio required. For example, NeRV-based models [10, 13, 17, 23, 24, 26, 27] typically contain millions of parameters. Due to the larger size of the models, more complicated compression techniques, such as fine-grained quantization and entropy parameters, are beneficial. Therefore, INR-based image and video compression tasks introduce different challenges, and we think that the fully end-to-end optimization could play a more important role for the video compression task compared to image coding, which allows us to achieve improved overall rate distortion performance.\n\n***Q: DCVC-DC results in Fig. 3***\n\nA: We appreciate the reviewer's attention to detail. We have identified that this inconsistency is due to the incorrect use of the model checkpoints. We have corrected the results (as included in the rebuttal results) and will update them in the revised paper. We have also cross-checked the remaining results and confirmed their correctness.\n\n***Q: Evalution (and ablation study) on more content***\n\nA: This is a valid observation. We have provided preliminary coding results on the MCL-JCV and JVET CTC Class B datasets in the rebuttal (see the attached pdf) and will include full results in the revised paper. Here we used the JVET CTC Class B rather then JCT-VC CTC Class B (HEVC-B) as the former is the latest HD testset for VVC. We agree that performing the ablation study on more sequences is important. However, due to the limited time available, we have not been able to generate these additional ablation study results during the rebuttal. We commit to providing full results on the whole UVG dataset in the paper.\n\n***Q: Anchor methods in Table 1***\n\nA: In Table 1, we compare NVRC (test) against each of the baseline models (anchor) to calculate the BD-rate results. We will make this clear in the paper.\n\n***Q: Encoding time in Table 2***\n\nA: In Table 2, we provide the encoding (training) speed in terms of frames per second (FPS) per each training epoch. Therefore, the total enc time = (#frames x #epoches)/enc FPS, e.g., the total time for encoding a 600 frame sequences with 390 epochs is around 10 hours, when the encoding speed is 6.4 FPS.\n\nWe reported the enc speed in this manner instead of the total time, for two reasons: i) this is commonly used in related works (e.g. [23] reported enc time per 1K steps and [24] reported the time per step), ii) the number of epochs of INR-based approaches is different with varying quality. \n\nWe have reported both the i) entropy en/decoding times for the whole model ('Model Compression Enc/Dec Time' in Table 2), ii) INR decoding time for reconstructing a video frame ('Frame Dec Time' in Table 2). For example, it takes 37 seconds for entropy decoding the smallest model, and this decoding process only needs to be performed once, and the decoded model can be used for reconstructing up to 600 frames in our experiment. Additional parallelism can be further achieved for faster coding with the autoregressive model, but we have not implemented it in our experiment: i) parallel en/decoding can be enabled because different resolution grids are independently coded, ii) parallel entropy decoding and frame reconstruction is also possible, as the decoding a single video frame does not require the full grids to be decoded.\n\nWe will clarify these points in the final paper and report the total en/decoding times as suggested by the reviewer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97K0T89MEC", "reviewer": "Reviewer_m68p", "summary": "This paper proposes a INR-based video compression framework, Neural Video Representation Compression (NVRC), targeting compression of the implicit representation.  \nBased on the proposed novel entropy coding and quantization models, NVRC is able to optimize an INR-based codec in a fully end-to-end manner.  \nThey also proposed a model compression framework for coding all the network, quantization and entropy model parameters hierarchically.  \nExperiments show that NVRC outperforms many conventional and learning-based codecs, with a 24% overall coding gain over VTM (Random Access) on the UVG dataset, measured in PSNR.", "questions": "Please refer to the weakness part. Especially about the complexity and technical novelty comparision with previous works.", "limitations": "The limitation is addressed in the manuscript.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The manuscript is well written and easy to follow.    \nThough the components exist in previous learned compression works, the proposed overall framework has novelty, which builds an elegant and efficient dependency among grid parameter, neural network parameter and hyperprior.    \nThe rate distortion performance is outstanding, making INR based methods better than VTM.", "weaknesses": "Only UVG dataset is evaluated in the experiment. Please consider using more dataset as previous works do.    \n\nIn VAE based deep video compression, there exists a line of works like ‘Bit allocation using optimization, ICML 2023’, which is based on overfitting the latent representation to individual input video/GoP during encoding. This line of research is closely related to INR based video compression and should be discussed in the related works part.   \n\nThe complexity in Table 2 should be compared with previous methods like figure 3.    \n\nThe components used in this paper exist in previous works and should be discussed as previous works, including group based quantization (Fully quantized network for object detection, CVPR 2019), patch or block based parallel context model (Riddle: Lidar data compression with range image deep delta encoding, CVPR 2022; Checkerboard context model for efficient learned image compression, CVPR 2021).  It is better to highlight the technical novelty in each component by comparing with those previous methods.   \n\nMinor:  \nL95, [11] is cited twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q: Evaluation on more datasets***\n\nA: Thank you for this suggestion. We agree that evaluation on more datasets is important, and we have included additional results (MCL-JCV and JVET CTC Class B, RGB setting) in the rebuttal results (see the attached pdf). Here we used the JVET CTC Class B rather then JCT-VC CTC Class B (HEVC-B) as the former is the latest HD testset for VVC. It can be observed that promising results have been observed in both dataset. The performance of NVRC is significantly better than HiNeRV, the previous best INR-based method in the MCL-JCV dataset. It is noted that the performance of NVRC with the MCL-JCV dataset is worse compare with the other datasets. We believe that this is mainly due to the shorter length of the sequences, and the above observation is consistent with previous work [24]. Full results on these datasets will also be provided in the final version of the paper.\n\n***Q: Discussing related works on VAE based deep video compression***\n\nA: We appreciate this suggestion. In the paper, we have referred to related work on overfitting-based VAE for image [55] and video [52] compression, but we agree that it would be beneficial to include further contributions in this class including the paper suggested by the reviewer [Xu2023].\n\n***Q: Complexity versus Rate plots ***\n\nA: We agree that complexity versue rate plots will provide a better illustration of performance. We have included these in the rebuttal results (see the attached pdf) and will of course include them in the revised paper.\n\n***Q: Technical novelty in each component over existing works.***\n\nA: Thank you for highlighting this. We agree that group-based quantization and patch/block-based context models have been employed in previously published papers, and we will include discussion of this in the revised paper.\n\nRegarding the novelty of our approaches over these works, although the aforementioned components do exist in the literature, existing INR-base video codecs have rarely employed these models, especially to combine learned quantization with different entropy models.\n\nIn terms of the use of quantization and entropy models, we would like to highlight that there is a gap in the design of these components between conventional end-to-end approaches and INR-based compression models; existing INR-based models only opt for simple solutions, e.g., conditional Gaussian or simple auto-regressive models. Our work exploits the use of more complex models, i.e., the axis-based conditional Gaussian and block-based auto-regressive models, which offer advantages over existing INR-based approaches. However, we do understand that these entropy models are not SOTA choices when considering all learning-based compression models, e.g. the checkerboard model [He2021].\n\nWhile our technical novelties include the use of both better quantization and entropy models for INR-based video compression, more importantly, we have combined different quantization and entropy models into a fully optimized framework and propose encoding the whole model (the INR + quantization/entropy models) in a hierarchical manner. Previous work has usually utilized a single entropy model for encoding different types of parameters. For example, [23, 27] only use a learned context model for encoding latent grids but employs non-learned quantization and entropy model for coding network parameters; [57] only uses a conditional Gaussian model for encoding all types of parameters. In our work, we use group-based quantization for network parameters and block-based context models for feature grid encoding. This offers better coding efficiency, as the parameters are encoded by a class of models that are likely to be more efficient. Furthermore, in INR-based compression, introducing more complex quantization and compression models can introduce significant overhead, because the parameters of these models have also to be transmitted to the decoder side. Thus, we introduce the hierarchical parameter coding structure into INR-based compression, which enables the use of more complicated quantization and compression models.\n\nWe will include the above discussion in the paper.\n\n***Q: [11] is cited twice***\n\nA: Thank you for pointing out this issue. We will fix this in the camera-ready version.\n\n***Refs***\n\n[Xu2023] Xu, Tongda, et al. \"Bit allocation using optimization.\" International Conference on Machine Learning. PMLR, 2023.\n\n[He2021] He, Dailan, et al. \"Checkerboard context model for efficient learned image compression.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you the reviewers for thorough feedback to our submission. We will address the concerns individually.

Meta-review: The paper introduces NVRC, an INR-based video codec with reported excellent performance.

The authors provided satisfactory responses and convinced 
- Reviewer csHr to increase the score to Accept 
- Reviewer BGkd to increase the score to Borderline Accept
- Reviewer m68p to increase the score to Weak Accept
- Reviewer YLxP to maintain Accept

Thus, for this work there is an acceptance consensus.

After reading the paper and carefully checking the reviews and the authors' responses the ACs agree with the reviews that the present work makes important contributions and is of interest for the community.

The authors are invited to further refine their paper for the camera ready by including (part of) information/details from their responses to the reviewers' comments.
Decision: Accept (poster)