HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation
Authors: Ho Man Kwan
Ge Gao
Fan Zhang
Andy Gower
David Bull
Venue: NeurIPS 2023 poster (2023)
Keywords: Video compression
Implicit neural representations

Abstract: Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3\% overall bit rate saving over HNeRV and 43.4\% over DCVC on the UVG dataset, measured in PSNR).

HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation
Ho Man Kwanf, Ge Gaot, Fan Zhangi, Andrew Gowerf, David Bulli
Visual Information Lab, University of Bristol, UK
Immersive Content & Comms Research, BT, UK
{mm. kwan, gel.gao, fan. zhang, dave. wullebbrrllbslllllo.. ac uk, andrew, gowerabt.com
Abstract
Learning- based video compression currently popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, propose HiNeRV, an INR that combines light weight layers with novel hicrarchical posi- tional encodings. We employs depth-wise wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing meth- ods. We further build video codec based HiNeRV and refined pipeline for training, pruning and quantization that can better preserve HiNeRV's 's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3% overall bit rate saving over HNcRV and 43.4% over DCVC on the UVG dataset, measured in PSNR).
1 Introduction
Implicit neural representations (INRs) have become popular due to their ability to represent and encode various scenes (37), images 45 and videos 45 [1]. INRs typically learn coordinate to value mapping (e.g. mapping a pixel or voxel index to its color and/or occupancy) to support implicit reconstruction of the original signal. While these representations are usually instantiated as multilayer perceptrons (MLPs), existing MLP-based network can only represent video content with a low reconstruction quality and speed M. To address this limitation, recent works have employed Convolutional Neural Networks (CNNs) to perform frame index to video frame mapping
[T] 29 5 25 10). These CNN-based INRs are capable of reconstructing video content with higher quality and with a faster decoding speed, when compared to
approaches [45]. When using
INRs for encoding videos, video compression can then be achieved by performing model compression for the individual input video. However, existing
algorithms remain significantly inferior
state-of-tt -th--arttt standard -ased 552 47 and learning based codees [26 41 (7 28 35). For
example, none of these INR-bases codecs can compete with HEVC x265
(veryslow preset).
Projecct page: https://hmkx. http::////mnkkggthuuubbb/b/aaaaa
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Runny
UVG
12
30-
PONE
2
2
12
10
100
150 200
250
300
20
100
150
200
250
300
HINeRY
HINeRV (ouns)
Epochs
Epochs
31.4dB PSNR 101bpp
36.6dB PS.R
Figure I: (Left) Visual comparison between HNeRV 10 and HiNeRV (ours) for compressed content (cropped). HiNeRV offers improved visual quality with approximately half the bit rate compared to HNeRV (PSNR and bitrate values are for the whole sequence). (Right) Video regression with different epochs representation task. HiNeRV (ours) with only 37 epochs achieves similar reconstruction quality to HNeRV [T] with 300 epochs.
employ conventional convolutional layers or sub-pixel convolutional layers 42, which are less parameter efficient, and hence limit representation capacity within a given storage budget. In addition, most existing work employs Fourier-based positional encoding [37]; this has long training time and can only achieve sub-optimal reconstruction quality [11 29 55. In video compression, the training of INR models is equivalent to the encoding process, implying that most INR-based codecs require long encoding runtime obtain satisfactory rate-quality performance [II]. However, some recent non-video INR models have utilized feature grids or a combination of grids and MLPs the representation speed up the convergence of INRs;
Most INR-based models for videos 29 5 25 10
this has improved the encoding speed by several orders of magnitude [T5 48 38 9
In this paper, we propose for video compression, HiNeRV. We replace commonly used sub-pixel conventional layers [42 in existing INRs for upsampling [ 29 25 [0] by new upsampling layer which embodies bilinear interpolation with hierarchical encoding that is sampled from multi-resolution local feature grids. These local grids offer increased parameter efficiency, the number of parameters increases with the upsampling factor instead of the resolution. Moreover, the network is primarily based on MLPs and depth-wise convolutional layers (rather than dense convolutional layers). This enhances the representation capability and maximizes the performance for given parameter count. This architectural design allows us build much deeper and wider network which offers a significantly better video encoding performance when compared to state-of- the-trr INR-based coding approaches.
INR model based Hierarchically-encone
Neural Representation
Furthermore, instead of learning frame- [I] patch-wise [3] representation, by simply training with overlapped patches, HiNeRV can be seamlessly switched between both representation types, achieving a unified representation with improved performance compared to both frame-based and patch-based settings. This also provides flexibility for hardware implementation, where encoding and decoding processes can be performed either using frames to optimize the computational complexity, or as patches to minimize the memory footprint.
show that
To achieve competitive coding performance, also refine the model compression pipeline in [I]. where pruning and fine-tuning are followed by model training, before quantization is applied. First, used adaptive pruning technique reduce the negative impact of model pruning. Secondly, quantization-awee -aware training is applied for fine-tuning the model performance before quantization. This enables lower bit depth quantization which achieves improve rate-distone dieetrsttoionn trade-off.
The proposed method has been tested against existing -NR-based video coding methods and state-of- the-art conventional and learning-based video codecs on the UVG [36] and MCL-JCV (51) datasets. Notwithstanding the fact that HiNeRV has not been end-to-end optimized for video compression (i.e. pruning, quantization and entropy coding are not included in the training loop), it still significantly outperforms all INR-baser methods (c.g., 72.3% overall BD-rate gain over HNcRV on the UVG database, measured in PSNR), and is also competitive compared with existing conventional and end-to-end learning- based video codec (e.g., 43.4% over DCVC, 38.7% over x265 veryslow).
The primary contributions of this work are summarized below:
1) We propose HiNeRV, new INR employing hierarchical encoding based neural representation.
2) We employ unified representation by adding padding, which trades small computation overhead for additional flexibility and performance gain.

---

3) We build video codec based on HiNeRV and refine the model compression pipeline to better preserve the reconstruction quality of INRs by using adaptive pruning and quantization-awae training.
4) The compression performance of the proposed method is superior to existing INR models, and is comparable to many conventiona//eamin--aaan video coding algorithms. As far are aware, it is the first INR-based codec to significantly outperform HEVC (x265 veryslow) [3
2 Related work
2.1 Video compression
Video compression has long been a fundamental task in the field of computer vision and multimedia processing. As alternatives to the already popularized conventional codecs such as H.264/AVC [52], H.265/HEVC [47, and H.266/VVC [8). there has been a rapid increase in the adoption of deep learning techniques for video compression in recent years. This has typically involved replacing
certain modules (e.g., motion compensation [ [4), transform coding [55] 16 and entropy coding
(6] in the conventional pipeline with powerful learning- baaneng-based models (33 35
In contrast, there has also been significant activity focused on the development of new coding architectures which allow end-to-end optimization. Lu et al. (32) proposed DVC that was further extended to enable major operations in both the pixel and the feature spaces (2 20 An alternative
approach has focused on conditional [26 30 instead of predictive coding to reduce the overall bitrate by estimating the probability model over several video frames. Furthermore, the characteristics of the differentiable frameworks have been exploited by 19 49 44, where both encoder and decoder (typically signaled by model stream containing updated parameters) are overfitted the video data during the evaluation to further enhance compression efficiency.
43 28 claiming to outperform the latest compression standards, these methods still follow the pipeline of conventional codecs, which may constrain the development of neural video compression methods. Moreover, learning-based video compression methods tend to be much more computational complex and often yield much slower decoding speed than conventional codecs. This often renders them impractical for real-time applications. especially considering the prevalence of high-qualit, and high-resolution videos consumed nowadays.
While effective, with some recent work [53
2.2 Implicit neural representation
Implicit neural representations (INRs) are being increasingly used to represent complicated natural signals such as images 45 12
14 23 videos [45] I1) and valued, volumetric content [37]. This type of approach benefits from incorporating positional encoding a technique that embeds the positional input into a higher-dimensional feature space. Periodic functions [45] 33 37 have first been utilized to improve the network's 's capability for learning high frequency information, and
grid features [48] 38 9 have then been applied to address their slow convergence speed and further improve the reconstruction quality.
has re-formulated the INR for video signals to be frame-wise, achieving competitive reconstruction performance with very high decoding speed. NeRV approaches have inspired a trend of utilizing CNNs to encode the RGB values of videos with 1D frame coordinates [1]
More recently, Neural Representations for Videos (NeRV)
29 25
I0] or with 3D patch coordinates [5]. and
have demonstrated promise in various video tasks, including denoising []], frame interpolation
29 25 10]. inpainting [5] [0], super-resolution
13 and video compression
29 5 25 10]
When INRs are applied for image and video compression, they typically convert the signal com- pression task into model compression problem by incorporating weight pruning, quantization and entropy coding [17). NeRV [IT] and related works (29 [5 [0] adopt the above approach for video compression. Although these have demonstrated fast decoding capability, they do not yet achieve a rate-distortion performance comparable to either conventional or learning- -bsed codecs.
3 Method
Following the approach adopted in previous work 45 we consider a video regression task where neural network encodes a video V by mapping coordinates to either individual frames, patches or
3

---

pixels, where V E RT<MXWxC, T. H, W and C' are the number of frames in V, the height, the width and the number of channels of the video frames, respectively.
Fig. 2 (top) illustrates the high level structure of the proposed model, HiNeRV, which contains a hase encoding layer, a stem layer, N HiNeRV blocks, and head layer. In HiNeRV, each RGB video frame is spatially segmented into patches of size M x M, where each patch is reconstructed by one forward pass. The model first takes patch coordinate (i, t) to compute the base feature maps No with size Mo x Mo x Co. Here we always refer the coordinates to the integer index, such that 0 S < T. S j < and 0 S < M The following N HiNeRV blocks then upsample and process the feature maps progressively, where the n-th block produces the intermediate feature maps X, that have the size Mn M,, x Cn (MN M). Finally, a head layer is used project the feature maps to the output, Y. with the target size M x M x C.
W.
3.1 Base encoding and stem
HiNeRV first maps the input patch coordinates into the base feature maps, Xo. by
Xo = F' s'ttem (Tbase (i. (.)).
(I)
To compute the base feature maps, we first calculate the pixel coordinates (related to the corresponding
video frame) of the patch. For a patch with size Mo x Mo. the frame-based pixel coordinates (uram Uframe) can be computed by the patch- Upatch. Ppatch MM. such that üframe = i Mo + Upatch and U/rame = X Mo + Upatch- Then, by using the frame-based pixel coordinates, the positional encoding Tbase ( f) can be interpolated from the learned feature grids [25] After that, we employ stem convolutional layer Fstem for projecting the feature maps to a desired number of channels Co.
pixel coordinates (Upotek Upateh) for 0 S
It is noted that most existing INRs for video [1] 29 5. I0 utilize the Fourier style encoding, i.e. they
apply sin and cos functions to map coordinates into the positional encoding. However, such encoding contains only positional information, which requires additional layers (e.g. MLPs) to transform it into informative features. In contrast, we adopt grid-based encoding (25] as they that contain richer information than the Fourier encoding. Specifically, we use the multi-resolution temporal grids that were introduced in FFNeRV (25), where the various feature grids have different temporal resolutions. In FFNeRV, linear interpolation over the temporal dimension is used to obtain slice that is used as the input feature map. In our case, we utilize both of the frame index and the frame-based coordinates, i.e. (Urrame: Uframe: t), for interpolating the feature patches.
Although both high temporal resolution and large number of channels are desirable for enhancing the expressiveness of the feature grids, this can result in greater model sizes and hence higher bitrates when the model is used for compression tasks. To maintain compact multi-resolution grid, we increase the number of channels when reducing the temporal resolution at each grid level. i.c. the size of grid is Tarid x Hgrid W grid (Cgrid x 2 for 0 < < Lgrid- Here Ligii is the number of grids and Tgrid x Hgria X Worid& X C Cgrid is the size of the first level grid.
3.2 HiNeRV block
The obtained base feature maps are then processed by N HiNeRV blocks, which progressively upsample and process the feature maps. Specifically, the n-th HiNeRV block, where 0 < S N, upsamples the input feature maps Xn-1 with size Mn-1 X Mn-1 x C.-1 through bilinear interpolation Un, with a scaling factor S., such that M,, = Mn-1 x Sn. We use bilinear interpolation, mainly due to its low computational cost and its capability to compute smooth upsampled maps. The HiNeRV block then computes hierarchical encoding 7 (i, matches its number of channels by linear layer Fene and adds it to the upsampled feature maps, where (n, j. t) matches the upsampled feature map size (see Section 3.4). Finally, it applies set of network layers Fn to enhance the representation to obtain the output X,,, where we specify the number of layers by D... For all the HiNeRV blocks except the first one (1 < < N), the first layer in F,, also reduces the number of Co
channels by factor, R, such that C, 7 T to save the computational cost for processing high spatial resolution maps. The output of the n th HiNeRV block, X,, has a size of M, x Mn x Cn. and can be written as
X, (Un + Penc(Tr <
(2)

---

Nx BINoRY Blocks
Prach Inder (5,,
Grd
Stem
X2 Xx-
Xay
HINeRV Block
Local Gesd
Xn-1
Bilinesr Interpolation
SILF Cuur
MLLP Cour
Inter
X
16
(4,1,
Lovel Grid
MLP
Figure 2: Top: The HiNeRV architecture. Bottom left: The HiNeRV block. HiNeRV block take feature maps Xn-1 and patch index (i,j.() input, upsample the feature maps, enhances with the hierarchical encoding, then computes transformed maps X... Bottom right: local grid. In HiNeRV, hierarchical encoding computed by performing interpolation from the local grid, where the modulo the coordinates is being used.
Figure 2 (bottom-left) shows the structure HiNeRV block. Due to its observed superior performance, we employ ConvNeXt [31] as the network block in F., a combination of the MLP layer with depth- wise convolution. We also apply Layer Normalization 4 before the interpolation and the MIP layers, and we only use shortcut connections when the input and output dimensions are matched.
3.3 Head layer
The final output patch Y is computed by applying a linear layer with sigmoid activation, denoted by Fhead, the output of the N-th HiNeRV block,
Y Freea (X))
3.4 Upsampling with the hierarchical encoding
10 use the sub-pixel convolutional layer 42 for feature map up-scaling. However, this has a high parameter complexity: K2 S² x C1 Cz, where K, S, Ci and C2 represent the kernel size, the upsampling factor, the number of input and output channels, respectively. While neighboring features are highly correlated, the convolutional layer does not take this into account and learns dense weight matrix to perform upsampling. This is inefficient especially when the model size the concern for tasks like compression (IT), because the low parameter efficiency limits the maximum depth and width of the networks, and thus the capacity.
Existing NeRV-based approaches [1 29 25
Previous work [I]] has shown that bilinear interpolation does not perform well convolutional layers. However, we observed that it is actually a better choice when the parameter count is fixed. By performing interpolation, we can utilize the saved parameter budget build network with higher capacity. While we can generate high-resolution maps using parameter- free bilinear interpolation, the resulting maps are smoothed, and subsequent neural network layers may struggle produce high-frequency output from them. One way to model high-frequency signals is to introduce positional encoding [37] during upsampling. As mentioned in Section 3. grid-based encoding is preferred due to the good representation performance. However, adopting it to enhance high-resolution feature maps can be costly, as it requires high-dimenn gigh-dimensional grids. To address this limitation, we introduce novel grid-based encoding approach called hierarchical encoding, which boosts the upsampling capability of bilinear interpolation without significantly increasing the storage cost.
Unlike normal grid- grid-based encoding, which computes the encoding using global coordinates, hier- archical encoding utilizes local coordinates to encode relative positional information. Specifically, the local coordinate is the relative position of pixel in the upsampled feature map to its nearest pixel in the original feature map. Because local coordinates have a much smaller range of values, the required feature grid is also much smaller. Moreover, when both the base encoding and hierarchical encoding are used, every position can represented hierarchically, allowing us to efficiently encode positional information high-resolution feature maps. During upsampling, the upsampled feature
5

---

maps are first produced through bilinear interpolation with an up-scaling factor Sn. Then, for all frame-based pixel coordinates (u frame; frame) in the upsampled feature maps, we compute the corresponding local pixel coordinates by Wocal ffrme mod Sn, and Viocal Uframe mod Sn. and employ them to compute the encoding.
It is noted that the above encoding approach is similar to applying a convolutional layer over a constant feature map. To further enhance the capacity of this encoding, we model the feature grids as multi-temppral resolution grids [25], which can provide richer temporal information, similar to the base encoding. In the n-th HiNeRV block, there are Lincal levels of grids and the I-level grid has a size of| Thcee Sn, S,, x (L Clocal
2 The size of the local grids is scaled with the factor S, and can be adjusted by the hyper-parament T Tloeat and Clocat- The number of channels of the grids also scaled in proportion the width of the HiNeRV block, i.c. by the reduction factor R. To obtain the hierarchical encoding, we perform trilinear interpolation by utilizing the frame index t with the local coordinate, i.e. [Wiocal, Voccl,, t), extract encodings from all levels, then concatenate the encodings and apply linear layer Fene to match the encoding channels to the feature maps. To distinguish the grids for interpolating the hierarchical encoding from the one for the base encoding, i.e. the temporal grids, we refer these grids as the temporal local grids, because the hierarchical encoding interpolated from these grids by using the local pixel coordinates. In Section 4.3] we demonstrated that the hierarchical encoding contributes to the superior performance HiNeRV. The process of upsampling with local encoding shown in Figure 2 (bottom right).
3.5 Unifying frame-wise and patch-wise representations
Recent publications on INR for video can be classified into frame-wise 11 representations [5]. Actually, in many of these networks, the initial feature maps can be easily computed either frame-wise or patch-wise, as the positional encoding depends only on the corre- sponding pixel coordinates. However, these two types of representations are not switchable because of boundary effects. In HiNeRV, we adopt simple technique to unify frame-wise and patch-wise representations, When configuring HiNeRV as a patch-wise representation, we perform computation in overlapped patches, where we refer to the overlapped part as paddings, and the amount of padding pixels depends on the network configuration (e.g. the kernel sizes and/or the number of bilinear interooltion/convoluttonn layers). Such overlapped patches have previously been used for tasks such as super-resolution (22), but have not been applied in NeRV-based methods. When performing encoding in patches without proper padding, missing values for operations such as convolution can result in discontinuities between patch boundaries. Moreover, networks trained in patches do not perform well when inferencing in frames due to boundary effects. In our implementation, we perform intermediate computation with padded patches and crop the non-overlapped parts as the output patches, while with the frame configuration, paddings are not required. By adding paddings, ensure generation of the same output for both frame- wise and patch- wise configurations.
29 25
10
patch-wise
Although adding paddings introduces additional computational overheads, it does provide the fol- lowing benefits: (i) it allows parallel computation within the same frame, and reduces the minimum memory requirement for performing forward and backward passes. This shares a design concept with conventional block-based video codecs, and can potentially benefit the compression of immersive video content with higher spatial resolutions (where performing frame-wise calculation may not be supported by the hardware); (ii) It improves the training efficiency when compared to frame-wiss representation, as we can randomly sample patches during training (22], which can better approximate the true gradient and speed up the convergence; (iii) It can also enhance the final reconstruction quality compared to a patch-wise representation without suffering boundary effects. By applying the above technique, HiNeRV can be flexibly configured as either frame-based patch-based representation without retraining. Our ablation study verifies these benefits by comparing this approach to both the Trame-based and patch-based variants (see Section 4.3
3.6
The model compression pipeline
To further enhance the video compression performance, we refined the original model compression pipeline in NeRV [T] which has been used in a series of related work 29
5
25
10
In (..
model training is followed by weight pruning with fine-tuning to lower model sizes. Post-training
quantization then utilized reduce the precision of each weight, while entropy coding is further employed for lossless compression. In this work, we made two primary changes to this pipeline:

---

(i) applying an adaptive weighting to the parameters in different layers for pruning, and (ii) using quantization-awee
-warre aware training with Quant-Noise [6
to minimize the quantization error.
25 10 is typically performed globally with respect to the magnitude of individual weights, with fine-tuning applied after pruning [18]. While various non- magnitude based pruning methods exist (e.g. OBD [24)), here we developed simple, modified magnitude-based method for network pruning. Intuitively, we assume that wider layers have more redundancy within their parameters hence pruning these layers tends to have less impact than on the shallower layers. To alleviate the negative impact of pruning, we weight each neuron using both its é1 norm and the size of the corresponding layer. Specifically, for a layer with P parameters, 18,1 which reflects the neuron importance, where À is
Model Pruning in NeRV 11 5
0p. 0 P S P. we compute score, PA parameter (0.5 in our experiments). The pruning is then performed as usual. By applying this weighting scheme, layers with fewer parameters, such as depth-wise convolutional layers and output layers, have less chance to be pruned, while layers in the early stage of the network are more likely to be pruned.
Weight quantization plays an important role in model compression, significantly reducing final model size (T]. While the results in [TT] have shown that the quantization error is not significant when reducing the weight bit depth to 8 bits, further increasing the quantization level is still meaningful for compression tasks. Unlike other related works [1] 29 25 10] that adopted 8 bit quantization, we found that an improved rate-distortion trade-off can be achieved by using bits quantization if a quantization-awwen -warr training methodology is applied. In particular, we perform a short fine-tuning with Quant-Noise [46] after weight pruning, which can effectively reduce the quantization error. However, unlike the original implementation, we do not use STE for computing the gradient of the quantized weights due to its inferior performance.
4 Experiments
4.1 Video representation
To evaluate the effectiivneess of the proposed model we benchmarked HiNeRV against five related works: NeRV [.. E-NeRV (29,, PS-NeRV [5), FFNeRV [25] and HNeRV [IO] on the Bunny M (1280 X 720 with 132 frames) and the UVG datasets (36] (7 videos 1920 x 1080 with total of 3900 frames). For each video, we trained all networks at multiple scales, and kept their number of parameters similar at each scale. Three scales were set up to target the S/M/L scales in NeRV [ for the UVG datasets, while two different scales XXS/XS together with scale S were configured for the Bunny dataset. We reported the encoding and decoding speeds in frames per second, measured with A100 GPU. The number of parameters corresponding to each scale are reported in Table and
For all models tested, we set the number of training epochs to 300 and batch size (in video frames) to 1. We used the same optimizer as in [I]] and employed the same learning objectives for all NeRV-based methods as the original literature [1] 29 5 25 10]. For HiNeRV, we empirically found that it is marginally better to adopt a larger learning rate of 2e 3 with global norm clipping which is commonly used for Transformer-basee which is a common choice for the other networks as in their original literature [1 29 5 23.. We
networks 50 We used the learning rate of 5e
also adopted a combination of E1 loss and -SS-M loss (with a small window size of 5 rather than 11 x 11) for HiNeRV, as we observed that the MS-SSIM loss with a small window size leads to a better performance. For HiNeRV and PS-NeRV (5), we randomly sample patches instead of frames during training, but we scale the number of patches in each batch to keep the same effective batch size. It is noted that the original configuration of HNeRV [10] has an input/output size of 1280 x 640/1920 960 with strides (5, 4, 4, 2, 2)/(5 4, 4, 3, 2). and we pad the frames of HNeRV in order to fit the 1280 x 720/1920 x 1080 videos. This was found to work better than changing the strides in HNeRV. Detailed configuration of HiNeRV is summarized in the Supplementary Material.
can be observed [ that our proposed HiNeRV outperforms all benchmarked models in terms of reconstruction quality at each scale on both Bunny [1] and UVG (36] datasets. We also note that HiNeRV performs better than all the other methods on all test sequences (with various spatial and temporal characteristics) in the UVG database, exhibiting better reconstruction quality in terms of PSNR. In particular, for some video sequences where existing INRs performed poorly (10) HiNeRV offers significant improvement 7.99 over NeRV on ReadySetGo). While the encoding
7

---

Table 1: Video representation results the Bunny dataset ① (for XXS/XS/S scales).
Model
Encodiing
Decoding HPS
NeRV
0.83M/L 64M/33200
25G/57G/1016
131,9/9...81.
308 5/229, 1/202.3
26.82/29 61/32.56
E.NeRV
0.88M/I 65M/3.31M
26G/10GG/104
104, 7/68 0/68.7
254 254.3/175.8/1 75.8//14488
13.03/31.55/6..4
PS-NeRV
0.90M/1 68M/3, 35M
29G/238G/240G
90.6/36. 1/36.0
228. 1/96. 1/96.0
28.47/30,31/34,77
HNeRV
0.82M/L 666666666888.
100.0/80 8/64.2
317.4/2516/1922
31.08/3368/36
FENeRV
0.91M/L 16M/3.19M
26G/58G/102G
62.1/99
108. 2/9../
N0.37/33.83//77.
HiNeRV
23G/47G/96
37.6/27.7/20.0
132.1/103. 9/76.79766.
36.37/38.94/41.1
Table 2: Video representation results with the UVG dataset 36 the encoding/decoding rate.
(for S/M/L scales). Results are in PSNR. FPS
Model
Size
MACs
FPS
Beauty
Bosph.
Honey:
Jockey
Ready.
Shake.
Yacht.
Avg.
NeRV
3.31M
227G
32.4/90.0
32.83
32.20
30.30
23.62
33.24
26.43
30.97
E-NeRV
3.29M
2300
20.7/75.9
33.13
33.38
38.87
30.61
24.53
34,26
26.87
31.75
PS-NeRV
538G
HNeRV
3,26M
175G
24.6/93.4
33.56
35.03
39.28
31.58
25.45
34.89
28.98
32.68
FFNeRV
3.40M
228G
19.0/49.3
33.57
35.03
38.95
31.57
25.92
34.41
28.99
32.63
HiNeRV
3.19M
181G
10.1/35.5
34,08
38.68
39.71
36.10
31.53
35.85
30.95
35.27
NeRV
33.67
E-NeRV
6.54M
245G
20.5/74.6
33.97
35.83
39.75
33.56
26.94
35.57
28.79
33.49
PS-NeRV
6.57M
564G
14.6/42.0
33.77
34.84
39.02
33.34
26.09
35.01
28.43
32.93
HNeRV
6,40M
349G
20.1 166.5
33.99
36.45
39.56
33.56
27.38
35.93
30.48
33.91
FFNeRV
644M
229G
18.9/49.3
33.98
36.63
39.58
33.58
27.39
35.91
30.51
33.94
HiNeRV
368G
8.4/29.1
34.33
34.54
37.04
NeRV
13.01M
230G
31.7/89.8
34.15
36,96
39.55
35,80
35,90
30.39
34.49
E NERR
13.02M
285G
21 1
34.25
37.61
39.74
35.45
29.17
36.97
30.76
34.85
PS-NeRV
13.07M
608G
14.1/41,4
34.50
37.28
39.58
35.34
28.56
36.51
30.28
34.61
HNeRV
12.87M
701G
15.6/52.7
34.30
29.67
37.16
32.31
FENeRV
12.66M
232G
18.4/49.3
34.28
38.48
39.74
36.72
30.75
37.08
32.36
35.63
12.82M
718G
5.5/19.9
37.32
38.19
35.20
and decoding speeds of HiNeRV are slower than that of other methods, HiNeRV achieves higher overall PSNR figure with fewer MACs. Further optimization may help reduce its latency. In the Supplementary Material, we conducted experiments with faster variants of HiNeRV, demonstrating that HiNeRV can achieve satisfactory trade-off among latency, model size and reconstruction quality simultaneously.
Figure (right) shows the performance of HiNeRV, NeRV and HNeRV (at scale S) in terms of the reconstruction quality with various epochs of training. We observe that our model with 37 epochs achieves similar reconstruction quality of HNeRV with 300 epochs on both datasets.
4.2 Video compression
To evaluate vidco compression performance, we compared HiNeRV with two INR-based models: NeRV [II and HNeRV [00]; with two conventional codecs: HEVC/H.265 HM 18.0 (Main Profile with Random Access)
40 and x265 (veryslow preset with B frames) 2 and with two state-
of-the-art learning- based codecs: DCVC (26). DCVC-HEM 27 and VCT [35]. These were all compared using two test databases: UVG [6) and MCL-JCV [1I] Unlike previous work [[]]) which concatenates the videos and uses single network to compress multiple videos, we train all the models for each video separately. For training of NeRV, HNeRV and HiNeRV, we use the configurations described in Section 4.1 but with two more scales, namely XI. and XXI., for encoding videos with the highest rate (See the Supplementary Material). We apply pruning and prune these three models to remove 15% of their weights and fine-tune the models for another 60 epochs. These models are further optimized with Quant-Noise [46] with 90% noise ratio for 30 epochs. Here use the same learning rate scheduling for fine-tuning, but employ 10% of the original learning rate in the optimization with Quant-Noise. To obtain the actual rate, perform arithmetic entropy coding 34 and combine all essential information including the pruning masks and the quantization parameters into bitstreams.
quantization as described in Section 3.6 In particular,
Figure 3 reports the overall rate quality performance on the UVG (36 and the MCL-JCV [51 datasets. Table summarizes the average Bjøntegaard Delta (BD) rate results for both databases. results show that HiNeRV offers competitive coding efficiency compared to most conventional codecs

---

Table 3: BD-Rate (Measured in PSNR/MS-SSIM) results the UVG [6] and MCL-JCV [S]] datasets.
x265 (veryzlow)
DCVC
DCVC-HEM
NeRV
HNeRV
PSNR
33.66%
7.54%
43.44%
25,23%
34.28%
-74.12%
-72.29%
UVG
MS-SSIM
-62.70%
41.41%
-3.50%
49.03%
23.69%
-73.76%
-83.86%
-23,39%
31.09%
-24,59%
35.83%
-17.03%
19%
-66.50%
MCL-ICV
MS-SSIM
-44.12%
-2.65%
-17.32%
80.73%
12.10%
-82.28%
-79.42%
UVG
UVG
40
0.98
38
MS SSIS
e5NR
x205 Iveryslone
HM (RA)
0.94
-X-
DCVC
DCVE-NEM
0.92
0.90
0.00
0.05
0.10
0.15
0.20
0.25
0.00
0.05
0.10
0.15
0.20
0.25
Bpp
Bpp
MCL-JCV
MCL-JCV
0.98
36
MS-SSIM SSISSSIM
34
0.94
32
0.92
30
0.90
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Bpp
Bpp
Figure 3: Video compression results on the UVG
36 and the MCL--CV datasets 51
and learning- based methods. This represents significant improvement over existing NeRvd approaches (this is also confirmed by the visual comparison with HNeRV Figure IT. In particular, it is observed that HiNeRV outperforms x265 (veryslow) [2). DCVC (26] and VCT 135 based on PSNR. As far as we are aware, this the first INR-based codec which can achieve such performance. We also observe that HiNeRV offers better performance compared to H.265 HM (Random Access) based on MS-SSIM. It should be noted that the results of each learning- based codec reported here are based on two model checkpoints, one for optimizing PSNR and the other for MS-SSIM, while all the results for HiNeRV are however based on the same checkpoint.
Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization are not optimized in the loop), it nonetheless outperforms many stateoof-the-ar end-to- end optimized learning- based approaches. This demonstrates the significant potential of utilizing INRs for video compression applications. In the Supplementary Material, comparison with additional baseline is provided.
4.3 Ablation study
To verify the contribution of various components in HiNeRV, generated number of variants of the original model, and evaluated them for video representation on the UVG dataset (36] (all the sequences were down-sampled 1280 x 720 for reduction the amount of computation). For all experiments, we followed the settings in Section 4.1 and performed training targeting scale (by adjusting the width of the network keep the similar model sizes). All results shown in Table 4 We also included the results NeRV [I] and HNeRV [00 for reference.
Bilinear interpolation with hierarchical encoding. The contribution of bilinear interpolation with hierarchical encoding was verified by comparing it with alternative upsampling layers, sub-pixel convolutional layer 42 with (V1) and 3 3 (V2) kernel sizes.

---

Table 4: Ablation studies of HiNeRV with the UVG dataset [3]
Results in PSNR.
Model
Beauty
Bosph.
Honey.
Jockey
Ready.
Stake.
Yacht.
Avg.
NeRV
3.20M
34.03
32.77
39.59
30.39
23.88
33.85
26.88
31.63
HNeRV
3.22M
35.04
35.72
41.11
32.20
25.88
35.75
29.69
33.63
HINERV
3.17M
35.67
39.37
41.61
36,94
31.98
36,74
31.57
36.27
(V1) Sub-Couvixi
3.16M
35.28
36.63
41.58
34.64
29.12
36.31
29.91
34.78
(V2) w/ Sab-Conv3x3
3.15M
34.96
35.35
41.14
32.80
27.18
35.34
29.14
33.70
(V3) wil Encoding
3.17M
35.64
39.18
41.58
36. 16
30.92
36,68
31.50
35.95
(V4)
enc.
3.17M
39.07
30,91
36.81
31.47
35.92
(V5) w/ Fourier (local)
3.17M
35.59
38.99
41.54
35.77
30.61
36.57
31.30
35.77
(V6) Grid (local) enc.
3.19M
35.65
39.26
41.58
36.17
30.93
36.72
31.55
35.98
(V7) w/ MLP
3.19M
35.10
37.17
41.35
34.77
29.10
35.58
29.76
34,69
(V8) Conv3x3
3.17M
37.86
41.37
35.13
29.70
36.10
30.31
35.12
(V9) w/ Frame wise
3.17M
35.68
39.22
41
36.69
31.49
36.54
31.54
36.10
(V10) Patch-wise
3.17M
35.46
38.30
41.55
35.04
30.06
36.51
30.77
35.38
(V1 Nearest Noighbor
3.17M
39.12
41.64
36.52
31.51
36.82
31.33
36.08
Upsampling encodings. Four variants are trained to confirm the effectiveness of the upsampling encodings including (V3) w/o encoding; (V4) with the Fourier encoding [37]; (V5) using Fourier encoding with local coordinates (computed in the hicrarchical encoding); ( (V6) using the grid-based encoding with local coordinates, i.e. the hierarchical encoding without the temporal dimension.
ConvNeXt block. The employed ConvNeXt block [3T has been compared with (V7) the MLP block in transformer [50]: (V8) the block containing two convolutional layers with where we use 3 x 3 and 1 X kernel size to keep the receptive field consistent with the ConvNeXt block used in our paper.
Unified representations. V9 and V10 have been generated for frame- and patch-wise configurations.
Interpolation methods. V11 replaces the bilinear interpolation by the nearest neighbor interpolation.
The ablation study results are presented in Table. 4 which shows that the full HiNeRV model outperforms all alternatives (VI-V1I) on the UVG dataset in terms of the overall reconstruction quality. This confirms the contribution of each primary component of the design. More discussion regarding the results can be found in the Supplementrry Material.
5 Conclusion
In this paper, neural representation model, HiNeRV, has been proposed for video compression, which exhibits superior coding performance over many conventional and learning-based video codecs (including those based on INRs). The improvements demonstrated are associated with new innovations including bilinear interpolation based hierarchical encoding, a unified representation and a refined model compression pipeline.
Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization not optimized in the loop), it nonetheless achieves comparable performance to state-of-the-an end-to-end optimized learning- existing NeRV-based algorithms. This demonstrates the great potential of utilizing INRs for video compression applications. For example, this is the first INR-based video codec which can outperform HEVC HM Random Access mode based MS-SSIM.
approaches, with significant improvement over
Future work should focus on incorporation entropy coding and quantization to achieve full end-to-end optimization.
Acknowledgment
This work was jointly funded by UK EPSRC (ICASE Awards), BT, the UKRI MyWorld Strength in Places Programme and the University of Bristol. We also thank the support by the Advanced Computing Research Centre, University of Bristol, for providing the computational facilities.
10

---

References
[1] scikit-video: Video Processing in Python. http:::/www..ciiiittttttaaaaaaaa
org/
x265. http://www.ee...eeee/eeeee/////a html
[3] E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. Hwang, and G. Toderici. Scale-Space Flow for End-to-End Optimized Video Compression. CVPR, pages 8500-8509. Computer Vision Foundation / IEFE, 2020.
[4] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer Normalization, 2016.
[5] Y. Bai, C. Dong, C. Wang, and C. Yuan. PS-NeRV: Patch-wise Stylized Neural Representations for Videos.
In 2023 IEEE International Conference on Image Processing (ICIP), pages 41-45. IERE, 2023.
[6] J. Ballé, D. Minnen, S. Singh, S. Hwang, and N. Johnston. Variational image compression with scale hyperprior. In ICLR. OpenReview.net, 2018.
[7] Y. Bengio. Léonard, and A. Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013.
[8] B. Bross, Y. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J. Ohm. Overview of the Versatile Vidco Coding (VVC) Standard and its Applications. IEEE Trans. Circuits Syst. Video T 31100)33736-376 2021.
[9] A. Chen, Z. Xu, A. Geiger, Yu, and H. Su. TensoRF: Tensorial Radiance Fields. In ECCV (32), volume 13692 of Lecture Notes in Computer Science, pages 333-350. Springer, 2022.
[10] H. Chen, M. Gwilliam, Lim, and A. Shrivastava. HNeRV: Hybrid Neural Representation for Videos. In CVPR, pages 10270-10279, IEEE, 2023.
[11] H. Chen, B. He, H. Wang. Y. Ren, S. Lim, and A. Shrivastava. NeRV; Neural Representations for Videos. In NeurlPS, pages 21557-21156, 2021.
[12] Y. Chen, S. Liu, and X. Wang. Learning Continuous Image Representation With Local Implicit Image Function. In CVPR, pages 8628 8638. Computer Vision Foundation IEEE, 2021.
[13] Z. Chen, Y. Chen, J. Liu, X. Xu, V. Goel, Z. Wang. H. Shi, and X. Wang. VidcoINR: Learning Vidco Implicit Neural Representation Continuous Space-Time Super- Resolution. In CVPR, pages 2037-2041 2047. IEEE, 2022.
[14] F. Dupont, A. Golinski, M. Alizadch, Y. W. Tch, and A. Doucet. COIN: COmpression with Implicit Neural representations, 2021.
[15] S. Fridovich- Keil, A. Yu, M. Tancik, Q Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance Fields without Neural Networks. In CVPR, pages 5491-5500 IEEE, 2022.
[16] G. Gao, P. You, R. Pan, S. Han, Y. Zhang, Y. Dai, and H. Lee. Neural Image Compression via Attentional Multi-scale Back Projection and Frequency Decomposition. In ICCV, pages 14657-14666. IEEE, 2021.
[17] S. Han, Mao, and W. J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, 2016.
[18] S. Han, J. Pool, J. Tran, and W. J. Dally. Learning both Weights and Connections for Efficient Neural Network. In NIPS, pages 11 133--113, 2015.
[19] G. He, C. Wu, L. Li, J. Zhou, X. Wang, Y. Zhengg B. Yu, and W. Xic. A Vidco Compression Framework Using an Overfitted Restoration Neural Network. In CVPR Workshops, pages 393-597. Computer Vision Foundation IEEE, 2020.
[20] Z. Hu, Lu, J. Guo, Liu, Jiang, D. Xu. Coarse-To- To-Fine Deep Video Coding with Hyperprior- Guided Mode Prediction.
CVPR, pages 591 1-5920. IEFE, 2022.
[21] Z. Hu, G. Lu, and D. Xu. FVC: New Framework Towards Deep Video Compression in Feature Space. In CVPR, pages 1502-1511 15. Computer Vision Foundation IEEE, 2021.
[22] A. Kappeler, S. Yoo, Q Dai, and A. K. Katsaggelos. Video Super- Resolution With Convolutional Neural Networks. IEEE Trans. Computational Imaging, 2(2):109 12000112222222000 2016.

---

[23] T. Ladune, P. Philippe. F. Henry, G. Clare, and T. Leguay. COOL-CHIC: CBOLLCCHICCCooddinatebbie Coordinate-baseel Low Complexity Hierarchical Image Codec, 2023.
[24] Y. LeCun, J. S. Denker, and S. A. Solla. Optimal Brain Damage. In NIPS, pages Morgan Kaufmann, 1989.
[25] J. C. Lee, D. Rho, H. Ko, and E. Park. FFNeRV: Flow-Guided Trame-Wise Neural Representations for Videos. In ACM Multimedia. ACM, 2023.
[26] J. Li, Li, and Y. Lu. Deep Contextual Video Compression. In NewrlPS. pages 18114-18125, 2021.
[27] J. Li, B. Li, and Y. Lu. Hybrid Sempoal-Tempora Entropy Modelling for Neural Video Compression. In ACM Multimedia, pages 1503-1511. 1. ACM, 2022.
[28] J. Li, B. Li, and Y. Lu. Neural Video Compression with Diverse Contexts. In CVPR, pages 22616-22626. IEEE, 2023.
[29] Z. Li, M. Wang, H. Pi, K. Xu, J. Mei, and Y. Liu. E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporaa Context. In ECCV (35), volume 13695 of Lecture Notes Computer Science, pages 267-284, Springer, 2022.
[30] J. Liu, S. Wang. W. Ma, M. Shah, R. Hu, Dhawan, and R. Urtasun. Conditional Entropy Coding for Efficient Video Compression. In ECCV (17), volume 12362 of Lecture Notes Computer Science, pages 453-468. Springer, 2020.
[31] Z. Liu, H. Mao, Wu, Feichtenhofer, Darrell, and Xie. ConvNet the 2020s. CVPR, pages 11966-11976. IEFF, 2022.
[32] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao. DVC: An End-To-End Deep Video Compression Framework. In CVPR, pages 1100--11015. Computer Vision Foundation/ IEEE, 2019.
[33] I. Mehta. M. Gharbi, C. Barnes, E. Shechtman, R. Ramamoorthi. and M. Chandraker. Modulated Periodic Activations for Generalizable Local Functional Representations. In ICCV, pages 14194-14203. IEEE, 2021.
[34] F. Mentzer, F. Agustsson, M. Tschannen, R. Timofte, and L. V. Gool. Practical Full Resolution Leamndd Lossless Image Compression. CVPR, pages 10629-10638, Computer Vision Foundation TEEF, 2019.
[35] F. Mentzer, G. Toderici, D. Minnen, S. Caclles, S. J. Hwang, M. Lucic, and E. Agustsson. VCT: A Vidco Compression Transformer. In NeurlPS, 2022.
[36] A. Mercat, M. Viitanen, and J. Vanne. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In MMSys, pages 297-302, ACM, 2020.
[37] B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV volume 12346 of Lecture Notes in Computer Science, pages 405-421, Springer, 2020.
[38] T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with Multiresolution Hash Encoding. ACM Trans. Graph., 1000::10 2022.
[39] C. R. K. Sharman, R. Sjoberg. and G. Sullivan. High Efficiency Video Coding (HEVC) Test Model 16 (HI 16) Improved Encoder Description Update 14. In the JVET meeting, number JVET-ANÇ002. ITU-T and ISОПEEĆ 2020.
[40] K. Sharman and K. Suchring. Common test conditions for HM video coding experiments. In the JCT-VC meeting, number JCTVC-AF110N ITU-T, ISO/TEC, 2018.
[41] X. Sheng. J. Li, B. Li, I. Li, D. Liu, and Y. Lu. Temporal Context Mining for Learned Video Compression. IEEE Transactions on Multimedia, pages 1 1-12, 2022.
[42] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-Time Single Image and Video Susor-Resolltion Using Efficient Sub-Pixel Convolutional Neural Network. In CVPR, pages 1874-1883. IEEE Computer Society, 2016.
[43] Y. Shi, Y. Ge, J. Wang, and J. Mao. VC:haVC: High- Perforaerfo and Efficient Learned Video Compression. In ECCV (19), volume 13679 of Lecture Notes Computer Science, pages 616-631. Springer. 2022.
[44] M. K. Shirkoohi, V. Sivaraman, and M. Alizadeh. Efficient Video Compression via Content-Adas Adaptive Super-Resolutton. In ICCV, pages 4501 4510. IEEE, 2021.
12

---

[45] V. Sitzmann, J. N. P. Martel. A. W. Bergman, D. B. Lindell, and G. Weizstein. Implicit Neural Representa- tions with Periodic Activation Functions. In NeurIPS, 2020.
[46] P. Stock, A. Fan, B. Graham, E. Grave, R. Gribonval, H. Jégou, and A. Joulin. Training with Quantization Noise for Extreme Model Compression. In ICLR. OpenReview.net, 2021.
[47] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Trans. Circuits Syst. Video Technol., 22( 2211):1649-1168 2012.
[48] C. Sun, M. Sun, and H. Chen. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In CVPR, pages 5449-5459. IEEE, 2022.
[49] T. van Rozendaal, J. Brehmer, Y. Zhang, R. Pourreza, A. Wiggers, and T. S. Cohen. Instance-Adaptive Video Compression: Improving Neural Codees by Training on the Test Set, 2023.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkorcit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. In NIPS, pages 5998-6008, 2017.
[51] H. Wang. W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C. J. Kuo. MCL.JICV IND-based H.264/AVC video quality assessment dataset. In ICIP, pages 1509-1513. TERE, 2016.
[52] T. Wiegand, G. Sullivan, G. Bjentegaard, and Luthra. Overview of the H.264/AVC Video Coding Standard. IEEE Trans. Circuits Syxt. Video Technol., 13(7)56005576, 2003.
[53] Xiang, K. Tian, and Zhang. MIMT: Masked Image Modeling Transformer for Video Compression. In ICLR. OpenReview.net, 2023.
[54] R. Yang, F. Mentzer, L. V. Gool, and R. Timofte. Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement. CVPR, pages 6627-6636. Computer Vision Foundation TEEE, 2020.
[55] Y. Zhu, Yang, and Cohen, Transformer- based Transform Coding, In ICLR. OpenReview.net, 2022.
13


Reviews:
[{"review_id": "KGXgacQNoF", "reviewer": "Reviewer_boCg", "summary": "This paper introduces a hierarchical encoding block for INR-based video compression. In particular, the authors replace the commonly-used sub-pixel/transpose convolution for upsampling with the bi-linear interpolation. The saved parameters are used to increase the capacity of the MLP encoders in order to achieve higher expressiveness of the model. Reportedly, this work is the first INR-based method that achieves comparable rate-distortion performance to the state-of-the-art VAE-based learned methods.", "questions": "See my comments in the weaknesses section.", "limitations": "The encoding time is not addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1) The paper is easy to follow. \n(2) The coding results are promising.", "weaknesses": "(1) The novelty of this paper is rather now. Essentially, the authors replace the commonly-used sub-pixel/transpose convolution for upsampling with the bilinear interpolation. In doing so, the saved parameters are used to increase the number of MLP encoders. The gain appears to come mainly from a simple tweak to the network architecture.\n\n(2) The network architecture of the proposed method is not new, and is more like a trivial combination of several existing works. For example, the Grid and Stem modules are from [1] and [2]. After index transformation, the feature upsampling part is from [3]. \n\n[1] J. C. Lee, D. Rho, J. H. Ko, and E. Park. FFNeRV: Flow-guided frame-wise neural representations for videos. arXiv preprint arXiv:2212.12294, 2022.\n\n[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016\n\n[3] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n(3) The authors should include the comparison with DCVC-HEM in the main paper. Not sure why DCVC (which is a weaker baseline) is used instead. Also, please clarify the GOP size. It seems that GOP=12 is adopted for DCVC-HEM. It was argued in several papers (including DCVC-HEM) that a larger GOP should be used.  \n\n(4) The comparison with HNeRV may be unfair, because the network of HNeRV is designed to represent the whole dataset. As such, its network is quite heavy, and can actually be reduced considerably when it is used to represent only one sequence. In this sense, the per-sequence rate reported for HNeRV may be overestimated. \n\n[4] H. Chen, M. Gwilliam, S.-N. Lim, and A. Shrivastava. HNeRV: Neural representations for videos. In CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Novelty of the paper**\n\nA: The novelty of this work is three-fold. (i) Starting with NeRV, all INR-based codecs employ sub-pixel convolution; this is one of the main limitations that has not been addressed yet. In this work, we introduce parameter efficient layers: this is a simple but effective idea which is shown to be critical for INR based video compression. (ii) While this modification of the architecture does contribute to the overall coding gains achieved, we emphasise that our use of hierarchical encoding also offers significant gains on some video sequences (Jockey/ReadySetGo), where existing INRs (including NeRV and HNeRV) perform poorly. (iii) Finally, we proposed a unified representation, which provides additional parallelism alongside performance gains. \n\nWe will revise the text in the paper to make the above contributions clearer. Furthermore, in order to better characterise the significance of each contribution, we will provide more comprehensive ablation study results including those from NeRV and HNeRV for comparison, as shown in the global response section (Table 1). It can be observed that, when replacing the bilinear interpolation in HiNeRV with a sub-pixel convolutional layer with 1x1/3x3 kernel size (Variant 1/2), HiNeRV still outperforms NeRV. When comparing with HNeRV, the variant with 1x1 kernel size is superior, while the variant with 3x3 kernel size is comparable with it based on PSNR (the MS-SSIM results are even better, i.e., 0.9664 vs 0.9382). We also note that HNeRV employs an additional encoder network and it is tweaked heavily -  the authors tuned the reduction ratio and kernel sizes across layers to obtain better performance.\n\nIt should also be noted in Table 1 of the global response and from the ablation study results in the original paper that other elements in HiNeRV (encoding, choice of block and unified representation) also affect the performance significantly. Specifically, we found that encoding helps more with videos that exhibit fast motion - e.g. it improves the PSNR of Jockey/ReadySetGo by 0.78/1.06 dB. With the default choice (using the ConvNeXT block) HiNeRV obtain delivers an increase of 1.15 dB in terms of average PSNR compared to using normal convolutional blocks. By training as a unified representation, while still performing computation in patches, HiNeRV obtains a 0.89 dB PSNR gain over a normal patch-wise representation.\n\nAll these results clearly demonstrate that all the design elements contribute to the overall performance improvements in HiNeRV. \n\n\n**Q: Novelty of the architecture**\n\nA: We agree that the Grid/Stem modules and feature upsampling are not new -- we do not claim these as novel contributions in this work. Instead, as  mentioned above, the novelty of this paper is based on: (i) applying a parameter efficient network architecture for video compression, (ii) hierarchical encoding and (iii) a unified representation. All of these contribute to the overall performance of HiNeRV based on the ablation study results provided in the global response section (and in the paper). Detailed justification can be found above.\n\n\n**Q: Comparison with DCVC-HEM**\n\nA: Following the suggestion of the reviewer, we will include the comparison with DCVC-HEM in the main paper (also shown in the global results section of the rebuttal). For DCVC-HEM, we use GOP=32. Please note that, in our paper, we are using full video sequences for evaluation rather than 96 frames as in the original DCVC-HEM paper. This may lead to different results.\n\n\n**Q: Comparison with HNeRV**\n\nA: The authors of HNeRV have specified that their experiments were performed by training one model for one video. Quoting from the HNeRV paper: \\textit{``Note that HNeRV achieves this using a small model for each video, while NeRV fits a big model on concat videos (for better compression) which greatly slows down the encoding and decoding speed.\"}. In our experiment, we always use one model for one video for both HiNeRV and baseline models in order that their rate won't be overestimated.\n\n\n**Q: The encoding time is not addressed**\n\nA: We did report the encoding time in the supplementary material associated with the submitted paper. Specifically, HiNeRV with scale L takes around 12 hours for encoding 1080p videos with 600 frames. This is the time required for the full training following the experimental settings for video compression. We will extend this to provide the estimated encoding time comparison of different INRs with the UVG dataset in the revised paper, as shown in the global response section of the rebuttal (Table 2). The results are based on 300 epochs for training with 90 epochs of fine-tuning. Although HiNeRV is slower in terms of encoding time, the faster variant, HiNeRV-B, which we also reported in the supplementary material, is able to perform faster encoding with a smaller model. For example, HiNeRV-B (scale-M) is faster than HNeRV (scale-L), while the model size is smaller and the reconstruction quality is better (reported in Table 5 of the supplementary material).\n\nHiNeRV is also able to obtain superior reconstruction quality in a much shorted time, as shown in Figure 1 (right) of the main paper."}, {"review_id": "0BcPF1wKhm", "reviewer": "Reviewer_zt3G", "summary": "The authors propose a 2D implicit neural compression method that combines bilinear interpolation and hierarchical encoding. This method is based on the premise that bilinear interpolation is a better choice for neural compression than convolution operations.", "questions": "Does the training method in this paper involve training only one model on the entire dataset for inference, as mentioned in NeRV[10]? How does this encoding/decoding process apply to actual scenarios? If so, how is the compression performance on a single-sequence dataset like bunny?", "limitations": "While the paper effectively improves the compression efficiency of the implicit codec, there is scope for future research in optimizing encoding time and latency.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method distinguishes itself from other existing methods by utilizing bilinear interpolation for feature upsampling.\n\n2. The paper investigates model compression techniques, including pruning and quantization, to further reduce the size of the model.\n\n3. The effectiveness of the proposed modules is convincingly demonstrated through an ablation study.", "weaknesses": "1. The explanation of the hierarchical coding process in section 3.4 and Fig.2 is unclear and difficult to follow, requiring further clarification.\n2. It is crucial for the paper to include results on encoding and decoding time, as this information holds significant relevance in real-world scenarios.\n3. The use of deep models with overlapped patches may lead to an unfair comparison with other implicit compression methods due to the increase in training pixels it implies. However, the paper lacks sufficient explanation on this matter, particularly regarding how the final reconstructed pixels are determined by overlapping blocks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: The explanation of the hierarchical coding is not clear**\n\nA: In the revised paper, we will modify the relevant text to better describe the hierarchical coding process. We will also re-draw Fig. 2 to make it easier to understand. Furthermore, we will release the code when paper is accepted, which will help others to understand the implementation details.\n\n\n**Q: Encoding/decoding time is not given**\n\nA: We agree with this reviewer regarding the importance of reporting the encoding/decoding time in the paper. Actually we did report encoding time in the supplemental material for one scale (line 140). We will include more comprehensive encoding runtime results in the revised version, as provided in the global response section (Table 2), where the detailed encoding time comparison results between different INRs are summarised.  For the decoding time results, we have already provided these in the main paper (Table 1-2, in FPS). \n\n\n**Q: The overlapped patches lead to unfair comparison**\n\nA: Although overlapped patches provide performance improvements for HiNeRV, HiNeRV still significantly outperforms NeRV and HNeRV without them. In the global response section (Table 1), we provide more additional ablation results, showing that HiNeRV outperforms NeRV and HNeRV with either frame or patch configurations.\n\n\n**Q: How to obtain final pixels from the overlapped patches**\n\nA: Regarding how to determine the final reconstructed pixels, we did briefly mention this in the supplementary materials (Line 43). The final reconstructed pixels are simply obtained by center cropping, as the amount of padding is equal on each side. To clarify this, we will include a more detailed description in the revised paper.\n\n\n**Q: Do the networks encode the entire dataset/individual videos?**\n\nA: In our work, we train one model for each video (as mentioned in the main paper, Line 305), which differs from the NeRV paper. This, in fact, makes HiNeRV more practical. In Section 4.1 and Table 1 of the main paper, we provide HiNeRV results for the Bunny dataset. HiNeRV is able to outperform all baselines significantly. For example, HiNeRV with 0.77M parameters is able to obtain a reconstruction quality to very close to HNeRV with 3.28M parameters.\n\n\n**Q: Future research of encoding time and latency**\n\nA: We totally agree that reducing the encoding time and latency will be an important direction for future work. We will mention this in the revised paper."}, {"review_id": "IAKpaQlnaf", "reviewer": "Reviewer_JKXU", "summary": "This paper presents a new type of implicit neural representations (INR) with hierarchical positional encoding for video compression, named HiNeRV. Four points are proposed: (1) a new upsampling layer with bilinear interpolation and hierarchical encoding of feature grids, (2) the use of depth-wise convolution and MLP layers instead of conventional convolution layers, (3) the unified representation of frame-wise and patch-wise INR by adding padding, and (4) refined training pipeline for HiNeRV model compression. The proposed method achieves significant improvement over all existing INRs baselines and competitive performance to recent neural video codecs.", "questions": "Below are some further questions: \n(1)\tWhy the proposed method performs a bit worse on MCL-JCV dataset? \n(2)\tIs it possible to provide the best/worst compression cases? In HNeRV, the authors show their model has limited performance on highly dynamic scenes. Is there a similar issue in HiNeRV?\n(3)\tIs it possible to provide the comparison of encoding complexity with recent neural video codecs?\n(4)\tHow do you compress the feature grid? How much bitrate cost of the feature grid?\n(5)\tSome typos in the paper, e.g. Line 60: before quantiszation -> before quantization\n", "limitations": "none", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Significant rate-distortion performance as an INR-based video codec. Largely closing the gap between INR-based method and the SOTA neural video codecs.", "weaknesses": "Although the proposed method achieves very significant video compression performance, the motivations behind the major contributions are not clearly discussed. Specifically, as the origin of HiNeRF, the hierarchically-encoded neural representation comes from the proposed new upsampling layer with hierarchical encoding on local feature grids. So why this design provides better performance? The paper claims that this design is similar to a convolutional layer over constant feature map. As one of the major contributions, it would be better to provide more comprehensive discussion and comparison.\n\nMoreover, according to the ablation studies, the major performance gain compared to previous methods seems to mainly come from the use of bilinear interpolation, which is a little bit incremental.\n\nFinally, it would be better to improve the paper writing. For example, as one of the contributions, the unified frame-wise and patch-wise representation is not mentioned in the abstract. And the analysis in ablation studies can be further improved. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: The motivations of the major components are not clear.**\n\nA: The advantage of the hierarchical encoding can be considered from different perspectives:\nFirstly, existing works have shown that positional encoding is helpful in learning high frequency information. In HiNeRV, when performing upsampling with the bilinear interpolation layer, it generates high dimensional but smoothed feature maps. By introducing the hierarchical encoding, we are able to encoded the relative positional information, which complements the interpolation layer. In our ablation studies, we found HiNeRV with MLP block still obtain acceptable performance, which is different from observations reported in previous works (the NeRV paper). We expect that this is primarily due to the use of the hierarchical encoding.\n\nWhen combining the input feature grids with multiple local feature grids (for extracting the hierarchical encoding) at different scales, we are able to represent different spatial locations in a hierarchical manner, which is a more efficient representation than using high dimension 'global' grids. For example, instead of having a global grid of size $T \\times H \\times W \\times C$, we can have a global grid with multiple local grids with a total size of $\\frac{T}{R} \\times \\frac{H}{S_0 \\times S_1 \\times ...} \\times \\frac{W}{S_0 \\times S_1 \\times ...} \\times C + T \\times S_0 \\times S_0 \\times C + ...$, where $R$ is a reduction factor to reduce the temporal resolution of the `global' grid, and the $S_0, S_1, ...$ are the up-scaling ratios, which are also the spatial sizes of the local grid as we use the local grid to encode the relative positional information during up-scaling. It also allows us to represent temporal signals at a higher resolution, given the same parameter budget. Our ablation studies have shown that the hierarchical encoding specifically benefit video sequences with fast motion (Jockey/ReadySetGo).\n\nTo make these points clearer, we will provide detailed justification regarding the motivation of our designs in the revised paper.\n\n\n**Q: The major gain is coming from the use of bilinear interpolation.**\n\nA: While we agree that bilinear interpolation plays an important role in our work, it can also be seen that other elements in HiNeRV (encoding, choice of block and unified representation) also affect the performance significantly. Specifically, we found that encoding helps with videos with fast motion -  e.g. improving the PSNR of Jockey/ReadySetGo by 0.78/1.06 dB. With the default choice (the ConvNeXT block) HiNeRV produces an increase of 1.15 dB in terms of the average PSNR compared to using normal convolutional blocks. By training as a unified representation, while still performing computation in patches, HiNeRV can obtain a 0.89 dB PSNR gain over a normal patch-wise representation.\n\nWe will make this point clearer in the revised paper.\n\n\n**Q: The paper writing has to be improved.**\n\nA: We commit to fully revising the writing of the paper to improve clarity and to include the use of the unified representation as one of the contributions in the abstract. We will also provide a more detailed discussion regarding the results of the ablation studies.\n\n\n**Q: Why the performance is worse on MCL-JCV dataset?**\n\nA: This is an interesting point and may be due to the different video frame-rates. In the UVG dataset, videos are captured at 120fps, whereas the frame rates for MCL-JCV are 25-30fps. INR-based approaches may perform better for content with more temporal redundancy compared to the anchors. We will comment on this in the revised paper.\n\n\n**Q: What is the best/worst compression cases?**\n\nA: Again, a good point. We will provide examples of best/worst cases in the revised paper, as shown in Fig 2-3 of the global response section. For HiNeRV, the best/worst performances are for the HoneyBee/Beauty sequences in the UVG dataset, and the videoSRC29/videoSRC08 sequences in the MCL-JCV dataset, respectively. We also found that, despite having worse PSNR performance than conventional codecs in the worst cases, it still outperforms all previous INR-based methods including NeRV and HNeRV in most cases. We also provide MS-SSIM results for these sequences in the global response, where HiNeRV outperforms both of the conventional codecs (HM (Random access) and x265 (veryslow)) and INR-based methods for most rates.\nUnlike HNeRV, we found that HiNeRV also performs well in representing highly dynamic scenes. As we can see from the results in the submitted paper (Table 2), HiNeRV outperforms HNeRV significantly on the ReadySetGo sequence.\n\n\n**Q: Compare encoding complexity with neural codes**\n\nA: We will provide more details regarding the encoding complexity in the revised paper, as shown in Table 2 in the general response section. And we will also provide comparison with neural video codecs in the revised paper.\n\nWe emphasize that encoding complexity is currently one of the major limitations of INR-base methods, due to the need for model training. However, INRs typically achieve better decoding speed. As mentioned in the supplementary materials (Section E), HiNeRV can achieve up to 10.9 FPS with the scale L (which corresponds to the middle/largest model for UVG/MCL-JCV, respectively), where other state-of-the-art neural codecs have a much slower decoding speed, e.g. DCVC-HEM can only achieve 1.9 FPS.\n\n\n**Q: How to compress the feature grid? How large is it?**\n\nA: We consider feature grids as normal parameters in this work, simply calculating the frequency of the quantized symbols and performing arithmetic coding. They contribute less than 20\\% of the total bitrate. We will revise the paper and provide this information.\n\n\n**Q: Typos in the paper.**\n\nA: Thank you for pointing this out. We will fix these typos and check throughout the paper."}, {"review_id": "c21EWnrwDl", "reviewer": "Reviewer_zJ4f", "summary": "This paper presents a novel method for video compression using implicit neural representation (INR), where video information is stored in the model weights. The authors explore various advanced techniques to enhance the architecture's capabilities. The primary contribution is the utilization of bilinear interpolation with hierarchical encoding for upsampling, which enables the construction of a higher-capacity network compared to using sub-pixel convolutional layers. Experimental results demonstrate that the proposed method outperforms existing INR models and is the first INR-based video codec to outperform HEVC.", "questions": "- Is the architecture the same for videos with a different number of frames? If not, how is the model's size adjusted? \n\n- Please include a comparison with FFNeRV, as it also outperforms HEVC.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-organized and easy to follow. \n\n- The authors achieve state-of-the-art INR-based video compression performance, particularly surpassing HEVC for the first time. \n\n- Detailed experimental settings and comparisons are provided.", "weaknesses": "- The authors do not provide the source code. \n\n- The comparison with FFNeRV, which also exhibits superior performance to HEVC, is missing. \n\n- The analysis of different interpolation methods, such as nearest neighbor and linear interpolation, is not included", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: Source code is not provided.**\n\nA: As mentioned in the supplementary material (line 4), we will provide the full implementation of HiNeRV when the paper is accepted.\n\n\n**Q: Comparison to FFNeRV is missing.**\n\nA: We did provide benchmark results of FFNeRV as shown in Section 4.1 (Table 1-2) for the video representation task.\n\nWe will also include its results for video compression in the revised version, as shown in the global response section (Figure 1). It is noted that FFNeRV is only comparable with the HEVC x265 with medium preset, where HiNeRV outperforms the HEVC x265 with the veryslow preset and is comparable with the HEVC HM with the randomaccess profile. In this case, we are confident in our claim that HiNeRV performs better than FFNeRV for the video compression task as well. \n\n\n**Q: Analysis of different interpolation methods is missing.**\n\nA: Following the reviewer's suggestion, we will provide the comparison results with the nearest neighbor up-sampling filter in the revised paper, as shown in the global response section (Table 1). Our results have shown that, even with nearest neighbor up-sampling, HiNeRV still exhibits superior performance, an average PSNR that is only slightly worse than with the default settings.\nIt is noted that, as we are using the same spatial grid size for both the first level feature maps and the grids, the employed trilinear interpolation is effectively the same as performing linear interpolation in the temporal dimension.\n\n\n**Q: How to adapt the network to different video lengths?**\n\nA: In our experiments, we share the same configurations for all videos in the same dataset for simplicity. For example, we use the same configurations for all 7 videos in the UVG dataset, although there are videos with 300/600 frames. For different datasets (e.g., UVG vs MCL-JCV), we did change the hyper parameters of the model, as provided in the supplementary material (Table 1)"}, {"review_id": "tmm3ZyzbrN", "reviewer": "Reviewer_H9aP", "summary": "This paper presents HiNeRV, a new INR model for video compression based on Hierarchically-encoded Neural Representation. HiNeRV introduces a new upsampling layer that replaces conventional sub-pixel layers in existing INRs. This layer combines bilinear interpolation with hierarchical encoding, utilizing multi-resolution local feature grids. Additionally, HiNeRV incorporates depth-wise convolutional and MLP layers to construct a deep and wide network architecture with higher capacity. Sufficient experiments on several datasets and tasks have demonstrated the effectiveness of the proposed method.", "questions": "-\tI am confused why the performance of HNeRV will be worse than NeRV performance, and whether the experiment is correct.\n-\tLimitations or bad case discussion and analysis are expected.\n-\tOther concerns have already been mentioned in Weakness.\n", "limitations": "\tYes, the authors have addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe ideas in this paper appear to be novel (to my knowledge) and straightforward and should be easily re-implement given the paper’s descriptions. \n-\tThe authors have made many technical improvements and engineering efforts to achieve competitive coding performance.\n-\tExtensive experiments have been done to demonstrate the effectiveness and compactness of the proposed method. Although HiNeRV has not been end-to-end optimized for video compression, it still significantly outperforms previous approaches with comparable bit rates.\n", "weaknesses": "-       There are several figures that may be improved for better illustration to the readers. In particular, I don't like Figure 2 since it seems too involved and without any summary in the caption. \n-\tAlthough the proposed method outperforms other methods in terms of encoding quality, the proposed method introduces more computational complexity, which makes encoding and decoding slower and is not conducive to real-time applications.\n-\tThere are some errors in the references, for example, reference 9 should be “HNeRV: a hybrid neural representation of videos” instead of  “HNeRV: neural representations for videos”. Please check your citations.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q: The figures require better illustration.**\n\nA: We will re-draw these figures and provide a more detailed description in the caption for improved clarity.\n\n\n**Q: The proposed method does not fit real-time application.**\n\nA: We agree that encoding and decoding are not yet sufficiently fast for real-time applications. However, it should be noted that HiNeRV achieves a much faster decoding speed than other learning-based codecs (such as DCVC/DCVC-HEM) as mentioned in the supplementary material (Section E). Furthermore, see Section D.3 of the supplementary material, we show that, by adjusting the architecture, HiNeRV has a similar decoding speed to HNeRV (up to 56.9 FPS with a modern GPU) with both better compression performance and lower theoretical complexity. \n\nThe runtime performance of the proposed model can be further improved by optimizing the implementation, inference with lower precision, structural pruning, etc. We will highlight this in the revised paper as our future work.\n\n\n**Q: Errors in the references.**\n\nA: Thank you for pointing out these issues. We will check all the citations thoroughly.\n\n\n**Q: Performance issue of HNeRV.**\n\nA: There are multiple possible reasons leading to this issue. In our experiments, we use both of the official implementations and configurations of NeRV and HNeRV. We notice that the configuration of NeRV in the NeRV repository is different to that of NeRV in the HNeRV repository. Specifically, the configuration of NeRV in the NeRV repository limits the minimum width of the convolutional layers to 96, where such width is only 12 in both configurations for NeRV/HNeRV in the HNeRV repository. Setting the minimum width to a larger value can lead to a better performance, as more computation will be done in higher resolutions. This could contribute to part of the performance difference between our results and the HNeRV paper.\n\nAlso, we have different experiment settings. Due to the stride ratios, HNeRV with the original configuration cannot represent videos with 1920x1080 resolution. In the original HNeRV paper, the authors crop the videos to 1920x960, while we perform padding to make HNeRV compatible with 1920x1080 resolution.\n\nFurthermore, in the HNeRV paper, the authors did not provide results of HNeRV for bpp $>$ 0.06 in the UVG dataset; we observed that NeRV performs better than HNeRV in this extended range. \n\nLastly, we observed that HNeRV is not stable when training with float16 for some sequences in the MCL-JCV dataset, as mentioned in Section A of the supplementary material. When training with float32, the performance of HNeRV can be improved, but still incomparable with HiNeRV. We will update the result of HNeRV with float32 for a fair comparison. We also reported the result shown in the global response section (Figure 1) with the updated numbers.\n\n\n**Limitation and worst case**\n \nWe will include a discussion regarding the limitations and the worst cases in  the revised paper.\n \nLimitations: \nAlthough our work has shown promising results in compression performance, similar to existing INR base approaches, the encoding time is still a limitation. The encoding time is due to the requirement for model training. Reducing encoding time will be an important direction for future work.\n\nWorst cases:\nIn the global response, we have provided the best/worst case (PSNR) performance for HiNeRV in the UVG/MCL-JCV datasets.  For the UVG dataset, we found that HiNeRV performs worst for the Beauty sequence; however the performance is still far better than for other INR based methods including NeRV and HNeRV, as well as for x265 (veryslow). For the MCL-JCV dataset, HiNeRV perform worst with the videoSRC08 sequence. In this sequence, HiNeRV is not as good as x265 (veryslow), but still far better than NeRV/HNeRV.\n\nFor both cases we found that HiNeRV still obtains superior performance in terms of MS-SSIM outperforming most baselines including HM (randomaccess) in the Beauty sequence, and comparable to HM (randomaccess) in the videoSRC08 sequence.\n\nAlso, as pointed out by reviewer JKXU, HNeRV has poor performance in representing high dynamic scenes. In Section 4.1 Table 2 of the paper, we show that HiNeRV has an average of 8.35 and 6.95 dB PSNR improvement over NeRV and HNeRV, respectively, for the highly dynamic scene for which HNeRV performed worst (ReadySetGo). Also, in contrast to HNeRV, we did not observe any clear disadvantages in representing specific sequence classes in HiNeRV."}]

Author rebuttal:
We thank all the reviewers for reviewing and providing detailed feedback to our submission. We will address the concerns individually.

Meta-review: This paper presents a simple method to improve INR-based video compression by introducing lightweight architectural changes and heirarchical positional encoding. It further investigates pruning and quantization to reduce the size of the model. These changes are validated by extensive experiments and comparisons. The authors addressed most reviewer concerns, and reviewers agreed that this work should be accepted by the end of the discussion phase.
Decision: Accept (poster)