Title: HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video compression

Implicit neural representations
Abstract: Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3\% overall bit rate saving over HNeRV and 43.4\% over DCVC on the UVG dataset, measured in PSNR).

HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation

Abstract

Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3% overall bit rate saving over HNeRV and 43.4% over DCVC on the UVG dataset, measured in PSNR). 1

1 Introduction

Implicit neural representations (INRs) have become popular due to their ability to represent and encode various scenes [37], images [45] and videos [45, 11]. INRs typically learn a coordinate to value mapping (e.g. mapping a pixel or voxel index to its color and/or occupancy) to support implicit reconstruction of the original signal. While these representations are usually instantiated as multilayer perceptrons (MLPs), existing MLP-based network can only represent video content with a low reconstruction quality and speed [11]. To address this limitation, recent works have employed Convolutional Neural Networks (CNNs) to perform a frame index to video frame mapping [11, 29, 5, 25, 10]. These CNN-based INRs are capable of reconstructing video content with higher quality and with a faster decoding speed, when compared to MLP-based approaches [45]. When using INRs for encoding videos, video compression can then be achieved by performing model compression for the individual input video. However, existing INR-based algorithms remain significantly inferior to state-of-the-art standard-based [52, 47, 8] and learning-based codecs [26, 41, 27, 28, 35]. For example, none of these INR-based codecs can compete with HEVC x265 [2] ( veryslow preset).

HNeRV

HiNeRV (ours)

UVG

40

35

88

34

33

2

2

PSNR

PSNR

32

32

31

30

NeRV

30

28

HNeRV

HiNeRV (ours)

29

28

50

100

150

200

250

300

50

100

150

200

250

300

Epochs

Epochs

Most INR-based models for videos [11, 29, 5, 25, 10] employ conventional convolutional layers or sub-pixel convolutional layers [42], which are less parameter efficient, and hence limit representation capacity within a given storage budget. In addition, most existing work employs Fourier-based positional encoding [37]; this has a long training time and can only achieve sub-optimal reconstruction quality [11, 29, 5]. In video compression, the training of INR models is equivalent to the encoding process, implying that most INR-based codecs require a long encoding runtime to obtain a satisfactory rate-quality performance [11]. However, some recent non-video INR models have utilized feature grids or a combination of grids and MLPs as the representation to speed up the convergence of INRs; this has improved the encoding speed by several orders of magnitude [15, 48, 38, 9].

In this paper, we propose a new INR model based on Hierarchically-encoded Neural Representation for video compression, HiNeRV. We replace commonly used sub-pixel conventional layers [42] in existing INRs for upsampling [11, 29, 5, 25, 10] by a new upsampling layer which embodies bilinear interpolation with hierarchical encoding that is sampled from multi-resolution local feature grids. These local grids offer increased parameter efficiency, as the number of parameters increases with the upsampling factor instead of the resolution. Moreover, the network is primarily based on MLPs and depth-wise convolutional layers (rather than dense convolutional layers). This enhances the representation capability and maximizes the performance for a given parameter count. This architectural design allows us to build a much deeper and wider network which offers a significantly better video encoding performance when compared to state-of-the-art INR-based coding approaches.

Furthermore, instead of learning a frame- [11] or patch-wise [5] representation, we show that by simply training with overlapped patches, HiNeRV can be seamlessly switched between both representation types, achieving a unified representation with improved performance compared to both frame-based and patch-based settings. This also provides flexibility for hardware implementation, where encoding and decoding processes can be performed either using frames to optimize the computational complexity, or as patches to minimize the memory footprint.

To achieve competitive coding performance, we also refine the model compression pipeline in [11], where pruning and fine-tuning are followed by model training, before quantization is applied. First, we used an adaptive pruning technique to reduce the negative impact of model pruning. Secondly, quantization-aware training is applied for fine-tuning the model performance before quantization. This enables lower bit depth quantization which achieves an improve rate-distortion trade-off.

The proposed method has been tested against existing INR-based video coding methods and state-ofthe-art conventional and learning-based video codecs on the UVG [36] and MCL-JCV [51] datasets. Notwithstanding the fact that HiNeRV has not been end-to-end optimized for video compression (i.e. pruning, quantization and entropy coding are not included in the training loop), it still significantly outperforms all INR-based methods (e.g., 72.3% overall BD-rate gain over HNeRV on the UVG database, measured in PSNR), and is also competitive compared with existing conventional and end-to-end learning-based video codec (e.g., 43.4% over DCVC, 38.7% over x265 veryslow ).

The primary contributions of this work are summarized below:

We propose HiNeRV, a new INR employing hierarchical encoding based neural representation.

We employ a unified representation by adding padding, which trades a small computation overhead for additional flexibility and performance gain.

3) We build a video codec based on HiNeRV and refine the model compression pipeline to better preserve the reconstruction quality of INRs by using adaptive pruning and quantization-aware training.

4) The compression performance of the proposed method is superior to existing INR models, and is comparable to many conventional/learning-based video coding algorithms. As far as we are aware, it is the first INR-based codec to significantly outperform HEVC (x265 veryslow ) [39].

2 Related work

2.1 Video compression

Video compression has long been a fundamental task in the field of computer vision and multimedia processing. As alternatives to the already popularized conventional codecs such as H.264/A VC [52], H.265/HEVC [47], and H.266/VVC [8], there has been a rapid increase in the adoption of deep learning techniques for video compression in recent years. This has typically involved replacing certain modules (e.g., motion compensation [3, 54], transform coding [55, 16] and entropy coding [6]) in the conventional pipeline with powerful learning-based models [53, 35].

In contrast, there has also been significant activity focused on the development of new coding architectures which allow end-to-end optimization. Lu et al. [32] proposed DVC that was further extended to enable major operations in both the pixel and the feature spaces [21, 20]. An alternative approach has focused on conditional [26, 30] instead of predictive coding to reduce the overall bitrate by estimating the probability model over several video frames. Furthermore, the characteristics of the differentiable frameworks have been exploited by [19, 49, 44], where both encoder and decoder (typically signaled by a model stream containing updated parameters) are overfitted to the video data during the evaluation to further enhance compression efficiency.

While effective, with some recent work [53, 43, 28] claiming to outperform the latest compression standards, these methods still follow the pipeline of conventional codecs, which may constrain the development of neural video compression methods. Moreover, learning-based video compression methods tend to be much more computational complex and often yield much slower decoding speed than conventional codecs. This often renders them impractical for real-time applications, especially considering the prevalence of high-quality and high-resolution videos consumed nowadays.

2.2 Implicit neural representation

Implicit neural representations (INRs) are being increasingly used to represent complicated natural signals such as images [45, 12, 14, 23], videos [45, 11], and vector-valued, volumetric content [37]. This type of approach benefits from incorporating positional encoding - a technique that embeds the positional input into a higher-dimensional feature space. Periodic functions [45, 33, 37] have first been utilized to improve the network's capability for learning high frequency information, and grid features [48, 38, 9] have then been applied to address their slow convergence speed and further improve the reconstruction quality.

More recently, Neural Representations for Videos (NeRV) [11] has re-formulated the INR for video signals to be frame-wise, achieving competitive reconstruction performance with very high decoding speed. NeRV approaches have inspired a trend of utilizing CNNs to encode the RGB values of videos with 1D frame coordinates [11, 29, 25, 10] or with 3D patch coordinates [5], and have demonstrated promise in various video tasks, including denoising [11], frame interpolation [11, 29, 25, 10], inpainting [5, 10], super-resolution [13] and video compression [11, 29, 5, 25, 10].

When INRs are applied for image and video compression, they typically convert the signal compression task into a model compression problem by incorporating weight pruning, quantization and entropy coding [17]. NeRV [11] and related works [29, 5, 25, 10] adopt the above approach for video compression. Although these have demonstrated fast decoding capability, they do not yet achieve a rate-distortion performance comparable to either conventional or learning-based codecs.

3 Method

Following the approach adopted in previous work [45, 11], we consider a video regression task where a neural network encodes a video V by mapping coordinates to either individual frames, patches or

pixels, where V ∈ R T × H × W × C , T , H , W and C are the number of frames in V , the height, the width and the number of channels of the video frames, respectively.

Fig. 2 (top) illustrates the high level structure of the proposed model, HiNeRV, which contains a base encoding layer, a stem layer, N HiNeRV blocks, and a head layer. In HiNeRV, each RGB video frame is spatially segmented into patches of size M × M , where each patch is reconstructed by one forward pass. The model first takes a patch coordinate ( i, j, t ) to compute the base feature maps X 0 with size M 0 × M 0 × C 0 . Here we always refer the coordinates to the integer index, such that 0 ≤ t < T , 0 ≤ j < H M and 0 ≤ i < W M . The following N HiNeRV blocks then upsample and process the feature maps progressively, where the n -th block produces the intermediate feature maps X n that have the size M n × M n × C n ( M N = M ). Finally, a head layer is used to project the feature maps to the output, Y , with the target size M × M × C .

3.1 Base encoding and stem

HiNeRV first maps the input patch coordinates into the base feature maps, X 0 , by

To compute the base feature maps, we first calculate the pixel coordinates (related to the corresponding video frame) of the patch. For a patch with size M 0 × M 0 , the frame-based pixel coordinates ( u frame , v frame ) can be computed by the patch-based pixel coordinates ( u patch , v patch ) for 0 ≤ u patch , v patch < M 0 , such that u frame = i × M 0 + u patch and v frame = j × M 0 + v patch . Then, by using the frame-based pixel coordinates, the positional encoding γ base ( i, j, t ) can be interpolated from the learned feature grids [25]. After that, we employ a stem convolutional layer F stem for projecting the feature maps to a desired number of channels C 0 .

It is noted that most existing INRs for video [11, 29, 5, 10] utilize the Fourier style encoding, i.e. they apply sin and cos functions to map coordinates into the positional encoding. However, such encoding contains only positional information, which requires additional layers (e.g. MLPs) to transform it into informative features. In contrast, we adopt grid-based encoding [25] as they that contain richer information than the Fourier encoding. Specifically, we use the multi-resolution temporal grids that were introduced in FFNeRV [25], where the various feature grids have different temporal resolutions. In FFNeRV, linear interpolation over the temporal dimension is used to obtain a slice that is used as the input feature map. In our case, we utilize both of the frame index and the frame-based coordinates, i.e., ( u frame , v frame , t ) , for interpolating the feature patches.

Although both high temporal resolution and a large number of channels are desirable for enhancing the expressiveness of the feature grids, this can result in greater model sizes and hence higher bitrates when the model is used for compression tasks. To maintain a compact multi-resolution grid, we increase the number of channels when reducing the temporal resolution at each grid level, i.e. the size of a grid is ⌊ T grid 2 l ⌋ × H grid × W grid × ( C grid × 2 l ) , for 0 ≤ l < L grid . Here L grid is the number of grids and T grid × H grid × W grid × C grid is the size of the first level grid.

3.2 HiNeRV block

The obtained base feature maps are then processed by N HiNeRV blocks, which progressively upsample and process the feature maps. Specifically, the n -th HiNeRV block, where 0 < n ≤ N , upsamples the input feature maps X n -1 with size M n -1 × M n -1 × C n -1 through bilinear interpolation U n with a scaling factor S n , such that M n = M n -1 × S n . We use bilinear interpolation, mainly due to its low computational cost and its capability to compute smooth upsampled maps. The HiNeRV block then computes a hierarchical encoding γ n ( i, j, t ) , matches its number of channels by a linear layer F enc and adds it to the upsampled feature maps, where γ n ( i, j, t ) matches the upsampled feature map size (see Section 3.4). Finally, it applies a set of network layers F n to enhance the representation to obtain the output X n , where we specify the number of layers by D n . For all the HiNeRV blocks except the first one ( 1 < n ≤ N ), the first layer in F n also reduces the number of channels by a factor, R , such that C n = ⌊ C 0 R n -1 ⌋ , to save the computational cost for processing high spatial resolution maps. The output of the n -th HiNeRV block, X n , has a size of M n × M n × C n , and can be written as

Figure 2 (bottom-left) shows the structure of HiNeRV block. Due to its observed superior performance, we employ ConvNeXt [31] as the network block in F n , a combination of the MLP layer with depthwise convolution. We also apply Layer Normalization [4] before the interpolation and the MLP layers, and we only use shortcut connections when the input and output dimensions are matched.

3.3 Head layer

The final output patch Y is computed by applying a linear layer with sigmoid activation, denoted by F head , on the output of the N -th HiNeRV block,

3.4 Upsampling with the hierarchical encoding

Existing NeRV-based approaches [11, 29, 5, 25, 10] use the sub-pixel convolutional layer [42] for feature map up-scaling. However, this has a high parameter complexity: K 2 × S 2 × C 1 × C 2 , where K , S , C 1 and C 2 represent the kernel size, the upsampling factor, the number of input and output channels, respectively. While neighboring features are highly correlated, the convolutional layer does not take this into account and learns a dense weight matrix to perform upsampling. This is inefficient especially when the model size is the concern for tasks like compression [11], because the low parameter efficiency limits the maximum depth and width of the networks, and thus the capacity.

Previous work [11] has shown that bilinear interpolation does not perform as well as convolutional layers. However, we observed that it is actually a better choice when the parameter count is fixed. By performing interpolation, we can utilize the saved parameter budget to build a network with higher capacity. While we can generate high-resolution maps using parameter-free bilinear interpolation, the resulting maps are smoothed, and subsequent neural network layers may struggle to produce high-frequency output from them. One way to model high-frequency signals is to introduce positional encoding [37] during upsampling. As mentioned in Section 3.1, grid-based encoding is preferred due to the good representation performance. However, adopting it to enhance high-resolution feature maps can be costly, as it requires high-dimensional grids. To address this limitation, we introduce a novel grid-based encoding approach called hierarchical encoding, which boosts the upsampling capability of bilinear interpolation without significantly increasing the storage cost.

Unlike normal grid-based encoding, which computes the encoding using global coordinates, hierarchical encoding utilizes local coordinates to encode relative positional information. Specifically, the local coordinate is the relative position of a pixel in the upsampled feature map to its nearest pixel in the original feature map. Because local coordinates have a much smaller range of values, the required feature grid is also much smaller. Moreover, when both the base encoding and hierarchical encoding are used, every position can be represented hierarchically, allowing us to efficiently encode positional information on high-resolution feature maps. During upsampling, the upsampled feature

maps are first produced through bilinear interpolation with an up-scaling factor S n . Then, for all frame-based pixel coordinates ( u frame , v frame ) in the upsampled feature maps, we compute the corresponding local pixel coordinates by u local = u frame mod S n and v local = v frame mod S n , and employ them to compute the encoding.

It is noted that the above encoding approach is similar to applying a convolutional layer over a constant feature map. To further enhance the capacity of this encoding, we model the feature grids as multi-temporal resolution grids [25], which can provide richer temporal information, similar to the base encoding. In the n -th HiNeRV block, there are L local levels of grids and the l -level grid has a size of ⌊ T local 2 l ⌋ × S n × S n × ( ⌊ C local R n -1 ⌋ × 2 l ) . The size of the local grids is scaled with the factor S n and can be adjusted by the hyper-parameter T local and C local . The number of channels of the grids is also scaled in proportion to the width of the HiNeRV block, i.e. by the reduction factor R . To obtain the hierarchical encoding, we perform trilinear interpolation by utilizing the frame index t with the local coordinate, i.e., ( u local , v local , t ) , to extract encodings from all levels, then concatenate the encodings and apply a linear layer F enc to match the encoding channels to the feature maps. To distinguish the grids for interpolating the hierarchical encoding from the one for the base encoding, i.e. the temporal grids , we refer these grids as the temporal local grids , because the hierarchical encoding is interpolated from these grids by using the local pixel coordinates. In Section 4.3, we demonstrated that the hierarchical encoding contributes to the superior performance HiNeRV. The process of upsampling with local encoding is shown in Figure 2 (bottom right).

3.5 Unifying frame-wise and patch-wise representations

Recent publications on INR for video can be classified into frame-wise [11, 29, 25, 10] or patch-wise representations [5]. Actually, in many of these networks, the initial feature maps can be easily computed either frame-wise or patch-wise, as the positional encoding depends only on the corresponding pixel coordinates. However, these two types of representations are not switchable because of boundary effects. In HiNeRV, we adopt a simple technique to unify frame-wise and patch-wise representations. When configuring HiNeRV as a patch-wise representation, we perform computation in overlapped patches, where we refer to the overlapped part as paddings, and the amount of padding pixels depends on the network configuration (e.g. the kernel sizes and/or the number of bilinear interpolation/convolutional layers). Such overlapped patches have previously been used for tasks such as super-resolution [22], but have not been applied in NeRV-based methods. When performing encoding in patches without proper padding, missing values for operations such as convolution can result in discontinuities between patch boundaries. Moreover, networks trained in patches do not perform well when inferencing in frames due to boundary effects. In our implementation, we perform intermediate computation with padded patches and crop the non-overlapped parts as the output patches, while with the frame configuration, paddings are not required. By adding paddings, we ensure generation of the same output for both frame-wise and patch-wise configurations.

Although adding paddings introduces additional computational overheads, it does provide the following benefits: (i) it allows parallel computation within the same frame, and reduces the minimum memory requirement for performing forward and backward passes. This shares a design concept with conventional block-based video codecs, and can potentially benefit the compression of immersive video content with higher spatial resolutions (where performing frame-wise calculation may not be supported by the hardware); (ii) It improves the training efficiency when compared to a frame-wise representation, as we can randomly sample patches during training [22], which can better approximate the true gradient and speed up the convergence; (iii) It can also enhance the final reconstruction quality compared to a patch-wise representation without suffering boundary effects. By applying the above technique, HiNeRV can be flexibly configured as either frame-based or patch-based representation without retraining. Our ablation study verifies these benefits by comparing this approach to both the frame-based and patch-based variants (see Section 4.3).

3.6 The model compression pipeline

To further enhance the video compression performance, we refined the original model compression pipeline in NeRV [11], which has been used in a series of related work [29, 5, 25, 10]. In [11], model training is followed by weight pruning with fine-tuning to lower model sizes. Post-training quantization is then utilized to reduce the precision of each weight, while entropy coding is further employed for lossless compression. In this work, we made two primary changes to this pipeline:

(i) applying an adaptive weighting to the parameters in different layers for pruning, and (ii) using quantization-aware training with Quant-Noise [46] to minimize the quantization error.

Model Pruning in NeRV [11, 29, 5, 25, 10] is typically performed globally with respect to the magnitude of individual weights, with fine-tuning applied after pruning [18]. While various nonmagnitude based pruning methods exist (e.g. OBD [24]), here we developed a simple, modified magnitude-based method for network pruning. Intuitively, we assume that wider layers have more redundancy within their parameters - hence pruning these layers tends to have less impact than on the shallower layers. To alleviate the negative impact of pruning, we weight each neuron using both its ℓ 1 norm and the size of the corresponding layer. Specifically, for a layer with P parameters, θ p , 0 < p ≤ P , we compute a score, | θ p | P λ , which reflects the neuron importance, where λ is a hyper-parameter (0.5 in our experiments). The pruning is then performed as usual. By applying this weighting scheme, layers with fewer parameters, such as depth-wise convolutional layers and output layers, have less chance to be pruned, while layers in the early stage of the network are more likely to be pruned.

Weight quantization plays an important role in model compression, significantly reducing final model size [11]. While the results in [11] have shown that the quantization error is not significant when reducing the weight bit depth to 8 bits, further increasing the quantization level is still meaningful for compression tasks. Unlike other related works [11, 29, 5, 25, 10] that adopted 8 bit quantization, we found that an improved rate-distortion trade-off can be achieved by using 6 bits quantization if a quantization-aware training methodology is applied. In particular, we perform a short fine-tuning with Quant-Noise [46] after weight pruning, which can effectively reduce the quantization error. However, unlike the original implementation, we do not use STE [7] for computing the gradient of the quantized weights due to its inferior performance.

4 Experiments

4.1 Video representation

To evaluate the effectiveness of the proposed model we benchmarked HiNeRV against five related works: NeRV [11], E-NeRV [29], PS-NeRV [5], FFNeRV [25] and HNeRV [10] on the Bunny [1] ( 1280 × 720 with 132 frames) and the UVG datasets [36] (7 videos at 1920 × 1080 with a total of 3900 frames). For each video, we trained all networks at multiple scales, and kept their number of parameters similar at each scale. Three scales were set up to target the S/M/L scales in NeRV [11] for the UVG datasets, while two different scales XXS/XS together with scale S were configured for the Bunny dataset. We reported the encoding and decoding speeds in frames per second, measured with A100 GPU. The number of parameters corresponding to each scale are reported in Table 1 and 2.

For all models tested, we set the number of training epochs to 300 and batch size (in video frames) to 1. We used the same optimizer as in [11], and employed the same learning objectives for all NeRV-based methods as in the original literature [11, 29, 5, 25, 10]. For HiNeRV, we empirically found that it is marginally better to adopt a larger learning rate of 2 e -3 with global norm clipping which is commonly used for Transformer-based networks [50]. We used the learning rate of 5 e -4 which is a common choice for the other networks as in their original literature [11, 29, 5, 25]. We also adopted a combination of ℓ 1 loss and MS-SSIM loss (with a small window size of 5 × 5 rather than 11 × 11 ) for HiNeRV, as we observed that the MS-SSIM loss with a small window size leads to a better performance. For HiNeRV and PS-NeRV [5], we randomly sample patches instead of frames during training, but we scale the number of patches in each batch to keep the same effective batch size. It is noted that the original configuration of HNeRV [10] has an input/output size of 1280 × 640 / 1920 × 960 with strides (5, 4, 4, 2, 2)/(5, 4, 4, 3, 2), and we pad the frames of HNeRV in order to fit the 1280 × 720 / 1920 × 1080 videos. This was found to work better than changing the strides in HNeRV. Detailed configuration of HiNeRV is summarized in the Supplementary Material .

It can be observed (Table 1 and 2) that our proposed HiNeRV outperforms all benchmarked models in terms of reconstruction quality at each scale on both Bunny [1] and UVG [36] datasets. We also note that HiNeRV performs better than all the other methods on all test sequences (with various spatial and temporal characteristics) in the UVG database, exhibiting better reconstruction quality in terms of PSNR. In particular, for some video sequences where existing INRs performed poorly [10], HiNeRV offers significant improvement (>7.9dB over NeRV on ReadySetGo). While the encoding

and decoding speeds of HiNeRV are slower than that of other methods, HiNeRV achieves a higher overall PSNR figure with fewer MACs. Further optimization may help reduce its latency. In the Supplementary Material , we conducted experiments with faster variants of HiNeRV, demonstrating that HiNeRV can achieve a satisfactory trade-off among latency, model size and reconstruction quality simultaneously.

Figure 1 (right) shows the performance of HiNeRV, NeRV and HNeRV (at scale S) in terms of the reconstruction quality with various epochs of training. We observe that our model with 37 epochs achieves similar reconstruction quality of HNeRV with 300 epochs on both datasets.

4.2 Video compression

To evaluate video compression performance, we compared HiNeRV with two INR-based models: NeRV [11] and HNeRV [10]; with two conventional codecs: HEVC/H.265 HM 18.0 (Main Profile with Random Access ) [39, 40] and x265 ( veryslow preset with B frames) [2]; and with two stateof-the-art learning-based codecs: DCVC [26], DCVC-HEM [27] and VCT [35]. These were all compared using two test databases: UVG [36] and MCL-JCV [51]. Unlike previous work [11, 5], which concatenates the videos and uses a single network to compress multiple videos, we train all the models for each video separately. For the training of NeRV, HNeRV and HiNeRV, we use the configurations described in Section 4.1, but with two more scales, namely XL and XXL, for encoding videos with the highest rate (See the Supplementary Material ). We apply pruning and quantization as described in Section 3.6. In particular, we prune these three models to remove 15% of their weights and fine-tune the models for another 60 epochs. These models are further optimized with Quant-Noise [46] with 90% noise ratio for 30 epochs. Here we use the same learning rate scheduling for fine-tuning, but employ 10% of the original learning rate in the optimization with Quant-Noise. To obtain the actual rate, we perform arithmetic entropy coding [34] and combine all essential information including the pruning masks and the quantization parameters into bitstreams.

Figure 3 reports the overall rate quality performance on the UVG [36] and the MCL-JCV [51] datasets. Table 3 summarizes the average Bjøntegaard Delta (BD) rate results for both databases. All results show that HiNeRV offers competitive coding efficiency compared to most conventional codecs

UVG

UVG

40

0.98

38

0.96

36

S-SSIM

PSNR

x265(veryslow)

34

HM (RA)

MS-

0.94

-X-DCVC

-X-DCVC-HEM

32

-×-VCT

0.92

NeRV

HNeRV

30

HiNeRV

0.90

0.00

0.05

0.10

0.15

0.20

0.25

0.00

0.05

0.10

0.15

0.20

0.25

MCL-JCV

MCL-JCV

40

0.98

38

36

MS-SSIM

0.96

PSNR

34

0.94

32

0.92

30

0.90

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

0.00 0.05 0.10 0.15 0.20 0.25 0.30 0.35

and learning-based methods. This represents a significant improvement over existing NeRV-based approaches (this is also confirmed by the visual comparison with HNeRV in Figure 1). In particular, it is observed that HiNeRV outperforms x265 ( veryslow ) [2], DCVC [26] and VCT [35] based on PSNR. As far as we are aware, this is the first INR-based codec which can achieve such performance. We also observe that HiNeRV offers better performance compared to H.265 HM ( Random Access ) based on MS-SSIM. It should be noted that the results of each learning-based codec reported here are based on two model checkpoints, one for optimizing PSNR and the other for MS-SSIM, while all the results for HiNeRV are however based on the same checkpoint.

Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization are not optimized in the loop), it nonetheless outperforms many state-of-the-art end-toend optimized learning-based approaches. This demonstrates the significant potential of utilizing INRs for video compression applications. In the Supplementary Material , comparison with additional baseline is provided.

4.3 Ablation study

To verify the contribution of various components in HiNeRV, we generated a number of variants of the original model, and evaluated them for video representation on the UVG dataset [36] (all the sequences were down-sampled to 1280 × 720 for reduction the amount of computation). For all experiments, we followed the settings in Section 4.1, and performed training targeting scale S (by adjusting the width of the network to keep the similar model sizes). All results are shown in Table 4. We also included the results of NeRV [11] and HNeRV [10] for reference.

Bilinear interpolation with hierarchical encoding. The contribution of bilinear interpolation with hierarchical encoding was verified by comparing it with alternative upsampling layers, sub-pixel convolutional layer [42] with 1 × 1 (V1) and 3 × 3 (V2) kernel sizes.

Upsampling encodings. Four variants are trained to confirm the effectiveness of the upsampling encodings including (V3) w/o encoding; (V4) with the Fourier encoding [37]; (V5) using Fourier encoding with local coordinates (computed in the hierarchical encoding); (V6) using the grid-based encoding with local coordinates, i.e. the hierarchical encoding without the temporal dimension.

ConvNeXt block. The employed ConvNeXt block [31] has been compared with (V7) the MLP block in transformer [50]; (V8) the block containing two convolutional layers with , where we use 3 × 3 and 1 × 1 kernel size to keep the receptive field consistent with the ConvNeXt block used in our paper.

Unified representations. V9 and V10 have been generated for frame- and patch-wise configurations.

Interpolation methods. V11 replaces the bilinear interpolation by the nearest neighbor interpolation.

The ablation study results are presented in Table. 4 which shows that the full HiNeRV model outperforms all alternatives (V1-V11) on the UVG dataset in terms of the overall reconstruction quality. This confirms the contribution of each primary component of the design. More discussion regarding the results can be found in the Supplementary Material .

5 Conclusion

In this paper, a new neural representation model, HiNeRV, has been proposed for video compression, which exhibits superior coding performance over many conventional and learning-based video codecs (including those based on INRs). The improvements demonstrated are associated with new innovations including bilinear interpolation based hierarchical encoding, a unified representation and a refined model compression pipeline.

Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization are not optimized in the loop), it nonetheless achieves comparable performance to state-of-the-art end-to-end optimized learning-based approaches, with significant improvement over existing NeRV-based algorithms. This demonstrates the great potential of utilizing INRs for video compression applications. For example, this is the first INR-based video codec which can outperform HEVC HM Random Access mode based on MS-SSIM.

Future work should focus on incorporation of entropy coding and quantization to achieve full end-to-end optimization.

Acknowledgment

This work was jointly funded by UK EPSRC (iCASE Awards), BT, the UKRI MyWorld Strength in Places Programme and the University of Bristol. We also thank the support by the Advanced Computing Research Centre, University of Bristol, for providing the computational facilities.

References

scikit-video: Video Processing in Python. http://www.scikit-video.org/ .

x265. https://www.videolan.org/developers/x265.html .

E. Agustsson, D. Minnen, N. Johnston, J. Ballé, S. J. Hwang, and G. Toderici. Scale-Space Flow for End-to-End Optimized Video Compression. In CVPR , pages 8500-8509. Computer Vision Foundation / IEEE, 2020.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer Normalization, 2016.

Y. Bai, C. Dong, C. Wang, and C. Yuan. PS-NeRV: Patch-wise Stylized Neural Representations for Videos. In 2023 IEEE International Conference on Image Processing (ICIP) , pages 41-45. IEEE, 2023.

J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston. Variational image compression with a scale hyperprior. In ICLR . OpenReview.net, 2018.

Y. Bengio, N. Léonard, and A. Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013.

B. Bross, Y. Wang, Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J. Ohm. Overview of the Versatile Video Coding (VVC) Standard and its Applications. IEEE Trans. Circuits Syst. Video Technol. , 31(10):3736-3764, 2021.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. TensoRF: Tensorial Radiance Fields. In ECCV (32) , volume 13692 of Lecture Notes in Computer Science , pages 333-350. Springer, 2022.

H. Chen, M. Gwilliam, S. Lim, and A. Shrivastava. HNeRV: A Hybrid Neural Representation for Videos. In CVPR , pages 10270-10279. IEEE, 2023.

H. Chen, B. He, H. Wang, Y. Ren, S. Lim, and A. Shrivastava. NeRV: Neural Representations for Videos. In NeurIPS , pages 21557-21568, 2021.

Y. Chen, S. Liu, and X. Wang. Learning Continuous Image Representation With Local Implicit Image Function. In CVPR , pages 8628-8638. Computer Vision Foundation / IEEE, 2021.

Z. Chen, Y. Chen, J. Liu, X. Xu, V. Goel, Z. Wang, H. Shi, and X. Wang. VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution. In CVPR , pages 2037-2047. IEEE, 2022.

E. Dupont, A. Goli´ nski, M. Alizadeh, Y . W. Teh, and A. Doucet. COIN: COmpression with Implicit Neural representations, 2021.

S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance Fields without Neural Networks. In CVPR , pages 5491-5500. IEEE, 2022.

G. Gao, P. You, R. Pan, S. Han, Y. Zhang, Y. Dai, and H. Lee. Neural Image Compression via Attentional Multi-scale Back Projection and Frequency Decomposition. In ICCV , pages 14657-14666. IEEE, 2021.

S. Han, H. Mao, and W. J. Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding, 2016.

S. Han, J. Pool, J. Tran, and W. J. Dally. Learning both Weights and Connections for Efficient Neural Network. In NIPS , pages 1135-1143, 2015.

G. He, C. Wu, L. Li, J. Zhou, X. Wang, Y. Zheng, B. Yu, and W. Xie. A Video Compression Framework Using an Overfitted Restoration Neural Network. In CVPR Workshops , pages 593-597. Computer Vision Foundation / IEEE, 2020.

Z. Hu, G. Lu, J. Guo, S. Liu, W. Jiang, and D. Xu. Coarse-To-Fine Deep Video Coding with HyperpriorGuided Mode Prediction. In CVPR , pages 5911-5920. IEEE, 2022.

Z. Hu, G. Lu, and D. Xu. FVC: A New Framework Towards Deep Video Compression in Feature Space. In CVPR , pages 1502-1511. Computer Vision Foundation / IEEE, 2021.

A. Kappeler, S. Yoo, Q. Dai, and A. K. Katsaggelos. Video Super-Resolution With Convolutional Neural Networks. IEEE Trans. Computational Imaging , 2(2):109-122, 2016.

T. Ladune, P. Philippe, F. Henry, G. Clare, and T. Leguay. COOL-CHIC: Coordinate-based Low Complexity Hierarchical Image Codec, 2023.

Y. LeCun, J. S. Denker, and S. A. Solla. Optimal Brain Damage. In NIPS , pages 598-605. Morgan Kaufmann, 1989.

J. C. Lee, D. Rho, J. H. Ko, and E. Park. FFNeRV: Flow-Guided Frame-Wise Neural Representations for Videos. In ACM Multimedia . ACM, 2023.

J. Li, B. Li, and Y . Lu. Deep Contextual Video Compression. In NeurIPS , pages 18114-18125, 2021.

J. Li, B. Li, and Y . Lu. Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression. In ACM Multimedia , pages 1503-1511. ACM, 2022.

J. Li, B. Li, and Y . Lu. Neural Video Compression with Diverse Contexts. In CVPR , pages 22616-22626. IEEE, 2023.

Z. Li, M. Wang, H. Pi, K. Xu, J. Mei, and Y. Liu. E-NeRV: Expedite Neural Video Representation with Disentangled Spatial-Temporal Context. In ECCV (35) , volume 13695 of Lecture Notes in Computer Science , pages 267-284. Springer, 2022.

J. Liu, S. Wang, W. Ma, M. Shah, R. Hu, P. Dhawan, and R. Urtasun. Conditional Entropy Coding for Efficient Video Compression. In ECCV (17) , volume 12362 of Lecture Notes in Computer Science , pages 453-468. Springer, 2020.

Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A ConvNet for the 2020s. In CVPR , pages 11966-11976. IEEE, 2022.

G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and Z. Gao. DVC: An End-To-End Deep Video Compression Framework. In CVPR , pages 11006-11015. Computer Vision Foundation / IEEE, 2019.

I. Mehta, M. Gharbi, C. Barnes, E. Shechtman, R. Ramamoorthi, and M. Chandraker. Modulated Periodic Activations for Generalizable Local Functional Representations. In ICCV , pages 14194-14203. IEEE, 2021.

F. Mentzer, E. Agustsson, M. Tschannen, R. Timofte, and L. V. Gool. Practical Full Resolution Learned Lossless Image Compression. In CVPR , pages 10629-10638. Computer Vision Foundation / IEEE, 2019.

F. Mentzer, G. Toderici, D. Minnen, S. Caelles, S. J. Hwang, M. Lucic, and E. Agustsson. VCT: A Video Compression Transformer. In NeurIPS , 2022.

A. Mercat, M. Viitanen, and J. Vanne. UVG Dataset: 50/120fps 4K Sequences for Video Codec Analysis and Development. In MMSys , pages 297-302. ACM, 2020.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV (1) , volume 12346 of Lecture Notes in Computer Science , pages 405-421. Springer, 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. , 41(4):102:1-102:15, 2022.

C. R. K. Sharman, R. Sjoberg, and G. Sullivan. High Efficiency Video Coding (HEVC) Test Model 16 (HM 16) Improved Encoder Description Update 14. In the JVET meeting , number JVET-AN§002. ITU-T and ISO/IEC, 2020.

K. Sharman and K. Suehring. Common test conditions for HM video coding experiments. In the JCT-VC meeting , number JCTVC-AF1100. ITU-T, ISO/IEC, 2018.

X. Sheng, J. Li, B. Li, L. Li, D. Liu, and Y . Lu. Temporal Context Mining for Learned Video Compression. IEEE Transactions on Multimedia , pages 1-12, 2022.

W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop, D. Rueckert, and Z. Wang. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In CVPR , pages 1874-1883. IEEE Computer Society, 2016.

Y. Shi, Y . Ge, J. Wang, and J. Mao. AlphaVC: High-Performance and Efficient Learned Video Compression. In ECCV (19) , volume 13679 of Lecture Notes in Computer Science , pages 616-631. Springer, 2022.

M. K. Shirkoohi, V. Sivaraman, and M. Alizadeh. Efficient Video Compression via Content-Adaptive Super-Resolution. In ICCV , pages 4501-4510. IEEE, 2021.

V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representations with Periodic Activation Functions. In NeurIPS , 2020.

P. Stock, A. Fan, B. Graham, E. Grave, R. Gribonval, H. Jégou, and A. Joulin. Training with Quantization Noise for Extreme Model Compression. In ICLR . OpenReview.net, 2021.

G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Trans. Circuits Syst. Video Technol. , 22(12):1649-1668, 2012.

C. Sun, M. Sun, and H. Chen. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In CVPR , pages 5449-5459. IEEE, 2022.

T. van Rozendaal, J. Brehmer, Y. Zhang, R. Pourreza, A. Wiggers, and T. S. Cohen. Instance-Adaptive Video Compression: Improving Neural Codecs by Training on the Test Set, 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C. J. Kuo. MCL-JCV: A JND-based H.264/AVC video quality assessment dataset. In ICIP , pages 1509-1513. IEEE, 2016.

T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra. Overview of the H.264/AVC Video Coding Standard. IEEE Trans. Circuits Syst. Video Technol. , 13(7):560-576, 2003.

J. Xiang, K. Tian, and J. Zhang. MIMT: Masked Image Modeling Transformer for Video Compression. In ICLR . OpenReview.net, 2023.

R. Yang, F. Mentzer, L. V. Gool, and R. Timofte. Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement. In CVPR , pages 6627-6636. Computer Vision Foundation / IEEE, 2020.

Y. Zhu, Y. Yang, and T. Cohen. Transformer-based Transform Coding. In ICLR . OpenReview.net, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HiNeRV, a new INR model for video compression based on Hierarchically-encoded Neural Representation. HiNeRV introduces a new upsampling layer that replaces conventional sub-pixel layers in existing INRs. This layer combines bilinear interpolation with hierarchical encoding, utilizing multi-resolution local feature grids. Additionally, HiNeRV incorporates depth-wise convolutional and MLP layers to construct a deep and wide network architecture with higher capacity. Sufficient experiments on several datasets and tasks have demonstrated the effectiveness of the proposed method.\n\nSTRENGTHS:\n-\tThe ideas in this paper appear to be novel (to my knowledge) and straightforward and should be easily re-implement given the paper’s descriptions. \n-\tThe authors have made many technical improvements and engineering efforts to achieve competitive coding performance.\n-\tExtensive experiments have been done to demonstrate the effectiveness and compactness of the proposed method. Although HiNeRV has not been end-to-end optimized for video compression, it still significantly outperforms previous approaches with comparable bit rates.\n\nWEAKNESSES:\n-       There are several figures that may be improved for better illustration to the readers. In particular, I don't like Figure 2 since it seems too involved and without any summary in the caption. \n-\tAlthough the proposed method outperforms other methods in terms of encoding quality, the proposed method introduces more computational complexity, which makes encoding and decoding slower and is not conducive to real-time applications.\n-\tThere are some errors in the references, for example, reference 9 should be “HNeRV: a hybrid neural representation of videos” instead of  “HNeRV: neural representations for videos”. Please check your citations.\n\nQUESTIONS:\n-\tI am confused why the performance of HNeRV will be worse than NeRV performance, and whether the experiment is correct.\n-\tLimitations or bad case discussion and analysis are expected.\n-\tOther concerns have already been mentioned in Weakness.\n\nLIMITATIONS:\nYes, the authors have addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a novel method for video compression using implicit neural representation (INR), where video information is stored in the model weights. The authors explore various advanced techniques to enhance the architecture's capabilities. The primary contribution is the utilization of bilinear interpolation with hierarchical encoding for upsampling, which enables the construction of a higher-capacity network compared to using sub-pixel convolutional layers. Experimental results demonstrate that the proposed method outperforms existing INR models and is the first INR-based video codec to outperform HEVC.

STRENGTHS:
- This paper is well-organized and easy to follow. 

- The authors achieve state-of-the-art INR-based video compression performance, particularly surpassing HEVC for the first time. 

- Detailed experimental settings and comparisons are provided.

WEAKNESSES:
- The authors do not provide the source code. 

- The comparison with FFNeRV, which also exhibits superior performance to HEVC, is missing. 

- The analysis of different interpolation methods, such as nearest neighbor and linear interpolation, is not included

QUESTIONS:
- Is the architecture the same for videos with a different number of frames? If not, how is the model's size adjusted? 

- Please include a comparison with FFNeRV, as it also outperforms HEVC.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new type of implicit neural representations (INR) with hierarchical positional encoding for video compression, named HiNeRV. Four points are proposed: (1) a new upsampling layer with bilinear interpolation and hierarchical encoding of feature grids, (2) the use of depth-wise convolution and MLP layers instead of conventional convolution layers, (3) the unified representation of frame-wise and patch-wise INR by adding padding, and (4) refined training pipeline for HiNeRV model compression. The proposed method achieves significant improvement over all existing INRs baselines and competitive performance to recent neural video codecs.\n\nSTRENGTHS:\nSignificant rate-distortion performance as an INR-based video codec. Largely closing the gap between INR-based method and the SOTA neural video codecs.\n\nWEAKNESSES:\nAlthough the proposed method achieves very significant video compression performance, the motivations behind the major contributions are not clearly discussed. Specifically, as the origin of HiNeRF, the hierarchically-encoded neural representation comes from the proposed new upsampling layer with hierarchical encoding on local feature grids. So why this design provides better performance? The paper claims that this design is similar to a convolutional layer over constant feature map. As one of the major contributions, it would be better to provide more comprehensive discussion and comparison.\n\nMoreover, according to the ablation studies, the major performance gain compared to previous methods seems to mainly come from the use of bilinear interpolation, which is a little bit incremental.\n\nFinally, it would be better to improve the paper writing. For example, as one of the contributions, the unified frame-wise and patch-wise representation is not mentioned in the abstract. And the analysis in ablation studies can be further improved.\n\nQUESTIONS:\nBelow are some further questions: \n(1)\tWhy the proposed method performs a bit worse on MCL-JCV dataset? \n(2)\tIs it possible to provide the best/worst compression cases? In HNeRV, the authors show their model has limited performance on highly dynamic scenes. Is there a similar issue in HiNeRV?\n(3)\tIs it possible to provide the comparison of encoding complexity with recent neural video codecs?\n(4)\tHow do you compress the feature grid? How much bitrate cost of the feature grid?\n(5)\tSome typos in the paper, e.g. Line 60: before quantiszation -> before quantization\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
The authors propose a 2D implicit neural compression method that combines bilinear interpolation and hierarchical encoding. This method is based on the premise that bilinear interpolation is a better choice for neural compression than convolution operations.

STRENGTHS:
1. The proposed method distinguishes itself from other existing methods by utilizing bilinear interpolation for feature upsampling.

2. The paper investigates model compression techniques, including pruning and quantization, to further reduce the size of the model.

3. The effectiveness of the proposed modules is convincingly demonstrated through an ablation study.

WEAKNESSES:
1. The explanation of the hierarchical coding process in section 3.4 and Fig.2 is unclear and difficult to follow, requiring further clarification.
2. It is crucial for the paper to include results on encoding and decoding time, as this information holds significant relevance in real-world scenarios.
3. The use of deep models with overlapped patches may lead to an unfair comparison with other implicit compression methods due to the increase in training pixels it implies. However, the paper lacks sufficient explanation on this matter, particularly regarding how the final reconstructed pixels are determined by overlapping blocks.

QUESTIONS:
Does the training method in this paper involve training only one model on the entire dataset for inference, as mentioned in NeRV[10]? How does this encoding/decoding process apply to actual scenarios? If so, how is the compression performance on a single-sequence dataset like bunny?

LIMITATIONS:
While the paper effectively improves the compression efficiency of the implicit codec, there is scope for future research in optimizing encoding time and latency.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a hierarchical encoding block for INR-based video compression. In particular, the authors replace the commonly-used sub-pixel/transpose convolution for upsampling with the bi-linear interpolation. The saved parameters are used to increase the capacity of the MLP encoders in order to achieve higher expressiveness of the model. Reportedly, this work is the first INR-based method that achieves comparable rate-distortion performance to the state-of-the-art VAE-based learned methods.\n\nSTRENGTHS:\n(1) The paper is easy to follow. \n(2) The coding results are promising.\n\nWEAKNESSES:\n(1) The novelty of this paper is rather now. Essentially, the authors replace the commonly-used sub-pixel/transpose convolution for upsampling with the bilinear interpolation. In doing so, the saved parameters are used to increase the number of MLP encoders. The gain appears to come mainly from a simple tweak to the network architecture.\n\n(2) The network architecture of the proposed method is not new, and is more like a trivial combination of several existing works. For example, the Grid and Stem modules are from [1] and [2]. After index transformation, the feature upsampling part is from [3]. \n\n[1] J. C. Lee, D. Rho, J. H. Ko, and E. Park. FFNeRV: Flow-guided frame-wise neural representations for videos. arXiv preprint arXiv:2212.12294, 2022.\n\n[2] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016\n\n[3] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.\n\n(3) The authors should include the comparison with DCVC-HEM in the main paper. Not sure why DCVC (which is a weaker baseline) is used instead. Also, please clarify the GOP size. It seems that GOP=12 is adopted for DCVC-HEM. It was argued in several papers (including DCVC-HEM) that a larger GOP should be used.  \n\n(4) The comparison with HNeRV may be unfair, because the network of HNeRV is designed to represent the whole dataset. As such, its network is quite heavy, and can actually be reduced considerably when it is used to represent only one sequence. In this sense, the per-sequence rate reported for HNeRV may be overestimated. \n\n[4] H. Chen, M. Gwilliam, S.-N. Lim, and A. Shrivastava. HNeRV: Neural representations for videos. In CVPR, 2023.\n\nQUESTIONS:\nSee my comments in the weaknesses section.\n\nLIMITATIONS:\nThe encoding time is not addressed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}