SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Although adding paddings introduces additional computational overheads, it does provide the following benefits: (i) it allows parallel computation within the same frame, and reduces the minimum memory requirement for performing forward and backward passes. This shares a design concept with conventional block-based video codecs, and can potentially benefit the compression of immersive video content with higher spatial resolutions (where performing frame-wise calculation may not be supported by the hardware); (ii) It improves the training efficiency when compared to a frame-wise representation, as we can randomly sample patches during training [22], which can better approximate the true gradient and speed up the convergence; (iii) It can also enhance the final reconstruction quality compared to a patch-wise representation without suffering boundary effects. By applying the above technique, HiNeRV can be flexibly configured as either frame-based or patch-based representation without retraining. Our ablation study verifies these benefits by comparing this approach to both the frame-based and patch-based variants (see Section 4.3).
3.6 The model compression pipeline
To further enhance the video compression performance, we refined the original model compression pipeline in NeRV [11], which has been used in a series of related work [29, 5, 25, 10]. In [11], model training is followed by weight pruning with fine-tuning to lower model sizes. Post-training quantization is then utilized to reduce the precision of each weight, while entropy coding is further employed for lossless compression. In this work, we made two primary changes to this pipeline:
(i) applying an adaptive weighting to the parameters in different layers for pruning, and (ii) using quantization-aware training with Quant-Noise [46] to minimize the quantization error.
Model Pruning in NeRV [11, 29, 5, 25, 10] is typically performed globally with respect to the magnitude of individual weights, with fine-tuning applied after pruning [18]. While various non-magnitude based pruning methods exist (e.g. OBD [24]), here we developed a simple, modified magnitude-based method for network pruning. Intuitively, we assume that wider layers have more redundancy within their parameters - hence pruning these layers tends to have less impact than on the shallower layers. To alleviate the negative impact of pruning, we weight each neuron using both its  $\ell 1$  norm and the size of the corresponding layer. Specifically, for a layer with  $P$  parameters,  $\theta_p, 0 < p \leq P$ , we compute a score,  $\frac{|\theta_p|}{P^{\lambda}}$ , which reflects the neuron importance, where  $\lambda$  is a hyper-parameter (0.5 in our experiments). The pruning is then performed as usual. By applying this weighting scheme, layers with fewer parameters, such as depth-wise convolutional layers and output layers, have less chance to be pruned, while layers in the early stage of the network are more likely to be pruned.
Weight quantization plays an important role in model compression, significantly reducing final model size [11]. While the results in [11] have shown that the quantization error is not significant when reducing the weight bit depth to 8 bits, further increasing the quantization level is still meaningful for compression tasks. Unlike other related works [11, 29, 5, 25, 10] that adopted 8 bit quantization, we found that an improved rate-distortion trade-off can be achieved by using 6 bits quantization if a quantization-aware training methodology is applied. In particular, we perform a short fine-tuning with Quant-Noise [46] after weight pruning, which can effectively reduce the quantization error. However, unlike the original implementation, we do not use STE [7] for computing the gradient of the quantized weights due to its inferior performance.
4.1 Video representation
To evaluate the effectiveness of the proposed model we benchmarked HiNeRV against five related works: NeRV [11], E-NeRV [29], PS-NeRV [5], FFNeRV [25] and HNeRV [10] on the Bunny [1]  $(1280\times 720$  with 132 frames) and the UVG datasets [36] (7 videos at  $1920\times 1080$  with a total of 3900 frames). For each video, we trained all networks at multiple scales, and kept their number of parameters similar at each scale. Three scales were set up to target the S/M/L scales in NeRV [11] for the UVG datasets, while two different scales XXS/XS together with scale S were configured for the Bunny dataset. We reported the encoding and decoding speeds in frames per second, measured with A100 GPU. The number of parameters corresponding to each scale are reported in Table 1 and 2.
For all models tested, we set the number of training epochs to 300 and batch size (in video frames) to 1. We used the same optimizer as in [11], and employed the same learning objectives for all NeRV-based methods as in the original literature [11, 29, 5, 25, 10]. For HiNeRV, we empirically found that it is marginally better to adopt a larger learning rate of  $2e - 3$  with global norm clipping which is commonly used for Transformer-based networks [50]. We used the learning rate of  $5e - 4$  which is a common choice for the other networks as in their original literature [11, 29, 5, 25]. We also adopted a combination of  $\ell 1$  loss and MS-SSIM loss (with a small window size of  $5\times 5$  rather than  $11\times 11$ ) for HiNeRV, as we observed that the MS-SSIM loss with a small window size leads to a better performance. For HiNeRV and PS-NeRV [5], we randomly sample patches instead of frames during training, but we scale the number of patches in each batch to keep the same effective batch size. It is noted that the original configuration of HNeRV [10] has an input/output size of  $1280\times 640 / 1920\times 960$  with strides  $(5,4,4,2,2) / (5,4,4,3,2)$ , and we pad the frames of HNeRV in order to fit the  $1280\times 720 / 1920\times 1080$  videos. This was found to work better than changing the strides in HNeRV. Detailed configuration of HiNeRV is summarized in the Supplementary Material.
It can be observed (Table 1 and 2) that our proposed HiNeRV outperforms all benchmarked models in terms of reconstruction quality at each scale on both Bunny [1] and UVG [36] datasets. We also note that HiNeRV performs better than all the other methods on all test sequences (with various spatial and temporal characteristics) in the UVG database, exhibiting better reconstruction quality in terms of PSNR. In particular, for some video sequences where existing INRs performed poorly [10], HiNeRV offers significant improvement ( $>7.9\mathrm{dB}$  over NeRV on ReadySetGo). While the encoding
Table 1: Video representation results on the Bunny dataset [1] (for XXS/XS/S scales).
Model	Size	MACs	Encoding FPS	Decoding FPS	PSNR
NeRV	0.83M/1.64M/3.20M	25G/57G/101G	131.9/93.6/81.8	308.5/229.1/202.3	26.82/29.61/32.56
E-NeRV	0.88M/1.65M/3.31M	26G/101G/104G	104.7/68.0/68.7	254.3/175.8/174.8	29.03/31.75/36.69
PS-NeRV	0.90M/1.68M/3.35M	29G/238G/240G	90.6/36.1/36.0	228.1/96.1/96.0	28.47/30.31/34.78
HNeRV	0.82M/1.66M/3.28M	23G/48G/94G	100.0/80.8/64.2	317.4/251.6/192.5	31.08/33.68/36.95
FFNeRV	0.91M/1.66M/3.19M	26G/58G/102G	62.1/51.5/47.9	108.4/95.2/90.5	30.37/33.83/37.01
HiNeRV	0.77M/1.59M/3.25M	23G/47G/96G	37.6/27.7/20.0	132.1/103.9/76.7	36.37/38.94/41.14
Table 2: Video representation results with the UVG dataset [36] (for S/M/L scales). Results are in PSNR. FPS is the encoding/decoding rate.
Model	Size	MACs	FPS	Beauty	Bosph.	Honey.	Jockey	Ready.	Shake.	Yacht.	Avg.
NeRV	3.31M	227G	32.4/90.0	32.83	32.20	38.15	30.30	23.62	33.24	26.43	30.97
E-NeRV	3.29M	230G	20.7/75.9	33.13	33.38	38.87	30.61	24.53	34.26	26.87	31.75
PS-NeRV	3.24M	538G	14.7/42.6	32.94	32.32	38.39	30.38	23.61	33.26	26.33	31.13
HNeRV	3.26M	175G	24.6/93.4	33.56	35.03	39.28	31.58	25.45	34.89	28.98	32.68
FFNeRV	3.40M	228G	19.0/49.3	33.57	35.03	38.95	31.57	25.92	34.41	28.99	32.63
HiNeRV	3.19M	181G	10.1/35.5	34.08	38.68	39.71	36.10	31.53	35.85	30.95	35.27
NeRV	6.53M	228G	32.0/90.1	33.67	34.83	39.00	33.34	26.03	34.39	28.23	32.78
E-NeRV	6.54M	245G	20.5/74.6	33.97	35.83	39.75	33.56	26.94	35.57	28.79	33.49
PS-NeRV	6.57M	564G	14.6/42.0	33.77	34.84	39.02	33.34	26.09	35.01	28.43	32.93
HNeRV	6.40M	349G	20.1/68.5	33.99	36.45	39.56	33.56	27.38	35.93	30.48	33.91
FFNeRV	6.44M	229G	18.9/49.3	33.98	36.63	39.58	33.58	27.39	35.91	30.51	33.94
HiNeRV	6.49M	368G	8.4/29.1	34.33	40.37	39.81	37.93	34.54	37.04	32.94	36.71
NeRV	13.01M	230G	31.7/89.8	34.15	36.96	39.55	35.80	28.68	35.90	30.39	34.49
E-NeRV	13.02M	285G	21.0/68.1	34.25	37.61	39.74	35.45	29.17	36.97	30.76	34.85
PS-NeRV	13.07M	608G	14.1/41.4	34.50	37.28	39.58	35.34	28.56	36.51	30.28	34.61
HNeRV	12.87M	701G	15.6/52.7	34.30	37.96	39.73	35.47	29.67	37.16	32.31	35.23
FFNeRV	12.66M	232G	18.4/49.3	34.28	38.48	39.74	36.72	30.75	37.08	32.36	35.63
HiNeRV	12.82M	718G	5.5/19.9	34.66	41.83	39.95	39.01	37.32	38.19	35.20	38.02
and decoding speeds of HiNeRV are slower than that of other methods, HiNeRV achieves a higher overall PSNR figure with fewer MACs. Further optimization may help reduce its latency. In the Supplementary Material, we conducted experiments with faster variants of HiNeRV, demonstrating that HiNeRV can achieve a satisfactory trade-off among latency, model size and reconstruction quality simultaneously.
Figure 1 (right) shows the performance of HiNeRV, NeRV and HNeRV (at scale S) in terms of the reconstruction quality with various epochs of training. We observe that our model with 37 epochs achieves similar reconstruction quality of HNeRV with 300 epochs on both datasets.
4.2 Video compression
To evaluate video compression performance, we compared HiNeRV with two INR-based models: NeRV [11] and HNeRV [10]; with two conventional CODECs: HEVC/H.265 HM 18.0 (Main Profile with Random Access) [39, 40] and x265 (veryslow preset with B frames) [2]; and with two state-of-the-art learning-based CODEcs: DCVC [26], DCVC-HEM [27] and VCT [35]. These were all compared using two test databases: UVG [36] and MCL-JCV [51]. Unlike previous work [11, 5], which concatenates the videos and uses a single network to compress multiple videos, we train all the models for each video separately. For the training of NeRV, HNeRV and HiNeRV, we use the configurations described in Section 4.1, but with two more scales, namely XL and XXL, for encoding videos with the highest rate (See the Supplementary Material). We apply pruning and quantization as described in Section 3.6. In particular, we prune these three models to remove  $15\%$  of their weights and fine-tune the models for another 60 epochs. These models are further optimized with Quant-Noise [46] with  $90\%$  noise ratio for 30 epochs. Here we use the same learning rate scheduling for fine-tuning, but employ  $10\%$  of the original learning rate in the optimization with Quant-Noise. To obtain the actual rate, we perform arithmetic entropy coding [34] and combine all essential information including the pruning masks and the quantization parameters into bitstreams.
Figure 3 reports the overall rate quality performance on the UVG [36] and the MCL-JCV [51] datasets. Table 3 summarizes the average Bjøntegaard Delta (BD) rate results for both databases. All results show that HiNeRV offers competitive coding efficiency compared to most conventional codecs
Table 3: BD-Rate (Measured in PSNR/MS-SSIM) results on the UVG [36] and MCL-JCV [51] datasets.
Dataset	Metric	x265 (veryslow)	HM (RA)	DCVC	DCVC-HEM	VCT	NeRV	HNeRV
UVG	PSNR	-38.66%	7.54%	-43.44%	25.23%	-34.28%	-74.12%	-72.29%
MS-SSIM	-62.70%	-41.41%	-34.50%	49.03%	-23.69%	-73.76%	-83.86%
MCL-JCV	PSNR	-23.39%	31.09%	-24.59%	35.83%	-17.03%	-80.19%	-66.56%
MS-SSIM	-44.12%	-2.65%	-17.32%	80.73%	12.10%	-82.28%	-79.42%
Figure 3: Video compression results on the UVG [36] and the MCL-JCV datasets [51].
and learning-based methods. This represents a significant improvement over existing NeRV-based approaches (this is also confirmed by the visual comparison with HNeRV in Figure 1). In particular, it is observed that HiNeRV outperforms x265 (veryslow) [2], DCVC [26] and VCT [35] based on PSNR. As far as we are aware, this is the first INR-based codec which can achieve such performance. We also observe that HiNeRV offers better performance compared to H.265 HM (Random Access) based on MS-SSIM. It should be noted that the results of each learning-based codec reported here are based on two model checkpoints, one for optimizing PSNR and the other for MS-SSIM, while all the results for HiNeRV are however based on the same checkpoint.
Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization are not optimized in the loop), it nonetheless outperforms many state-of-the-art end-to-end optimized learning-based approaches. This demonstrates the significant potential of utilizing INRs for video compression applications. In the Supplementary Material, comparison with additional baseline is provided.
To verify the contribution of various components in HiNeRV, we generated a number of variants of the original model, and evaluated them for video representation on the UVG dataset [36] (all the sequences were down-sampled to  $1280 \times 720$  for reduction the amount of computation). For all experiments, we followed the settings in Section 4.1, and performed training targeting scale S (by adjusting the width of the network to keep the similar model sizes). All results are shown in Table 4. We also included the results of NeRV [11] and HNeRV [10] for reference.
Bilinear interpolation with hierarchical encoding. The contribution of bilinear interpolation with hierarchical encoding was verified by comparing it with alternative upsampling layers, sub-pixel convolutional layer [42] with  $1 \times 1$  (V1) and  $3 \times 3$  (V2) kernel sizes.
Table 4: Ablation studies of HiNeRV with the UVG dataset [36]. Results are in PSNR.
Model	Size	Beauty	Bosph.	Honey.	Jockey	Ready.	Shake.	Yacht.	Avg.
NeRV	3.20M	34.03	32.77	39.59	30.39	23.88	33.85	26.88	31.63
HNeRV	3.22M	35.04	35.72	41.11	32.20	25.88	35.75	29.69	33.63
HiNeRV	3.17M	35.67	39.37	41.61	36.94	31.98	36.74	31.57	36.27
(V1) w/ Sub-Conv1x1	3.16M	35.28	36.63	41.58	34.64	29.12	36.31	29.91	34.78
(V2) w/ Sub-Conv3x3	3.15M	34.96	35.35	41.14	32.80	27.18	35.34	29.14	33.70
(V3) w/o Encoding	3.17M	35.64	39.18	41.58	36.16	30.92	36.68	31.50	35.95
(V4) w/ Fourier enc.	3.17M	35.62	39.07	41.59	36.00	30.91	36.81	31.47	35.92
(V5) w/ Fourier (local) enc.	3.17M	35.59	38.99	41.54	35.77	30.61	36.57	31.30	35.77
(V6) w/ Grid (local) enc.	3.19M	35.65	39.26	41.58	36.17	30.93	36.72	31.55	35.98
(V7) w/ MLP	3.19M	35.10	37.17	41.35	34.77	29.10	35.58	29.76	34.69
(V8) w/ Conv3x3	3.17M	35.35	37.86	41.37	35.13	29.70	36.10	30.31	35.12
(V9) w/ Frame-wise	3.17M	35.68	39.22	41.54	36.69	31.49	36.54	31.54	36.10
(V10) w/ Patch-wise	3.17M	35.46	38.30	41.55	35.04	30.06	36.51	30.77	35.38
(V11) w/ Nearest Neighbor	3.17M	35.60	39.12	41.64	36.52	31.51	36.82	31.33	36.08
Upsampling encodings. Four variants are trained to confirm the effectiveness of the upsampling encodings including (V3) w/o encoding; (V4) with the Fourier encoding [37]; (V5) using Fourier encoding with local coordinates (computed in the hierarchical encoding); (V6) using the grid-based encoding with local coordinates, i.e. the hierarchical encoding without the temporal dimension.
ConvNeXt block. The employed ConvNeXt block [31] has been compared with (V7) the MLP block in transformer [50]; (V8) the block containing two convolutional layers with, where we use  $3 \times 3$  and  $1 \times 1$  kernel size to keep the receptive field consistent with the ConvNeXt block used in our paper.
Unified representations. V9 and V10 have been generated for frame- and patch-wise configurations.
Interpolation methods. V11 replaces the bilinear interpolation by the nearest neighbor interpolation.
The ablation study results are presented in Table. 4 which shows that the full HiNeRV model outperforms all alternatives (V1-V11) on the UVG dataset in terms of the overall reconstruction quality. This confirms the contribution of each primary component of the design. More discussion regarding the results can be found in the Supplementary Material.
In this paper, a new neural representation model, HiNeRV, has been proposed for video compression, which exhibits superior coding performance over many conventional and learning-based video CODECs (including those based on INRs). The improvements demonstrated are associated with new innovations including bilinear interpolation based hierarchical encoding, a unified representation and a refined model compression pipeline.
Despite the fact that HiNeRV has not been fully optimized end-to-end (entropy encoding and quantization are not optimized in the loop), it nonetheless achieves comparable performance to state-of-the-art end-to-end optimized learning-based approaches, with significant improvement over existing NeRV-based algorithms. This demonstrates the great potential of utilizing INRs for video compression applications. For example, this is the first INR-based video codec which can outperform HEVC HM Random Access mode based on MS-SSIM.
Future work should focus on incorporation of entropy coding and quantization to achieve full end-to-end optimization.
This work was jointly funded by UK EPSRC (iCASE Awards), BT, the UKRI MyWorld Strength in Places Programme and the University of Bristol. We also thank the support by the Advanced Computing Research Centre, University of Bristol, for providing the computational facilities.
[45] V. Sitzmann, J. N. P. Martel, A. W. Bergman, D. B. Lindell, and G. Wetzstein. Implicit Neural Representations with Periodic Activation Functions. In NeurIPS, 2020.
[46] P. Stock, A. Fan, B. Graham, E. Grave, R. Gribonval, H. Jégou, and A. Joulin. Training with Quantization Noise for Extreme Model Compression. In ICLR. OpenReview.net, 2021.
[47] G. J. Sullivan, J. Ohm, W. Han, and T. Wiegand. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Trans. Circuits Syst. Video Technol., 22(12):1649-1668, 2012.
[48] C. Sun, M. Sun, and H. Chen. Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction. In CVPR, pages 5449-5459. IEEE, 2022.
[49] T. van Rozendaal, J. Brehmer, Y. Zhang, R. Pourreza, A. Wiggers, and T. S. Cohen. Instance-Adaptive Video Compression: Improving Neural Codes by Training on the Test Set, 2023.
[50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, pages 5998-6008, 2017.
[51] H. Wang, W. Gan, S. Hu, J. Y. Lin, L. Jin, L. Song, P. Wang, I. Katsavounidis, A. Aaron, and C. J. Kuo. MCL-JCV: A JND-based H.264/AVC video quality assessment dataset. In ICIP, pages 1509-1513. IEEE, 2016.
[52] T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra. Overview of the H.264/AVC Video Coding Standard. IEEE Trans. Circuits Syst. Video Technol., 13(7):560-576, 2003.
[53] J. Xiang, K. Tian, and J. Zhang. MIMT: Masked Image Modeling Transformer for Video Compression. In ICLR. OpenReview.net, 2023.
[54] R. Yang, F. Mentzer, L. V. Gool, and R. Timofte. Learning for Video Compression With Hierarchical Quality and Recurrent Enhancement. In CVPR, pages 6627-6636. Computer Vision Foundation / IEEE, 2020.
[55] Y. Zhu, Y. Yang, and T. Cohen. Transformer-based Transform Coding. In ICLR. OpenReview.net, 2022.