SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ho Man Kwan†, Ge Gao†, Fan Zhang†, Andrew Gower‡, David Bull†
† Visual Information Lab, University of Bristol, UK
$\ddagger$  Immersive Content & Comms Research, BT, UK
{hm.kwan, ge1.gao, fan.zhang, dave.bull}@bristol.ac.uk,
andrew.p.gower@bt.com
Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3% overall bit rate saving over HNeRV and 43.4% over DCVC on the UVG dataset, measured in PSNR).<sup>1</sup>
Implicit neural representations (INRs) have become popular due to their ability to represent and encode various scenes [37], images [45] and videos [45, 11]. INRs typically learn a coordinate to value mapping (e.g. mapping a pixel or voxel index to its color and/or occupancy) to support implicit reconstruction of the original signal. While these representations are usually instantiated as multilayer perceptrons (MLPs), existing MLP-based network can only represent video content with a low reconstruction quality and speed [11]. To address this limitation, recent works have employed Convolutional Neural Networks (CNNs) to perform a frame index to video frame mapping [11, 29, 5, 25, 10]. These CNN-based INRs are capable of reconstructing video content with higher quality and with a faster decoding speed, when compared to MLP-based approaches [45]. When using INRs for encoding videos, video compression can then be achieved by performing model compression for the individual input video. However, existing INR-based algorithms remain significantly inferior to state-of-the-art standard-based [52, 47, 8] and learning-based CODECs [26, 41, 27, 28, 35]. For example, none of these INR-based CODECs can compete with HEVC x265 [2] (veryslow preset).
1Project page: https://hmkx.github.io/hinerv/
HNeRV 31.4dB PSNR@0.101bpp
HiNeRV (ours) 36.6dB PSNR@0.051bpp
Figure 1: (Left) Visual comparison between HNeRV [10] and HiNeRV (ours) for compressed content (cropped). HiNeRV offers improved visual quality with approximately half the bit rate compared to HNeRV (PSNR and bitrate values are for the whole sequence). (Right) Video regression with different epochs for a representation task. HiNeRV (ours) with only 37 epochs achieves similar reconstruction quality to HNeRV [10] with 300 epochs.
Most INR-based models for videos [11, 29, 5, 25, 10] employ conventional convolutional layers or sub-pixel convolutional layers [42], which are less parameter efficient, and hence limit representation capacity within a given storage budget. In addition, most existing work employs Fourier-based positional encoding [37]; this has a long training time and can only achieve sub-optimal reconstruction quality [11, 29, 5]. In video compression, the training of INR models is equivalent to the encoding process, implying that most INR-based CODECs require a long encoding runtime to obtain a satisfactory rate-quality performance [11]. However, some recent non-video INR models have utilized feature grids or a combination of grids and MLPs as the representation to speed up the convergence of INRs; this has improved the encoding speed by several orders of magnitude [15, 48, 38, 9].
In this paper, we propose a new INR model based on Hierarchically-encoded Neural Representation for video compression, HiNeRV. We replace commonly used sub-pixel conventional layers [42] in existing INRs for upsampling [11, 29, 5, 25, 10] by a new upsampling layer which embodies bilinear interpolation with hierarchical encoding that is sampled from multi-resolution local feature grids. These local grids offer increased parameter efficiency, as the number of parameters increases with the upsampling factor instead of the resolution. Moreover, the network is primarily based on MLPs and depth-wise convolutional layers (rather than dense convolutional layers). This enhances the representation capability and maximizes the performance for a given parameter count. This architectural design allows us to build a much deeper and wider network which offers a significantly better video encoding performance when compared to state-of-the-art INR-based coding approaches.
Furthermore, instead of learning a frame- [11] or patch-wise [5] representation, we show that by simply training with overlapped patches, HiNeRV can be seamlessly switched between both representation types, achieving a unified representation with improved performance compared to both frame-based and patch-based settings. This also provides flexibility for hardware implementation, where encoding and decoding processes can be performed either using frames to optimize the computational complexity, or as patches to minimize the memory footprint.
To achieve competitive coding performance, we also refine the model compression pipeline in [11], where pruning and fine-tuning are followed by model training, before quantization is applied. First, we used an adaptive pruning technique to reduce the negative impact of model pruning. Secondly, quantization-aware training is applied for fine-tuning the model performance before quantization. This enables lower bit depth quantization which achieves an improve rate-distortion trade-off.
The proposed method has been tested against existing INR-based video coding methods and state-of-the-art conventional and learning-based video CODECs on the UVG [36] and MCL-JCV [51] datasets. Notwithstanding the fact that HiNeRV has not been end-to-end optimized for video compression (i.e. pruning, quantization and entropy coding are not included in the training loop), it still significantly outperforms all INR-based methods (e.g.,  $72.3\%$  overall BD-rate gain over HNeRV on the UVG database, measured in PSNR), and is also competitive compared with existing conventional and end-to-end learning-based video codec (e.g.,  $43.4\%$  over DCVC,  $38.7\%$  over x265 veryslow).
The primary contributions of this work are summarized below:
1) We propose HiNeRV, a new INR employing hierarchical encoding based neural representation.
2) We employ a unified representation by adding padding, which trades a small computation overhead for additional flexibility and performance gain.
3) We build a video codec based on HiNeRV and refine the model compression pipeline to better preserve the reconstruction quality of INRs by using adaptive pruning and quantization-aware training.
4) The compression performance of the proposed method is superior to existing INR models, and is comparable to many conventional/learning-based video coding algorithms. As far as we are aware, it is the first INR-based codec to significantly outperform HEVC (x265 veryslow) [39].
2.1 Video compression
Video compression has long been a fundamental task in the field of computer vision and multimedia processing. As alternatives to the already popularized conventional CODECs such as H.264/AVC [52], H.265/HEVC [47], and H.266/VVC [8], there has been a rapid increase in the adoption of deep learning techniques for video compression in recent years. This has typically involved replacing certain modules (e.g., motion compensation [3, 54], transform coding [55, 16] and entropy coding [6]) in the conventional pipeline with powerful learning-based models [53, 35].
In contrast, there has also been significant activity focused on the development of new coding architectures which allow end-to-end optimization. Lu et al. [32] proposed DVC that was further extended to enable major operations in both the pixel and the feature spaces [21, 20]. An alternative approach has focused on conditional [26, 30] instead of predictive coding to reduce the overall bitrate by estimating the probability model over several video frames. Furthermore, the characteristics of the differentiable frameworks have been exploited by [19, 49, 44], where both encoder and decoder (typically signaled by a model stream containing updated parameters) are overfitted to the video data during the evaluation to further enhance compression efficiency.
While effective, with some recent work [53, 43, 28] claiming to outperform the latest compression standards, these methods still follow the pipeline of conventional codecs, which may constrain the development of neural video compression methods. Moreover, learning-based video compression methods tend to be much more computational complex and often yield much slower decoding speed than conventional codecs. This often renders them impractical for real-time applications, especially considering the prevalence of high-quality and high-resolution videos consumed nowadays.
2.2 Implicit neural representation
Implicit neural representations (INRs) are being increasingly used to represent complicated natural signals such as images [45, 12, 14, 23], videos [45, 11], and vector-valued, volumetric content [37]. This type of approach benefits from incorporating positional encoding - a technique that embeds the positional input into a higher-dimensional feature space. Periodic functions [45, 33, 37] have first been utilized to improve the network's capability for learning high frequency information, and grid features [48, 38, 9] have then been applied to address their slow convergence speed and further improve the reconstruction quality.
More recently, Neural Representations for Videos (NeRV) [11] has re-formulated the INR for video signals to be frame-wise, achieving competitive reconstruction performance with very high decoding speed. NeRV approaches have inspired a trend of utilizing CNNs to encode the RGB values of videos with 1D frame coordinates [11, 29, 25, 10] or with 3D patch coordinates [5], and have demonstrated promise in various video tasks, including denoising [11], frame interpolation [11, 29, 25, 10], inpainting [5, 10], super-resolution [13] and video compression [11, 29, 5, 25, 10].
When INRs are applied for image and video compression, they typically convert the signal compression task into a model compression problem by incorporating weight pruning, quantization and entropy coding [17]. NeRV [11] and related works [29, 5, 25, 10] adopt the above approach for video compression. Although these have demonstrated fast decoding capability, they do not yet achieve a rate-distortion performance comparable to either conventional or learning-based CODECs.
Following the approach adopted in previous work [45, 11], we consider a video regression task where a neural network encodes a video  $V$  by mapping coordinates to either individual frames, patches or
pixels, where  $V \in \mathbb{R}^{T \times H \times W \times C}$ ,  $T, H, W$  and  $C$  are the number of frames in  $V$ , the height, the width and the number of channels of the video frames, respectively.
Fig. 2 (top) illustrates the high level structure of the proposed model, HiNeRV, which contains a base encoding layer, a stem layer,  $N$  HiNeRV blocks, and a head layer. In HiNeRV, each RGB video frame is spatially segmented into patches of size  $M \times M$ , where each patch is reconstructed by one forward pass. The model first takes a patch coordinate  $(i,j,t)$  to compute the base feature maps  $X_0$  with size  $M_0 \times M_0 \times C_0$ . Here we always refer the coordinates to the integer index, such that  $0 \leq t < T$ ,  $0 \leq j < \frac{H}{M}$  and  $0 \leq i < \frac{W}{M}$ . The following  $N$  HiNeRV blocks then upsample and process the feature maps progressively, where the  $n$ -th block produces the intermediate feature maps  $X_n$  that have the size  $M_n \times M_n \times C_n$  ( $M_N = M$ ). Finally, a head layer is used to project the feature maps to the output,  $Y$ , with the target size  $M \times M \times C$ .
3.1 Base encoding and stem
HiNeRV first maps the input patch coordinates into the base feature maps,  $X_0$ , by
$$
X _ {0} = F _ {\text {s t e m}} \left(\gamma_ {\text {b a s e}} (i, j, t)\right). \tag {1}
$$
To compute the base feature maps, we first calculate the pixel coordinates (related to the corresponding video frame) of the patch. For a patch with size  $M_0 \times M_0$ , the frame-based pixel coordinates  $(u_{frame}, v_{frame})$  can be computed by the patch-based pixel coordinates  $(u_{patch}, v_{patch})$  for  $0 \leq u_{patch}, v_{patch} < M_0$ , such that  $u_{frame} = i \times M_0 + u_{patch}$  and  $v_{frame} = j \times M_0 + v_{patch}$ . Then, by using the frame-based pixel coordinates, the positional encoding  $\gamma_{base}(i,j,t)$  can be interpolated from the learned feature grids [25]. After that, we employ a stem convolutional layer  $F_{stem}$  for projecting the feature maps to a desired number of channels  $C_0$ .
It is noted that most existing INRs for video [11, 29, 5, 10] utilize the Fourier style encoding, i.e. they apply  $\sin$  and  $\cos$  functions to map coordinates into the positional encoding. However, such encoding contains only positional information, which requires additional layers (e.g. MLPs) to transform it into informative features. In contrast, we adopt grid-based encoding [25] as they that contain richer information than the Fourier encoding. Specifically, we use the multi-resolution temporal grids that were introduced in FFNeRV [25], where the various feature grids have different temporal resolutions. In FFNeRV, linear interpolation over the temporal dimension is used to obtain a slice that is used as the input feature map. In our case, we utilize both of the frame index and the frame-based coordinates, i.e.,  $(u_{frame}, v_{frame}, t)$ , for interpolating the feature patches.
Although both high temporal resolution and a large number of channels are desirable for enhancing the expressiveness of the feature grids, this can result in greater model sizes and hence higher bitrates when the model is used for compression tasks. To maintain a compact multi-resolution grid, we increase the number of channels when reducing the temporal resolution at each grid level, i.e. the size of a grid is  $\left\lfloor \frac{T_{grid}}{2^l}\right\rfloor \times H_{grid} \times W_{grid} \times (C_{grid} \times 2^l)$ , for  $0 \leq l < L_{grid}$ . Here  $L_{grid}$  is the number of grids and  $T_{grid} \times H_{grid} \times W_{grid} \times C_{grid}$  is the size of the first level grid.
3.2 HiNeRV block
The obtained base feature maps are then processed by  $N$  HiNeRV blocks, which progressively upsample and process the feature maps. Specifically, the  $n$ -th HiNeRV block, where  $0 < n \leq N$ , upsamples the input feature maps  $X_{n-1}$  with size  $M_{n-1} \times M_{n-1} \times C_{n-1}$  through bilinear interpolation  $U_n$  with a scaling factor  $S_n$ , such that  $M_n = M_{n-1} \times S_n$ . We use bilinear interpolation, mainly due to its low computational cost and its capability to compute smooth upsampled maps. The HiNeRV block then computes a hierarchical encoding  $\gamma_n(i,j,t)$ , matches its number of channels by a linear layer  $F_{enc}$  and adds it to the upsampled feature maps, where  $\gamma_n(i,j,t)$  matches the upsampled feature map size (see Section 3.4). Finally, it applies a set of network layers  $F_n$  to enhance the representation to obtain the output  $X_n$ , where we specify the number of layers by  $D_n$ . For all the HiNeRV blocks except the first one ( $1 < n \leq N$ ), the first layer in  $F_n$  also reduces the number of channels by a factor,  $R$ , such that  $C_n = \left\lfloor \frac{C_0}{R^{n-1}} \right\rfloor$ , to save the computational cost for processing high spatial resolution maps. The output of the  $n$ -th HiNeRV block,  $X_n$ , has a size of  $M_n \times M_n \times C_n$ , and can be written as
$$
X _ {n} = F _ {n} \left(U _ {n} \left(X _ {n - 1}\right) + F _ {\text {e n c}} \left(\gamma_ {n} (i, j, t)\right)\right), 0 <   n \leq N \tag {2}
$$
Figure 2: Top: The HiNeRV architecture. Bottom left: The HiNeRV block. HiNeRV block take feature maps  $X_{n-1}$  and patch index  $(i,j,t)$  as input, upsample the feature maps, enhances it with the hierarchical encoding, then computes the transformed maps  $X_{n}$ . Bottom right: The local grid. In HiNeRV, the hierarchical encoding is computed by performing interpolation from the local grid, where the modulo of the coordinates is being used.
Figure 2 (bottom-left) shows the structure of HiNeRV block. Due to its observed superior performance, we employ ConvNeXt [31] as the network block in  $F_{n}$ , a combination of the MLP layer with depthwise convolution. We also apply Layer Normalization [4] before the interpolation and the MLP layers, and we only use shortcut connections when the input and output dimensions are matched.
3.3 Head layer
The final output patch  $Y$  is computed by applying a linear layer with sigmoid activation, denoted by  $F_{head}$ , on the output of the  $N$ -th HiNeRV block,
$$
Y = F _ {\text {h e a d}} \left(X _ {N}\right) \tag {3}
$$
3.4 Upsampling with the hierarchical encoding
Existing NeRV-based approaches [11, 29, 5, 25, 10] use the sub-pixel convolutional layer [42] for feature map up-scaling. However, this has a high parameter complexity:  $K^2 \times S^2 \times C_1 \times C_2$ , where  $K$ ,  $S$ ,  $C_1$  and  $C_2$  represent the kernel size, the upsampling factor, the number of input and output channels, respectively. While neighboring features are highly correlated, the convolutional layer does not take this into account and learns a dense weight matrix to perform upsampling. This is inefficient especially when the model size is the concern for tasks like compression [11], because the low parameter efficiency limits the maximum depth and width of the networks, and thus the capacity.
Previous work [11] has shown that bilinear interpolation does not perform as well as convolutional layers. However, we observed that it is actually a better choice when the parameter count is fixed. By performing interpolation, we can utilize the saved parameter budget to build a network with higher capacity. While we can generate high-resolution maps using parameter-free bilinear interpolation, the resulting maps are smoothed, and subsequent neural network layers may struggle to produce high-frequency output from them. One way to model high-frequency signals is to introduce positional encoding [37] during upsampling. As mentioned in Section 3.1, grid-based encoding is preferred due to the good representation performance. However, adopting it to enhance high-resolution feature maps can be costly, as it requires high-dimensional grids. To address this limitation, we introduce a novel grid-based encoding approach called hierarchical encoding, which boosts the upsampling capability of bilinear interpolation without significantly increasing the storage cost.
Unlike normal grid-based encoding, which computes the encoding using global coordinates, hierarchical encoding utilizes local coordinates to encode relative positional information. Specifically, the local coordinate is the relative position of a pixel in the upsampled feature map to its nearest pixel in the original feature map. Because local coordinates have a much smaller range of values, the required feature grid is also much smaller. Moreover, when both the base encoding and hierarchical encoding are used, every position can be represented hierarchically, allowing us to efficiently encode positional information on high-resolution feature maps. During upsampling, the upsampled feature
maps are first produced through bilinear interpolation with an up-scaling factor  $S_{n}$ . Then, for all frame-based pixel coordinates  $(u_{frame}, v_{frame})$  in the upsampled feature maps, we compute the corresponding local pixel coordinates by  $u_{local} = u_{frame} \mod S_{n}$  and  $v_{local} = v_{frame} \mod S_{n}$ , and employ them to compute the encoding.
It is noted that the above encoding approach is similar to applying a convolutional layer over a constant feature map. To further enhance the capacity of this encoding, we model the feature grids as multi-temporal resolution grids [25], which can provide richer temporal information, similar to the base encoding. In the  $n$ -th HiNeRV block, there are  $L_{local}$  levels of grids and the  $l$ -level grid has a size of  $\left\lfloor \frac{T_{local}}{2^l} \right\rfloor \times S_n \times S_n \times \left( \left\lfloor \frac{C_{local}}{R^{n-1}} \right\rfloor \times 2^l \right)$ . The size of the local grids is scaled with the factor  $S_n$  and can be adjusted by the hyper-parameter  $T_{local}$  and  $C_{local}$ . The number of channels of the grids is also scaled in proportion to the width of the HiNeRV block, i.e. by the reduction factor  $R$ . To obtain the hierarchical encoding, we perform trilinear interpolation by utilizing the frame index  $t$  with the local coordinate, i.e.,  $(u_{local}, v_{local}, t)$ , to extract encodings from all levels, then concatenate the encodings and apply a linear layer  $F_{enc}$  to match the encoding channels to the feature maps. To distinguish the grids for interpolating the hierarchical encoding from the one for the base encoding, i.e. the temporal grids, we refer these grids as the temporal local grids, because the hierarchical encoding is interpolated from these grids by using the local pixel coordinates. In Section 4.3, we demonstrated that the hierarchical encoding contributes to the superior performance HiNeRV. The process of upsampling with local encoding is shown in Figure 2 (bottom right).
3.5 Unifying frame-wise and patch-wise representations
Recent publications on INR for video can be classified into frame-wise [11, 29, 25, 10] or patch-wise representations [5]. Actually, in many of these networks, the initial feature maps can be easily computed either frame-wise or patch-wise, as the positional encoding depends only on the corresponding pixel coordinates. However, these two types of representations are not switchable because of boundary effects. In HiNeRV, we adopt a simple technique to unify frame-wise and patch-wise representations. When configuring HiNeRV as a patch-wise representation, we perform computation in overlapped patches, where we refer to the overlapped part as paddings, and the amount of padding pixels depends on the network configuration (e.g. the kernel sizes and/or the number of bilinear interpolation/convolitional layers). Such overlapped patches have previously been used for tasks such as super-resolution [22], but have not been applied in NeRV-based methods. When performing encoding in patches without proper padding, missing values for operations such as convolution can result in discontinuities between patch boundaries. Moreover, networks trained in patches do not perform well when inferencing in frames due to boundary effects. In our implementation, we perform intermediate computation with padded patches and crop the non-overlapped parts as the output patches, while with the frame configuration, paddings are not required. By adding paddings, we ensure generation of the same output for both frame-wise and patch-wise configurations.