# Neural Network-Based Reference Block Quality Enhancement for Motion Compensation Prediction
## 2 Related Work  
In inter prediction, the closer the reference CU is to the current CU, the less the residuals are that are required for coding, and, thus, the lower the bit rate. However, due to the discretization of digital video, the current CU and the reference CU cannot be completely aligned. For this reason, existing video coding standards usually adopt a set of fixed sub-pixel filtering coefficients to interpolate the reference CUs [4]. In H.266/High Efficiency Video Coding (HEVC) and H.266/VVC, discrete cosine transform-based half or quarter interpolation filters are adopted. Specifically, to improve the coding performance, H.266/VVC applies one-sixth sub-pixel precision interpolation filters and affine motion compensation. Although the fractional interpolation filter achieves good performance in ME and MC, it still cannot adapt to complex video content well. Therefore, many researchers have used DL to refine the sub-pixel blocks. In these methods, [5; 6; 7; 8; 9; 10; 11; 12; 13] are for HEVC/H.265 [14], while [15; 16; 17; 18; 19] are for H.266/VVC [20]. [PERSON] et al. [5; 6; 7] regarded the fractional interpolation filter as an image regression problem and trained neural networks for each fractional interpolation filter. In [5], the author replaced three half-pixel fractional filters with three super-resolution convolutional neural network (SRCNN) [21] models, which resulted in a 0.9% reduction in BD rate under the LDP configuration of H.265/HEVC. In [6], the author used the reference CU as the input, the original CU as the ground truth, and trained 15 networks, whose structures were similar to a variable-filter-size residue learning convolutional neural network (VRCNN) [22], to replace the fractional filters of half- and quarter-pixel fractional shifts. In this method, there are a total of 120 natural networks for uniprediction, biprediction, and four different quantization parameters (QPs). This method saves 3.9%, 2.7%, and 1.3% in BD rates on average for LDP, low-delay B (LDB), and random-access (RA) configurations of H.265/HEVC, respectively. [PERSON] et al. [7] also proposed a reversible training method that is based on the assumption that integer pixels and fractional pixels can be derived from each other. This method can save 4.7% and 3.6% in BD rates under the configurations of LDB and RA for H.265/HEVC, respectively. In [8], a group variational transformation neural network (GVTNN) was proposed to replace the fractional interpolation in H.265/HEVC, and it can achieve a 1.9% BD rate reduction under the LDP configuration. [PERSON] et al. [9] optimized a neural network based on [8] and analyzed the influence of a fuzzy kernel on the task. In [10], [PERSON] et al. deemed that fractional interpolation can obtain more information from the reconstructed area around the current CU. The authors then put the current CU, together with the left and top reconstructed areas, into a larger square area as the network input and the original CU as the ground truth label. This method reduces the BD rate by 1.8% in the LDP configuration of H.265/HEVC. Similarly, [PERSON] et al. [11] reshaped the left and the upper reconstructed regions of the current CU into images with the same size, and then input them into the neural network together. This method can achieve 4.6%, 3.0%, and 2.7% BD rate reductions on average under LDP, LDB, and RA configurations, respectively, compared to H.265/HEVC. [PERSON] et al. [12] trained separate networks for luma and chroma channels. The method in [13] is similar to a loop filter; it uses the optical flow network (PWC-Net) [23] to generate an optical flow graph from the video frames before the reference frames, then uses the optical flow graph and the reference frame in a neural network [24] for quality enhancement. T he reference CUs are then extracted from the quality-enhanced reference frame for MC. In the study of H.266/VVC, [PERSON] et al. [15; 16] designed a three-layer neural network that only contains convolutional layers.