# Neural Network-Based Reference Block Quality Enhancement for Motion Compensation Prediction
###### Abstract  
Inter prediction is a crucial part of hybrid video coding frameworks, and it is used to eliminate redundancy in adjacent frames and improve coding performance. During inter prediction, motion estimation is used to find the reference block that is most similar to the current block, and the following motion compensation is used to shift the reference block fractionally to obtain the prediction block. The closer the reference block is to the original block, the higher the coding efficiency is. To improve the quality of reference blocks, a quality enhancement network (RBNN) that is dedicated to reference blocks is proposed. The main body of the network consists of 10 residual modules, with two convolution layers for preprocessing and feature extraction. Each residual module consists of two convolutional layers, one ReLU activation, and a shortcut. The network uses the luma reference block as input before motion compensation, and the enhanced reference block is then filtered by the default fractional interpolation. Moreover, the proposed method can be used for both conventional motion compensation and affine motion compensation. Experimental results showed that RBNN could achieve a \(-1.35\%\) BD rate on average under the low-delay P (LDP) configuration compared with the latest H.266/VVC.  
H.266/VVC; inter prediction; motion compensation; deep learning; image enhancement +
Footnote †: journal: Medical  
generate a bit stream. The decoder finds the corresponding reference block by the MV and adds residuals to reconstruct the current block.  
In recent years, deep learning (DL) has made great achievements in many fields with its strong learning ability and robustness. Especially in the field of computer vision, deep convolutional neural networks have played an interesting role in solving image processing problems. Recently, Moving Picture Expert Group has been investigating deep learning-based coding tools to further improve coding efficiency. DL-based methods can be integrated into a specific video coding module, such as intra prediction, inter prediction, loop filter, and post-processing. Specifically, in inter prediction, DL can be used for prediction, quality enhancement, fractional interpolation, and reference frame generation.  
In view of the successful application of DL in image quality enhancement, we proposed a quality enhancement method for CU before MC, which was added to the H.266/VVC framework. Specifically, we designed a reference-block-enhanced neural network (RBENN) to enhance the quality of the reference block. Fractional interpolation and other subsequent operations were then used on the quality-enhanced reference block. Compared with the standard H.266/VVC framework, the proposed method can reduce the Bjentegaard delta rate (BD rate) [3] in the low-delay P (LDP) configuration by 1.35%. The contributions of this paper are as follows.  
* A convolutional neural network with residual structure was used to enhance the quality of reference blocks before MC.
* The proposed method not only enhanced the reference CU before conventional motion compensation, but it also enhanced the reference CU before affine motion compensation.
* Experimental results showed that the proposed method achieved a 1.35% BD rate saving compared with the H.266/VVC standard under LDP test conditions.  
The remainder of this paper is organized as follows. Section 2 provides a brief review of the related work, Section 3 describes the proposed methodology and network architecture, Section 4 shows experimental results, and the work is concluded in Section 5.