# Cloud Gaming Video Coding Optimization Based on Camera Motion-Guided Reference Frame Enhancement
## 3 The Proposed Coding Optimization Algorithm and Network Architecture
### Network Architecture  
We assume that the reference frames in the reference frame list are denoted as \(I_{R_{1}}\), \(I_{R_{2}}\), \(I_{R_{3}}\), and \(I_{R_{4}}\), which are arranged in order of the POC (from the smallest to the largest). Considering that the reconstructed frame of the previous frame and those with the highest quality in each previous GOP are generally selected as reference frames under the LDP configuration, \(I_{R_{4}}\) has the highest temporal correlation and similarity with the frame to be encoded while \(I_{R_{1}}\), \(I_{R_{2}}\), and \(I_{R_{3}}\) have a higher objective quality. We take the high-quality reference frames \(I_{R_{1}}\), \(I_{R_{2}}\), and \(I_{R_{3}}\) to enhance the quality of \(I_{R_{4}}\), using the temporal correlation between them to generate a new reference frame denoted as \(I_{R_{4}}^{E}\), with both high-quality and high-temporal correlation. Thus, the proposed model can be expressed as:  
\[I_{R_{4}}^{E}=f_{\theta}\Big{(}\{I_{R_{i}}\}_{1}^{4}\big{\{}V_{R_{i}R_{4}}\}_{ 1}^{4}\Big{)} \tag{1}\]  
where \(V_{R_{i}R_{4}},i\in\{1,2,3,4\}\) represents the camera motion indicating the transformation from the position of \(I_{R_{i}}\) to \(I_{R_{4}}\). \(I_{R_{4}}^{E}\) is the enhanced target frame (i.e., the output of the CMGNet). \(I_{R_{i}}\) and \(V_{R_{i}R_{4}}\) are the inputs of the proposed CMGNet and \(\theta\) represents the set of the learnable model parameters. The structure of the proposed CMGNet is shown in Figure 3, which consists of a camera motion-guided prediction module, an alignment and fusion module, and a reconstruction module.  
As shown in Figure 3, \(I_{R_{i}}\), \(I_{R_{4}}\), and \(V_{R_{i}R_{4}}\) are first fed into the camera motion-guided prediction (i.e., CMGP in Figure 2) block to predict the pixel offset \(p_{R_{i}}\) for the alignment of the input frame to the target frame, and the weight mask \(m_{R_{i}}\) for the frame fusion. Then, four pairs of offsets and masks, i.e., \(\big{(}p_{R_{i}},m_{R_{i}}\big{)}\), together with \(I_{R_{i}}\), for each \(i\in\{1,2,3,4\}\), are fed into the alignment and fusion module to obtain the fused feature. We use the deformable convolution (DCNv2) [48] to perform frame alignment and fusion, which has been proved effective. Finally, the fused feature is input to the reconstruction module to fully explore the information contained in the feature map, therefore improving the quality of the reference frame. In our implementation, we select a simple yet effective eight-layer convolution with residual learning [49] as the reconstruction module due to its versatility and efficiency (eight layers for the luma component and four layers for the chroma component). Thus, the model can be re-formulated as:  
\[I_{R_{4}}^{E}=f_{\text{Rect}}\Big{(}f_{DCN}\Big{(}f_{CMGP}\Big{(}\big{\{}I_{R _{i}},V_{R_{i}R_{4}}\big{\}}_{1}^{4},I_{R_{4}}\Big{)},\big{\{}I_{R_{i}}\big{\}} _{1}^{4}\Big{)},I_{R_{4}}\Big{)} \tag{2}\]  
Figure 2: Overview of the proposed optimization compression pipeline.  
where \(f_{CMGP}(\cdot)\), \(f_{DCN}(\cdot)\), and \(f_{Rect}(\cdot)\) represent the CMGP module, the alignment and fusion module, and the reconstruction module, respectively. The outputs of \(f_{CMGP}(\cdot)\) are offset \(p_{R_{i}}\) and mask \(m_{R_{i}}\) required by DCNv2. We will describe the CMGP in detail below.