# Cloud Gaming Video Coding Optimization Based on Camera Motion-Guided Reference Frame Enhancement
## 3 The Proposed Coding Optimization Algorithm and Network Architecture
### Network Architecture
#### 3.2.1 Camera Motion-Guided Prediction (CMGP)  
It is well-known that effectively using the temporal correlation between frames for frame alignment and fusion is a key issue. The non-consecutive inputs with varying degrees of motion offset further increase the difficulty for frame alignment and fusion. Deformable convolution has shown advanced performance in frame alignment and fusion, so we use it in our network. In deformable convolution operations, obtaining accurate alignment offsets and fusion masks is crucial. Inspired by the excellent performance of flow-guided deformable alignment [47], we adopt the flow-based method to predict the basic offset in our paper. Meanwhile, benefiting from the rendering and coding mechanism of the cloud gaming system, the camera motions that accurately record displacements between frames can be directly obtained. To some extent, the camera motion between the input frame and the target frame reflects the corresponding offset between them, and input frames with different camera motions relative to the target frame could bring different degrees of reference and correlation. Therefore, we use this information to guide the offset and mask prediction process. Figure 4 gives the illustration of the proposed CMGP. One CMGP block is selected as an example in the following discussion for simplification, and the operation is identical for all blocks. Given a pair of reconstructed reference frames \(I_{R_{i}}\) and \(I_{R_{4}}\), we first use a pre-trained flow prediction network to calculate the optical flow \(F_{R_{i}\to R_{4}}\). The well-known pyramid structure SPyNet [50] obtains final optical flow by generating flows at different scales and gradually refining them. Its smaller model parameters with advantages in terms of accuracy and speed attract us to adopt it. Then the warped frame \(I_{R_{i}\to R_{4}}^{W}\) is obtained by warping \(I_{R_{i}}\) to \(I_{R_{4}}\) through the optical flow \(F_{R_{i}\to R_{4}}\). The above process can be expressed by the following formulas:  
\[F_{R_{i}\to R_{4}} = SPyNet\left(I_{R_{i}},I_{R_{4}}\right) \tag{3}\] \[I_{R_{i}\to R_{4}}^{W} = warp\left(I_{R_{i}},F_{R_{i}\to R_{4}}\right)\]  
Figure 3: The proposed camera motion-guided network CMGNet. We regard the optical flow as the basic offset. Then we use the warped frame \(I_{R_{i}\to R_{4}}^{W}\) and the target frame \(I_{R_{4}}\) to predict the residual offset \(\Delta p_{R_{i}}\) and fusion mask \(m_{R_{i}}\). As shown in Figure 4, we use an encoder-decoder architecture to predict them, where convolution and deconvolution layers with a stride of 2 are used for downsampling and upsampling, respectively. In the convolution layers with a stride of 1, we use zero padding to keep the resolution of features unchanged. There is also a camera motion-guided transformation layer inspired by [51] before the last downsampling layer. The camera motion information \(V_{R_{i}R_{4}}\) participates in the prediction process by guiding the production of the affine transformation parameters for the downsampling feature in this layer. Such projection then guides the feature adjustment based on the accurate prior information, leading to more precise residual offset and mask prediction. Since we focus on the rotation movement of cloud gaming videos in this paper, the camera motion \(V_{R_{i}R_{4}}\) can be represented by a quaternion in the shape of \(4\times 1\times 1\). Then in the camera motion-guided transformation layer, we use a fully connected layer and a repeat operation to make \(V_{R_{i}R_{4}}\) the same dimension as that of the input feature \(f_{in}\). The transformation parameters \(\gamma_{R_{i}\to R_{4}}\) and \(\beta_{R_{i}\to R_{4}}\) are produced by a mapping function:  
\[\big{(}\gamma_{R_{i}\to R_{4}},\beta_{R_{i}\to R_{4}}\big{)}=\Psi\big{(} \big{[}f_{in},V_{R_{i}R_{4}}\big{]}\big{)} \tag{4}\]  
where the camera motion and input feature are concatenated together for guidance.