# Video Multi-Scale-Based End-to-End Rate Control in Deep Contextual Video Compression
## 3 The Proposed Algorithm
### End-to-End Rate Control  
To make full use of the temporal correlation, a two-branch residual-based network for bit rate ratio prediction is designed, and it is shown in Figure 3.  
In Figure 3, the residual block is designed to extract high-level semantic features. \(R_{F}(n-1)\), \(D_{F}(n-1)\), and \(\lambda_{F}(n-1)\) represent the bit rate, distortion, and Lagrangian multiplier of the previous encoded frame, respectively. \(R_{G}\) represents the target bit rate of the current GoP. \(W\) is the output of the network, which represents the predicted bit rate ratio of every frame. The up branch is used to extract the deep features correlated with the content of the original frame. The down branch will build a learning vector of the coding parameters from the previous encoded frame. Since the outputs of the two branches have a strong temporal correlation, a multiplication operation is used to fuse the output features. Then, fusion features are further extracted via a residual block and finally converted to bit rate ratio \(W\).  
From the GoP bit allocation, \(R_{G}\) can be expressed as  
\[R_{G}=\frac{R_{\text{target}}\cdot(n_{\text{encoded}}+N_{SW})-R_{\text{encoded}}}{N_{SW}}\cdot N_{G} \tag{13}\]  
Figure 3: Two-branch residual-based network.  
Figure 2: Frame super-resolution-based end-to-end rate control.  
where \(R_{\text{target}}\) and \(R_{\text{encoded}}\) are the target bit rate and total used bit rate, respectively. \(N_{G}\) is the number of frames in the GoP. \(N_{SW}\) is the smooth window, which is set to 40. Then, the bit allocation of a frame \(n\) can be expressed as  
\[R_{F}(n)=\frac{R_{G}-R_{\text{encoded}-\text{G}}}{\sum\limits_{i=n}^{N_{G}}W_{ i}}\cdot W_{n} \tag{14}\]  
where \(R_{\text{encoded}-\text{G}}\) is the used bit rate of the frames in the GoP. \(W_{n}\) is the bit rate ratio of the frame \(n\), which can be predicted from the two-branch residual-based network in Figure 3. The loss function is defined as  
\[Loss_{BTR}=\frac{1}{N}\cdot\sum\limits_{i=1}^{N}\left(W_{i}-\hat{W}_{i}\right) ^{2} \tag{15}\]  
where \(\hat{W}_{i}\) is the actual bit rate ratio. \(N\) is the number of frames for training.  
To predict the Lagrange multiplier \(\lambda\) of the current frame, a two-branch regression network is designed in Figure 4. Since the residual feature of a frame, which is the difference between the predicted and original frames, can indicate the correlation between the adjacent frames, the residual frame is used as an input of the up branch. The bit allocation of the current frame \(R_{F}(n)\) calculated via Equation (14) and the bit cost \(R_{F}(n-1)\) and distortion \(D_{F}(n-1)\) and \(\lambda_{F}(n-1)\) of the previously encoded frame are used as inputs of the down branch. Then, the fusion features of the two branches are inputted into the regression block. Finally, the optimal \(\lambda\) value of the current frame can be predicted using the network.  
Different from the two-branch residual-based network, the two-branch regression-based network for \(\lambda\) is trained using a multitasking loss function, which is defined as  
\[Loss_{\lambda}=\gamma\Bigg{(}\frac{\left|R_{F}-R_{F}(\lambda)\right|}{R_{F}} \Bigg{)}^{2}+(1-\gamma)\hat{D}_{F}(\lambda) \tag{16}\]  
where \(\gamma\) is set as 0.4 empirically, and \(R_{F}\) is the calculated bit in Equation (14). \(\hat{R}_{F}(\lambda)\) and \(\hat{D}_{F}(\lambda)\) are the actual bit and distortion, respectively.