## 4 Joint Decoding-Complexity and Rate Control
### CTU-Level Rate and Decoding-Complexity Allocation  
Adopting a similar approach to the rate controller in the HM reference encoder [47; 48], the group of picture (GOP)-level and frame-level bit rate and decoding-complexity targets can be used to derive their CTU-level allocations. In this case, the target number of bits for the GOP can be expressed as  
\[R_{\mathrm{T}}^{\mathrm{GOP}}=\psi\left\{\frac{B_{l}-B_{a}(M_{l}-W)}{W}\right\}, \tag{7}\]  
where \(\psi\), \(B_{l}\), \(B_{a}\), \(M_{l}\), and \(W\) are the GOP size, bits remaining, average bits per frame, frames remaining, and window size, respectively [48]. (In this case, it is assumed that the bit rate and the number of frames to be encoded in the sequence is known ahead of the encoding process. e.g., an on-demand video streaming scenario.) The average bits per frame is given by \(B_{a}=B_{\mathrm{T}}/M\), where \(B_{\mathrm{T}}\) is the total number of bits assigned to the sequence and \(M\) is the total number of frames in the sequence. Similarly, the target decoding-complexity for the GOP becomes  
\[C_{\mathrm{T}}^{\mathrm{GOP}}=\psi\left\{\frac{C_{l}-C_{a}(M_{l}-W)}{W}\right\}, \tag{8}\]  
Figure 1: The decoding-complexity (cpp), rate (bpp) and distortion (MSE) behavior of inter-predicted (**top row**) and intra-predicted (**bottom row**) frames. where \(C_{l}\) and \(C_{a}\) are the total decoding-complexity budget left over and the average complexity per frame, which is given by \(C_{a}=C_{\rm T}/M\). Here, \(C_{\rm T}\) represents the total decoding-complexity assigned to the sequence. (The available decoding-complexity budget in this case is defined as the total number of CPU cycles that can be spent for the purpose of decoding a particular sequence with a known number of frames. This can be determined based on the remaining energy capacity of the battery powered decoding device, and is considered outside the scope of this work.) Throughout the remainder of the manuscript, \(W=40\) as in the default configuration used in the HM 16.0 [49] encoder implementation. Next, the bits and decoding-complexities allocated for the GOP in (7) and (8), respectively, must be distributed across the individual frames in the GOP. In this case, a similar approach is adopted for both quantities (i.e., bits and decoding-complexity) leading to a weighted distribution of the GOP-level allocation to each frame as shown in (9). For the \(j\)th frame in the GOP, the bit and decoding-complexity allocations are given by  
\[X_{\rm T}^{\rm Frame}(j)=X_{\rm T}^{\rm GOP}\times\frac{\omega_{X}^{\rm Frame }(j)}{\sum_{j=1}^{\Psi}\omega_{X}^{\rm Frame}(j)}, \tag{9}\]  
where \(X\in\{R,C\}\) and the weights are defined as  
\[\omega_{X}^{\rm Frame}(j)=\varrho(j)+\eta_{X}^{m}. \tag{10}\]  
Here, \(\varrho(j)\) is the default weighting factor defined in HM16.0 reference encoder implementation [48; 50] for the \(j\)th frame in the GOP. The weighting factor \(\eta_{X}^{m}\) is experimentally determined based on the ratio of bits and decoding-complexities consumed by intra-predicted and inter-predicted frames in a typical video sequence. In this case, the numbers of bits and the decoding-complexities utilized within intra-predicted and inter-predicted frames were averaged across 50 frames for the six test sequences (Section 3.2) to determine \(\eta_{X}^{m}\), which is defined as  
\[m=\text{Intra-frame}: \eta_{R}^{m}=50,\,\eta_{C}^{m}=80,\] \[m=\text{Inter-frame}: \eta_{R}^{m}=3,\,\eta_{C}^{m}=10.\]  
Finally, bit rates and decoding-complexity targets are allocated to the individual CTUs based on the MSE of the previous co-located CTU, as is often done in traditional rate control [51; 52].