# Video Multi-Scale-Based End-to-End Rate Control in Deep Contextual Video Compression
## 1 Introduction  
In recent years, in order to achieve high-quality, high-resolution video storage and transmission, the Video Coding Experts Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) has released a series of international video coding standards: the H.26x standards and MPEG-x. ITU-T formulates H.261, H.262, H.263, H.263+, and H.264, which mainly aim at real-time video communication. ISO/IEC formulates MPEG-I, MPEG-2, and MPEG-4, which mainly aim at video storage and video streaming applications. On the other hand, ITU-T and ISO/IEC established a joint working group to formulate MPEG-2/H.262, H.264/advanced video coding (AVC), H.265/high-efficiency video coding (HEVC), and the newest H.266/versatile video coding (VVC). Rate control is a critical part of every video coding standard, especially in bandwidth-limited tasks such as live broadcasts. Many excellent rate control algorithms are proposed for the different video coding standards: the Variable Bit Rate (VBR) model [1] for MPEG-1; Test Model 5 (TM5) [2] for MPEG-2, which consists of bit allocation, rate control, and modulation; scalable rate control (VM8) [3] for MPEG-4, which uses a second-order rate distortion (RD) model; Reference Model 8 (RMS) [4] for H.261; and Test Model Near-term 8 (TMN8) [5] for H.263 and JVT-F086 [6] for H.264/AVC, with all adopting a hybrid coding framework. To further improve rate control performance, Unified Rate Quantization (URQ) [7] was first developed in H.264/AVC, in which Quantization Parameters (QPs) areregarded as the critical factor in determining the bit rate. Then, the URQ model continues to be used in H.265/HEVC. However, [PERSON] et al. [8] suggest that the bit rate is determined by QP only when all other coding parameters except QP are not too flexible, which indicates that the URQ model cannot work for increasingly flexible coding schemes. Therefore, the Lagrange multiplier \(\lambda\), which is derived from the RD optimization to balance the bit rate and coding distortion, will be used as the most critical factor for controlling the bit rate. Then, the R-\(\lambda\) rate control model [9] was proposed. By considering the distortion effect in the temporal domain, [PERSON] et al. [10] provided a coding distortion propagation equation for calculating the optimal \(\lambda\) for every frame. Due to the excellent coding performance of the R-\(\lambda\) model, it is also adopted in H.266/VVC. In [11], an improved R-\(\lambda\)mode with a quality-dependent relationship is proposed to achieve better coding performance. As a convolutional neural network (CNN) [12] can obtain foreknowledge by extracting image textures, many rate control algorithms use CNN to obtain prior information for coding, such as predicting optimal \(\lambda\)[13], obtaining the RD performance [14], and deriving rate-quantization (R-Q) and distortion-quantization (D-Q) relationships [15]. However, CNN-based rate control methods are always carried out on the hybrid coding framework, which will limit coding performance improvement. With the development of deep learning methods, researchers observed that all modules of the traditional coding framework can be replaced by different types of neural networks, and this can achieve good coding performance; this neural network coding framework is called end-to-end coding. In [16], an end-to-end image compression framework is proposed, which outperforms the traditional image coding standards. In [17], a more advanced entropy model is adopted for end-to-end image compression. In [18], the first end-to-end framework for deep video compression (DVC) is proposed, and the key components of traditional video compression are replaced by neural networks. To further improve end-to-end video compression, [PERSON] et al.