Query: YUV  deep video compression

title: A Combined Deep Learning based End to End Video Coding Architecture for YUV Color Space
abstract: Most of the existing deep learning based end to end video coding (DLEC) architectures are designed specifically for RGB color format, yet the video coding standards, including H.264/AVC, H.265/HEVC and H.266/VVC developed over past few decades, have been designed primarily for YUV 4:2:0 format, where the chrominance (U and V) components are subsampled to achieve superior compression performances considering the human visual system. While a broad number of papers on DLEC compare these two distinct coding schemes in RGB domain, it is ideal to have a common evaluation framework in YUV 4:2:0 domain for a more fair comparison. This paper introduces a new DLEC architecture for video coding to effectively support YUV 4:2:0 and compares its performance against the HEVC standard under a common evaluation framework. The experimental results on YUV 4:2:0 video sequences show that the proposed architecture can outperform HEVC in intra frame coding, however inter frame coding is not as efficient on contrary to the RGB coding results reported in recent papers.

title: Transform Network Architectures for Deep Learning Based End to End Image/Video Coding in Subsampled Color Spaces
abstract: Most of the existing deep learning based end to end image/video coding (DLEC) architectures are designed for non subsampled RGB color format. However, in order to achieve a superior coding performance, many state of the art block based compression standards such as High Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266) are designed primarily for YUV 4:2:0 format, where U and V components are subsampled by considering the human visual system. This paper investigates various DLEC designs to support YUV 4:2:0 format by comparing their performance against the main profiles of HEVC and VVC standards under a common evaluation framework. Moreover, a new transform network architecture is proposed to improve the efficiency of coding YUV 4:2:0 data. The experimental results on YUV 4:2:0 datasets show that the proposed architecture significantly outperforms naive extensions of existing architectures designed for RGB format and achieves about 10% average BD rate improvement over the intra frame coding in HEVC.

title: Subband Decomposition for High Resolution Color in HEVC and AVC 4:2:0 Video Coding Systems
abstract: High resolution color video (in 4:4:4 format) is becoming increasingly important for screen content and graphics. We have developed a frame packing arrangement (FPA) scheme that enables transmission of 4:4:4 luma chroma (YUV or YCoCg) content through conventional subsampled chroma (4:2:0) codecs as an approach for low cost support of 4:4:4 content. We focus on the "band separation" method initially discussed in  and evaluate new scenarios with the 4:4:4 and 4:2:0 versions at similar compression levels. We use the HEVC HM12.0 reference software for evaluating the FPA scheme. We vary the luma QP for the main 4:2:0 representation from 10 to 34, keeping the QP for the main frame's chroma and auxiliary frame's luma and chroma (containing three fourths of the 4:4:4 chroma data) at the same value as the luma QP. We then compare the final 4:4:4 chroma rate distortion performance of band separation, direct frame packing, and native HM 4:4:4 encoding. We also show preliminary results for lifting based band separation (with clipping) which eliminates the inherent rounding errors of other band separation filters and provides better rate distortion performance. Fig. 1 provides an example for a typical case of the new results showing that both the "band separation" and "lifting based band separation" approaches have much better performance than previously reported, and are significantly better than the earlier developed "direct" frame packing approach. Although the rate distortion performance is inferior to that of native HM 4:4:4 coding, the FPA scheme enables the use of existing 4:2:0 decoding hardware, thus allowing for considerable battery power savings in mobile devices. A more complete discussion and evaluation is available in

title: Relocated I Frames Detection in H.264 Double Compressed Videos Based on Genetic CNN
abstract: Analyzing the appearance of Relocated I frames is a vital step in double compression detection in Group of Pictures (GOP) non aligned videos. In this work, a frame wise relocated I frames detection method in H.264 double compressed videos based on Genetic CNN is proposed. Video clips which contain three adjacent frames are used as the input of network to separate image content from noise. A preprocessing operation is adopted by extracting the noise residual. The genetic algorithm is applied to verify the possibility of automatically designing deep network structures. The network optimization operation mainly includes CNN encoding, initialization, the construction of fitness function and genetic operations, e.g. selection, mutation and crossover. By testing on a data set composed of published YUV sequences, the results clearly demonstrate the efficacy of the proposed approach and show that the generated CNN can achieve better performance than previous method investigated.

title: DVC: An End To End Deep Video Compression Framework
abstract: Conventional video compression approaches use the predictive coding architecture and encode the corresponding motion information and residual information. In this paper, taking advantage of both classical architecture in the conventional video compression method and the powerful non linear representation ability of neural networks, we propose the first end to end video compression deep model that jointly optimizes all the components for video compression. Specifically, learning based optical flow estimation is utilized to obtain the motion information and reconstruct the current frames. Then we employ two auto encoder style neural networks to compress the corresponding motion and residual information. All the modules are jointly learned through a single loss function, in which they collaborate with each other by considering the trade off between reducing the number of compression bits and improving quality of the decoded video. Experimental results show that the proposed approach can outperform the widely used video coding standard H.264 in terms of PSNR and be even on par with the latest standard H.265 in terms of MS SSIM. Code is released at https:/github.com/GuoLusjtu/DVC.

title: High Quality Single Model Deep Video Compression with Frame Conv3D and Multi frame Differential Modulation
abstract: Deep learning (DL) methods have revolutionized the paradigm of computer vision tasks and DL based video compression is becoming a hot topic. This paper proposes a deep video compression method to simultaneously encode multiple frames with Frame Conv3D and differential modulation. We first adopt Frame Conv3D instead of traditional Channel Conv3D for efficient multi frame fusion. When generating the binary representation, the multi frame differential modulation is utilized to alleviate the effect of quantization noise. By analyzing the forward and backward computing flow of the modulator, we identify that this technique can make full use of past frames' information to remove the redundancy between multiple frames, thus achieves better performance. A dropout scheme combined with the differential modulator is proposed to enable bit rate optimization within a single model. Experimental results show that the proposed approach outperforms the H.264 and H.265 codecs in the region of low bit rate. Compared with recent DL based methods, our model also achieves competitive performance.

title: BVI DVC: A Training Database for Deep Video Compression
abstract: Deep learning methods are increasingly being applied in the optimisation of video compression algorithms and can achieve significantly enhanced coding gains, compared to conventional approaches. Such approaches often employ Convolutional Neural Networks (CNNs) which are trained on databases with relatively limited content coverage. In this paper, a new extensive and representative video database, BVI DVC, is presented for training CNN based video compression systems, with specific emphasis on machine learning tools that enhance conventional coding architectures, including spatial resolution and bit depth up sampling, post processing and in loop filtering. BVI DVC contains 800 sequences at various spatial resolutions from 270p to 2160p and has been evaluated on ten existing network architectures for four different coding tools. Experimental results show that this database produces significant improvements in terms of coding gains over three existing (commonly used) image/video training databases under the same training and evaluation configurations. The overall additional coding improvements by using the proposed database for all tested coding modules and CNN architectures are up to 10.3% based on the assessment of PSNR and 8.1% based on VMAF.

title: Improving Deep Video Compression by Resolution adaptive Flow Coding
abstract: In the learning based video compression approaches, it is an essential issue to compress pixel level optical flow maps by developing new motion vector (MV) encoders. In this work, we propose a new framework called Resolution adaptive Flow Coding (RaFC) to effectively compress the flow maps globally and locally, in which we use multi resolution representations instead of single resolution representations for both the input flow maps and the output motion features of the MV encoder. To handle complex or simple motion patterns globally, our frame level scheme RaFC frame automatically decides the optimal flow map resolution for each video frame. To cope different types of motion patterns locally, our block level scheme called RaFC block can also select the optimal resolution for each local block of motion features. In addition, the rate distortion criterion is applied to both RaFC frame and RaFC block and select the optimal motion coding mode for effective flow coding. Comprehensive experiments on four benchmark datasets HEVC, VTL, UVG and MCL JCV clearly demonstrate the effectiveness of our overall RaFC framework after combing RaFC frame and RaFC block for video compression.

title: Content Adaptive and Error Propagation Aware Deep Video Compression
abstract: Recently, learning based video compression methods attract increasing attention. However, the previous works suffer from error propagation due to the accumulation of reconstructed error in inter predictive coding. Meanwhile, the previous learning based video codecs are also not adaptive to different video contents. To address these two problems, we propose a content adaptive and error propagation aware video compression system. Specifically, our method employs a joint training strategy by considering the compression performance of multiple consecutive frames instead of a single frame. Based on the learned long term temporal information, our approach effectively alleviates error propagation in reconstructed frames. More importantly, instead of using the hand crafted coding modes in the traditional compression systems, we design an online encoder updating scheme in our system. The proposed approach updates the parameters for encoder according to the rate distortion criterion but keeps the decoder unchanged in the inference stage. Therefore, the encoder is adaptive to different video contents and achieves better compression performance by reducing the domain gap between the training and testing datasets. Our method is simple yet effective and outperforms the state of the art learning based video codecs on benchmark datasets without increasing the model size or decreasing the decoding speed.

title: Deep Learning Based Technology in Responses to the Joint Call for Proposals on Video Compression With Capability Beyond HEVC
abstract: Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes) and deep network based coding tools that are embedded into traditional coding schemes (deep tools) Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC) a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC) In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network based intra prediction, convolutional neural network (CNN) based in loop filtering, and CNN based block adaptive resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning based future video coding, towards which a lot of open problems call for further study.