Versatile Learned Video Compression

Learned video compression methods have demonstrated great promise in catching1 up with traditional video codecs in their rate-distortion ( R-D ) performance . How-2 ever , existing learned video compression schemes are limited by the binding of3 the prediction mode and the fixed network framework . They are unable to support4 various inter prediction modes and thus inapplicable for various scenarios . In this5 paper , to break this limitation , we propose a versatile learned video compression6 ( VLVC ) framework that uses one model to support all possible prediction modes.7 Specifically , to realize versatile compression , we first build a motion compensation8 module that applies multiple 3D motion vector fields ( i.e. , voxel flows ) for weighted9 trilinear warping in spatial-temporal space . The voxel flows convey the information10 of temporal reference position that helps to decouple inter prediction modes away11 from framework designing . Secondly , in case of multiple-reference-frame predic-12 tion , we apply a flow prediction module to predict accurate motion trajectories with13 a unified polynomial function . We show that the flow prediction module can largely14 reduce the transmission cost of voxel flows . Experimental results demonstrate that15 our proposed VLVC not only supports versatile compression in various settings but16 also achieves comparable R-D performance with the latest Versatile Video Coding17 ( VVC ) standard in terms of MS-SSIM.18 1 Introduction19 Video occupies more than 80 % of network traffic and the amount of video data is increasing rapidly20 [ 1 ] . Thus , the storage and transmission of video become more challenging . A series of hybrid video21 coding standards have been proposed , such as AVC/H.264 [ 2 ] , HEVC/H.265 [ 3 ] and the latest video22 coding standard VVC/H.266 [ 4 ] . These traditional standards are manually designed , evolving for23 decades . However , the development within the hybrid coding framework is gradually saturated.24 Recently , the performance of video compression is mainly improved by designing more complex25 prediction modes , leading to increased coding complexity.26 Deep neural networks are currently promoting the development of data compression . Despite the27 remarkable progress on the field of learned image compression [ 5–10 ] , the area of learned video28 compression is still in early stages . Existing methods for learned video compression can be grouped29 into three categories , including frame interpolation-based methods [ 11 , 12 ] , 3D autoencoder-based30 methods [ 13 , 14 ] , and predictive coding methods with optical flows [ 15–17 ] . So far , among them,31 video compression with optical flow presents the best performance [ 18 ] , where optical flow represents32 pixel-wise motion vector ( MV ) fields utilized for inter frame prediction . In this paper , we also focus33 on this predictive coding architecture . Previous works with optical flow are proposed to support34 specific prediction mode , including unidirectional or bidirectional , single or multiple frame prediction.35 They are too cumbersome to support versatile compression in various settings since they bind the inter36 prediction mode with the fixed network framework . It is important to design a more flexible model to37 handle all possible settings like traditional codecs . In this paper , we propose a versatile learned video38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . compression ( VLVC ) framework that achieves coding flexibility as well as compression performance.39 A voxel flow based motion compensation module is adopted for higher flexibility , which is then40 extended into multiple voxel flows to perform weighted trilinear warping . In addition , in case of41 multiple-reference-frame prediction , a polynomial motion trajectories based flow prediction module42 is designed for better compression performance . Our motivations are described as follows.43 Motion compensation with multiple voxel flows . Previous works such as [ 15 ] apply 2D optical44 flows for low-delay prediction using single reference frame ( unidirectional prediction , see Fig . 1a ) .45 For the practical random access scenario , bidirectional reference frames are available for more46 accurate frame interpolation [ 12 ] ( Fig . 1b ) . However , the reference positions in these works are47 determined by pre-defined prediction modes . They can not adapt to various inter prediction modes48 where reference positions are various . In this paper , we apply 3D voxel flows to describe not only49 the spatial MVs , but also the information of temporal reference positions ( Fig . 1c ) . We perform50 voxel flow based motion compensation via trilinear warping , which is applicable to single or multiple,51 unidirectional or bidirectional reference frames . Unlike [ 16 ] that adopts scale space flow with trilinear52 warping , we apply voxel flows for inter prediction in spatial-temporal space , which naturally renders53 our model more robust to different coding scenarios . Furthermore , beyond using single MV in every54 position of the current frame , we propose to use multiple voxel flows to describe multiple possible55 reference relationships ( Fig . 1d ) . Then the target pixel is synthesized by weighted fusing of the56 warping results . We show that without increasing the coding cost of motion information , the motion57 compensation is thus more accurate , yielding less residuals and more efficient compression.58 Flow prediction with polynomial motion trajectories . Exploiting multiple reference frames usu-59 ally achieves better compression performance since more reference information is provided . A60 versatile learned video compression model should cover this multi-reference case . While previous61 work [ 17 ] designs a complex flow prediction network to reduce the redundancies of 2D MV fields , the62 number and structure of reference frames are inherent and fixed within the framework . In this paper,63 we design a more intelligent method for flow prediction , i.e. , modelling the prediction modes with64 polynomial coefficients . We formulate different motion trajectories in a time interval by a unified65 polynomial function . The polynomial coefficients are solved by establishing a multivariate equation66 ( see Section 3.2 ) . Since this polynomial function models the accurate motion trajectories , it serves as67 a basic discipline that constrains the predicted motion to be reasonable . We show the transmission68 cost of voxel flows is reduced obviously with the help of additional motion trajectory information.69 Thanks to the above two technical contributions , our proposed VLVC is not only applicable for various70 practical compression scenarios with different inter prediction modes , but also delivers impressive71 R-D performance on standard test sequences . Extensive experimental results demonstrate that our72 method is the first one to achieve comparable performance with VVC in terms of MS-SSIM in both73 low delay and random access configurations . Comprehensive ablation studies and discussions are74 provided to verify the effectiveness of our method.75 2 Related Work76 Learned Image Compression Recent advances in learned image compression [ 5–7 ] , have shown77 the great success of nonlinear transform coding . Many existing methods are built upon hyperprior-78 based coding framework [ 6 ] , which are improved with more efficient entropy models [ 7 , 8 ] , variable-79 rate compression [ 19 ] and more effective quantization [ 9 , 10 ] . While the widely used autoregressive80 entropy models provide significant performance gain in image coding , the high decoding complexity81 is not suitable for practical video compression . We thus employ the hyperprior model [ 6 ] without82 context models in our video compression framework.83 Learned Video Compression Existing approaches [ 11–15 , 17 , 18 , 20–24 ] can be roughly divided84 into three categories : frame interpolation-based methods [ 11 , 12 , 24 ] , 3D autoencoder-based methods85 [ 13 , 14 ] , and predictive coding methods with optical flows [ 15–17 ] . Currently , researchers are more86 interested in the latter two methods . Although 3D autoencoder-based methods requires less time87 complexity , they barely achieve comparable performance with x265 in MS-SSIM [ 14 ] . Meanwhile,88 predictive coding methods with optical flows have outperformed HM in terms of PSNR [ 18 ] .89 Predictive-based video compression approaches [ 15 , 20–24 , 11 , 12 , 17 ] sequentially perform motion90 estimation , motion compression , motion compensation and residual compression . Chen et al . [ 24 ] 91 first propose to predict block of pixels using learned neural network ( DNN ) , and the residual is92 compressed by a RNN-based autoencoder . Wu et al . [ 11 ] propose a interpolation-based approach93 using traditional MVs . Lu et al . [ 15 ] propose an fully end-to-end trainable framework , where all key94 components in the classical video codec are implemented with neural networks . Rippel et al . [ 21 ] 95 jointly compress the motion and residual information , and propose a latent state to memorize the96 information from the past . Djelouah et al . [ 12 ] perform interpolation by the decoded optical flow97 and blending coefficients . They reuse the same autoencoder of I-frame compression and directly98 quantize the corresponding latent space residual . Liu et al . [ 23 ] combine the optical flow estimation99 and motion compression networks into one-stage , and remove the redundancy of quantized flow100 representations using joint spatial-temporal priors . Yang et al . [ 20 ] propose a video compression101 framework with three hierarchical quality layers and recurrent enhancement . In [ 17 ] , multiple frames102 motion prediction are introduced into the P-frame coding . Lu et al . [ 22 ] propose an content adaptive103 and error propagation aware method to reduce error accumulation and achieve adaptive coding.104 Agustsson et al . [ 16 ] replace the bilinear warping operation with scale-space flow which allows105 the model adaptively blur the reference content for better warping results . However , most existing106 methods are designed for particular prediction modes , resulting in inflexibility for different scenarios.107 Video Interpolation The task of video interpolation is closely related to video compression . One108 pioneering work [ 25 ] proposes to use deep voxel flow to synthesize new video frames . Some works109 of video interpolation [ 26–28 ] directly generate the spatially-adaptive convolutional kernels for each110 motion vectors by neural networks . Most recently , [ 29 , 30 ] proposed to relax the kernel shape,111 allowing the models to freely select multiple sampling points in space or space-time . In this paper,112 our employed multiple voxel flows is motivated by the accurate interpolation result in [ 30 ] .113 3 Versatile Learned Video Compression114 To compress video , the original video sequence is first divided into groups of pictures ( gop ) . Let115 x = { x1 , x2 , ... , xT } denote the frames in one gop unit where the gop size is T . To take advantage116 of previous decoded frames , our model predicts the current frame xt from n reference frame ( s ) ,117 i.e. , the lossy reconstruction results compared to the original frames . Here , we denote the reference118 frames as { x̂t1 , x̂t2 , ... , x̂tn } , where { t1 , t2 , ... , tn } is the index of temporal reference position . If119 multiple frames are taken as the reference ( i.e. , n > 1 ) , the reference frames are divided into two120 groups : one is used only for flow prediction , and the other is used for both flow prediction and motion121 compensation . In other words , the reference involved for motion compensation is only a sub-set of122 { x̂t1 , x̂t2 , ... , x̂tn } , which could be concatenated into a volume denoted by X̂t . If only one reference123 frame is available , the volume for warping X̂t = { x̂t−1 } .124 An overview of our video compression framework is shown in Fig . 2 . Previous work [ 16 ] demon-125 strates that an implicit flow encoder can outperform a pre-trained optical flow network and simplify126 the network structures simultaneously . In our paper , we also abandon the use of a pre-trained optical127 flow network in motion encoder . The motion encoder and decoder are similar to image compression128 network [ 5 ] . While the work of [ 16 ] sends current frame and previous reconstruction into motion129 encoder , we make some modifications on the input of motion encoder . Specifically , in our framework,130 the motion encoder is fed with the current frame xt concatenated with predicted frames ( represented131 as x′t in Fig . 2 ) . Here , the predicted frame x ′ t is an estimation of current frame xt . The flow prediction132 module will predict 2D optical flows ft→ti to warp corresponding reference frames x̂ti , each of133 which will generate a predicted frame x′ti . The predicted frames reveal how much information the134 decoder knows about current frame.135 On the decoder side , a motion decoder will generate voxel flows which is used for motion compensa-136 tion via trilinear warping ( details are explained in Section 3.1 ) . In addition , when multiple reference137 frames are available , the flow prediction module is turned on ( the red dashed box ) . As shown in138 Fig . 2 , the flow prediction module is auxiliary for motion encoding and decoding , which reduces139 the transmission cost of the quantized motion latent m̂t . The specific mechanism of flow prediction140 can be found in Section 3.2 . After motion compensation , we obtain a prediction of current frame as141 x̄t . The residual encoder and decoder are then used to compress the remaining residuals between the142 original frame xt and the predicted frame x̄t , yielding the final reconstructed frame x̂t and quantized143 latent r̂t.144

Summary: This paper introduces multi-frame references for learned video coding, which estimates multiple optical flows as voxel flows for motion compensation. With the weighted map, it can achieve weighted trilinear warping to combine the multiple predictions into  the final prediction. It is a good idea to obtain a better predcition to search information across multiple frames.  The author introduces  the polynominal motion modeling to use multiple backward optical flow to generate the single optical flow. Based on the method of softmax spaltting, they convert the backward flow into a forward flow using the flow reversal layer. Then use the forward flow and softmax spaltting for motion compensation. The overall architecture is a hybrid video coding system and the main contribution is inter coding. Residual coding is applied on the feature domain.