Versatile Learned Video Compression

Learned video compression methods have demonstrated great promise in catching1 up with traditional video codecs in their rate-distortion ( R-D ) performance . How-2 ever , existing learned video compression schemes are limited by the binding of3 the prediction mode and the fixed network framework . They are unable to support4 various inter prediction modes and thus inapplicable for various scenarios . In this5 paper , to break this limitation , we propose a versatile learned video compression6 ( VLVC ) framework that uses one model to support all possible prediction modes.7 Specifically , to realize versatile compression , we first build a motion compensation8 module that applies multiple 3D motion vector fields ( i.e. , voxel flows ) for weighted9 trilinear warping in spatial-temporal space . The voxel flows convey the information10 of temporal reference position that helps to decouple inter prediction modes away11 from framework designing . Secondly , in case of multiple-reference-frame predic-12 tion , we apply a flow prediction module to predict accurate motion trajectories with13 a unified polynomial function . We show that the flow prediction module can largely14 reduce the transmission cost of voxel flows . Experimental results demonstrate that15 our proposed VLVC not only supports versatile compression in various settings but16 also achieves comparable R-D performance with the latest Versatile Video Coding17 ( VVC ) standard in terms of MS-SSIM.18 1 Introduction19 Video occupies more than 80 % of network traffic and the amount of video data is increasing rapidly20 [ 1 ] . Thus , the storage and transmission of video become more challenging . A series of hybrid video21 coding standards have been proposed , such as AVC/H.264 [ 2 ] , HEVC/H.265 [ 3 ] and the latest video22 coding standard VVC/H.266 [ 4 ] . These traditional standards are manually designed , evolving for23 decades . However , the development within the hybrid coding framework is gradually saturated.24 Recently , the performance of video compression is mainly improved by designing more complex25 prediction modes , leading to increased coding complexity.26 Deep neural networks are currently promoting the development of data compression . Despite the27 remarkable progress on the field of learned image compression [ 5–10 ] , the area of learned video28 compression is still in early stages . Existing methods for learned video compression can be grouped29 into three categories , including frame interpolation-based methods [ 11 , 12 ] , 3D autoencoder-based30 methods [ 13 , 14 ] , and predictive coding methods with optical flows [ 15–17 ] . So far , among them,31 video compression with optical flow presents the best performance [ 18 ] , where optical flow represents32 pixel-wise motion vector ( MV ) fields utilized for inter frame prediction . In this paper , we also focus33 on this predictive coding architecture . Previous works with optical flow are proposed to support34 specific prediction mode , including unidirectional or bidirectional , single or multiple frame prediction.35 They are too cumbersome to support versatile compression in various settings since they bind the inter36 prediction mode with the fixed network framework . It is important to design a more flexible model to37 handle all possible settings like traditional codecs . In this paper , we propose a versatile learned video38 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . compression ( VLVC ) framework that achieves coding flexibility as well as compression performance.39 A voxel flow based motion compensation module is adopted for higher flexibility , which is then40 extended into multiple voxel flows to perform weighted trilinear warping . In addition , in case of41 multiple-reference-frame prediction , a polynomial motion trajectories based flow prediction module42 is designed for better compression performance . Our motivations are described as follows.43 Motion compensation with multiple voxel flows . Previous works such as [ 15 ] apply 2D optical44 flows for low-delay prediction using single reference frame ( unidirectional prediction , see Fig . 1a ) .45 For the practical random access scenario , bidirectional reference frames are available for more46 accurate frame interpolation [ 12 ] ( Fig . 1b ) . However , the reference positions in these works are47 determined by pre-defined prediction modes . They can not adapt to various inter prediction modes48 where reference positions are various . In this paper , we apply 3D voxel flows to describe not only49 the spatial MVs , but also the information of temporal reference positions ( Fig . 1c ) . We perform50 voxel flow based motion compensation via trilinear warping , which is applicable to single or multiple,51 unidirectional or bidirectional reference frames . Unlike [ 16 ] that adopts scale space flow with trilinear52 warping , we apply voxel flows for inter prediction in spatial-temporal space , which naturally renders53 our model more robust to different coding scenarios . Furthermore , beyond using single MV in every54 position of the current frame , we propose to use multiple voxel flows to describe multiple possible55 reference relationships ( Fig . 1d ) . Then the target pixel is synthesized by weighted fusing of the56 warping results . We show that without increasing the coding cost of motion information , the motion57 compensation is thus more accurate , yielding less residuals and more efficient compression.58 Flow prediction with polynomial motion trajectories . Exploiting multiple reference frames usu-59 ally achieves better compression performance since more reference information is provided . A60 versatile learned video compression model should cover this multi-reference case . While previous61 work [ 17 ] designs a complex flow prediction network to reduce the redundancies of 2D MV fields , the62 number and structure of reference frames are inherent and fixed within the framework . In this paper,63 we design a more intelligent method for flow prediction , i.e. , modelling the prediction modes with64 polynomial coefficients . We formulate different motion trajectories in a time interval by a unified65 polynomial function . The polynomial coefficients are solved by establishing a multivariate equation66 ( see Section 3.2 ) . Since this polynomial function models the accurate motion trajectories , it serves as67 a basic discipline that constrains the predicted motion to be reasonable . We show the transmission68 cost of voxel flows is reduced obviously with the help of additional motion trajectory information.69 Thanks to the above two technical contributions , our proposed VLVC is not only applicable for various70 practical compression scenarios with different inter prediction modes , but also delivers impressive71 R-D performance on standard test sequences . Extensive experimental results demonstrate that our72 method is the first one to achieve comparable performance with VVC in terms of MS-SSIM in both73 low delay and random access configurations . Comprehensive ablation studies and discussions are74 provided to verify the effectiveness of our method.75 2 Related Work76 Learned Image Compression Recent advances in learned image compression [ 5–7 ] , have shown77 the great success of nonlinear transform coding . Many existing methods are built upon hyperprior-78 based coding framework [ 6 ] , which are improved with more efficient entropy models [ 7 , 8 ] , variable-79 rate compression [ 19 ] and more effective quantization [ 9 , 10 ] . While the widely used autoregressive80 entropy models provide significant performance gain in image coding , the high decoding complexity81 is not suitable for practical video compression . We thus employ the hyperprior model [ 6 ] without82 context models in our video compression framework.83 Learned Video Compression Existing approaches [ 11–15 , 17 , 18 , 20–24 ] can be roughly divided84 into three categories : frame interpolation-based methods [ 11 , 12 , 24 ] , 3D autoencoder-based methods85 [ 13 , 14 ] , and predictive coding methods with optical flows [ 15–17 ] . Currently , researchers are more86 interested in the latter two methods . Although 3D autoencoder-based methods requires less time87 complexity , they barely achieve comparable performance with x265 in MS-SSIM [ 14 ] . Meanwhile,88 predictive coding methods with optical flows have outperformed HM in terms of PSNR [ 18 ] .89 Predictive-based video compression approaches [ 15 , 20–24 , 11 , 12 , 17 ] sequentially perform motion90 estimation , motion compression , motion compensation and residual compression . Chen et al . [ 24 ] 91 first propose to predict block of pixels using learned neural network ( DNN ) , and the residual is92 compressed by a RNN-based autoencoder . Wu et al . [ 11 ] propose a interpolation-based approach93 using traditional MVs . Lu et al . [ 15 ] propose an fully end-to-end trainable framework , where all key94 components in the classical video codec are implemented with neural networks . Rippel et al . [ 21 ] 95 jointly compress the motion and residual information , and propose a latent state to memorize the96 information from the past . Djelouah et al . [ 12 ] perform interpolation by the decoded optical flow97 and blending coefficients . They reuse the same autoencoder of I-frame compression and directly98 quantize the corresponding latent space residual . Liu et al . [ 23 ] combine the optical flow estimation99 and motion compression networks into one-stage , and remove the redundancy of quantized flow100 representations using joint spatial-temporal priors . Yang et al . [ 20 ] propose a video compression101 framework with three hierarchical quality layers and recurrent enhancement . In [ 17 ] , multiple frames102 motion prediction are introduced into the P-frame coding . Lu et al . [ 22 ] propose an content adaptive103 and error propagation aware method to reduce error accumulation and achieve adaptive coding.104 Agustsson et al . [ 16 ] replace the bilinear warping operation with scale-space flow which allows105 the model adaptively blur the reference content for better warping results . However , most existing106 methods are designed for particular prediction modes , resulting in inflexibility for different scenarios.107 Video Interpolation The task of video interpolation is closely related to video compression . One108 pioneering work [ 25 ] proposes to use deep voxel flow to synthesize new video frames . Some works109 of video interpolation [ 26–28 ] directly generate the spatially-adaptive convolutional kernels for each110 motion vectors by neural networks . Most recently , [ 29 , 30 ] proposed to relax the kernel shape,111 allowing the models to freely select multiple sampling points in space or space-time . In this paper,112 our employed multiple voxel flows is motivated by the accurate interpolation result in [ 30 ] .113 3 Versatile Learned Video Compression114 To compress video , the original video sequence is first divided into groups of pictures ( gop ) . Let115 x = { x1 , x2 , ... , xT } denote the frames in one gop unit where the gop size is T . To take advantage116 of previous decoded frames , our model predicts the current frame xt from n reference frame ( s ) ,117 i.e. , the lossy reconstruction results compared to the original frames . Here , we denote the reference118 frames as { x̂t1 , x̂t2 , ... , x̂tn } , where { t1 , t2 , ... , tn } is the index of temporal reference position . If119 multiple frames are taken as the reference ( i.e. , n > 1 ) , the reference frames are divided into two120 groups : one is used only for flow prediction , and the other is used for both flow prediction and motion121 compensation . In other words , the reference involved for motion compensation is only a sub-set of122 { x̂t1 , x̂t2 , ... , x̂tn } , which could be concatenated into a volume denoted by X̂t . If only one reference123 frame is available , the volume for warping X̂t = { x̂t−1 } .124 An overview of our video compression framework is shown in Fig . 2 . Previous work [ 16 ] demon-125 strates that an implicit flow encoder can outperform a pre-trained optical flow network and simplify126 the network structures simultaneously . In our paper , we also abandon the use of a pre-trained optical127 flow network in motion encoder . The motion encoder and decoder are similar to image compression128 network [ 5 ] . While the work of [ 16 ] sends current frame and previous reconstruction into motion129 encoder , we make some modifications on the input of motion encoder . Specifically , in our framework,130 the motion encoder is fed with the current frame xt concatenated with predicted frames ( represented131 as x′t in Fig . 2 ) . Here , the predicted frame x ′ t is an estimation of current frame xt . The flow prediction132 module will predict 2D optical flows ft→ti to warp corresponding reference frames x̂ti , each of133 which will generate a predicted frame x′ti . The predicted frames reveal how much information the134 decoder knows about current frame.135 On the decoder side , a motion decoder will generate voxel flows which is used for motion compensa-136 tion via trilinear warping ( details are explained in Section 3.1 ) . In addition , when multiple reference137 frames are available , the flow prediction module is turned on ( the red dashed box ) . As shown in138 Fig . 2 , the flow prediction module is auxiliary for motion encoding and decoding , which reduces139 the transmission cost of the quantized motion latent m̂t . The specific mechanism of flow prediction140 can be found in Section 3.2 . After motion compensation , we obtain a prediction of current frame as141 x̄t . The residual encoder and decoder are then used to compress the remaining residuals between the142 original frame xt and the predicted frame x̄t , yielding the final reconstructed frame x̂t and quantized143 latent r̂t.144

Summary: The authors introduce VLVC, a new neural video compression codec. Thanks to a new 3D motion compensation structure based on spatio-temporal interpolation, a single trained model can be run in arbitrary prediction modes (such as low-delay or random-access settings). In addition, each frame can be compressed with an arbitrary number of reference frames. There are a number of smaller changes to established neural video codecs. The resulting VLVC codec is demonstrated on standard datasets, showing good rate-distortion results (in particular when trained and evaluated on MS-SSIM).