Hierarchical Autoregressive Modeling for Neural Video Compression

1 INTRODUCTION . Recent advances in deep generative modeling have seen a surge of applications , including learningbased compression . Generative models have already demonstrated empirical improvements in image compression , outperforming classical codecs ( Minnen et al. , 2018 ; Yang et al. , 2020d ) , such as BPG ( Bellard , 2014 ) . In contrast , the less developed area of neural video compression remains challenging due to complex temporal dependencies operating at multiple scales . Nevertheless , recent neural video codecs have shown promising performance gains ( Agustsson et al. , 2020 ) , in some cases on par with current hand-designed , classical codecs , e.g. , HEVC . Compared to hand-designed codecs , learnable codecs are not limited to specific data modality , and offer a promising approach for streaming specialized content , such as sports or video chats . Therefore , improving neural video compression is vital for dealing with the ever-growing amount of video content being created . Source compression fundamentally involves decorrelation , i.e. , transforming input data into white noise distributions that can be easily modeled and entropy-coded . Thus , improving a model ’ s capability to decorrelate data automatically improves its compression performance . Likewise , we can improve the associated entropy model ( i.e. , the model ’ s prior ) to capture any remaining dependencies . Just as compression techniques attempt to remove structure , generative models attempt to model structure . One family of models , autoregressive flows , maps between less structured distributions , e.g. , uncorrelated noise , and more structured distributions , e.g. , images or video ( Dinh et al. , 2014 ; 2016 ) . The inverse mapping can remove dependencies in the data , making it more amenable for compression . Thus , a natural question to ask is how autoregressive flows can best be utilized in compression , and if mechanisms in existing compression schemes can be interpreted as flows . This paper draws on recent insights in hierarchical sequential latent variable models with autoregressive flows ( Marino et al. , 2020 ) . In particular , we identify connections between this family of models and recent neural video codecs based on motion estimation ( Lu et al. , 2019 ; Agustsson et al. , 2020 ) . By interpreting this technique as an instantiation of a more general autoregressive flow transform , we propose various alternatives and improvements based on insights from generative modeling . In more detail , our main contributions are as follows : 1 . A new framework . We interpret existing video compression methods through the more general framework of generative modeling , variational inference , and autoregressive flows , allowing us to readily investigate extensions and ablations . In particular , we compare fully data-driven approaches with motion-estimation-based neural compression schemes , and consider a more expressive prior model for better entropy coding ( described in the second bullet point below ) . This framework also provides directions for future work . 2 . A new model . Following the predictive coding paradigm of video compression ( Wiegand et al. , 2003 ) , Scale-Space Flow ( SSF ) ( Agustsson et al. , 2020 ) uses motion estimation to predict the frame being compressed , and further compresses the residual obtained by subtraction . Our proposed model extends the SSF model with a more flexible decoder and prior , and improves over the state of the art in rate-distortion performance . Specifically , we • Incorporate a learnable scaling transform to allow for more expressive and accurate reconstruction . Augmenting a shift transform by scale-then-shift is inspired by improvements from extending NICE ( Dinh et al. , 2014 ) to RealNVP ( Dinh et al. , 2016 ) . • Introduce a structured prior over the two sets of latent variables in the generative model of SSF , corresponding to jointly encoding the motion information and residual information . As the two tend to be spatially correlated , encoding residual information conditioned on motion information results in a more informed prior , and thus better entropy model , for the residual information ; this cuts down the bit-rate for the latter that typically dominates the overall bit-rate . 3 . A new dataset . The neural video compression community currently lacks large , highresolution benchmark datasets . While we extensively experimented on the publicly available Vimeo-90k dataset ( Xue et al. , 2019 ) , we also collected and utilized a larger dataset , YouTube-NT1 , available through executable scripts . Since no training data was publicly released for the previous state-of-the-art method ( Agustsson et al. , 2020 ) , YouTube-NT would be a useful resource for making and comparing further progress in this field . 2 RELATED WORK . We divide related work into three categories : neural image compression , neural video compression , and sequential generative models . Neural Image Compression . Considerable progress has been made by applying neural networks to image compression . Early works proposed by Toderici et al . ( 2017 ) and Johnston et al . ( 2018 ) leveraged LSTMs to model spatial correlations of the pixels within an image . Theis et al . ( 2017 ) first proposed an autoencoder architecture for image compression and used the straight-through estimator ( Bengio et al. , 2013 ) for learning a discrete latent representation . The connection to probabilistic generative models was drawn by Ballé et al . ( 2017 ) , who firstly applied variational autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) to image compression . In subsequent work , Ballé et al . ( 2018 ) encoded images with a two-level VAE architecture involving a scale hyper-prior , which can be further improved by autoregressive structures ( Minnen et al. , 2018 ; Minnen & Singh , 2020 ) or by optimization at encoding time ( Yang et al. , 2020d ) . Yang et al . ( 2020e ) and Flamich et al . ( 2019 ) demonstrated competitive image compression performance without a pre-defined quantization grid . Neural Video Compression . Compared to image compression , video compression is a significantly more challenging problem , as statistical redundancies exist not only within each video frame ( exploited by intra-frame compression ) but also along the temporal dimension . Early works by Wu et al . ( 2018 ) ; Djelouah et al . ( 2019 ) and Han et al . ( 2019 ) performed video compression by predicting future frames using a recurrent neural network , whereas Chen et al . ( 2019 ) and Chen et al . ( 2017 ) used convolutional architectures within a traditional block-based motion estimation approach . These early approaches did not outperform the traditional H.264 codec and barely surpassed the MPEG-2 codec . Lu et al . ( 2019 ) adopted a hybrid architecture that combined a pre-trained Flownet ( Dosovitskiy et al. , 2015 ) and residual compression , which leads to an elaborate training scheme . Habibian et al . ( 2019 ) and Liu et al . ( 2020 ) combined 3D convolutions for dimensionality reduction with expressive autoregressive priors for better entropy modeling at the expense of parallelism and runtime efficiency . Our method extends a low-latency model proposed by Agustsson et al . ( 2020 ) , which allows for end-to-end training , efficient online encoding and decoding , and parallelism . 1https : //github.com/privateyoung/Youtube-NT Sequential Deep Generative Models . We drew inspiration from a body of work on sequential generative modeling . Early deep learning architectures for dynamics forecasting involved RNNs ( Chung et al. , 2015 ) . Denton & Fergus ( 2018 ) and Babaeizadeh et al . ( 2018 ) used VAE-based stochastic models in conjunction with LSTMs to model dynamics . Li & Mandt ( 2018 ) introduced both local and global latent variables for learning disentangled representations in videos . Other video generation models used generative adversarial networks ( GANs ) ( Vondrick et al. , 2016 ; Lee et al. , 2018 ) or autoregressive models and normalizing flows ( Rezende & Mohamed , 2015 ; Dinh et al. , 2014 ; 2016 ; Kingma & Dhariwal , 2018 ; Kingma et al. , 2016 ; Papamakarios et al. , 2017 ) . Recently , Marino et al . ( 2020 ) proposed to combine latent variable models with autoregressive flows for modeling dynamics at different levels of abstraction , which inspired our models and viewpoints . 3 VIDEO COMPRESSION THROUGH DEEP AUTOREGRESSIVE MODELING . We identify commonalities between hierarchical autoregressive flow models ( Marino et al. , 2020 ) and state-of-the-art neural video compression architectures ( Agustsson et al. , 2020 ) , and will use this viewpoint to propose improvements on existing models . 3.1 BACKGROUND . We first review VAE-based compression schemes ( Ballé et al. , 2017 ) and formulate existing lowlatency video codecs in this framework ; we then review the related autoregressive flow model . Generative Modeling and Source Compression . Given a a sequence of video frames x1 : T , lossy compression seeks a compact description of x1 : T that simultaneously minimizes the description length R and information loss D. The distortion D measures the reconstruction error caused by encoding x1 : T into a lossy representation z̄1 : T and subsequently decoding it back to x̂1 : T , while R measures the bit rate ( file size ) . In learned compression methods ( Ballé et al. , 2017 ; Theis et al. , 2017 ) , the above process is parameterized by flexible functions f ( “ encoder ” ) and g ( “ decoder ” ) that map between the video and its latent representation z̄1 : T = f ( x1 : T ) . The goal is to minimize a rate-distortion loss , with the tradeoff between the two controlled by a hyperparameter β > 0 : L = D ( x1 : T , g ( bz̄1 : T e ) ) + βR ( bz̄1 : T e ) . We adopt the end-to-end compression approach of Ballé et al . ( 2017 ) , which approximates the rounding operations b·e ( required for entropy coding ) by uniform noise injection to enable gradientbased optimization during training . With an appropriate choice of probability model p ( z1 : T ) , the relaxed version of above R-D ( rate-distortion ) objective then corresponds to the VAE objective : L̃ = Eq ( z1 : T |x1 : T ) [ − log p ( x1 : T |z1 : T ) − log p ( z1 : T ) ] . ( 1 ) In this model , the likelihood p ( x1 : T |z1 : T ) follows a Gaussian distribution with mean x̂1 : T = g ( z1 : T ) and diagonal covariance β2 log 2I , and the approximate posterior q is chosen to be a unit-width uniform distribution ( thus has zero differential entropy ) whose mean z̄1 : T is predicted by an amortized inference network f . The prior density p ( z1 : T ) interpolates its discretized version , so that it measures the code length of discretized z̄1 : T after entropy-coding . Low-Latency Sequential Compression We specialize Eq . 1 to make it suitable for low-latency video compression , widely used in both conventional and recent neural codecs ( Rippel et al. , 2019 ; Agustsson et al. , 2020 ) . To this end , we encode and decode individual frames xt in sequence . Given the ground truth current frame xt and the previously reconstructed frames x̂ < t , the encoder is restricted to be of the form z̄t = f ( xt , x̂ < t ) , and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding , x̂t = g ( x̂ < t , bz̄te ) ) . Existing codecs usually condition on a single reconstructed frame , substituting x̂ < t by x̂t−1 in favor of efficiency . In the language of variational inference , the sequential encoder corresponds to a variational posterior of the form q ( zt|xt , z < t ) , i.e. , filtering , and the sequential decoder corresponds to the likelihood p ( xt|z≤t ) = N ( x̂t , β2 log 2I ) ; in both distributions , the probabilistic conditioning on z < t is based on the observation that x̂t−1 is a deterministic function of z < t , if we identify bz̄te with the random variable zt and unroll the recurrence x̂t = g ( x̂ < t , zt ) . As we show , all sequential compression approaches considered in this work follow this paradigm , and the form of the reconstruction transform x̂ determines the lowest hierarchy of the corresponding generative process of video x. Masked Autoregressive Flow ( MAF ) . As a final component in neural sequence modeling , we discuss MAF ( Papamakarios et al. , 2017 ) , which models the joint distribution of a sequence p ( x1 : T ) in terms of a simpler distribution of its underlying noise variables y1 : T through the following autoregressive transform and its inverse : xt = hµ ( x < t ) + hσ ( x < t ) yt ; ⇔ yt = xt−hµ ( x < t ) hσ ( x < t ) . ( 2 ) The noise variable yt usually comes from a standard normal distribution . While the forward MAF transforms a sequence of standard normal noises into a data sequence , the inverse flow “ whitens ” the data sequence and removes temporal correlations . Due to its invertible nature , MAF allows for exact likelihood computations , but as we will explain in Section 3.3 , we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods .

Summary: In this paper, the authors provide a new interpretation of existing video compression models. Their perspective is that a video decoder is a stochastic temporal autoregressive model with latent variables. The introduced latent variables could be either used for providing more expressive power for 1) motion estimation&compensation modeling and 2) residual noise modeling, which are two key components of traditional video codecs. The proposed method shows favorable results when the bitrate is higher than 0.12 bits per pixel on the public benchmarks.