Deep Probabilistic Video Compression

We propose a variational inference approach to deep probabilistic video compression . Our model uses advances in variational autoencoders ( VAEs ) for sequential data and combines it with recent work on neural image compression . The approach jointly learns to transform the original video into a lower-dimensional representation as well as to entropy code this representation according to a temporally-conditioned probabilistic model . We split the latent space into local ( per frame ) and global ( per segment ) variables , and show that training the VAE to utilize both representations leads to an improved rate-distortion performance . Evaluation on small videos from public data sets with varying complexity and diversity show that our model yields competitive results when trained on generic video content . Extreme compression performance is achieved for videos with specialized content if the model is trained on similar videos . 1 INTRODUCTION . The transmission of video content is responsible for up to 80 % of the consumer internet traffic , and both the overall internet traffic as well as the share of video data is expected to increase even further in the future ( Cisco , 2017 ) . Improving compression efficiency is more crucial than ever . Today , a variety of video codecs exists that have reached an impressive performance . The most commonly used standard is H.264 ( Wiegand et al. , 2003 ) , also known as Advanced Video Coding ( AVC ) . More recent codecs include H.265 ( Sullivan et al. , 2012 ) , also known as High Efficiency Video Coding ( HEVC ) , and VP9 ( Mukherjee et al. , 2015 ) . All of these existing codecs follow the same block based hybrid structure ( Musmann et al. , 1985 ) which essentially emerged from engineering out and refining this concept over decades . From a high level perspective , they differ in a huge number of smaller design choices and have grown to become more and more complex systems . While there is room for improving the block based hybrid approach even further ( Fraunhofer , 2018 ) , the question is how much longer significant improvements can be obtained when following the same concept . Interestingly , in the context of image compression , deep learning approaches that are fundamentally different to existing codecs have already shown promising results ( Ballé et al. , 2018 ; 2016 ; Theis et al. , 2017 ; Agustsson et al. , 2017 ; Minnen et al. , 2018 ) , mostly within the past year . Motivated by these successes for images , we propose a first step towards innovating beyond blockbased hybrid codecs by framing video compression in a deep probabilistic context . To this end , we propose an unsupervised deep learning approach to encoding video . The approach simultaneously learns the optimal transformation of the video to a low-dimensional representation and a powerful predictive model that assigns probabilities to video segments , allowing us to efficiently entropy-code the discretized latent representation into a short code length . Our end-to-end neural video compression scheme is based on sequential variational autoencoders ( Bayer & Osendorfer , 2014 ; Chung et al. , 2015 ; Li & Mandt , 2018 ) and the approach of Ballé et al . ( 2016 ) for discretizing and entropy coding a continuous latent representation . The transformations to and from the latent representation , known as the encoder and decoder , are parameterized by deep neural networks and are learned by unsupervised training on videos . We introduce both local latent variables , which are inferred from a single frame , and a global state , inferred from an entire segment , to efficiently store a video sequence . Furthermore , the trajectory of the latent variables is modeled stochastically by a deep probabilistic model . After training , the context-dependent predictive model is used to entropy code the latent variables into binary with an arithmetic coder . As the first step towards a new approach , we focus on small resolution video ( 64 × 64 ) and aim to efficiently capture temporal correlations . Figure 1 shows a test example of the possible performance improvements using our approach if the model is trained on similar content . The plots show two frames of a video , compressed and reconstructed by our approach and classical video codecs . One sees that fine granular details , such as the hands of the cartoon character , are lost in the classical approach due to artifacts from block motion estimation ( low bitrate regime ) , whereas our deep learning approach successfully captures these details with less than 10 % of the file length . Our contributions can be understood as follows : 1 ) Deep probabilistic video compression . To the best of our knowledge , this is the first work to employ a variational autoencoder ( VAE ) in conjuction with discretization and entropy coding to build an end-to-end trainable video codec . 2 ) Global inference . Temporal redundancy in a video can be taken into account by a temporal prior on each frame or through an architectural design that encodes an entire video segment into a global state . We propose a model which incorporates both global and local latent variables and show that this produces a shorter code length for a given image quality . 3 ) Small bit rates . We perform experiments on three large public data sets of varying complexity and diversity . Performance is evaluated by rate-distortion curves . Our method is competitive with traditional codecs on small videos after training and testing on a diverse set of videos . Extreme compression performance can be achieved for videos with specialized content if the model is trained on similar videos . Paper Organization . In Section 2 , we summarize important related works before describing our method for deep video compression in Section 3 . Section 4 discusses our experimental results , including quantitative results and a qualitative discussion . 2 RELATED WORK . The approaches related to our method fall into three categories : deep generative video models , neural image compression , and neural video compression . Deep Generative Video Models . Several works have applied the variational autoencoder ( VAE ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) to stochastically model sequences ( Bayer & Osendorfer , 2014 ; Chung et al. , 2015 ) . Babaeizadeh et al . ( 2018 ) ; Xu et al . ( 2018 ) use a VAE for stochastic video generation . He et al . ( 2018 ) and Denton & Fergus ( 2018 ) apply a long short term memory ( LSTM ) in conjunction with a sequential VAE to model the evolution of the latent space across many video frames . Li & Mandt ( 2018 ) separate latent variables of a sequential VAE into local and global variables in order to learn a disentangled representation for video generation . Vondrick et al . ( 2016 ) generate realistic videos by using a GAN ( Goodfellow et al. , 2014 ) to learn to separate foreground and background , and Lee et al . ( 2018 ) combine variational and adversarial methods to generate realistic videos . This paper also employs a deep generative model to model the sequential probability distribution of frames from a video source . In contrast to other work , our work learns a continuous latent representation that can be discretized with minimal information loss , required for further compression . Furthermore , our objective is to convert the original video into a short binary description rather than to generate new videos . Neural Image Compression . There has been significant work on applying deep learning to image compression . In Toderici et al . ( 2016 ; 2017 ) , a LSTM based codec is used to capture spatial correlations of pixel values and can achieve different bit-rates without having to retrain the model . Ballé et al . ( 2016 ) perform image compression with a VAE and demonstrate how to approximately discretize the VAE latent space by introducing noise during training . This work is refined by ( Ballé et al. , 2018 ) which improves the prior model ( used for entropy coding ) beyond the mean-field approximation by transmitting side information in the form of a hierarchical model . Minnen et al . ( 2018 ) consider an autoregressive model to achieve a similar effect . These image codecs encode each image independently and therefore their probabilistic models are stationary with respect to time . In contrast , our method performs compression according to a non-stationary , time-dependent probability model which has much lower entropy per pixel . Neural Video Compression . The use of deep neural networks for video compression is relatively new . Wu et al . ( 2018 ) perform video compression through image interpolation between reference frames using a predictive model based on a deep neural network . Chen et al . ( 2017 ) and Chen et al . ( 2018 ) use a deep neural architecture to predict the most likely frame with a modified form of block motion prediction and store residuals in a lossy representation . Since these works are based on block prediction , they are similar in function and in performance to existing codecs . Our method is not based on block motion estimation , and the full inferred probability distribution over the space of plausible subsequent frames ( rather than residuals ) is used for entropy coding . 3 DEEP PROBABILISTIC VIDEO COMPRESSION . The objective of lossy video compression can be defined as finding the shortest description of a video while tolerating a certain level of information loss . Classical video codecs try to predict intermediate frames based on block motion estimates , since information which can be predicted does not need to be stored . The residual error is then stored in a lossy transform representation . An end-to-end machine learning approach to encoding video , however , should simultaneously learn the appropriate predictive model and the optimal lossy transformation . This allows both to transform the video into a low dimensional latent representation and to then use the jointly learned predictive model to remove the remaining redundancy in the latents by entropy coding them to a short binary representation ( Huffman , 1952 ; Langdon , 1984 ) . The challenge over image compression is that the data is now sequences of images . These exhibit strong temporal correlations in addition to the spatial correlations already present in images . A naive approach to neural video compression would be to encode the video frame-by-frame , using the marginal distribution of images as done in VAE image compression . This distribution does not capture the temporal correlations and therefore tends to have high entropy , leading to long code lengths . On the other hand , treating an entire video segment as an independent data point in the latent representation leads to data sparseness and poor generalization performance . Therefore , we propose to use a temporally-conditioned prior distribution parameterized by a deep generative model to efficiently code the latent variables associated with each frame . By conditioning on nearby frames in the sequence , the predictive model can be more certain about the next frame , thus achieving a smaller entropy and code length . As detailed below , in addition to using a deep sequential probabilistic model , we propose an architecture that combines local and global information in the video . A global variable stores information that is common to the sequence of frames , while a local variable stores additional dynamical content . In the following paragraphs , we describe our approach ( see Fig . 3 ) in more detail . We describe the encoder and decoder models , the objective function , and the interplay between our deep probabilistic sequential model and entropy coding scheme . Decoder . We propose a stochastic recurrent variational autoencoder to transform a sequence of frames x1 : T = ( x1 , · · · , xT ) into a compressed representation of local latent variables z1 : T = ( z1 , · · · , zT ) , where each zi only depends on xi . This model is refined to additionally include a global state f similar to Li & Mandt ( 2018 ) , resulting in the following probabilistic deep generative model : pθ ( x1 : T , z1 : T , f ) = pθ ( f ) pθ ( z1 : T ) T∏ t=1 pθ ( xt | zt , f ) ( 1 ) Above , θ is shorthand for the parameters . Each frame xt at time t depends on the corresponding latent variables zt and global variables f . The frame likelihood pθ ( xt|f , zt ) for reconstruction is the Laplace distribution , Laplace ( µθ ( zt , f ) , λ −11 ) . The reason for its choice is that its log likelihood results in an ` 1 loss , which typically produces sharper images than the ` 2 loss for autoencoding images ( Isola et al. , 2016 ; Zhao et al. , 2015 ) . The decoder µθ ( · ) is a function parameterized by neural networks . The prior distributions pθ ( f ) and pθ ( z1 : T ) will be discussed separately below . After training , the reconstructed frame in image space is obtained from taking the most likely frame x̃t = argmax pθ ( xt|f , zt ) = µθ ( zt , f ) . Crucially , the decoder is conditioned both on global code f and time-local code zt . Encoder . We employ amortized variational inference ( Blei et al. , 2017 ; Zhang et al. , 2017 ; Marino et al. , 2018 ) to predict a distribution over latent codes given the input video , qφ ( z1 : T , f | x1 : T ) = qφ ( f | x1 : T ) T∏ t=1 qφ ( zt | xt ) . ( 2 ) The global variables f are inferred from all video frames in a sequence and may thus contain static information , while zt is only inferred from a single frame xt . As will be explained in the paragraph on model-based entropy coding below , modifications to standard variational inference are required for compression . Instead of sampling from Gaussian distributions with learned variances , here we employ uniform distributions centered at their means : f̃ ∼ qφ ( f | x1 : T ) = U ( f̂ − 1 2 , f̂ + 1 2 ) ; z̃t ∼ qφ ( zt | xt ) = U ( ẑt − 1 2 , ẑt + 1 2 ) . ( 3 ) The means are predicted by additional encoder neural networks f̂ = µφ ( x1 : T ) , ẑt = µφ ( xt ) with parameters φ . This corresponds to adding random noise i ∼ U ( − 12 , 1 2 ) to f̂ and µ̂ in the inference process . The encoder and decoder neural networks are described in more detail in Appendix B . Prior Models . The models that parameterize the learned prior distributions are ultimately used for entropy coding . Each dimension of the latent space has its own density model : pθ ( f ) = dim ( f ) ∏ i pθ ( f i ) ∗ U ( −1 2 , 1 2 ) ; pθ ( z1 : T ) = T∏ t dim ( z ) ∏ i pθ ( z i t | ct ) ∗ U ( − 1 2 , 1 2 ) . ( 4 ) Above , indices refer to the dimension index of the latent variable and ct is a time-dependent context . The convolution with uniform noise is to allow the priors to better match the true marginal distribution when working with the box-shaped approximate posterior in Eq . 3 ( see Ballé et al . ( 2018 ) Appendix 6.2 ) . This convolution has an analytic form in terms of the cumulative probability density . The stationary density pθ ( f i ) is parameterized by a flexible non-parametric , fully-factorized model from ( Ballé et al. , 2018 ) . The density is defined by its cumulative and is built out of compositions of nonlinear probability densities , similar to the construction of a normalizing flow ( Rezende & Mohamed , 2015 ) . Two dynamical models are considered to model the sequence z1 : T . We propose a LSTM prior architecture which conditions on all previous frames in a segment : pθ ( zit | ct ) ≡ pθ ( zit | z < t ) . We also considered a simpler model , which we compare against , with a single frame context : pθ ( zit | zt−1 ) which is essentially a deep Kalman filter ( Krishnan et al. , 2015 ) . A discussion of the connection between the amount of conditional context and the entropy is given in appendix A. Variational Objective . The encoder ( variational model ) and decoder ( generative model ) can be learned jointly by minimizing the VAE loss function which consists of the KL divergence between the approximate and true posterior . In compression applications , however , one needs to adjust the trade-off between the bit rate and distortion . This is achieved by the β-VAE loss ( Higgins et al. , 2016 ; Mandt et al. , 2016 ) , which takes the form ( up to constant terms ) − Ef̃ , z̃1 : T∼q [ log pθ ( x1 : T |f̃ , z̃1 : T ) ] − β Ef̃ , z̃1 : T∼q [ log pθ ( f̃ , z̃1 : T ) ] = Ef̃ , z̃1 : T∼q T∑ t=1 ‖x̃t − xt‖1 + βH [ qφ ( z̃1 : T , f̃ | x1 : T ) , pθ ( f̃ , z̃1 : T ) ] ( 5 ) where the reconstructed frame x̃t = µθ ( z̃t , f̃ ) and we have introduced a parameter β to control the rate-distortion trade-off ( Alemi et al. , 2018 ) . The Laplace parameter λ was set to one . The first term corresponds to the distortion and the second term is the cross entropy between the approximate posterior and the prior . The latter has the interpretation of the expected code length when using the prior distribution p ( f , z1 : T ) to entropy code the latent variables . This term is minimized for p ( f , z1 : T ) = Ex1 : T [ q ( f , z1 : T |x1 : T ) ] , that is , when the empirical distribution of codes matches the prior model . For our choice of generative model , the cross entropy separates into two terms H [ qφ ( f |x1 : T ) , pθ ( f ) ] and H [ qφ ( z1 : T |x1 : T ) , pθ ( z1 : T ) ] . Model-based Entropy Coding . Sequential VAEs can be used to reduce the dimensionality of a video by performing a lossy transformation . However , the reduction in dimensionality does not mean the video is optimally compressed since there still exists redundancy in the form of temporal correlations in the sequence of the latent space variables . Sequential VAEs may include a temporal prior to model this redundancy , but to actually remove this redundancy and achieve a compact binary representation , the latent space has to be entropy coded . This is the distinguishing element between variational autoencoders and compression algorithms . Crucially , the entropy coder needs a discrete vocabulary , which is obtained by rounding the latent state ( f̂ and ẑ1 : T ) after training . Care must be taken such that the quantization is approximated in a differentiable way during training . Ballé et al . ( 2016 ) address the problem of discretization of a continuous latent space in the context of VAE image compression by introducing noise in the inference process . By adding uniform noise to the most-likely inferred latent variables , the VAE is prevented from storing information in the latent space on length scales smaller than the discretization bin size . As such , rounding ( after training ) does not significantly affect the image reconstruction . Our VAE framework exactly leads to such injection of noise with width one ; it corresponds to our choice of a box-shaped approximate posterior distribution , centered at the maximally-likely values for the latent variables . Besides dealing with quantization , for efficient entropy coding we also need an estimate of the frequency of atoms in order to obtain short file sizes . We can obtain this from our prior model .

Summary: This work on video compression extends the variational autoencoder of Balle et al. (2016; 2018) from images to videos. The latent space consists of a global part encoding information about the entire video, and a local part encoding information about each frame. Correspondingly, the encoder consists of two networks, one processing the entire video and one processing the video on a frame-by-frame basis. The prior over latents factorizes over these two parts, and an LSTM is used to model the coefficients of a sequence of frames. The compression performance of the model is evaluated on three datasets of 64x64 resolution: sprites, BAIR, and Kinetics600. The performance is compared to H.264, H.265, and VP9.