Year: 2021

Abstract:
Recent work by Marino et al. (2020) showed improved performance in sequential density estimation by combining masked autoregressive flows with hierarchical latent variable models. We draw a connection between such autoregressive generative models and the task of lossy video compression. Specifically, we view recent neural video compression methods (Lu et al., 2019; Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight. Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.

heading: 1 INTRODUCTION
text: Recent advances in deep generative modeling have seen a surge of applications, including learningbased compression. Generative models have already demonstrated empirical improvements in image compression, outperforming classical codecs (Minnen et al., 2018; Yang et al., 2020d), such as BPG (Bellard, 2014). In contrast, the less developed area of neural video compression remains challenging due to complex temporal dependencies operating at multiple scales. Nevertheless, recent neural video codecs have shown promising performance gains (Agustsson et al., 2020), in some cases on par with current hand-designed, classical codecs, e.g., HEVC. Compared to hand-designed codecs, learnable codecs are not limited to specific data modality, and offer a promising approach for streaming specialized content, such as sports or video chats. Therefore, improving neural video compression is vital for dealing with the ever-growing amount of video content being created.
Source compression fundamentally involves decorrelation, i.e., transforming input data into white noise distributions that can be easily modeled and entropy-coded. Thus, improving a model’s capability to decorrelate data automatically improves its compression performance. Likewise, we can improve the associated entropy model (i.e., the model’s prior) to capture any remaining dependencies. Just as compression techniques attempt to remove structure, generative models attempt to model structure. One family of models, autoregressive flows, maps between less structured distributions, e.g., uncorrelated noise, and more structured distributions, e.g., images or video (Dinh et al., 2014; 2016). The inverse mapping can remove dependencies in the data, making it more amenable for compression. Thus, a natural question to ask is how autoregressive flows can best be utilized in compression, and if mechanisms in existing compression schemes can be interpreted as flows.
This paper draws on recent insights in hierarchical sequential latent variable models with autoregressive flows (Marino et al., 2020). In particular, we identify connections between this family of models and recent neural video codecs based on motion estimation (Lu et al., 2019; Agustsson et al., 2020). By interpreting this technique as an instantiation of a more general autoregressive flow transform, we propose various alternatives and improvements based on insights from generative modeling.
In more detail, our main contributions are as follows:
1. A new framework. We interpret existing video compression methods through the more general framework of generative modeling, variational inference, and autoregressive flows, allowing us to readily investigate extensions and ablations. In particular, we compare fully data-driven approaches with motion-estimation-based neural compression schemes, and
consider a more expressive prior model for better entropy coding (described in the second bullet point below). This framework also provides directions for future work.
2. A new model. Following the predictive coding paradigm of video compression (Wiegand et al., 2003), Scale-Space Flow (SSF)(Agustsson et al., 2020) uses motion estimation to predict the frame being compressed, and further compresses the residual obtained by subtraction. Our proposed model extends the SSF model with a more flexible decoder and prior, and improves over the state of the art in rate-distortion performance. Specifically, we
• Incorporate a learnable scaling transform to allow for more expressive and accurate reconstruction. Augmenting a shift transform by scale-then-shift is inspired by improvements from extending NICE (Dinh et al., 2014) to RealNVP (Dinh et al., 2016).
• Introduce a structured prior over the two sets of latent variables in the generative model of SSF, corresponding to jointly encoding the motion information and residual information. As the two tend to be spatially correlated, encoding residual information conditioned on motion information results in a more informed prior, and thus better entropy model, for the residual information; this cuts down the bit-rate for the latter that typically dominates the overall bit-rate.
3. A new dataset. The neural video compression community currently lacks large, highresolution benchmark datasets. While we extensively experimented on the publicly available Vimeo-90k dataset (Xue et al., 2019), we also collected and utilized a larger dataset, YouTube-NT1, available through executable scripts. Since no training data was publicly released for the previous state-of-the-art method (Agustsson et al., 2020), YouTube-NT would be a useful resource for making and comparing further progress in this field.

heading: 2 RELATED WORK
text: We divide related work into three categories: neural image compression, neural video compression, and sequential generative models.
Neural Image Compression. Considerable progress has been made by applying neural networks to image compression. Early works proposed by Toderici et al. (2017) and Johnston et al. (2018) leveraged LSTMs to model spatial correlations of the pixels within an image. Theis et al. (2017) first proposed an autoencoder architecture for image compression and used the straight-through estimator (Bengio et al., 2013) for learning a discrete latent representation. The connection to probabilistic generative models was drawn by Ballé et al. (2017), who firstly applied variational autoencoders (VAEs) (Kingma & Welling, 2013) to image compression. In subsequent work, Ballé et al. (2018) encoded images with a two-level VAE architecture involving a scale hyper-prior, which can be further improved by autoregressive structures (Minnen et al., 2018; Minnen & Singh, 2020) or by optimization at encoding time (Yang et al., 2020d). Yang et al. (2020e) and Flamich et al. (2019) demonstrated competitive image compression performance without a pre-defined quantization grid.
Neural Video Compression. Compared to image compression, video compression is a significantly more challenging problem, as statistical redundancies exist not only within each video frame (exploited by intra-frame compression) but also along the temporal dimension. Early works by Wu et al. (2018); Djelouah et al. (2019) and Han et al. (2019) performed video compression by predicting future frames using a recurrent neural network, whereas Chen et al. (2019) and Chen et al. (2017) used convolutional architectures within a traditional block-based motion estimation approach. These early approaches did not outperform the traditional H.264 codec and barely surpassed the MPEG-2 codec. Lu et al. (2019) adopted a hybrid architecture that combined a pre-trained Flownet (Dosovitskiy et al., 2015) and residual compression, which leads to an elaborate training scheme. Habibian et al. (2019) and Liu et al. (2020) combined 3D convolutions for dimensionality reduction with expressive autoregressive priors for better entropy modeling at the expense of parallelism and runtime efficiency. Our method extends a low-latency model proposed by Agustsson et al. (2020), which allows for end-to-end training, efficient online encoding and decoding, and parallelism.
1https://github.com/privateyoung/Youtube-NT
Sequential Deep Generative Models. We drew inspiration from a body of work on sequential generative modeling. Early deep learning architectures for dynamics forecasting involved RNNs (Chung et al., 2015). Denton & Fergus (2018) and Babaeizadeh et al. (2018) used VAE-based stochastic models in conjunction with LSTMs to model dynamics. Li & Mandt (2018) introduced both local and global latent variables for learning disentangled representations in videos. Other video generation models used generative adversarial networks (GANs) (Vondrick et al., 2016; Lee et al., 2018) or autoregressive models and normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Kingma et al., 2016; Papamakarios et al., 2017). Recently, Marino et al. (2020) proposed to combine latent variable models with autoregressive flows for modeling dynamics at different levels of abstraction, which inspired our models and viewpoints.

heading: 3 VIDEO COMPRESSION THROUGH DEEP AUTOREGRESSIVE MODELING
text: We identify commonalities between hierarchical autoregressive flow models (Marino et al., 2020) and state-of-the-art neural video compression architectures (Agustsson et al., 2020), and will use this viewpoint to propose improvements on existing models.

heading: 3.1 BACKGROUND
text: We first review VAE-based compression schemes (Ballé et al., 2017) and formulate existing lowlatency video codecs in this framework; we then review the related autoregressive flow model.
Generative Modeling and Source Compression. Given a a sequence of video frames x1:T , lossy compression seeks a compact description of x1:T that simultaneously minimizes the description length R and information loss D. The distortion D measures the reconstruction error caused by encoding x1:T into a lossy representation z̄1:T and subsequently decoding it back to x̂1:T , while R measures the bit rate (file size). In learned compression methods (Ballé et al., 2017; Theis et al., 2017), the above process is parameterized by flexible functions f (“encoder”) and g (“decoder”) that map between the video and its latent representation z̄1:T = f(x1:T ). The goal is to minimize a rate-distortion loss, with the tradeoff between the two controlled by a hyperparameter β > 0:
L = D(x1:T , g(bz̄1:T e)) + βR(bz̄1:T e). We adopt the end-to-end compression approach of Ballé et al. (2017), which approximates the rounding operations b·e (required for entropy coding) by uniform noise injection to enable gradientbased optimization during training. With an appropriate choice of probability model p(z1:T ), the relaxed version of above R-D (rate-distortion) objective then corresponds to the VAE objective:
L̃ = Eq(z1:T |x1:T )[− log p(x1:T |z1:T )− log p(z1:T )]. (1) In this model, the likelihood p(x1:T |z1:T ) follows a Gaussian distribution with mean x̂1:T = g(z1:T ) and diagonal covariance β2 log 2I, and the approximate posterior q is chosen to be a unit-width uniform distribution (thus has zero differential entropy) whose mean z̄1:T is predicted by an amortized inference network f . The prior density p(z1:T ) interpolates its discretized version, so that it measures the code length of discretized z̄1:T after entropy-coding.
Low-Latency Sequential Compression We specialize Eq. 1 to make it suitable for low-latency video compression, widely used in both conventional and recent neural codecs (Rippel et al., 2019; Agustsson et al., 2020). To this end, we encode and decode individual frames xt in sequence. Given the ground truth current frame xt and the previously reconstructed frames x̂<t, the encoder is restricted to be of the form z̄t = f(xt, x̂<t), and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding, x̂t = g(x̂<t, bz̄te)). Existing codecs usually condition on a single reconstructed frame, substituting x̂<t by x̂t−1 in favor of efficiency. In the language of variational inference, the sequential encoder corresponds to a variational posterior of the form q(zt|xt, z<t), i.e., filtering, and the sequential decoder corresponds to the likelihood p(xt|z≤t) = N (x̂t, β2 log 2I); in both distributions, the probabilistic conditioning on z<t is based on the observation that x̂t−1 is a deterministic function of z<t, if we identify bz̄te with the random variable zt and unroll the recurrence x̂t = g(x̂<t, zt). As we show, all sequential compression approaches considered in this work follow this paradigm, and the form of the reconstruction transform x̂ determines the lowest hierarchy of the corresponding generative process of video x.
Masked Autoregressive Flow (MAF). As a final component in neural sequence modeling, we discuss MAF (Papamakarios et al., 2017), which models the joint distribution of a sequence p(x1:T ) in terms of a simpler distribution of its underlying noise variables y1:T through the following autoregressive transform and its inverse:
xt = hµ(x<t) + hσ(x<t) yt; ⇔ yt = xt−hµ(x<t)hσ(x<t) . (2)
The noise variable yt usually comes from a standard normal distribution. While the forward MAF transforms a sequence of standard normal noises into a data sequence, the inverse flow “whitens” the data sequence and removes temporal correlations. Due to its invertible nature, MAF allows for exact likelihood computations, but as we will explain in Section 3.3, we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods.

heading: 3.2 A GENERAL FRAMEWORK FOR GENERATIVE VIDEO CODING
text: We now describe a general framework that captures several existing low-latency neural compression methods as specific instances and gives rise to the exploration of new models. To this end, we combine latent variable models with autoregressive flows into a joint framework. We consider a sequential decoding procedure of the following form:
x̂t = hµ(x̂t−1,wt) + hσ(x̂t−1,wt) gv(vt,wt). (3)
Eq. 3 resembles the definition of the MAF in Eq. 2, but augments this transform with two sets of latent variables wt,vt ∼ p(wt,vt). Above, hµ and hσ are functions that transform the previous reconstructed data frame x̂t−1 along with wt into a shift and scale parameter, respectively. The function gv(vt,wt) converts these latent variables into a noise variable that encodes residuals with respect to the mean next-frame prediction hµ(x̂t−1,wt).
This stochastic decoder model has several advantages over existing generative models for compression, such as simpler flows or sequential VAEs. First, the stochastic autoregressive transform hµ(x̂t−1,wt) involves a latent variable wt and is therefore more expressive than a deterministic transform (Schmidt & Hofmann, 2018; Schmidt et al., 2019). Second, compared to MAF, the additional nonlinear transform gv(vt,wt) enables more expressive residual noise, reducing the burden on the entropy model. Finally, as visualized in Figure 2, the scale parameter hσ(x̂t−1,wt) effectively acts as a gating mechanism, determining how much variance is explained in terms of the autoregressive transform and the residual noise distribution. This provides an added degree of flexibility, in a similar fashion to how RealNVP improves over NICE (Dinh et al., 2014; 2016).
Our approach is inspired by Marino et al. (2020) who analyzed a restricted version of the model in Eq. 3, aiming to hybridize autoregressive flows and sequential latent variable models for video prediction. In contrast to Eq. 3, their model involved deterministic transforms as well as residual noise that came from a sequential VAE.

heading: 3.3 EXAMPLE MODELS AND EXTENSIONS
text: Next, we will show that the general framework expressed by Eq. 3 captures a variety of state-of-theart neural video compression schemes and gives rise to extensions and new models.
Temporal Autoregressive Transform (TAT). The first special case among the class of models that are captured by Eq. 3 is the autoregressive neural video compression model by Yang et al. (2020b), which we refer to as temporal autoregressive transform (TAT). Shown in Figure 1(a), the decoder g implements a deterministic scale-shift autoregressive transform of decoded noise yt,
x̂t = g(zt, x̂t−1) = hµ(x̂t−1) + hσ(x̂t−1) yt, yt = gz(zt). (4)
The encoder f inverts the transform to decorrelate the input frame xt into ȳt and encodes the result as z̄t = f(xt, x̂t−1) = fz(ȳt), where ȳt =
xt−hµ(x̂t−1) hσ(x̂t−1)
. The shift hµ and scale hσ transforms are parameterized by neural networks, fz is a convolutional neural network (CNN), and gz is a deconvolutional neural network (DNN) that approximately inverts fz .
The TAT decoder is a simple version of the more general stochastic autoregressive transform in Eq 3, where hµ and hσ lack latent variables. Indeed, interpreting the probabilistic generative process of
x̂, TAT implements the model proposed by Marino et al. (2020), as the transform from y to x̂ is a MAF. However, the generative process corresponding to compression (reviewed in Section 3.1) adds additional white noise to x̂, with x := x̂ + , ∼ N (0, β2 log 2I). Thus, the generative process from y to x is no longer an autoregressive flow. Regardless, TAT was shown to better capture the low-level dynamics of video frames than the autoencoder (fz, gz) alone, and the inverse transform decorrelates raw video frames to simplify the input to the encoder fz (Yang et al., 2020b).
DVC (Lu et al., 2019) and Scale-Space Flow (SSF, Agustsson et al. (2020)). The second class of models captured by Eq. 3 belong to the conventional video compression framework based on predictive coding (Cutler, 1952; Wiegand et al., 2003; Sullivan et al., 2012); both models make use of two sets of latent variables z1:T = {w1:T ,v1:T } to capture different aspects of information being compressed, where w captures estimated motion information used in warping prediction, and v helps capture residual error not predicted by warping.
Like most classical approaches to video compression by predictive coding, the reconstruction transform in the above models has the form of a prediction shifted by residual error (decoded noise), and lacks the scaling factor hσ compared to the autoregressive transform in Eq. 3
x̂t = hwarp(x̂t−1, gw(wt)) + gv(vt,wt). (5)
Above, gw and gv are DNNs, ot := gw(wt) has the interpretation of an estimated optical flow (motion) field, hwarp is the computer vision technique of warping, and the residual rt := gv(vt,wt) = x̂t − hwarp(x̂t−1,ot) represents the prediction error unaccounted for by warping. Lu et al. (2019) only makes use of vt in the residual decoder gv , and performs simple 2D warping by bi-linear interpretation; SSF (Agustsson et al., 2020) augments the optical flow (motion) field ot with an additional scale field, and applies scale-space-warping to the progressively blurred versions of x̂t−1 to allow for uncertainty in the warping prediction. The encoding procedure in the above models compute the variational mean parameters as w̄t = fw(xt, x̂t−1), v̄t = fv(xt − hwarp(x̂t−1, gw(wt))), corresponding to a structured posterior q(zt|xt, z<t) = q(wt|xt, z<t)q(vt|wt,xt, z<t). We illustrate the above generative and inference procedures in Figure 1(b).
Proposed: models based on Stochastic Temporal Autoregressive Transform. Finally, we consider the most general models as described by the stochastic autoregressive transform in Eq. 3, shown in Figure 1(c). We study two main variants, categorized by how they implement hµ and hσ:
STAT uses DNNs for hµ and hσ as in (Yang et al., 2020b), but complements it with the latent variable wt that characterizes the transform. In principle, more flexible transforms should give better compression performance, but we find the following variant more parameter efficient in practice: STAT-SSF: a less data-driven variant of the above that still uses scale-space warping (Agustsson
et al., 2020) in the shift transform, i.e., hµ(x̂t−1,wt) = hwarp(x̂t−1, gw(wt)). This can also be seen as an extended version of the SSF model, whose shift transform hµ is preceded by a new learned scale transform hσ .
Structured Prior (SP). Besides improving the autoregressive transform (affecting the likelihood model for xt), one variant of our approach also improves the topmost generative hierarchy in the form of a more expressive latent prior p(z1:T ), affecting the entropy model for compression. We observe that motion information encoded in wt can often be informative of the residual error encoded in vt. In other words, large residual errors vt incurred by the mean prediction hµ(x̂t−1,wt) (e.g., the result of warping the previous frame hµ(x̂t−1)) are often spatially collocated with (unpredictable) motion as encoded by wt. The original SSF model’s prior factorizes as p(wt,vt) = p(wt)p(vt) and does not capture such correlation. We therefore propose a structured prior by introducing conditional dependence between wt and vt, so that p(wt,vt) = p(wt)p(vt|wt). At a high level, this can be implemented by introducing a new neural network that maps wt to parameters of a parametric distribution of p(vt|wt) (e.g., mean and variance of a diagonal Gaussian distribution). This results in variants of the above models, STAT-SP and STAT-SSF-SP, where the structured prior is applied on top of the proposed STAT and STAT-SSF models.

heading: 4 EXPERIMENTS
text: In this section, we train our models both on the existing dataset and our new YouTube-NT dataset. Our model also improves over state-of-the-art neural and classical video compression methods when evaluated on several publicly available benchmark datasets. Lower-level modules and training scheme for our models largely follow Agustsson et al. (2020); we provide detailed model diagrams and schematic implementation, including the proposed scaling transform and structured prior, in Appendix A.4. We also implement a more computationally efficient version of scale-space warping (Agustsson et al., 2020) based on Gaussian pyramid and interpolation (instead of naive Gaussian blurring of Agustsson et al. (2020)); pseudocode is available in Appendix A.3.

heading: 4.1 TRAINING DATASETS
text: Vimeo-90k (Xue et al., 2019) consists of 90,000 clips of 7 frames at 448x256 resolution collected from vimeo.com, which has been used in previous works (Lu et al., 2019; Yang et al., 2020a; Liu et al., 2020). While other publicly-available video datasets exist, they typically have lower
resolution and/or specialized content. e.g., Kinetics (Carreira & Zisserman, 2017) only contains human action videos, and previous methods that trained on Kinetics (Wu et al., 2018; Habibian et al., 2019; Golinski et al., 2020) generally report worse rate-distortion performance on diverse benchmarks (such as UVG, to be discussed below), compared to Agustsson et al. (2020) who trained on a significantly larger and higher-resolution dataset collected from youtube.com.
YouTube-NT. This is our new dataset. We collected 8,000 nature videos and movie/video-game trailers from youtube.com and processed them into 300k high-resolution (720p) clips, which we refer to as YouTube-NT. We release YouTube-NT in the form of customizable scripts to facilitate future compression research. Table 1 compares the current version of YouTube-NT with Vimeo-90k (Xue et al., 2019) and with Google’s closed-access training dataset (Agustsson et al., 2020). Figure 5b shows the evaluation performance of the SSF model architecture after training on each dataset.

heading: 4.2 TRAINING AND EVALUATION PROCEDURES
text: Training. All models are trained on three consecutive frames and batchsize 8, which are randomly selected from each clip, then randomly cropped to 256x256. We trained on MSE loss, following similar procedure to Agustsson et al. (2020) (see Appendix A.2 for details).
Evaluation. We evaluate compression performance on the widely used UVG (Mercat et al., 2020) and MCL JCV (Wang et al., 2016) datasets, both consisting of raw videos in YUV420 format. UVG is widely used for testing the HEVC codec and contains seven 1080p videos at 120fps with smooth and mild motions or stable camera movements. MCL JCV contains thirty 1080p videos at 30fps, which are generally more diverse, with a higher degree of motion and a more unstable camera.
We compute the bit rate (bits-per-pixel, BPP) and the reconstruction quality (measured in PSNR) averaged across all frames. We note that PSNR is a more challenging metric than MS-SSIM (Wang et al., 2003) for learned codecs (Lu et al., 2019; Agustsson et al., 2020; Habibian et al., 2019; Yang et al., 2020a;c). Since existing neural compress methods assume video input in RGB format (24bits/pixel), we follow this convention in our evaluations for meaningful comparisons. We note that HEVC also has special support for YUV420 (12bits/pixel), allowing it to exploit this more compact file format and effectively halve the input bitrate on our test videos (which were coded in YUV420 by default), giving it an advantage over all neural methods. Regardless, we report the performance of HEVC in YUV420 mode (in addition to the default RGB mode) for reference.

heading: 4.3 BASELINE ANALYSIS
text: We trained our models on Vimeo-90k to compare with the published results of baseline models listed in Table 2. Figure 3a compares our proposed models (STAT-SSF, STAT-SSF-SP) with previous state-of-the-art classical codec HEVC and neural codecs on the UVG test dataset. Our STAT-SSFSP model provides superior performance at bitrates≥ 0.07 BPP, outperforming conventional HEVC
even in its favored YUV 420 mode and state-of-the-art neural method SSF (Agustsson et al., 2020), as well as the established DVC (Lu et al., 2019) model, which leverages a more complicated model and multi-stage training procedure. We also note that, as expected, our proposed STAT model improves over TAT (Yang et al., 2020b), with the latter lacking stochasticity in the autoregressive transform compared to our proposed STAT and its variants.
Figure 3a shows that the performance ranking on MCL JCV is similar to on UVG, despite MCL JCV having more diverse and challenging (e.g., animated) content (Agustsson et al., 2020). We provide qualitative results in Figure 2 and 4, offering insight into the behavior of the proposed scaling transform and structured prior, as well as visual qualities of the top-performing methods.

heading: 4.4 ABLATION ANALYSIS
text: Using the baseline SSF (Agustsson et al., 2020) model, we study the performance contribution of each of our proposed components, stochastic temporal autoregressive transform (STAT) and structured prior (SP), in isolation. We trained on YouTube-NT and evaluated on UVG. As shown in Figure 5a, STAT improves performance to a greater degree than SP, while SP alone does not provide noticeable improvement over vanilla SSF (however, note that when combined with STAT, SP offers higher improvement over STAT alone, as shown by STAT-SSF-SP v.s. STAT-SSF in Figure 3a).
To quantify the effect of training data on performance, we compare the test performance (on UVG) of the SSF model trained on Vimeo-90k (Xue et al., 2019) and YouTube-NT. We also provide the reported results from Agustsson et al. (2020), which trained on a larger (and unreleased) dataset. As seen from the R-D curves in Figure 5b, training on YouTube-NT improves rate-distortion performance over Vimeo-90k, in many cases bridging the gap with the performance from the larger closed-access training dataset of Agustsson et al. (2020). At a higher bitrate, the model trained on Vimeo-90k(Xue et al., 2019) tends to have a similar performance to YouTube-NT. This is likely because YouTube-NT currently only covers 8000 videos, limiting the diversity of the short clips.

heading: 5 DISCUSSION
text: We provide a unifying perspective on sequential video compression and temporal autoregressive flows (Marino et al., 2020), and elucidate the relationship between the two in terms of their underlying generative hierarchy. From this perspective, we consider several video compression methods, particularly a state-of-the-art method Scale-Space-Flow (Agustsson et al., 2020), as sequential variational autoencoders that implement a more general stochastic temporal autoregressive transform, which allows us to naturally extend the Scale-Space-Flow model and obtain improved ratedistortion performance on standard public benchmark datasets. Further, we provide scripts to generate a new high-resolution video dataset, YouTube-NT, which is substantially larger than current publicly-available datasets. Together, we hope that this new perspective and dataset will drive further progress in the nascent yet highly impactful field of learned video compression.

heading: 6 ACKNOWLEDGEMENTS
text: We gratefully acknowledge extensive contributions from Yang Yang (Qualcomm), which were indispensable to this work. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA). Yibo Yang acknowledges funding from the Hasso Plattner Foundation. Furthermore, this work was supported by the National Science Foundation under Grants 1928718, 2003237 and 2007719, as well as Intel and Qualcomm.

heading: A APPENDIX
text: A.1 COMMAND FOR HEVC CODEC
To avoid FFmpeg package taking the advantage of the input file color format (YUV420), we first need to dump the video.yuv file to a sequence of lossless png files:
ffmpeg -i video.yuv -vsync 0 video/%d.png
Then we use the default low-latency setting in ffmpeg to compress the dumped png sequences:
ffmpeg -y -i video/%d.png -c:v libx265 -preset medium \ -x265-params bframes=0 -crf {crf} video.mkv
where crf is the parameter for quality control. The compressed video is encoded by HEVC with RGB color space.
To get the result of HEVC (YUV420), we directly execute:
ffmpeg -pix_fmt yuv420p -s 1920x1080 -i video.yuv \ -c:v libx265 -crf {crf} -x265-params bframes=0 video.mkv
A.2 TRAINING SCHEDULE
Training time is about four days on an NVIDIA Titan RTX. Similar to Agustsson et al. (2020), we use the Adam optimizer (Kingma & Ba, 2015), training the models for 1,050,000 steps. The initial learning rate of 1e-4 is decayed to 1e-5 after 900,000 steps, and we increase the crop size to 384x384 for the last 50,000 steps. All models are optimized using MSE loss.
A.3 EFFICIENT SCALE-SPACE-FLOW IMPLEMENTATION
Agustsson et al. (2020) uses a simple implementation of scale-space flow by convolving the previous reconstructed frame x̂t−1 with a sequence of Gaussian kernel σ2 = {0, σ20 , (2σ0)2, (4σ0)2, (8σ0)2, (16σ0)2}. However, this leads to a large kernel size when σ is large, which can be computationally expensive. For example, a Gaussian kernel with σ2 = 256 usually requires kernel size 97x97 to avoid artifact (usually kernel size = (6 ∗ σ + 1)2). To alleviate the problem, we leverage an efficient version of Gaussian scale-space by using Gaussian pyramid with upsampling. In our implementation, we use σ2 = {0, σ20 , σ20 + (2σ0)2, σ20 + (2σ0)2 + (4σ0)2, σ20 +
(2σ0) 2 + (4σ0) 2 + (8σ0) 2, σ20 + (2σ0) 2 + (4σ0) 2 + (8σ0) 2 + (16σ0) 2}, because by using Gaussian pyramid, we can always use Gaussian kernel with σ = σ0 to consecutively blur and downsample the image to build a pyramid. At the final step, we only need to upsample all the downsampled images to the original size to approximate a scale-space 3D tensor. Detailed algorithm is described in Algorithm 1.
Algorithm 1: An efficient algorithm to build a scale-space 3D tensor Result: ssv: Scale-space 3D tensor Input: input input image; σ0 base scale; M scale depth; ssv = [input]; kernel = Create Gaussian Kernel(σ0); for i=0 to M-1 do
input = GaussianBlur(input, kernel); if i == 0 then
ssv.append(input); else
tmp = input; for j=0 to i-1 do
tmp = UpSample2x(tmp); {step upsampling for smooth interpolation}; end ssv.append(tmp);
end input = DownSample2x(input);
end return Concat(ssv)
A.4 LOWER-LEVEL ARCHITECTURE DIAGRAMS
Figure 6 illustrates the low-level encoder, decoder and hyper-en/decoder modules used in our proposed STAT-SSF and STAT-SSF-SP models, as well as in the baseline TAT and SSF models, based on Agustsson et al. (2020). Figure 7 shows the encoder-decoder flowchart for wt and vt separately, as well as their corresponding entropy models (priors), in the STAT-SSF-SP model.

Review summaries:
In this paper, the authors provide a new interpretation of existing video compression models. Their perspective is that a video decoder is a stochastic temporal autoregressive model with latent variables. The introduced latent variables could be either used for providing more expressive power for 1) motion estimation&compensation modeling and 2) residual noise modeling, which are two key components of traditional video codecs. The proposed method shows favorable results when the bitrate is higher than 0.12 bits per pixel on the public benchmarks.

In this paper, the authors focus on the problem of lossy video compression. To this end they propose the application of latent variable sequential generative models, specifically autoregressive flows to compress video streams. They evaluate variations of these models quantitatively including their own proposed version of scale space flow. They also introduce a new dataset named Youtube-NT and show promising quantitative performance.