Hierarchical Autoregressive Modeling for Neural Video Compression
Authors: Ruihan Yang
Yibo Yang
Joseph Marino
Stephan Mandt
Venue: ICLR 2021 Poster (2021)
Keywords: Compression
Video Compression
Generative Models
Autoregressive Models

Abstract: Recent work by Marino et al. (2020) showed improved performance in sequential density estimation by combining masked autoregressive flows with hierarchical latent variable models. We draw a connection between such autoregressive generative models and the task of lossy video compression. Specifically, we view recent neural video compression methods (Lu et al., 2019; Yang et al., 2020b; Agustssonet al., 2020) as instances of a generalized stochastic temporal autoregressive transform, and propose avenues for enhancement based on this insight. Comprehensive evaluations on large-scale video data show improved rate-distortion performance over both state-of-the-art neural and conventional video compression methods.

Published as a conference paper at ICLR 2021
HIERARCHICAL AUTOREGRESSIVE MODELING FOR NEURAL VIDEO COMPRESSION
Ruihan Yang', Yibo Yang', Joseph Marino and Stephan Mandt
Department of Computer Science, UC Irvine'
Computation & Neural Systems, California Institute of Technology? {ruihan. yang, yibo. yang, mandt Jouci edu, jmarino@caltech.ed
ABSTRACT
Recent work by Marino et al. (2020) showed improved performance in sequential density estimation by combining masked autoregressive flows with hierarchical latent variable models. We draw a connection between such autoregressive gener- ative models and the task of lossy video compression. Specifically, we view recent neural video compression methods (Lu et al., 2019; Yang et al., 2020b; Agustsson et al., 2020) as instances of a generalized stochastic temporal autoregressive trans- form, and propose avenues for enhancement based on this insight. Comprehensive evaluations on large-scall scale video data show improved rate- distoriioo performance over both nate-of-the-ard neural and conventional video compression methods.
INTRODUCTION
Recent advances in deep generative modeling have seen a surge of applications, including learning- based compression. Generative models have already demonstrated empirical improvements image compression, outperforming classical codecs (Minnen et al., 2018; Yang et al., 2020d), such as BPG (Bellard, 2014). In contrast, the less developed area of neural video compression remains chal- lenging due to complex temporal dependencies operating at multiple scales. Nevertheless, recent neural video codecs have shown promising performance gains (Agustsson et al., 2020), in some cases on par with current hand-designed, classical codecs, e.g., HEVC. Compared to hand-designed codecs, learnable codecs are not limited to specific data modality, and offer a promising approach for streaming specialized content, such as sports or video chats. Therefore, improving neural video compression vital for dealing with the ever-growing amount of video content being created.
Source compression fundamentally involves decorrelation, i.e., transforming input data into white noise distributions that can be easily modeled and entropy-coded. Thus, improving a model's ca- pability decorrelate data automatically improves its compression performance. Likewise, improve the associated entropy model (i.c., the model's prior) to capture any remaining dependen- cies. Just as compression techniques attempt remove structure, generative models attempt to model structure. One family of models, autoregressive flows, maps between less structured distribu- tions, e.g., uncorrelated noise, and more structured distributions, e.g., images video (Dinh al., 2014; 2016). The inverse mapping can remove dependencies in the data, making it more amenable for compression. Thus, a natural question ask is how autoregressive flows can best be utilized in compression, and if mechanisms in existing compression schemes can be interpreted as flows.
This paper draws on recent insights in hierarchical sequential latent variable models with autoregres- sive flows (Marino et al., 2020). In particular, we identify connections between this family of models and recent neural video codees based on motion estimation (Luu al., 2019; Agustsson al., 2020). By interpreting this technique we propose various alternatives and improvements based on insights from generative modeling.
instantiation of more general autoregressive flow transform,
In more detail, our main contributions are as follows:
1. A new framework. We interpret existing video compression methods through the more general framework of generative modeling, variational inference, and autoregressive flows, allowing us to readily investigate extensions and ablations. In particular, we compare fully data-driven approaches with motion-estimattion-aasss neural compression schemes, and

---

Published as a conference paper at ICLR 2021
consider more expressive prior model for better entropy coding (described in the second bullet point below). This framework also provides directions for future work.
2. A new model. Following the predictive coding paradigm of video compression (Wiegand et al., 2003), Scale-Space Flow (SSF) et al., 2020) uses motion estimation to predict the frame being compressed, and further compresses the residual obtained by sub- traction. Our proposed model extends the SSF model with a more flexible decoder and prior, and improves over the state of the art in rate- distorston performance. Specifically, we
Incorporate a learnable scaling transform to allow for more expressive and accurate reconstruction. Augmenting shift transform by scale-then-shift is inspired by im- provements from extending NICE (Dinh al., 2014) to RealNVP (Dinh et al., 2016).
Introduce a structured prior over the two sets of latent variables in the generative model of SSF, corresponding to jointly encoding the motion information and residual information. As the two tend to be spatially correlated, encoding residual information conditioned on motion information results in a more informed prior, and thus better entropy model, for the residual information; this cuts down the bit-rate for the latter that typically dominates the overall bit-rate.
3. A new dataset. The neural video compression community currently lacks large, high- resolution benchmark datasets. While we extensively experimented on the publicly avail- able Vimeo-90k dataset (Xue et al., 2019), we also collected and utilized larger dataset, YouTube-NT', available through executable scripts. Since no training data was publicly released for the previous state-of-the-ar method (Agustsson et al., 2020), YouTube-NY would be useful resource for making and comparing further progress in this field.
2
RELATED WORK
We divide related work into three categories: neural image compression, neural video compression, and sequential generative models.
Neural Image Compression. Considerable progress has been made by applying neural networks to image compression. Early works proposed by Toderici et al. (2017) and Johnston et al. (2018) leveraged I. LSTMs model spatial correlations of the pixels within image. Theis al. (2017) first proposed an autoencoder architecture for image compression and used the straight-through estima- tor (Bengio et al.. 2013) for learning discrete latent representation. The connection to probabilistic generative models was drawn by Ballé et al. (2017), who firstly applied variational autoencoders (VAEs) (Kingma & Welling, 2013) to image compression. In subsequent work, Ballé al. (2018) encoded images with two-level VAE architecture involving scale hyper-prior, which can be fur- ther improved by autoregressive structures (Minnen al., 2018; Minnen & Singh, 2020) by optimization at encoding time (Yang et al., 2020d). Yang et al. (2020e) and Flamich et al. (2019) demonstrated competitive image compression performance without pre-defined quantization grid.
Neural Video Compression. Compared to image compression, video compression is signifi- cantly more challenging problem, as statistical redundancics cxist not only within each video frame (exploited by intra-frame compression) but also along the temporal dimension. Early works by Wu et al. 20; Djelouah et al. (2019) and Han et al. (2019) performed video compression by predict- ing future frames using recurrent neural network, whereas Chen et al. (2019) and Chen et al. (2017) used convolutional architectures within a traditional block-based motion estimation approach. These early approaches did not outperform the traditional H.264 codec and barely surpassed the MPEG-2 codec. Lu et al. (2019) adopted a hybrid architecture that combined pre-trained Flownet (Dosovit- skiy et al., 2015) and residual compression, which leads to an elaborate training scheme. Habibian et al. (2019) and Liu et al. (2020) combined 3D convolutions for dimensionality reduction with ex- pressive autoregressive priors for better entropy modeling at the expense of parallelism and runtime efficiency. Our method extends a low-latency model proposed by Agustsson et al. (2020), which allows for end-to-end training, efficient online encoding and decoding, and parallelism.
https:///gittubo.oo thub. http::/h...bb...coooorttttttotrt
2

---

Published as a conference paper at ICLR 2021
Sequential Deep Generative Models. We drew inspiration from body of work sequential generative modeling. Early deep learning architectures for dynamics forecasting involved RNNs (Chung al., 2015). Denton & Fergus (2018) and Babacizadeh et al. (2018) used VAE-based stochastic models in conjunction with LSTMs to model dynamics. Li & Mandt (2018) introduced both local and global latent variables for learning disentangled representations in videos. Other video generation models used generative adversarial networks (GANs) (Vondrick et al., 2016; Lee al., 2018) or autoregressive models and normalizing flows (Rezende & Mohamed, 2015; Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018; Kingma et al., 2016; Papamakarios et al., 2017). Recently, Marino al. (2020) proposed to combine latent variable models with autoregressive flows for modeling dynamics at different levels of abstraction, which inspired our models and viewpoints.
3 VIDEO COMPRESSION THROUGH DEEP AUTOREGRESSIVE MODELING
We identify commonalities between hierarchical autoregressive flow models (Marino et al., 2020) and state-of- ofate-of-the-art neural video compression architectures (Agustsson al., 2020), and will use this viewpoint to propose improvements on existing models.
3.1 BACKGROUND
We first review VAE-based compression schemes (Ballé al., 2017) and formulate existing low- latency video codecs in this framework; we then review the related autoregressive flow model.
Generative Modeling and Source Compression. Given sequence of video frames X1:7. lossy compression seeks a compact description of X1:7 that simultaneously minimizes the description length R and information loss D. The distortion D measures the reconstruction error caused by encoding X into a lossy representation Z and subsequently decoding it back to &1.T, while R measures the bit rate (file size). In learned compression methods (Ballé al., 2017: Theis et al., 2017), the above process is parameterized by flexible functions f ("encoder") and ("decoder") that map between the video and its latent representation ž1.T = fxx:r)) The goal is to minimize rate-distortion loss, with the tradeoff between the two controlled by a thperpprameeter 3 > 0:
= D(x1:1 9 [Z1rr] 1)) + ( [.rr]:
We adopt the end-to-end compression approach of Ballé et al. (2017), which approximates the rounding operations 1.1 (required for entropy coding) by uniform noise injection to enable gradient- based optimization during training. With an appropriate choice of probability model p(zı.T), the relaxed version of above R-D (rate-distortion) objective then corresponds to the VAE objective:
C | log (Z.r) log p(z).)).
In this model, the likelihood p(x (Z::) follows a Gaussian distribution with mean &1.7 g(zrr) and diagonal covariance Io and the approximate posterior q is chosen be a unit-width uni- form distribution (thus has zero differential entropy) whose mean ž1.7 is predicted by an amortized inference network f. The prior density p(Z1:T) interpolates its discretized version, so that it mea- sures the code length of discretized ž1.T after antropy-coding
Low-Latency Sequential Compression We specialize Eq. to make it suitable for low-latency video compression, widely used in both conventional and recent neural codecs (Rippel al., 2019; Agustsson et al., 2020). To this end, we encode and decode individual frames x, in sequence. Given the ground truth current frame x, and the previously reconstructed frames & the encoder is restricted to be of the form 2, f(x;, & and similarly the decoder computes reconstruction sequentially based on previous reconstructions and the current encoding, & = g(&ct. [z Ex- single reconstructed frame, substituting 8 by & in favor
isting codecs usually condition of efficiency. In the language of variational inference, the sequential encoder corresponds to a vari- ational posterior of the form (Zz |x,, z<c), i.e., filtering, and the sequential decoder corresponds to g I): in both distributions, the probabilistic conditioning on the likelihood p(x; zzz)) (Xx 2log
Zet is based on the observation that - is a deterministic function of zet, if we identify [ž, with the random variable z, and unroll the recurrence & g( 2,). As we show, all sequential com- pression approaches considered in this work follow this paradigm, and the form of the reconstruction transform & determines the lowest hierarchy of the corresponding generative process of video x.
3

---

Published as a conference paper at ICLR 2021
Masked Autoregressive Flow (MAF). As final component in neural sequence modeling, discuss MAF (Papamakarios et al., 2017), which models the joint distribution a sequence p(X1:r) in terms of simpler distribution of its underlying noise variables yı.r through the following au- toregressive transform and its inverse:
h1 (xer) + (x<x)) Ys < (x-h,((xe))
(2)
(x(x)e))
The noise variable y, usually comes from standard normal distribution. While the forward MAF transforms a sequence of standard normal noises into a data sequence, the inverse flow "whitens" the data sequence and removes temporal correlations. Due its invertible nature, MAF allows for exact likelihood computations, but as we will explain in Section 3.3, we will not exploit this aspect in compression but rather draw on its expressiveness in modeling conditional likelihoods.
3.2 GENERAL FRAMEWORK FOR GENERATIVE VIDEO CODING
We now describe general framework that captures several existing low-latency neural compression methods as specific instances and gives rise to the exploration of new models. To this end, we combine latent variable models with autoregressive flows into a joint framework. We consider sequential decoding procedure of the following form:
& hp (& w.) + (8 We) @ 9v we).
Eq. resembles the definition of the MAF Eq. 2, but augments this transform with two sets of latent variables W,, V2 ~ p(W,, vz). Above, hy and h are functions that transform the previous reconstructed data frame & along with w, into a shift and scale parameter, respectively. The function go (Vt, we) converts these latent variables into a noise variable that encodes residuals with respect to the mean next-frame prediction h (X, w,).
This stochastic decoder model has several advantages over existing generative models for com- pression, such as simpler flows or sequential VAEs. First, the stochastic autoregressive transform (8,-1 w.) involves latent variable W; and is therefore more expressive than deterministic transform (Schmidt & Hofmann, 2018; Schmidt et al., 2019). Second, compared to MAF, the addi- tional nonlinear transform go (Ve, we) enables more expressive residual noise, reducing the burden on the entropy model. Finally, as visualized in Figure 2, the scale parameter h, (X, w.) effec- tively acts as gating mechanism, determining how much variance explained in terms of the autoregressive transform and the residual noise distribution. This provides an added degree of flexi- bility, in a similar fashion to how RealNVP improves over NICE (Dinh et al., 2014; 2016).
Our approach is inspired by Marino et al. (2020) who analyzed restricted version of the model Eq. 3, aiming to hybridize autoregressive flows and sequential latent variable models for video prediction. In contrast to Eq. 3, their model involved deterministic transforms well as residual noise that came from a sequential VAE.
3.3 EXAMPLE MODELS AND EXTENSIONS
Next, we will show that the general framework expressed by Eq. captures variety of state-of-the- art neural video compression schemes and gives rise to extensions and new models.
Temporal Autoregressive Transform (TAT). The first special case among the class of models that are captured by Eq. 3 is the autoregressive neural video compression model by Yang et al. (2020b), which we refer to as temporal autoregressive transform (TAT). Shown in Figure ((a), the decoder yg implements deterministic scale-shift autoregressive transform of decoded noise y..
* = g(z,, 1 = hn (x + he (R,-1) Yt. = (z.).
(4)
The encoder f inverts the transform to decorrelate the input frame x, into y. and encodes the result - b. (2, as z, fx,, x (F.). where y. he (8.1)) The shift h, and scale h transforms are parameterized by neural networks, fa is a convolutional neural network (CNN), and ga is a deconvolutional neural network (DNN) that approximately inverts fa-
The TAT decoder is a simple version of the more general stochastic autoregressive transform in Eq 3, where hy and hr lack latent variables. Indeed, interpreting the probabilistic generative process of
4

---

Published as a conference paper at ICLR 2021
(b)
(c)
Spit
Figure I: Model diagrams for the generative and inference procedures of the current frame x₃, for various neural video compression methods. Random variables are shown in circles, all other quan- tities are deterministically computed; solid and dashed arrows describe computational dependency during generation (decoding) and inference (encoding), respectively. Purple nodes correspond to neural encoders (CNNs) and decoders (DNNs), and green nodes implement temporal autoregressive transform. (a) TAT; (b) SSF; (c) STAT or STAT-SSF; the magenta box highlights the additional proposed scale transform absent in SSF, and the red arrow from W; to v. highlights the proposed (optional) structured prior. See Appendix Fig. 7 for computational diagrams of the structured prior.
8, TAT implements the model proposed by Marino et al. (2020), the transform from y to & is a MAF. However, the generative process corresponding to compression (reviewed in Section 3.1) adds additional white noise to &, with x : & 6, (O, 2log I). Thus, the generative process from to X is no longer an autoregressive flow. Regardless, TAT was shown to better capture the low-level dynamics of video frames than the autoencoder (f..9z) alone, and the inverse transform decorrelates raw video frames to simplify the input to the encoder f= (Yang et al., 2020b).
DVC (Lu et al., 2019) and Scale-Space Flow (SSF, Agustsson et al. (2020)). The second class of models captured by Eq. belong to the conventional video compression framework based on predictive coding (Cutler, 1952; Wiegand et al., 2003; Sullivan et al., 2012); both models make use of two sets of latent variables 1 [W1-r. VI.T] } to capture different aspects of information being compressed, where w captures estimated motion information used in warping prediction, and helps capture residual error not predicted by warping.
Like most classical approaches to video compression by predictive coding, the reconstruction trans- form in the above models has the form of a prediction shifted by residual error (decoded noise), and lacks the scaling factor ha compared to the autoregressive transform in Eq. 3
& hwarr ( Iw (w.)) + go. we).
(5)
Above, Iw and 9o are DNNs, O, := Iw (W.) has the interpretation of an estimated optical flow (mo- tion) field, hwarp is the computer vision technique of warping, and the residual r, := (V:, w.) = 8, A 0,) represents the prediction error unaccounted for by warping. Lu et al. (2019) only makes use of V. in the residual decoder Jv. and performs simple 2D warping by bi-linear inter- pretation; SSF (Agustsson et al., 2020) augments the optical flow (motion) field o, with additional scale field, and applies ocalesspace-warping to the progressively blurred versions of &,- to allow for uncertainty in the warping prediction. The encoding procedure in the above models compute the variational mean parameters as w, = fu (X,, (X,, 1),, V. fx( - hwarr (&1-1, Iw (w:))) cor- responding to a structured posterior q(z, [x,, Z<z) = q(w, I Xx,, Z<z (ev)ggvelee...... X Zer). We illustrate the above generative and inference procedures in Figure 1(b).
Proposed: models based on Stochastic Temporal Autoregressive Transform. Finally, we con- sider the most general models described by the stochastic autoregressive transform in Eq. 3, shown in Figure ((c). We study two main variants, categorized by how they implement hy. and hg:
STAT uses DNNs for h2 and h as in (Yang et al., 2020b), but complements it with the latent variable we that characterizes the transform. In principle, more flexible transforms should give better compression performance, but we find the following variant more parameter efficient in practice: STAT-SSF: less data-driven variant of the above that still uses scale-space warping (Agustsson
5

---

Published as a conference paper at ICLR 2021
Previous reconstruction R STAT SSF-SP
Magnitude of the proposed scale parameter dt 1.18. (8.-1. (9x))
Rate savings residual encoding 9i] relative STAT-SSF (BPP-0.0)
Rate savings in resiual encoding [9)
relative SSF (8PP-0.075)
30
Hi
50
SSF warping (maan) prediction 1.18.11
Current reconstruction by STAT-SSF-SP (8PP=0.045
Decoded
s.(. (w,])
Decoded dusi G.d
Figure 2: Visualizing the proposed STAT-SSF-SP model on one frame of UVG video "Shake- NDry". Two methods in comparison, STAT-SSF (proposed) and SSF (Agustsson et al., 2020). have comparable reconstruction quality to STAT-SSF-SP but higher bit-rate; the (BPP, PSNR) for STAT- SSF-SP, STAT-SSF, and SSF are (0.046, 36.97), (0.053, 36.94), and (0.075, 36.97), respectively. In this example, the warping prediction fa, = (8:-1, [W:) incurs large error around the dog's 's mov- ing contour, but models the mostly static background well, with the residual latents [v,] taking up an order of magnitude higher bit-rate than [w.] in the three methods. The proposed scale parameter o, gives the model extra flexibility when combining the noise y, (decoded from (1 ([v,), [w 1)) with the warping prediction für (decoded from [w,] only) to form the reconstruction & f4 o; @ J:: the scale o, downweights contribution from the noise ý, in the foreground where it very costly, and reduces the residual bit-rate R([v;]) (and thus the overall bit-rate) compared to STAT-SSF and SSF (with similar reconstruction quality), as illustrated in the third and fourth figures in the top row.
et al., 2020) in the shift transform, i.e., (8,,,, ,,,,),,,) = hweap (R, (&,-1, In (w.)). This can also be seen as an extended version the SSF model, whose shift transform hp is preceded by a new learned scale transform h..
Structured Prior (SP). Besides improving the autoregressive transform (affecting the likelihood model for xt). one variant of our approach also improves the topmost generative hierarchy in the form of more expressive latent prior p(Z1.r) ), affecting the entropy model for compression. We ob- serve that motion information encoded in w, can often be informative of the residual error encoded in In other words, large residual errors v, incurred by the mean prediction h (8,-1, w.) (e.g.. the result of warping the previous frame h (x,-1) are often spatially collocated with (unpredictable) motion as encoded by W,. The original SSF model's prior factorizes p Wr. V.)
((wz)p(v)) v.)
and does not capture such correlation. We therefore propose a structured prior by introducing condi- tional dependence between W, and Vs, so that p(We, V+) p(w,)) (Wve) //e/). At a high level, this can be implemented by introducing a new neural network that maps W, to parameters of a parametric distribution of (w,) (e.g., mean and variance of diagonal Gaussian distribution). This results in variants of the above models, STAT-SP and STAT-SSE-SP. where the structured prior is applied on top of the proposed STAT and STAT-SSF models.
EXPERIMENTS
In this section, we train our models both on the existing dataset and our new YouTube-NT dataset. Our model also improves over state-of-the-art neural and classical video compression methods when evaluated on several publicly available benchmark datasets. Lower-level modules and training scheme for our models largely follow Agustsson et al. (2020); we provide detailed model diagrams and schematic implementation, including the proposed scaling transform and structured prior, in Appendix A.4. We also implement more computationally efficient version of scale-space warping (Agustsson et al., 2020) based on Gaussian pyramid and interpolation (instead of naive Gaussian blurring of Agustsson al. (2020)); pseudocode is available in Appendix A.3.
4.1 TRAINING DATASETS
Vimeo-90k (Xue et al., 2019) consists of 90,000 clips of 7 frames at 448x256 resolution collected from vimeo. com, which has been used in previous works (Lu et al., 2019; Yang et al., 2020a; Liu et al., 2020). While other publicl--aviilbll video datasets exist, they typically have lower

---

Published as a conference paper at ICLR 2021
resolution and/or specialized content. e.g., Kinetics (Carreira & Zisserman, 2017) only contains human action videos, and previous methods that trained on Kinetics (Wu et al., 2018; Habibian diverse
et al., 2019; Golinski et al., 2020) generally report worse rate-distortion performance benchmarks (such as UVG, to be discussed below), compared to Agustsson et al. (2020) who trained on significantly larger and higher-resolution dataset collected from youtube.. com.
You To. This is our new dataset. We collected 8,000 nature videos and movie/video-gan -amme trailers from youtube. com and processed them into 300k high-resolution (720p) clips, which we refer to as YouTube-NT. We release YouTube-NT in the form of customizable scripts to facilitate future compression research. Table 1 compares the current version of YouTube-NT with Vimeo- 90k (Xue et al.. 2019) and with Google's 's closed-access training dataset (Agustsson et al., 2020). Figure 5b shows the evaluation performance of the SSF model architecture after training on each dataset.
Table 1: Overview of Training Datasets.
Dataset name
Clip length
Resolution
# of clips
# of videos
Public
Configurable
Vimeo-90k
frames
448x256
90,000
5,000
YouTube-NTT (ours)
6-10 frames
1280x
300,000
8,000
Agustsson 2020 et al.
frames
1280x720
700,000
700,000
X
4.2 TRAINING AND EVALUATION PROCEDURES
Training. All models are trained on three consecutive frames and batchsize 8, which are randomly selected from each clip, then randomly cropped 256x256. We trained on MSE loss, following similar procedure to Agustsson et al. (2020) (see Appendix A.2 for details).
Evaluation. We evaluate compression performance on the widely used UVG (Mercat et al., 2020) and MCL.JVV (Wang et al., 2016) datasets, both consisting of raw videos in YUV420 format. UVG widely used for testing the HEVC codec and contains seven 1080p videos at 120fps with smooth and mild motions or stable camera movements. MCL contains thirty 1080p videos at 30fps, which are generally more diverse, with a higher degree of motion and a more unstable camera.
We compute the bit rate (bits-per-pixel, BPP) and the reconstruction quality (measured in PSNR) averaged across all frames. We note that PSNR is a more challenging metric than MS-SSIM (Wang et al., 2003) for learned codecs (Lu et al., 2019; Agustsson et al., 2020; Habibian et al., 2019; Yang et al., 2020a;c). Since existing neural compress methods assume video input in RGB format (24bits/pixel),
follow this convention in our evaluations for meaningful comparisons. We note that HEVC also has special support for YUV420 (12bits/pixel), allowing it to exploit this more compact file format and effectively halve the input bitrate on our test videos (which were coded in YUV420 by default), giving it an advantage over all neural methods. Regardless, we report the performance of HEVC in YUV420 mode (in addition the default RGB mode) for reference.
Table 2: Overview of compression methods and the datasets trained on (if applicable).
Model Name
Category
Vimeo 90kk
Youtube-NT
Remark
STAU-SSF
Proposed
Proposed autoregrosivv transform (on top our officient scale-space flow implememttiion) Same as above (S STAT-SSF) but combined with our proposed strictured priar
STATSSF
Proposed
SSF
Biseline
Aguatsion ot (2020) CVPR
BVC
CVPR
VCHI
Biseline
Wa (2018) BCCV (triindd on Kinecties darases)
DGVE
Biseline
/
8
Hon et al. (2019) NewrIPS modifedd for low -wy-Aateccy compression setup (no future framos used)
TAT
Riseline
Yang al, (2000)) ICML workshop
HEVC
Biseline
N/A
ofa
codec RGB 4:44 space inpur
HEVCTYUV)
Kiseline
N/
atole-ff-hh-ar conventional pade YUV polor space Input
STAT
Ablation
7
Koplice senle spice flw in STAT-SS with neunl network Scale space flow model with sttuctured
SSF-SP
Ablation
X
4.3 BASELINE ANALYSIS
We trained our models on Vimeo-90k to compare with the published results of baseline models listed in Table 2. Figure 3a compares our proposed models (STAT-SSE, STAT-SSF-SP) with previous state-of-the-ane classical codec HEVC and neural codecs on the UVG test dataset. Our STAT-SSF- SP model provides superior performance bitrates > 0.07 BPP, outperforming conventional HEVC
7

---

Published as a conference paper at ICLR 2021
50-
Propusol)
agatnn al 2005)
STAT- SSF SP(Phepuol)
DICI al 20-3)
STAT SSF(Prapzsol)
HEYC/RCB444)
RGB444)
VCTOW st 2015)
SSRI Aasttts al 2123)
SIAT(A
S1Al(A2btion)
TATEYAR
DOV 2019
DOVC
30
(a)
(b)
Rate-Distortion Performance of various models and ablations. Results evaluated (a) UVG and (b) MCLJCV datasets. All the learning- models (except VCII (Wu et al., 2018))
Figure 3:
trained Vimeo-90k. STAT-SSF-SP (proposed) achieves the best performance.
(a) HEVC: BPP 0.087,
(b) SSF: BPP 0.082, PSNR 38.099
(c) STAT-SSF (ours); BPP 0.0768.
(d) STAT-SSF-ST SP (ours): BPP 0.0551. PSNR = 38.097
PSNR = 37.440
PSNR 38.108
Figure 4: Qualitative comparisons of various methods on frame from MCL-JCV video 30. Fig- ures in the bottom row focus on the same image patch on top. Here, models with the proposed scale transform (STAT-SSF and STAT-SSF-SP) outperform the ones without, yielding visually more detailed reconstructions at lower rates; structured prior (STAT-SSF-SP) reduces the bit-rate further.
even in its favored YUV 420 mode and state-of-the-art neural method SSF (Agustsson et al., 2020), as well as the established DVC (Lu et al., 2019) model, which leverages a more complicated model and multi-stage training procedure. We also note that, expected, our proposed STAT model improves over TAT (Yang et al., 2020b), with the latter lacking stochasticity in the autoregressive transform compared proposed STAT and its variants.
8

---

Published as a conference paper at ICLR 2021
40
30-
55rlAp-mmer
traned Youtsb-NN
SSFiAgsatsonn 2000)
STAT-SSF traind on VostbeNT
50P/Ag 2000 riledd Youness
03
33
(a)
(b)
Figure 5: Ablations & Comparisons. (a) An ablation study proposed components. (b) Performance of SSF (Agustsson et al., 2020) trained on different datasets. Both sets of results are evaluated on UVG.
Figure 3a shows that the performance ranking MCLJCV is similar to UVG, despite MCLJCVV having more diverse and challenging (e.g., animated) content (Agustsson et al., 2020). We provide qualitative results in Figure 2 and 4, offering insight into the behavior of the proposed scaling transform and structured prior, as well as visual qualities the top-performing methods.
4.4 ABLATION ANALYSIS
Using the baseline SSF (Agustsson et al., 2020) model, we study the performance contribution of each of our proposed components, stochastic temporal autoregressive transform (STAT) and struc- tured prior (SP), in isolation. We trained on YouTube-NT and evaluated on UVG. As shown in Figure 5a, STAT improves performance to greater degree than SP, while SP alone does not provide noticeable improvement over vanilla SSF (however, note that when combined with STAT, SP offers higher improvement over STAT alone, shown by STAT-SSF-SP
STAT-SSF in Figure 3a).
To quantify the effect training data on performance, we compare the test performance (on UVG) of the SSF model trained on Vimeo-90k (Xue al., 2019) and YouTube-NT. We also provide the reported results from Agustsson et al. (2020), which trained on a larger (and unreleased) dataset. As seen from the R-D curves in Figure 5b, training YouTube-NT improves rate-distortion per- formance over Vimeo-90k, in many cases bridging the gap with the performance from the larger closed-access training dataset of Agustsson et (2020). At higher bitrate, the model trained on Vimeo-90k et al., 2019) tends to have similar performance to YouTube-NT.. This is likely because YouTube-NTI currently only covers 8000 videos, limiting the diversity of the short clips.
DISCUSSION
We provide unifying perspective on sequential video compression and temporal autoregressive flows (Marino et al., 2020), and elucidate the relationship between the two in terms of their un- derlying generative hierarchy. From this perspective, we consider several video compression meth- ods, particularly a state-of-the-art method Scale-Space-Flow ( (Agustsson et al., 2020), as sequential variational autoencoders that implement a more general stochastic temporal autoregressive trans- form, which allows us to naturally extend the Scale-Space-Flov model and obtain improved rate- distortion performance on standard public benchmark datasets. Further, we provide scripts to gen- erate new reghlresolution video dataset, YouTube-NT, which is substantially larger than current publicly-avan available datasets. Together, we hope that this new perspective and dataset will drive further progress in the nascent yet highly impactful field of learned video compression.

---

Published as a conference paper at ICLR 2021
6 ACKNOWLEDGEMENTS
We gratefully acknowledge extensive contributions from Yang Yang (Qualcomm), which were in- dispensable to this work. This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0021, Any opinions, findings and conclusions or recommendations expressed this material are those of the author(s) and do not necessarily reflect the views of the Defense Advanced Research Projects Agency (DARPA), Yibo Yang acknowledges funding from the Hasso Plattner Foundation. Furthermore, this work was sup- ported by the National Science Foundation under Grants 1928718, 2003237 and 2007719, well as Intel and Qualcomm.
REFERENCES
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5503-8512, 2020.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representa- tions, 2018.
Johannes Ballé, Valero Laparra, and Ecro P Simoncelli. End-to-end optimized image compression. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational im- age compression with a scale hyperprior. International Conference on Learning Representations, 2018
Fabrice Bellard. Bpg image format, 2014. URL htt.tt/ss//bblllard..r/////////p txt.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 100. 3432, 2013. 1rrXiv:308..4..
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 1299-6308 2017.
T. Chen, H. Liu, Q. Shen, T. Yue, X. Cao, and Z. Ma. Deepcoder: A deep neural network based video compression. In 2017 IEFE Visual Communications and Image Processing (VCIP), pp- 1-4, 2017.
Zhibo Chen, Tianyu He, Xin Jin, and Feng Wu. Learning for video compression. IEEE Transactions on Circuits and Systems for Video Technology, 30(2):566-576, 2019.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Cassius C Cutler. Differential quantization of communication signals, July 29 1952. US Patent 2,605,361.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In International Conference on Machine Learning, pp. 1174-1183, PMLR, 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti- mation. arXiv preprini 140.8816, 2014.
Laurent Dinh, Jascha Dickkiics,,,,, and Samy Bengio. Density estimation using real nvp. arXiv preprint arXiv:1605 1605008033, 08803, 2016.
10

---

Published as a conference paper at ICLR 2021
A. Djelouah, J. Campos. S. Schaub-Meyer and C. Schroers. Neural inter-frame compression for video coding. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp- 1420-6428, 2019.
Alexey Dosovitskiy, Philipp Fischer, Eddy llg. Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 2758-2766 2015.
Gergely Flamich, Marton Havasi, and José Miguel Hernández-Lobato Compression without quan- tization. In OpenReview, 2019.
Adam Golinski, Reza Pourreza, Yang Yang, Guillaume Sautiere. and Taco S Cohen. Feedback recurrent autoencoder for video compression. arXiv preprint arXiv:2004.04533 2020.
Amirhossein Habibian, Ties van Rozendaal, Jakub Tomczak, and Taco S Cohen. Video compres- sion with rate-distortion autoencoders. In Proceedings of the IEEE International Conference Computer Vision, pp. 7033-7042, 2019.
Jun Han, Salvator Lombardo, Christopher Schroers, and Stephan Mandt. Deep generative video compression. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Nick Johnston, Damien Vincent, David Minnen, Michele Covell, Saurabh Singh, Troy Chinen, Sung Jin Hwang. Joel Shor, and George Toderici. Improved lossy image compression with priming and spatially adaptive bit rates for recurrent networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3385-439) 2018
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes.
arXiv preprint
arXiv: 14, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ix1 convolutions. In Advances in Neural Information Processing Systems, pp. 10215-10224, 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in neural information processing systems, pp. 4743-4751, 2016.
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 2018.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5670-5679, Stockholmsmässsan Stockholm Sweden, 10-15 Jul 2018. PMLR.
Haojie Liu, Han Shen, Lichao Huang, Ming Lu, Tong Chen, and Zhan Ma. Learned video compres- sion via joint seapoal-temporal correlation exploration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 11580-11587, 2020.
Guo Lu. Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end- to-end deep video compression framework. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10055-1001,, 2019.
Joseph Marino, Lei Chen, Jiawei He, and Stephan Mandt. Improving sequential latent variable models with autoregressive flows. In Symposium on Advances in Approximate Bayesian Inference, pp. 1-16, 2020.
11

---

Published as a conference paper at ICLR 2021
Alexandre Mercat, Marko Viitanen, and Jamo Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compression. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3339- 3343. IEEE, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hicrarchical priors for learned image compression. In Advances in Neural Information Processing Systems, pp- 10771-10780, 2018.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pp. 2338-2347, 2017.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference International Conference Machine Learning- Volume 37, pp. 1530-155 1538, 2015.
Oren Rippel, Sanjay Nair, Carissa Lew, S. Branson, Alexander G. Anderson, and Lubomir D. Bour- dev. Learned video compression. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3453-3462, 2019.
Florian Schmidt and Thomas Hofmann. Deep state space models for unconditional word generation. In Advances Neural Information Processing Systems, pp. 6158-6168, 2018.
Florian Schmidt, Stephan Mandt, and Thomas Hofmann. Autoregressive text generation beyond feedback loops. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3391-3397, 2019.
Gary Sullivan, Jens Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (heve) standard. IEEE Transactions circuits and systems for video technology, 22(11):1649 649-1668, 2012.
Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. International Conference on Learning Representations, 2017.
George Toderici, Damien Vincent, Nick Johnston, Sung Jin Hwang, David Minnen, Joel Shor, and Michele Covell. Full resolution image compression with recurrent neural networks. In Proceed- ings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pp. 613-621, 2016.
Haiqiang Wang. Weihao Gan, Sudeng Hu, Joe Yuchich Lin, Lina Jin, Longguang Song, Ping Wang, loannis Katsavounidis, Anne Aaron, and C-C Jay Kuo. Mcl-jev: jnd-based h. 26A/ave video quality assessment dataset. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 1513. IEEE, 2016.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The SSeeenh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pp. 1398-1402. Ieee, 2003.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7): 560-576, 2003.
Chao- Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image inter- polation. Proceedings of the European Conference on Computer Vision (ECCV), pp. 416-431, 2018.
12

---

Published as a conference paper at ICLR 2021
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. International Journal of Computer Vision (WCV), 127(8):1 100--115, 2019.
Ren Yang, Fabian Mentzer, Lue Van Gool, and Radu Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Ruihan Yang, Yibo Yang, Joseph Marino, Yang Yang, and Stephan Mandt. Deep generative video compression with temporal autoregressive transforms. ICML 2020 Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020b.
Yang Yang, Guillaume Sautière, Jon Ryu, and Taco Cohen. Feedback recurrent autoencoder. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3347-3351. IEEE, 2020c.
Yibo Yang, Robert Bamler, and Stephan Mandt. Improving inference for neural image compression. Advances in Neural Information Processing Systems, 33, 2020d.
Yibo Yang, Robert Bamler, and Stephan Mandt. Variational bayesian quantization. In International Conference on Machine Learning, 2020e.
A APPENDIX
A.1 COMMAND FOR HEVC CODEC
To avoid FFmpeg package taking the advantage of the input file color format (YUV420), we first need dump the video. yuv file to sequence of lossless png files:
Empeg video. yuv -vyyne video (oddeo/dd.ppo png
Then we use the default low-natency setting ffmpeg to compress the dumped png sequences:
fmpeg -y -i video/$d. -cvv 1ibx265 -preset medium
-x26 params bframes=0 -crf (crf) video.mke
where cr is the parameter for quality control. The compressed video is encoded by HEVC with RGB color space.
To get the result of HEVC (YUV420), we directly execute:
fmpeg -pix_ fnt yuv420p -8 1920x1080 video. yuv -C:v 1ibx265 -crf {crf) -x265-params bframes=0 video.mky
A.2 TRAINING SCHEDULE
Training time is about four days on an NVIDIA Titan RTX. Similar to Agustsson et al. (2020), we the Adam optimizer (Kingma & Ba, 2015), training the models for 1,050,000 steps. The initial learning rate of le-4 is decayed to le-5 after 900,000 steps, and we increase the crop size to 384x384 for the last 50,000 steps. All models are optimized using MSE loss.
A.3 EFFICIENT SCALE-SPACE-RLO IMPLEMENTATION
Agustsson et al. (2020) uses simple implementation of scale-space flow by convoly- ing the previous reconstructed frame 8,-1 with a sequence of Gaussian kernel o² {0, oe, (2 2 (8oo)², (16og)²). However, this leads to large kernel size when is large, which can be computationally expensive. For example, a Gaussian kernel with σ² = 256 usually requires kernel size 97x97 to avoid artifact (usually kernel_size problem, we leverage an efficient version of Gaussian scale-space by using Gaussian pyramid with upsampling. In our implementation,
(6 + 1)2). To alleviate the
o² of of (2oo)² 2 oB + (2op)² (4ar)² )2 a +
13

---

Published as a conference paper at ICLR 2021
(20o))+ (40))² (8or)² ) because by using Gaussian pyramid, we can always use Gaussian kernel with o do to consecutively blur and downsample the image to build pyramid. At the final step, only need to upsample all the downsampled images to the original size to approximate scale-space 31) tensor. Detailed algorithm is described in Algorithm 1.
(20u)2 (4ou)² (8ou))
Algorithm 1: An efficient algorithm to build scale-space 3D tensor
Result: ssv: Scale-space 3D tensor
Input:
input imput image; do base scale; M scale depth: ssv = [input];
kernel = C-eerteeeaaaiiae GaussiannKemme((r))) Kernel(oo):
for i=0 to M-/ do
input GaussianBluriinput, kernel);
if = then
ssv.append(input);
else
tmp input: for j0 to i-/ do
tmp UpSample2x(mpp)) (step upsampling for smooth interpolation);
end
ssv.append(tmp);
end
input= = DownSample2x(inpu))
end
return Concat(ssv)
A.4 LOWER-LEVEL ARCHITECTURE DIAGRAMS
Figure illustrates the low-level encoder, decoder and hyper-en/decoder modules used our pro- posed STAT-SSF and STAT-SSF-SP models, as well as in the baseline TAT and SSF models, based on Agustsson et al. (2020). Figure 7 shows the encoder-decoder flowchart for w, and Ve separately, well their corresponding entropy models (priors), in the STAT-SSF-SH model.
DeConv 5x5/2, 128
DeConv 5x5/2, 192
Convv 5x5/2, 128
Conv 5x5/2,
ReLU
ReLU
ReLU
ReLU
DeConv 5x5/2, 128
DeConv 5x5/2, 192
Conv 5x5/2, 128
Conv 5x5/2, 192
ReLU
ReLU
ReLU
ReLU
DeConv 5x5/2, 128
Conv 5x5/2, 128
DeConv 5x5/2, 384
Conv 5x5/2, 192
ReLU
ReLU
Hyper Encoder
Hyper Decoder
DeConv 5x5/2, channel
Conv 5x5/2, 192
Encoder
Decoder
Figure 6: Backbone module architectures, where "5x5/2, 128" means 5x5 convolution kernel with stride 2; the number filters is 128.
14

---

Published as a conference paper at ICLR 2021
AT
AT-1
Enooder
Encoder
Encoder
Encoder
HyperEncader
HyperEncoder
Decoder SSF warping
Fectorized Prior
Factorized
W:
Decoder
w/
HyperDecoder
HyperDecoder
AT
W:
AT
21
N (pau Fw)
Ni o.)
En Decoder We
En/Decoder for V4
Prions We
Priors Vt
Figure 7: Computational flowchart for the proposed STAT-SSF-SP model. The left two subfig- ures show the decoder and encoder flowcharts for w, and v,, respectively, with "AT" denoting autoregressive transform. The right two subfigures show the prior distributions that are used for en- tropy coding w, and Vt, respectively, with p(w, w!) p(w)))((w; wh), and p(v,, v/ v/(ww, w!) p(v])p((v vf, W2, wh),, with wi and vi denoting hyper latents (see (Agustsson et al., 2020) for description of hyper-priors); note that the priors in the SSF and STAT-SSF models (without the proposed structured prior) correspond to the special case where the HyperDecoder for va does not receive wh and w, as inputs, that the entropy model for Ve independent of wi plve, v!t) p(v))p(v.[v))
15


Reviews:
[{"review_id": "cjP_wsA1Q6J", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes improved version of Scale-Space Flow (SSF) model. Authors add two enhancements to SSF for improve Rate-Distortion performance: (i) learnable scale transform and (ii) structured prior (SP). The paper also introduce a dataset for neural video compression that collected from youtube.com.\n\nPros: \n- Proposal seems better results than conventional SSF in some bit-rate and dataset conditions (Figure 3).\n- Publishing a new dataset may help the community.\n\nCons: \n- I think the introduction of scale parameter is the most important point of this paper. Authors says it acts as a gating mechanism. Figure 2 shows the case of an image, but no comparison with traditional SSF has been made. Figure 4 shows final RD-curve comparison from traditional SSF but I don't know how the gating mechanism works. For example, showing some kind of residual noise amount of conventional SSF (Agustsson et al., 2020) and show a reduce effect by scale parameter will make the claim of the paper credible.\n- There needs to be a more specific and clear explanation of how the SP is processed.\n- Figure 4 shows the effect of SP is that the image quality is better than SSF at high bitrates, but the image quality is worse at low bitrates. I would like to see a discussion on the reason for this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GZ5TYqNMAe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors focus on the problem of lossy video compression. To this end they propose the application of latent variable sequential generative models, specifically autoregressive flows to compress video streams. They evaluate variations of these models quantitatively including their own proposed version of scale space flow. They also introduce a new dataset named Youtube-NT and show promising quantitative performance.\n\nPros:\n\n1. The proposed problem of video compression has direct societal applications.\n\n2. The application of generative models, especially flow based models, to video compression is a novel and relatively underexplored topic in the community.\n\n3. Quantitative performance is promising.\n\n4. New dataset released will be useful for future work.\n\nCons:\n\n1. There ideally should be more qualitative evaluation. How do the reconstructions of all these approaches look side by side?\n\n2. The new dataset is fairly small by the standards of video datasets. For example, Kinetics and Youtube 8m have hundreds of thousands or even millions of examples.\n\nIn summary, the paper explores an interesting and novel application of flow based models to video compression. While I think the paper could be strengthened by more qualitative examples and a larger dataset, I think this could be a good contribution to the conference.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7-Z1EqFgza0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary\nIn this paper, the authors provide a new interpretation of existing video compression models. Their perspective is that a video decoder is a stochastic temporal autoregressive model with latent variables. The introduced latent variables could be either used for providing more expressive power for 1) motion estimation&compensation modeling and 2) residual noise modeling, which are two key components of traditional video codecs. The proposed method shows favorable results when the bitrate is higher than 0.12 bits per pixel on the public benchmarks.\n\n#### Strength\nThe proposed method has the following strengths.\n* The new interpretation of the video compression model gives a unified framework and insights. Although I am not entirely familiar with the learned video compression field, the interpretation looks quite interesting. It would be helpful for other researchers in the field.\n* New moderately high-resolution dataset is provided for training learned video compression models. This is a good contribution to the community.\n* The proposed model improves performance over the baselines. They propose two new components: 1) a learnable scaling transform for modeling motion estimation/compensation, and 2) structured priors for modeling the entropy.\n\nTo make the work stronger, I have a few suggestions as follows.\n* Study on the effect of using a pre-trained optical flow model. It seems that the parameters for motion estimation (g_w, f_w, w_t) are learned end-to-end. However, it might be challenging to learn motion estimation parameters end-to-end with other parameters. I am curious how much gain we can get if we plug in the pre-trained optical flow estimator. Intuitively it should give some boost, as one major component of the video codec is the motion estimator.\n* More failure mode analyses. The proposed models, STAT and STAT-SSF, show favorable results when the bitrate is high but underperforms baselines in low bitrate. It would be better to have some discussion about why the case is. \n* High latency extension. It would be interesting to see how the proposed framework can be extended for high-latency compression. In traditional video codecs, high latency compressions are also widely used to achieve high rate-distortion performance. However, in this work, only the frame t and t-1 are considered for compression. \n\n\n#### Minor comments\n* I am not sure the authors can cite papers in the abstract.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3qK18Rk2P6G", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a lossy video coding scheme using autoregressive generative models. \n+ It is good to see that the paper includes comprehensive reviews regarding neural network based compression schemes and autoregressive models.\n\n- The contribution has been depicted in aspects of 1) a new framework, 2) a new model, and 3) a new dataset. However, beside to 3), it is hard to see the other two aspects have significant novelty both in a neural video coding architecture (Agustsson et al. 2020) and in temporal autoregressive models, etc. \n\n- Furthermore, current video codecs (i.e. HEVC) are developed for YUV4:2:0 because the current broadcasting system has supported the video format. The evaluation should be conducted in YUV 4:2:0 domain for fair and reasonable comparisons.  \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommend acceptance. The authors have addressed several of the reviewers' concerns in their comments, conducted additional experiments, and updated the manuscript accordingly.

A concern was raised regarding the size of the dataset introduced and used by the authors for this work. However, I agree with the authors that it doesn't necessarily make sense to compare this to datasets designed for training video classification and/or generation models; In the compression setting, the quality of individual data points matters much more than their quantity, as the authors argue.

Reviewer 2 was curious about the potential of a pre-trained optical flow module. I believe the authors have convincingly argued that end-to-end learning is likely to be more effective and practical (and indeed, there is plenty of evidence for this in other ML contexts where training data is not scarce). I agree that a direct comparison in the paper would have been interesting, but this would constitute a significant investment of time and effort on the authors' part (as they also point out, training such a module separately could actually be more difficult), and I think it would be unreasonable to make this a condition for acceptance.
Decision: Accept (Poster)