Scaling Autoregressive Video Models
Authors: Dirk Weissenborn
Oscar Täckström
Jakob Uszkoreit
Venue:  (2020)
Keywords: autoregressive models
video prediction
generative models
video generation

Abstract: Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models attempt to address these issues by combining sometimes complex, often video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple, autoregressive video generation models based on a three-dimensional self-attention mechanism achieve highly competitive results across multiple metrics on popular benchmark datasets for which they produce continuations of high fidelity and realism. Furthermore, we find that our models are capable of producing diverse and surprisingly realistic continuations on a subset of videos from Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. To our knowledge, this is the first promising application of video-generation models to videos of this complexity.

Published as a conference paper at ICLR 2020
SCALING AUTOREGRESSIVE VIDEO MODELS
Dirk Weissenborn*
Oscar Täckström Sana Labs oscar arcar@sanalabe com
Jakob Uszkoreit Google Research us usz0google.co com
Google Research
diwe@google.com
ABSTRACT
Due the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architec- tures, latent variable models, adversarial training and range of other methods. Despite their often high complexity, these approaches still fall short of generat- ing high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present re- sults from training models on Kinetics, large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, which include occasional realistic continuations encourage further research on comparatively complex, large scale datasets such as Kinetics.
show that conceptually simple autoregressive video
INTRODUCTION
Generative modeling of video holds promise for applications such as content creation, forecasting, transfer learning and model-based reinforeement learning (Srivastava al.] 2015 Carl Vondrick 2016 Oh et al. 2015 Kaiser et al. 2019). While recently there has been lot of progress on generative models for text, audio and images, video generation remains challenging. To some extent this is simply due to the large amount of data that needs to be produced. Autoregressive models suffer from this particularly in their generation speed. On the other hand, they have number of desirable attributes, such as their conceptual simplicity and tractable likelihood, which enables straightforward evaluation their ability to model the entire data distribution.
Moreover, recent results image generation by Menick & Kalchbrenner (2019) show that pixel- level autoregressive models are capable generating images with high fidelity. These findings motivate the question of how far one push such autoregressive models in the more general task of video generation when scaling recent advances in neural architectures to modern hardware accelerators.
In this work, we introduce a generalization of the Transformer architecture of Vaswani et al. (2017) using three-dimensional, block-local self-attention. In contrast to the block-local attention mecha- nism of Parmar et al. (2018), our formulation can be implemented efficiently on Tensor Processing 2017). To further reduce the memory footprint, we combine this with
Units, or TPUs (Jouppi a three-dimensional generalization of methods from Menick & Kalchbrenner (2019), who generate images as sequences of smaller, sub-scaled image slices.
Together, these techniques allow us to efficiently model videos as 3D volumes instead of sequences of still image frames, with direct interactions between representations of pixels across the spatial and temporal dimensions.
"Equal contribution.
Work done while Google Research.

---

Published as a conference paper at ICLR 2020
We obtain strong results popular benchmarks (Section 4.2 Appendix A) and produce high fidelity video continuations on the BAIR robot pushing dataset (Ebert et al. 2017) exhibiting plausible ob- ject interactions. Furthermore, model achieves almost 50% reduction in perplexity compared to prior work on autoregressive models on another robot pushing dataset.
Finally, we apply models down-sampled videos from the Kinetics-600 dataset (Carreira al., 2018) (Section 4.3). While modeling the full range of Kinetics-600 videos still poses a major chal- lenge, we see encouraging video continuations for a more limited subset, namely cooking videos. These feature camera movement, complex object interactions and still cover diverse subjects.
We hope that these initial results will encourage future video generation work to evaluate models on more challenging datasets such as Kinetics.
RELATED WORK
Our setup is closely related to that of Kalchbrenner et al. (2016), who extend work on pixel-level autoregressive image generation (van den Oord et al. 2016b] a) to videos. However, whereas they model the temporal and spatial dimensions separately with dilated convolutions and convolutional LSTMs, respectively, our model is conceptually simpler in that we do not make any distinction be- tween temporal and spatial dimensions and instead rely almost entirely multi-head self-attention (Vaswani et al. 2017) within the 3D video volume. For comparability, we provide results on Mov- ing MNIST and another robot pushing dataset (Finn et al. 2016a) which model achieves almost 50% reduction in perplexity (see Appendix A
One major drawback of autoregressive models is their notoriously slow generation speed. However, we believe that further research into (partially) parallelizing sampling (Stern et al. (0188) and future hardware accelerators will help alleviate this issue and eventually make autoregressive modeling viable solution even for extremely high- -igg-ddmeniional data such as videos.
To reduce the generally quadratic space complexity of the self-attention mechanism, we use block- local self-attention, generalizing the image generation approaches of Parmar et al. (2018) Chen al. (2018) to 3D volumes. In concurrent work. Child et al. 2019 instead use sparse attention after linearizing images to a sequence of pixels.
To further reduce memory requirements, we generalize sub-scaling (Menick & Kalchbrenner, 2019)
to video. An alternative approach is optionally hierarchical multi-scale generation, which has re- cently been explored for both image generation (Reed et 2017 De Fauw et al., 2019) as well as video generation (Mathieu et al. 2016).
Earlier work on video generation mostly focused on deterministic approaches (Srivastava et al.) 2015 Carl Vondrick 2016) Xingjian et al. 2015 Liu et al.) 2017: Jia et al. 2016), which fail to capture the high degree of stochasticity inherent in video. In response, a popular research direction has been that of generative latent-variable video models. In contrast to pixel-level autoregressive models, these posit an underlying latent process in tandem with the observed pixel values. Work in this category includes variants of variational autoencoders (Babaeizadeh al., 2018 Denton & Fergus 2018). To address the issues inherent in these models, most notably the tendency to generate blurry outputs possibly due to restricted modeling power, inadequate prior distributions, or optimization of lower bound in place of the true likelihood, various directions have been explored, including the use of adversarial objectives (Mathieu al., 2016; Vondrick et al., 2016 Lee al. 2018), hierarchical latent- variables (Castrejón et al., 2019), or flow-based models (Kumar et al., 2.19). All of these approaches admit significantly faster generation. However, in the adversarial case, they tend to only focus on subset of the modes in the empirical distribution while flow- based models struggle with limited modeling power even when using a large number of layers and parameters.
A large fraction of earlier work on video generation has encoded specific intuitions about videos, such as explicit modeling of motion (Finn et al. 6b Denton & Fergus 2018) or generation of optical flow (Päträucean et al. 2016). The conceptual simplicity of our model, however, is more in line with recent approaches to video classification that process videos by means of 3D convolutions
(Carreira & Zisserman 2017 Xie et al. 2018) or, similar to this work, spatiotemporal self-attention Girrhaar 2018))
2

---

Published as a conference paper at ICLR 2020
Figure Top: Illustration the subscale video transformer architecture and process flow. We incrementally generate 4 2 2 = 16 video slices. The video slices and their respective generation order are derived from sub- scaling. In each iteration, we first pro- cess the partially padded video (illus- trated for slice index (1, 0, 1), black means padding and gray means already generated or visible) by an encoder, the output of which is used as condition- ing for decoding the current video slice. After generating slice replace the respective padding the video with the generated output and repeat the process for the next slice. Bottom: Subscal- ing in 3D (best viewed in color). The 3D volume evenly divided by a given subscale factor, here = 24, 2), and the respective slices are extracted. The whole volume is generated by in- crementally predicting the individual, much smaller slices, starting at slice (00,0,0) (yellow), followed by 2(0,0.1) (green), 2(0,1.0) (red), etc., in raster- order.
Subseale Video Transformer
Blocks Self-attention
Layernorm
Layernorm
Conv3D (3x3x3)
Stride & (4,2.2)
ConvSD (3x3x3)
Puddead video (lark pund
Video alice
Repent a times
Sulscale Slices
aejunut
VIDEO TRANSFORMER
explicitly model videos represented as three-dimensional spatiotemporal volumes, without resorting to sequential lineariza- tion of the positions in the volume (Child et al. 2019p. This allows for maintaining spatial neigh- borhoods around positions, which is important as the large number of individual positions to be predicted in video requires limiting the receptive field of the self-attention mechanism to neigh- borhood around every position to avoid the quadratic blow-up memory consumption of naive fully-connected attention.
We generalize the one-dimensional Transformer (Vaswani al. 2017)
We model the distribution p(æ) over videos æ E RT*HHxxX,, with time, height, width and channel dimensions, respectively by means of pixel-channel level autoregressive factorization That is, the joint distribution over pixels is factorized into product of channel intensities for all Ne channels, for each of the N₂ = T И W pixels, with respect an ordering π over pixels:
N₂ N
p(z) = II
II A=
p(a o (
(1)
==0
The ordering π is given by combination of subscale- and raster-scan ordering, as detailed in 3.2
3.1 BLOCK-LOCAL SELF-ATEER ATTENTION
The attention mechanism of the original Transformer lets each element in a set of N, elements connect to every other element, via the fully-connected weighted adjacency (attention) matrix A E R.N,NN, with Ai representing attention weights from element to element j. Because grows quadratically with the number of elements becomes prohibitively large for objects such as videos, which typically consist of hundreds of thousands of pixels or more. Therefore, similar in spirit to Parmar et al. (2018), propose to use local self-attention by dividing a video into much smaller non-overlapping sub-volumes, or 3D blocks. We then apply self-attention separately within each block. This approach is conceptually simple and amenable to highly efficient implementation
In the following, denote general tensors by boldface lowercase letters and matrices by capital letters.
3

---

Published as a conference paper at ICLR 2020
TPUs, which enables to scale our models substantially while maintaining comparatively high training speed with only a modest sacrifice in expressive power.
The Video Transformer consists of multiple stacked self-attention layers. Each layer divides the overall video volume of shape (T' H, W) into smaller blocks of shape (t, h, w) of length n, = t h- w, and performs attention within each block independently. Given (flattened) block representation 2 E IR x of hidden size d as input, this amounts to:
(q. k, v] = layernorm(22 (
q. k. R.ydd Wyku E Rdx3d.
(2)
= softmax
qk /Vdo+B)
A.BER
(3)
attentionn2 = Av
The input is first projected to query, key and value representations (Eq. 2). The attention matrix A is then formed as the scaled dot-pro product between all query-key pairs adding a relative position bias B (Parikh et al. (016) (Eq, 3). The bias Bij is defined as the sum of per-dimension relative distance biases between element and j. along each of the time- and spatial dimensions. Finally, the values aggregated with respect to the attention weights (Eq. 4).
Following Vaswani et al. (2017). we concatenate the output of na parallel attention heads in each layer and project the result by a linear transformation ( before applying residual connection. Finally, the output of the multi-head self-attention layer is passed through another dense laver with ReLU activation, followed by final linear transformation and residual connection (Eq. 6p:
z |attention (z); attention,, (=)] Wp +
Wp R (Raa-da))xd
(5)
relu(layernorm (;) +
Tí, T2 Idddd
(6)
where overloading notation, attention(z) denotes the blockwise application of self-attention to z. Similar to Baevski & Auli (2019), we found that applying layer normalization before each block, rather than after each block as proposed by Vaswani et al. (2017), improves training.
Connectivity. Operating on 3D sub-volumes (blocks) of videos means that there is no direct in- formation exchange between blocks. However, this can be addressed by varying the block sizes between each layer. To achieve this, we define blocks that stretch over the entire extent of least single dimension in each layer. Following this procedure, we effectively connect all pixel positions in the encoder, but due to masking some dependencies are missed the decoder. How- in our experiments these did not produce any visible, systematic artifacts. We discuss missing dependencies and potential remedies in Appendix C
Efficiency. Running block-local self-attention is very efficient in practice the cost of splitting videos into blocks is negligible. The approach Parmar et al. (2018) uses overlapping 2D image blocks in each layer. We found this prohibitive as the required data copying is comparatively pensive. To avoid the need for overlaps to connect pixels across blocks, we simply vary block sizes between layers, which is highly efficient and, as our results show, works well practice.
3.2 SPATIOTEMPORAL SUBSCALING
Menick & Kalchbrenner 2019) recently proposed generating images as sequence of subscaled image slices. We similarly define a subscale factor s (St, 8h, 8w) which divides video into 8h, = (St 3h sw) sub-sampled videos (slices), each of resolution (T/st, H W/sw), as depicted in the bottom part of Figure The slices are generated order according to their respective offsets, Generating all slices one at a time in this way drastically reduces the number of pixels in memory to Np/s, which enables scaling our architectures by factor of 8. Each slice internally generated according to the raster-scan order. In the following we explain how slices generated and how they are conditioned on already decoded slices. An overview illustrated the upper part of Figure
such that we first generate slice ((0,0,0). then L((,,,1), up until slice fit.-1.ss-1....
The current slice (a,b.c) is generated conditioned the encoded pixels from pre- ceding slices as follows. First, we create a partially masked video, where only the pixels of preceding visible. The partially masked video is then embedded by concatenating the hot encoding of the discretized pixel intensities of each channel. Subsequently, 3D convolution with kernel size ls- (ki, K2, ka) and stride (the sub-scaling factor) results in an encoded video
Slice Encoder.
slices ((,,h.))
4

---

Published as a conference paper at ICLR 2020
of resolution (T/st, H/ss. w/sw). We apply convolution padding depending the current slice index (a, c). In particular, we pad with [[k]/]] - a, | 2| - b, | c). which "centers" the convolution kernel on the pixels of the current slice. Finally, each axis, as well as embeddings for the current slice index (a, b, c). to the output this strided convolution. The result is an initial encoder representation 20 (a.bb.c E RT/ x H/+a W//s xd=, where de is the embedding size. We can optionally condition on auxiliary information, such per-frame action values of robot arm, by concatenating this information the initial encoder representation.
add positional embeddings for
This representation is further transformed by linear projection to hidden size d, before being fed as input to stack of L block-local self attentio layers as described in [3.1 Each layer is parame- terized by a different block size and number of attention heads. The resulting output ((a,b,e) is used conditional input to the subscale slice decoder, which generates the pixels of the current slice (a, b, c).
Slice Decoder. The pixel values of the current slice (a,bbe)) predicted conditioned on the encoder representation = ((.bbe)) The decoder is almost identical to the encoder in structure, except for the use of masking in the decoder as defined by the generation order. First, we embed (a.bb.c) by summing N₂ channel embeddings of size de at every pixel, before applying a 3x3x3 masked convolution (van den Oord et al.] 2016a) on the embedded pixels, effectively representing each pixel by its already generated, immediate neighbors. Similar to the encoder, we add positional embeddings for the space- and time dimensions to the output of this masked convolution. As the encoder, this results in an initial decoder representation y(ab..)) E TR xt//
To condition on the encoder state, linear projection of ((0,b.o) is added to Vlab.e) and the resulting representation is fed through stack of L block-local self-attention layers, with masking, to produce a state ( on which the final channel predictions are conditioned.
3.3 CHANNEL PREDICTION & LOss FUNCTION.
The per-pixel channel intensities x (a.b.e) (we omit the slice index (a, b, e) in the following) for each channel A: < No are predicted by MLPs with single hidden layer (Eq. conditioned on the flattened final decoder state yd E Rd which itself conditioned on i(a,b.c) and hence on prior slices (aa.b,)))) one-hot vectors. Finally, the per video slice loss defined as the negative log-likelihood as in Eq.
well the preceding channels (x (=)))=1..A--1 for each pixel, encoded as
u [aaernorm (y²): oonehot (ax (ar)):... ;onehol (zx-1)) Uk
(7)
p (a le <k azi)) softmax (relu(u,)) ))
PE RaxN, UR E (R(d((N-1)-NN)))
(8)
Rpp 11 N Σ
f(x)= =
In Inp(
2r lnp xes))
(9)
We found that splitting the color channel values of the videos into coarse and fine bits helps slightly in terms of performance. Specifically, split the 3 x 8-bit RGB channels into 6 x 4-bit channels (N. = 6, N, = 16), such that the coarse bits all three channels are predicted before the fine bits. Furthermore, splitting channels this way at the input level considerably lowers memory footprint when encoding videos as onchot vectors on TPUs.
4 EXPERIMENTS
Below, we provide details on the model variants considered, our training setup and the evaluation metrics used. We focus our evaluation on the BAIR Robot Pushing and Kinetics datasets. Addi- tional results Moving MNIST and another robot pushing dataset are provided in Appendix for reference. Sample videos strips of each model and dataset can be found Appendix F and sample videos at wttptttt://tt //bit. l 1y/22b017F
5

---

Published as a conference paper at ICLR 2020
4.1 MODELS & SETUP
Unless specified otherwise, we model video slices of 4 frames with a spatial resolution of 32x32.
Both the encoder and decoder consist of 8 layers and have nearly identical structure, except for the use of masking in the decoder, as described in Section 3.2 We apply block-local self-attention with the following block sizes h, w). Layers 1-4: (4, 8, 4); (4, 4, 8); (1, 32, 4); and (1, 4, 32). Intuitively, layers and 2 are responsible for gathering temporal information whereas layers 3 and 4 gather spatial information of the entire frame. Layer 3 has access the entire height and layer 4 to the entire width of frame. The remaining 4 layers have the same block sizes, but in reverse order. However, as discussed in Appendix [B] this particular choice of block size ordering is not crucial. There are = 8 attention heads, each with hidden size du = 128. Our base models are trained with embedding size de = 128 and hidden size of d = 512 (46M parameters). Based on ablations in Appendix B we observed that increasing the hidden dimension is preferable to using deeper networks. Hence, we increase the hidden size to d = 2048 and use n. = 16 instead of heads for the last 4 encoder/decoder layers in our large models (373M parameters).
Models. To assess the effect of subscaling, we explore the following variants. These differ mainly in the subscaling factor 8 as well as the context kernel size k. defaulting to k =
Spatiotemporal Subscaling. The subscale video transformer with full spatiotemporal subscaling applies subscaling in every dimension. For instance, 16x64x64 video is subscaled by factors 8 = (4, 2, 2) to 16 slices of 4x32x32.
Spatial Subscaling. This model uses no temporal subscaling and only subscales individual frames to a resolution of 32x32. For instance, a 4x64x64 video is subscaled by factors s = (1, to slices of 4x32x32.
Single Frame. This model uses no subscaling. Instead, we here model an entire single frame at time, conditioned only the previous three frames to limit memory consumption. The model uses no actual subscaling. Instead, one can imagine a 16x64x64 video to be subscaled by factors (16, I, 1) 16 slices of 1x64x64 frames. The context kernel size is k (6, 1, 1) which means that we merely condition on a context of 3 past frames, as the current and future frames are always masked when the temporal subscaling factor equals the full video length. Self-attention blocks are adapted as follows: Layers 1-4: (1, 8, 16); (1, 16, 8); (1, 2. 64); (1, 64, 2). For the remaining 4 layers we use the same blocks, again in reverse order.
Training. All models are trained with RMSProp TTellmaan & Hinton 2012) with fixed learning rate of 2 10-5, decay of 0.95 and momentum of 0.9. We use a batch size of 64 video slices, not stated otherwise, and shuffle the slices to avoid having all slices in batch correspond to the same video. The smaller models are trained for 300K steps and the larger ones for 1M steps. No explicit regularization is applied as we could not observe any form of over-fitting Videos longer than the training resolution are cropped randomly in time to the defined training length. If not stated otherwise, models are conditioned on the first frame during training, which is achieved by masking the loss corresponding to this frame. In preliminary experiments, this gave a minor improvement over computing the training loss across all frames.
Intrinsic Evaluation. Most results are reported as bits per dimension (bits/dim), the average neg- ative lrobabibyyyylity assigned by the model per (RGB) channel, averaged across all pixels in the video. This corresponds directly to the loss optimized by the model. In all experiments, we condi- tion (prime) on a specified number of initial frames. The probbbbbabilssses corresponding these frames are excluded from this average.
Extrinsic Evaluation. Prior work mainly reported results on the peak signal-to-noise ratio (PSNR) and mean-structural similarity (SSIM) metrics (Wang et al.) 2004b). However, these metrics were developed for images and have serious flaws when applied videos (Wang et al. 2004a| Wang & Lij 2007 /Zhang et al. 2018 Lee et al.) 2018). Conceptually, PSNR has a strong preference for blurry videos as it is based on pixel-level mean squared error. Similarly, SSIM does not correlate well with perceptual quality either. For instance, variational autoencoders show very strong performance on this metric despite producing blurry videos (Lee et al. 2018)). Hence, we focus on the Fréchet Video Distance (FVD), which recently proposed by Unterthiner et al. (2018) as a qualitative metric sensitive to visual quality, temporal coherence and diversity of samples. This is the spatiotemporal counterpart to the Fréchet Inception Distance (Heusel et al.) 2017), replacing the ImageNet-trained

---

Published as a conference paper at ICLR 2020
Table 1: Quantitative results on BAIR Robot Pushing (left) and Kinetics (right).
Models
Bits/dim
FVD
FVD (Avg)
Single Frame
1.49
104=4
99-22
Models
Bita/dim
FVD
FVD (Avg)
Spatiotemp. Suh.
Spatial Sub.
1.57
1.53
111:4
106
108::
10642
Single Frame
Sputial Sub.
1.47
1.40
263-26
243-26
413+11
450+:5
Spatiotemp. Sub. (L)
94:22
96:12
Spatiotemp, Sub.
1.49
195=7
375
SV2P [1]
263
Single frume (1.)
1.14
207=
33
SAVP [2]*
116
Spatiotemp, Sub. (L)
1.19
170=5
316+12
VideaFlow [3]
1.87
(b) Kinetics. Bits/dim averaged actoss 15 subsequent Trames when priming with initial frame, FVD and unrolled average FVD scores when priming with frames. Best results in bold.
(a) BAIR Robot Pushing, Bitskdim averaged across 15 sabsequent frames when priming with initial frame, FVD and unrolled average
[VD scores. Best results in hold. Results from [Unterthierr et
2018). Results are strictly comparable (see text details).
[ Babgeizzadhh al. 2018), [2] et al. (018) [3] Kumar
2019
Inception network of the latter with an I3D Network trained on Kinetics, Despite sharing the known drawbacks of FID (Binkooski et al. 2018), FVD has shown correlate much stronger with human raters compared to both PSNR and SSIM (Unterthiner et al. (018)). We report the FVD of the first 16 frames, as well as the "unrolled" average FVD across all contiguous subsequences of 16 frames. In each case, we report the mean and standard deviation of 20 trials.
Sampling time. Sampling from autoregressive models is notoriously slow. However, because our decoders are not very deep (8 layers) we are able to sample a batch of four 30x64x64 videas in acceptable time (approx. 8 minutes) with our large models on a Nvidia Tesla V100. Though this might still be impractical we argue that further advances in parallel sampling strategies (Stern al., 2018) and future hardware will alleviate this disadvantage significantly.
4.2
BAIR ROBOT PUSHING
BAIR Robot Pushing (Ebert al. 2017) shows robotic arm pushing and grasping objects in box. It consists of roughly 40K training- and 256 test videos. We prime on the first frame for training and evaluation.
Empirical Results. All variants of the Video Transformer achieve strong results compared to prior work in terms of both intrinsic and extrinsic metrics. From Table Ta we see that the small models already reduce the perplexity in terms of bits/dim by almost 20% compared to the recently proposed VideoFlow model (Kumar et al. 2019) with our large model (L) reducing perplexity even further to a 25% improvement. Similar to Menick & Kalchbrenner 2019). we find that subscaling can have a slightly negative effect on bits/dim. In terms of perceptual quality, every incarnation of model obtains lower (better) FVD score compared to all models evaluated by Unterthiner et al. (2018), which notably includes adversarial networks with no guarantees of covering the full empirical distribution. These results are not strictly comparable, since prior work has used longer priming sequences of two (Babaeizadeh et al. 2018 Lee et al. (018)) or three (Kumar et al.) 2019) frames, whereas our models (to our disadvantage) see single prime frame. Note that we sample with temperature 0.9 for the extrinsic metrics as we observed improved qualitative results at this temperature on the validation set. This corresponds to mild form of mode dropping and is common practice to improve sampling quality. For fair comparison we also tweaked the "temperature" of SAVP by scaling the variance of its normal distribution when sampling. This, however, did not result in any improvements for FVD.
Further results on an earlier version of robot pushing (Finn et al. 2016a) and Moving MNIST (Sri- vastava et al. 2015) can be found in Appendix [A] for brevity. In summary, like Kalchbrenner et al. (2016), we match the lower bound on Moving MNIST while obtaining an almost 50% reduction in bits/dim on robotic pushing which demonstrates the superiority of our models against prior work autoregressive video modeling.
Qualitative Observations. All variants of our model reach similar quantitative results on these benchmarks and we observe no immediate differences in fidelity. However, there are some notable differences. First, whereas the spatiotemporal subscaling model is able to capture temporal depen-
7

---

Published as a conference paper at ICLR 2020
Unrolled FVD
Unrolled FVD
800
150
Spa
Saage frome
Speaswerp
Spatotampp quuditttnn (14
110
400
100
Prams offset
Frame octset
(a)
(b)
Figure 2: Unrolled FVD metries on BAIR Robot Pushing (left) and Kinetics (right).
dencies across up to 16 frames (given subscaling in time by a factor four), the remaining models can only capture dependencies across four frames. This can, for example, result in deformation of occluded objects (e.g., Figure of the Appendix). However, due to the simplicity of the benchmark datasets, this is not appropriately reflected in the metrics including better unrolled FVD curves for the single frame base model in Figure 2a Second, we observe that lowering the sampling tem- perature from 1.0 to 0.9 consistently improves results. Notably, spatiotemporal subscaling seems more robust to sampling ertors its performance decays less when sampling with temperature 1.0 (122 Avg. FVD) compared to the spatial subscaling (134=4) and single frame models (153+t7). We attribute this finding to the difference in generation order when spatiotemporal subscaling is employed as it predicts pixels over the entire extend of the 3D video volume early and thereby effectively anchors future predictions around these pixels. Finally, considering that our results on BAIR Robot Pushing in terms of FVD are on par with those between two ground-truth subsamples (Figure 4 of [Uterrhiner al. (2018). we may be approaching the limit of this benchmark. On the other hand, it could be that FVD suffers out-of-domain and is not sufficiently sensitive to long- range temporal dynamics, since is trained perform human action recognition, which is known to predominantly rely on local features (Carreira & Zisserman 2017: Xie et al. 2018).
4.3 KINETICS
Moving from a constrained to real world setting, we next apply our models to the Kinetics dataset (Kay et al. 2017), a large scale cecion-recognitionn dataset consisting of YouTube videos. Specifi- cally, we use Kinetics-600 -00,, which contains roughly 400K training videos ranging over 600 action classes (Carreira et al. 2018). We center-crop and down-sample each frame to 64x64 with a width-3 Lanczos filter and anti-aliasing.
We introduce a slight change to our setup by using a separate decoder for the first slice (0000,)) This decoder can be twice deep (16 instead of 8 layers) as the original subscale decoder, because it does not rely on any encoder. For all other slices we train a regular subscale model (8 layers in both encoder and decoder) before. Using a separate first-slice decoder means that there is no wasted encoder computation on the first slice and that there are additional parameters. Furthermore, for our large models we scale the batch size 256 by training in parallel on 128 TPU v3 instances for IM steps.
Empirical Results. Results for our base models are shown in the upper part of Table Tb In line with results on BAIR pushing, we find that the single frame model obtains better performance in terms of bits/dim. In contrast, we observe that the spatiotemporal subscaling model generates better and more robust video continuations which is reflected by its superior FVD scores. Our large models (L) show much stronger performance across the board (see lower half of Table Tb and Figures 2b). lowering the perplexity to 1.14 bits/dim for the single frame model. While the spatiotemporal subscaling model obtains slightly worse perplexity of 1.19 bits/dim, improves FVD to 170. Despite its good performance on bits/dim, even with temperature of 0.9, samples from the large single frame model are prone to instability and in many cases we observe color "explosions" (Figure [2 in the Appendix shows an example) which is reflected in its significantly higher FVD score. Although much less pronounced we observed such instability already when sampling with temperature 1.0 on BAIR pushing which clearly indicates the benefits of temporal subscaling for video generation.
8

---

Published as a conference paper at ICLR 2020
(a) Zoom
(b) Perspective
(c) Camera movement
roulag roulag
(d) Fire
(e) Scene change
(f) Object interaction
(g) Fingers
(h) Mouth closing
(i) Yawning
Figure 3: Selected Kinetics continuations from of 128 videos and 16 samples which showcase variety of
natural, sedeospe phenomena our model learns to generate. We used our large spatiotemporal subscaling model and prime generation with 5 frames (0-4) to include the first two frames in subscale order (0, 4). Samples are generated with temperature of 0.9. The examples depict frames 0, 5, 10 and 15.
Qualitative Observations. Figure 3 shows samples from cooking subset of Kinetics that de- scribe in Appendix EE] These are selected to showcase different aspects of real-world vidcos learned by the large spatiotemporal subscaling model. Figures Sa and [c demonstrate the model's ability to handle camera movement. We find that camera movement seems to be learned early in training, possibly since is a major source of uncertainty. This requires transforming pixels correctly while hallucinating new pixels at the edges. Similarly, object movement resulting, for instance, in a change of perspective is predicted quite well (Figure 3b). Highly stochastic motion such fire (Figure (3d] or steam is modeled surprisingly well. Videos in Kinetics sometimes contain scene changes and our model. too, occasionally generates videos with jumps to completely new scenes (Figure Be). Motion of human fingers and faces seems challenging to model. Nevertheless, in number of samples the model is able generate somewhat believable continuations as can seen Figures 3g 3h or 3i
These selected examples show only small subset of the interesting phenomena handled by the model and illustrate the sheer complexity involved in modeling this dataset. Appendix [P] we pro- vide multiple samples, primed with the same initial frames to illustrate the diversity of the generated samples.
Limitations. While obtained the occasional encouraging sampl, would like to point out that the diversity of Kinetics still poses a major challenge. Failure modes range from freezing movement object distortions to continuations that "wash out" entirely after few frames. We firmly beleive that yet larger datasets and/or models will be required to capture the complexity of even short clips from YouTube videos. With this work merely provide initial baseline, hoping highlight both the potential and the enormous room for improvement.
CONCLUSION
We presented an autoregressive model of videos based almost entirely on variant of block-local self-attention that can easily be implemented efficiently on TPUs. Combined with spatiotemporal subscaling, our models can be scaled up substantially while retaining the ability to capture longer range spatiotemporal dependencies.
Empirically, we obtain state-of-the-art results across a range of video generation benchmarks, while the scalability of our approach enables us to make an initial attempt at modeling videos of unusually high complexity and diversity found in the Kinetics dataset. Our models occasionally generate encouraging continuations, especially on a subset of cooking videos, yet we find modeling the full range of such videos clearly remains major challenge.

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
This work benefited from numerous conversations with Nal Kalchbrenner, as well as discussions with Jacob Menick, Mohammad Taghi Saffar and Niki Parmar. We would also like thank Chelsea Finn and Tom Kwiatkowski for thoughtful comments on an earlier draft.
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2018
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In ICLR, 2019.
Mikotaj Binkowski, Dougal J. Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In ICLR, 2018.
Antonio Torralba Carl Vondrick, Hamed Pirsiavash. Anticipating visual representations from unla- beled video. In CVPR, 2016.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
João Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv, abs/1 808.01340, 2018.
Lluís Castrejón, Nicolas Ballas, and Aaron Courville. Improved conditional vrnns for video predic- tion. arXiv, abs/ hats/1904.10 12165, 2019.
XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An improved autoregressiie generative model. In /CML, 2018.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. OpenAl Preprint, 2019.
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. In CVPR, 2019.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In JCML, 2018.
Frederik Ebert, Chelsea Finn, Alex Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Conference on Robot Learning (CoRL), 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, 2016a.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In NIPS, 2016b.
Rohit Girdhar, João Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. arXiv, abs/1812.02707.. 2018
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In NIPS, 2017.
Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks. In Advances in Neural Information Processing Systems, 2016.
Norman P. Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Ba- jwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, Rick Boyle, Pierre-luc Cantin, Clifford Chao, Chris Clark, Jeremy Coriell, Mike Daley, Matt Dau, Jeffrey Dean, Ben Gelb, Tara Vazir Ghaemmaghami, Rajendra Gottipati, William Gulland, Robert Hagmann, C. Richard Ho, Doug Hogberg, John Hu, Robert Hundt, Dan Hurt, Julian Ibarz, Aaron Jaffey, Alek Jaworski,
10

---

Published as a conference paper at ICLR 2020
Alexander Kaplan, Harshit Khaitan, Daniel Killebrew, Andy Koch, Naveen Kumar, Steve Lacy, James Laudon, James Law, Diemthu Le, Chris Leary, Zhuyuan Liu, Kyle Lucke, Alan Lundin, Gordon MacKean, Adriana Maggiore, Maire Mahony, Kieran Miller, Rahul Nagarajan, Ravi Narayanaswami, Ray Ni, Kathy Nix, Thomas Norrie, Mark Omernick, Narayana Penukonda, Andy Phelps, Jonathan Ross, Matt Ross, Amir Salek, Emad Samadiani, Chris Severn, Gregory Sizikov, Matthew Snelham, Jed Souter, Dan Steinberg, Andy Swing, Mercedes Tan, Gregory Thorson, Bo Tian, Horia Toma, Erick Tuttle, Vijay Vasudevan, Richard Walter, Walter Wang, Eric Wilcox, and Doe Hyun Yoon. In-datacenter performance analysis of tensor processing unit. In ISCA, 2017.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model-based reinforcement learning for atari. arXiv, 0oss/193.00031 2019.
Nal Kalchbrenner, Aäron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals, Alex Graves, and Koray Kavukcuoglu. Video pixel networks. ICML, 2016.
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Apostol Natsev, Mustafa Suleyman, and An- drew Zisserman. The kinetics human action video dataset. arXiv, abs/1705.06650, 2017.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Lau- rent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv, abs/ htbs/1903.01444, 2019.
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv, abs/1804.01523, 2018.
Ziwei Liu, Raymond A Ych, Xiaoou Tang, Yiming Liu, and Aseem Agarwala. Video frame synthesis using deep voxel flow. In ICCV, 2017
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. CVPR, 2017.
Michaël Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In ICLR, 2019.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditional video prediction using deep networks in atari games. In NIPS, 2015.
Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. EMNLP, 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, and Alexander Ku. Image transformer. In ICML, 2018.
Viorica Päträucean, Ankur Handa, and Roberto Cipolla. Spatio-temporal video autoencoder with differentiable memory. In ICLR (Workshop track), 2016.
Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gómez Colmenarejo, Ziyu Wang, Yutian Chen, Dan Belov, and Nando Freitas. Parallel multiscale autoregressive density estimation. In ICMI., 2017.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In ICML, 2015.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. In NeurlPS, 2018.
11

---

Published as a conference paper at ICLR 2020
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5 6.s-ros coursera: Neural networks for machine learning. University of Toronto, Technical Report, 2012.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv, abs/1812.01777, 2018.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. In NIPS, 2016a.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In ICML, 2016b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NIPS, 2016.
Jiheng Wang and Qiang Li. Video quality assessment using a statistical model human visual speed perception. Journal of the Optical Society of America. A, Optics, image science, and vision, 24 12:B61-9, 2007.
Jiheng Wang, Ligang Lu, and Alan C. Bovik. Video quality assessment based on structural distortion measurement. Sig, Proc.: Image Comm., 19, 2004a.
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4): 600-612, 2004b.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offss in video classification. In ECCV, 2018
SHI Xingjian, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In NIPS, 2015.
Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2020
Figure 4: Samples (showing every 5th frame horizontally) illustrating occlusion effects on BAIR Robot Pushing. Models without temporal subscaling (rows 3-4) fail on occlusions, whereas the model with temporal subscaling (row 2) correctly maintains objects from the ground truth video (row 1). Notice the green ball deformation on rows and and the hallucinated green ball on the right edge of row 3, which are caused by missing temporal dependencies across the duration of occlusion.
Table 2: Moving MNIST. Nats frame averaged across 10 subsequent frames when priming with 10 initial frames. Best results in bold. t The lower bound reported in (Kalchbrenner et al. 2016) is slightly higher than ours.
Models
Nats/Frame (1)
Single Frame
86.2
Spatial Subscaling
91.8
Spatiotemporal Subscaling
90.0
VPN Kalchbrenner al. 2016)
87.6
Lower bound
85.1 (86.3)1
A FURTHER BENCHMARKS
A.1 MOVING MNIST
Moving MNIST (Srivastava et al. 2015) consists of 100K training- and 10K validation/test videos of two handwritten digits from MNIST benchmark that move deterministically across the frame, crossing each other and bouncing off the borders. The partial occlusion of crossing digits makes this dataset challenging. To be comparable with Kalchbrenner et al. (2016), we use the first ten frames priming and predict the subsequent ten frames.
To allow direct comparison with Kalchbrenner et al. (2016), we change our loss to a "deterministic" loss (and derived nats-per-frame metric) which defined as: H(2 -E, In yi (1 2)) In(1 - ys). where 2; are the gray-scale targets between 0.0 and 1.0, and y are the predicted scalar intensities.
From Table 2 we find that like Kalchbrenner et al. (2016) our single frame prediction model (i.c., subscaling) virtually solves the task in the sense that almost matches the lower bound of the
13

---

Published as a conference paper at ICLR 2020
Table 3:
Ablation of syyee--arameee settings in terms of bits per dimension for models on 256 BAIR Robot Pushing validation videos. All models were primed on I frame and trained for 300K steps with a batch size of 64.
Layers
Heads
Hidden size
1.63
4
1.59
256
1.65
8
1.55
8
1.55
512
1.55
1.4816
1.51
1024
1.47
24
1.4524
1.47
2048
1.40
loss. However, this is not true for our subscaling models. Employing spatial subscaling on this task gives aliasing artifacts that make harder predict future frames. Although this finding is limited to Moving MNIST, it suggests that spatial subscaling can potentially hurt generation.
A.2 ROBOTIC PUSHING.
Robotic Pushing (Finn et al. 2016a was used in prior work on autoregressive video generation Kalchbrenner al. 2016). The videos show robotic arm pushing and grasping objects in box and there are roughly 50K training videos and 1500 test videos with seen and novel objects, respectively. Following prior work,
the initial two frames for priming and condition the robot arm action for each frame as described in Section We use the same setup as (Kalchbrenner al. 2016) with videos of twenty frames down-sampled 64x64 with Lanczos filter and anti- aliasing.
We report results to compare with prior work on autoregressive video generation by Kalchbrenner et al. (2016). who achieve 0.92 bits/dim (0.64 nats/dim) with 2 frames of priming on each of the test splits (one with objects seen during training and one with novel objects). We trained a large (2048 dimensional) spatiotemporal subscaling model which achieves 0.51 bits/dim on the subset with seen objects and 0.47 bits/dim on the subset with new objects, which corresponds to an almost 50% reduction in perplexity.
B
SWEEPS
Table shows the impact of different architectural settings. We see that the hidden size has the biggest impact followed by the number of layers and heads. This is an interesting as well as im- portant finding because inereasing the hidden size (wider networks) requires more parallel compute which modern Deep Learning hardware excels at. Computation time grows sub-linear, memory lin- and parameters partially quadratically. In contrast all of these aspects grow linearly with deep networks. For scaling up architectures depth is therefore not the preferred option as we suffer much more in terms of computation time while having less parameters.
In another experiment, we shuffle the arrangement of block sizes between layers and found that did not really matter, that is, all results were within 0.01 bits/dim. However, our setup had the best overall performance.
Finally, we tried sampling temperature 0.9 and 1.0 only on the BAIR Robot Pushing validation set and found that temperature 0.9 consistently gave more robust predictions and better results on all extrinsic metries.
C CONNECTIVITY IN BLOCK-LOCAL SELF-ATTENTION
Blind Spots. Varying block sizes between layers in block-local self-attention can efficiently con- nect every pixel with every other pixel when no masking is employed. If masking is employed to respect the generation order (as our slice decoder) block-local self attention produces "blind spots" which leads to independence assumptions. To exemplify these special cases, consider posi- tion (1, 0), the left pixel of the second frame, and direct predecessor in generation order (0 1, w 1), the bottom-right pixel of the first frame. The only way establish connection
14

---

Published as a conference paper at ICLR 2020
between these two positions is through direct connection, because masking prevents any indirect connection. Thus, there has to be one layer in which both of these pixels are in the same block. This block must least stretch over the entire extent of both width and height (i.e., the full frame) well as at least time steps. Running full self-attention in such blocks can easily become prohibitive for large h and w.
Remedies, There seems to be no simple solution that solves the problem of blind spots completely. However, we can make sure that local dependencies up to certain distance are all covered by increasing the kernel size of the initial, masked convolution in the decoder. It is also possible to combine block-local self-attention with its dual form, dilated self-attention in n dimensions which connects all pixels at the same relative position within their respective block with each other. Finally, we find that it is important to avoid blocks of small sizes any dimension (e.g., 1). That means, even if we stretch block to the full extent of one dimension it is important to define sizes at least larger than on all other dimensions to limit the number unconnected pixels.
On the other hand, the independence assumptions due to masking do not seem produce any systematic, visible artifacts in our samples. We believe this to be an interesting finding by itself as it shows that there is potential for parallelizing autoregressive video generation by systematically exploring further independence assumptions.
D ADDITIONAL FINDINGS
Below, we summarize some additional findings that may of interest to some readers:
We found that using blocks stretching across single time-foow/collmm- dimension, is substantially worse than using blocks that stretch at least to some extent in all directions. This is likely due the fact that future masking in the decoder imposes strong indepen- dence assumptions in this case, as discussed in Appendix C
We found that RMSProp with momentum converges significantly faster than ADAM, which tried with different learning rates and settings for 31 and (32-
We tried using continuous, rather than discretized one-hot, input channel representations, but this had an overall negative impact on both performance and sample quality.
We experimented with gating mechanism in Eq. B] such that the attention matrix A is masked elementwise with (1 I) to allow for not attending to any element, similar to sentinel attention (Lu al. 2017 However, this had no effect generation quality.
E KINETICS COOKING
We found that for many video-prefixes in Kinetics it is very hard for model to predict continu- ations. For instance, main objects in the videos are too small or movement is too fast which results in very blurry frames there is little to movement at all. Figure 13] shows some examples. Therefore, we created a subset of cooking videos that we found to exhibit these problems to lesser degree.
In particular we filtered videos whose label matched the following regular expression:
(baking barbequing /breading cooking cutting pancake vegetables)
meat | cake sandwi ch pizza sushi tea ppeeling fruit leggs salldd).
Note that we still train on the full Kinetics training set and only use the cooking set to showcase samples in some cases.
F SAMPLES
Figures show samples from spatiotemporal subscaling and large spatiotemporal subscaling models on BAIR Robot Pushing. Figures 6 illustrate the fidelity and realism of the generated samples, whereas Figures 7 and illustrate the diversity of samples.
15

---

Published as a conference paper at ICLR 2020
Figures 11 show samples from spatiotemporal subscaling model cooking videos for Kinetics-6ll 600, while Figure [2 depicts samples from the single frame model. In each case, we prime frames and sample the next 11 frames. Each figure shows 16 different samples from the same model. As can be seen, the model is able to generate diverse continuations while retaining fidelity. For the single frame model we observe strange color artifacts (exploding colors) which attribute to the standard, raster-scan generation order of this model.
16

---

Published as a conference paper at ICLR 2020
Figure 5: Samples of 30 future frames (showing every 4th frame) for 12 test videos with the spa-
tiotemporal subscaling model, using I prime frame and temperature 0.9 on BAIR Robot Pushing.
17

---

Published as a conference paper at ICLR 2020
Figure 6: Samples of 30 future frames (showing every 4th frame) for 12 test videos with the large spatiotemporal subscaling model, using prime frame and temperature 0.9 on BAIR Robot Pushing.
18

---

Published as a conference paper at ICLR 2020
Figure 7: 11 samples of 30 future frames (showing every 4th frame) for I test video (top row)
with the spatiotemporal subscaling model, using I prime frame and temperature 0.9 on BAIR Robot Pushing.
19

---

Published as a conference paper at ICLR 2020
Figure 8: 11 samples of 30 future frames (showing every 4th frame) for 1 test video (top row) with
the large spatiotemporal subscaling model, using prime frame and temperature 0.9 on BAIR Robot Pushing.
20

---

Published as a conference paper at ICLR 2020
Figure 9: Samples of 11 future frames from the spatiotemporal subscaling model with 5 prime frames on 64x64 Kinetics.
21

---

Published as a conference paper at ICLR 2020
Figure 10: Samples of 11 future frames from the spatiotemporal subscaling model with 5 prime frames on 64x64 Kinetics.
22

---

Published as a conference paper at ICLR 2020
Figure 11: Samples of 11 future frames from the spatiotemporal subscaling model with 5 prime frames on 64x64 Kinetics.
23

---

Published as a conference paper at ICLR 2020
Figure 12: Samples of future frames from the single frame model with 5 prime frames on 64x64 Kinetics exhibiting strange color artifacts.
(a) Blurry and fast camera movement.
(b) Blurry and fast camera movement.
(c) Very little movement.
(d) Very little movement and small objects.
Figure 13: Ground-truth (top) and 2 samples of 30 future frames (showing every 4th frame) demon- strating that random Kinetics videos do not always lend themselves as good prefixes for generating continuations.
24


Reviews:
[{"review_id": "ryeoIONqKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary \nThis papers presents a pixel-autoregressive model for video generation, in the spirit of VPN (Kalchbrenner’16). The proposed method uses video transformers and is made computationally efficient by extending block-local attention (Parmar’18, Chen’18) and sub-scaling (Menick’19) to 3D volumes. The block-local attention is separable, meaning that in theory it is possible to connect every two pixels through a sequence of block-local layers. However, for efficient parallelization implement via masking mechanism it is necessary to ignore certain connections, introducing independence assumptions.  The model is shown to substantially exceed state-of-the-art in terms of likelihood as well as quantitative and qualitative visual quality on several datasets, including the very challenging Kinetics-600. Interestingly, it is shown that the model with spatiotemporal subscaling is more robust to higher generation temperatures, which could imply robustness to accumulating errors. \n\nDecision\nThe paper proposes a well-motivated method backed by solid state-of-the-art results. I recommend accept.\n\nPros\n- The proposed method is relevant and well-motivated.\n- The experimental results are strong.\n\nCons\n- The paper novelty is somewhat limited as it is mostly a combination of previously existing techniques.\n- The paper does not provide code which makes the results not easily reproducible. I think a minimal example of the code should be provided that is trainable at least on a simple dataset.\n\nQuestions\n- No videos are provided. Please provide an (anonymous immutable) link to video results.\n- Strong aliasing artifacts can be seen in the supplement on the Kinetics data, such as vegetables becoming increasingly “blocky” as well as general cube-like aliasing artifacts in Fig. 9. This indicates that the introduced independence assumptions are likely hurting the video quality. The paper discusses this in the appendix C, stating that there seems to be no remedy for the independence assumptions that does not increase the computational cost. However, this is exactly the problem that latent variable models such as variational inference or normalizing flows are designed to address. Would a certain combination of latent variable models with the proposed autoregressive approach alleviate these issues?\n\nMinor comments\n- Contrary to the summary in the related work section, Kumar’19 does not use variational inference and operates purely on the normalizing flows technique. Similarly, Mathieu’16 and Vondrick’16 do not use variational inference either instead relying on adversarial techniques. The paper correctly states that Lee’18, Castrejon’19 use variational inference.\n- Figure 2 is never referred to in the text. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1gMc17TtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for scalable autoregressive models for video synthesis. Key to the approach is a form of 3D (2 space and one time) self-attention that operates in a softly local manner (through a bias on the attention weights that makes them tend to prefer nearby connections), and also limits its field of view to a specific 3D sub-\"block\" of video at each layer for scalability. They also propose a clever ordering for autoregressive synthesis of the video subsampling spatially and temporally to generate multiple slices that are synthesized autoregressively one after the other. Each of these ideas is individually close to ideas proposed elsewhere before in other forms, as the authors themselves acknowledge [Vaswani et al 2017, Parmar et al 2018, Parikh et al 2016, Menick et al 2019], but this paper does the important engineering work of selecting and combining these ideas in this specific video synthesis problem setting.\n\nResults on standard datasets for video generation match up to and/or surpass prior methods, in line with prior work on autoregressive image generation that has been shown to do very well on similar metrics (perplexity and FID). What is perhaps more interesting is that this paper presents initial promising results for open-world Youtube video settings (Kinetics dataset) that have not been evaluated systematically in any prior work in this area, to my knowledge.\n\nThe downsides of this paper are largely common to this method class (autoregressive generative models): training time (one of their models is \"trained in parallel on 128 TPU v3 instances for 1M steps\"), inference time (four short 64x64 video clips of 30 frames take 8 mins to generate on a Tesla V100), and model sizes (373M parameters for the Kinetics model). However, this does not take away from the contributions made here, that make it possible at all to train an autoregressive model of this size. \n\nOn the experiments, some questions, comments, and suggestions that the authors might consider addressing:\n- How well do methods like SVG, SAVP, SV2P do on Kinetics, for comparison? It would be still more interesting if those models were scaled to have similar sizes to the large model in this work. While these methods have never been evaluated before on such unconstrained data, it is not clearly established that they do not work at all.\n- To what extent does the blocking help, and when does it breaks down? e.g. how many layers/how large do blocks have to be for the idea of using different block sizes to suffice for smooth video synthesis? What happens when the blocking idea is not used at all?\n- Other choices that aren't ablated in experiments: the choice of a local preference using the bias term in attention, the Transformer-style multi-attention heads. I do understand that these models are expensive to train and evaluate, but perhaps a smaller dataset might still suffice to demonstrate the value of these choices.\n- Why is the proposed approach evaluated only on video prediction? Could it not be used for video generation without conditioning or with class conditioning?\n- It is surprising to me that the perplexity of Kinetics models is lower than BAIR. Is there a reasonable explanation?\n\nWriting and presentation are good for the most part, despite the main paper being dense with details and multiple fairly involved ideas. I particularly enjoyed parts of related work, the illustration of slicing in Fig 1, and the illustrative examples in Fig 3. \n\nI would suggest however, that the paper might benefit from placing Sec 3.2 which describes the framework, before Sec 3.1. Fig 1 also belongs closer to Sec 3.2 anyway.\n\nThere are also terms/phrases I don't understand despite being reasonably familiar with the field like \"positional embbeddings\" (Sec 3.2). I also don't understand the need for \"one-hot encoding of the discretized pixel intensities\" (in that same paragraph). As a more minor comment, a footnote 1 before Eq 1 declares that capital letters denote matrices right before using capital letters to denote constants (T, H, W etc.). ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1g90yoRtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes an autoregressive video generation model, which is based on a newly proposed three-dimensional self-attention mechanism. It generalizes the Transformer architecture of Vaswani et al. (2017) to spatiotemporal data (video). The original Transform implies self-attention among different words in a sentence. Considering the larger scale of video, this work proposes to divide it into small blocks, and apply the self-attention (part of block-local self-attention modular) on each block. At the same time, it addresses the information exchange between blocks problem, by spatiotemporal sub-scaling (described in section 3.2). The proposed method achieves competitive results across multiple metrics on popular benchmark datasets (BAIR Robot Pushing and KINETICS), for which they produce continuations of high fidelity.\n\nSome questions:\n-      The proposed model is claimed to work on competitive results across multiple metrics on popular benchmark datasets. However, it only compares with stat-of-the-art models on BAIR Robot Pushing dataset (for the other dataset, the author only compares with the variations of the proposed model). Further, the author only reports the result of Bits/dim and FVD, instead of PSNR and SSIM, which are reported in the original papers. Any justification for this? Though FVD has its own advantages, showing PSNR and/or SSIM at the same time would help us get better sense of the performance. \n-      In table 1 (left) in section 4.2, the author mentions that results from all the stat-of-the-art models are not strictly comparable, since prior works use longer priming sequences of two or three frames, whereas the proposed models only observe a single prime frame. I am confused of why the proposed model can only see a single frame.\n-      The proposed block-local self-attention modular works on divideding video into small blocks, which seems to be a matrix of 3 or 4 dimensions (t,h,w,c). However, in the experiment, the input of the model for the BAIR Robot Pushing dataset is the first frame. How can this frame be feed into the block-local self-attention modular?\n-     In section 3.3, it splits the 3x8-bit RGB channel into 6x4-bit channels. It would be better if the author can show an example and clarify the advantages.\n-     In section 3.3, U_k, N_v and P seems not defined in the context.\n-    Videos are divided into small blocks and feed into the block-local self-attention modular separately. Then, I’m confused on how to aggregate these different blocks together to predict future frames.\n\nI would like to raise up my score if the author can address my questions.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach for scalable autoregressive video generation based on a three-dimensional self-attention mechanism. As rightly pointed out by R3, the proposed approach ’is individually close to ideas proposed elsewhere before in other forms ... but this paper does the important engineering work of selecting and combining these ideas in this specific video synthesis problem setting.’ 
The proposed method is relevant and well-motivated, and the experimental results are strong. All reviewers agree that experiments on the Kinetics dataset are particularly appealing. In the initial evaluation, the reviewers have raised several concerns such as performance metrics, ablation study, training time comparison, empirical evaluation of the baseline methods on Kinetics, that were addressed by the authors in the rebuttal. 
In conclusion, all three reviewers were convinced by the author’s rebuttal, and AC recommends acceptance of this paper – congratulations to the authors!
Decision: Accept (Spotlight)