Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work proposes an autoregressive video generation model, which is based on a newly proposed three-dimensional self-attention mechanism. It generalizes the Transformer architecture of Vaswani et al. (2017) to spatiotemporal data (video). The original Transform implies self-attention among different words in a sentence. Considering the larger scale of video, this work proposes to divide it into small blocks, and apply the self-attention (part of block-local self-attention modular) on each block. At the same time, it addresses the information exchange between blocks problem, by spatiotemporal sub-scaling (described in section 3.2). The proposed method achieves competitive results across multiple metrics on popular benchmark datasets (BAIR Robot Pushing and KINETICS), for which they produce continuations of high fidelity.

Some questions:
-      The proposed model is claimed to work on competitive results across multiple metrics on popular benchmark datasets. However, it only compares with stat-of-the-art models on BAIR Robot Pushing dataset (for the other dataset, the author only compares with the variations of the proposed model). Further, the author only reports the result of Bits/dim and FVD, instead of PSNR and SSIM, which are reported in the original papers. Any justification for this? Though FVD has its own advantages, showing PSNR and/or SSIM at the same time would help us get better sense of the performance. 
-      In table 1 (left) in section 4.2, the author mentions that results from all the stat-of-the-art models are not strictly comparable, since prior works use longer priming sequences of two or three frames, whereas the proposed models only observe a single prime frame. I am confused of why the proposed model can only see a single frame.
-      The proposed block-local self-attention modular works on divideding video into small blocks, which seems to be a matrix of 3 or 4 dimensions (t,h,w,c). However, in the experiment, the input of the model for the BAIR Robot Pushing dataset is the first frame. How can this frame be feed into the block-local self-attention modular?
-     In section 3.3, it splits the 3x8-bit RGB channel into 6x4-bit channels. It would be better if the author can show an example and clarify the advantages.
-     In section 3.3, U_k, N_v and P seems not defined in the context.
-    Videos are divided into small blocks and feed into the block-local self-attention modular separately. Then, I’m confused on how to aggregate these different blocks together to predict future frames.

I would like to raise up my score if the author can address my questions.

review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper presents an approach for scalable autoregressive models for video synthesis. Key to the approach is a form of 3D (2 space and one time) self-attention that operates in a softly local manner (through a bias on the attention weights that makes them tend to prefer nearby connections), and also limits its field of view to a specific 3D sub-"block" of video at each layer for scalability. They also propose a clever ordering for autoregressive synthesis of the video subsampling spatially and temporally to generate multiple slices that are synthesized autoregressively one after the other. Each of these ideas is individually close to ideas proposed elsewhere before in other forms, as the authors themselves acknowledge [Vaswani et al 2017, Parmar et al 2018, Parikh et al 2016, Menick et al 2019], but this paper does the important engineering work of selecting and combining these ideas in this specific video synthesis problem setting.

Results on standard datasets for video generation match up to and/or surpass prior methods, in line with prior work on autoregressive image generation that has been shown to do very well on similar metrics (perplexity and FID). What is perhaps more interesting is that this paper presents initial promising results for open-world Youtube video settings (Kinetics dataset) that have not been evaluated systematically in any prior work in this area, to my knowledge.

The downsides of this paper are largely common to this method class (autoregressive generative models): training time (one of their models is "trained in parallel on 128 TPU v3 instances for 1M steps"), inference time (four short 64x64 video clips of 30 frames take 8 mins to generate on a Tesla V100), and model sizes (373M parameters for the Kinetics model). However, this does not take away from the contributions made here, that make it possible at all to train an autoregressive model of this size. 

On the experiments, some questions, comments, and suggestions that the authors might consider addressing:
- How well do methods like SVG, SAVP, SV2P do on Kinetics, for comparison? It would be still more interesting if those models were scaled to have similar sizes to the large model in this work. While these methods have never been evaluated before on such unconstrained data, it is not clearly established that they do not work at all.
- To what extent does the blocking help, and when does it breaks down? e.g. how many layers/how large do blocks have to be for the idea of using different block sizes to suffice for smooth video synthesis? What happens when the blocking idea is not used at all?
- Other choices that aren't ablated in experiments: the choice of a local preference using the bias term in attention, the Transformer-style multi-attention heads. I do understand that these models are expensive to train and evaluate, but perhaps a smaller dataset might still suffice to demonstrate the value of these choices.
- Why is the proposed approach evaluated only on video prediction? Could it not be used for video generation without conditioning or with class conditioning?
- It is surprising to me that the perplexity of Kinetics models is lower than BAIR. Is there a reasonable explanation?

Writing and presentation are good for the most part, despite the main paper being dense with details and multiple fairly involved ideas. I particularly enjoyed parts of related work, the illustration of slicing in Fig 1, and the illustrative examples in Fig 3. 

I would suggest however, that the paper might benefit from placing Sec 3.2 which describes the framework, before Sec 3.1. Fig 1 also belongs closer to Sec 3.2 anyway.

There are also terms/phrases I don't understand despite being reasonably familiar with the field like "positional embbeddings" (Sec 3.2). I also don't understand the need for "one-hot encoding of the discretized pixel intensities" (in that same paragraph). As a more minor comment, a footnote 1 before Eq 1 declares that capital letters denote matrices right before using capital letters to denote constants (T, H, W etc.). 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
Summary 
This papers presents a pixel-autoregressive model for video generation, in the spirit of VPN (Kalchbrenner’16). The proposed method uses video transformers and is made computationally efficient by extending block-local attention (Parmar’18, Chen’18) and sub-scaling (Menick’19) to 3D volumes. The block-local attention is separable, meaning that in theory it is possible to connect every two pixels through a sequence of block-local layers. However, for efficient parallelization implement via masking mechanism it is necessary to ignore certain connections, introducing independence assumptions.  The model is shown to substantially exceed state-of-the-art in terms of likelihood as well as quantitative and qualitative visual quality on several datasets, including the very challenging Kinetics-600. Interestingly, it is shown that the model with spatiotemporal subscaling is more robust to higher generation temperatures, which could imply robustness to accumulating errors. 

Decision
The paper proposes a well-motivated method backed by solid state-of-the-art results. I recommend accept.

Pros
- The proposed method is relevant and well-motivated.
- The experimental results are strong.

Cons
- The paper novelty is somewhat limited as it is mostly a combination of previously existing techniques.
- The paper does not provide code which makes the results not easily reproducible. I think a minimal example of the code should be provided that is trainable at least on a simple dataset.

Questions
- No videos are provided. Please provide an (anonymous immutable) link to video results.
- Strong aliasing artifacts can be seen in the supplement on the Kinetics data, such as vegetables becoming increasingly “blocky” as well as general cube-like aliasing artifacts in Fig. 9. This indicates that the introduced independence assumptions are likely hurting the video quality. The paper discusses this in the appendix C, stating that there seems to be no remedy for the independence assumptions that does not increase the computational cost. However, this is exactly the problem that latent variable models such as variational inference or normalizing flows are designed to address. Would a certain combination of latent variable models with the proposed autoregressive approach alleviate these issues?

Minor comments
- Contrary to the summary in the related work section, Kumar’19 does not use variational inference and operates purely on the normalizing flows technique. Similarly, Mathieu’16 and Vondrick’16 do not use variational inference either instead relying on adversarial techniques. The paper correctly states that Lee’18, Castrejon’19 use variational inference.
- Figure 2 is never referred to in the text. 

review_title: Official Blind Review #1

Meta-review: This paper presents an approach for scalable autoregressive video generation based on a three-dimensional self-attention mechanism. As rightly pointed out by R3, the proposed approach ’is individually close to ideas proposed elsewhere before in other forms ... but this paper does the important engineering work of selecting and combining these ideas in this specific video synthesis problem setting.’ 
The proposed method is relevant and well-motivated, and the experimental results are strong. All reviewers agree that experiments on the Kinetics dataset are particularly appealing. In the initial evaluation, the reviewers have raised several concerns such as performance metrics, ablation study, training time comparison, empirical evaluation of the baseline methods on Kinetics, that were addressed by the authors in the rebuttal. 
In conclusion, all three reviewers were convinced by the author’s rebuttal, and AC recommends acceptance of this paper – congratulations to the authors!
Decision: Accept (Spotlight)