Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This submission presents a conditional video synthesis model. This work is largely based on the recent progress of the Transformer-based autoregressive model [17] but with video extension. Differences over [17] are 1) learning-based flow module for temporal consistency, 2) conditioning for controllability, and 3) temporal embedding.

Overall, the submission contains a few technical novelties while being questionable whether significant, and the organization could be further improved.
Other than those, the results seem reasonable, and the analyses are well conducted.


=============================================

<After the rebuttal and discussion phase>

The authors' responses have addressed most of this reviewer's concerns, which increases my rating. 
The authors are strongly required to reflect all the rebuttal responses in the paper; especially missing reference, acknowledging the conventional works, the additional results, and the preliminary results with the SPADE version.
More discussion about the flow module is recommended.

main_review: 
Pros
- Video extension of [17] introducing flow module, controllability, and temporal embedding
- Simple and general conditioning method using Transformer (frame, sound and pose conditioning)
- Sufficient experiments and study. Nice ablation
- Good quantitative results

Cons
- Questionable effectiveness of the flow module in temporal consistency
- Paper organization and writing could be improved (the use of terminologies are confusing, and notations are abused)
- Hard to feel improvement over the state-of-the-art methods in qualitative results.


<Detail comments>

* Sound-conditioned video synthesis results: Multi-modal controllability is very interesting in generative modeling.
It would be more interesting to show the diverse synthesis results obtained from the fixed visual condition and different sound contexts (e.g., different drum playing sounds). This would allow to clearly understand behaviors and insights of the proposed method w.r.t. conditioning.

* Layout-conditioned video synthesis results: Compared to [40], the results obtained by the proposed method suffer from wobble effects and unstable temporal consistency.
Then, it is questionable what is the benefit of using the flow module.
This is concerning in that the flow-module is one of the main technical contributions and the most technical distinctive extension over [17], which seems to be neutralized by failing to show its effectiveness.

* BAIR results look interesting. The proposed method seems to understand physical interaction well to some extent.

* Missing reference: Flow-based feature warping modules have been used for temporal consistency, e.g., [Semantic Video CNNs through Representation Warping, ICCV 2017]. However, in Line 154, the references are missing, and the discussion about the history is misleading. 


<Writing issues>

* Terminologies are confusing: The authors interchangeably used {encoding, embedding, feature, code}, {annotation, condition, state}, etc. Those badly affect the readability of the submission.

* Flow module: By reading about the proposed flow module in the main paper, this reviewer failed to understand how the flow module is operated clearly, and found that the description in the supplementary material is much more specific and clear to understand.
The writing should be reorganized to convey the clear elaboration of the module.

* Fig. 3: The y-axis scale annotation is confusing (log_10). Is it 10 FPS or exp(10) FPS? 

* Notation abuse: Dimensions defined in the submission are confusingly described. The authors seem to intend this to appear to be general, but it disturbs understanding about the dimensions. Also, it leads to duplicated descriptions about dimensions in Sec. 3.1 
and Sec. 3.2, which can be much condensed. It would be more informative to bring some of the supplementary material contexts by reorganizing the descriptions to make more space.




* Question: Why should the flow inversion be approximated? This reviewer cannot see any reason to use this approximation rather than directly learning to extract backward flow.



limitations_and_societal_impact: * The authors argued a positive effect on energy consumption by faster training thanks to the compact code.
* Misuse case discussion is included, and the authors argued that the positive applications outweigh the potential ethical concerns.

The discussions sound reasonable.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for video synthesis that combines an encoder / decoder architecture that computes discrete representation of images with a transformer network that predicts those discrete representations into the future. The encoder and decoder are connected through a novel optical flow module that warps encoder features into the decoder in order to predict how objects will move. In experiments, the authors perform thorough ablations of the proposed method, and show that they outperform most baselines.
main_review: Strengths:
+ New feature level warping strategy for future video synthesis
+ Thorough ablations
+ Outperforms most baselines

Weaknesses:

- Flow module and latent variables:
In Figure 1, we can see that there are two sources of information that contribute to generating frame X_s. First source is Z_s and the second source is F. From observing the provided videos, it looks like the synthesized frames are mostly a result of warping which means that Z_s is potentially being completely ignored, and thus, there is no point in having it to begin with since F provides all necessary information. The authors claim that FVD should provide a measure of diversity since we are comparing the generative distribution and real distribution. However, since the entire dataset is used to compute this number, and not individual videos, then it is hard to claim diversity unless the generation scenario is unconditional (i.e., no encoder, so F becomes impossible to apply). Is it possible to provide diversity metrics for the conditioned scenarios? For example, sample multiple trajectories from the same condition and measure the distance between them. It would be good to provide this for all datasets, and not just the BAIR dataset, to make sure this applies to all experiments.


- How is the flow module learned?
It is not clear to me how the optical flow module is learned. Is this completely learned using the frame reconstruction signal? So, features from the encoder are warped into the decoder using the optical flow module and the reconstruction error signal flows through this operation?


- Visual comparisons against baselines (or best baseline)?
I was not able to find any visual comparisons between the proposed method and baselines. This would be good to have to see if/how the baselines fail where the proposed method succeeds.


- Thin experimental section except for BAIR experiments:
The experimental section presents a good amount of content for the BAIR experiments. However, the subsections for Kinetics and AudioSet-Drum are very thin. There are no real discussions or analysis of what was observed in these experiments. I understand that the page limits are usually not ideal, but this is where authors have to figure out how to organize the paper in a way that sections do not look too thin.


Conclusion:
The proposed method is able to generate good looking videos by combining discrete variables, warping, transformers and Adversarial objectives. Having said that, there are a few questions that I have about the method to get a good idea about the performance of the proposed method. My current score is slightly above borderline, but I am willing to increase my score depending on the author’s response.

limitations_and_societal_impact: They have discussed potential negative societal impact, but as far as I can tell there is no discussion of limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduced a conditional video prediction algorithm based on VQ-VAE [57] and Vision Transformer (ViT). Similar to VQ-VAE, the frame-level discrete representation is learned by autoencoder with vector quantization, whose (conditional) prior on the latent representation is modeled by the variants of ViT. The authors also incorporated a useful inductive bias for video prediction by warping features based on optical flow estimation, as in [63]. The practical impacts of the proposed method are demonstrated in various short-term video prediction tasks, showing competitive performance to the previous SOTAs.
main_review: Pros:
- This paper proposes a general video prediction framework that can incorporate various structured conditional signals. The experiment results show that indeed the generated results are reasonable across diverse conditioning signals.
- The performance of the proposed method seems to be superior/competitive to the previous SOTAs on video prediction. Thorough ablation studies were helpful to understand the contribution of the model.

Cons:
- The proposed method cleverly combined many “working ideas” from the previous methods. Although its motivation is sound and performance is decent, it (1) lacks novel technical contributions and (2) often misses proper citations. In terms of technical novelty, it follows the idea of combining VQ-VAE with the Transformer as in previous works (e.g., [46]). Although the authors improved its architecture by modifying both the Transformer and VQ-VAE autoencoders, the modifications are also largely based on the existing techniques (ViT [A1] for Transformer and feature warping [63] for autoencoder). In terms of citation, the paper misses many important citations, making readers unfamiliar with this domain difficult to assess its contribution. For instance, the paper misses citation to ViT [A1] although it largely borrows their architecture. The feature warping in Eq.(4) was also introduced without proper citation to its original paper ([63];  Eq.(4)). 

[A1] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, In ICLR, 2021

- In principle, the proposed method can generate diverse futures. It would be great if authors can provide some qualitative results of diverse synthesis results (images or videos) conditioned on the same input. The current supplementary file provides results only on different conditionings. 

- It would be interesting to see if the proposed ViT+VQ-VAE architecture has a strength in long-term generalization in testing (e.g., learned to generate 15 frames into the future and evaluated to generate 50 frames in testing). Some experiments/comparisons to the other methods would be helpful to get more insights.

- The current draft misses important implementation details in the main paper; for instance, it was really unclear to me how the flow estimator is trained (Ln 189-199). 

===== Post-rebuttal update ======

The authors addressed some of my major concerns in the rebuttal. I think the paper misses proper citations/discussions to the prior works (even if some of them are rather "conventional"). Although the technical contribution of this paper seems to be not significant, the results are promising. Overall, it seems to me that it is reasonable submission to NeurIPS. I mildly recommend the acceptance.

limitations_and_societal_impact: Yes.
review_title: null

Meta-review: The paper proposed a conditional video synthesis model based on the vision transformer architecture and the VQVAE architecture All the reviewers considered the paper above the bar. The rebuttal successfully answered several questions raised by the reviewers, with two reviewers upgraded the score to more positive ratings. Overall, the reviewers consider the paper a welcomed extension of the transformer plus VQVAE paradigm for video synthesis. The quantitative results were convincing. The meta-reviewer agrees with the assessment and would like to recommend its acceptance.
Decision: Accept (Poster)