Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: An interested outsider trying to exercise best judgment here.

Certainly interesting idea with great results. The way this paper approaches video synthesis is to decouple that into two steps: 1) learning motion (residuals between the latent codes of consecutive frames) from a sequence of past motions, and 2) synthesizing a new image using the learnt latent code. Would you agree with this layman's assessment?

Reasons to accept the paper:
* The decoupling idea is new and interesting from an interested outsider's perspective.
* Evaluation is pretty comprehensive and the results are decent.

I have a few clarification questions though.
* The related work section claims that "In this paper, we focus on generating realistic videos using manageable computational resources" Do you have results to show that your approach is computationally more efficient?
* Intuitively, why do you think future video frames can be predicted? That is, fundamentally, why do you think motion can be learnt?If I understand you correctly, you basically claim that given a sequence of motion from the training data, we should be able to learn the subsequent motion. Isn't it that there could be many kinds of subsequent motion given a sequence of past motions? I could understand how you are using it in the context of image synthesis, i.e., as long as the predicted motion can be used to generate realistic images you'd be fine with it, but precisely predicting the next frames seems to suggest that there is one fixed motion given the past motion sequence.
* The whole motion diversity thing feels quite hand-wavy and a bit empirical to me. It feels like a hack. Why is it that motion needs to be diverse? In fact, how could you precisely quantify diverse?
* Equation 4 (and the last paragraph of Section 3.1) shows that the motion generation is evaluated with the loss using the image generated from the image generator. Do you have a more direct way of quantifying the quality of the learnt motion? Very minor though.
review_title: Interesting idea decent results

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: **Summary**
This paper addresses the problem of video synthesis --- generating diverse, realistic videos. This paper's core idea is to leverage a fixed, pre-trained GAN model for image synthesis and train a motion generator to produce a sequence of latent vectors to generate image sequences (using the pretrained GAN and the generated latent vectors) are temporally coherent. The specific technical novelties lie in (1) predicting the motion residual and (2) adding contrastive image discriminator to ensure that generated contents in a video are similar. The paper provides an extensive set of experiments demonstrating the proposed method's effectiveness over the state-of-the-art video synthesis models.

**Strength**
+ The quantitative and visual results are extensive. The performance over existing methods across multiple datasets is significant. The paper also provides an ablation study (Table 4, 5) highlighting the importance of individual components.
+ The capability of cross-domain video synthesis could be beneficial, particularly when high-quality video datasets are difficult to collect.
+ The appendix and the supplementary material provide extensive details about the experimental settings and results that would help reproduce the results.


**Weakness**
- When discussing the difference over [Tulyakov et al. 2018], the paper states “…applies h_t as the motion code for the frame to be generated, while the content code is fixed for all frames. However, such a design requires a recurrent network to estimate the motion while preserving consistent content from the latent vector, … difficult to learn in practice”. I do not fully understand why this is the case. It would be clearer if the paper can explain why such a design causes difficulty in learning and why the proposed design could alleviate such problems.

- For motion diversity, why maximizing the mutual information between the hidden vector and the noise vector can prevent mode collapse?

- It seems to me that the proposed method can only handle 1) “subtle” motion, such as facial expressions and 2) short video sequences (e.g., 16 frames). One can see the problem in the synthesized results for UCF-101: inconsistent motion, changing color, or object disappearing over time. It would be interesting to videos with a longer duration (by running the LSTM over many time steps). 

In sum, this is a paper with an interesting idea and extensive experiments. While the results are still not perfect and seem to handle subtle motion, the quantitative and qualitative evaluation show clearly improved results over the previous state-of-the-art.  
review_title: Interesting approach and extensive experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This article proposes to retrofit a pretrained image generator for video generation. 

______
Pros:
- Novel objective formulation that controls for motion diversity, disentanglement and content matching.
- High resolution video generation at 1024x1024.
- Cross domain video generation (i.e. use pretrained image generator from one domain on the motion from another).
- Using a pretrained image encoder lowers the training cost and improves efficiency.
- Cost estimates for training the proposed models on commodity hardware are provided.




______
Cons:
- Prior art missing:
  - Proposed approach seems very related to DrNet and a discussion on this is not included (https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video)
  - Proposed approach is missing prior art on efficient video GAN training - (https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397)


_________
Questions:
- Are the LSTM encoder and decoder implemented in practice by the same neural network component?
  - If they are separate components, why not use just one LSTM for both encoding and decoding (the noise vector could be 0 for the first timestep)?
- How will such an approach handle occlusions, during video. A benchmark on the BAIR pushing dataset would be useful to help clarify this.
- "Additionally, each frame in the input video sequence is conditioned on the first frame, as it falls into the distribution of the pretrained image generator, for more stable training."
  - please clarify this statement
- How does this work relate to DrNET?
 - Unsupervised Learning of Disentangled Representations from Video - https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video
 - This is a very related piece of work and it should be included and discussed in the related works section. 
 - DrNET seems capable of challenging most the main claims made in this paper.
- How does the proposed approach perform on more complex datasets such as Kinetics-600?
  - It doesn't appear that there is much change in the morphology of video objects temporally, this can be seen on samples from more complex datasets such as UCF-101. 
- What is the baseline performance of the model trained on the UCF dataset of videos, if you do not do any temporal modelling but instead tile the first image temporally to make a video ?
  - I suspect that for more complex video with complex motion, there is little in the way of motion modelling happening and the proposed model is simply copying and jittering the initial image.
  - This should be investigated in the paper so as to ascertain the limits of this approach. 
 
_______
Missing References:
-  MDP-GAN: Markov Decision Process for Video Generation - https://openaccess.thecvf.com/content_ICCVW_2019/html/HVU/Yushchenko_Markov_Decision_Process_for_Video_Generation_ICCVW_2019_paper.html
  - Related prior art for video generation
- LDVD-GAN: Lower Dimensional Kernels for Video Discriminators - https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397
  - Related prior art for video generation
  - Relevant to discussion on high resolution video generation as it held the previous state-of-the-art at 512x512 video generation.
  - Table 1 is missing the LDVD-GAN result of 22.91 +/- .19
  - Relevant to the discussion on video GAN efficiency since LDVD-GAN is a single-GPU model.
- DrNET: Unsupervised Learning of Disentangled Representations from Video - https://papers.nips.cc/paper/7028-unsupervised-learning-of-disentangled-representations-from-video
  - Related prior art for video generation
  - Related prior art for proposed disentanglement and training approach.
_______
Overall this work tries to take on video modelling from a different angle with some success but also leaves many questions unanswered due to gaps in the experimental design and related work (discussed above). I will increase my score should these questions and the other concerns be addressed.


__________________________________________________________________________________________________________________________________________
**Updated review** following revisions:
- A5: For future reference, I believe the kinetics dataset is now downloadable from a google drive folder: https://github.com/activitynet/ActivityNet/issues/28#issuecomment-602838701

- A6: I am not convinced by your hypothesis in Appendix (Section E) as to why UCF-101 is so challenging. Surely if UCF-101 is not diverse enough, then a model based on stylegan should perform well on it. In-fact I would argue that it is the opposite, that the UCF-101 is very diverse for such a small dataset.
  - Observe that the datasets compared against UCF-101 in Table 7 are not that diverse, there is a dataset of just faces (FaceForensics), of just sky time-lapses and of just dogs. These are all uni-modal datasets. On the other hand, compare just a subset of the 101 classes in UCF-101; Horse Riding, Military Parade, Baseball Pitch, Billiards Shot, Brushing Teeth,...

  - I would argue that the limitation of your approach (and that of DVDGAN [Clark et al., 2019]) on this dataset stems from the fact that UCF-101 is a small but very diverse dataset. On average, just over 100 samples per class.


Overall, the authors have adequately addressed my questions and concerns. They also appear to have done so for all the other reviewers too.

My recommendation is to **accept** this work for publication to ICLR 2021.

I recommend that the authors open-source their code and pretrained models. 
review_title: You need more than a good image generator 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

This paper proposes a method to disentangle content and motion from videos for high-resolution video synthesis. The proposed method consists of a motion generator, pre-trained generator, image discriminator, and video discriminator. The motion generator predicts the latent motion trajectory z, which is residually updated over time. Then the image generator produces each individual frame from the motion trajectory. For training, five types of loss functions are combined. In experiments, video generation by the proposed method is performed on UCF-101, FaceForensics, and Sky time-Lapse datasets. Also, cross-domain video generation and more ablation studies were conducted to show the effectiveness of the proposed method.

Overall,  about this paper, I am leaning on the positive side. I summarize both the strength and weakness of this paper that I felt.

Strength

The main benefits of this paper are covering high-resolution video synthesis and disentangling motion and contents. The proposed method was tested on the various datasets and showed experiments about cross-domain video synthesis. Also, ablation studies were performed to check the effects of each loss function.


Weakness


As mentioned in the Introduction, the second desired property for generated video is temporal coherency.  The motion generator may be helpful to find a motion trajectory that makes a temporally consistent video. However, it is unclear whether the meaning of "temporarily constant video" means that the content of the video is consistent over frames in video or there is no flickering effect in the video. Also, there seems to be a lack of experiments to directly verify the temporal coherency.

About the paragraph “Motion Disentanglement” in section 3.1, how to decide variable “m” for PCA? Also, what is the reason why using motion residual is helpful to motion disentanglement?



For evaluation, is there a reason for using different evaluation metrics for each dataset? Also, for each dataset, methods for comparison are different from each other. Is there a special reason to use different methods for comparisons? For, UCF101, IS and FVD were used while FVD and ACD were used for the FaceForensis dataset. Also, FVD, PSNR, and SSIM were used for Sky Time-lapse dataset. It is better to conduct a comparative experiment with the same metric and the same compared methods for all datasets.


There are qualitative results in section4.2 about cross-domain video generation. Are there results that have been verified quantitatively?

review_title: Review on Paper751 

Meta-review: All the reviewers are positive about the paper; R2 and R3 voted for clear accept. Overall, all the reviewers feel that evolution is comprehensive and the results are decent. There is a novel objective formulation that controls for motion diversity, disentanglement and content matching, outperforming existing methods across multiple datasets. High-res videos at 1024x1024 are generated and there is cross-domain video generation. Many good questions were raised by the reviewers, and they were addressed in details in the rebuttal. In particular, the question about subtle motion and short video sequences was raised (which was the concern that the AC had). The AC agrees with the reviewers that the paper warrants a publication. Please address the questions raised by the reviewers in the final version.

Decision: Accept (Spotlight)