Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper tackles the problem of efficient video generation. The authors present a Dual-Video-Discriminator Generative Adversarial Network (DVD-GAN) composed of an image-level spatial discriminator and video-level temporal discriminator. DVD-GAN achieves state-of-the-art results when benchmarked against the FID, IS and FVD quantitative metrics. Compared to previous video generation works, DVD-GAN is the first model to present compelling qualitative results on the UCF-101 and Kinetics-600 dataset.

Motivation:
Efficient video generation using GANs remains a significant challenge as it exacerbates all the issues associated with image generation using GANs. There are also increased memory and computation costs due to the 3D nature of video and the requirement for temporal modelling.

Main Contributions:
1. The authors propose to spatially down-sample the input into the video-level temporal discriminator. 
2. The authors propose to temporally sub-sample video frames for the image-level spatial discriminator

Secondary Contributions:
1. The authors set a benchmark for video generation on the Kinetics-600 dataset. This is significant due to the scale and complexity of this dataset in comparison to other datasets in the video generation literature.

----------------
Pros:
+ The paper is fairly well-written and clear
+ The paper recognizes and tackles a significant challenge in video generation when using GANs.
+ The ablation studies and experiments offer clarity with regard to the performance of the proposed model on different datasets and with different sampling strategies.
+ This paper presents compelling results for high-resolution (i.e. > 64x64) video generation on complex datasets.


Cons:
The main weakness of this paper is that it does not represent a significant advancement in our understanding of video generation using GANs. The novelty of DVD-GAN is also limited with respect to prior video GAN literature. DVD-GAN appears to be a straight-forward application of the BIG-GAN [5] family of models to video generation. Subsequent experiments and results are tailored for this particular model with little to no applicability to prior models or generalization. This is a significant weakness given that this paper's main contribution is a discriminator component for video GAN architectures.

Other Observations:
- Dual Video Discriminator GANs were introduced in MoCoGAN [1], the naming of this model may mislead readers into thinking that this is the first such discriminator architecture.
- Subsampling of the discriminator has already been explored in TGANv2 [2].

Points of Improvement:
- Given that the video discriminator is the main contribution, it should be benchmarked against previous video generator models in the literature such as TGAN [3], MoCoGAN [1] and TGANv2 [2]
- Given that the reproducibility of these results (at this moment in time) is difficult outside of a few organisations, can the authors provide a more thorough exposition of the efficiency aspects of their proposed discriminator. Currently, the best argument for the efficiency claims is the 58% reduction in pixels processed per video but what does this mean concretely, with respect to memory, computation and time efficiency?


Final notes:
- Authors should clear up the confusion with class-conditional vs unconditional UCF101 results by clearly labeling class-conditional vs unconditional results. Furthermore, they should provide a detailed section on how the UCF101 metrics were achieved, it is known that video GAN metrics are highly sensitive to even the decision of when to normalize inputs to the evaluation network (see Appendix B of [4])

- As impressive as the results are on Kinetics-600, they ultimately are dampened if they cannot be reproduced or built upon. The original Kinetics-600 dataset (based on youtube clips) is no longer available as youtube has either blocked, geo-blocked or removed a significant portion of this dataset. In light of this, do the authors plan to provide access to this dataset or should we place less significance on the Kinetics results?

-------

Current Review Decision:
Although this paper does present state-of-the-art results for all video generation datasets and carries out thorough experimentation to demonstrate this. The DVD-GAN model rides significantly on the backbone of the BIG-GAN model to achieve this. The Dual Video Discriminator by itself does not provide a novel contribution for a conference such as ICLR as it has already been proposed [1] and discriminator subsampling has been explored in different forms previously [2]. Crucially, the claims on efficiency are not sufficiently explored and it is unclear what they are and how they would translate to other video GAN models in the literature. As such, I lean towards rejecting this paper in its current form.


----------------
[1] - https://arxiv.org/abs/1707.04993
[2] - https://arxiv.org/abs/1811.09245
[3] - https://arxiv.org/abs/1611.06624
[4] - https://arxiv.org/abs/1909.12400
[5] - https://arxiv.org/abs/1809.11096

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes a class-conditional GAN model for video generation called DVD-GAN. The generator uses a single latent variable and uses ConvGRU modules and ResNet blocks to generate N frames. The model uses a dual discriminator, with one discriminator that discriminates individual frames, i.e. an image discriminator, and one that discriminates the whole video. This is similar to the MoCoGAN model, with the main difference being that the video discriminator operates on a smaller resolution video, thus reducing the dimensionality of the input to discriminate. The model is used to generate videos after being trained on the large-scale Kinetics-600 dataset, which contains multiple examples and has a lot of variability across videos. The main contribution of the paper is to successfully train this large GAN model on the very large-scale Kinetics dataset. The samples from the model are very visually appealing and are qualitatively  superior to any previous video prediction model.

While the paper mostly focuses on scaling up current models, it achieves significantly better qualitative results than previous models on a very challenging dataset, and therefore I believe it should be accepted as it is a significant advance in the field which probably will lead to follow-up work based on the model proposed here. 

However, there are a number of things that could be improved/minor comments:

- Further details about the generator should be included in the main body of the paper, only having a figure to describe its architecture is not enough when the model and how to scale it up are key contributions.
- The authors introduce a FID score for video which is similar to FVD, but FID is only used to report results in one experiment, while FVD is used for the rest of the experiments. Since the community uses FVD and there is a publicly available implementation of this metric, I'd suggest that the authors also include FVD scores in Table 1 to help reproduce the results. This is important since the FID metric is not explained thoroughly in the paper and small implementation details matter when using these metrics.
- The related work section is missing many references to video prediction models, please add them to the paper. Some examples include:
Decomposing motion and content for natural video sequence prediction. Villegas et al. ICLR 2017
Unsupervised Learning of Disentangled Representations from Video. Denton and Birodkar. NIPS 2017
Predrnn: Recurrent neural networks for predictive learning using spatiotemporal lstms. Wang et al. NIPS 2017 
PredRNN++, ContextVP, ...
- Additional metrics for the BAIR experiment, including LPIPS and SSIM. While FVD correlates well with human judgement, LPIPS does so as well and provides another evaluation of the model. Furthermore, metrics such as SSIM as used in SVG-LP can help better understand how well do the models cover the ground-truth sequence for given context frames.
- Qualitative results for the BAIR experiment. Since most current models are not trained on Kinetics, qualitative samples for the BAIR dataset would help to qualitatively compare current methods to DVD-GAN.
- In practice people have found that it is very difficult to train BigGAN-like models on images and videos. A common difficulty is that training diverges after a number of iterations, with the model starting to show mode collapse and big oscillations in terms of FID scores. Since the models are trained for a big number of iterations, have you observed these kind of issues with different hyperparameter configurations? If so, did you find any strategies to address it?
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
Summary:
This paper presents a method for training a generative adversarial network on high resolution videos and complex datasets. They propose decomposing the discriminator in Adversarial Networks into a spatial and temporal discriminator similar to previous works, however, the temporal discriminator downsamples the input using average pooling before forwarding it through the network. In experiments, the presented method outperforms previous state-of-the-art methods in the used metrics. In addition, the videos generated from the Kinetic-600 dataset in a non-conditional setting are the most realistic looking up to date. 


Pros:
+ Best generated video quality out there (In the Kinetics-600 dataset)
+ Numerically outperforms baselines in all datasets

Weaknesses / comments:
* What is the message of the paper?
- It is not clear to me whether this paper is about a new method or if it’s about an empirical study of the network size for video prediction algorithms. If the paper is about a new method, then the novelty is lacking. Decomposing the discriminator into space and time discriminators was originally proposed by MoCoGAN. Therefore, the only difference I can see is that the input to the time discriminator passes through a downsampling function (average pooling). On the other hand, if the paper is about showing that a bigger video model generates better video, the paper is lacking a more in depth evaluation. It seems that the size (frame resolution and number of frames) is only done on the Kinetics-600 dataset (Table 1). However, it is not clear to me whether there is a conclusion to be made from these results about the model size. The FID seems to favor smaller models and IS favors bigger models? But the last (biggest) model does not increase the IS? I am not sure what to make of these results. In addition, there is no such results on the other datasets (UCF101 and BAIR) or on the frame conditioned experiments for Kinetics-600. Do the authors have any comments or clarifications on this?


* Lacking proof that the network is not just overfitting to each class.
- It seems that the authors train separate models for each video class. The generated videos are the best I have seen in such complicated dataset. The authors claim that the fact that the Kinetics-600 dataset is large automatically removes the concern of overfitting. However, I would like to see  evidence that this is true. Can the authors do something like sample a video and find its nearest neighbor in the training data and compare them to see if the model is simply memorizing the training set? Something like this could serve as evidence that the network is not simply memorizing the training data.


* FVD Evaluation on Robot Push (marginal improvement based on FVD paper)
- In Table 3, the authors present quantitative comparisons for frame-conditioned video prediction with state-of-the-art methods. The authors of the FVD paper state the following: 

“In Figure 5 it can be seen that when the difference in FVD is smaller than 50, the agreement with human raters is close to random (but never worse).”

The difference between DVD-GAN and SAVP / Video Transformer is below 50. Does this mean that this result may not significant? It seems that basically the top 3 methods are performing similarly. Can the authors comment on this?


* Comparing 11 frame conditional generation with 16 frame unconditional generation (DVD-GAN)?
- In Table 4, the authors are comparing DVD-GAN-FP, Video Transformer and DVD-GAN. DVD-GAN-FP clearly outperforms Video Transformer. However, the authors make a comment about the DVD-GAN result being much better than the other two and that “the synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames.”. However, the DVD-GAN model generates 16 frames, and not 11 like the other 2. Is this comparison correct? Also the other 2 methods (DVD-GAN-FP and Video Transformer) are constrained by what they must predict given input frames, does this fact affect the comparison / conclusion by the authors?


* Claim of diversity but no evidence in the video prediction setting (frame conditional generation).
- The authors claim that their method generates diverse videos, however, there is no palpable evidence that this is true. Something like showing examples of different videos generated given the same condition should be good or per-frame evaluations like PSNR and SSIM with error bars of the N number of samples generated given the same condition could also serve as proof that the generation is diverse.


* Video results only provided for Kinetics-600
- It looks like the authors only provide actual video files for the un-conditional generative model for Kinetics-600. Could the authors provide videos of the frame conditioned experiments as well (UCF101, BAIR and Kinetics-600)?


* Argument in paper: “The synthesis model’s improved performance on this task seems to indicate that the advantage of being able to select videos to generate is greater than the advantage of having a ground truth distribution of starting frames”
- I do not fully agree with this argument. Optimally, a frame-conditional model should identify all objects/background observed in the input frame and object/background dynamics if multiple frames are given. Given the optimally identified factors, video generation/prediction should be much better or the same as a purely generative model. The fact that the experiments in this paper show that the conditional model is not as good as the non-conditional model is most likely due to the model and/or objective function and not the generation scenario. Do the authors have any comments on this?


Conclusion:
This paper does indeed present the best video generation up-to-date given the dataset difficulty. However, the paper itself has multiple issues as stated above. Please try to address these in the next revision.

review_title: Official Blind Review #1

Meta-review: This paper addresses the tasks of video generation and prediction and shows impressive results on the datasets such as Kinetics-600. There is a reviewer disagreement on this paper. AC can confirm that all three reviewers have read the rebuttal and have contributed to a long discussion. The reviewers have raised the following concerns that were viewed as critical issues when making the final decision: R1 and R3 expressed the concerns regarding limited technical novelty of the proposed approach in light of the prior works, e.g. MoCoGAN and TGANv2. R3 suggests, that the proposed method shows advantage that might be due to the large computational resources available to train the model. Providing a comparison of the proposed model and the relevant baselines on the Kinetics dataset is desirable to access the benefits of the proposed approach (R1). 
AC also agrees with the R2 about the potential impact this work could have in the community. However, given that the reviewers have raised important concerns and have given suggestions, the paper needs too many revisions for acceptance at this time. We hope the reviews are useful for improving and revising the paper.
Decision: Reject