Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: **Paper Contributions**

The paper proposes SSW-GAN, an adversarial generative model of video which proposes a new generator architecture along with splitting the training into multiple stages.

**Strong points of the paper**

* The results are very strong.
* Prior work has focused on efficient decomposition of the discriminator, this work focuses on decomposition of the generator (effectively), and this is an extremely reasonable direction to take adversarial video research in.
* The claim that this requires substantially less computational cost is grounded.

**Weak points of the paper**

* A major departure from prior work is training stage 1 of SSW-GAN separate of stage 2, but this is independent of the computational benefits present from the generator architecture innovations. It is missing an important ablation showing the results of training the stages jointly.
* The generator architectures changes from prior work are quite concrete, but the high level description doesn't clearly reflect that.
* Some of the comparisons and model descriptions are lacking details, which make it difficult to understand experiments.

**Clearly state your recommendation (accept or reject) with one or two key reasons for this choice.**

I believe this paper is an accept (7), however I think there are a number of places where the description of the architecture and experiments needs more detail, and for my final rating I would like to see these addressed in rebuttal. 

Furthermore, I think there is a key experiment missing. Were that to be added I would strongly consider moving to clear accept (8).

**Supporting arguments for your recommendation.**

There are two key innovations in the SSW-GAN model:
* A generator architecture which provides output to discriminators that does not require an entire high-resolution full-length sample to be generated (because between stage 1 and 2 you select only a window of frames to run stage 2 on).
* Splitting the training into two phases, where at first you only train stage 1 and then you train stage 2.

These are very interesting ideas, since either one substantially reduces the training-cost of the video model (something aptly described in the paper), and not something strongly touched on by previous work. In addition, the metric scores of the SSW-GAN model are very nice, state of the art in almost all cases. For that reason I think the work in this paper is worthy of acceptance.

However there are a couple points of clarification and increased description which I think are needed, and I would like to see the following points addressed in the rebuttal:

* The core idea of judging fixed-length output upscaling as a generative problem seems like a very clear and reasonable idea. Most of the high-level description of the most omits this in place of a general “multi stage definition” of the SSW-GAN model. I think the introduction and abstract would benefit about being more clear with regards to this change.

* In section 4, it is unclear if all four bold sections are trained independently or not. I think it would be good to be clear in the paper-structure which components are trained together.

* In general, the paper is not super clear where upsampling (both in time and space) occurs. I think it would be good to describe this in the paper text and in the architecture figures.

* I am not super clear on the comparisons between SSW-GAN and DVD-GAN in “comparison with prior work”. When you say "our model trained to generate 128x128/12 videos “ is this the model which had a first stage trained on 32s32/25 and then you trained the second stage on input windows of 6 frames, generating 128x128/12, and took just single samples from that to compare?

* Similarly, when you say “However, our model is only trained on 128x128/12 outputs, as it is unrolled and applied convolutionally over the first stage output to generate 48 frames.”, isn’t it the case that the first stage is trained on longer sequences?

*  In Table 1, the numbers for DVD-GAN seem lifted from the paper, which I believe is using a Kinetics-600 trained I3D for metric calculation. This means the FID numbers are comparable, but in section 5 when describing IS you say you are using a Kinetics-400 trained I3D (like in FVD). Is this correct (in which case the numbers are not quite comparable) or is this is just mis-explained, and all numbers in Table 1 come from Kinetics-600 trained I3Ds?

* I think the IS/FID metrics you pick in Table 1 are the better metrics, but could you also add FVD numbers? That would allow you to compare against TriVD-GAN [1], which outperforms DVD-GAN. I think this is important because that paper also discusses modifications which reduce the memory requirement of DVD-GAN.

Finally, there is a major question this paper does not address: **is the two-stage training of SSW-GAN necessary, or can it be trained in a single pass** (but with the generator decomposition as described)? Doing so would be simpler, and also might potentially reduce the need for the Matching Discriminator, which the paper contains an ablation for, but I think needs a further ablation when the model is not trained in two stages.

I believe a key experiment would be training an SSW-GAN architecture but in a single pass, the results of that experiment would mean quite a lot for interpreting the changes described in this paper. In particular, the paper's title and abstract focus on the multi-stage aspect of the model, but skim over the substantial change of making the generator architecture more modular and scalable. I think it would be very beneficial to understand how each of these changes independently effect the performance of the model.

I do not think this ablation is required to maintain an accept rating, but including it would push my rating closer to clear accept, and be quite a strong addition to the paper's content.
 
[1] https://arxiv.org/abs/2003.04035

review_title: Official Review of SSW-GAN: Scalable Stage-wise Training of Video GANs

final_score: rating: 3: Clear rejection
review_content: Summary:
The paper proposes a stage-wise training pipeline for training 128x128 resolution videos of up to 100 frames. It starts by generating low resolution and temporally downsampled videos, and upsample the results in a stage-wise manner. Experimental results on Kinetics-600 and BDD100K demonstrate that the network is effective in generating higher resolution videos.

Strengths:
The idea is easy to understand. The paper is well written and easy to follow. Quantitative results show that the proposed method is superior than existing methods under some circumstances.

Weaknesses:
1.	The novelty is very low. Stage-wise and progressive training have been proposed for such a long time, they have been used everywhere. The way the authors use them don’t really exhibit anything novel to me.
2.	The resolution of the outputs (128x128) is lower than prior works (e.g. DVD-GAN has 256x256 outputs). Since the paper claims the computation cost is lower, one would expect the model can generate higher resolution and much longer duration videos, but in fact it’s quite the opposite. To prove the effectiveness, I feel the authors need to show something higher than 256x256, say 512 or 1024 resolution. On the other hand, the hardware requirement is still high (128 GPUs) instead of some normal equipment that everyone can have, so I really don’t see any benefit of the model. If the authors can train DVD-GAN using only a handful of GPUs, that might also be a contribution, but it’s not the case now.
3.	Output quality is reasonable, but still far from realistic. Recent GAN works have shown amazing quality in synthesized results, and the bar has become much higher than a few years ago. In that aspect, I feel there’s still much room for improvement for the result quality.

Overall, given the limited novelty, low resolution output and still high hardware requirement, I’m inclined to reject the paper.

review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: The method shows promising results on on generating high duration (up to 100 frames) class conditional videos with convincing Inception scores, indicating quality similar to DVD-GAN, while consuming less memory and with better coherence. 
While the contribution of the paper is mainly to improve the DVD-GAN architecture to reduce training time and memory consumption, the reviewer believes that the paper would be a good contribution to the venue.
Below there are some questions on the methodology:
1. Is there any way to tell does the matching discriminator actually only estimates the ability to upsample $\hat{x}_w$ from the previous low resolution sample $x_w^l$? From the architecture it is not evident whether or not it only does this or it is also entangled with assessment of how good the low-resolution sample $x_w^l$  was. In other words, if the low resolution sample scores good (e.g. because it's the real-world data) but the upsampling does not match,  would the objective of the matching descriptor training still score it as a good upscaling?  Or is there any reason preventing from this type of behaviour? 
2. Although, as mentioned in the introduction, it may not be as big problem as for VAE-based models, the problem of blurring might exist for DVD-GAN-like models. It is written in the caption of Figure 5 that 'Despite the two stages of local upsampling, the frame quality does not degrade noticeably through time.’ Although the reviewer appreciates that previous work reported only IS/FID/FVD metrics and that defining proper evaluation metrics for generative models is an open question, it might be a good idea to show some other quantitative metrics such as power spectral density (PSD) plots similar to figure 5 from [1]. This would help get an idea how it compares to the real-world video in terms of blurring of the results. 
3. Given that the generation of videos is class-conditional, is it possible to show the metrics per class? Are the scores per class similar or does the method score better for larger classes or the classes with specific motion dynamics? 

[1] Ayzel et al (2020) RainNet v1.0: a convolutional neural network for radar-based precipitation nowcasting
review_title: Good contribution in improvement of DVD-GAN

final_score: rating: 3: Clear rejection
review_content: Pros:
1. A stage-wise approach to train GANs for video is defined to reduce the computational costs needed to generate long high resolution videos.
2. The authors provide some quality results of the proposed approach.

Cons:
1. The contribution of this paper is very limited. The authors just do some incremental improvement based on current GAN models, and the theoretical analysis for the stage-wise training approach is not enough. 
2. The experiments are not convincing. The authors only compared the baseline methods in the experiments. Besides, the proposed training strategy should be applied in different generation models based on GAN to show the effectiveness in different cases. 
3. This paper aims to reduce the computation cost of the model training, but do not achieve significant effect, which takes 23 days for model training.

review_title: This paper proposes a stage-wise strategy to train Generative Adversarial Networks for videos. The contribution of this paper is very limited and the experiments are not convincing.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a GAN-based model which generates videos in multiple stages. The main idea is the upsampling of the spatio-temporal resolution upon addition of a stage. This is the key feature of the proposed model allowing the model to generate videos of higher temporal resolution while using significantly less computational resources. 

**Strengths**
+ The paper is clearly written
+ The model performs competitively with relevant baselines with respect to quantitative metrics
+ The evaluation of the model has been conducted on real world datasets
+ Implementation details have been mentioned clearly

**Weaknesses**
- There have been earlier attempts for multi-stage video generation  [1,2]. However, the paper misses citations in this direction. Also, apart from condition for the generation, how is the proposed model different from the existing multi-stage ones?
- The generated samples for Kinetics dataset are not temporally consistent and misses several details especially for smaller entities in video. To list a few: in Figure 2 row 4, the baby's face looks distorted and different in every frame; in Figure 3 row 2 and in Figure 2 row 1, the face of the person is completely incomprehensible.
- The generated samples in the paper do not have a lot of perceived motion in them. How does the model perform when the input class is supposed to possess huge temporal variations?

Overall, the paper presents a scalable way to generate video with higher temporal resolution. However, the generated results do not look realistic and lot of important details are missing in the generated samples. Therefore, my initial rating for this paper is 4. 


*References used in the review:*

[1] Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018

[2] Zhao, L., Peng, X., Tian, Y., Kapadia, M. and Metaxas, D.N., Towards Image-to-Video Translation: A Structure-Aware Approach via Multi-stage Generative Adversarial Networks. International Journal of Computer Vision, 2020



============================================**Post-Rebuttal Comments**==================================

I appreciate the revisions and additional results presented by the authors. The authors have addressed my concerns as well as improved the clarity of the model description in the revised version of the paper. While that results are not perfect, I acknowledge that the problem of video generation is difficult and I believe such multi-stage model can motivate future methods in this direction of scalable video generation. Therefore, I would like to improve my score to 6 and would recommend acceptance of this paper.

review_title: Generated samples not temporally consistent

Meta-review: This paper proposes a GAN for video generation based on stagewise training over different resolutions, addressing scalability issues with previous approaches. Reviewers noted that the paper is clearly written, proposes a method that improves upon the DVD-GAN architecture by reducing training time and memory consumption, and has competitive quantitative results.

On the other hand, the more negative reviewers are concerned that the empirical improvements demonstrated are somewhat incremental, and that there is not much novelty as the proposed approach is similar to other methods that decompose the generation process into multiple stages at different temporal window lengths and/or spatial resolutions. The authors argue that these criticisms are subjective and non-actionable. I sympathize with their frustration, but an acceptance decision for a competitive conference like ICLR does involve some subjective judgment as to whether the method and/or results meet a high bar beyond mere correctness. For this submission that's a close call, but between the novelty/incrementality concerns and the other more minor issues raised by reviewers (e.g., missing frame-conditional evaluation) I believe this paper could benefit from another round of revisions and improvements and recommend rejection.

I hope the authors will consider improving the submission based on the reviewers' feedback and resubmitting to a future venue, as the paper certainly has merit. To this end I have a few concrete recommendations for the authors which could have flipped my recommendation to an accept if implemented:

* Report results in the frame-conditional setting for comparison with DVD-GAN and other methods that operate in this setting.
* Proofread the paper more thoroughly. I noticed several typos while skimming the paper, e.g. in the theory section, the second term of eq. 6 confusingly uses $\rho$ instead of $\log$. (Relatedly, given that appendix B.1 reports that the hinge loss is used, I'm not sure whether $\log$ is correct in the first place -- this probably deserves further explanation or correction.)
* Demonstrate/argue more convincingly (in one way or another) that SSW-GAN's improved efficiency really expands the frontier of what was possible before. It is true that the 128x128/100 video samples contain 2x as many total pixels as DVD-GAN's 256x256/12 samples, but this isn't a *strict* improvement as the spatial resolution is smaller, and a 2x difference leaves space for reviewers to reasonably wonder whether previous methods really couldn't have matched this if pushed. Some possible examples of this: show that SSW-GAN can generate longer 256x256 videos (a strict improvement over what was possible with DVD-GAN), or orders of magnitude longer (e.g., 1 minute) but still temporally coherent videos at 128x128, or videos with substantially improved subjective sample quality at the same (or higher) resolution.
* The paper notes that "DVD-GAN models do not unroll well and tend to produce samples that become motionless past its training horizon". If this were quantified, e.g. by additionally reporting IS/FID/FVD separately for different timestep ranges, it could make a more compelling argument in favor of SSW-GAN.
Decision: Reject