Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a GAN-based video generative model that is based upon StyleGAN-V while producing videos with significantly fewer artifacts. The paper first outlined several common artifacts seen in the videos produced by StyleGAN-V, and then provided an extensive set of targeted solutions:
1. Alias-free techniques from StyleGAN3 is adopted to reduce the texture sticking artifacts at the cost of a slight drop in quantitative scores.
2. To alleviate the quality drop, the author proposed pretraining the later layers on a image generation task and then finetuning the early layers on the actual video generation task.
3. A *temporal shift module*, originally developed for video understanding, is incorporated into the discriminator, improving the quality further.
4. To enable jitter-free long video generation, the motion latent space is instead modeled with a B-Spline.
5. To reduce the periodic repeating artifact introduced by 4, a low-rank matrix is used to limit the effect of temporal features on the output frames.

In terms of quantitative results, the proposed method with improvement 1, 2, and 3 achieved the best video quality on an existing benchmark but inferior image quality. It however outperformed prior art in every aspect on a new, self-collected dataset. 

Adding improvement 4 and 5 will improve visual quality but reduce quantitative scores.
strength_and_weaknesses: ## Strength
* The proposed method significantly improves upon the previous works in terms of video quality. What's more important, the paper conveyed a message that good quantitative scores does not always equal good visual quality perceived by humans. 
* The paper is easy to follow, and includes comprehensive quantitative experiments and good visualization.

## Weaknesses
* Most of the techniques used in the paper are not new. They are borrowed from related works.
* The effect of setting F is not clear. There is no figures or videos showing what referred to as *similar content periodically appears*.
* The new techniques introduced in the paper (setting E and F) significantly hurt the quantitative performance. Although claimed to have the benefit of reducing the jitter artifact, the non-visual evidence is weak (Table 4). It will be more persuasive if there are better evaluations, such as a user study.
clarity,_quality,_novelty_and_reproducibility: The paper is well written. The backgrounds are well-covered. The paper included comprehensive quantitative evaluations but qualitative ones are lacking. The techniques used in the paper are not original. However, it is inspirational to see that these techniques can be used together to obtain a more powerful model. The authors promised in the paper to release code.
summary_of_the_review: I liked how the paper cast light on the practical issues in the previous works and then tacked them one-by-one, instead of focusing only on getting better numerical scores. This, as well as the techniques proposed in the paper, can serve as a good reference point for the future works. There are a few improvements that are not as well-justified as others, but it does not diminish other contributions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a method for smooth video synthesis. They build upon the SyleGan-V and progressively improve the baseline by proposing the alias-free operation, adding a temporal shift module, and incorporating the B-Spline based motion representation. It achieves improved temporal smoothness in the generated video without hurting the per-frame quality much. The experiments on SkyTimelapse and YouTube Driving datasets demonstrate the contributions of each component of the method. This work also considers long video generation which requires continuously sampling the temporal latent codes.
strength_and_weaknesses: Strength
+ They propose multiple techniques to siginficantly improve the video synthesis quality. Especially the FVD scores improved much compared to the prior arts.

+ It resolves the brush effect which has been a problem in the previous methods, due to the generated contents being bound to the spatial coordinates in the image/frame.

+ The visual quality of the generated video outperforms those of the previous methods.

+ Long video generation is also considered, and is come up with novel methods to handle this.

Weakness
- While the proposed multiple techniques do improve over the baseline, the conclusion from the result is not very clear. In Table 1. Config-D shows the best FVD scores on the two datasets but not the methods with the best FID score are different over these two datasets. Why is the Config-E and F worse than D? What is the best Config for users?

- While the paper argues the proposed method does not sacrifice the per-frame quality, the FID scores are worse than the baseline StyleGan-V. 

- The visual comparison (Figure 2, 3, A4, A5) shows the results on different sequence examples, which makes it hard to directly compare the performance between the baseline vs proposed method.




clarity,_quality,_novelty_and_reproducibility: - It is not clear what “our method” refers to among the Config-B - F. Is it Config-D for short videos, and Config-F for long videos?

- It would be also better to have a name for the proposed method instead of Config-A, B, … 
summary_of_the_review: The discussion on the worse per-frame FID score, and visual comparison on the same examples, and the best method among the Configs for short / long video will make this paper more complete.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper improves the previous video generation model StyleGAN-V with several existing techniques and model-specific fixes. The proposed method shows better FVDs on the Sky Time-lapse dataset and a self-collected Driving dataset.
strength_and_weaknesses: [Strengths]

1. The paper is very well-organized. The author gave a distinct description of the baseline StyleGAN-V model and clear discussions about the problems that existed in this model and how they propose to fix them. 

2. The motivations for adopting the temporal shift module (TSM) in the discriminator and pretraining the model on images first are reasonable, adapting techniques proposed in StyleGAN3 to video generation is also useful for the community. The experiments showing improvements brought by these modifications are relatively sound.

[Weaknesses]

There are several flaws in the experiments of this paper:

1. Only two landscape video datasets are used to evaluate the method. It is often the case that people are also interested in generating videos with human motions, which could imply different difficulties from landscapes. There are common video generation benchmark datasets for that purpose such as UCF-101, Tahich-HD, or FaceForensics where previous methods are evaluated on. However, none of these datasets is used in the paper. In addition, since the I3D model used to compute FVD scores is trained on the Kinetics dataset, I personally think that this metric will be more meaningful on the human motion datasets.

2. Even though only two datasets are considered, the trends of the results on them are not well aligned. For instance, pretraining model on images helps on the driving dataset but not the sky dataset, while using TSM only improves the sky dataset. Is this because of the large variance? Probably more datasets could also help demystify these observations.

3. For the main results, the authors adopt only those baselines used in the StyleGAN-V paper. However, since some of the results are focused on long video synthesis, the recently published methods on this problem should be considered [1, 2]. 

4. The most novel part of the method contains configurations E and F. The main goal of these two fixes is to resolve a known issue of StyleGAN-V which generates cyclic videos when extending to a longer duration. However, these fixes don't seem to improve the model in terms of quantitative evaluation. In addition, only a limited number of qualitative results (in the format of images) are given to demonstrate the improvements. Therefore, the claimed improvements are not convincing to me. Probably some metrics to detect the recurrence or human evaluation could better show the advantage of the proposed method.

In addition, the contribution of the proposed YouTube Driving dataset is not significate to me. There are multiple existing video datasets for driving or car dash cameras, e.g. BDD-100K [3] or Cityscapes[4]. There are also video generation works that are trained on these datasets. The paper lacks a comparison of the collected driving dataset with these existing datasets.

[1] Long video generation with time-agnostic vqgan and time-sensitive transformer. ECCV 2022.

[2] Generating long videos of dynamic scenes. NeurIPS 2022. 

[3] Bdd100k: A diverse driving dataset for heterogeneous multitask learning. CVPR 2020.

[4] The cityscapes dataset for semantic urban scene understanding. CVPR 2016.
clarity,_quality,_novelty_and_reproducibility: [Clarity]

This paper is generally well-written and easy to follow. I have some questions regarding the method details:

1. When pretraining the model on images, what is input as the motion embedding?
2. When loading the pretrained parameters to the model, are the shallow layers initialized and trained from scratch?
3. When generating infinite length of video, what happens if $t$ is sampled to be larger than $N_A-1$?
4. Which layers are TSM added? The notations are a bit confusing in section 2.3 since I previously thought that $y_i$ are the output of the discriminator convolution layers and input to the MLP layers. But it seems that from the top of page 5, more convolution layers are applied to $y_i$?

[Quality]

As stated in the weaknesses, I have several concerns about the experiments done in the paper.

[Novelty]

The proposed methods are incremental but relatively novel in their current forms compared with existing methods.

[Reproducibility]

The code is not provided as part of the supplementary material. I have raised a few questions regarding the details of the method in the clarity section, which could make the method less reproducible.

summary_of_the_review: I value some of the contributions of the paper such as adapting StyleGAN3 to video generation. However, I have multiple concerns regarding the experimental results as well as the conclusions drawn from them. I'm happy to hear from the authors about their thoughts on these concerns and reconsider my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper deals with generative modelling of videos, and specifically proposes a variety of improvements over StyleGAN-V. The improvements are intellectually appealing, and quite convincingly demonstrated. While Arxiv is starting to be full of video papers, many of which exceed the quality of this paper, they may not strictly speaking constitute prior art as even the NeurIPS decisions are not officially out yet. As such, I'm ok with the scientific novelty in this one. 
strength_and_weaknesses: Strengths:
- Clear contributions
- New ideas are explained well

Weaknesses:
- Prior art description (Sec 2.1) is confusing. What data is used to compute the anchor frame embeddings? I suspect the current description cannot be understood by anyone who doesn't already know how it works.
- The result quality is still pretty low -- don't get me wrong, it's better than StyleGAN-V, but we definitely still see jittering in the longer videos and content warps around like crazy. These needs to be admitted and discussed in the paper.
- "Infinite" video is a dubious concept. If the authors want to talk about it, then it needs to be defined. What constitutes an infinite video and what does not? Surely any interesting "infinite" video will need to introduce infinite amount of novel content over time? Otherwise it's hardly infinite -- although just ping ponging a short clip may technically be infinite, but that's hardly a productive definition. 


clarity,_quality,_novelty_and_reproducibility: The novelty and clarity are fine, evaluation is ok. I guess I'm a bit underwhelmed by the result quality in the longer videos, though. 
summary_of_the_review: This paper describes a clear step over StyleGAN-V, and as such I'm in favour of acceptance. Details are listed above. Obviously we need better quality metrics for video. To the best of my knowledge Sec. 2.2--2.4 are novel when compared to officially published work, but I may not know everything.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a generative model for videos built on StyleGAN-V. In initial reviews all reviewers found the paper to have adequate contributions, thought that it was mostly well-written, and presented results which were a clear improvement over StyleGAN. Several reviewers were confused about Configurations E and F. Reviewer yrRK also pointed out several other important weaknesses: the paper only evaluated on two datasets of landscape videos, and should have compared with some more recent prior work. These issues were mostly resolved by the author's responses, and in the end all reviewers agreed that the paper should be accepted.
justification_for_why_not_higher_score: Reviewers agreed that the results were an improvement over StyleGAN-V and the paper was ok to accept, but no reviewers were particularly enthusiastic about the paper; the results are better than StyleGAN-V but not dramatically so, and the technical ideas were mostly adapted from other problems rather than presenting some dramatic new techniques.
justification_for_why_not_lower_score: All reviewers recommended acceptance.
Decision: Accept: poster