Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper discusses a method for generating long videos in which new content is introduced as the camera moves forwards. Training on such videos especially in high resolution is prohibitively expensive. One of the possible ways to mitigate the issue is to train a two stage architecture, in which dynamics is trained on low-res, followed by super-resolving the low-res long video with the second stage. To show advantages of their approach the authors collect two new datasets, in which the camera moves forward. According to the results, the method outperformed current works on these two datasets, and finishes second on previously available datasets. 
strengths_and_weaknesses: Strengths. The paper has some:

S1. The paper shows that treating temporal noise is important. The idea of filtering it with a low-pass filter is interesting. This way only low-frequency, longer events are captured by the model.
S2. Results are quite impressive on the two new datasets. Very impressive!

There also are weaknesses, unfortunately:

W1. Subsampling in space and time has been proposed in the previous literature. It has been shown that training at high res both in space and time is prohibitively expensive. In [TGAN-V2] for example, they generate longer videos at low resolution and as they increase resolution with more generators they drop the frame rate. This way they achieved efficient training of 256^2 resolution--highest resolution of this paper--in 2018. Another similar idea was used in a discriminator of DVD-GAN [9]. They downsampled the resolution for video discriminator, and reduced frame-rate for image discriminator. [TGAN-V2] is not cited, DVD-GAN is cited but the similarities/differences are not discussed. 

W2. The proposed framework is reasonable and seems to be working well on the proposed datasets. The key new interesting idea is temporal filtering, the rest of the framework from the high level is known (see W1). I acknowledge the amount of effort the authors put into making it work, of course. 

W3. Other video datasets. The paper fails to report UCF, FaceForensics and others from stylegan-v & tats, arguing that these datasets contain less new content and camera movements. While it might be case, the method only reports better scores on their own proposed datasets. The problem with this is that it takes a lot of effort to tune each method to each dataset, making it not very clear if the proper and sufficient tuning for stylegan-v was made. If one compares to existing numbers on existing datasets, such as UCF, on which TATS and stylegan-v show reasonable performance, one can guarantee that the method is evaluated against the numbers in which proper time was invested. Otherwise it's always possible to select a dataset on which the method scores best.

W4. Other resolutions. Stylegan-v and mocogan-hd show training on much higher resolutions, as high as 1024. While this method focuses on the temporal part of videos, it's not clear what's the resolution upper bound. It's important to understand that even for this work, since scaling GANs is a non-trivial task both in terms of computational resources and quality. 


[TGAN-V2] Saito, Masaki, et al. "Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan." International Journal of Computer Vision 128.10 (2020): 2586-2606.
questions: Please see above.
limitations: Please see above
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Brief Summary: The paper addresses the problem of video generation with a focus on longer time horizon videos which requires consistency. To this end, the authors introduce two new benchmark datasets on horse-riding and mountain biking. The key observation is that the main components for temporal consistency are preserved at lower spatial resolution. The authors therefore propose using a hierarchical architecture to first create long low-resolution video, followed by sliding windows to create higher resolution videos at shorter time-steps.

The authors have further performed a human evaluation on mechanical turk to find their method was preferred over 80% of the time.  
strengths_and_weaknesses: Pros:

1. The idea is simple yet elegant. Implementation-wise it is quite interesting to find that a straightforward extension of stylegan to videos where input images are simply concatenated leads to promising results. 

2. New datasets contributions are always welcome. The two contributed datasets on mountain biking and horseback riding could be useful for future research especially to evaluate temporal consistency.

3. The authors have done a human eval which is very important in video generation, and found their method was preferred 80% of the time. 

4. The authors also show color-change as a heuristic for temporal which clearly show the benefits of hierarchical training.

5. The visualizations provided in the supplementary are very cool!

Cons:
1. For the user study, the authors should have provided "equally same" as an option, and used sanity check that "equally same" is picked when both videos are obtained from the same model. I am not sure if this could have created any issues in the assessment (my guess is the effect could be mild, but not negligible).

2. I am wondering if a better evaluation for long-term consistency would be conditional generation, where first few and last few seconds of real video are provided. For instance, if a tree is seen at the last frame (fig-1, 10s), one could have a soft-metric that the tree be seen (albeit at lower resolution) at some intermediate frame. Could make the task on human evaluation easier as well given that they would be comparing more similar videos.
questions: Q1. The idea of using low and high resolution for long and small temporal segments reminds me of SlowFast networks [Ref1], where the slow path samples sparsely with more channel dimension, while fast path samples densely with less channel dimension. Obviously, it is not 1-1 correspondence, but some kind of discussion could be useful (maybe in supplementary).

[Ref1]: Feichtenhofer, Christoph, Haoqi Fan, Jitendra Malik, and Kaiming He. "Slowfast networks for video recognition." In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211. 2019.
limitations: I think authors have done a good job of highlighting failure cases as well as pointing out where FVD metric is not very predictive of human eval performance.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a video generation model that is capable of producing new content (e.g., new object or scenery), object motion, and changes in camera viewpoint over time for longer videos than prior works. In order to achieve this, temporal modeling is emphasized in the proposed architecture, unlike existing works in which frame quality is often prioritized. As their main contribution, the authors redesign the temporal latent representation and train the carefully-designed model, which has the capability to operate over long time scales with a vast temporal receptive field, on longer videos at a low resolution and shorter videos at a high resolution. Two new benchmark datasets are introduced to best evaluate the proposed model since there are no existing datasets with long enough videos. Evaluated on 4 datasets with different characteristics, the proposed model outperforms prior methods especially qualitatively on aspects including generating plausible dynamics and object persistence, producing new content while maintaining consistencies over time.

strengths_and_weaknesses: Strength:
+ The paper has suggested several interesting insights and many useful practices for video generation. For example, a multi-resolution two-stage strategy might be a good solution for training and deployment for models to handle long videos; the low-resolution generator should be fully convolutional over time to learn long-term temporal correlations whereas the super-resolution generator can operate in a frame-by-frame basis. Videos with longer sequence length tend to exacerbate the issue of overfitting and therefore some strong augmentations might be required. 

+ The paper has provided a deep investigation on the metrics of video generation. Particularly, they propose to analyze color change over time as a simple way to diagnose potential bias captured by the different models. In addition, existing commonly-used metrics such as FVD and LPIPS are also discussed, and the authors have found these metrics to agree less with the qualitative results or user study results.

+ The qualitative results are convincing. It was shown that the proposed model is capable of generating videos with rich motion and scenery changes. Existing methods are incapable of generating realistic long videos, and explanations and analysis presented in the paper are reasonable.

+ Two new datasets are proposed which might be beneficial for future researchers. 



Weaknesses:
- An in-depth failure case analysis is absent, which might be interesting to have for readers.

- The proposed architecture seems to heavily depend on the correct data augmentation in use.


questions: 1. According to the color similarity plots, the standard deviations of Dataset are quite high, and often higher than that of models. Does it tell anything? 

2. The “swirly” artifacts visible in some of the videos might be attributed to the multi-resolution strategy, and it was referred to as “RGB bottleneck” (line 305). Could you further explain this in detail?

limitations: The authors have discussed the limitations and potential negative societal impact. From the qualitative results, it seems that the model also struggles when objects in the scene interact with each other.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper takes the initiative to improve the problem of video generation in terms of the dynamics of the generated videos. The main strategy proposed by the author is to increase the real video lengths seen by the discriminator by reducing the resolution. The reduced resolution is compensated separately with a super-resolution network.

The authors provide some analysis of the color dynamics of the real dataset and the generated videos using different methods.

strengths_and_weaknesses: Strengths:

1- The literature has overlooked video dynamics and temporal modeling for video generation in generative models. This research has selected the correct challenge in video generation to address.

2- Great writing and supplementary materials.

3- Decoupling the temporal dynamics from frame resolution has some novelties. However, it has been remotely mentioned by previous research but has never been explicitly modeled with two networks in two different stages.

Weaknesses:

1- Although I respect the authors' braveness and choice of problem, I believe the paper conveys such a message that the main reason for low dynamics in video generation models is data. Since we are not showing long videos to the discriminator, the generated videos do not have enough dynamics or look repetitive. Although I agree with this argument, IMO, the main challenge is modeling the temporal dynamics. `We can still (wisely) select shorter videos but with enough frames change.` But we need better structures to model the temporal dynamics. 

2- I appreciate the honest selection of the generated video samples in the supplementary materials. It is clear that in some scenarios, like horse riding, we see novel objects enter the scene, and the motion is not repetitive as the baselines. However, in some cases like `ACID: acid_grid.mp4, second row`,  we see a lot of distortions in the generated video. Although the method works better for high dynamics, it may not handle low dynamics. 


questions: 1 - Fig. 5 is a little hard to read due to the mix of green and blue colors. However, it seems that although the color average between the proposed method is the closest to the real data, the variance is much lower. What does that imply?

2 - I am wondering if the color similarity based on the frame separation is a good indicator/metric for better scene dynamics. Why did the authors select this metric?

3 - I found this statement confusing `time offset of the generated video can be controlled by shifting and reshaping the temporal noise, respectively.` How shifting the noise, can decide the time offset?

4 - I am wondering if a pre-trained super-resolution network (perhaps on image datasets) can perform nearly as well as what authors trained on the video examples. 

5 - What about the number of generated shots? Applying a shot detector on the real vs generated data, how much the distribution of the number of shots per clip is similar per class? What about the distribution of consecutive shots similarity? Do authors think it will be a complementary indication to the color examination?  

limitations: I believe the authors are honest about the negative societal impacts. I have no more points to add here.
review_title: null

Meta-review: metareview: All four reviewers enjoyed this paper and were particularly impressed by the videos provided in the supplementary material. The results are very impressive indeed. The reviewers also agreed that using a multi stage approach was interesting and effective. The two new datasets were deemed useful to the generation community and the proposed metrics and human evaluations were appreciated by the reviewers. A few smaller concerns included a missing failure analysis and some clarifications questions which were addressed in the rebuttal. Given the above, I recommend acceptance.
confidence: Certain
Decision: Accept