Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces an INR-based design that utilizes an MLP network to encode spatio-temporal dynamics of video. The authors also propose an efficient discriminator to detect unnatural motions. The design is an extension of INR-GAN (Skorokhodov et al., 2021) architecture. The paper claims that the method obtains SOTA performance on the UCF101 dataset. 
main_review: The paper is well-written, easy to follow, and provides experiments on multiple datasets.
#### Strengths
- The paper provides a comparison to recent methods on IS, FVD, and KVD evaluation metrics. This helps to understand better the performance of the introduced model on video generation.
- Introducing an INR-based design for long-term video generation seems a good idea.
- Non-autoregressive generation. This is the feature I like the most. Being able to generate arbitrary time or predict past or future frames. 

#### Weaknesses
 - Decomposing signal into motion and content is interesting however this is not new and previously studied by MoCoGAN[1] and Temporal Shift GAN [2] (which is not cited in the paper). We don't see any discussion in the related works on decomposition and similarity between works. 
- Although extending INR-based image generator to video generator is interesting, it's a trivial modification and not significantly novel.
-  The paper states that "the proposed generator encourages the temporal coherency of videos by regulating the
variations of motion features and enhancing the expressive power of motions with an extra nonlinear mapping", however, this is not very clear that how is this done in the design (explicitly or implicitly)? Is there any specific term to control the temporal coherency of video in the generator? How do you "regulate variations of motion features"? I couldn't find the details in the paper.
- There is no information on computational complexity, training time, and the number of GPUs used for training. 
- I really don't think that the discriminator design works for long-term video generation. Without seeing a good amount of video frames how do you expect the discriminator to evaluate the action? let's assume that in the long-term video a person doing something in the first 4 seconds and then goes out of the scene and another person starts doing something else! Then only two frames and temporal difference can't help the generator to create better motions. I think this discriminator works if the video is short and there is no repetition or high similarity in the video frames. To evaluate, a good experiment would be to generate random videos from UCF101 dataset and try to evaluate the performance with the metric introduced in [3] or a similar metric (S3) used in [2]. Or at least qualitatively show long videos from UCF101.
- Missing ref [4]



#### Ref:
1-MoCoGAN: Decomposing Motion and Content for Video Generation, Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, Jan Kautz\
2-Temporal Shift GAN for Large Scale Video Generation, Andres Munoz, Mohammadreza Zolfaghari, Max Argus, Thomas Brox\
3-Classification Accuracy Score for Conditional Generative Models, Suman Ravuri, Oriol Vinyals\
4-StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN, Gereon Fox, Ayush Tewari, Mohamed Elgharib, Christian Theobalt
summary_of_the_review: Video generation is a very hard problem and the authors try to introduce a more efficient approach for this problem. I believe INR-based video generation is a valuable extension of the image generator but this is not significant. Also, the experiments cannot fully support the design choices for the discriminator.  Additionally, the paper misses some related works (and discussion on differences) and details on computational complexity and training time. I don't think the paper is ready to be accepted to ICLR. However, I'm open to discussions and will consider the author's response.

Update:
The author's response address almost all my questions, however, I'm not convinced about the discriminator choice and the author's explanation is not clear and convincing. This is also confirmed by their qualitative results on long videos such as the UCF101 video (64 frames). The result is much worse than DVD-GAN (Clark et al 2019). 
I believe this work has positive aspects in terms of efficiency and ability to generate frames in parallel. Therefore, I'm increasing my rating from reject to "marginally below the acceptance threshold".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper present a new method to video generation. They built on top of the INR-GAN framework and extend it to temporal domain. They, therefore, get continuous time and space image generation--something all other video methods lack (technically one should be able to interpolate motion codes of existing CNN-based works, but spatially none of them is continuous to my knowledge). They model a video as a function $f(x,y,t)$. They condition this functions on motion codes and on content codes. The upper layers get information from the content, the lower layers get information from both content and motion. They further introduce a discriminator that is conditioned on the time difference between frames. It allows them to train on longer videos. 

main_review: The paper presents a good submission, is well written, well motivated and evaluated. 

Strengths:
1. Idea of representing videos as continuous functions. I believe this is an important idea not addressed in the past
2. The idea of temporally conditioned video discriminator. A heavy weight discriminator is a typical problem in video synthesis, as it makes training of the whole pipeline longer. I wonder if a similar discriminator can be used to make previous models be able to generate longer sequences.  
3. Results and applications. The reported results show significant numerical improvements over the state-of-the-art methods. What's more interesting, similar to other INR methods, the present work can extrapolate videos in space and time. It can also do video in beetwening. 

Weaknesses:
1. The generator part is adaptation of the INR-GAN to the temporal domain. The extension is simple.
2. The paper doesn't discuss the limitations. I wonder if the method can be trained on extremely long sequences, or perhaps can generate a couple of tens of seconds at inference time? The provided website contain a limited of 2 seconds long examples and two more 4 seconds long examples, which look like interpolation. 
3. Consider two frames of a video, for example, when the hand is up and the other one when the hand is down. Since the action can be performed fast (or slow), there can be arbitrary many (or few) frames between these two frames. It's not clear if this adds any difficulties for the video discriminator, since the conditioning can be seen as incorrect. Could the authors comment?
4. I wonder if the authors thought of any metric that show that the proposed motion is motion indeed, rather than interpolation between two or more frames. For example, in Fig. 12 the generated results look like interpolations between two frames. See the example on the top, the right hand grows, not moves. One could have an experiment, in which indeed an interpolation in StyleGAN space is made and then compared to these results.

UPDATE:

I read the authors' response. It seems to me, that both the generator and discriminator work because of the relative simplicity of commonly used benchmarks for video synthesis. It means that a dataset can be well described by a pair of frames sampled many times from the data.   UCF-101 is a way more challenging dataset (too challenging) for current video synthesis methods. This is, however, not an issue of this paper but of the field in general. 

It seems odd to me, that authors showed only 2 examples of long video generation on TaiChi (where the method seems to work well) and 100 examples on UCF-101 where the method works very poorly, generates repetitive motions, generates videos that don't make sense (yet better than state-of-the-art). How representative are the two examples? I had the same question in my initial review. The authors had two 4-second examples, now they have 2 8-second example which doesn't bring much difference.

So I believe my initial evaluation was correct, the paper is slightly more on the positive side of the bar.
summary_of_the_review: I believe, the paper is a good piece of work and is above the bar.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper leverages the implicit neural representations paradigm to build generative adversarial networks for video generation. Implicit neural representations encode continuous signals into parametrized neural networks, and the authors claim that this mitigates the issue related to the inefficient modeling of videos as 3d tensors of RGB values. Their GAN includes an INR-based video generator, and a motion discriminator that is more efficient at identifying unnatural motions. Their approach yields FVD improvements in various existing datasets.
main_review: ## General Thoughts
The paper is interesting and the method propose definitely shows promise. Applying INRs to this task seems like an excellent direction, and avoids many computational problems of video generation. I would have loved to see a more compelling set of visual examples to highlight the possibilities of the technique, and a potential comparison to other works that have trained on larger portions of Kinetics-600.

## Strengths
- The proposed alternative has several positive poperties, such as non-autoregessive generation.
- Decomposition of spatial and temporal factors in the INR
- Good variety in datasets tested
- Strong improvements in quantitative metrics
- INRs seem like a very good approach for this task, love the concept

## Weaknesses
- A few typos and writing could be improved (last paragraph of pages 4 and 5 for example)
- Would love to see an ablation study showing that including the motion + content vector as input to the motion generator (instead of just the motion vector) improves performance
- The idea of comparing two frames and their time difference is nice, but its power is only truly leveraged if many different $\Delta t$ are shown to the discriminator - is this the case? How are you handling this during training?
- Would love to see more results of the effects of sampling different motion vectors - Figure 6 is not compelling enough to believe the claim that the presented motion vector setup is beneficial. There is only one example in the anonymized webpage.
- Why was only TaiChi used for the interpolation task? This technique should handle interpolation much better than its counterparts, and showing more results there seems like a winning move. Why were the other datasets omitted?
- Fig 5 is not very compelling, would modify or add more examples. Would have loved to see more examples in the appendix or webpage.
- Comparisons to MoCoGAN-HD's 1024x1024 videos would have been nice, to really test the extrapolation abilities of INRs


## Questions
- What is Lambda in figure 9? Can't find it in the main text
- Figure 10: compelling, but I'm left to wonder if the discriminator does this even with a fake pair of frames: For a fake video, when the frames are far apart, we'd like the discriminator to still say that the input is fake, right?
- Discriminator input channels increased from 3 to 7 -> why do this instead of siamese nets to compare the two frames? 



summary_of_the_review: UPDATE: The provided rebuttal answers most of my concerns, and my confidence about the strength of the paper has increased. I am updating my recommendation to 8 - accept.

The paper discusses a very relevant concept in video generation: utilizing INRs to improve performance and efficiency. The authors show strong quantitative results and a few architectural novelties. The work is solid, but there is a worrying lack of compelling qualitative examples to back up certain claims. If the authors can provide more examples, I believe this could be a strong addition to ICLR.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper proposes a video generation approach based on implicit neural representations (INRs). They construct a "dynamics-aware" GAN; where by the generator model used derives from the coordinate-based models used to capture continuous representations of images in prior  INR work. The discriminator architecture derives from the 2D discriminator traditionally used in image GAN research; they simply expand the input channels from the traditional 3 channels (i.e. rgb) used; to 7 input channels. This discriminator processes a stack of images from one video simultaneously, two frames from different time points in the video and their difference image.
main_review: Strengths:

1. This work proposes a simple extension of INRs to video generation.
INRs have thus far remained in the realm of image generation. To my knowledge, this is the first work to extend this approach to video.
In hindsight, this may seem like a trivial extension, but the introduction of INRs to video generation provides a solution to many hard problems in video generation work.

2. As a result of being the first work to present INRs as a viable method for tackling video generation, it benefits from the efficiency advantages inherent to using INRs for videos, which include but are not limited to: long horizon video generation (can predict up 128 frames in future at a resolution of 128x128), time interpolation and extrapolation (can predict interim frames and also predict far out into the future), non-autoregressive generation (can generate frames for specific time points of interest without generating intermediate frames), space interpolation and extrapolation (can up sample and zoom out of generated video)

3. The work is well motivated, written and benchmarks across all the standard video generation datasets. It also includes the relevant ablation studies. It is also timely, given the need for more efficient video GAN models.



Weaknesses:
1. This work suffers from one fatal flaw. Some of its experiments train the proposed model on the full dataset, not the training split as in prior work, yet it presents results comparing performance to models trained on only the training split of the dataset. I have to admit that I missed this as a reviewer of the prior work that was submitted and accepted to ICLR last year, but since it was not explicitly stated, it was assumed that the prior work followed proper experimentation procedure when benchmarking against prior art. Particularly for UCF101, the results where models train on the full 13320 videos should be explicitly differentiated from models that train on the "trainlist01" training split of UCF101 comprised of only 9537 videos. To allow for honest and accurate comparison to the majority of prior work, I would strongly encourage the authors to benchmark models trained on the training data split for the related datasets, in particular, the "trainlist01" split of the UCF101 dataset. They should also highlight and mark (e.g. star) results that come from models trained on a full dataset, that compare their results to prior art that only trains on a split of that dataset.

2. This work is lacking when it comes to experiments comparing the compute, memory, time or energy efficiency of the proposed approach. This would greatly strengthen the proposed contribution since it is so naturally amenable to great gains in efficiency across all four metrics.
summary_of_the_review: This is good work, a simple idea extended from the image to the video domain. It suffers a fatal flaw that prevents me from recommending it for acceptance. 

In its current form, this paper is "marginally below the acceptance threshold". If the issues with benchmarking are resolved, then I will recommend it for acceptance. If additional experiments exploring the efficiency of the proposed approach are provided, then I will strongly recommend it for acceptance.


------

***EDIT****

The authors have sufficiently addressed all of my concerns (and those of the other reviewers it seems).
I strongly recommend this paper for acceptance to ICLR22, it is a meaningful step in a promising direction for long-horizon GAN-based video generation.
review_title: null

Meta-review: This work tackles video generation using implicit representations, and demonstrates that using these representations enables improvements to long-term coherence of the generated videos.

Reviewers praised the writing, the thorough experimental evaluation, and the strong quantitative results. Some concerns were raised about a lack of discussion of relevant related work, novelty/significance, model architecture, and a lack of qualitative examples, many of which the authors have tried to address during the discussion phase. Several reviewers raised their ratings as a result.

Personally I certainly believe that exploring implicit representations for video is important, and I know of no published prior work in this direction, which amplifies the potential significance of this work. Even if results are qualitatively worse than previous work in some ways, this exploration is still valuable and worth publishing.

While the paper ultimately received one reject rating, another reviewer chose to champion this work and award it the highest possible rating. Combined with the other positive reviews, this provides plenty of convincing evidence for me to recommend acceptance. That said, given the rating spread, I would like to encourage the authors to consider the reviewers' comments further as they prepare the final version of the manuscript. Especially providing more qualitative results would be a welcome addition.
Decision: Accept (Poster)