Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
Summary:
This paper proposes a training objective for higher quality video generation for the tasks of Video Super Resolution (VSR) and Unpaired Video Translation (UVT) and also two evaluation metrics tOF and tLP. They provide a comprehensive ablative study of the proposed method and also show comparisons against baselines for the tasks of VSR and UVT.


Pros:
+ Novel video generation method for VSR and UVT.
+ Novel metrics
+ Well written paper

Weaknesses / comments:

- Confused about inputs to discriminator (I_{w,g} and I_{w,b})
I am not sure I understand the inputs I_{w,g} and I_{w,b} to the discriminator network. Based on the description, I_{w,g} = {W(g_{t-1}, v_t), g_t, W(g_{t+1}, v_t^’)} = {g_t, g_t, g_t}, assuming v_t^” means reverse flow. A similar process seems to be happening with I_{w,b}. If this is the case, will the discriminator optimally get the same frame concatenated together (i.e. {g_t, g_t, g_t})? Now, if that’s the case, how is the discriminator modeling temporal consistency other than making the generator learn to put pixels forward and back in place since the “original triplets” (real data) are {g_{t-1}, g_t, g_{t+1}} and {b_{t-1}, g_t, g_{t+1}}, respectively. This is very confusing to me. It would be good if the authors can clarify this in the rebuttal.


- How do you prevent the zero output in PP loss?
The PP-loss compares forward and backward prediction by || g_t - g_t^’ ||. If not careful, the neural network can just learn to output zeros or the same frame for the full video. Did the authors see any behavior like this? Or did the proposed formulations prevent this? Or was there a very small weight applied to this loss?



- TecoGAN vs baselines generator parameters (rather than TecoGAN^{-}).
Based on the experimental section, it looks like TecoGAN is the main network being compared to the baseline methods. TecoGAN has more parameters in the generator compared to TecoGAN^{-}. Did the authors make sure that the generator had the same number of parameters as the other methods? The authors mention a performance difference between TecoGAN and DUF due to DUF having more parameters, but what about the other baselines?

- Evaluation metric contributions in the supplementary material?
The description of the two proposed metrics tOF and tLP have been placed in Appendix B. These are mentioned as contributions of the paper so their description should be in the main text. Please make sure that Appendix B is in the main text in future versions of this paper.


- UVT task evaluation in the supplementary material?
The UVT task is mentioned in the abstract and as a target task in this work, however, the evaluations for this are in the Appendix. Please move them to the main text in future versions of this paper. Secondary experiments should be in the Appendix but I feel this is primary.


- UVT task only evaluated with the proposed evaluations?
The UVT evaluation in the Appendix only has the proposed metrics as evaluation. I understand that, since it’s an unpaired video translation task, there is no ground truth to compare against. However, a human based study could be done where human raters would judge for the more realistic of N videos.


- The same qualitative comparisons in the paper are not in the provided website.
The provided website does not have the same qualitative evaluations as the paper does. For example: There are 7 methods being compared for the same video in Figure 8 in the supplementary material, but I don’t see anything like that in the website.


- For the UVT task, we omit the DsDtPP model because …….
In the paper, the authors mention that they don’t provide the DsDtPP baselines because it requires a lot of computation. However, I feel these missing baselines make the experiments incomplete since it’s a different task.


- Figure 5 has no point of reference with which to compare the frames.
Figure 5 points out that the video b) is better than video a). While it is true that one seems more noisy than the other, there should be a point of reference for readers to make sure of it.


Conclusion:
In conclusion, the paper seems to present a novel method and evaluation metrics but has many issues as stated above. It would make the submission better if the authors can address them in the rebuttal.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper presents video generation method with spacio-temporally consistent features. This is done through: a) temporal adversarial learning, b) Ping Pong loss, and c) metrics that quantify the quality. The methods are evaluated on two datasets and user studies.

The idea is interesting and the paper is well written. The results are convincing.

The originality of the concatenation of several frames is somewhat limited, since it is a standard procedure in other domains such as robotics. Nevertheless the results are positive.

Seems like the metrics definitions were not included in the main body of the paper - the authors should either include them to remove from the contributions. 
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: 
The paper presents a novel method for training video-to-video translation (vid2vid) models. The authors introduce a spatio-temporal adversarial discriminator for GAN training, that shows significant benefits over prior methods, in particular, parallel (as opposed to joint) spatial and temporal discriminators. In addition the authors introduce a self-supervised objective based on cycle dependency that is crucial for producing temporally consistent videos. A new set of metrics is introduced to validate the claims of the authors.

I really like this paper. Although the method is a rather complex mix of multiple losses, these are justified in detail, both intuitively and empirically. The appendix is filled with much more detail about implementation, architecture and more results. Finally the results show that the proposed method is superior across the board compared to previous approaches from the literature.


Strenghts:
- The approach works on two distinct applications of vid2vid.
- Detailed ablations justifying the introduction of every single part of the overall objective.
- Strong results.
- Clear writing and presentation.
- A lot of additional details and results in the appendix for the more interested reader.

Weaknesses:
- Rather complex overall objective
- Seems to need a lot of tweaking

Questions:
- Isn't the PP loss just an incarnation of cycle consistency loss? If so, maybe there is no need for the introduction of a new name for it.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Contribution
Augments the loss of video generation systems with a discriminator that considers multiple frames (as opposed to single frames independently) and a new objective termed ping-pong loss which is introduced in order to deal with “artifacts” that appear in video generation. The paper also proposes a few automatic metrics with which to compare systems. Although the performance does not convincingly exceed its competitors, the contribution seems to be getting the spatio-temporal adversarial loss to work at all.
 
Overall
I found the video generations impressive, and the addition of the discriminator seems to improve sharpness of the individual frames over methods trained with non-adversarial losses, in particular DUF. Although TecoGAN does not beat its competitors (for example DUF) on most proposed automatic metrics, it appears to be ranked better in expectation in user studies (Table 2). However, I am concerned about generalizability of the method.
 
Decision
As someone not in this field, I find it hard to judge the performance of the system from so few samples. I also found the differences in the losses between tasks to be worrisome, as it indicates this method does not generalize without heavy tweaking of the loss. My current decision is weak reject, since the generations look quite good but I have concerns about generalization of the approach to other video generation tasks as well as the justification of the ping-pong loss.

Questions
- Is RecycleGAN unable to be applied to video super-resolution? Why is it not compared to in Table 2?
- Is L_Phi the perceptual loss? I don't think it was mentioned in the body of the text.
- I found the ping-pong (PP) loss to be unjustified. The loss is motivated by the issue of "artifacts", which I assume are poor generations due to lacking a good model of the world. The paper says this issue could be alleviated by training with longer video sequences, but that would prevent the generator from working with sequences of arbitrary length. I do not believe the ping-pong loss allows the generator to work with arbitrary sequences, as training only on short sequences and their reverse should not allow the model to generalize to longer sequences. Additionally, this approach would likely not scale to long sequences as well, since it requires doubling the sequence length. Would you be able to train on longer sequences?
review_title: Official Blind Review #1

Meta-review: The paper presents an architecture for conditional video generation tasks with temporal self-supervision and temporal adversarial learning. The proposed architecture is reasonable but looks somewhat complicated. In terms of technical novelty, the so-called "ping-pong" loss looks interesting and novel, but other parts are more-or-less some combinations of existing techniques. Experimental results show promise of the proposed method against selected baselines for video super-resolution (VSR) and unpaired video-to-video translation tasks (UVT). In terms of weakness, (1) the technical novelty is not very high; (2) the final loss is a combination of many losses with many hyperparameters; (3) experimentally the proposed method is not compared against recent SOTA methods on VSR and UVT. 

The proposed method should be compared against more recent SOTA baselines for VSR tasks (see examples of references below):

EDVR: Video Restoration with Enhanced Deformable Convolutional Networks
https://arxiv.org/abs/1905.02716

Progressive Fusion Video Super-Resolution Network via Exploiting Non-Local Spatio-Temporal Correlations
ICCV 2019

Recurrent Back-Projection Network for Video Super-Resolution
CVPR 2019

The same comment would apply for baselines for UVT tasks:

Mocycle-GAN: Unpaired Video-to-Video Translation
https://arxiv.org/abs/1908.09514

Preserving Semantic and Temporal Consistency for Unpaired Video-to-Video Translation
https://arxiv.org/abs/1908.07683

Particularly for UVT, the evaluated dataset seems limited in terms of scope as well (i.e., evaluations on more popular benchmarks, such as Viper would be needed for further validation). Overall, given that the contribution of this work is an empirical performance with a rather complex architecture/loss, more comprehensive empirical evaluations on SOTA baselines are warranted.

Decision: Reject