Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of this paper: In this work, the authors propose a method to learn to generate long-range video sequences. The general idea is starting from a prior work (Video Textures) and extending this work with a learning framework. Specifically, during training a model is used to learn the transition probability between different video segments. During inference, long-range video synthesis is achieved through iterative sampling of new video segments. To guarantee the smoothness of the transition between different segments, an existing interpolation method is used to connect these video segments in a sequential order.

Pros:

+ quality: This paper is overall easy to read. The motivation behind this work is clearly presented, i.e., to synthesize long-range video sequences. In the introduction part, the authors present the basis of their work (Video Textures) and present a comprehensive comparison with previous works. The authors also present sufficient qualitative results to demonstrate the superiority of their work.

+ clarity: The pipeline of the proposed method is clearly presented in the method part. The general framework is very straightforward. The analysis of the quantitative and qualitative results is convincing and logical. 

Cons:

- originality: This work is more like a simple extension of the previous work (Video Textures) with limited novelty. I am confused about the difference of the basic formulation of the core  transition probabilities (Eq. 1 and 2). Are they different from the work (Video Textures) or not? It is highly recommended that the authors could present more comparison with the previous baselines in both general idea and model details. Moreover, the video interpolation is directly borrowed from previous work without further improvements, where I think is still challenging and worth to explore. The audio conditioned video synthesis part also seems like an extra module which does not influence the completeness of the whole model if not included.

- significance: I have carefully checked the quality of the generated video sequences, which are not so satisfying. First, these are noticeable discontinuity between sampled video segments. Second, this method seems to be example-specific, which needs retraining if fed a new video sequence. The scalability of this method is limited. Third, the video content is directly sampled from seen sequence, where the diversity is constrained to the given video.
review_title: A simple method with non-parametric method for long-range video generation, expecting for more comprehensive experimental results and better results. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: ** Strengths

(1) Improve the classic video texture synthesis method Video Textures by replacing pixel similarity with a distance metric learning to measure the transition probabilities 

(2) Extend the proposed approach to audio conditioned video synthesis

(3) Outperform the competing algorithms on a set of evaluations


** Weaknesses

(1) It seems a strong limitation that the proposed approach is not able to generalize to different videos or has to be video-specific (i.e., train a model on each input video). 

(2) It is not true that existing methods fail to generate more than a short sequence of frames, e.g., (Lee et al. 2019) in theory can generate videos with arbitrary lengths. 

(3) The pre-trained interpolation network (Jiang et al. 2018) seems a quite important component for the proposed algorithm to generate smooth videos. No analysis is provided to show the contribution of this module.

(4) Lacks of analysis or comparisons to justify some important hyper-parameters, e.g., (1) how the softmax temperature term impacts the synthesis results, (2) how t% is chosen to threshold the transition probabilities.   

(5) For video encoding, two separate subnetworks are used to break the symmetry between query and target embeddings, which makes sense considering temporal ordering. However, why audio embeddings take a same encoding network?

(6) For unconditional video synthesis, the musical instrument playing videos mostly contain small motions. Unclear how this approach could handle general videos exhibiting with large actions. 

(7) It is claimed that the approach is able to produce infinite video, however the content is constrained in the input video, so the variation is limited. 

(8) What are the results if the classic methods also use the interpolation network?
review_title: This paper presents Contrastive Video Textures for video synthesis by resampling frames from an input video. The model is video-specific and represents a video as a graph where nodes indicate individual frames and edges are transition probabilities. An output video is generated by traversing edges with high transition probabilities. A deep video interpolation model is used to make visual smoothness for large transitions. This model can be extended to an audio conditioned video synthesis task. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors proposed a non-parametric approach for video generation, i.e., video frame (un)conditional resampling. The proposed method is inspired by Video Textures (Sch¨odl et al., 2000), which synthesizes new videos by stitching together snippets of an existing video. Comparing to existing 'video textures' methods, the authors mainly made two improvements/contributions. (i) a new pipeline for modeling and calculating probabilities of transitioning between frames of the same videos. Specifically, giving a video clip, the authors first extract overlapping segments from it and fit a bi-gram model. The adjacent segments are regarded as positive pairs with high transitioning probability, yet other random sampled pairs are negative pairs. Similar to contrastive learning works, NCE loss is utilized to train the bi-gram model. (ii) Extending the model to a conditional situation and performing the task of audio conditioned video synthesis. The authors made a trade-off between the audio conditioning signal and the learned transition probabilities. Finally, experiments including multiple qualitative resampled videos and quantitative user studies were provided. And the proposed method demonstrated promising performance. 

Overall, this work is well organized and easy to follow. The motivation is clear and ideas are simple and useful. According to the provided video examples, the proposed method achieved great performance. However, I still have some concerns:

In the experiments of unconditional setting, the authors included Classic, Classic+, and Classical++ as their comparisons. However, all recent (here, I mean in the past decade) methods that related to video generation are missed. First, although there may not exist any resampling method that can directly perform the video texture synthesis, I believe many related graph-based methods could be used to model the transition probabilities of frames. I think the authors may at least discuss these works. Second, although the authors pointed out that the video resampling (textures) strategy is different from the recent generation-based strategy, they should provide visual results/comparisons to support their claims. In recent years, some methods were proposed to synthesize videos with dynamic textures, such as flame, wave, flash, etc. I am interested in if the proposed method can perform these tasks. In addition, the authors may further discuss the limitations of the proposed method. 
review_title: Promising results, more discussions/comparisons are requested

Meta-review: The paper initially had mixed reviews (4,5,6).  The main issues raised were:
1) limited novelty (re-using/integrating components) [R2];
2) limited generalization ability since the model needs to be retrained on every video [R2, R3];
3) limited applicability - experiments limited to certain domain of video, while results on videos with large motion are not convincing [R2, R3];
4) missing ablation studies / experiments [R3, R4].

The author response partially addressed some concerns, but the main points 1-3 are still problematic. In addition, the AC noted that the technical aspect was lacking:

- Training with contrastive loss on a single video may likely overfit the embedding to the video, which leads to a meaningless embedding where all non-neighboring segments are orthogonal in the embedding space. While changing the softmax temperature can yield higher entropy transition probabilities, the induced probability distribution is probably highly noisy. It would be better to train this on a large video corpus, which will prevent overfitting. Also contrastive loss is typically used to build a discriminative embedding space for classification/recognition, not a smooth embedding space for generation (where distances between embedding vectors are strongly correlated to similarity). Thus some other embedding smoothness terms could be added during contrastive learning.
- The learning is only on the transition probabilities, while the video generation is separate. It would have been more convincing to learn the transition probabilities with the video generation process in an end-to-end manner. Perhaps a discriminator could be placed after the video generator so that the transition probabilities could be learned so as to better mimic real video. Other loss terms based on video temporal smoothness could also be added ensure smoother transitions between clips (e.g., motion consistency).

The negative reviewers remained unconvinced by the author response, and the AC agreed with their concerns. Thus, the paper was recommended for rejection.
Decision: Reject