Contrastive Video Textures
Authors: Medhini Narasimhan
Shiry Ginosar
Andrew Owens
Alexei A Efros
Trevor Darrell
Venue:  (2021)
Keywords: Video Textures
Audio Conditioned Video Synthesis
Self-Supervised learning
Contrastive Learning

Abstract: Existing methods for video generation struggle to generate more than a short sequence of frames. We introduce a non-parametric approach for infinite video generation based on learning to resample frames from an input video. Our work is inspired by Video Textures, a classic method relying on pixel similarity to stitch sequences of frames, which performs well for videos with a high degree of regularity but fails in less constrained settings. Our method learns a distance metric to compare frames in a manner that scales to more challenging dynamics and allows for conditioning on heterogeneous data, such as audio. We learn representations for video frames and probabilities of transitioning by fitting a video-specific bi-gram model trained using contrastive learning. To synthesize the texture, we represent the video as a graph where the nodes are frames and edges are transitions with probabilities predicted by our video-specific model. By randomly traversing edges with high transition probabilities, we generate diverse temporally smooth videos with novel sequences and transitions. The model naturally extends with no additional training to handle the task of Audio Conditioned Video Synthesis, when conditioned on an audio signal. Our model outperforms baselines on human perceptual scores, can handle a diverse range of input videos, and can combine semantic and audio-visual cues in order to synthesize videos that synchronize well with an audio signal.

Under review conference paper at ICLR 2021
CONTRASTIVE VIDEO TEXTURES
Anonymous authors
Paper under double-blind review
ABSTRACT
Existing methods for video generation struggle to generate more than a short sequence of frames. We introduce a non-parametric approach for infinite video generation based on learning to resample frames from an input video. Our work is inspired by Video Textures, classic method relying on pixel similarity to stitch sequences of frames, which performs well for videos with high degree of regularity but fails in less constrained settings. Our method learns a distance metric to compare frames in a manner that scales to more challenging dynamics and allows for conditioning representations for video frames and probabilities of transitioning by fitting a video-specific bi-gram model trained using contrastive learning. To synthesize the texture, represent the video as a graph where the nodes are frames and edges are transitions with probabilities predicted by our video-specific model. By randomly traversing edges with high transition probabilities, we generate diverse temporally smooth videos with novel sequences and transitions. The model naturally extends with no additional training to handle the task of Audio Conditioned Video Synthesis, when conditioned on an audio signal. Our model outperforms baselines on human perceptual scores, can handle diverse range of input videos, and can combine semantic and audio-visual cues in order to synthesize videos that synchronize well with an audio signal.
heterogeneous data, such as audio. We learn
1 INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow al., 2014) and Variational Autoencoders (VAEs) (Kingma & Welling, 2013) have achieved great success in generating images "from scratch". While one might have hoped that video generation would be simple extension of image-generation methods, this has not been the case. A major reason is that videos are much higher dimensional than images, and producing correct transitions between frames is a difficult problem. While video generation (Vondrick et al., 2016; Mallya et al., 2020; Lee et al., 2019; Wang et al., 201 8a;b) has shown some success, videos generated using such methods are relatively short and are unable to match the realism of actual videos. In comparison, classic non-parametric video synthesis methods from two decades ago, most notably Video Textures (Schödl et al., 2000), are much simpler and can often produce videos of arbitrary lengths.
In these models, new plausible video is generated by stitching together snippets of an existing video. While video textures have been very successful on simple videos with high degree regularity, they use simple Euclidean pixel distance a similarity metric between frames, which causes them to fail for less constrained videos containing irregularities and chaotic movements, such as dance or playing a musical instrument. They are also sensitive to subtle changes in brightness and often produce jarring transitions.
In this work, we propose Contrastive Video Textures, a non parametric learning- baasd approach for video texture synthesis that overcomes the limitations of classic video textures. As in Schödl et al. (2000), we synthesize textures by resampling frames from the input video. However, as opposed to using pixel similarity, we learn feature representations and distance metric to compare frames by training deep model single

---

Under review conference paper at ICLR 2021
Prediction
Video Textures
Ours: Contrastive Video Textures
NCE
Find good toos Interpoiate
DNa 2020)
Video Generation
[Lex 2015, Malge 23201
Non- Man-Parametric Texture Synthenis
lopat Video
Outpur. Texture
Nobe
(Scnel at 2000
Training
Inference
[Wedkcc
Figure I: Video Texture Synthesis. Prior video prediction (Xu et al., 2020) and generation (Vondrick et al., 2016; Lee et al., 2019; Mallya et al., 2020) fail generate long sequences and high resolution images. Classic video textures (Schödl et al., 2000) (middle) can generate infinite sequences by resampling frames, but uses fixed representations which are not robust to varying domains. Our method (right) learns a representation and non-parametric method for infinite video generation based on resampling frames from an input video.
inpul video. The network is trained using contrastive learning to fit an example-specific bi-gram model (i.e. a Markov chain).
To synthesize the video texture, between frames of the same video. We represent the video as a graph where the individual frames are nodes and the edges represent transition probabilities predicted by our video-specific model. We generate output videos (or textures) by randomly traversing edges with high transition probabilities. We additionally incorporate deep video interpolation into our contrastive video textures framework to suppress visual discontinuities and to allow for large transitions. Our proposed method able to synthesize realistic, smooth, and diverse output textures variety of domains, including dance and music videos as shown this website. Fig. illustrates the distinction between video cenneratioo/predictionn video textures and our contrastive model.
the video-specific model to compute probabilities of transitioning
We also extend our model to audio conditioned video synthesis task. Given a source video with associated audio and a new conditioning audio not in the source, we synthesize a new video that approximately matches the conditioning audio. demonstration of this task is shown at this link. We modify the inference algorithm to include an additional constraint that the predicted frame's 's audio should match the conditioning audio. We trade off between temporal coherence (frames predicted by the constrastive video texture model) and audio similarity (frames predicted by the audio matching algorithm) to generate videos which align well with the conditioning audio and are also temporally smooth.
We assess the perceptual quality of the synthesized textures by conducting human perceptual evaluations comparing our method to a number of baselines. In the case of unconditional video texture synthesis, we compare to the classic video texture algorithm (Schödl et al., 2000) and variations to this which we describe Sec. 4. For the audio conditioning setting, we compare to three different baselines: classic video textures with audio conditioning, visual rhythm and beat (Davis & Agrawala, 2018), and a random baseline. Our results confirm that our method perceptually better than all the baselines.
2 CONTRASTIVE VIDEO TEXTURES
We propose a parametric learning-basee approach for video texture synthesis. At a high-level, we fit example-specift bi-gram model (i.e. Markov chain) and use it re-sample input frames, producing diverse and temporally coherent video. In the following, we first define the bi-gram model, and then describe how to train and sample from it.
2

---

Under review conference paper at ICLR 2021
NCE Loss
((V))
(V
(Van)
[(V)
vV
Vis
V
o o(V241)
Input
Bi-Gram Model
Output
Contrastive Model
Training
Video Texture Synthesis
Figure 2: Contrastive Video Textures: We extract overlapping segments from the video and fit bi-gram model trained using NCE loss which learns representations for query/target pairs such that given a query segment Vi, <(V.) is similar to positive segment vb(4+)1) and dissimilar to negative segment vb(V)) where j [ []] and j j i, i 1. Video Texture Synthesis. During inference, we start with random segment Vi [1], compute (V2) and (V,) and calculate the edge weights similarity between (VV)) and (V,). We randomly traverse to (purple arrow) one of the edges that has high probability to reach [2]. We denote higher weight edges in green and lower weighted edges red and the thickness correlates with the probability. No
edge indicates zero similarity. The randomly chosen segment [2] is appended to the output as V4+1 and the process is repeated (shown by the orange arrow) with [2] as the query.
Given an input video, we extract N overlapping segments denoted by V; where E [], with sliding window of length W and stride Consider these segments to be the states of a Markov chain, where the probability transition computed by a deep similarity function parameterized by encoders & and d/:
P(V++ x exp(sim(((V))) (V[++1))/))
(1)
Fitting the transition probabilities amounts to fitting the parameters of á and dh which here will take form of a 3D convolutional network, by maximizing the log-likelihood of the sequence under the model:
N
N llog exp(sim(((V))) (V++1)))/)
C(V.0) =
log P(V+nlV))
(2)
10 cxp(sim( (V(V.) (V.))/r
where denotes temperature term that modulates the sharpness of the softmax distribution. As the complexity increases with number of negatives in the denominator, for efficiency, we use negative sampling (Mikolov al., 2013) to approximate
Eq 2. Fitting the encoder in this manner amounts to learning a video representation by contrastive learning, where the positive is the segment that follows, and negatives are sampled from the set of all other segments. The encoder thus learns features useful for predicting the dynamics of phenomena specific to the input video.
Given that we the model on a single video, it is important that we ensure there is enough entropy in the transition distribution in order to ensure diversity in samples synthesized during inference. While assume that our imput video sequence exhibits sufficient hicrarchical, periodic structure to ensure repetition and
we also directly adjust the conditional entropy of the model through the softmax temperature term т. As we will see in Sec. 2, the encoder used for conditioning and prediction also plays a role in ensuring diversity in the sampling distribution. An overview of method is provided in Fig. 2.
3

---

Under review conference paper at ICLR 2021
Video Texture Synthesis. To synthesize the texture, we represent the video as graph, with nodes as segments and edges indicating the transition probabilities computed by our model. We randomly select a query segment Vi among the segments of the video and set the output sequence to all the W frames in Vi. Next, our contrastive model computes (VV) and (V2)) for all segments in the video and updates the edges of the graph with the transition probabilities, given by sim( (mV(),)) (V;)). The target segment with the highest transition probability is chosen as the positive segment. We then append the last number of frames in the positive segment to the output. This predicted positive segment Vi+1 is again fed into the network as the query and this is repeated to generate the whole output in autoregressive fashion. This approach would regurgitate the original sequence, as the model was trained to predict Vi+1 as the positive segment given V as the query. The edge with the maximum weight is always directed to the next segment in the video.
In order to introduce variance in the generated textures, we select segments which are similar to the positive segment V4+1. First, we vary the temperature term r to adjust the weights of the graph. The temperature term controls the entropy of the output distribution. A lower temperature would flatten the prediction probabilities/neeress the entropy and reduce the difference in probabilities of the positive segment and segments similar to it. We then threshold the probabilities and set values to zero if they are less than 5% of the max weighted edge connecting V to any other node V,, we set,
sim(o(V)). (VV)) 0 where (VV( < max (sim(Ø(V)), (Vi)) t% 1= I......N NN
Next, we randomly select a frame to transition to from the edges with non-zero probabilities. This introduces variance in the generated textures and also ensures that the transitions are smooth and coherent.
Video Encoding, We use the SlowFast (Feichtenhofer et al., 2019) action recognition framework for encoding the video segments. We introduce two separate query and target multi-layer perceptrons to break the symmetry between the query and target embeddings. This ensures sim(V, V(+1) # sim(V++) V2) which allows us to learn the arrow of time.
Interpolation. For smoother transitions, we also conditionally interpolate between frames of the synthesized texture when there are transitions to different parts of the video. We use a pre-trained interpolation network of Jiang et al. (2018). 1).. We include results both with and without interpolation to show that interpolation helps with smoothing.
3 AUDIO-CONDITIONE CONTRASTIVE VIDEO SYNTHESIS
We extend Contrastive Video Textures to synthesize videos that match a conditioning audio signal. Given an input video and a conditioning audio A we synthesize a new video that synchronized with the audio. We extract N overlapping segments from the conditioning audio, as before. We compute the similarity of the source audio segments A to the conditioning audio segment A by matching them in an embedding space and computing the similarity between the audio segments. We construct transition probability matrix T, in the audio space as in Eq.
(i( = sim(pp44)) a + (1 11-aay T)Ta.
(3)
(4)
We compute the transition probabilities T, for the target video segments given the previous predicted segment using the contrastive video textures model (Eq. 2). The joint transition probabilities for segment formulated as a trade-off between the audio conditioning signal and the temporal coherence constraint as in Eq. 4.
4

---

Under review conference paper at ICLR 2021
Table 1: Perceptual Studies for Unconditional Video Textures and Audio Conditioned Video Synthesis.
(a) We show MTurk evaluators textures synthesized by 5 methods ask them to pick the most realistic one. We also report the chance evaluators chose any of the variation of the classic model.
(b) Unconditional: Real Fake study, show evaluators a pair of videos (generated and real video) without labels, ask them to pick the real one. Our method fools evalua- tors more times than Classic.
Real vs. Fake study. We show evaluators pair of videos (generated and real video) without la- bels and ask them to pick the real one. Our method fooled evaluators more of- than did baselines.
(c) Conditional:
Method
Preference %
Method
Real vs. Fake
Method
Real vs Fake
Classic
3.33 2.42 %
Classic Deep
6.66 + 3.37 %
Classic+
11.4 + 4.30%
Random Clip
15.33 +: 5.76% 20.4 6.63%
Classic+
10.95 + 4.22
Classic+
15.7 = 4.92 %
Audio NN
Classic++
1:977 %
Contrastive
45.7 + 4.3%
Contrastive
26.74 + 6.14%
Any Classic
30.48 6.22
Contrastive
69.52 = 6.22 %
Audio encoding. We embed the audio segments using the VGGish model (Hershey et al., 2017) pretrained on AudioSet (Gemmeke et al., 2017). We remove the last fully connected layer from the model and use the output of the final convolutional layer as audio features. We describe details of the implementation of our method in Sec. A.1.
4 EXPERIMENTS
We curate a dataset of 70 videos from different domains such as dance and musical instruments including piano, guitar, suitar, drums, flute, ukelele, and harmonium. A subset of these vidcos were randomly sampled from the PianoYT dataset (Koepke et al., 2020) and the rest were downloaded from YouTube. We first worked with 40 of the 70 videos and used those to tune our hyperparameters. We then tested it on the remaining 30 without any tuning. Our dataset consists of both short videos which are 2-3 minutes long and long videos ranging from 30- 60 mins' We conduct perceptual evaluations on Amazon MTTurk to qualitatively compare the results from our method to different baselines both the aforementioned cases.
4.1 UNCONDITIONAL VIDEO TEXTURE SYNTHESIS
To show the effectiveness of our method, we compare our results to the classic video textures algorithm (Schödl al., 2000). Additionally, we compare three variations of the algorithm. Classict appends multiple frames to the output sequence instead of single frame, Classic++ adds stride while filtering the distance matrix and Classic Deep uses ImageNet pretrained ResNet features instead of raw pixel values. The algorithm and its variants are described in Sec. A.2.
Table reports the results from a perceptual study on Amazon MTurk where evaluators where shown textures generated by all five methods and asked to choose the one they found most realistic. Our contrastive model surpasses all baselines by a large margin and was chosen 69. 69.52% the time. Since the classic models are similar, also report all variations of classic combined. They chosen 30.48% of the time.
We include qualitative video results for Classic, Classic+, Classic+ and contrastive in Fig. 3. Video in Fig. 3a produced by contrastive method is most realistic and has seamless transitions. We notice in Fig. 3b that
We will releasc the vidcos along with the trained models for each, along with reference implementation of our method.
5

---

Under review conference paper at ICLR 2021
the classic algorithm often transitions frames in and around the target, producing textures which are jarring. Classic+ shown in Fig. 3c has slightly improved quality compared to classic but is still nowhere close to our contrastive method. Classic shown in Fig. 3d shows no significant improvement.
Additionally, we conduct real vs. fake studies where the evaluators are shown the ground truth video and synthesized texture and asked to pick the one they think real. Our method able to fool evaluators 45.7% of the time whereas the best baseline (Classic+) is able to fool the evaluators only 15.7% of the time.
Both the qualitative and quantitative comparisons clearly highlight the issues with the classic model and emphasize the need to learn the feature representations and the distance metric as we do in our contrastive method.
(a) Contrastive
(b) Classic
(c) Classic+
(d) Classic++
Figure To play the videos, please view Acrobat Reader and click on the figure. Videos are also included at this [website]. Qualitative comparison of method (Contrastive) the baselines Classic, Classic+ and Classic++ are shown. The red bar at the bottom indicates the part the original video being played. The results from all 3 baselines choppy. Our method selects good transitions in the video be seen by the red bar moving but the transition is seamless.
4.2 AUDIO CONDITIONED CONTRASTIVE VIDEO TEXTURES
For audio conditioned video synthesis, we choose subset of 30 videos from the above 50 and randomly paired them up with songs from the same domain (e.g. source piano video is paired with conditioning audio of piano). Using this strategy, we created 50 source video conditioning audio pairs. As described in Sec. 3, we extend the contrastive method to synthesize textures given a conditioning audio signal. We compare audio conditioned video textures synthesized by our method to four baselines and report results from a perceptual evaluation.
To show it is easy to identify when a video and the corresponding audio are out of syne, we randomly choose a portion of the source video to match the conditioning audio.
Random Clip.
Video Textures with Audio Conditioning. We add audio conditioning to the classic video textures algorithm. For this, we divide the conditioning audio into segments and find neighbous the source audio. Then we combine these distances with the distance matrix D calculated by video textures.
Visual Rhythm and Beat. We use the approach Davis & Agrawala (2018) to synchronize the source video with the audio beats. This method works by changing the pacing of the video (slowing it down and speeding it up) so that the visual and audio beats are more closely aligned.
Audio Nearest Neighbours. We include comparisons to nearest neighbor baseline that works by computing the similarity between the conditioning audio signal and segments of the source audio the same length, and then choosing the video clip of the closest match.
We conduct perceptual studies comparing the audio conditioned video textures synthesized by our contrastive shown two videos with the conditioning audio,
model and all of the baselines. The evaluators synthesized by our method and the other by the baseline. They were asked to pick the video that they felt was more in sync with the audio. Our method was chosen 92% of the time when compared with Classic+Audie
6

---

Under review conference paper at ICLR 2021
84% of the time when compared with VRB, 70% of the time when compared with Random Replay and 66% of the time when compared with Audio NN. As shown in Tab. Ic, we conducted real vs. fake study comparing the ground truth videos with the synthesized videos from contrastive and the two best baselines (Random Clip and Audio NN). While Random Clip and Audio NN beat the ground truth only 15.33% and 20.4% respectively, our method was able to fool evaluators 26.7 74% of the time.
No Soand. Cop Mart.
Depasted
Conditiorng
Conditioning Autio
Audi Conditonod Video Texture
Conditioeed
Contrastive
Hanca struming..
Same note reasated twic.
Classic Audio
Strummiing
Nate is not repeatedd
Hardom note playes.
VR&B
Talerg
Playa section
(A)
(B)
Figure 4: Qualitative comparison of audio conditioned video textures synthesized by Classic, Random, Visual Rhythm and Beat (VRB) and our Contrastive model. (A) The conditioning audio waveform shows a gap in the audio where no music is being played. Our model is able to pick up on that and the corresponding video that is synthesized has hands in the air and strumming. However both Random and Classic+Audio show strumming, which is not in sync with the audio. The result using VRB shows the person talking. (B) The conditioning audio waveform has the same chord repeated twice. The video synthesized by our model reflects this, and we observe the same frames (1 and 2) repeated again. Classic+Audio plays the note just once and Random plays different note. VRB result contains a region without audio where the person isn't playing anything, which is out of sync with the conditioning audio.
We show qualitative results comparing our method the five baselines described above in Fig. 4 and also include additional results at this [website]. In Fig. 4 (A), the conditioning audio signal has gap/break in the audio where no music being played. We see the output produced by the contrastive model semantically meaningful and aligns best with the audio. Random Clip chooses random segment which has strumming and thus fails to align with the audio. Similarly class+audio chooses frames that don't 't correlate with the audio. VRB doesn't It capture semantics as it only speeds up or slows down the video to better match the audio beats. Similarly, in Fig. 4(B) we see that our contrastive method able to pick up on repeated chords in the conditioning audio signal while no other method is able to do that. Furthermore, importance of adding video interpolation through examples show here. Through more examples listed at this website we show that the videos synthesized by contrastive model are more in syne with the conditioning audio. For example, it identifies gaps in the audio, repeated chords, and change of pace. Our model better learning audio-visual correlations.
also illustrate the
7

---

Under review conference paper at ICLR 2021
5 RELATED WORK
Texture Synthesis. All texture synthesis methods aim to produce textures which are sufficiently different from the source yet appear to be produced by the same underlying stochastic process. Texture synthesis methods can be broadly classified into two categories: non-parametric and parametric. Non-parametric methods focus modeling the conditional distribution of the input images and sample information directly from the input. The sampling could be done pixel-wise (Efros & Leung, 1999; Wei & Levoy, 2000) or patch-wise (Efros & Freeman, 2001; Kwatra et al., 2003) for image texture synthesis. Inspired by these con-mmaate approach for synthesizing video texture with by
works, (Schödl et al., 2000) proposed finding novel, plausible transitions in an input video. (Schödl & Essa, 2001; 2002; Efros et al., 2003) were interesting extensions of the same. (Wei et al., 2009) provides an extensive review of example-based texture synthesis methods. Parametric approaches, on the other hand, focus on explicitly modeling the underlying texture synthesis process. (Heeger & Bergen, 1995; Portilla & Simoncelli, 2000) were the first to propose parametric image texture synthesis by matching statistics image features between source and target images. This later inspired (Gatys et al., 2015), which used features learned using convolutional neural network for image texture synthesis.
Video Generation and Video Prediction. The success of (GANs) (Goodfellow et al., 2014) and Variational Autoencoders (VAEs) (Kingma & Welling, 2013) in image generation (Zhu et al., 2017; Karras et al., 2019; Park et al., 2019) inspired several video generation methods, both unconditional (Vondrick et al., 2016; Clark et al., 2019; Tulyakov et al., 2018; Saito et al., 2017 7) and conditional (Wang et al., 2018;; Chen et al., 2019; Wang al., 2019; Zhou et al., 2019; Gafni et al., 2020; Mallya et al., 2020). A common type of conditional video synthesis includes future frame prediction given past frames (Denton & Birodkar, 2017; Srivastava al., 2015; Kalchbrenner et al., 2017). Even the most recent video prediction (Ye al., 2019; Xu et al., 2020) techniques produce blurry outputs and fail to generate frames beyond a few seconds. These methods are far from generating realistic videos and oftentimes produce outputs which are blurry and low-resolution, especially in the unconditional case. This is because videos are higher dimensional and modeling spatio- temporal changes and transition dynamies is more complex. As such, these methods are expected to fail when applied to our task of video texture synthesis. There are also a few recent works which condition the video generation on an input signal such as text (Li et al., 2018), or speech (Oh et al., 2019; Kim et al., 2018; Ephrat et al., 2018), 8), or single image (Shaham et al., 2019). Our method is similar to SinGAN (Shaham et al., 2019) in that we train our network on a single input, though on a video instead of an image and without an adversarial loss.
Contrastive Learning. Recent contrastive learning approaches (Hénaff et al., 2019; Chen et al., 2019; 2020; He et al., 2020) have achieved success in classic vision tasks proving the usefulness of the learned representations. (Misra et al., 2016) train a network to determine the temporal ordering of frames in a video and (Wei et al., 2018)'s self-supervised model learns to tell if a video playing forwards/backwards. Here, we use contrastive learning to fit a video-specific bi-gram model. Our network maximizes similarity between learned representations for the current and next frame. Unlike (Oord et al., 2018), our goal is not to generate frames from latent representations, but rather to resample from the input video.
6 CONCLUSION
We presented contrastive video textures, a bearning-based approach for texture synthesis applied to audio conditioned video generation. Our method fits an input-specific bi-gram model to capture the dynamics of a video, and uses it to generate diverse and temporally coherent textures. Further, we introduce the task of audio conditioned video texture synthesis useful application of video textures. We show that model outperforms a number of baselines on perceptual studies. We hope this work inspires research in texture synthesis based generation.
8

---

Under review conference paper at ICLR 2021
REFERENCES
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297 2020.
Yang Chen, Yingwei Pan, Ting Yao, Xinmei Tian, and Tao Mei. Mocycle-gan ganee::egan:.... Unpaired video-to-video translation. 2019. 8
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets. arXiv preprint arXin:1907.065771 2019. 8
Abe Davis and Maneesh Agrawala. Visual rhythm and beat. ACM Transactions on Graphics (TOG), 2018. 2,
6
Emily L Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 8
Alexei A Efros and William T Freeman. Image quilting for texture synthesis and transfer. In ACM SIGGRAPH, 2001.
Alexei A Efros and Thomas K Leung. Texture synthesis by non-parametric sampling. In IEEE International Conference on Computer Vision (ICCV), 1999.
Alexei A. Efros, Alexander C. Berg, Greg Mori, and Jitendra Malik. Recognizing action distance. In
IEEE International Conference on Computer Vision (ICCV), pp. 726-733, Nice, France, 2003, 8
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation. ACM SIGGRAPH,, 2018.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In IEEE International Conference on Computer Vision (ICCV), 2019. 4
Oran Gafni, Lior Wolf, and Yaniv Taigman. Vid2game: Controllable characters extracted from real-world videos. In International Conference on Learning Representations (ICLR), 2020. 8
Leon Gatys, Alexander Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks.
In Advances in Neural Information Processing Systems (NeurIPS), 2015. 8
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labele dataset for audio events. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017. 5
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems (NeurIPS), 2014. 1,8
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020. 8
David Heeger and James R Bergen. Pyramid- texture amalysis/synthesis. In ACM SIGGRAPH, pp- 229-238, 238, 1995.
9

---

Under review conference paper at ICLR 2021
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:19050092 2019.
Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort Gemmeke, Aren Jansen, R Channing Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Scybold, et al. Can architectures for large-scale audio classification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
2017. 5
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming- Ming-Hsuan Yang, Erik Learned and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 4
Nal Kalchbrenner, Aaron van den Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals. Alex Graves, and Koray Kavukcuoglu. Video pixel networks. International Conference on Machine Learning (ICML), 2017.
8
Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019. 8
Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojeiech Matusik. On learning associations faces and voices. In Asian Conference on Computer Vision, 2018. 8
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations (ICLR), 2013. 1,8
A Sophia Koepke, Olivia Wiles, Yael Moses, and Andrew Zisserman. Sight sound: An end-to-end approach for visual piano transcription. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020. 5
Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. Graphcut textures: image and video synthesis using graph cuts. ACM Transactions Graphics (TOG), 2003. 8
Hiin-Ying Lee, Xiaodong Yang. Ming- Yu Liu, Ting-Chun Wang. Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing to music. In Advances in Neural Information Processing Systems (NeurIPS), 2019. 1,2
Yitong Li, Martin Rengiang Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In AAAI, 2018.
Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming-Yo Yu Liu. World-consistent video-to-video synthesis. In European Conference on Computer Vision (ECCV), 2020. 1, 2, 8
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (NeurIPS), 2013. 3
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In European Conference on Computer Vision (ECCV). Springer, 2016.
Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T Freeman, Michael Rubinstein, and Wojciech Matusik. Speech2face: Learning the face behind voice. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 8
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 100.00/807003740 2018. 8
10

---

Under review conference paper at ICLR 2021
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu, Semantic image synthesis with spatially- adaptive normalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 8
Javier Portilla and Eero P Simoncelli. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision (ICC), 2000. 8
Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In IEEE International Conference on Computer Vision (ICCV), 2017.
Arno Schödl and Irfan A Essa. Machine learning for video-based rendering. In Advances in Neural Information Processing Systems (NeurIPS), 2001. 8
Arno Schödl and Irfan A Essa. Controlled animation video sprites. In ACM SIGGRAPH. 2002.
Arno Schödl, Richard Szeliski, David H Salesin, and Irfan Essa. Video textures. In ACM SIGGRAPH, 2000. 1, 5, 8, 12, 13
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from single natural image. IEEE International Conference on Computer Vision (ICCV), 2019.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations
using LSTMs. In International Conference on Machine Learning (ICML), 2015. 8
Sergey Tulyakov, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 8
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 1, 8
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018a.
Ting-Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis. Advances in Neural Information Processing Systems (NeurIPS), 2018b. 1,
Ting-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8052-8060, 2018.
Li-Yi Wei and Mare Levoy. Fast texture synthesis using tree-structured vector quantization. In ACM SIGGRAPH, 2000.
Li-Yi Wei, Sylvain Lefebvre, Vivek Kwatra, and Greg Turk. State of the art in example-based texture synthesis. 2009.
Jingwei Xu, Huazhe Xu, Bingbing Ni, Xiaokang Yang, and Trevor Darrell. Video prediction via example guidance. International Conference on Machine Learning (ICML), 2020. 2,
Yufei Ye, Maneesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. IEEE International Conference Computer Vision (ICCV), 2019.
11

---

Under review conference paper at ICLR 2021
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Dance dance generation: Motion transfer for internet videos. arXiv preprint arXiv: 1:XX::/904000/999 2019. 8
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In IEEE International Conference on Computer Vision (ICCV), 2017.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
We divide the video into overlapping segments using a window of length 0.5 seconds and a stride of 0.2 seconds. Depending on the frame rate of the video, this yields segments with varying number of frames.
Video Encoding. Each of these segments then encoded by SlowFast model into R512, Next the query and target are passed through two separate MLPs, each consisting of 3 linear layers interspersed with ReLU activations. The MLP maintains the size of the embedding such that the final outputs, (55) and (5)) are in R5122 We initialize the SlowFast model with weights pretrained on Kinetics dataset and fine tune the whole network end-to-end. We use SGD optimizer with a learning rate le-4 for the Slow Fast model and a rate le-3 for the MLP.
Audio Encoding, The VGGish model is initialized with weights pretrained on AudioSet. The learned audio representations for the source audio segments ((Aa) and the conditioning audio segments (p(Af) are in IR
Interpolation. We typically set the number of interpolated frames to be added to be 4. This increases the FPS of the synthesized video by a factor of (i.e. 2 frames is converted to 4). When there is no jump, the frames are repeated 3 times, to ensure the overall FPS of the video the same.
Temperature tuning and threshold. For training the contrastive video texture model, experimented with multiple values of temperature (r) and found 0.1 to work the best. At test time, setting the temperature to 0.1 and threshold (th) to 0.0 regurgitates the original video. Increasing the temperature and threshold increases the entropy and allows for more random transitions in the output. We found that the number of transitions the temperature is fairly constant across all videos and include details in Sec. 4. We found temperature of 0.3 and a temperature of th is optimal for synthesizing videos which are temporally smooth yet different from the original video.
Combining Ta and T.. Smaller values of a allow for better audio-video synchronization but at the cost of continuity the video. For most results reported here, we set a to either 0.5 or 0.7.
A.2 UNCONDITIONAL VIDEO TEXTURES: BASELINES
We first provide an overview of the classic video texture algorithm introduced in (Schödl et al., 2000) followed by the descriptions of the baselines.
Classic Video Textures The classic video textures algorithm proposed in (Schödl et al., 2000) computes a distance matrix D of pairwise distances between all frames in the video. The distance is computed as the 1.2norm of the difference RGB values between pairs of frames. Next, the distance matrix D filtered with 2 4-tap filter with binomial weights to produce matrix D'. The stride used while filtering is 1. If the input video is short, oftentimes this approach would not be able to find good transitions from the last frame and reaches dead end. To avoid this, they use Q-learning to predict the anticipated (increased) "future cost" of choosing a given transition, given the future transitions that such a move might necessitate. This gives rise D" The transition probabilities P" are computed from D" as PM sxp(-DD'a,,/0
12

---

Under review conference paper at ICLR 2021
To synthesize a texture, frame i is chosen at random. This is added to the output sequence of frames. After displaying frame i, the next frame is selected according to Pt.j To improve the quality of the textures and suppress non-optimal transitions they adopt two step pruning strategy. First, they choose the optimal transition with the maximum transition probability, next they set all probabilities below some threshold to zero and pick a random transition from the non-zero probabilities. The output sequence is generated one frame at time.
We generate textures using the algorithm above. Following the convention in (Schödl et al., 2000), we set sigma to be small multiple of the average (non-zero) values in the distance matrix. We tune this small multiple and the threshold on the train set and use the same values on the test set.
For an apples-to-apples comparison, we fix some of the shortcomings of the classic algorithm and compare to these modified versions described below.
Classic+. During inference, the number of frames appended to the output texture is the stride with which the initial video was segmented. While this stride is I for the classic algorithm, it is greater than I for our contrastive method. To ensure the difference in perceptual quality isn't due just the changes in stride length, we modify the classic algorithm to increase the stride during inference to be the same as our contrastive model. The distance matrix is still computed pairwise between frames but instead of appending single frame, we append stride number of frames to the output. This stride is set to be the same value as our contrastive model.
Classic++. To further reduce the gap between classic+ and contrastive method, we apply stride while filtering the distance matrix D with the tap filter. This is equivalent to the approach we use in contrastive, which is dividing the video into overlapping segments of window W and stride s.
Deep Classic. Additionally, we also tried replacing the frame-wise features in the classic algorithm with learned representations from pre-trained resnet.
A.3 TRANSITION PROBABILITIES
We compare the transition probability matrices generated by both classic and contrastive methods. Fig. 5a shows the transition probability matrices for two different videos generated by Classic (la, 2a) and Contrastive (Ib, 2b) methods. It can be observed from the diagonal lines in the figure that the classic method assigns the same value to multiple frames whereas our method picks up on subtle differences and assigns different scores. This emphasizes that the distance metrie learned by our method is better at distinguishing frames.
Fig. 5b shows the variation in the number of transitions with sigma for the classic technique and with temperature for the contrastive technique. Number of transitions increases linearly with temperature for contrastive method whereas for the classic technique we found no such correlation. Moreover, a temperature of 0.3 and threshold of 0.01 results in 15-20 jumps across all videos. There was no such strong correlation for the classic technique, making it necessary to tune hyperparameters on a per video basis.
Failure Cases. As shown here, our method fails when the changes in background/lighting are too large be smoothened by a interpolation model. This is observed in dance videos where there's 's large movements across segments. VRB and Random Replay work well in such cases. VRB is designed to align dance moves with the music and hence clearly performs the best. Random works well too as good music and good dancing go well together.
Fig. 6 shows some transitions in the video textures generated by contrastive model.
13

---

Under review conference paper at ICLR 2021
Figure 5: Left. Transition probability matrix for different videos (in each row) for both classic and contrastive methods. Right. Number of transitions vs Sigma for Classic and Number of transitions vs Temperature for Contrastive.
Classic
Contrastive
Classio
Contrastive
Video
Vidwo
1(+)
1/63
Ogmn
Texterstars
Video
Video
2(8]
Teepester
(b)
Figure 6: The figure shows frames from two different videos synthesized by our method. Red bar indicates position of the original video being played. The transition happens at the third frame and is seamless in both cases. The first forward jump and the second is backward jump.
14


Reviews:
[{"review_id": "zLpFPhQFarT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a non-parametric approach for video generation, i.e., video frame (un)conditional resampling. The proposed method is inspired by Video Textures (Sch¨odl et al., 2000), which synthesizes new videos by stitching together snippets of an existing video. Comparing to existing 'video textures' methods, the authors mainly made two improvements/contributions. (i) a new pipeline for modeling and calculating probabilities of transitioning between frames of the same videos. Specifically, giving a video clip, the authors first extract overlapping segments from it and fit a bi-gram model. The adjacent segments are regarded as positive pairs with high transitioning probability, yet other random sampled pairs are negative pairs. Similar to contrastive learning works, NCE loss is utilized to train the bi-gram model. (ii) Extending the model to a conditional situation and performing the task of audio conditioned video synthesis. The authors made a trade-off between the audio conditioning signal and the learned transition probabilities. Finally, experiments including multiple qualitative resampled videos and quantitative user studies were provided. And the proposed method demonstrated promising performance. \n\nOverall, this work is well organized and easy to follow. The motivation is clear and ideas are simple and useful. According to the provided video examples, the proposed method achieved great performance. However, I still have some concerns:\n\nIn the experiments of unconditional setting, the authors included Classic, Classic+, and Classical++ as their comparisons. However, all recent (here, I mean in the past decade) methods that related to video generation are missed. First, although there may not exist any resampling method that can directly perform the video texture synthesis, I believe many related graph-based methods could be used to model the transition probabilities of frames. I think the authors may at least discuss these works. Second, although the authors pointed out that the video resampling (textures) strategy is different from the recent generation-based strategy, they should provide visual results/comparisons to support their claims. In recent years, some methods were proposed to synthesize videos with dynamic textures, such as flame, wave, flash, etc. I am interested in if the proposed method can perform these tasks. In addition, the authors may further discuss the limitations of the proposed method. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "F2UGOUpcnlr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Strengths\n\n(1) Improve the classic video texture synthesis method Video Textures by replacing pixel similarity with a distance metric learning to measure the transition probabilities \n\n(2) Extend the proposed approach to audio conditioned video synthesis\n\n(3) Outperform the competing algorithms on a set of evaluations\n\n\n** Weaknesses\n\n(1) It seems a strong limitation that the proposed approach is not able to generalize to different videos or has to be video-specific (i.e., train a model on each input video). \n\n(2) It is not true that existing methods fail to generate more than a short sequence of frames, e.g., (Lee et al. 2019) in theory can generate videos with arbitrary lengths. \n\n(3) The pre-trained interpolation network (Jiang et al. 2018) seems a quite important component for the proposed algorithm to generate smooth videos. No analysis is provided to show the contribution of this module.\n\n(4) Lacks of analysis or comparisons to justify some important hyper-parameters, e.g., (1) how the softmax temperature term impacts the synthesis results, (2) how t% is chosen to threshold the transition probabilities.   \n\n(5) For video encoding, two separate subnetworks are used to break the symmetry between query and target embeddings, which makes sense considering temporal ordering. However, why audio embeddings take a same encoding network?\n\n(6) For unconditional video synthesis, the musical instrument playing videos mostly contain small motions. Unclear how this approach could handle general videos exhibiting with large actions. \n\n(7) It is claimed that the approach is able to produce infinite video, however the content is constrained in the input video, so the variation is limited. \n\n(8) What are the results if the classic methods also use the interpolation network?", "extra_scores": {}, "extra_text": {}}, {"review_id": "DNFQvDfn0Ai", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of this paper: In this work, the authors propose a method to learn to generate long-range video sequences. The general idea is starting from a prior work (Video Textures) and extending this work with a learning framework. Specifically, during training a model is used to learn the transition probability between different video segments. During inference, long-range video synthesis is achieved through iterative sampling of new video segments. To guarantee the smoothness of the transition between different segments, an existing interpolation method is used to connect these video segments in a sequential order.\n\nPros:\n\n+ quality: This paper is overall easy to read. The motivation behind this work is clearly presented, i.e., to synthesize long-range video sequences. In the introduction part, the authors present the basis of their work (Video Textures) and present a comprehensive comparison with previous works. The authors also present sufficient qualitative results to demonstrate the superiority of their work.\n\n+ clarity: The pipeline of the proposed method is clearly presented in the method part. The general framework is very straightforward. The analysis of the quantitative and qualitative results is convincing and logical. \n\nCons:\n\n- originality: This work is more like a simple extension of the previous work (Video Textures) with limited novelty. I am confused about the difference of the basic formulation of the core  transition probabilities (Eq. 1 and 2). Are they different from the work (Video Textures) or not? It is highly recommended that the authors could present more comparison with the previous baselines in both general idea and model details. Moreover, the video interpolation is directly borrowed from previous work without further improvements, where I think is still challenging and worth to explore. The audio conditioned video synthesis part also seems like an extra module which does not influence the completeness of the whole model if not included.\n\n- significance: I have carefully checked the quality of the generated video sequences, which are not so satisfying. First, these are noticeable discontinuity between sampled video segments. Second, this method seems to be example-specific, which needs retraining if fed a new video sequence. The scalability of this method is limited. Third, the video content is directly sampled from seen sequence, where the diversity is constrained to the given video.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially had mixed reviews (4,5,6).  The main issues raised were:
1) limited novelty (re-using/integrating components) [R2];
2) limited generalization ability since the model needs to be retrained on every video [R2, R3];
3) limited applicability - experiments limited to certain domain of video, while results on videos with large motion are not convincing [R2, R3];
4) missing ablation studies / experiments [R3, R4].

The author response partially addressed some concerns, but the main points 1-3 are still problematic. In addition, the AC noted that the technical aspect was lacking:

- Training with contrastive loss on a single video may likely overfit the embedding to the video, which leads to a meaningless embedding where all non-neighboring segments are orthogonal in the embedding space. While changing the softmax temperature can yield higher entropy transition probabilities, the induced probability distribution is probably highly noisy. It would be better to train this on a large video corpus, which will prevent overfitting. Also contrastive loss is typically used to build a discriminative embedding space for classification/recognition, not a smooth embedding space for generation (where distances between embedding vectors are strongly correlated to similarity). Thus some other embedding smoothness terms could be added during contrastive learning.
- The learning is only on the transition probabilities, while the video generation is separate. It would have been more convincing to learn the transition probabilities with the video generation process in an end-to-end manner. Perhaps a discriminator could be placed after the video generator so that the transition probabilities could be learned so as to better mimic real video. Other loss terms based on video temporal smoothness could also be added ensure smoother transitions between clips (e.g., motion consistency).

The negative reviewers remained unconvinced by the author response, and the AC agreed with their concerns. Thus, the paper was recommended for rejection.
Decision: Reject