Contrastive Video Textures

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) and Variational Autoencoders ( VAEs ) ( Kingma & Welling , 2013 ) have achieved great success in generating images “ from scratch ” . While one might have hoped that video generation would be a simple extension of image-generation methods , this has not been the case . A major reason is that videos are much higher dimensional than images , and producing correct transitions between frames is a difficult problem . While video generation ( Vondrick et al. , 2016 ; Mallya et al. , 2020 ; Lee et al. , 2019 ; Wang et al. , 2018a ; b ) has shown some success , videos generated using such methods are relatively short and are unable to match the realism of actual videos . In comparison , classic non-parametric video synthesis methods from two decades ago , most notably Video Textures ( Schödl et al. , 2000 ) , are much simpler and can often produce videos of arbitrary lengths . In these models , a new plausible video is generated by stitching together snippets of an existing video . While video textures have been very successful on simple videos with a high degree of regularity , they use simple Euclidean pixel distance as a similarity metric between frames , which causes them to fail for less constrained videos containing irregularities and chaotic movements , such as dance or playing a musical instrument . They are also sensitive to subtle changes in brightness and often produce jarring transitions . In this work , we propose Contrastive Video Textures , a non-parametric learning-based approach for video texture synthesis that overcomes the limitations of classic video textures . As in Schödl et al . ( 2000 ) , we synthesize textures by resampling frames from the input video . However , as opposed to using pixel similarity , we learn feature representations and a distance metric to compare frames by training a deep model on a single input video . The network is trained using contrastive learning to fit an example-specific bi-gram model ( i.e . a Markov chain ) . To synthesize the video texture , we use the video-specific model to compute probabilities of transitioning between frames of the same video . We represent the video as a graph where the individual frames are nodes and the edges represent transition probabilities predicted by our video-specific model . We generate output videos ( or textures ) by randomly traversing edges with high transition probabilities . We additionally incorporate deep video interpolation into our contrastive video textures framework to suppress visual discontinuities and to allow for large transitions . Our proposed method is able to synthesize realistic , smooth , and diverse output textures on a variety of domains , including dance and music videos as shown at this website . Fig . 1 illustrates the distinction between video generation/prediction , video textures and our contrastive model . We also extend our model to an audio conditioned video synthesis task . Given a source video with associated audio and a new conditioning audio not in the source , we synthesize a new video that approximately matches the conditioning audio . A demonstration of this task is shown at this link . We modify the inference algorithm to include an additional constraint that the predicted frame ’ s audio should match the conditioning audio . We trade off between temporal coherence ( frames predicted by the constrastive video texture model ) and audio similarity ( frames predicted by the audio matching algorithm ) to generate videos which align well with the conditioning audio and are also temporally smooth . We assess the perceptual quality of the synthesized textures by conducting human perceptual evaluations comparing our method to a number of baselines . In the case of unconditional video texture synthesis , we compare to the classic video texture algorithm ( Schödl et al. , 2000 ) and variations to this which we describe in Sec . 4 . For the audio conditioning setting , we compare to three different baselines : classic video textures with audio conditioning , visual rhythm and beat ( Davis & Agrawala , 2018 ) , and a random baseline . Our results confirm that our method is perceptually better than all the baselines . 2 CONTRASTIVE VIDEO TEXTURES . We propose a non-parametric learning-based approach for video texture synthesis . At a high-level , we fit an example-specific bi-gram model ( i.e . a Markov chain ) and use it to re-sample input frames , producing a diverse and temporally coherent video . In the following , we first define the bi-gram model , and then describe how to train and sample from it . Given an input video , we extract N overlapping segments denoted by Vi where i ∈ [ 1 , ... N ] , with a sliding window of length W and stride s. Consider these segments to be the states of a Markov chain , where the probability of transition is computed by a deep similarity function parameterized by encoders φ and ψ : P ( Vi+1|Vi ) ∝ exp ( sim ( φ ( Vi ) , ψ ( Vi+1 ) ) /τ ) ( 1 ) Fitting the transition probabilities amounts to fitting the parameters of φ and ψ , which here will take form of a 3D convolutional network , by maximizing the log-likelihood of the sequence under the model : L ( V , φ ) = N∑ i=1 − logP ( Vi+1|Vi ) = N∑ i=1 −log exp ( sim ( φ ( Vi ) , ψ ( Vi+1 ) ) /τ ) ∑N j=1 [ j 6=i , i+1 ] exp ( sim ( φ ( Vi ) , ψ ( Vj ) ) /τ ) ( 2 ) where τ denotes a temperature term that modulates the sharpness of the softmax distribution . As the complexity increases with number of negatives in the denominator , for efficiency , we use negative sampling ( Mikolov et al. , 2013 ) to approximate in Eq 2 . Fitting the encoder in this manner amounts to learning a video representation by contrastive learning , where the positive is the segment that follows , and negatives are sampled from the set of all other segments . The encoder thus learns features useful for predicting the dynamics of phenomena specific to the input video . Given that we fit the model on a single video , it is important that we ensure there is enough entropy in the transition distribution in order to ensure diversity in samples synthesized during inference . While we assume that our input video sequence exhibits sufficient hierarchical , periodic structure to ensure repetition and multi-modality , we can also directly adjust the conditional entropy of the model through the softmax temperature term τ . As we will see in Sec . 2 , the encoder used for conditioning and prediction also plays a role in ensuring diversity in the sampling distribution . An overview of our method is provided in Fig . 2 . Video Texture Synthesis . To synthesize the texture , we represent the video as a graph , with nodes as segments and edges indicating the transition probabilities computed by our model . We randomly select a query segment Vt among the segments of the video and set the output sequence to all the W frames in Vt. Next , our contrastive model computes φ ( Vt ) and ψ ( Vj ) for all segments in the video and updates the edges of the graph with the transition probabilities , given by sim ( φ ( Vt ) , ψ ( Vj ) ) . The target segment with the highest transition probability is chosen as the positive segment . We then append the last s number of frames in the positive segment to the output . This predicted positive segment Vt+1 is again fed into the network as the query and this is repeated to generate the whole output in an autoregressive fashion . This approach would regurgitate the original sequence , as the model was trained to predict Vi+1 as the positive segment given Vi as the query . The edge with the maximum weight is always directed to the next segment in the video . In order to introduce variance in the generated textures , we select segments which are similar to the positive segment Vi+1 . First , we vary the temperature term τ to adjust the weights of the graph . The temperature term controls the entropy of the output distribution . A lower temperature would flatten the prediction probabilities/increase the entropy and reduce the difference in probabilities of the positive segment and segments similar to it . We then threshold the probabilities and set values to zero if they are less than t % of the max weighted edge connecting Vt to any other node Vj , we set , sim ( φ ( Vt ) , ψ ( Vj ) ) = 0 ∀ j , where sim ( φ ( Vt ) , ψ ( Vj ) ) < max l=1 , ... , N ( sim ( φ ( Vt ) , ψ ( Vl ) ) − t % Next , we randomly select a frame to transition to from the edges with non-zero probabilities . This introduces variance in the generated textures and also ensures that the transitions are smooth and coherent . Video Encoding . We use the SlowFast ( Feichtenhofer et al. , 2019 ) action recognition framework for encoding the video segments . We introduce two separate query and target multi-layer perceptrons to break the symmetry between the query and target embeddings . This ensures sim ( Vi , Vi+1 ) 6= sim ( Vi+1 , Vi ) which allows us to learn the arrow of time . Interpolation . For smoother transitions , we also conditionally interpolate between frames of the synthesized texture when there are transitions to different parts of the video . We use a pre-trained interpolation network of Jiang et al . ( 2018 ) . We include results both with and without interpolation to show that interpolation helps with smoothing . 3 AUDIO-CONDITIONED CONTRASTIVE VIDEO SYNTHESIS . We extend Contrastive Video Textures to synthesize videos that match a conditioning audio signal . Given an input video and a conditioning audio Ac we synthesize a new video that is synchronized with the audio . We extract N overlapping segments from the conditioning audio , as before . We compute the similarity of the source audio segments As to the conditioning audio segment Ac by matching them in an embedding space and computing the similarity between the audio segments . We construct a transition probability matrix Ta in the audio space as in Eq . 3 . Ta ( i , j ) = sim ( ϕ ( A c i ) , ϕ ( A s j ) ) ( 3 ) T = αTv + ( 1− α ) Ta ( 4 ) We compute the transition probabilities Tv for the target video segments given the previous predicted segment using the contrastive video textures model ( Eq . 2 ) . The joint transition probabilities for a segment are formulated as a trade-off between the audio conditioning signal and the temporal coherence constraint as in Eq . 4 . Table 1 : Perceptual Studies for Unconditional Video Textures and Audio Conditioned Video Synthesis . ( a ) We show MTurk evaluators textures synthesized by all 5 methods and ask them to pick the most realistic one . We also report the chance evaluators chose any of the variation of the classic model . Method Preference % Classic 3.33 ± 2.42 % Classic Deep 6.66 ± 3.37 % Classic+ 10.95 ± 4.22 % Classic++ 9.52 ± 3.97 % Any Classic 30.48 ± 6.22 Contrastive 69.52 ± 6.22 % ( b ) Unconditional : Real vs . Fake study . We show evaluators a pair of videos ( generated and real video ) without labels , ask them to pick the real one . Our method fools evaluators more times than Classic . Method Real vs . Fake Classic++ 11.4 ± 4.30 % Classic+ 15.7 ± 4.92 % Contrastive 45.7 ± 4.3 % ( c ) Conditional : Real vs . Fake study . We show evaluators a pair of videos ( generated and real video ) without labels and ask them to pick the real one . Our method fooled evaluators more often than did baselines . Method Real vs Fake Random Clip 15.33 ± 5.76 % Audio NN 20.4 ± 6.63 % Contrastive 26.74 ± 6.14 % Audio encoding . We embed the audio segments using the VGGish model ( Hershey et al. , 2017 ) pretrained on AudioSet ( Gemmeke et al. , 2017 ) . We remove the last fully connected layer from the model and use the output of the final convolutional layer as audio features . We describe details of the implementation of our method in Sec . A.1 .

Summary: In this paper, the authors proposed a non-parametric approach for video generation, i.e., video frame (un)conditional resampling. The proposed method is inspired by Video Textures (Sch¨odl et al., 2000), which synthesizes new videos by stitching together snippets of an existing video. Comparing to existing 'video textures' methods, the authors mainly made two improvements/contributions. (i) a new pipeline for modeling and calculating probabilities of transitioning between frames of the same videos. Specifically, giving a video clip, the authors first extract overlapping segments from it and fit a bi-gram model. The adjacent segments are regarded as positive pairs with high transitioning probability, yet other random sampled pairs are negative pairs. Similar to contrastive learning works, NCE loss is utilized to train the bi-gram model. (ii) Extending the model to a conditional situation and performing the task of audio conditioned video synthesis. The authors made a trade-off between the audio conditioning signal and the learned transition probabilities. Finally, experiments including multiple qualitative resampled videos and quantitative user studies were provided. And the proposed method demonstrated promising performance.