CCVS: Context-aware Controllable Video Synthesis

1 Introduction . Feeding machines with extensive video content , and teaching them to create new samples on their own , may deepen their understanding of both the physical and social worlds . Video synthesis has numerous applications from content creation ( e.g. , deblurring , slow motion ) to human-robot interaction ( e.g. , motion prediction ) . Despite the photo-realistic results of modern image synthesis models [ 38 ] , video synthesis is still lagging behind due to the increased complexity of the additional temporal dimension . An emerging trend is to use autoregressive models , for example transformer architectures [ 68 ] , for their simplicity , and their ability to model long-range dependencies and learn from large volumes of data [ 4 , 16 ] . First introduced for natural language processing ( NLP ) and then succesfully applied to visual data [ 18 ] , the strength of transformers is grounded in a self-attention mechanism which considers all pairwise interactions within the data . The price to pay is a computational complexity which grows quadratically with the data size , which itself depends linearly on the temporal dimension and quadratically on the spatial resolution in the image domain . Although there have been some efforts to reduce the complexity of self-attention [ 11 , 39 , 50 ] , using such methods directly on visual data is still limited to low resolutions and impractical without considerable computational power [ 9 , 79 ] . Some recent works [ 20 , 53 ] address this problem by using an autoencoder to compress the visual data , and apply the autoregressive model in the latent space . This greatly reduces the memory footprint and computational cost , yet , the greater the compression , the harder it is to faithfully reconstruct frames . The corresponding trade-offs may undermine the practical usability of these approaches . GANs [ 26 ] mitigate this issue by “ hallucinating ” plausible local details in image synthesis [ 20 ] . But latent video ∗corresponding author : guillaume.le-moing @ inria.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . transformers [ 53 ] decode frames independently , which prevents local details from being temporally coherent . Hence , using GANs in this setting may result in flickering effects in the synthesized videos . We follow the problem decomposition from [ 20 , 53 ] , but introduce a more elaborate compression strategy with CCVS ( for Context-aware Controllable Video Synthesis ) , an approach that takes advantage of “ context ” frames ( i.e. , both input images and previously synthesized ones ) to faithfully reconstruct new ones despite lossy compression . As shown in Figure 1 , CCVS relies on optical flow estimation between context and new frames , within temporal skip connections , to let information be shared across timesteps . New content , which can not be retrieved from context , is synthesized directly from latent compressed features , and adversarial training [ 26 ] is used to make up realistic details . Indeed , information propagates in CCVS to new frames as previously synthesized ones become part of the context . Like other video synthesis architectures based on autoregressive models [ 53 , 79 ] , CCVS can be used in many tasks besides future video prediction . Any data which can be expressed in the form of a fixed-size sequence of elements from a finite set ( aka , tokens ) can be processed by a transformer . This applies to video frames ( here , via compression and quantization ) and to other types of data . Hence , one can easily fuse modalities without having to build complex or task-specific architectures . This idea has been used to control image synthesis [ 20 , 54 ] , and we extend it to a variety of video synthesis tasks by guiding the prediction with different annotations as illustrated in Figure 2 . Code , pretrained models , and video samples synthesized by our approach are available at the url https : //16lemoing.github.io/ccvs . Our main contributions are as follows : 1. an optical flow mechanism within an autoencoder to better reconstruct frames from context , 2. the use of ancillary information to control latent temporal dynamics when synthesizing videos , 3. a performance on par with or better than the state of the art , while being more memory-efficient . 2 Related Work . Video synthesis . In its simplest form , videos are produced without prior information about their content . GAN-based approaches map Gaussian noise into a visually plausible succession of frames . For example , VGAN [ 71 ] and ProgressiveVGAN [ 1 ] , adapt the traditional GAN framework [ 26 ] from image to video synthesis by simply using 3D instead of 2D convolutions . These approaches , including recent attempts such as G3AN [ 76 ] , are computationally expensive , and , by nature , restricted to synthesizing a fixed number of frames due to the constraints of their architecture . Other approaches predict latent motion vectors with a CNN [ 56 ] , or a recurrent neural network ( RNN ) [ 12 , 57 , 65 ] , and generate frames with individual 2D operations . To avoid the shortcomings of information loss in the sequential processing of RNNs , we use an attention-based autoregressive model instead . Since we forecast temporal dynamics in a compressed space , a large temporal window can be used when predicting new frames , without having to resort to expensive 3D computations . Previous works have attempted to scale video synthesis to higher resolution by using progressive training [ 1 ] , subsampling [ 56 , 57 ] , reducing the dimension of the discriminator [ 36 ] , or redefining the task as finding a trajectory in the latent space of a pre-trained image generator [ 63 ] . Compression , together with efficient context-aware reconstruction allows us to synthesize videos at high resolution . Controllable video synthesis . Some of the aforementioned works [ 12 , 56 , 57 , 76 ] handle synthesis conditioned on a class label . Another popular control is to use a few priming frames to set off the generation process . This task , known as future video prediction , has received a lot of attention recently . Methods based on variational autoencoders ( VAE ) [ 2 , 15 ] have been proposed to account for the stochastic nature of future forecasting , i.e. , the plurality of possible continuations , disregarded in deterministic predictive models [ 21 , 47 , 70 ] . Yet , their blurry predictions have motivated the incorporation of adversarial training [ 44 ] , hierarchical architectures [ 7 , 80 ] , fully latent temporal models [ 22 ] , or normalizing flows [ 42 ] . Another line of work infers spatial transformation parameters ( e.g. , optical flow ) , and predicts the future by warping past frames ( i.e. , grid sampling and interpolation as in [ 33 ] ) in RGB space [ 21 , 24 , 28 , 72 , 81 ] or in feature space [ 45 ] , typically using a refinement step to handle occlusions . Lately , autoregressive methods [ 53 , 79 ] leveraging a self-attention mechanism [ 68 ] have also been applied to this task . Our method benefits from the context-efficiency of approaches based on spatial transformation modules and the modeling power of autoregressive networks . In the meantime , other forms of control with interesting applications have emerged . Point-to-point generation [ 75 ] , a variant of future video prediction , specifies both start and end frames . State or action-conditioned synthesis [ 28 , 48 ] guides the frame-by-frame evolution with high-level commands . Additional works consider video synthesis based on one [ 51 ] or multiple layouts [ 46 , 74 ] , another video [ 8 ] , or sound [ 10 , 34 , 73 ] . To account for the variety of potential controls , we leverage the flexibility of transformers , and propose an unifying approach to tackle all of these tasks . 3 Context-aware controllable video synthesis . 3.1 Overview of the proposed approach . We consider video data , such that an individual frame x is natively an element of X = RH×W×3 , which can be encoded as some feature z in Z = Rh×w×F with reduced spatial resolution and increased number of channels . We assume that we are given Xc in Xm corresponding to m successive frames , and our goal is to synthesize a plausible representation of the following n frames Xs which lies in Xn . Given Xc , we can compute the corresponding features Zc in Zm using some encoder E : X → Z on each frame individually , then use an autoregressive model ( a transformer coupled with a quantization step in our case ) to predict features Zs in Zn formed by the n features corresponding to time ticks m + 1 to m + n. ( Note that the values of m and n can be arbitrary , using [ by now ] traditional masking and sliding window techniques . ) These features can finally be converted one by one into the corresponding frames Xs using some decoder D : Z → X . The overall approach is illustrated by Figure 1 . The use of a quantizer over a learned codebook in our implementation complicates the architecture a bit , but has several advantages , including the reuse of familiar NLP technology [ 68 ] and , perhaps more importantly , affording simple mechanisms for handling different types of inputs ( from images to sound for example [ 9 , 31 ] ) and the intrinsically uncertain and multimodal nature of future prediction ( by sampling different codebook elements according to their likelihood [ 30 ] ) . Concretely , this choice simply amounts to inserting in our architecture , right after the encoder E , a nearest-neighbor quantizer Q : RF → J1 , qK parameterized by an F × q codebook which , given the embedding z of a frame , returns a h × w matrix of corresponding tokens , that is , the indices of the closest entry in the codebook for feature vector zi , j for all spatial locations ( i , j ) in J1 , hK × J1 , wK . We abuse notation , and identify Q with its parameterization by this codebook , so we can optimize over Q just as we optimize over E instead of naming explicitly its parameters in the rest of this presentation . An “ un-quantizer ” U : J1 , qK → RF , also parameterized ( implicitly ) by the codebook and associating with each token the corresponding entry of the codebook , is also inserted right before the decoder D. In this setting , T : J1 , qKm×h×w → J1 , qKn×h×w takes as input a sequence of tokens , and outputs the tokens for subsequent frames , each one of them chosen among q possibilities as either the one with the highest score , or drawn randomly from the top-k scores to account for the multimodal nature of future forecasting ( here k ≤ q is some predefined constant , see [ 30 ] for related approaches ) . The elements of the architecture described so far are by now ( individually ) rather classical , with learnable , parametric functions E , D , Q and T . Besides putting them all together , and as detailed in the rest of this section , we add several original elements : ( a ) The encoding/decoding scheme is improved by the use of two discriminators , ∆i and ∆t respectively , trained in an adversarial manner to ensure that the predicted frames are realistic and temporally consistent . ( b ) The context frames Xc are themselves updated each time a new frame is predicted in an autoregressive manner ( iteratively fill the sequence up to some predefined capacity , then shift to the left , forgetting the first frame and adding the latest synthetic one on the right ) . ( c ) The encoder and decoder are linked through a learnable flow module F , allowing the context frames to guide the prediction of the synthetic ones . ( d ) Additional control variables , ranging from object trajectories to audio tracks , can be used in the form of sequence- or frame-level annotations to drive the synthesis by adding the corresponding tokens to the ones passed on to the autoregressive model T . We detail in this section the concrete components of the approach sketched above , including the autoencoder and quantizer architectures and their training procedure ( Section 3.2 ) , and the implementation of the autoregressive model by a transformer [ 68 ] , illustrated in Figure 2 , which we adapt to account for outside control signals ( Section 3.3 ) , once again with the corresponding procedure . Further architectural choices are also detailed in Appendix A .

Summary: This paper proposes a method for video synthesis that combines an encoder / decoder architecture that computes discrete representation of images with a transformer network that predicts those discrete representations into the future. The encoder and decoder are connected through a novel optical flow module that warps encoder features into the decoder in order to predict how objects will move. In experiments, the authors perform thorough ablations of the proposed method, and show that they outperform most baselines.