CCVS: Context-aware Controllable Video Synthesis

1 Introduction . Feeding machines with extensive video content , and teaching them to create new samples on their own , may deepen their understanding of both the physical and social worlds . Video synthesis has numerous applications from content creation ( e.g. , deblurring , slow motion ) to human-robot interaction ( e.g. , motion prediction ) . Despite the photo-realistic results of modern image synthesis models [ 38 ] , video synthesis is still lagging behind due to the increased complexity of the additional temporal dimension . An emerging trend is to use autoregressive models , for example transformer architectures [ 68 ] , for their simplicity , and their ability to model long-range dependencies and learn from large volumes of data [ 4 , 16 ] . First introduced for natural language processing ( NLP ) and then succesfully applied to visual data [ 18 ] , the strength of transformers is grounded in a self-attention mechanism which considers all pairwise interactions within the data . The price to pay is a computational complexity which grows quadratically with the data size , which itself depends linearly on the temporal dimension and quadratically on the spatial resolution in the image domain . Although there have been some efforts to reduce the complexity of self-attention [ 11 , 39 , 50 ] , using such methods directly on visual data is still limited to low resolutions and impractical without considerable computational power [ 9 , 79 ] . Some recent works [ 20 , 53 ] address this problem by using an autoencoder to compress the visual data , and apply the autoregressive model in the latent space . This greatly reduces the memory footprint and computational cost , yet , the greater the compression , the harder it is to faithfully reconstruct frames . The corresponding trade-offs may undermine the practical usability of these approaches . GANs [ 26 ] mitigate this issue by “ hallucinating ” plausible local details in image synthesis [ 20 ] . But latent video ∗corresponding author : guillaume.le-moing @ inria.fr 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . transformers [ 53 ] decode frames independently , which prevents local details from being temporally coherent . Hence , using GANs in this setting may result in flickering effects in the synthesized videos . We follow the problem decomposition from [ 20 , 53 ] , but introduce a more elaborate compression strategy with CCVS ( for Context-aware Controllable Video Synthesis ) , an approach that takes advantage of “ context ” frames ( i.e. , both input images and previously synthesized ones ) to faithfully reconstruct new ones despite lossy compression . As shown in Figure 1 , CCVS relies on optical flow estimation between context and new frames , within temporal skip connections , to let information be shared across timesteps . New content , which can not be retrieved from context , is synthesized directly from latent compressed features , and adversarial training [ 26 ] is used to make up realistic details . Indeed , information propagates in CCVS to new frames as previously synthesized ones become part of the context . Like other video synthesis architectures based on autoregressive models [ 53 , 79 ] , CCVS can be used in many tasks besides future video prediction . Any data which can be expressed in the form of a fixed-size sequence of elements from a finite set ( aka , tokens ) can be processed by a transformer . This applies to video frames ( here , via compression and quantization ) and to other types of data . Hence , one can easily fuse modalities without having to build complex or task-specific architectures . This idea has been used to control image synthesis [ 20 , 54 ] , and we extend it to a variety of video synthesis tasks by guiding the prediction with different annotations as illustrated in Figure 2 . Code , pretrained models , and video samples synthesized by our approach are available at the url https : //16lemoing.github.io/ccvs . Our main contributions are as follows : 1. an optical flow mechanism within an autoencoder to better reconstruct frames from context , 2. the use of ancillary information to control latent temporal dynamics when synthesizing videos , 3. a performance on par with or better than the state of the art , while being more memory-efficient . 2 Related Work . Video synthesis . In its simplest form , videos are produced without prior information about their content . GAN-based approaches map Gaussian noise into a visually plausible succession of frames . For example , VGAN [ 71 ] and ProgressiveVGAN [ 1 ] , adapt the traditional GAN framework [ 26 ] from image to video synthesis by simply using 3D instead of 2D convolutions . These approaches , including recent attempts such as G3AN [ 76 ] , are computationally expensive , and , by nature , restricted to synthesizing a fixed number of frames due to the constraints of their architecture . Other approaches predict latent motion vectors with a CNN [ 56 ] , or a recurrent neural network ( RNN ) [ 12 , 57 , 65 ] , and generate frames with individual 2D operations . To avoid the shortcomings of information loss in the sequential processing of RNNs , we use an attention-based autoregressive model instead . Since we forecast temporal dynamics in a compressed space , a large temporal window can be used when predicting new frames , without having to resort to expensive 3D computations . Previous works have attempted to scale video synthesis to higher resolution by using progressive training [ 1 ] , subsampling [ 56 , 57 ] , reducing the dimension of the discriminator [ 36 ] , or redefining the task as finding a trajectory in the latent space of a pre-trained image generator [ 63 ] . Compression , together with efficient context-aware reconstruction allows us to synthesize videos at high resolution . Controllable video synthesis . Some of the aforementioned works [ 12 , 56 , 57 , 76 ] handle synthesis conditioned on a class label . Another popular control is to use a few priming frames to set off the generation process . This task , known as future video prediction , has received a lot of attention recently . Methods based on variational autoencoders ( VAE ) [ 2 , 15 ] have been proposed to account for the stochastic nature of future forecasting , i.e. , the plurality of possible continuations , disregarded in deterministic predictive models [ 21 , 47 , 70 ] . Yet , their blurry predictions have motivated the incorporation of adversarial training [ 44 ] , hierarchical architectures [ 7 , 80 ] , fully latent temporal models [ 22 ] , or normalizing flows [ 42 ] . Another line of work infers spatial transformation parameters ( e.g. , optical flow ) , and predicts the future by warping past frames ( i.e. , grid sampling and interpolation as in [ 33 ] ) in RGB space [ 21 , 24 , 28 , 72 , 81 ] or in feature space [ 45 ] , typically using a refinement step to handle occlusions . Lately , autoregressive methods [ 53 , 79 ] leveraging a self-attention mechanism [ 68 ] have also been applied to this task . Our method benefits from the context-efficiency of approaches based on spatial transformation modules and the modeling power of autoregressive networks . In the meantime , other forms of control with interesting applications have emerged . Point-to-point generation [ 75 ] , a variant of future video prediction , specifies both start and end frames . State or action-conditioned synthesis [ 28 , 48 ] guides the frame-by-frame evolution with high-level commands . Additional works consider video synthesis based on one [ 51 ] or multiple layouts [ 46 , 74 ] , another video [ 8 ] , or sound [ 10 , 34 , 73 ] . To account for the variety of potential controls , we leverage the flexibility of transformers , and propose an unifying approach to tackle all of these tasks . 3 Context-aware controllable video synthesis . 3.1 Overview of the proposed approach . We consider video data , such that an individual frame x is natively an element of X = RH×W×3 , which can be encoded as some feature z in Z = Rh×w×F with reduced spatial resolution and increased number of channels . We assume that we are given Xc in Xm corresponding to m successive frames , and our goal is to synthesize a plausible representation of the following n frames Xs which lies in Xn . Given Xc , we can compute the corresponding features Zc in Zm using some encoder E : X → Z on each frame individually , then use an autoregressive model ( a transformer coupled with a quantization step in our case ) to predict features Zs in Zn formed by the n features corresponding to time ticks m + 1 to m + n. ( Note that the values of m and n can be arbitrary , using [ by now ] traditional masking and sliding window techniques . ) These features can finally be converted one by one into the corresponding frames Xs using some decoder D : Z → X . The overall approach is illustrated by Figure 1 . The use of a quantizer over a learned codebook in our implementation complicates the architecture a bit , but has several advantages , including the reuse of familiar NLP technology [ 68 ] and , perhaps more importantly , affording simple mechanisms for handling different types of inputs ( from images to sound for example [ 9 , 31 ] ) and the intrinsically uncertain and multimodal nature of future prediction ( by sampling different codebook elements according to their likelihood [ 30 ] ) . Concretely , this choice simply amounts to inserting in our architecture , right after the encoder E , a nearest-neighbor quantizer Q : RF → J1 , qK parameterized by an F × q codebook which , given the embedding z of a frame , returns a h × w matrix of corresponding tokens , that is , the indices of the closest entry in the codebook for feature vector zi , j for all spatial locations ( i , j ) in J1 , hK × J1 , wK . We abuse notation , and identify Q with its parameterization by this codebook , so we can optimize over Q just as we optimize over E instead of naming explicitly its parameters in the rest of this presentation . An “ un-quantizer ” U : J1 , qK → RF , also parameterized ( implicitly ) by the codebook and associating with each token the corresponding entry of the codebook , is also inserted right before the decoder D. In this setting , T : J1 , qKm×h×w → J1 , qKn×h×w takes as input a sequence of tokens , and outputs the tokens for subsequent frames , each one of them chosen among q possibilities as either the one with the highest score , or drawn randomly from the top-k scores to account for the multimodal nature of future forecasting ( here k ≤ q is some predefined constant , see [ 30 ] for related approaches ) . The elements of the architecture described so far are by now ( individually ) rather classical , with learnable , parametric functions E , D , Q and T . Besides putting them all together , and as detailed in the rest of this section , we add several original elements : ( a ) The encoding/decoding scheme is improved by the use of two discriminators , ∆i and ∆t respectively , trained in an adversarial manner to ensure that the predicted frames are realistic and temporally consistent . ( b ) The context frames Xc are themselves updated each time a new frame is predicted in an autoregressive manner ( iteratively fill the sequence up to some predefined capacity , then shift to the left , forgetting the first frame and adding the latest synthetic one on the right ) . ( c ) The encoder and decoder are linked through a learnable flow module F , allowing the context frames to guide the prediction of the synthetic ones . ( d ) Additional control variables , ranging from object trajectories to audio tracks , can be used in the form of sequence- or frame-level annotations to drive the synthesis by adding the corresponding tokens to the ones passed on to the autoregressive model T . We detail in this section the concrete components of the approach sketched above , including the autoencoder and quantizer architectures and their training procedure ( Section 3.2 ) , and the implementation of the autoregressive model by a transformer [ 68 ] , illustrated in Figure 2 , which we adapt to account for outside control signals ( Section 3.3 ) , once again with the corresponding procedure . Further architectural choices are also detailed in Appendix A .

Summary: This submission presents a conditional video synthesis model. This work is largely based on the recent progress of the Transformer-based autoregressive model [17] but with video extension. Differences over [17] are 1) learning-based flow module for temporal consistency, 2) conditioning for controllability, and 3) temporal embedding.  Overall, the submission contains a few technical novelties while being questionable whether significant, and the organization could be further improved. Other than those, the results seem reasonable, and the analyses are well conducted.   =============================================  <After the rebuttal and discussion phase>  The authors' responses have addressed most of this reviewer's concerns, which increases my rating.  The authors are strongly required to reflect all the rebuttal responses in the paper; especially missing reference, acknowledging the conventional works, the additional results, and the preliminary results with the SPADE version. More discussion about the flow module is recommended.