CCVS: Context-aware Controllable Video Synthesis
Authors: Guillaume Le Moing
Jean Ponce
Cordelia Schmid
Venue: NeurIPS 2021 Poster (2021)
Keywords: Video Synthesis
Multimodal Generative Modeling
Vision Transformers
Vector-Quantized Generative Adversarial Networks

Abstract: This presentation introduces a self-supervised learning approach to the synthesis of new videos clips from old ones, with several new key elements for improved spatial resolution and realism: It conditions the synthesis process on contextual information for temporal continuity and ancillary information for fine control. The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module. Adversarial training of the autoencoder in the appearance and temporal domains is used to further improve the realism of its output. A quantizer inserted between the encoder and the transformer in charge of forecasting future frames in latent space (and its inverse inserted between the transformer and the decoder) adds even more flexibility by affording simple mechanisms for handling multimodal ancillary information for controlling the synthesis process (e.g., a few sample frames, an audio track, a trajectory in image space) and taking into account the intrinsically uncertain nature of the future by allowing multiple predictions. Experiments with an implementation of the proposed approach give very good qualitative and quantitative results on multiple tasks and standard benchmarks.

CCVS: Context-aware Controllable Video Synthesis
Guillaume Le Moing Jean Ponce!.2 Cordelia Schmid!
Inria and Department of Computer Science, ENS, CNRS, PSL Research University Center for Data Science, New York University
Abstract
This presentation introduces self-supervised learning approach to the synthesis of new video clips from old ones, with several new key elements for improved spatial resolution and realism: It conditions the synthesis process contextual information for temporal continuity and ancillary information for fine control. The prediction model is doubly autoregressive, in the latent space of an autoencoder for forecasting, and in image space for updating contextual information, which is also used to enforce spatio-temporal consistency through a learnable optical flow module. Adversarial training of the autoencoder in the appearance and temporal domains is used to further improve the realism of its output. A quantizer inserted between the encoder and the transformer in charge of forecasting future frames in latent space (and its inverse inserted between the transformer and the decoder) adds even more flexibility by affording simple mechanisms for handling multimodal ancillary information for controlling the synthesis process (e.g., few sample frames, audio track, trajectory in image space) and taking into account the intrinsically uncertain nature of the future by allowing multiple predictions. Experiments with an implementation of the proposed approach give very good qualitative and quantitative results on multiple tasks and standard benchmarks.
Introduction
Feeding machines with extensive video content, and teaching them to create new samples on their own, may deepen their understanding of both the physical and social worlds. Video synthesis has numerous applications from content creation (e.g., deblurring, slow motion) to human-robot interaction (e.g., motion prediction). Despite the photo-realistic results of modern image synthesis models [38], video synthesis is still lagging behind due to the increased complexity of the additional temporal dimension.
An emerging trend is to use autoregressive models, for example transformer architectures [68], for their simplicity, and their ability to model long-range dependencies and learn from large volumes of data [4, 16]. First introduced for natural language processing (NLP) and then succesfully applied to visual data [18], the strength of transformers is grounded in self-attention mechanism which considers all pairwise interactions within the data. The price pay is computational complexity which grows quadratically with the data size, which itself depends linearly on the temporal dimension and quadratically the spatial resolution in the image domain. Although there have been some efforts to reduce the complexity of self-attention [11, 39, 50], using such methods directly on visual data is still limited low resolutions and impractical without considerable computational power [9, 79].
Some recent works [20, 53] address this problem by using an autoencoder to compress the visual data, and upply the autoregressive model in the latent space. This greatly reduces the memory footprint and computational cost, yet, the greater the compression, the harder is to faithfully reconstruct frames. The corresponding trade-offs may undermine the practical usability of these approaches. GANs [26] mitigate this issue by "hallucinating" plausible local details in image synthesis [20]. But latent video
corresponniig author: guillaume. moing
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

transformers [53] decode frames independently, which prevents local details from being temporally coherent. Hence, using GANs in this setting may result in flickering effects in the synthesized videos.
We follow the problem decomposition from [20, 53], but introduce a more elaborate compression strategy with CCVS (for Context-aware Controllable Video Synthesis), approach that takes advantage of "context" frames (i.e., both input images and previously synthesized ones) to faithfully reconstruct new ones despite lossy compression. As shown in Figure CCVS relies on optical flow estimation between context and new frames, within temporal skip connections, to let information be shared across timesteps. New content, which cannot be retrieved from context, synthesized directly from latent compressed features, and adversarial training [26] is used to make up realistic details. Indeed, information propagates in CCVS to new frames as previously synthesized ones become part of the context. Like other video synthesis architectures based on autoregressive models [53, 79], CCVS can be used in many tasks besides future video prediction. Any data which can be expressed in the form of a fixed-size sequence of elements from finite set (aka, tokens) can be processed by a transformer. This applies to video frames (here, via compression and quantization) and to other types of data. Hence, one can easily fuse modalities without having to build complex task-specific architectures. This idea has been used to control image synthesis [20, 54], and we extend to a variety of video synthesis tasks by guiding the prediction with different annotations as illustrated in Figure 2. Code, pretrained models, and video samples synthesized by our approach are available at the url https: bttpps///1tt / github. io/ ccvs.. Our main contributions are as follows:
1. an optical flow mechanism within an autoencoder to better reconstruct frames from context, 2. the use of ancillary information to control latent temporal dynamics when synthesizing videos,
3. performance on par with or better than the state of the art, while being more memory-efficient.
2 Related Work
Video synthesis. In its simplest form, videos are produced without prior information about their content. GAN-based approaches map Gaussian noise into visually plausible succession of frames. For example, VGAN [71] and ProgressiveVGAN [1], adapt the traditional GAN framework [26] from image to video synthesis by simply using 3D instead of 2D convolutions. These approaches, including recent attempts such as AeAN [76], are computationally expensive, and, by nature, restricted to synthesizing a fixed number of frames due to the constraints of their architecture. Other approaches predict latent motion vectors with a CNN [56], or recurrent neural network (RNN) [12, 57, 65], and generate frames with individual 2D operations. To avoid the shorteomings of information loss in the sequential processing of RNNs, we use an attention iidentioo-bbsedd autoregressive model instead. Since forecast temporal dynamics in a compressed space, a large temporal window can be used when predicting new frames, without having to resort to expensive 3D computations. Previous works have attempted scale video synthesis to higher resolution by using progressive training [1], sub- sampling [56, 57], reducing the dimension of the discriminator [36]. or redefining the task as finding trajectory in the latent space of pre-trained image generator [63]. Compression, together with efficient context-aware reconstruction allows us to synthesize videos at high resolution.
Controllable video synthesis. Some of the aforementioned works [12, 56, 57, 76] handle synthesis conditioned on a class label. Another popular control is to use a few priming frames to set off the generation process. This task, known as future video prediction, has received lot of attention recently. Methods based on variational autoencoders (VAE) [2. 15] have been proposed to account for the stochastic nature of future forecasting, i.e., the plurality of possible continuations, disregarded in deterministic predictive models [21, 47, 70]. Yet, their blurry predictions have motivated the incorpo- ration of adversarial training [44], hicrarchical architectures [7, 80], fully latent temporal models [22], or normalizing flows [42]. Another line of work infers spatial transformation parameters (e.g., optical flow). and predicts the future by warping past frames (i.e., grid sampling and interpolation as in [33])
in RGB space [21, 24, 28, 72, 81] or in feature space [45], typically using a refinement step to handle occlusions. Lately, autoregressive methods [53, 79] leveraging a self-attention mechanism [68] have also been applied to this task. Our method benefits from the context- context-efficiency of approaches based on spatial transformation modules and the modeling power of autoregressive networks. In the meantime, other forms of control with interesting applications have emerged. Point-to-point generation [75], a variant of future video prediction, specifies both start and end frames. State or action-condittoned synthesis [28, 48] guides the frame-by-frame frame evolution with high-level commands.

---

Q
T
u
encoder
A
decoder
Zo
7
F flow modulo
T transformer
3
F
D
S "shhif--add-ale module
A image discriminator
A temporal discriminator quantizer un qunattiazz
A
X
S
X
Figure Proposed architecture. Here X, and X, respectively stand for the (input) context and (output) synthesized video. Learnable encoder and decoder modules E and D are linked by learnable flow estimation module ensuring spatio-temporal consistency between context and synthesized frames. The architecture is doubly autoregressive, with transformer T responsible for predicting the features Z2 associated with future frames X, from the features Ze associated with context frames X and simple, parameterless, "shift-and-add" module S updating X, each new frame is generated. The architecture trained in two steps: The parameters of ε, D and F are first estimated (without any future forecasting from the transformer) in adversarial manner using two discriminators A; and A, to ensure that the frames synthesized are both realistic (A,) and temporally coherent (A,). The two discriminators with &, F and 'D frozen. At inference time, the transformer is used only once, latter frames being estimated in an autoregressive manner by the quadruple D 8, E, F. See text for more details.
then discarded, and the parameters of the transformer 'T are estimated
Additional works consider video synthesis based on one [51] or multiple layouts [46, 74], another video [8], sound [10, 34, 73]. To account for the variety of potential controls, we leverage the flexibility of transformers, and propose an unifying approach to tackle all of these tasks.
Context-aware controllable video synthesis
3.1 Overview of the proposed approach
We consider video data, such that an individual frame x is natively an element of X' = # which can be encoded as some feature increased number of channels. We assume that we are given X₂ in X corresponding to m successive frames, and our goal is to synthesize a plausible representation of the following n frames X, which lies in X". Given X we can compute the corresponding features Ze in 2d using some encoder E: + Z on each frame individually, then use an autoregressive model (a transformer coupled with quantization step in our case) to predict features Z, in 2" formed by the features corresponding to time ticks m + m + n. (Note that the values of m and n can be arbitrary, using [by now] traditional masking and sliding window techniques.) These features can finally be converted one by one into the corresponding frames X, using some decoder D Z X.
Z RxxuxF with reduced spatial resolution and
The overall approach is illustrated by Figure 1, The use of a quantizer over learned codebook in our implementation complicates the architecture bit, but has several advantages, including the reuse of familiar NLP technology [68] and, perhaps more importantly, affording simple mechanisms for handling different types of inputs (from images sound for example [9, 31]) and the intrinsically uncertain and multimodal nature of future prediction (by sampling different codebook elements according to their likelihood [30]). Concretely, this choice simply amounts to inserting our
architecture, right after the encoder E, nearest-neighhor quantizer Q RR - [1, q] parameterized by an x q codebook which, given the embedding z of a frame, returns corresponding tokens, that is, the indices of the closest entry in the codebook for feature vector zij for all spatial locations (i,j) in [1, h]] x [1, w]. We abuse notation, and identify Q with its parameterization by this codebook,
x w matrix of
optimize over Q just as we optimize over & instead of naming explicitly its parameters the rest of this presentation. An "un-quantizer"
[1, q] - R²., also parameterized (implicitly) by the codebook and associating with each token the corresponding entry of the codebook, is also inserted right before the decoder D. In this setting, T: [1, nm xmm xhhx [1, g] x bx takes as input sequence of tokens, and outputs the tokens for subsequent frames, each one of them chosen among g possibilities as either the one with the highest score, or drawn randomly

---

from the top-k scores to account for the multimodal nature of future forecasting (here ls S g is some predefined constant, see [30] for related upproaches).
The elements of the architecture described so far are by now (individually) rather classical, with learnable, parametric functions &, D, Q and T. Besides putting them all together, and as detailed in the rest of this section, we add several original elements: (a) The encoding/decoding scheme is improved by the use of two discriminators, A; and A, respectively, trained in an adversarial manner to ensure that the predicted frames are realistic and temporally consistent.
(b) The context frames X. are themselves updated each time new frame is predicted an autoregressive manner (iteratively fill the sequence up to some predefined capacity, then shift to the left, forgetting the first frame and adding the latest synthetic one on the right). (c) The encoder and decoder are linked through a learnable flow module F, allowing the context frames to guide the prediction of the synthetic ones. (d) Additional control variables, ranging from object trajectories to audio tracks, can be used in the form of sequence- or frame-level annotations to drive the synthesis by adding the corresponding tokens to the ones passed on to the autoregressive model T.
We detail in this section the concrete components of the approach sketched above, including the autoencoder and quantizer architectures and their training procedure (Section 3.2), and the imple- mentation of the autoregressive model by a transformer [68], illustrated in Figure 2, which we adapt to account for outside control signals (Section 3.3), once again with the corresponding procedure. Further architectural choices are also detailed in Appendix A.
3.2 First stage: training the context-aware autoencoder and the quantizer
Architecture. E and D respectively decreases and increases the spatial resolution by using convolu- tional residual blocks [29]. with (rk) the K corresponding resolution levels (r&- = h2 x wx). It is common practice to augment, in U-Net [55], the autoencoder with long skip connections between & and D to share information across the two models at these intermediate levels and escape the lossy compression bottleneck. Although we cannot apply this directly our setting since infor- mation only flows through 'D for predicted timesteps, such skip connections can be established from the encoding stage of context frame Te to the decoding stage of a new frame T, (resulting from features z,). Similar mechanisms [15, 19] have been proposed in the past for video synthesis but they are only copying static background features from a single context frame. We follow works on semantic segmentation [23] and face frontalization [78] and use a flow module F to warp features and produce temporally consistent outputs despite motion. We extend this to with significant performance gains and no additional parameter to be learned.
contexts,
Concretely, let e be features being encoded from Z,, and d: features being decoded from 2, at given intermediate resolution F&- We first compute all intermediate context features et: by applying E to Te. We then progressively decode features d: for the new frame from low (k- = 1) to high resolution (k- K) by iterating over the following steps: (a) apply one decoding sub-module to get d from d: (b) use F to refine the optical flow f (in 2x2 which estimates the displacement field from ex to d (as a proxy to the one from xe to X,), and a fusion mask m (in 1x which indicates the expected similarity between aligned features et = W(e,, and de (also as proxy for the one in image domain) with W corresponding to a standard warping operation, (c) use e and m update de with context information (see update rule (1) below), (d) move to resolution level r by going back to (a). We note that F estimates ft and m. in a coarse-to-fin fone fashion by refining f6-1 and m (see Appendix A for further details). Temporal skip connections at given resolution level r are defined as the following in place modification of d:.
d = (m²)) @ d: + (1 - (m²)) @ ett
(I)
where a is the Sigmoid function, and @ the element-wise product. We note that update rule (1) is quite standard for warping and fusing two streams of spatial information [23, 28, 74]. For concrete implementation of F, we build upon LiteFlowNet [32], an optical flow estimation model which also combines pyramidal extraction and progressive warping of features. For simple integration into our framework, we use features from E and D both in the mask and flow estimation, and in the update (1). This process readily generalizes to multi-frame contextual information (see Appendix D for details). Similar to spatial transformers [33], the warping operation W is differentiable. As result, gradients from the training losses can backpropagate from D to E through F. This allows end-to-end training of the autoencoder even with information from different timesteps in E and D.

---

Training procedure.
The global objective is the linear combination of four auxiliary ones:
L == Ag6u + X, + Aaha + Anhi,
namely quantization loss (L.g,, reconstruction loss (C,), adversarial loss (Ca), and contextual loss (Lc), detailed in the next paragraphs.
The codebook is trained by minimizing the reconstruction error between encoded features (x)
and quantized features Zn = (())))) (with notations introduced in Section 3.1):
Q) sg((=) 2 + 311 sg( zn) =112.
where sg(.) is the stop gradient operation which constrains its operand to remain constant during backpropagation. The first part moves the codebook entries closer to the encoded features. The second part, known the commitment loss [67], reverses the roles played by the variables.
In regions with complex textures and high frequency details, local patterns shifted by few pixels in and its reconstruction E- = DDzz)) may result in large pixel-to-pixe errors, while being visually satisfactory. We thus define the recovery objective as the L1 loss in both RGB space and between features from a VGG network [59] pretrained on ImageNet [14]:
(E, F, || I|l + VGG(z) - VGG(T)]l.
To tackle cases where information cannot be recovered from context due to occlusion, and the compressed features are insufficient to create plausible reconstructions due to lossy compression, we supplement our architecture with an image discriminator A,, made of downsampling residual blocks [29], to encourage realistic outputs. A, tries to distinguish real images from reconstructed ones (Ca), while & and D fools A; into assuming reconstructed images are good as real ones (C.):
(a( In(1 +e( In(1 +e (.))) (5)
(E Q, D) = In(1 + (.))))
(6)
We employ a similar strategy on sequences of consecutive frames to improve the temporal consisteney using a 3D temporal discriminator A,, direct extension 2D image discriminator A,,
The success of our method relies on accurate motion estimation in F. a difficult task which benefits from self-supervision [35]. Therefore, we train the autoencoder with augmented views of the input frames as context. Custom augmentations functions A X' + X' include: rotation, scaling, translation, elastic deformation, and combinations of these. Augmented views are obtained by warping x by the suitable flow ac: A(r) W(x, az), and the inverted flow fc from A(r) to can be approximated. I We resort flow inversion because directly reconstructing distorted views may encourage similar defects during inference. Moreover, we balance between self-recovery and context-recovery objectives by additionally applying a blurring function : + X' and an occlusion mask to the augmented frames 0, @ (x)))... This augmentation strategy is illustrated in Appendix F. We define the contextual loss as:
Lo(E, = 1.f Fell + 1o o (me)
where df W(o.. fc), and f.. and m. are the flow and mask estimated by F. In practice, this loss is applied at intermediate resolution levels Tı for improved training.
3.3 Second stage: predicting temporal dynamics with transformers
Architecture. We follow [20] and adopt an architecture similar to Image-GP") [9] for T. Instead of modeling a single annotated frame in [54], we design our model to account for sequences of N such frames to allow prediction of temporal dynamics controlled by ancillary information in the form of video- and frame-level annotations. We have shown, Section 3.2, how to represent a frame as a sequence of h. tokens (indices in [1, [1, through encoding and quantization. Similar strategies can be applied to cater to other types of data, with or without compression depending on their complexity, and, thereby, turn ancillary information into tokens as well. The capacity of the transformer (the maximum sequence length it can process) is = ly + (l, + f w), where 4 and I are the size of video- and frame-level annotations respectively. The final layer of the model predicts, for every i in [1, L - 1]. a vector o (of size q, the number of possible tokens) which scores the likelihood of the i + 1hh token given all preceding ones.
OUr implementation of flow inversion approximation is detailed in Appendix C.
5

---

spatial
T
embeddiing
3 encoder
0
& video anzotation encoder
0
61
11
E annotation encoder
temporal embedding
2
2
quantizer
video annotation quamtizer
E
3
8
3
E
D
2 Irame annotation quanlizer
T transformer
U1
un-quantizer
video annotation
frame annotation
frame annoration
R
frame annotation
D decoder
Conditioning input
Prediction
Figure 2: Illustration of the transformer to predict latent temporal dynamics with control-related data. For clarity, we omit the flow module between & and D. Input video-level annotation, frame-level annotations, and conditioning frames are encoded and quantized to form the initial token sequence, and mapped corresponding embeddings. Subsequent frame tokens are obtained autoregressively with T, while the ones corresponding to video- and frame-level annotations guide the prediction.
Training procedure. To learn the parameters T. load a complete sequence (N frames corresponding to L. tokens), and try to predict the T, last tokens based on the J, first ones. This is done by maximizing the log-likelihood of the data using the cross-entropy loss:
L
exp((ār-1, eT(i)))
C(T) = Σ log
(8)
Σ, exp((o--1..))
where T(i) is the 1h ground-truth token, and ej vector of O's with in its jth coordinate.
Inference. During inference we use T to complete autoregressively an input sequence tokens. To avoid known pitfalls (e.g., repetitive synthesis) and allow diverse outcomes, whereby the next token randomly chosen among the k most likely ones, weighted by their scores O. Although T processes sequences of N annotated frames, we predict ones of arbitrary length by using a temporal sliding window. Tokens corresponding to video- and frame-level annotations are given as input to T (not predicted, even for future frames) to guide the synthesis. We show in the experiments how this control can translates into a variety of interesting tasks.
top-k sampling
4 Experiments
assess the merits of CCVS in the light of extensive experiments on various video synthesis tasks.
Datasets. BAIR Robot Pushing [19] consists of 43k training and 256 test videos of robotic arm interacting with objects from a fixed viewpoint. A high-resolution version has recently been released. We manually annotate, in 500 frames, the (a, location of the arm in image space train position estimator, which we use for state-conditioned synthesis. To account for real world scenarios, we evaluate on Kinetics-600 [5], large and diverse action-recognittoo dataset with approximately 500K videos. We also test our method on AudioSet-Drumt [25] for sound-conditioned synthesis on music performance, containing 6k and 1 video clips in train and test splits respectively. Other datasets and tasks are covered in Appendix E.
Metrics. We use the Fréchet video distance (FVD) [66] which measures the distribution gap between real and synthetic videos in the feature space of an Inception3D network [6] pretrained on Kientics-400 [40]. It estimates the visual quality and temporal consistency of samples as well as the diversity in unconditioned scenarios. In conditioned ones, we use another metric for diversity (DIV) which is the mean pixel-wise distance among synthetic trajectories conditioned on the same input. For near-deterministie motions (e.g., in reconstructions, one-to-one mapping between real video frames and synthetic ones, we include pairwise image quality assessments: the structural similarity index measure (SSIM) [77] which evaluates per-frame
constrained tasks), there is

---

Table 1: Ablation study of the autoencoder on BAIR (256 x 256). We evaluate self- and context- recovery modules in different scenarios: synthesizing 16-frame videos from known compressed features ( ("Reconstruction)) by inferring compressed features with 7 given the real trajectory of the robotic arm "SState-condition" is used as initial context in all cases, except for "Unc." where it is synthesized by StyleGAN2 [38].
or without the trajectory ("Pred." and "Unc."). The first real frame
Self-recovery
CIxt-reeooveyy
Reconstruction
State-conditioned
Pred.
Unc.
RGB
VGG
A
A.
F
Sup.
Cist.
FVDL
PSNR
FVDJ
PSNR
FVD1
FVDJ
1200
20.0
1238
17.8
1265
1321
0
700+11
17.9
71415
17.4
704
765412
V
/
0
323
17.8
355
16.6
377:11
566
V
0
389+12
18.2
401+4
16.9
407
627
V
98+2
22.7
10642
22.2
142+6
350+11
87+3
24.4
97
22.1
128
301+10
8
62
25.4
76+3
22.7
109
299
15
60+1
25.6
75+2
22.8
110+3
297
Training longer (num. epochs x3)
45+1 4-1
26.8
67+1
22.3
100
293
"Sup." self-supervisioo of F: "Cixt." number context frames taken into account (in (F) when decoding current Trame (in D).
Table 2: Ablation study of the transformer on BAIR.
10 1
FPS
We adopt notations and evaluation setups from Table 1.
16
Architecture
Top-k
State-conditioned
Pred.
Unc.
Layer Head Doc. Frame State
FVD4
PSNR1
FVD1
FVDJ
73+2
23.0
281
474+17
64
n1
Tokens/Frame
Reconstruction
24
12
16
70
73+3
23.3
23.1
262
331
521+19
4354s
0.1
Video prediction
24
16
69=2
23.2
321
479434
256
FVD
24
16
10
65
22.4
127_7
308
100
200
300
100
67
22.3
121
314+12
24
16
V
100
10
67+1
22.3
100+2
293+7
Figure 3: Quality and speed of synthesis
"Dec!": spatio-temporal decomposition positional emheddings.
vs. compression Nvidia V100 GPU.
conformity (combination of luminance, contrast and structure), and the peak signal-to-noise ratio (PSNR) which is directly related to the root mean squared error. For each metric, we compute the mean and standard deviation (std) over evaluation runs (2 for Kinetics due its voluminous 50k video test set). For clarity, the std value is shown only when it is greater than the reported precision.
Training details. All our models are trained on 4 Nvidia V100 GPUs (32GB VRAM each), using ADAM [41] optimizer, for multiple 20 hour runs. We adapt the batch size to fill the memory available. We use a learning rate of 0.02 to train the autoencoder, and exponential moving average [83] to obtain its final parameters. We use weighting factors (1, 10, 1, 1) and 0.25 for (Ag. Ar, Aay Ao) and in Equations (2) and (3) respectively. We use a learning rate of 10 5 train the transformer.
4.1 Ablation study
We conduct ablation study of CCVS to show the individual contribution of the key components of the proposed autoencoder (Table 1), transformer (Table 2), and the effect of compression (Figure 3).
First, we fix the transformer and observe the incremental improvements in synthesis quality when adding self- and context-recovery modules to the autoencoder (Table 1). In particular, L₁ loss in the feature space of a VGG net [59] produces sharper videos than using the same loss in the RGB space alone, Predicted frames using image and temporal discriminators (A, and A,) display greater realism and temporal consistency. The flow module F significantly improves the performance on all metrics by allowing context frames to guide the reconstruction of synthetic ones. The self-supervision of F, the use of larger context windows, and longer training times, further improve the quality of the synthesis. A, seems to deteriorate the FVD first, but when all modules are combined it improves FVD by almost a factor 2 it encourages better temporal consistency in the presence of context.
7

---

Table 3: Future video prediction on BAIR (64 x 64), synthesizing 16-frame videos given a few conditioning frames ("Cond."). We include some extensions of our method at higher resolution.
Method
Cond.
FVD 4
Code Avail.
Memory, compute
10*
MoCoGAN [65]
503
16GB, 23h"
SVG-FP [15]
315
12GB, 24h"
GBxh
CDNA [21]
297
10GB, 20h
SV2P [2]
2
263
16GB, 24 to 48h"
SRVP [22]
2
181
36GB, 168h
VideoFlow [42]
131
128GB, 336h"
L.V [53]
12643
128GB, 48h
SAVP [44]
116
32GB, 144h
DVD-GAN-FP [12]
110
2TB, 24h"
Video Transformer (S) [79]
10643
256GB, 33h"
TrVDD-GAN-FP [45]
103
ITB, 280h
Low res. CCVS (ours)
9942
128GB, 40h
Vidco Transformer (L) [79]
9442
512GB, 336h"
SSIM 1 (f = 8) SSIM (f = 15)
High CCVS (ours)
80.3
0.729
0.683
end frame
12
0.766
0.839
state
50+1
0.885
0,863
": value confirmed by authors:
*1 training inference/ SSIM at
1=1
1=2
1=4
t=
6
1=
8
1=
10
1=
12
t=
14
t= 16
Future prediction
Point-to- point
State- conditioned
Real video
Figure 4: Qualitative samples for different types of control on BAIR (256 x 256). Zoom in for details.
We fix the autoencoder, and compare different architectures and sampling strategies for the transformer (Table 2). Increasing the model capacity by adding layers and increasing expressivity (number of attention heads) along with a spatio-tempore decomposition of positional embeddings (shown in Figure detailed in supplementary material) yields small improvements. Top- sampling is beneficial for stochastic tasks (video prediction and unconditional synthesis). whereas always selecting the most probable tokens results in little to motion. Guiding temporal dynamics with state-conditioned synthesis reduces the advantage of sampling by narrowing down possible outcomes.
Finally, we explore the effect of compression (in terms of the number of tokens for each frame) on synthesis speed (FPS) and quality (FVD) of reconstructed and predicted videos (Figure 3). We use a compression of 64 tokens in our default setup since it gives the best FVD while retaining reasonable speed. The critical drop of FPS for low compression ratios is due the pairwise consideration of all input tokens in 'T. Note that using smaller temporal window in T may allow additional speed-ups.

---

Table 4: Future video prediction on Kinetics (64 x 64)
Table 5: Codebook size on Kinetics.
of 16-frame videos from consecutive input frames.
Reconstruction
Pred.
Size
Method
FVD J
GPU/TPU Mem.
FVD 4
SSIM
PSNR
FVD
LVT [53]
225
128GB, 48h
1024
58
0.907
31.2
6642
Video Transformer [79]
170+5
2TB, 336h"
4096
54+1
0.917
31.6
6441
DVD-GAN-FP [12]
69+1
2TB, 144h"
16384
49+1
0.923
32.0
5541
CCVS (ours)
5541
128GB, 300h
65536
45+1
0.928
32.2
6141
TrVD-GAN-FFP [45]
26+1
16TB, 160h
8c
12+1
0.963
34.5
229+1
": value confirmed by authors.
Table 6: Sound-conditional video synthesis on AudioSet- Drumms (64 64).
SSIM
PSNR1
Method
Cond.
Audio
30
45
SVG-LP [15]
15
0.971=0.01
0.661 010 0.51010.008
30.0
16.610.3
13.5
Vougioukas al. [73]
15
V
0.910_0.001
0.904 007 0.89610.015
26.2+1.0
23.810.2
23.3
Sound2Sight [10]
007 0.947 0.95440.op 0.984_0.00p
33.2
27.9
27.0+0.3
CCVS (ours)
15
V
0.987
0. 2000
00.0ss
30.4
28 220.6
27. 3 +
4.2
Quantitative and qualitative studies
BATR. For future video prediction on BAIR (Table 3), CCVS trained at 64 x 64 resolution (low res.) on par with the best method (L-size version of [79]), but requires much less computing resources, and outperforms [79] under similar resources. We also propose high res. CCVS which is not strictly comparable to the prior arts as use 256 256 image resolution for training and test, and resize the synthesized frames to 64 x 64 for computing FVD. However, using this variant demonstrates the performance gains that can arise by scaling CCVS. We additionally address point-to-point synthesis (with the end frame as a video-level annotation) and state-conditioned synthesis (with the estimated 2D position of the arm as a frame-level annotation). Point-to-poin synthesis is more difficult than video prediction: Not only does it requires realistic video continuations, but also ones which explain the end position of all visible objects. Hence, FVD score is constant despite the additional input. Still, this yields better SSIM for mid-point and one- ane-before-last frames. State coatee-ondiiioeedd synthesis improves on FVD and mid-synthesis SSIM as motion becomes determiniitii.c. Some synthetic frames for these tasks are shown in Figure 4. CCVS creates plausible settings, and true interactions with objects compared to previous attempts [48] the same resolution.
videos in various
t=1 1
t=8 8
t= 12
t= 16
t= 1
t= 8
t= 12
t= 16
Figure 5: Qualitative samples on future video prediction on Kinetics (64 64). Zoom in for details.
Kinetics. CCVS ranks second on Kinetics video prediction benchmark. Kinetics contains more diversity than BAIR, and the reconstruction is thus more difficult. A solution [54] is to increase the codebook size (Table 5) but stops translating into better prediction FVD at some point. We also try removing the quantization step (equivalent to an infinite codebook), and directly regressing latent features with T (instead of ranking the likelihood of possible tokens). It allows better reconstructions, yet the prediction FVD is high. Figure 5 shows examples of synthetic continuations conditioned on

---

Input
Seed
Seed 2 f 15
Seed 3 f 15
Seed 1 t 15
Seed 5
Seed 6 15
DIV f
t 15
15
Whole frames
BAIR
19.53 60
Moving parts 136.88 12.227
Kineties
Whole frames 20.85
Moving parts 48. 18.163
AudioSet-
Whole frames
Drums
2. 2.
Moving parts
65. 45
the pixel-wise distances obtained by applying 1.0 10-3 factor (omitted for clarity).
Figure 6: Diversity results for conditioned scenarios. The diversity metric (DIV) measures the mean pixel-wise distance among 10 15-frame synthetic trajectories conditioned on the same frame. We report the mean and std over 100 runs: on whole frames as in [82], moving parts only by masking static regions (optical flow magnitude between consecutive frames < 20% of the max magnitude) as in [69]. We show, for the same input, the 15th frame of various randomly seeded synthetic trajectories.
5-frame unseen test sequences. CCVS produces realistic and temporally coherent outputs which display various types of motion (e.g.. body, hand, camera).
AudioSet-Drums, CCVS achieves top performance on oound-conditioned video synthesis on Au- dioSet Drums (Table 6). Figure 6 shows quantitative and qualitative insights on the diversity of the synthetic trajectories conditioned on the same input for the three datasets. The diversity metrics (DIV) computed on whole frames is lower on AudioSet Drums than on the other two datasets. This is explained by the fact that motion is quite repetitive and involves a limited portion of the frame. The same metric on moving parts only, and the end position of the drummer's hand and upper left cymbal in qualitative samples, show the diversity of synthetic trajectories. An ablation of CCVS with/without audio guidance as well as more qualitative results on diversity can be found in Appendix
5
Discussion
CCVS is on par or better than the state-of-the-art on standard benchmarks, uses less computational resources, and scales to high resolution. Training neural networks is environmentally costly, due to the carbon footprint to power processing hardware [17, 61]. Methods sparing GPU-hours like ours are crucial to make Al less polluting [43, 61]. and move from a "Red" to "Green" AI [58]. Future work will include exploring new codebook strategies and synthesis guided by textual information.
Limitations. CCVS uses a complex architecture and a two-stage training strategy. Simplification of both is an interesting direction for improving the method. Moreover, CCVS lacks global regularization of motion (flow computed on pairs of timesteps), and its efficiency relies on recycling context information such that synthesizing content from scratch (i.e., input frame given) remains difficult.
Broader impact. The increased accessibility and the many controls CCVS offers could accelerate the emergence of questionable applications, such as "deepfakes" (e.g.. video created from someone's s picture and an arbitrary audio) which could lead to harassment, defamation, or dissemination of fake news. On top of current efforts to automate their detection [49], it remains our responsibility to grow awareness of these possible misuses. Despite these worrying aspects, our contribution has plenty of positive applications which outweigh the potential ethical harms. Our efficient compression scheme is step in the direction of real-time solutions: e.g., enhancing human-robot interactions, or improving the safety of self-driving cars by predicting the trajectories of people and vehicles nearby.
10

---

Acknowledgements
This work was granted access the HPC resources of IDRIS under the allocation 2020-AD01101222 made by GENCI. It was funded in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d' d'avenir" program, reference ANR-19. P3IA-0001 (PRAIRIE 31A Institute). JP was supported in part by the Louis Vuitton/ENS chair in artificial intelligence and the Inria/NYU collaboration. We thank the reviewers for useful comments.
References
[1] Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein GANs, arXiv preprint, 2018.
[2] Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2018.
[3] Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint, 2013.
[4] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In NeurlPS, 2020.
[5] Joao Carreira, Eric Noland. Andras Banki-Horvale Horvath, Chloe Hillier, and Andrew Zisserman. A short note about Kinetics-600. arXiv preprint, 2018.
[6] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
[7] Lluis Castrejon, Nicolas Ballas, and Aaron Courville. Improved conditional VRNNs for video prediction. In ICCV, 2019.
[8] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros. Everybody dance now. In JCCV, 2019.
[9] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020.
[10] Anoop Cherian, Moitreya Chatterjee, and Narendra Ahuja. Sound2Sight: Generating visual dynamics from sound and context. In ECCV, 2020.
[11] Krzysztof Choromanski. Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers. In ICLR, 2021.
[12] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint, 2019.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hicrarchical image database. CVPR, 2009.
[15] Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In ICML, 2018.
[16] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidircctional transformers for language understanding. In ACL, 2019.
[17] Payal Dhar. The carbon impact of artificial intelligence. Nature Machine Intelligence, 2020.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.

---

[19] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, 2017.
[20] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[21] Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised leamnngg for physical interaction through video prediction. In NeurlPS, 2016.
[22] Jven-Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochas- tic latent residual video prediction. In ICML, 2020.
[23] Raghudeep Gadde, Varun Jampani, and Peter V. Gehler. Semantic video cans through representation warping. In ICCV, 2017.
[24] Hang Gao, Huazhe Xu, Qi-Zhi Cai, Ruth Wang. Fisher Yu, and Trevor Darrell. Disentangling propagation and generation for video prediction. In ICCV, 2019.
[25] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP, 2017.
[26] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurIPS, 2014.
[27] Lena Gorelick, Moshe Blank, Eli Shechtman, Michal Irani, and Ronen Basri. Actions as space-time shapes. TPAMI, 2007.
[28] Zekun Hao, Xun Huang, and Serge Belongie. Controllable vidco generation with sparse trajectories. In CVPR, 2018.
[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] Ari Holtzman, Jan Buys, Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020.
[31] Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthoree, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In ICLR, 2019.
[32] Tak-Wa Hui, Xiaoou Tang, and Chen Change Loy. LiteFlowNet: A lightweight convolutional neural network for optical flow estimation. In CVPR, 2018.
[33] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. networks. In NeurlPS, 2015.
Spatial transformer
[34] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. IJCV, 2019.
[35] Rico Jonschkowski, Austin Stone, Jonathan T. Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova. What matters in unsupervised optical flow. ECCV, 2020.
[36] Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video discrimina- tors. Neural Networks, 2020.
[37] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung. Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
[38] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[39] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In ICML, 2020.
[40] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vijaanarasimman. Fabio Viola, Tim Green. Trevor Back. Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action vidco dataset. arXiv preprint, 2017.
[41] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In ICLR, 2015.
12

---

[42] Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine. Laurent Dinh, and Durk Kingma. VideoFlow: A conditional flow-based model for stochastic video generation. In ICLR, 2020.
[43] Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions machine learning. In NeurlPS workshop. 2019.
[44] Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbecl, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction, arXiv preprint, 2018.
[45] Pauline Luc, Aidan Clark, Sander Dicleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-bann based adversarial vidco prediction on large-scale data. arXiv preprint, 2020.
[46] Arun Mallya, Ting-Chun Wang, Karan Sapra, and Ming-Yu Yu Liu. World-consistent video-to-video synthesis. In ECCV, 2020.
[47] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
[48] Willi Menapace. Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation. In CVPR, 2021.
[49] Thanh Thi Nguyen, Quoe Viet Hung Nguyen, Cuong M. Nguyen, Dung Nguyen, Duc Thanh Nguyen, and Saeid Nahavandi. Deep learning for deepfakes creation and detection: A survey. arXiv preprint, 2019.
[50] Anselm Levskaya Nikita Kitaev, Lukasz Kaiser, Reformer: The efficient transformer. In ICLR, 2020.
[51] Junting Pan, Chengyu Wang. Xu Jia, Jing Shao, Lu Sheng. Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. In CVPR, 2019.
[52] Taesung Park, Ming- Yu Liu, Ting-Chun adaptive normalization. In CVPR, 2019.
Chun Wang, and Jun Yan Zhu, Semantic image synthesis with spatially-
[53] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnacv, Latent vidco transformer. arXiv preprint, 2020.
[54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero-shot text-to-image generation. In ICMI., 2021.
[55] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[56] Masaki Saito, Eitchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In JCCV, 2017.
[57] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal GAN. IJC,, 2020.
[58] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green Al. ACM, 2020.
[59] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
[60] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint, 2012.
[61] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In ACL, 2019.
[62] Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmenta- tion. arXiv preprint, 2020.
[63] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. In ICLR, 2021.
[64] Du Tran, Lubomir Bourdev, Rob Fergus. Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.
[65] Sergey Tulyakov, Ming Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN; Decomposing motion and content for video generation. In CVPR, 2018
13

---

[66] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint. 2018.
[67] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
[69] Ruben Villegas, Jimei Yang. Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In ICLR, 2017.
[70] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016.
[71] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016.
[72] Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In CVPR, 2017.
[73] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. End-to-end specch-driven facial animation with temporal GANs. In BMVC, 2018.
[74] Ting-Chun Wang, Ming-Yos Yu Liu, Jan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Vidco-to-video synthesis. In NewrIPS, 2018.
[75] Ssun-Hsuan Wang. Yen-Chi Cheng, Chich Hubert Lin, Swwann-Tzone Chen, and Min Sun. Point-to-poii video generation. In ICCV, 2019.
[76] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva. G3AN: Disentangling appearance and motion for vidco generation. In CVPR, 2020.
[77] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. 77P, 2004.
[78] Yuxiang Wei, Ming Liu, Haolin Wang. Ruifeng Zhu, Guosheng Hu, and Wangmeng Zuo. Learning flow-based feature warping face frontalization with illumination inconsistent supervision. ECCV, 2020.
[79] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2020.
[80] Bohan Wu. Suraj Nair, Roberto Martin-Martin Fei-Fei, and Chelsea Finn. Greedy hierarchical variational autoencoders for large-scale video prediction. In CVPR, 2021.
[81] Yue Wu, Rongrong Gao, Jacsik Park, and Qifeng Chen. Future video synthesis with object motion prediction. CVPR, 2020.
[82] Dingdong Yang, Seunghoon Hong, Yunseok Jang. Tianchen Zhao, and Honglak Lee. Diversity conditional generative adversarial networks. ICLR, 2019.
[83] Yasin Yaz, Chuan-Sheng Foo, Stefan Winkler, Kim Hui Yap. Georgios Piliouras, and Vijay Chandrasekhar. The unusual effectiveness of averaging in GAN training. In ICLR, 2019.
14


Reviews:
[{"review_id": "i0oLB-IkXUB", "reviewer": "Reviewer_Q5Cg", "summary": "This submission presents a conditional video synthesis model. This work is largely based on the recent progress of the Transformer-based autoregressive model [17] but with video extension. Differences over [17] are 1) learning-based flow module for temporal consistency, 2) conditioning for controllability, and 3) temporal embedding.\n\nOverall, the submission contains a few technical novelties while being questionable whether significant, and the organization could be further improved.\nOther than those, the results seem reasonable, and the analyses are well conducted.\n\n\n=============================================\n\n<After the rebuttal and discussion phase>\n\nThe authors' responses have addressed most of this reviewer's concerns, which increases my rating. \nThe authors are strongly required to reflect all the rebuttal responses in the paper; especially missing reference, acknowledging the conventional works, the additional results, and the preliminary results with the SPADE version.\nMore discussion about the flow module is recommended.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros\n- Video extension of [17] introducing flow module, controllability, and temporal embedding\n- Simple and general conditioning method using Transformer (frame, sound and pose conditioning)\n- Sufficient experiments and study. Nice ablation\n- Good quantitative results\n\nCons\n- Questionable effectiveness of the flow module in temporal consistency\n- Paper organization and writing could be improved (the use of terminologies are confusing, and notations are abused)\n- Hard to feel improvement over the state-of-the-art methods in qualitative results.\n\n\n<Detail comments>\n\n* Sound-conditioned video synthesis results: Multi-modal controllability is very interesting in generative modeling.\nIt would be more interesting to show the diverse synthesis results obtained from the fixed visual condition and different sound contexts (e.g., different drum playing sounds). This would allow to clearly understand behaviors and insights of the proposed method w.r.t. conditioning.\n\n* Layout-conditioned video synthesis results: Compared to [40], the results obtained by the proposed method suffer from wobble effects and unstable temporal consistency.\nThen, it is questionable what is the benefit of using the flow module.\nThis is concerning in that the flow-module is one of the main technical contributions and the most technical distinctive extension over [17], which seems to be neutralized by failing to show its effectiveness.\n\n* BAIR results look interesting. The proposed method seems to understand physical interaction well to some extent.\n\n* Missing reference: Flow-based feature warping modules have been used for temporal consistency, e.g., [Semantic Video CNNs through Representation Warping, ICCV 2017]. However, in Line 154, the references are missing, and the discussion about the history is misleading. \n\n\n<Writing issues>\n\n* Terminologies are confusing: The authors interchangeably used {encoding, embedding, feature, code}, {annotation, condition, state}, etc. Those badly affect the readability of the submission.\n\n* Flow module: By reading about the proposed flow module in the main paper, this reviewer failed to understand how the flow module is operated clearly, and found that the description in the supplementary material is much more specific and clear to understand.\nThe writing should be reorganized to convey the clear elaboration of the module.\n\n* Fig. 3: The y-axis scale annotation is confusing (log_10). Is it 10 FPS or exp(10) FPS? \n\n* Notation abuse: Dimensions defined in the submission are confusingly described. The authors seem to intend this to appear to be general, but it disturbs understanding about the dimensions. Also, it leads to duplicated descriptions about dimensions in Sec. 3.1 \nand Sec. 3.2, which can be much condensed. It would be more informative to bring some of the supplementary material contexts by reorganizing the descriptions to make more space.\n\n\n\n\n* Question: Why should the flow inversion be approximated? This reviewer cannot see any reason to use this approximation rather than directly learning to extract backward flow.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "* The authors argued a positive effect on energy consumption by faster training thanks to the compact code.\n* Misuse case discussion is included, and the authors argued that the positive applications outweigh the potential ethical concerns.\n\nThe discussions sound reasonable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "BfWdRYxZfq", "reviewer": "Reviewer_mX9P", "summary": "This paper proposes a method for video synthesis that combines an encoder / decoder architecture that computes discrete representation of images with a transformer network that predicts those discrete representations into the future. The encoder and decoder are connected through a novel optical flow module that warps encoder features into the decoder in order to predict how objects will move. In experiments, the authors perform thorough ablations of the proposed method, and show that they outperform most baselines.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ New feature level warping strategy for future video synthesis\n+ Thorough ablations\n+ Outperforms most baselines\n\nWeaknesses:\n\n- Flow module and latent variables:\nIn Figure 1, we can see that there are two sources of information that contribute to generating frame X_s. First source is Z_s and the second source is F. From observing the provided videos, it looks like the synthesized frames are mostly a result of warping which means that Z_s is potentially being completely ignored, and thus, there is no point in having it to begin with since F provides all necessary information. The authors claim that FVD should provide a measure of diversity since we are comparing the generative distribution and real distribution. However, since the entire dataset is used to compute this number, and not individual videos, then it is hard to claim diversity unless the generation scenario is unconditional (i.e., no encoder, so F becomes impossible to apply). Is it possible to provide diversity metrics for the conditioned scenarios? For example, sample multiple trajectories from the same condition and measure the distance between them. It would be good to provide this for all datasets, and not just the BAIR dataset, to make sure this applies to all experiments.\n\n\n- How is the flow module learned?\nIt is not clear to me how the optical flow module is learned. Is this completely learned using the frame reconstruction signal? So, features from the encoder are warped into the decoder using the optical flow module and the reconstruction error signal flows through this operation?\n\n\n- Visual comparisons against baselines (or best baseline)?\nI was not able to find any visual comparisons between the proposed method and baselines. This would be good to have to see if/how the baselines fail where the proposed method succeeds.\n\n\n- Thin experimental section except for BAIR experiments:\nThe experimental section presents a good amount of content for the BAIR experiments. However, the subsections for Kinetics and AudioSet-Drum are very thin. There are no real discussions or analysis of what was observed in these experiments. I understand that the page limits are usually not ideal, but this is where authors have to figure out how to organize the paper in a way that sections do not look too thin.\n\n\nConclusion:\nThe proposed method is able to generate good looking videos by combining discrete variables, warping, transformers and Adversarial objectives. Having said that, there are a few questions that I have about the method to get a good idea about the performance of the proposed method. My current score is slightly above borderline, but I am willing to increase my score depending on the author’s response.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They have discussed potential negative societal impact, but as far as I can tell there is no discussion of limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "8kxDcTr3yDF", "reviewer": "Reviewer_ZPQv", "summary": "This paper introduced a conditional video prediction algorithm based on VQ-VAE [57] and Vision Transformer (ViT). Similar to VQ-VAE, the frame-level discrete representation is learned by autoencoder with vector quantization, whose (conditional) prior on the latent representation is modeled by the variants of ViT. The authors also incorporated a useful inductive bias for video prediction by warping features based on optical flow estimation, as in [63]. The practical impacts of the proposed method are demonstrated in various short-term video prediction tasks, showing competitive performance to the previous SOTAs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper proposes a general video prediction framework that can incorporate various structured conditional signals. The experiment results show that indeed the generated results are reasonable across diverse conditioning signals.\n- The performance of the proposed method seems to be superior/competitive to the previous SOTAs on video prediction. Thorough ablation studies were helpful to understand the contribution of the model.\n\nCons:\n- The proposed method cleverly combined many “working ideas” from the previous methods. Although its motivation is sound and performance is decent, it (1) lacks novel technical contributions and (2) often misses proper citations. In terms of technical novelty, it follows the idea of combining VQ-VAE with the Transformer as in previous works (e.g., [46]). Although the authors improved its architecture by modifying both the Transformer and VQ-VAE autoencoders, the modifications are also largely based on the existing techniques (ViT [A1] for Transformer and feature warping [63] for autoencoder). In terms of citation, the paper misses many important citations, making readers unfamiliar with this domain difficult to assess its contribution. For instance, the paper misses citation to ViT [A1] although it largely borrows their architecture. The feature warping in Eq.(4) was also introduced without proper citation to its original paper ([63];  Eq.(4)). \n\n[A1] Dosovitskiy et al., An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, In ICLR, 2021\n\n- In principle, the proposed method can generate diverse futures. It would be great if authors can provide some qualitative results of diverse synthesis results (images or videos) conditioned on the same input. The current supplementary file provides results only on different conditionings. \n\n- It would be interesting to see if the proposed ViT+VQ-VAE architecture has a strength in long-term generalization in testing (e.g., learned to generate 15 frames into the future and evaluated to generate 50 frames in testing). Some experiments/comparisons to the other methods would be helpful to get more insights.\n\n- The current draft misses important implementation details in the main paper; for instance, it was really unclear to me how the flow estimator is trained (Ln 189-199). \n\n===== Post-rebuttal update ======\n\nThe authors addressed some of my major concerns in the rebuttal. I think the paper misses proper citations/discussions to the prior works (even if some of them are rather \"conventional\"). Although the technical contribution of this paper seems to be not significant, the results are promising. Overall, it seems to me that it is reasonable submission to NeurIPS. I mildly recommend the acceptance.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a conditional video synthesis model based on the vision transformer architecture and the VQVAE architecture All the reviewers considered the paper above the bar. The rebuttal successfully answered several questions raised by the reviewers, with two reviewers upgraded the score to more positive ratings. Overall, the reviewers consider the paper a welcomed extension of the transformer plus VQVAE paradigm for video synthesis. The quantitative results were convincing. The meta-reviewer agrees with the assessment and would like to recommend its acceptance.
Decision: Accept (Poster)