MCVD - Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Authors: Vikram Voleti
Alexia Jolicoeur-Martineau
Christopher Pal
Venue: NeurIPS 2022 Accept (2022)
Keywords: score-based denoising diffusion
video prediction
video generation
video interpolation

Abstract: Video prediction is a challenging task. The quality of video frames from current state-of-the-art (SOTA) generative models tends to be poor and generalization beyond the training data is difficult. 
Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general-purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using a probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in a manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train a single model that is capable of executing a broad range of video tasks, specifically: future/past prediction -- when only future/past frames are masked; unconditional generation -- when both past and future frames are masked; and interpolation -- when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using $\le$ 4 GPUs. 

Project page: \url{https://mask-cond-video-diffusion.github.io}

Code: \url{https://mask-cond-video-diffusion.github.io/}

MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
Vikram Voleti*
Alexia Jolicoeur-Martinenn" Mila, University of Montreal Canada
Mila, University of Montreal
Canada
vikram.voleti mooleti@uontteal.ccmm ca
Christopher Pal
Mila, Polytechnique Montreal
Canada CIFAR AI Chair
ServiceNow Research
Abstract
Video prediction is challenging task. The quality of video frames from current state-of-the-ar (SOTA) generative models tends be poor and generalization beyond the training data is difficult. Furthermore, existing prediction frameworks are typically not capable of simultaneously handling other video-related tasks such as unconditional generation or interpolation. In this work, we devise a general- purpose framework called Masked Conditional Video Diffusion (MCVD) for all of these video synthesis tasks using probabilistic conditional score-based denoising diffusion model, conditioned on past and/or future frames. We train the model in manner where we randomly and independently mask all the past frames or all the future frames. This novel but straightforward setup allows us to train single model that is capable of executing a broad range of video tasks, specifically: future/past prediction when only future/past frames are masked; unconditional generation when both past and future frames are masked; and interpolation when neither past nor future frames are masked. Our experiments show that this approach can generate high-quality frames for diverse types of videos. Our MCVD models are built from simple non-recurrent 2D-convolutional architectures, conditioning on blocks of frames and generating blocks of frames. We generate videos of arbitrary lengths autoregressively in a block-wise manner. Our approach yields SOTA results across standard video prediction and interpolation benchmarks, with computation times for training models measured in 1-12 days using < 4 GPUs.
Project page: https: //a//maakaaaa/ mwndaaaoodddoaaaoeeooaoaaaaa diw.:oodddiiiiiiiii--d---------- ieaiiinngittttti github.ii io Code: https: htt:///maaaaaaaeee wwwo-ooddoddddieedooooeo diffusionn.oooooo github.is
Introduction
Predicting what one may visually perceive in the future is closely linked to the dynamics of objects and people. As such, this kind of prediction relates many crucial human decision-making tasks ranging from making dinner to driving a car. video models could generate full-fledged videos in pixel-level detail with plausible futures, agents could use them to make better decisions, especially safety-critical ones. Consider, for example, the task of driving car a tight situation at high speed. Having accurate model of the future could mean the difference between damaging car
Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

something worse. We can obtain some intuitions about this scenario by examining the predictions of our model in Figure where we condition on two frames and predict 28 frames into the future for car driving around corner. We can see that this is enough time for two different painted arrows to pass under the car. If one zooms in, one can inspect the relative positions of the arrow and the Mercedes hood ornament in the real versus predicted frames. Pixel-level models of trajectories, pedestrians, potholes, and debris on the road could one day improve the safety of vehicles.
Our approach generates high quality frames many steps into the future: Given two conditioning frames from the Cityscapes [Cordts al., 2016] validation set (top left), we show 7 predicted future frames in row below, then skip frames 20-28, autoregressively predicted in row 4. Ground truth frames are shown in rows and 3. Notice the initial large arrow advancing and passing under the car. In frame 20 (the far left of the 3rd and 4th row), the initially small and barely visible second arrow in the background of the conditioning frames has advanced into the foreground. Result generated by our MCVD concat model variant. Note that some Cityscapes videos contain brightness changes, which may explain the brightness change in this sample.
Figure 1:
Although beneficial to decision making, video generation is an incredibly challenging problem; not only must high-quality frames be generated, but the changes over time must be plausible and ideally drawn from an accurate and potentially complex distribution over probable futures. Looking far in time is exceptionally hard given the exponential increase in possible futures. Generating video from scratch or unconditionally further compounds the problem because even the structure of the first frame must be synthesized. Also related to video generation are the simpler tasks of a) video prediction, predicting the future given the past, and b) interpolation, predicting the in-between given past and future. Yet, both problems remain challenging. Specialized tools exist to solve the various video tasks, but they rarely solve more than one task at time.
Given the monumental task of general video generation, current approaches are still very limited despite the fact that many state of the art methods have hundreds of millions of parameters [Wu et al., 2021, Weissenborn et al., 2019, Villegas et al., 2019, Babaeizadeh et al., 2021]. While industrial research is capable of looking at even larger models, current methods frequently underfit the data, leading to blurry videos, especially in the longer-term future and recent work has examined ways in improve parameter efficiency [Babaeizadeh et al., 2021]. Our objective here to devise a video generation approach that generates high-quality. time-consistent videos within our computation budget of < GPU) and computation times for training models two weeks. Fortunately, diffusion models for image synthesis have demonstrated wide success, which strongly motivated our use of this approach. Our qualitative results in Figure also indicate that our particular approach does quite well at synthesizing frames in the longer-term future (i.c., frame 29 in the bottom right corner).
One family of diffusion models might be characterized as Denoising Diffusion Probabilistic Models (DDPMs) [Sohl- -Sohi-Dickstein et al., 2015, Ho et al., 2020, Dhariwal and Nichol, 2021], while another as Score-based Generative Models (SGMs) [Song and Ermon, 2019, Li et al., 2019, Song and Ermon, 2020, Molicoeur-Martineee et al., 2021a]. However, these approaches have effectively merged into a field we shall refer to as score-based diffusion models, which work by defining a stochastic process from data to noise and then reversing that process to go from noise to data. Their main benefits are that they generate very 1) high-quality and 2) diverse data samples. One of their drawbacks that solving the reverse process is relatively slow, but there are ways to improve speed [Song et al., 2020,

---

Jolicoeur-Martnnee et al., 2021b, Salimans and Ho, 2022, Liu et al., 2022, Xiao et al., 2022]. Given their massive success and attractive properties, focus here on developing our framework using score-based diffusion models for video prediction, generation, and interpolation.
Our work makes the following contributions:
1. A conditional video diffusion approach for video prediction and interpolation that yields SOTA results.
2. A conditioning procedure based on masking past and/or future frames in blockwise manner giving single model the ability to solve multiple video tasks: future/past prediction, unconditional generation, and interpolation.
3. A sliding window blockwise autoregressive conditioning procedure to allow fast and coherent long-term generation (Figure 2).
4. A convolutional U-net neural architecture integrating recent developments with a conditional normalization technique we call SPAce-TIme Teee--TmemA-apeee Adaptive Normalization (SPATIN) (Figure 3).
By conditioning on blocks of frames in the past and optionally blocks of frames even further in the future, we are able to better ensure that temporal dynamics are transferred across blocks of samples, ie. our networks can learn implicit models spatio-temporal dynamics to inform frame generation. Unlike many other approaches, we do not have explicit model components for spatio-temporal derivatives or optical flow or recurrent blocks.
2 Conditional Diffusion for Video
Let Xg E Rd be a sample from the data distribution Pdata- A sample xa can corrupted from 0 to t T through the Forward Diffusion Process (FDP) with the following transition kernel:
(x, x) (Xx;;; v 1, 3,T.
(1)
Furthermore, can be sampled directly from x using the following accumulated kernll:
qe (x, xa) = (xx; Vā,xo, (1 - a) = x = Vā + VI āGE
where a, = - (1 - B.). and c ~ (0,
Generating new samples can be done by reversing the FDP and solving the Reverse Diffusion Process (RDP) starting from Gaussian noise XT. It can be shown (Song et al. [2021], Ho et [2020]) that the RDP can be computed using the following transition kernel:
Pe (x X2, Xo) = -N (x,, Xo), 3,I)
13 X0 + Vagl 1 a(-1))
1 - āc-1 3,
where Xx(,
and 3.
(3)
1 Cr
1 at
Since Xy given x; is unknown, it can be estimated using eq. (2): xo (x VI - / /Va, where (o(x, [f) estimates using time-conditional neural network parameterized by 0. This allows us to reverse the process from noise data. The loss function of the neural network is:
L(0)= = E, xo xo er ||c- co coVox + /1 - ã.c tt[[2]
(4)
Note that estimating is equivalent to estimating scaled version of the score function (i.e., the gradient of the log density) the noisy data:
Xx, log ge (x: xo) - Ct.
(x:
/axxo)
(5)
VI
Thus, data generation through denoising depends on the score-function, and can be seen as noise- conditional score-based generation.
Score-based diffusion models can be straightforwardly adapted to video by considering the joint distribution of multiple continuous frames. While this is sufficient for unconditional video generation, other tasks such as video interpolation and prediction remain unsolved. A conditional video prediction model can be approximately derived from the unconditional model using imputation [Song et al., 2021]; I; indeed, the contemporary work of Ho et [2022] attempts to use this technique; however, their approach is based approximate conditional model.
3

---

2.1 Video Prediction via Conditional Diffusion
We first propose directly model the conditional distribution of video frames in the immediate future given past frames. Assume we have past frames p {p') and current frames in the
immediate future X {xb] 1= We condition the above diffusion models on the past frames to
predict the current frames:
Txidpred (0) Er, 6- |||- (/e-<ele -0 + vI dre p. |1
(6)
Given model trained above, video prediction for subsequent time steps be achieved by blockwise autoregressively predicting current video frames conditioned on previously predicted
frames (see Figure 2). We use variants of the network shown in Figure to model Eg in Equation here, and for Equation 7 and Equation 8 below.
Real
Prediction
Real Past
1
Prediction
Prediction
Prediction
10
I
6
8
9
10
/
11
12
Figure (Ahove) Blockwise autoregressive prediction with our model. (Right) shows this strategy where the top row and third row are ground truth, and the second and fourth rows show the blockwise autoregressively gener- ated frames using our approach.
2.2 Video Prediction + Generation via Masked Conditional Diffusion
Our approach above allows video prediction, but not unconditional video generation. As second approach, we extend the same framework to video generation by masking (zeroing-out) the past frames with probability Pmask = 1/2 using binary mask my. The network thus learns predict the noise added without any past frames for context. Doing so means that we can perform conditional as well as unconditional frame generation, i.c., video prediction and generation with the same network. This leads to the following loss (B is the Bernouilli distribution):
|||e co(va, + VI- āGe m.p.,t]l
Lvddgnn ()
(7)
We hypothesize that this dropout-like [Srivastava al., 2014] approach will also serve as a form of regularization, improving the model's ability to perform predictions conditioned on the past. We see positive evidence of this effect our experiments see the MCVD past-mask model variants in Tables 3 and 9 versus without past-masking Note that random masking is used only during training.
2.3 Video Prediction + Generation + Interpolation via Masked Conditional Diff'usion
We now have design for video prediction and generation, but it still cannot perform video interpo- lation nor past prediction from the future. As a third and final approach, we show how to build general model for solving all four video tasks. Assume we have p past frames, & current frames, and future frames = { i=1-" We randomly mask the p past frames with probability Pmack = 1/2, and similarly randomly mask the f future frames with the same probability (but sampled separately). Thus, future or past prediction is when only future or past frames are masked. Unconditional genera- tion when both past and future frames are masked. Video interpolation is when neither past nor future frames are masked. The loss function for this general video machinery is:
((0) = Ep ff e~ (O.T) (m.. m |||c colVaixo + /1 - ā.c mpp. P. mym f.t)]

---

Doring training
frames P
frames
Current frames No
level
Noise level
bxpx3xhxw
bxfx3xhxw
bxkx3xhxw
bxkx3xhxw
Past mask
Future mask
Redhape
Positional embedding e
"A (r 1)
1r or 1)
Notey current frames bx 3kxhxw
Reshapa
Reshape
frames bx xww
FC Act
bx 3tw
U-Net
FC
Concatenate
Residual
Conditional frames
Residual block
During sampling (T steps)
3x xhww
I 1T 1/T
Residual block
During training 1/T)
Residual block
SPATIN
GroupNorm
Residual block
SPATIN
Conv
Residual
Interpolate to correct size
SPATIN
GroupNorm
Predicted noise bx x kkw
FC
Conv
Dropout
Act
Denoise
Conv
Conv
Conv
Conv
Denaised current frames N..
bxkx3xhxw
Scale
Offsct
Figure 3: We give noisy current frames U-Net whose residual blocks receive conditional information from past/future frames and noise-level. The output is the predicted noise in the current frames, which denoise the current frames. At test time, we start from pure noise.
2.4 Our Network Architecture
For our denoising network we use a U-net architecture [Ronneberger et al., 2015, Honari et al., 2016, Salimans et al., 2017] combining the improvements from Song et al. [2021] and Dhariwal and Nichol [2021]. This architecture uses a mix of 2D convolutions [Fukushima and Miyake, 1982], multi-head self-attention [Cheng et al., 2016], and adaptive group-norm [Wu and He, 2018]. We use positional encodings the noise level (1 [0, 1)) and process it using a transformer style positional embedding:
e(t) = ...,
[...,cos cos (tc #²) ssin te
where d = I,..,,., D/2, D is the number of dimensions of the embedding, and c = 10000. This embedding vector passed through fully connected layer, followed by an activation function and another fully connected layer. Each residual block has an fully connected layer that adapts the embedding to the correct dimensionality.
To provide Xt, p. and to the network, we separately concatenate the past/future conditional frames and the noisy current frames in the channel dimension. The concatenated noisy current frames are directly passed as input to the network. Meanwhile, the concatenated conditional frames are passed through an embedding that influences the conditional normalization akin SPatially-Adapti (DE)normalization (SPADE) [Park et al., 2019]; to account for the effect of time/motion, we call this approach SPAce-TIme-Aree Addee-Adetive Normalization (SPATIN). In addition SPATIN, we also try directly concatenating the conditional and noisy current frames together and passing them as the input. In our experiments below we show some results with SPATIN and some with concatenation (concat). For simple video prediction with Equation 6, we experimented with 3D convolutions and attention However, this requires an exorbitant amount of memory, and we found no benefit in using 3D layers over 2D layers at the same memory (i.e., the biggest model that fits in 4 GPUs). Thus, we did not explore this idea further. We also tried and found no benefit from gamma noise [Nachmani et al., 2021], LI loss, and F-PNDM sampling [Liu et al., 2022].
5

---

3 Related work
Score-based diffusion models have been used for image editing [Meng et al., 2022, Saharia et al., 2021, Nichol et al., 2021] and our approach to video generation might be viewed as an analogy to classical image inpainting, but in the temporal dimension. The GLIDE or Guided Language to Image Diffusion for Generation and Editing approach of Nichol et al. [2021] uses CLIP-guided diffusion for image editing, while Denoising Diffusion Restoration Models (DDRM) Kawar et al. [2022] additionally condition on a corrupted image to restore the clean image. Adversarial variants of score-based diffusion models have been used to enhance quality Molicoeur-Martineu et al., 2021a] or speed [Xiao et al., 2022].
Contemporary work to our own such as that of Ho et al. [2022] and Yang et al. [2022] also examine video generation using score-based diffusion models. However, the Video Diffusion Models (VDMs) work of Ho et al. [2022] approximates conditional distributions using gradient method for conditional sampling from their unconditional model formulation. In contrast, our approach directly works with conditional diffusion model, which we obtain through masked conditional training, thereby giving us the exact conditional distribution as well as the ability to generate unconditionally. Their experiments focus on: a) unconditional video generation, and b) text-conditioned video generation, whereas our work focuses primarily on predicting future video frames from the past, using our masked conditional generation framework. The Residual Video Diffusion (RVD) of Yang et al. [2022] is only for video prediction, and it uses a residual formulation to generate frames autoregressively one at time. Meanwhile, ours directly models the conditional frames to generate multiple frames in a biock-wise autoregressive manner.
Recurrent neural network (RNN) techniques were early candidates for modern deep neural ar- chitectures for video prediction and generation. Early work combined RNNs with a stochastic la- tent variable (SV2P) Babaeizadeh et al. [2018a) 8a] and was optimized by variational inference. The stochastic video generation (SVG) approach of Denton and Fergus [2018] learned both prior and per time step latent variable model, which in- fluences the dynamics of an LSTM at each step. The model is also trained in manner similar to variational autoencoder, i.e., it was another form of variational RNN (vRNN). To address the fact that vRNNs tend to lead to blurry results, Castrejón et al. [2019] (Hier-v >RNN) increased the expressiveness of the latent distributions us- ing a hierarchy of latent variables. We com- pare qualitative result of SVG and Hier-vRNN with the MCVD concat variant of our method in Figure 4. Other vRNN- based models include SAVP Lee et al. [2018] 8]. SRVP Franceschi et al. [2020], SLAMP Akan et al. [2021].
Figure 4: Comparing future prediction methods on Cityscapes: SVG-LP (Top Row), Hier-vRNNs (Second Row), Our Method (Third Row), Ground Truth (Bottom Row). Frame 2, a ground truth conditioning frame is shown in first column, fol- lowed by frames: 3, 5, 10 and 20 generated by each method vs the ground truth at the bottom.
The well known Transformer paradigm [Vaswani al., 2017] from natural language processing has also been explored for video. The Video-GPT work of Yan et al. [2021] applied an autoregressive GPT style [Brown et al., 2020] transformer to the codes produced from a VQ-VAE [Van Den Oord et al., 2017]. The Video Transformer work of Weissenborn et al. [2019] models video using 3-D spatio- temporal volumes without linearizing positions in the volume. They examine local self-attention over small non-overlapping sub-volume or 3D blocks. This is done partly to accelerate computations on TPU hardware. Their work also observed that the peak signal-to-noise ratio (PSNR) metric and the mean-structt stran-struttural similarity (SSIM) metrics [Wang et al., 2004] were developed for images, and have serious flaws when applied to videos. PSNR prefers blurry videos and SSIM does not correlate well to perceptual quality. Like them, we focus on the recently proposed Frechet Video Distance (FVD) [Unterthiner et al., 2018), computed over entire videos and which is sensitive to visual quality, temporal coherence, and diversity of samples. Rakhimov et al. [2020] (LVT) used transformers to

---

predict the dynamics of video in latent space. Le Moing et al. [2021] (CCVS) also predict in latent space, that of an adversarially trained autoencoder, and also add a learnable optical flow module.
Generative Adversarial Network (GAN) based approaches to video generation have also been studied extensively. Vondrick et al. [2016] proposed early GAN architecture for video, using spatio-lemporal CNN. Villegas et al. [2017] proposed a strategy for separating motion and content into different pathways of a convolutional LSTM based encoder-decoder RNN. Saito et al. [2017] (TGAN) predicted sequence of latents using a temporal generator, and then the sequence of frames from those latents using an image generator. TGANx2 Saito et al. [2020] improved its memory efficiency. MoCoGAN Tulyakov et al. [2018] explored style and content separation, but within a CNN framework. Yushchenko et al. [2019] used the MoCoGAN framework by re-formulating the video prediction problem as a Markov Decision Process (MDP). FutureGAN Aigner and Körner [2018] used spatio-temporal 3D convolutions in an encoder decoder architecture, and elements of the progressive GAN Karras et al. [2018] approach to improve image quality. TS-GAN Munoz et al. [2021] facilitated information flow between consecutive frames. TriVD-GAN Luc et al. [2020] proposes novel recurrent unit in the generator to handle more complex dynamics, while DIGAN Yu et al. [2022] uses implicit neural representations in the generator.
Video interpolation was the subject of flurry of interest in the deep learning community a number of years ago [Niklaus et al., 2017, Jiang et al., 2018, Xue al., 2019, Bao et al., 2019]. However, these architectures tend to be fairly specialized to the interpolation task, involving optical flow or motion field modelling and computations. Frame interpolation is useful for video compression; therefore. many other lines of work have examined interpolation from compression perspective. However, these architectures tend to be extremely specialized to the video compression task [Yang et al., 2020].
The Cutout approach of DeVries and Taylor [2017] has examined the idea of cutting out small continuous regions of input image, such small squares. Dropout [Srivastava al., 2014] the MeatureMap level was proposed and explored under the name of SpatialDropout in Tompson et al. [2015]. Input Dropout [de Blois al., 2020] has been examined in the context of dropping different channels of multi-modal input imagery. such as the dropping of the RGB channels or depth map channels during training, then using the model without one of the modalities during testing, e.g. in their work they drop the depth channel.
Regarding block-autoregressiv approach, previous video prediction models were typically either 1) non-recurrent: predicting all n frames simultaneously with no way of adding more frames (most GAN-based methods), 2) recurrent in nature, predicting frame at time in an autoregressive fashion. The benefit of the non-recurrent type is that you can generate videos faster than 1 frame at a time while allowing for generating as many frames as needed. The disadvantage is that it is slower than generating all frames at once, and takes up more memory and compute at each iteration. Our model finds a sweet spot in between in that it block autoreerrree:::: generating time recurrently to finally obtain n frames.
frames at a
4 Experiments
We show the results of our video prediction experiments on test data that was never seen dur-
ing training in Tables 1 --4 for Stochastic Moving MNIST (SMMNIST) 2, KTH 3, BAIR 4 and Cityscapes rrspectively. We present unconditional generation results for BAIR in Table 5 and UCF-101 6 in Table 6, and interpolation results for SMMNIST, KTH, and BAIR in Table 7.
Datasets:
We generate 128x128 images for Cityscapes and 64x64 images for the other datasets. See our Appendix and supplementary material for additional visual results. Our choice of datasets is in order of progressive difficulty: 1) SMMNIST: black-and-white digits; 2) KTH: grayscale single-humans: 3) BAIR: color, multiple objects, simple scene; 4) Cityscapes: color, natural complex natural driving scene; 5) UCF101: color, 101 categories of natural scenes. We process these datasets similarly to prior works. For Cityscapes, each video is center-cropped, then resized to 128 x 128. For UCF101, each video clip is center-cropped at 240x240 and resized to 64x64, taking care to maintain the train-test splits.
[Denton and Fergus, 2018, Srivastava et al., 2015] [Schuldt et al., 2004]
[Eberr al,, 2017]
|Corddts
al., 2016] 6 [Soomro et al., 2012]
7

---

Unless otherwise specified, we set the mask probability to 0.5 when masking was used. For sampling, we report results using the sampling methods DDPM [Ho et al., 2020] or DDIM [Song et al., 2020] with only 100 sampling steps, though our mod- els were trained with 1000, to make sampling faster. We observe that the metrics are generally better using DDPM than DDIM (except for UCF. 101). Using 1000 sampling steps could yield better results.
Table I: Video prediction results on SMMNIST (64 x 64) for 10 predicted frames conditioned on 5 past frames. We predicted 10 trajectories per rear video, and report the average FVD and maximum SSIM, averaged across 256 test videos.
SMMNIST [5 - 10; trained on k]
FVD1
SSIM
SVG [Denion and Fergus, 2018]
10
90,81
0.688
vRNN IL [Castrejón et al., 2019]
10
63.81
0.763
Hier-vRNN [Castrejón et al., 20191
10
57.17
0.760
MCVD concat (Ours)
5
25.63
0.786
MCVD spatin (Ours)
5
23.86
0.780
Note that all our models are trained to predict only 4-5 current frames at time, unlike other models that predict 2 10. We use these models to then autoregressively predict longer sequences for prediction or generation. This was done in order to fit the models in our GPU memory budget. Despite this disadvantage, we find that our MCVD models perform better than many previous SOTA methods.
Metries: As mentioned earlier, we primarily use the FVD met- ric for comparison across mod- els as FVD measures both fidelity and diversity of the generated samples. Previous works com- pare Frechet Inception Distance (FID) [Heusel et al., 2017] and Inception Score (IS) [Salimans et al. 2016], adapted to videos by replacing the Inception network with 3D-convolutional network that takes video input. FVD is computed similarly to FID, but using an I3D network trained on the huge video dataset Kinetics-400. We also report PSNR and SSIM.
Table 2: Video prediction results on KTH (64 x 64), predicting 30 and 40 frames using models trained to predict k frames at time. All models condition on past frames, on 256 test videos.
KTH [10 + pred; trained on k]
k pred
FVD1
PSNRT
SSIM
SAVP [Lee ct al., 2018]
10
30
374 + 3
26.5
0.756
MCVD concat (Ours)
5
30
323
3
27.5
0.835
SLAMP [Akan al., 2021]
10 30
228
5
29.4
0.865
SRVP [Franceschi et al., 2020]
30
222 +
29.7
0.870
MCVD concat (Ours)
5 40
276.7
26.40
0.812
SAVP-VAE [Lee et al., 2018]
10
40
145.7
26.00
0.806
Grid-keypoints [Gao et al., 2021]
10
40
144.2
27.11
0.837
Ablation studies: In Table 3 we compare models that use concatenated raw pixels as input to U-Net blocks (concat) to SPATIN variants. We also compare no- no-masking to past-masking variants, i.e. models which are only trained predict the future vs. models which are regularized by being trained for prediction and unconditional generation. It can be seen that our model works across different choices of past frames and generates better quality for shorter videos. This is expected from models of this kind. Moreover, it can be seen that the model trained on the two tasks of Prediction and Generation (i.e., the models with past-mask) performs better than the model trained only on Prediction!
In addition, the appendix contains an ablation study in Table 9 on the different design choices: concat vs concat past-future-malt -aatr vs spatin vs spatin future-mask vs spatin past-future-mass mask. It can be seen that concat is, in general, better than spatin. It can also be seen that the east-future-matt which is a general model capable of all three tasks, performs better at the individual tasks than the models trained only on the individual task. This was demonstrated in Table as well. This shows that the model gains very helpful insights while generalizing to all three tasks, which it does not while training only on the individual task.
variant,
We conducted preliminary experiments with a larger number of frames. Since the models with a larger number of frames were bigger, we could only run them for a shorter time with a smaller batch size than the smaller models. In general, we found that larger models did not substantially improve the results. We attribute this to the fact that using more frames means that the model should be given more capacity, but we could not increase it due to our computational budget constraints. We emphasize that our method works very well with fewer computational resources.
Examining these results we remark that we have SOTA performance for prediction on SMMNIST, BAIR and the challenging Cityscapes evaluation. Our Cityscapes model yields an FVD of 145.5, whereas the best previous result of which we are aware is 418. The quality of our Cityscapes results illustrated visually in Figure and Figure 2 and in the additional examples provided in our Appendix. While our completely unconditional generation results are strong, we note that when
8

---

past masking is used to regularize future predicting models, we see clear performance gains in Table Finally, in Table we see that our interpolation results are SOTA by a wide margin, across experiments on SMMNIST, KTH and BAIR even compared to architectures much more specialized for interpolation.
It can be seen that our proposed method generates better quality videos, even though it was trained on shorter number of frames than other methods. It can also be seen that training on multiple tasks using random masking improves the quality of generated frames than training on the individual tasks.
Table 3: Video prediction results on BAIR (64 x 64) conditioning on p past frames and predicting pred frames in the future, using models trained to predict & frames at time.
BAIR (64 x 64) Ipast pred: trained on k]
k
pred
FVDJ
PSNR
SSIM
LVT [Rakhimov et al., 2020]
15
15
125.8
DVD-GAN-FP [Clark et al., 2019]
15
15
109.8
MCVD spatin (Ours)
5
15
103.8
18.8
0.826
THVD-GAN-FP [Lue al., 2020]
15
103.3
VideoGPT [Yan et al., 2021]
15
15
103.3
CCVS [Le Moing et al., 202
15
15
99.0
MCVD concat (Ours)
5
15
98.8
18.8
0.829
MCVD spatin past-mask (Ours)
15
96.5
18.8
0.828
MCVD concat past-mask (Ours)
15
95.6
0.832
Video Transformer [Weissenborn et al., 2019]
15
15
94-96*
Vit [Babaeizadeh et al., 2021]
15
15
93.6
MCVD concat past-future mask (Ours)
5
15
89.5
16.9
0.780
SAVP [Lee et al., 2018]
14
14
116.4
MCVD spatin (Ours)
14
94.1
19.1
0.836
MCVD spatin past-mask (Ours)
14
0.837
MCVD concat (Ours)
14
90.5
19.1
0.834
MCVD concat past-future-mast mask (Ours)
14
89.6
17.1
0.787
MCVD concat past-mask (Ours)
14
87.9
19.1
0.838
SAVP [Lee et al., 2018]
10
28
143.4
0.795
HliervRNN [Castrejón et al., 2019]
10
28
143.4
0.822
MCVD spatin (Ours)
28
132.1
17.5
0.779
MCVD spatin past-mask (Ours)
28
127.9
17.7
0.789
MCVD concat (Ours)
2
5
28
120.6
17.6
0.785
MCVD concat past-mask (Ours)
5
28
119.0
17.7
0.797
MCVD concat past-future-mask (Ours)
28
118.4
16.2
0.745
94 only the first frames, 96 all subsequences of test frames
Table 4: Video prediction on Cityscapes (128 x 128) conditioning on frames and predicting 28.
SPATIN produce drift towards brighter images with color balance shift in frames further from the start frame on Cityscapes, resulting in increased FVD for SPATIN than the CONCAT variant.
Cityscapes (128 128) + 28; trained on
k
FVDJ
LPIPS}
SSIMI
SVG-LP Denton and Fergus [2018]
10
1300.26
0.549 + 0.06
0.574 + 0.08
vRNN 1I. Castrejón et al. [2019]
10
682.08
0.304 + 0.10
0.609 + 0.11
Hier-vRNN Castrejón al. [2019]
10
567.51
0.264 0.07
0.628 + 0.10
GHVAE Wu al. [2021]
418.00
0.193
0.014
0.740 + 0.04
MCVD spatin past-mask (Ours)
5
184.81
0.121 + 0.05
0.720 1 0.11
MCVD concat past-mask (Ours)
141.31 5
0.112 - 0.05
0.690 1 0.12
5
Conclusion
We have shown how to obtain SOTA video prediction and interpolation results with randomly masked conditional video diffusion models using relatively simple architecture. We found that past-masking was able to improve performance across all model variants and configurations tested. We believe our approach may pave the way forward toward high quality larger-scale video generation.

---

Limitations. Videos generated by these models are still small compared to real movies, and they can still be- come blurry or inconsistent when the number of generated frames is very large. Our unconditional generation results on the highly diverse UCF-101 dataset are still far from perfect. More work is clearly needed to scale these
Table 5: Unconditional generation of BAIR video frames.
BAIR (64 64) [0 pred; trained on 5]
pred
FVD4
MCVD spatin past-mask (Ours)
16
267.8
MCVD concat past-mask (Ours)
16
228.5
MCVD spatin past-mask (Ours)
30
399.8
MCVD concat past-mask (Ours)
30
348.2
models to larger datasets with more diversity and with longer duration video. As has been the case in many other settings, simply using larger models with many more parameters is a strategy that is likely to improve the quality and flexibility of these models were limited to 4 GPUs for work here. There is also a need for faster sampling methods capable of maintaining quality over time.
Given our strong interpolation results, conditional diffusion models which generate skipped frames could make it possible to generate much longer, but consistent video through strat- egy of first generating sparse distant frames in block, followed by an in- terpolative diffusion step for the miss- ing frames.
Table 6: Unconditional generation of UCF-101 video frames.
UCF-101 (64 x 64) [0 + 16: trained on k]
|
k
FVD]
MoCoGAN-MDD [Yushchenko et al., 2019]
16
1277.0
MCVD concat past-mask (Ours)
4
1228.3
TGAN22 [Saito et al., 2020]
16
1209.0
MCVD spatin past-mask (Ours)
4
1143.0
DIGAN (Yu ct al., 2022]
16
655.0
Table 7: Video Interpolation results (64 x 64). Given p past + f future frames - interpolate & frames.
Reporting average of the best metrics out of n trajectories per test sample. 4 (p-++ and k is harder. We used MCVD spatin past-mask for SMMNIST and KTH, and MCVD concat past-future- mask for BAIR. We also include results SMMNIST for "pure" model trained without any masking.
SMMNIST (64 x 64)
KTH (64 X 64)
BAIR (64 x 64)
p+f
k
n
PSNR SSIM
p+f
n
PSNR SSIM
p+f & n
|SSNR SSIM
SVG-LP Deaton and Forgos [2018]
18
7
100
13.543
0.741
18
100
28.131
0.883
18 7 100
18.648
0.846
FSTN Lu al. [2017]
18
7
100
14.730
0.765
18
100
29.431
0.899
18 7 100
19,908
0.850
SepCony Niklaus et al. [2017]
18
7
100
14.759
0.775
18
100
29.210
0.904
18 7 100
21.615
0.877
SuperSloMo Jiang et al. [2018]
18
100
13.387
0.749
18
100
28.756
0.893
SDVI full Xu ct [2020]
18
7
100
16.025
0.842
18
100
29.190
0.901
18 7 100
21.432
0.880
SDVI Xu et al. [2020]
16
7
100
14.857
0.782
16
100
26.907
0.831
16 7 100
19.694 0.852
10
20.944
0.854
15
34.669
0.943
5 100
162 0.932
MCVD (Ours)
10
27.693
0,941
15
34.068
0.942
5 23.408 0.914
pure
18.385
0.802
10
35.611
0.963
Broader Impacts. High-quality video generation is potentially a powerful technology that could be used by malicious actors for applications such as creating fake video content. Our formulation focuses on capturing the distributions of real video sequences. High-quality video prediction could one day find use in applications such as autonomous vehicles, where the cost of errors could be high. Diffusion methods have shown great promise for covering the modes of real probability distributions. In this context, diffusion- future research where the ability to capture modes properly is safety critical. Another potential point of impact the amount of computational resources being spent for these applications involving the high fidelity and voluminous modality of video data. We emphasize the use of limited resources in achieving better or comparable results. Our submission provides evidence for more efficient computation involving fewer GPU hours spent in training time.
techniques for generative modelling may promising avenue for
Acknowledgments and Disclosure of Funding
We thank Digital Research Alliance of Canada for the GPUs which were used in this work. Alexia, Vikram thank their wives and cat for their support. We thank CIFAR for support under the Al Chairs program, and NSERC for support under the Discovery grants program, application ID 5018358.
10

---

References
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5213-3223 2016.
Bohan Wu, Suraj Nair, Roberto
Li Fei-Fei, and Chelsea Finn. Greedy hierarchi- cal variational autoencoders for large-scale video prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2318-2328, 2021.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint (0ti::/9000023444 2019.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Du- mitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.13111 13195, 2021.
Jascha Sohl-Dickstein, Erie Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning. pages 2266-2265. PMLR, 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 2019.
Zengyi Li, Yubei Chen, and Friedrich Sommer. Learning energy- based models in high-dimensional spaces with multi-scale denoising score matching. arXiv preprint arXiv: 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in Neural Information Processing Systems, 2020.
Alexia Jolicoeur-Martinee Rémi Piché-Taillefer, Rémi Tachet des Combes, and Ioannis Mitliagkas. Adversarial score matching and improved sampling for image generation. International Conference on Learning Representations, 2021a. URL arXivpreprintarXiv 2009 05475.
Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv:2010.02500, October 2020. URL //axxiv abs//b1bs20000 02500.0000.........222
Alexia
Ke Li, Rémi Piché- Taillefrr,, Tal Kachman, and Ioannis Mitliagkas. Gotta go fast when generating data with score-based models. arXiv preprint arXiv:2105.14080 2021b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.0057 00512 2022.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. arXiv preprint arXiv:2202.09778 2022.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. International Conference on Learning Representations (ICLR), 2022.
Yang Song, Jascha
Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference Learning Representations, 2021. URL axxixpreprintarxiv::2 13456.
11

---

Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022. URL https://tarttti org/ abs/2204.222222/////////////a 03458.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way prevent neural networks from overfitting. Journal of Machine Learning Research, (5(56):129-1155) 2014.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234 244.241.22 Springer, 2015.
Sina Honari, Jason Yosinski, Pascal Vincent, and Christopher Pal. Recombinator networks: Learning coarse-to-fine feature aggregation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 5743-5752, 2016.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1001.05517, 2017.
Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.
Jianpeng Cheng, Li Dong, and Mirella Lapata. Long short-term memory-networks for machine reading. arXiv preprint arXiv:1601.0611 2016.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
Taesung Park, Ming-Yo Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with apatially-adaptive normalization. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 2337-2346, 2019.
Eliya Nachmani, Robin San Roman, and Lior Wolf. Denoising diffusion gamma models. arXiv preprint ar a0.0v:2110.05948, 2021.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations 2022.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. arXiv preprint arXiv:2111.05826,, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 2021.
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. arXiv preprint arXiv:2201.117953 2022.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt, Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203. 09481, 2022.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Representations, 2018a.
Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In Proceedings of the 35th International Conference Machine Learning, pages 1174-1183, 2018
Lluís Castrejón, Nicolas Ballas, and Aaron Courville, Improved conditional vrans for video prediction. ArXiv, abs/1904.12165, 2019.
12

---

Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. ArXiv, abs/1804.01523, 2018.
Yven Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pages 5233-3246, PMLR, 2020.
Adil Kaan Akan, Erkut Erdem, Aykut Erdem, and Fatma Güney. Slamp: Stochastic latent appearance and motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14728-14737, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst., volume 30, 2017.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Adv. Neural Inform. Process. Syst., 33:1877-1901, 2020.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Adv. Neural Inform. Process. Syst., 30, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric challenges. arXiv preprint
2018
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. In International Conference on Learning Representations, 2017.
Masaki Saito, Eicchi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830-2839, 2017.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: eeemory--fficieen unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128(10):2586-20) 2020.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattem recognition, pages 1526-1535, 2018.
Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth. Markov decision process for video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
Sandra Aigner and Marco Körner. Futuregan: Anticipating the future frames of video sequences using spatio-temporal 3d convolutions in progressively growing gans, arXiv preprint arXiv: 1810,01325, 2018.
13

---

Tero Karras, Timo Aila. Samuli Laine. and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In Int. Conf. Learn. Represent. 2018.
Andres Munoz, Mohammadreza Zolfaghari, Max Argus, and Thomas Brox. Temporal shift gan for large scale video generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3179-3188, 2021.
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-basde adversarial video prediction on large-scale data. arXiv preprint arXiv:2003. 04035, 2020.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wor Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 670-679, 2017.
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang. Erik Lealned-Miller, and Jan Kautz. Super slomo: High quality estimation multiple intermediate frames for video interpolation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 9000-9008 2018.
Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. Int. J. Comput. Vis., 127(8): 106- 1125, 2019.
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth- aware video frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3703-3712, 2019.
Ren Yang, Fabian Mentzer, Lue Van Gool, and Radu Timofte. Learning for video compression with recurrent auto-encoder and recurrent probability model. IEEE Journal of Selected Topics in Signal Processing, 15(2):388-40)) 2020.
Terrance Vries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint / 2017.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In IEEE Conf. Comput. Vis. Pattern Recog.. pages 648-656, 2015.
Sébastien de Blois, Mathieu Garon, Christian Gagné, and Jean-François Lalonde. Input dropout for spatially aligned modalities. In IEEE Int. Conf. Image Process., pages 733-737. 2020.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pages 843-852. PMLR, 2015.
Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEEE, 2004.
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, pages 344-356, 2017.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint 12 2012.
Xiaojie Gao, Yueming Jin, Qi Dou, Chi- Wiin Fu, and Pheng-Ann Heng. Accurate grid keypoint learning for efficient video prediction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5908-5915. IEEE, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pages 6626 6626-6637, 2017.
14

---

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, 2016.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: 1.XXv06907.0677,,, 2019.
Chaochao Lu, Michael Hirsch, and Bernhard Scholkopf. Flexible spatio-tempore networks for video prediction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6523-6531, 2017.
Qiangeng Xu, Hanwang Zhang, Weiyue Wang, Peter Belhumeur, and Ulrich Neumann. Stochastic dynamics for video infilling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2714-2717 2423, 2020.
Seiya Tokui, Ryosuke Okuta, Takuya Akiba, Yusuke Niitani, Toru Ogawa, Shunta Saito, Shuji Suzuki, Kota Uenishi, Brian Vogel, and Hiroyuki Yamazaki Vincent. Chainer: deep learning framework for accelerating the research cycle. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 2002-2011, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. International Conference on Learning Representations, 2018b.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin Murphy, and Honglak Lee. Unsupervised learning of object structure and dynamics from videos. Advances in Neural Information Processing Systems, 32, 2019.
Beibei Jin, Yu Hu, Qiankun Tang, Jingyu Niu, Zhiping Shi, Yinhe Han, and Xiaowei Li. Exploring spatial-temporal multi-frequee frequency analysis for high-fidelity and temporal-consistte video predic- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4554 4563, 2020.
15


Reviews:
[{"review_id": "xz2yaLp8Np", "reviewer": "Reviewer_8MsY", "summary": "This paper introduces the modified U-Net architecture for taking video frame features and the conditional diffusion process for frame-level image generation. Modified U-Net architecture is equipped with conditioning components with non-recurrent architecture fusing frame-level features. The space-time fusing process is conducted in two-step; channel-wise concatenation of frame-level features and space-time adaptive normalization (SPATIN) inspired by SPADE. This non-recurrent and space-time conditioning U-Net architecture is evaluated on the several video generation, prediction, and interpolation datasets. Although the scale of the datasets is restricted, the versatility of the algorithm is validated. ", "questions": "### The effect of $k$ on the proposed algorithm\nThe experiments only show the maximum number of frames during training is 4, 5 for video prediction/generation, and 10 for interpolation. Compared to the other model with $k=10, 15$ for training, I guess that the proposed model seems to show better performance on lower $k$ as explained in lines 257~260. How about the smaller or larger number of frames affect the performance of the proposed algorithm?\n\n### Diversity issues in video generation/prediction\nWhat is the diversity of the generated videos? Most video generation works only report fidelity metrics, such as FVD, FID, and IS. However, the fidelity metrics do not show the diversity of the generated videos. The diversity metric introduced in [1] would be good.\n\n[1] Diverse Video Generation using a Gaussian Process Trigger, ICLR 21\n\n### The scale of the dataset\nThe experiments only contain small-scale datasets. How about applying the proposed algorithm to a larger scale dataset, such as Kinetics, Something-something v2? Is there any bottleneck of the proposed algorithm except the amount of computation? An algorithm with a non-recurrent and sliding window sampling may be weak in this case compared to recurrent counterparts. \n\n### Time coverage of snippets.\nSeveral works typically count the number of frames, not the time span measured in seconds. It would be good to the time span of the video snippets for generation, prediction and interpolation.\n", "limitations": "I’ve mentioned it in the question part.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The author validates the versatility of the proposed neural diffusion architecture for video generation on several video generation tasks--- unconditional generation, prediction, and interpolation.\n\n### Weaknesses\n- The scale of the datasets used in the experiments is small-scale compared to the other generation methods these days.\nThe disadvantage of non-recurrent and sliding window sampling is not explained. The word ‘non-recurrent’ only appears in the abstract.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cNMs3nbxeO2", "reviewer": "Reviewer_LFuX", "summary": "This work tackles the tasks of video generation, prediction and interpolation using a score-based diffusion modeling approach. The main innovation is in using diffusion models to block-wise autoregressively generate/predict/interpolate video both forward and backward in time efficiently (i.e. using <4GPUs in <12days).", "questions": "BAIR, KTH, UCF-101 are some of the standard video generation benchmarks. Although the results tables appear to be quite exhaustive for BAIR/KTH, UCF101 is lacking in comparison... which matters since it is the standard benchmark for unconditional video generation. Could you improve the UCF-101 benchmark by making the experimental procedure clearer (see Appendix A.2 in the [DIGAN](https://openreview.net/pdf?id=Czsdv-S4-w9) paper), also do provide FID and IS results on this dataset, appropriately presented to allow for easy comparison to prior work such as progressiveVGAN,LDVD-GAN/TGAN-F,TGAN-ODE, etc.\n\n\nThe introductory paragraph touching on vehicle safety is an interesting way to introduce the problem, but it isn't touched upon beyond the first paragraph. In the end, how does your work impact this?\n", "limitations": "More effort and thought has to be put into the broader impact statement. Two generic sentences on broader impact is unacceptable for this kind of work.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is well written, clear and provides a straight-forward extension of the diffusion modelling approach to video.\n2. The proposed approach is more computational efficient than concurrent works in the literature (i.e. using <4GPUs in <12days).\n3. Technical contributions include a more efficient blockwise generation mechanism and the ability to interpolate between frames. Concurrent works use similar architectures and are capable of video generation and prediction. This work combines all of these within one architecture.\n\nWeaknesess:\n1. This work misses comparisons to prior video GAN based work. Taking the UCF-101 experiments as an example, although this work motivates the case for focusing on FVD as a metric, prior work utilises FID and IS as metrics. Not evaluating on these metrics makes it difficult to compare this work against prior art. For historical comparison, I would strongly encourage the authors to extend or append their results to a table similar to the one found in other concurrent/prior works (e.g. Table 1 from the [Video Diffusion Models](https://arxiv.org/abs/2204.03458) or [DIGAN](https://openreview.net/forum?id=Czsdv-S4-w9) papers)\n2. This work is limited to 64x64 generation. Many GAN based approaches are more performant and efficient at this resolution. The 64x64 generation should be made explicitely clear in any results/discussions, especially when comparing results to other works. [TGAN-F/LDVD-GAN](https://www.sciencedirect.com/science/article/abs/pii/S0893608020303397) discussed how results from these metrics are dependant on resolution and the pre-processing pipeline.\n3. Although an interesting direction for diffusion models, the novelty of this work in terms of new capabilities is limited when compared to prior art or GAN-based approaches (e.g.  [Deep Video Generation, Prediction and Completion of Human Action Sequences - CVPR 2018](https://arxiv.org/abs/1711.08682)).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HBAa01cW2Ow", "reviewer": "Reviewer_h3Ht", "summary": "The authors proposed a DDPM based model for video prediction, called MCVD, based on randomly masking frames for conditioning. The random masking allows the model to be conditioned on past or future frames, enabling the model to perform a variety of tasks such as infilling, prediction and generation. Authors compared the performance of the proposed method in conditional (on BAIR and Cityescapes), inpainting (on SMMNIST, KTH and BAIR) and unconditional setup (on BAIR and UCF-101), demonstrating its capability on generating high quality videos.\n\n", "questions": "Please check weaknesses. \n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths And Weaknesses:\n===== Strengths\n- The paper is well written and is easy to follow. \n- The authors compared the proposed method to many other models in conditional and unconditional setups.\n- The proposed method is novel (however there are multiple similar parallel works).\n \n===== Weaknesses:\n- Limited datasets. Although, authors tested the proposed method on multiple datasets, I find the choice of datasets to be narrow. In both prediction and unconditional generation, the datasets are limited to smaller size dataset (BAIR and UCF) and there is no experiment on larger datasets such as RoboNet or Kinetics. This makes me wonder if the model is heavily overfitting on the training data and there is not much generalization. FitVid paper (Babaeizadeh et al) has an interesting discussion on how overfitted methods can generate realistic high quality videos but are not capable of generating anything new. \n\n- Analysis. There is limited analysis and ablation studies. While the paper shows great and promising results on video tasks, it is not clear how the design choices affect its results. e.g. why the masking should be random for inpainting? In absence of any ablations, I can argue that a fixed set of conditioned frames is a much better defined problem and may result in even higher quality predictions. Table 3 has some for masking strategies but the paper can use more targeted experiments along the same lines. Another angle is computational issues which is mostly ignored in the paper. One major issue in video prediction literature is the huge number of parameters required for getting good results. How is the situation in a score based model? Does a bigger model substantially improves the result or not? Are these models already overfitting? Such analysis can substantially improve the quality of the paper.\n\n- Limited qualitative results. Given the nature of videos, they are hard to compare on printed paper. I highly recommend authors to host predicted videos in an anonymous website and provide comparative videos with other baselines.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept