Diffusion Models for Video Prediction and Infilling
Authors: Tobias Höppe
Arash Mehrjou
Stefan Bauer
Didrik Nielsen
Andrea Dittadi
Venue: Accepted by TMLR ()

Abstract: Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain.
We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces a new conditioning technique during training.
By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for video prediction, on which we achieve state-of-the-art results, and one for video generation. High-resolution videos are provided at https://sites.google.com/view/video-diffusion-prediction.

Published in Transactions on Machine Learning Research (11/2022)
Diffusion Models for Video Prediction and Infilling
Tobias Höppe
Lobhooepe@gmaiilcccm com
KTH Stockholm
Arash Mehrjou' MPI for Intelligent Systems ETT Zürich
Stefan Bauer*
KTH Stockholm
Didrik Nielsen"
Norwegion Computing Center
Andrea Dittadi'
adit Siitidaa dlu dk
Technicol University of Denmork 8 MPI for Intelligent Systems
Reviewed on OpenReview: https: openreview. net/ forum? id= 1f0lr4AYM6
Abstract
Predicting and anticipating future outcomes or reasoning about missing information in a sequence are critical skills for agents to be able to make intelligent decisions. This requires strong, temporally coherent generative capabilities. Diffusion models have shown remarkable success in several generative tasks, but have not been extensively explored in the video domain. We present Random-Mask Video Diffusion (RaMViD), which extends image diffusion models to videos using 3D convolutions, and introduces new conditioning technique during training. By varying the mask we condition on, the model is able to perform video prediction, infilling, and upsampling. Due to our simple conditioning scheme, we can utilize the same architecture as used for unconditional training, which allows us to train the model in a conditional and unconditional fashion at the same time. We evaluate RaMViD on two benchmark datasets for vidco prediction, on which we achieve state-of-the-arl results, and one for video generation. High-resolution videos provided at https://sites. http://tw......eeeeee////////tta
1 Introduction
Videos contain rich information about the world, and a vast. amount of diverse video data is available. Training models that understand this data can be crucial for developing agents that interact with the surrounding world effectively. In particular, video prediction plays an increasingly important role: autonomous driving (Hu et al. 2020), anticipating events (Zeng et al. 2017), planning (Finn & Levine 2017) and reinforcement learning Haffne et al., 2019) are applications which can benefit from increasing performance of prediction models. On the other hand, video infilling i.e., observing a part of video and generating missing frames can be used for example in planning, estimating trajectories, and video processing. In addition, video models be valuable for downstream tasks such action recognition (Kong & Fu 2018 and pose estimation (Sahin ct al. 2020). However, there has not been extensive research on vidco infilling and most rescarch is focusing on generation or prediction.
Most recent approaches to video prediction are based on variational autoencoders (Bbaeizadeh et al. 2021 Saxena et al. 2021) or GANs (Clark et al. 2019 Luc et al. 2020). Diffusion models (Sohl-Dickstein al.
* "Equal advising.

---

Published in Transactions on Machine Learning Research (11/2022)
X
x
X
X
x
Figure 1: The first two and last two frames of video are given and our model does fill the missing frames very accurate and with much detail.
2015 Ho et al., 2020: Nichol & Dhariwal 2021 Song el al. 2021 Abstreiter et al., 2021 Mittal el al. 2022
Dockhorn et al. 2021 have recently seen tremendous progress on static visual data, even outperforming GANs in image synthesis (Dhariwal & Nichol 2021), but have not yet been extensively studied for videos. Considering their impressive performance on images, it is reasonable to believe that diffusion models may also be useful for tasks in the video domain.
In this paper, we extend diffusion models to the vidco domain via several technical contributions. We use 3D convolutions and a new conditioning procedure incorporating randomness. Our model is not only able to predict future frames of a video but also fill missing frames at arbitrary positions the sequence (see Fig. D. Therefore, our Random-Mask Video Diffusion (RaMViD) can be used for several video completion tasks. We summarize our technical contributions as follows
A novel diffusion-based architecture for video prediction and infilling.
Competitive performance with recent approaches across multiple datasets.
Introduce a schedule for the random masking-
The remainder of this paper is organized as follows: In Section 2 we provide the necessary background on
diffusion models and video prediction and outline relevant related work. Section 3 describes Random-Mask Video Diffusion (RaMViD). In Section 4 we present and discuss extensive experiments several benchmark datasets. We finally conclude with discussion in Section 5
2 Background and related work
Diffusion models. Diffusion-based models generally refer to the class of machine learning algorithms that consist of gradually transforming a complex distribution into unstructured noise and learning to reverse this process to recover the data generating distribution. They have attracted a great deal of attention after being successfully applied to a diverse range of tasks such as image generation (Song et al., 2021 Niu et al. 2020p, audio (Chen et al. 2021), ; graph and shape generation (Cai et al., 2020). The essence of these models is two stochastic (diffusion) processes implemented by Stochastic Differential Equations (SDEs), forward and a backward one. We explain the formulation in the abstract domain here and specialize it later according to the application of this work.
Let Xo E Rd be a sample from the empirical data distribution, i.e., Xo Pasta (Xo) and d be the data dimension. The forward diffusion process takes Xy as the starting point and creates the random trajectory X(o.T] from = 0 to the final time L t= T. The forward process is designed such that p(xr) xx) has simple unst ructured distribution. One example of such SDEs is
dx, = [xxx, + g(()dw == ([)
dt dw
(1)
where is the Brownian motion. A desirable property of this process is the fact that the conditional distribution p(x, Xo) takes a simple analytical form:
p(x, xo) N (xx: Xo, (t) o² 1).
(2)
"odee available at
2

---

Published in Transactions on Machine Learning Research (11/2022)
Upon learning the gradient of p(xx) for each one can reverse the above process and obtain the complex data distribution from pure noise as
dx. [f(x, 1) o2 ( log log + g(t)dw',
(3)
where w' is Brownian motion independent of the one in the forward direction. Hence, generating samples from the data distribution boils down to learning Vx log.
The original score matching objective Hyvärinen & Dayan 2005
Ex, (x,, t) Vx, log p
(4)
is the most intuitive way to learn the score function, but is unfortunately intractable. Denoising Score Matching (DSM) provides a tractable alternative objective function:
JPSM (0) Ex, Ex, [x:,) () - Vx, logp(x, x
(5)
whose equivalence with the original score matching objective was shown by Vincent (2011) and used to train energy models by [Sarmi et al. (2018). Similarly many recent works, we use the DSM formulation of score matching to learn the score function.
Video prediction and infilling. Research in video prediction has received more attention in the previous years, as the ability to predict videos can be used for several downstream tasks (Oprea et al. 20200). Video prediction can be modeled in a deterministic or stochastic form. Deterministic modeling [Walker et al.) 2015
2019 Due to the stochastic nature of the Vondrick & Torralba, 2017: Terwilliger et al. 2019; Sun et al.
2019) tries to predict the most likely future, but this often leads to averaging the future states (Li al.
future, generative models have lately shown to be more successful in capturing the underlying dynamics. For this approach, variational models [Babacizadeh et al. 2018 Saxena et al. 2021 Denton & Fergus 2018 Wu et al. 2021a). However, this often leads to blurry predictions due to underfitting, and [Babaeizadeh et al. (2021) have overcome this problem via architectural novelties. Blurry prediction is a less serious problem in GANs and promising results have been achieved especially on large datasets (Clark et al. of rescarch on vidco infilling is significantly more scarce, with most works in this area focusing on frame interpolation (Jiang et al. 2018). However, Xu el. al. (2018) have shown interesting results in infilling, by modeling the video stochastic generation process.
often used by modeling the stochastic content in latent variable
2019 Luc et al. 2020). On the other hand, the body
Concurrent work. Yang ct al. (2022 is the only work so far that has used diffusion models for autoregressive video prediction, by modeling residuals for a predicted frame. However, since their evaluation procedure and datasots are different, comparison with their work is not possible. A few concurrent works have recently considered diffusion models for video generation. 2Ho et al. (2022) focus on unconditional video generation, Harvey et al. (2022) use diffusion models to predict long videos, and Voleti et al. (0022). the most similar to our work, also consider video prediction and infilling.
3 Random-Mask Video Diffusion
Our method, Random-Mask Video Diffusion (RaMViD), consists of two main features. First, the way we introduce conditional information different from prior work. Sccond, by randomizing the mask, we can directly use the same approach for video prediction and video completion (infilling). In the following, we detail each of these aspects of RaMVID.
3.1 Conditional training
Let x e L.W.H.C. be video with length L. We partition the video xo into two parts: the unknown frames X E WW,HC and the conditioning frames xf E W.H.W.HC. where U and C are sets of indices such that
Unc = Ø and U UC = {0, I....,L- 1}. We write Xa = x @ xg with the following definition for @:
(a + bbye== afifiel
(6)
biife
3

---

the prior Published in Transactions on Machine Learning Research (11/2022)
X
X
X
X
X
X
Figure 2: An unconditionally trained model is used to predict 15 frames given one frame. Even with re-sampling can see, that objects in the background are not harmonized between the predicted and conditioned frames.
where the superscript indicates tensor indexing and our case corresponds to selecting a frame from a video. Here, indicates the diffusion step, with 0 corresponding to the data and Gaussian distribution.
If we use unconditionally trained model, we find that the predicted unknown frames xy do not harmonize well with the conditioning frames x6. as shown in Fig. 2 One solution for this would be re-sampling, as proposed by Lugmayr et al. (2022). In re-sampling, we take one step in the learned reversed diffusion (denoising) process and then go back by taking a step in the forward diffusion process (i.c., adding noise again). This repeated several times for each diffusion step, to make sure the predicted and conditioning frames are harmonized. However, this becomes computationally too expensive for videos, especially when using very few conditioning frames, the number of re-sampling steps needs to be increased. To mitigate this issue, propose to train the model conditionally with randomized masking.
Conditional diffusion models usually optimize
Exo
(xxixx |xa
[|se(x,, x6, Vx. logp(x, (x0) |x0)| [[]]]
where x is typically given separate input through additional layer (Chen et al. 2021 it is concatenated with the input (Saharia et al., 2021b Batzolis et al. 2021: Saharia et al.) 2021a). On the other hand, we feed the entire sequence to the network se but only add noise to the unmasked frames: x4 ~ N (xl (t² - (0) I). The input to the network is then a video where some frames are noisy and some are clean: X, = x @ x (see Fig. 3). The loss is computed only with respect to x²
[[so(xx,,)) t)at Vazi log xo) x001?
RRMVIDD (8) Ex, Ex
(8)
where (x t)M represents the output of the model with indices in U. Note that the score function V Var log p(x X2) has the same dimension as x",, whereas in Eq. 7 it has the dimension of the entire
video x, This leads to the forward diffusion process:
dxf f(x t)dt + g(t)dw
(9)
and the reversed diffusion process then becomes::
dxf [f(x ( log (xx)((/(1x))) + g(t)dw'
(10)
Similarly, Tashiro et al. 2021 compute the loss only on the unknown input. However, they also use concatenation and zero-padding bring x and x, to the same dimension. For more detailed schematic, see Appendix A In our implementation, we used discrete diffusion process with E 1.... T,,17
3.2 Randomization
As previously mentioned, the proposed model is able to perform several tasks. We achieve this by sampling C at random. At each training step, we first choose the number of conditioning frames |C| = (l..... K), where is a chosen hyperparameter. Then we define C by selecting random indices from {O..... 1}, and refrain from applying the diffusion process to the corresponding frames. Since the videos now consist.
4

---

Published in Transactions on Machine Learning Research (11/2022)
x
x
x
X
Figure 3: Example input of the network with {0,8).
of original and noisy frames in varying positions, the model has to learn to distinguish between them in order to use the frames x6 information for the reversed diffusion process. After training, we can use RaMViD by fixing C to the set of indices of the known frames (C can be any arbitrary subset of ( generating the unknown frames (those with indices in 2).
L- 1}) and
Our approach allows us use the exact same architecture unconditionalyy trained models, thus enabling mixed training, where we train the model conditionally and unconditionally at the same time. We set
C= Ø (i.e., the model does not have any conditional information xf) with probability pu, which is fixed
hyperparameter. If C = Ø, our objective in Eq. (8) becomes the same the objective in Eq. 5 used for unconditional training. The pscudocode for RaMViD is shown in Algorithm 1
Algorithm RaMViD.
Initialize model ~ 8q
T Number of diffusion steps
K Max number of frames to condition on
L Length of the video
while not converged do
(Pdut((x))
~~ Uniform( ((0, ( T})
Bernoull(pp)) if b then C= Ø
else
k Uniform((1......
C~ Uniform (fS C I- [S] = kt)
end if
u = {o.....
L 13
x N (xf;; x (0² -2((0)) I)
x,
+ x
Take gradient step on: VOlxs Ex [so(xx 1)4 Vxy logp(x? xo)>|
end while
4 Experiments
4.1 Experimental setup
Implementation details. Our implementation relies on the official code of Nichol & Dhariwal 2021
adapted to video data by using 3D) convolutions. Even though most previous work uses the cosine noise schodule, we found that the lincar noise schedule works better when training the model conditionally. Therefore, we use a linear diffusion schedule for our experiments. For the architecture, we also use the same as proposed by Nichol Dhariwal (2021): ): a U-Net with self-attention at the resolutions 16 and 8. We do not encode the time dimension. We use ResNet blocks per resolution for the BAIR dataset, and three blocks
5

---

Published in Transactions on Machine Learning Research (11/2022)
for Kinetics-600 and UCF-101. We set the learning rate for all our experiments to 2e-5, use a batch size of 32 for BATR and 64 for Kinetics-600 and UCF-101, and fix T = 1000. We found, especially on the more diverse datasets like Kinetics-600 and UCF-101, that larger batch sizes produce better results. Therefore, to increase the batch size, we use gradient accumulation by computing the gradients for micro-batches of size 2 and accumulate for several steps before doing proppppopag
Datascts and evaluations. To compare our model to prior work, we train it on the BAIR robot pushing dataset (Ebert et al. 2017). The dataset consists of short videos, with 64 x 64 resolution, manipulating different objects. For evaluation, we use the same setting as Rakhiimo et al. (2020), which is to predict the next 15 frames given one observed frame. We train on videos of length 20.
robot arm
Additionally, we evaluate our model on the Kinetics-600 datasel. (Carreira et al. 2018 which consists of roughly 500,000 10-second YouTube clips, also at 64 x 64 resolution, from 600 classes The size and the diversity of this dataset make it a perfect task to investigate if the model captures the underlying real-world dynamics. For downloading and preprocessing we use the dataset's public repository compare our model to concurrent work by predicting frames when conditioned on 5 frames (Lue et al. 2020). We additionally perform several ablation studies on video completion. We train on 16 frames and choose again K = 4.
g On Kinetics-600 600, we
To quantitatively evaluate the unconditional generation performance when using pv > 0, we also train on UCF-101 (Soomro et al. 2012), a common benchmark for unconditional video generation. It consists of 13,320 videos from 101 human action classes. We also rescale this dataset to 64 x 64 and train with K = 4.
To quantitatively evaluate prediction, we use the Fréchet Video Distance (FVD) (Unterthiner et al. 2018 which captures scmantic similarity and temporal coherence between vidcos by comparing statistics in the latent space of a Inflated 3D ConvNet (I3D) trained on Kinetics-400. To evaluate unconditional generation,
use the Inception Score (IS) (Salimans et al. 2016) to measure the quality and diversity of the generated vidcos. As we have to adapt the score to vidcos, we use the public repository from Saito et al. (000
4.2 BAIR
We train four models on the BAIR datasct with Pe E f0, 0.25, 0.5, 0.75) respectively. The models are trained for 250,000 iterations with a batch size of 32 on 8 GPUs.
First, we test our method with the typical evaluation protocol for BAIR (predicting 15 frames, given one conditional frame). With all values of pu, we can achieve state-of-the-art performance, as shown in Table [ By using pv 0, we can even increase the performance of our method. However, it seems that there is tipping point after which the increasing unconditional rate hurts the prediction performance of the model. Interestingly, we find that also the model trained with Pv = 0.75 overcomes the harmonization problem described in Section 3.1 We have trained more models with pu = 0.25 but varying K. When training with K=2 2 we found a slight drop in performance but with K = 8 a slight increase. Furthermore, we experiment with a task-specific model for prediction (i.e. setting = for all training steps), which we will call RaMViD fized. However, we find that this does not improve performance and does not appear to work reliably on other video completion tasks.
Since we train with randomized masking, can also perform video infilling with the same models, without retraining. We condition on the first and last frame (i.e., set C = (0, 15} for sampling) and compute the FVD of the 14 generated frames, Again we find that the performance is very similar for different values of po (see Table 2) and only a slight change in results when using diffcrent values for K. Intorestingly, RaMViD is also able to perform unconditional generation on BAIR for all considered values of pr, as shown in Appendix B.
So far, we have shown that our method works very well for prediction and infilling, However, since the BAIR dataset is arguably rather simple and not very diverse, we will now evaluate RaMViD the significantly
$
https://github..co/pnnec hew.:pcchtttrrreerrrcc////////

---

Published in Transactions on Machine Learning Research (11/2022)
Table 1: Prediction performance on BAIR. The values are taken from Babaeizadeh (2021) after inquiring about the evaluation procedure. Parameter counts were obtained either directly from the papers or by contacting the authors. Since our computational constraints did not allow us to do several runs each method, we only give error bars for RaMViD (pc 0.25)
Method
FVD (4)
# paramcters
Latent Video Transformer Rakhimov et al.
2020
125.8
SAVP (Lee et al. 2018
116.4
DVD-GAN-FP Clark et al.
2019
109.8
THVD-GAN-FP Luc ct al.
2020
103.3
VideoGPT Yan et al. 202T
103.3
40M
Video Transfomer Weissenborn et al.
2020
94.0
373M
Fit Vid Babaaizzdeel et al. 202
93.6
302M
MCVD Voleti et al. 2022
89.5
251.2M
NUWA [Wu et al. 2021b
86.9
RaMViD (pu = 0, K = 4)
86.41
235M
RaMViD (PU = 0.25, K = 4)
85.3 + 1.8
235M
RaMViD (pu = 0.5, K = 4)
85.03
235M
RaMViD (pu = 0.73, = =4)
86.05
235M
RaMViD (PU 0.25, K 2)
87.39
235M
RaMViD (PU 0.25, K
82.64
235M
RaMViD (pC 0.25, fired)
89.01
235M
x
X
X
X
X
X
Figure 4: Prediction of 11 frames given the first frames Kinetics-600 with RaMViD (pu 0.25).
more complex Kinetics-600 dataset. Since using K 4 appears to lead to stable performance on all tasks we evaluated our model on, we will use K = 4 for our experiments on Kinetics-600 and UCF-101.
Table Infilling performance on BAIR.
Method
FVD (1)
RaMViD (pu = 0. K 4)
85.68
RaMViD (pu = 00.25, K 4)
85.02
RaMViD (pu = -0.5, K = 4)
87.04
RaMViD (pU = =0.75, K = 4)
87.85
RaMViD (pU = -0.25, K = 2)
83.83
RaMViD (PU -0.25, K 8)
87.11
RaMViD (PU =0.25, fared)
119.76
7

---

Published in Transactions on Machine Learning Research (11/2022)
Table 3: Prediction performance on Kinetics-600, Values are taken from Moing et al. (2021) after inquiring about the evaluation procedure. Parameter counts were obtained either directly from the papers by contacting the authors. Since our computational constraints did not allow us to do several runs for each method, we only give error bars for RaMViD (pc 0.25)
Method
FVD (4)
# parameters
Video Transfomer Weissenborn et al.
2020
170 = 5
373M
DVD-GAN-FP (Clark al. 2019
69 + 1
CCVS (Moing et al. 202
55 + 1
366M
TMVD-GANNFF (Luc et al.
2020
26 = 1
RaMViD (PU = 0)
18.69
308M
RaMViD (pr = -0.25)
17.53 1.07
308M
RaMViD (PU =0.5)
17.61
308M
RaMViD (PC 0.75)
27.64
308M
Table 4: Performance of RaMViD Kinetics-600, when conditioning different frames.
Method
C = 14,
C = {0, 5, 10, 15)
RaMViD (pu = 0)
10.68
6.28
RaMViD (pu 0.25)
10.85
4.91
RaMViD (pu -0.5)
10.86
5.90
RaMViD (pv 0.75)
17.33
7.29
4.3 Kinetics-600
For the Kinetics-600 dataset, we increase the batch size to 64 and train for 500,000 iterations on 8 GPUs. First, evaluate the model on prediction with the setting described in Section 4.1 (predict 11 frames given 5 frames). When comparing results by a significant margin (see Table 3h. In Fig. [ we can see that the model produces temporally coherent outputs and is able to model details, especially in the background, such as clouds and patterns in the water. Nevertheless, it struggles with fast movements: objects moving quickly often get deformed, as can observed Appendix B.2] Similar to what we have seen Table [ having an unconditional rate pv > 0 increases the performance up to tipping point. However, differently from the model trained on BAIR, the FVD score now drops significantly with pa 0.75. We conjecture that this drop in performance is due the complexity of the data distribution. In BAIR, the conditional and unconditional distributions are rather similar, while this is not true for Kinetics-600.
models to concurrent work, find that RaMViD achicves stato-of-the-art
We also evaluate RaMViD on two video completion tasks on Kinetics-600. The first task is to fill in a video given the two first and last frames (i.c., C= fo, 1, 14, 15)): the challenge here is to harmonize the observed movement at the beginning with the movement observed at the end. In the second task, the conditioning frames are distributed evenly over the sequence (i.c., C = {0, 5, 10, 15)), hence the model has to infer the movement from the statie frames and harmonize them into one realistic video. RaMViD excels on both tasks, shown quantitatively in Table ! and qualitatively in Figs. II and 12 Especially when setting C = (0, 5, 10, 15) RaMViD is able to fill the missing frames with very high quality and coherence. This setting can be easily applied to upsampling by training a model on high-FPS videos and then sampling a sequence conditioned on a low-FPS video.
We find that only RaMVID (Pr 0.5) and RaMVID (pv 0.75) can generate unconditional vidcos on Kinetics-600. To quantify RaMViD's unconditional generation, we will evaluate these models on the UCF-101 dataset and compare it to other work.
8

---

Published in Transactions on Machine Learning Research (11/2022)
Table Generative performance of RaMViD on UCF-101. Note that the methods TGAN-F, VideoGITT and DVD-GAN in Table 5 are trained with 128 x 128 resolution, which gives them slight advantage, the IS score is computed with
x resolution.
Method
(t)
# parameters
resolution
VGAN Vondrick et al. MoCoGAN (Tulykovv et a.. 2018
2016
8.31 + 0.09
64 X 64
12.42 = 0.03
3.3M
64 x 64
TGAN-F (Kahembwe & Ramamoorthy
2020
13.62 = 0.06
17.5M
64 x 64
progressive VGAN (Acharya et al.
2018)
14.56 = 0.05
64 x 64
TGAN-F (Kahembwe Ramamoorl by
2020
22.91 + 0.19
70M
128 x 128
VideoGPT Yan et al. 2021
24.69 = 0.3
200M
128 128
TGANv2 (Saito al.
2020
26.60 = 0.47
200M
64 x 64
DVD-GAN [Clark et al
2019
32.97 + 1.7
128 128
RaMViD (pu = 0.5)
20.84 = 0.08
308M
64 x 64
RaMViD (Pu 0.75)
21.71 + 0.21
308M
64 64
X
10
12 X
x
Figure 5: Unconditional generation on the UCF-101 dataset. The first generation does not have much movement. and is generated very realistically. In the second video, the background is generated properly, but that the fast-moving people are unrealistically deformed.
4.4
UCF-101
We train RaMViD UCF-101 with the same setting as used for Kinetics-600 but for 450,000 iterations. Sshows that model achieves competilive performance unconditional video generation, although it does not reach srate-of-the-art The trained models can successfully generate scenes with static background and a human performing an action in the foreground, consistent with the training dataset (sce Figs. and 13 However, the actions are not always coherent and moving objects can deform over time. Note that UCF-F 101 is very small dataset given its complexity. Therefore we do observe some overfitting. Since for each action only have around 25 different settings, our model does not learn combine those but generates very similar videos to the training set. Due to the characteristies of this dataset think with more extensive hyperparameter tuning, one can achieve better results with RaMViD in unconditional generation. But our focus does not lie on this.
4.5 Autoregressive sampling
While we train our models only on 16 (Kinetics-600) or 20 (BAIR) frames, it is still possible to sample longer sequences autoregressively. By conditioning on the latest sampled frames, one can sample the next sequence and therefore generate arbitrarily long videos. In Fig, G we show examples of this autoregressive sampling with RaMViD (Pr 0.25) trained on Kinetics-600. However, we found that this rather challenging because, at each autoregressive step, the quality of the generated sequence slightly deteriorates. This amplifies over time, often resulting in poor quality after about 30 frames.

---

Published in Transactions on Machine Learning Research (11/2022)
X
X
12
Xg
18
%
24
Xg
30
Xg
36
X(
42
Figure 6: Autoregressive prediction of 45 frames conditioned on frames with RaMViD (pu 0.25) trained Kinetics-600.
5
Conclusion
We have shown that diffusion models, which have been demonstrated to be remarkably powerful for image generation, can be extended to videos and used for several video completion tasks. The way we introd conditioning information is novel, simple, and does not require any major modification to the architecture of existing diffuuson models, but it is nonetheless surprisingly effective. Although the proposed method targets conditional video generation, we also introduce an alternative masking schedule in an attempt to improve the unconditional generation performance without sacrificing performance on conditional generation tasks.
Since we have observed varying performance in different tasks using different masking schemes, an interesting direction for future research is to investigate which masking schedules are more suitable for each task. It would also be interesting to explore in future work whether our conditioning technique is also effective for completion on other data domains. Finally, the focus of this work has been on the diffusion- based algorithm for videos rather than on optimizing the quality of each frame. It has been shown in concurrent works that including super-resolution modules helps create high-resolution videos. Adding a resouutnnolution module to RaMViD would be a relevant direction for future work.
Broader impact statement
Generative models have been a cornerstone of AI research for many years. While some these models have been used for the benefit of society e.g,; in wildlife conservation, they can likewise be used with malicious intent such as for creating deepfakes. Similarly other generative models, the capabilities of our approach for filling and predicting videos can be used for the benefit of all well as unfortunately the opposite. In this work, we not see any specific negative impact beyond the general possibility of malicious users of AI algorithms. In addition, our models require large video datascts for training. For some of these curated datasets, the underlying distribution of samples across various groups might not be uniform and before deploying any machine learning model trained on these datasets these aspects need to be carefully evaluated.
Acknowledgments
This project was enabled by the Berzelius cluster at the Swedish National Supercomputer Center (NSC). We thank our anonymous reviewers for their constructive feedback.
References
Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf, and Arash Mehrjou. Diffusion-based represent ation learning. arXiv preprint arXiv:2105 14257, 2021.
Dinesh Acharya, Zhiwu Huang, Danda Pani Paudel, and Luc Van Gool. Towards high resolution video generation with progressive growing of sliced wasserstein gans, 2018
Mohammad Babacizadeh, Chelsea Finn, Dumitru Erhan, Roy H. Campbell, and Sergey Levine. Stochastic variational video prediction. In International Conference on Learning Regresentations, 2018.
10

---

Published in Transactions on Machine Learning Research (11/2022)
Mohammad Babacizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint or arxiv:206. 13195, 2021.
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. gencration with score-based diffusion models, 2021.
Conditional image
Ruojin Cai, Guandao Yang, Hadar Averbuch-Elor Zekun Hao, Serge Belongie, Noah Snavely, and Bharath Hariharan. Learning gradient fields for shape generation, 2020.
Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, 2018.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In International Conference on Learning Representations, 2021.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Efficient video generation on complex datasets. CoRR, 16651997.06671, 2019. URL httt:://axxiv.org/bbb/////
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Reseorch, pp. 1174-1183. PMLR, 7 2018.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vanghan (eds.), Advances in Neural Information Processing Systems, 2021.
Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Score-based generative modeling with intically-damper langevin diffusion. arXiv preprint ar arx:::1112.0000 07068, 2021.
Frederik Ebert, Chelsea Fimn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1sl Annual Conference on Robol Learning, volume 78 of Proceedings of Machine Learning Reseurch, pp- 344-356. PMLR, 11 2017.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2786-2793, 2017. doi: 10. 2CRA.2017..7999824
Danijar Hafner, Timothy Lillierap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2555 2565. PMLR, 6 2019.
William Harvey, Sacid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffission modeling of long videos, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851 Curran Associates, Inc., 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
Anthony Hu, Fergal Cotter, Nikhil Mohan, Corina Gurau, and Alex Kendall. Probabilistic future prediction for video scene understanding, 2020.
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
11

---

Published in Transactions on Machine Learning Research (11/2022)
Huaizu Jiang, Deqing Sun, Varun Jampani, Ming- Hsuan Yang, Erik Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9000 1000-000,,-000000000000000 2018.
Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video discriminators.
Newral Networks, 132:506-520) 2020. ISSN 0893-6080. doi: https://dbi.orrg/l m00.//...cmmc..220000
Yu Kong and Yun Fu. Human action recognition and prediction: A survey, 2018
Alex X. Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction, 2018.
Maomao Li, Chun Yuan, Zhihui Lin, Zhuobin Zheng, and Yangyang Cheng, Stochastic video generation with disentangled representations. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 224-22222220 229, 2019.
Pauline Lue, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transfrmmation- based adversarial video prediction on large-scale data, 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Lue Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models, 2022.
Sarthak Mittal, Guillaume Lajoie, Stefan Bauer, and Arash Mehrjou. From points to functions: Infinite- dimensional representations in diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. CCVS: Context-aware controllable video synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neurol Information Processing Systems, 2021.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved demoising diffusion probabilistic models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8162-8171. PMLR, 7 2021.
Chenhao NIu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, and Stefano Ermon. Permutation invariant graph generation via score-based generative modeling, 2020.
Sergiu Oprea, Pablo Martinez-Gonzalee Alberto Garcia- Garcia, John Alejandro Castro Castro-Vargas, Sergio Orts-Escolano, Jose Marrii-Roodiiguer and Antonis Argyros. A review on deep learning techniques for vidoo prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer, 2020.
Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Floet, and Mohammad Norouzi. Palette: Image-to-image diffusion models, 2021a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Flect, and Mohammad Norouzi. Image super-resolttion via iterative refinement, 2021b.
Caner Sahin, Guillermo
Juil Sock, and Taun Kim. A review on object pose recovery: from 3d bounding box detectors to full 6d pose estimators, 2020.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memoryy-fficient unsupervised training of high-resolution temporal GAN. International Journal of Computer Vision, 10- (10-11)::2586 2606, 5 2020. doi:
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved Techniques for Training GANs. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett. (eds.), Advances in Newral Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
12

---

Published in Transactions on Machine Learning Research (11/2022)
Saced Saremi, Arash Mehrjon, Bernhard Schölkopf, and Aapo Hyvärinen. Deep energy estimator networks. arXiv preprinl Xiv: 1 10011800.88 08306, 2018.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork varialional autoencoders. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Jascha Sohl-Dicksted Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, Lille, France, 7 2015. PMLR.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012.
Jiangxin Sun, Jiafeng Xie, Jian-Fang Hu, Zihang Lin, Jianhuang Lai, Wenjun Zeng, and Wei-shi Zheng. Predicting future instance segmentation with contextual pyramid convlstms. Proceedings of the 27th ACM International Conference on Multimedio, MM '19, pp. 2043 2051, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450368896. doi: 10.1145/3343031 /////33333113330000
Yusuke Tashiro, Jiarning Song, Yang Song, and Stefano Ermon. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Adam Terwilliger, Garrick Brazil, and Xiaoming Liu. Recurrent flow-guided semantic forecasting. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1703-1712, 2019. doi: 10.1109/WACV. 1A110//WWAC..221.001666
Sergey Tulyakov, Ming-Y Yu Liu, Xiaodong Yang, and Jan Kautz, Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 6 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of vidco: A new metric challenges. CoRR, abs/1812.017177 2018.
Pascal Vincent. A connection between score matching and denoising autoencoders. Newral Compulation, 23 (7):1661-1674, 2011. doi: 10.162/NECO0a_00012
Vikram Voleti, Alexia Jolicoeur-Martineuu, and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022.
Carl Vondrick and Antonio Torralba. Generating the future with adversarial transformers. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1902-3000, 2017. doi: 10.1109/ CVPR.2017.319.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In D. Lee, M. Sugiyama, U. Luxburg, T. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from static image. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2443-245) 2015. doi: 10.1109/ ICCV.2015.200
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In Interna- tional Conference Learning Representations, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2022)
Bohan Wu, Suraj Nair, Roberto Martin-Martin, Li Fei-Fei, and Chelsea Finn. Greedy hicrarchical variational autoencoders for large-scale video prediction. In 2021 IEFE/CVE Conference Computer Vision and Pottern Recognition (CVPR), pp. 2318 2328, 2021a. doi:
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NÜwa: Visual
synthesis pre-training for neural visual world creation, 2021b.
Qiangeng Xu, Hanwang Zhang, Weiyue Wang, Peter N. Belhumeur, and Ulrich Neumann. Stochastic dynamics for video infilling, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers, 2021.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation, 2022.
Kuo-Hao Zeng, William Shen, De-An Huang, Min Sun, and Juan Carlos Niebles. Visual lorecasting by imitating dynamics in natural sequences. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3018-3027, 2017. doi: 10.1109/CCVV.2017.32
14

---

Published in Transactions on Machine Learning Research (11/2022)
A
Implementation details
Fig. i presents a sketch of RaMViD's architecture. Thanks to the way introduce conditioning frames, the architecture does not need to be different from the one in unconditional models.
X
FC layer SILU FC layer
Residual block
bxexlxhxw
Residual block
h/2x/// xx22x1x//2xx//
bx 2x xx
Residual block
bx bx3exIxb/4xw/4
Residual block
bx bxdexIxh/8xxi
Residual block
bx bx w/8
Residual block
bx3xxxb//xx
Residual block
b bx2ex1xh/2xxx
Residual block
bxexixhxw
Residual block
50(x,, t)
so s0(xt, t)
Exo
Exlt
[soe (xxx t)4
x log Po (xff |xo) }
|
Figure 7: Sketch our method. In the last step we only compute the loss with respect to the frames that were corrupted with noise. The number of channels 128, and the video length.
15

---

Published in Transactions on Machine Learning Research (11/2022)
X
10 X
x0
12
14
Figure 8: Unconditional generation on the BAIR dataset sampled from RaMViD for pa 0 (first row) until po = 0.75 (last row). Due to the low complexity of the dataset, we can generate reasonable unconditional videos even with = O. However, the quality of details increases with increasing pu-
As mentioned, we use the linear noise schedule and the score-based ("simple") objective for all experiments. All models are trained with 1000 diffusion steps, for sampling we used 750 steps on BAIR, and 500 on Kinetics-600 and UCF-101.
As mentioned in Section we use the code basc from Nichol Dhariwal (2021) (MIT license) and their proposed architecture, except. that we only the spatial dimensions down to x 8 in three steps. We use 128 channels for the first block and increase it by 128 for each downsampling step. As mentioned in Section 4 use multi-head self-attention at the resolutions 16 and 8, cach with 4 attention heads. For sampling, we found it to be more beneficial to sample from the exponential moving average (EMA) of the weights (Nichol & Dhariwal 2021 We set the EMA rate to 0.9999.
3 x 3 convolution kernels. In the encoder, we downsample
B Additional results
B.1 Results on BAIR
Fig. shows that all of the models are also able to do unconditional video generation (even RaMViD (pu 0), we assume that this is due to the low diversity of the dataset). Qualitatively, we can see in Fig. S that videos generated by models with higher moving robot arm, only the models with pe 2 0.25 can properly generate the different objects in the box. However, we have no quantitative results on unconditional generation on BAIR
are better in generating details. While all models can generate the
B.2 Results on Kinetics-600
Kinetics-600 in practice appears be the most difficult dataset among those considered here. While our results are state-of thr- (see Table B), we do observe failure cases. One of the most common failure cases is fast, movement. In that case often deformation of the moving object (see Fig. 9
16

---

Published in Transactions on Machine Learning Research (11/2022)
x
x
X
X
Xg
Figure 9: Prediction of 11 frames given the first 5 frames Kinetics-600 when conditioning fast-moving objects. We can see that the background does get preserved well, while the object itself gets unrealistically deformed.
X
X
X
X
X
X
X
Figure Infilling on Kinetics. The conditioned frames are = (0. 1, 14, 15). The people frames and 1 are placed quite differently than in 14 and 15. The model is not able to generate the necessary camera movement and does simply interpolate between the frames.
Fast camera. movement can also be a problem for infilling. If an object is placed very different between the first and last frames, the model does not generate a harmonized movement but makes the object disappear in the first and appear in the last frames Fig. 10
C Qualitative comparison
The main qualitative improvement RaMViD compared to other methods is the decrease in occurence of deformed objects in the predictions on Kinetics-600 (see Fig. 15 While the predictions of several other methods suffer from object deformations, RaMViD only sulfers from this with fast-moving objects. On BAIR, on the other hand, our qualitative results are similar the ones of recent methods, although RaMViD can predict details in the interactions in more detail (see Fig. 14..
D Compute
Each model trained on 8 NVIDIA A100 GPUs with 40 GB of memory. The models on BAIR are trained with batch size of 32 and micro-batch size of 16 for 250k iterations (-3 days). All other models on Kinetics-600 and UCF-101 are trained with batch size of 64 and micro-batch size of 16. The models are trained for 500k iterations on Kinetics-600 (-10 days) and for 450k iterations on UCF-101 (-9 days).
E Datasets
The videos in all datasets have more frames than we train on. Therefore choose random sub-sequences of the desired length during training.
BAIR robot pushing. The BAIR robot pushing dataset can be used under an MIT license. We use the low resolution dataset (64 64). Since the data is already in the correct size, no prepossessing is necessary. For evaluation we predict one sequence for each of the 256 test videos and compare the FVD to the ground
17

---

Published in Transactions on Machine Learning Research (11/2022)
XD
X
X
X
X
X
X
Figure 11: Video infilling (C {0. 1, 14,, 15)) on Kinetica-600 with RaMViD (Po = 0.25).
x
x
x
Figure 12: Video completion (C (0,5, 10, 15)) on Kinetics-600 with RaMViD (po 0.25).
18

---

Published in Transactions on Machine Learning Research (11/2022)
21
20D0 J80N B009
x
x
x
10
x
12
14
Figure 13: Qualitative results of unconditional generation on UCF-101. Scenes with less movement are generated well, but are often close to the training set.
truth. To get a proper evaluation score, we do this 100 times and the final FVD score is the average over all 100 runs. We train on sequence length of 20.
Kinetics-600. The Kinetics-600 dataset has a Creative Commons Altribution 4.0 International License. The videos have different resolutions, which is why we reshape and center crop them to a 64 x 64 resolution. For evaluation we take 50,000 vidcos from the test set and predict a sequence for cach of the vidcos. We then compute the statistics for the ground truth and the predicted videos to obtain the FVD score. We train on a sequence length of 16.
UCF-101. We could not find license for the UCF-101 dataset. The original frames have resolution of 160 x 120, therefore we resize and center crop the videos to a 64 x 64 resolution. We train on the entire dataset of 13,320 videos. To evaluate the generative performance, we sample 10000 videos unconditionally and compute the Incoption Score (IS) This is repeated three times.
F Sampling Speed
Since sampling 10,000 videos takes about hours with the large model trained on Kinetics-600 and UCF-1011, and 7 hours with the smaller model trained BAIR using 500 sampling steps, it is crucial to know how many sampling steps are necessary to achieve satisfactory performance. We found that by using only 250 sampling steps for RaMViD (p = 0.25) trained on BAIR, our results drop significantly (to 101.54). However, when using 500 sampling steps, we achieve an FVD of 85.07 which is very similar to using 750 steps (84.20). We observe the same behaviour on Kinetics-600, where we achieve an FVD of 14 / 16 with 750 / 500 sampling steps but only an FVD of 49 with 250 steps. Therefore, when using the DDPM sampler with RaMViD, we recommend using a minimum of 500 sampling steps for video generation.
19

---

Published in Transactions on Machine Learning Research (11/2022)
x9
x²
x
x
10
X
X
Figure 14: A comparison between RaMViD (pu = 0.25, K = 8) and VPT Van et al. 2021 The first row shows our predictions and the second VideoGPT respectively. VideoGP T produces artefacts, especially when there are interacting objects.
V Visually, the difference is not aleniificantt
but we found that differente S
e
20

---

Published in Transactions on Machine Learning Research (11/2022)
10 X
X
X
X
X
Figure 15: Predicted videos on Kinetics-600 from RaMViD (pu = 0.25) the top row, and CCVS the bottom row. We found that model is better in modelling motion, whereas for static videos, the difference is visually not significant. Nevertheless, we also found that both models perform equally bad for fast object/camen movement.
21

---

Published in Transactions on Machine Learning Research (11/2022)
G Concurrent work
As mentioned in Section three concurrent works on diffusion models for vidcos were recently made public. Only Ho et al. (2022] and Voleti et al. (2022) consider similar tasks as we do. Ho et al. (022) appears to outperform RaMViD on unconditional video generation on UCF-101, which is not surprising, as we train with the mixed method and therefore the models are mostly trained for conditional generation. VVleti et al. (2022) evaluate their method on BAIR with the same procedure we used, and the results reported in their publication suggest that RaMViD outperforms their proposed method.
22


Reviews:
[{"review_id": "1S4gBKM1RF", "reviewer": "Reviewer_XQ2a", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- One of the earlier (concurrent) works that applies diffusion model to video prediction and infilling.\n- The empirical results are solid and outperform traditional methods. \n\nWeaknesses:\n- The technical novelty is a bit limited, as the approach is proposed in the Palette paper in inpainting. Despite what is claimed in the paper here, the inpainting method proposed by Palette also do not increase the dimensions of the model input, where random Gaussian noise are added to masked regions as well for inputs. \n- There is less discussion about what sampler is used in these tasks, which is very critical for time-sensitive applications. While it is acceptable to be slower than GANs, it is still helpful to discuss the time spent in these tasks and perform a small ablation study on top of it. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a diffusion model for video prediction, where both an \"unconditional model\" and a conditional model are trained jointly. The method achieves better performance on video prediction compared to some baselines in terms of FVD and IS.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Change the claims regarding Palette, which is not correct regarding inpainting.\n- Add some discussion about the speed/latency of the predictions, the sampler being used, and an ablation study about the trade-off between prediction quality versus speed.", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7TteE7z4R6", "reviewer": "Reviewer_LFdv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n\n1. The experiment results of the proposed approach is impressive. It outperforms existing baseline models by a significant margin on various tasks including on standard benchmarks datasets, including. The work also did extensive study on different settings and hyperparameters, including the conditioning frame numbers and propability of doing unconditional training.\n\n2. The use of diffusion model for video infilling and prediction is an interesting application of diffusion model. It is of great importance and relevance to both generative modeling and video prediction communities.\n\n3. The work is easy to follow and enjoyable to read. RamViD is well presented with the help of pseudocode.\n\n4. It should also be appreciated that the work also discusses the RaMViD's challenges when handling fast moving objects on Kinetics-600 and shows unconditional generation results on UCF-101 where RaMViD underperform existing models. \n\nWeakness:\n\n1. Despite impressive quantitaive experiment results, there're no comparison of qualitative sampels between RaMViD and other approaches. Comparisons on qualitative samples could help better demonstrates the model's strengths and weakness and provide more insights into the model.\n\n2. There are several presentation issues. Please see requested changes for more details.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed an approach for video infilling, prediction, and generation, RaMViD, based on 3D convolutional neural networks and diffusion generative models. Given observed conditional frames, 3D convolutional model learns to approximate the score function of the unknown frames. During training, the observed conditional frames are randomly sampled from each training sequence and the model is randomly switched between being trained to approximate conditional score functions and unconditional score functions with a certain probability. The work evalutes the model on various tasks including video prediction and infilling and unconditional generation using standard benchmarks and datasets, including BAIR, Kinetics-600 and UCF-101. The proposed approach demonstrates state-of-the-arts results on various tasks. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. The work need to include comparison of qualitative samples between RaMViD and baseline approaches as well as detailed discussion on comparison results.\n\n2. Equation 8 and Equation 9 can be confusing. In Equation 8, it is not clear what the second parenthesis in $s_\\theta (x_t, t)(x_t^U) $ means. Apart from the reversed diffusion process in Equation 9, the paper should also clearly presents the equation of the forward diffusion process (from data to noise) in the conditional setting.\n\n3. The numbers in Table 3 are not reported in the same format as the previous models, can the author explain why?\n\n4. Given the stochastic nature of diffusion models, the paper should report mean and standard deviations of results of multiple runs.\n", "broader_impact_concerns": "The broader impacts of the work has been fully discussed. The reviewer has no concern on this part.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jff1kcUNZu", "reviewer": "Reviewer_cFap", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe proposed model is conceptually simple -- the model itself is a modification of [1] to use 3D convolutional kernels instead of 2D kernels.\n- The paper introduces a simple and careful masking strategy to enable ease of training on unconditional and conditional examples. This enables the proposed model to achieve SOTA on several video generation benchmarks by training with modest resources.\n- The paper is generally well written and easy to follow.\n\n\n# Weaknesses\n\nThe evaluation and analysis of the model is fairly insufficient, with few ablative studies. There are several aspects that could be significantly improved:\n- It would be useful to include other diffusion models into the evaluation tables. In particular, all of the other baseline models in Tables 1, 3, and 5 are not diffusion models, and including some of the diffusion models (from the “concurrent work” paragraph of section 2) would be useful for an apples-to-apples comparison. Even if the results are considered concurrent for the purposes of SOTA, it would still be useful to provide the reader with a more comprehensive view of the diffusion modeling landscape.\n- It would be useful to include some measure of computational efficiency (in the form of FLOPS, inference throughput, training throughput) to compare the proposed model against baselines. It would be especially useful to include an efficiency comparison between diffusion based models and autoregressive models.\n- Related to the above point, it would also be useful to include the parameter count of each model being compared. In some prior papers (e.g., [2]), it appears that generative diffusion models achieve SOTA with a much lower parameter count compared to traditional GAN or autoregressive based approaches. Do we observe a similar trend here, for video generation?\n- The paper does not contain many ablations. It would be useful to conduct a study on the effect of the masking procedure against results (the only ablation on this is on the probability of training on unconditional examples). For example, it might be worthwhile to study masking strategies that do not sample uniformly at random -- would masking “chunks” rather than individual frames be helpful? Or would masking more frames at the beginning of the video be helpful/harmful as compared to masking at random?\n\n\n# References\n[1] Nichol, Alexander Quinn, and Prafulla Dhariwal. \"Improved denoising diffusion probabilistic models.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Saharia, Chitwan, et al. \"Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.\" arXiv preprint arXiv:2205.11487 (2022).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the extension of diffusion models to the domain of video prediction, generation, and infilling. The proposed method works for both unconditional video generation, as well as conditional video generation (i.e., conditioned on a set of initial frames). In order to achieve this, the paper introduces a new way to condition diffusion models on video information, by masking out frames at random. This enables their model to be jointly trained on conditional and unconditional examples, and perform a variety of tasks. The proposed method achieves state-of-the-art on several video benchmarks as measured by automatic metrics on video quality (FVD and IS).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "In general, I believe that the results and modeling from this paper are valuable contributions for the video generation and generative modeling community. The proposed model is simple and effective, trained on modest resources to achieve SOTA. However, I think that the paper would be significantly improved if the evaluation procedure in the paper was more comprehensive and robust:\n\n- As mentioned in the “weaknesses section”: It would be useful to include measures of efficiency and capacity of each model. Does the proposed model improve upon baselines simply because it has greater capacity, or is it due to diffusion-based training?\n- More ablation analysis on the masking strategy (which appears to be one of the primary contributions of the paper) is essential. In particular, running ablation experiments over $K$ (number of conditioning frames), the random masking strategy (uniform vs. non-uniform), masking chunks rather than individual frames.", "broader_impact_concerns": "The primary concern of the paper is on adversarial and unethical deployment of generative models, e.g., for creating deepfakes. This is well addressed by Section 6 of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, the paper presents a simple diffusion-based video prediction method with strong performance demonstrated in multiple settings. The paper is written clearly and authors have modified the manuscript in response to reviewer concerns. This includes adding discussion on sampling speed, updating notation for clarity, adding qualitative examples, adding model size discussion, and further examination of concurrent work. The AE believes the claims in this work are well-supported and there exists an audience of TMLR readers who would be interested in the method, analysis, and provided codebase. 
Decision: Accept (poster)