Video Diffusion Models
Authors: Jonathan Ho
Tim Salimans
Alexey A. Gritsenko
William Chan
Mohammad Norouzi
David J. Fleet
Venue: NeurIPS 2022 Accept (2022)
Keywords: diffusion
score
video
generative
text-to-video

Abstract: Generating temporally coherent high fidelity video is an important milestone in generative modeling research. We make progress towards this milestone by proposing a diffusion model for video generation that shows very promising initial results. Our model is a natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find to reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos we introduce a new conditional sampling technique for spatial and temporal video extension that performs better than previously proposed methods. We present the first results on a large text-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video prediction and unconditional video generation. Supplementary material is available at https://video-diffusion.github.io/.

Video Diffusion Models
Jonathan Ho jonathanho@google.com com
Tim Salimans* sal mallims@@googeeccommm com com
Alexey Gritsenko
agrittsenk@google..com
William Chan willaanhann@oogle.comm
Mohammad Norouzi morooziigooglle com
David Fleet davidf daviidllett@ooggle.com
Abstract
Generating temporally coherent high fidelity video is important milestone in generative modeling research. We make progress towards this milestone by propos- ing diffusion model for video generation that shows very promising initial results. Our model is natural extension of the standard image diffusion architecture, and it enables jointly training from image and video data, which we find reduce the variance of minibatch gradients and speed up optimization. To generate long and higher resolution videos introduce spatial and temporal video extension that performs better than previously proposed methods. We present the first results on large lext-conditioned video generation task, as well as state-of-the-art results on established benchmarks for video predic- tion and unconditional video generation. Supplementary material is available https: //vido diffusion, g thub.io/.
conditional sampling technique for
1 Introduction
Diffusion models have recently been producing high quality results in image generation and audio generation [e.g. 28, 39, 40, 16, 23, 36, 48, 60, 42, 10, 29], and there is significant interest in validating diffusion models in new data modalities. In this work, we present first results on video generation using diffusion models, for both unconditional and conditional settings.
We show that high quality vidcos can be generated using essentially the standard formulation of the Gaussian diffusion model [46], with little modification other than straightforward architectural changes to accommodate video data within the memory constraints deep learning accelerators. We train models that generate fixed number of video frames using 3D U-Net diffusion model architecture, and we enable generating longer videos by applying this model autoregressively using new method for conditional generation. We additionally show the benefits of joint training on video and image modeling objectives. We test our methods on video prediction and unconditional video generation, where achieve state-of-the-arr sample quality scores, and we also show promising first results on text-conditioned video generation.
2 Background
A diffuusion model [46, 47, 22] specified in continuous time [53, 48, 10, 28] is a generative model with latents z {z, |t [0, 1]} obeying forward process q(z(x) starting at data x ~ p(x). The forward process is Gaussian process that satisfies the Markovian structure:
( x) = N (,,
g(2,(2,)= = (2,,; (oz/ (v, (0,//,,),,,0 ,2,0
(1)
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where 0 S < < - -1-e and at, os specify a differentiable noise schedule whose log Sggnalto--nois--aate x loglo /o] decreases with & until g(z1) I..
Training Learning to reverse the forward process for generation can be reduced to learning to denoise Z, ~ q(z,) x) into an estimate {,(Z,, A x for all (we will drop the dependence on X to simplify notation). We train this denoising model Xs using weighted mean squared error loss
Er, [wo(Ae] (z())) - x112]
over uniformly sampled times / E [0, 1]. This reduction of generation to denoising can justified as optimizing weighted variational lower bound on the data log likelihhod under the diffusion model, or as form of denoising score matching [56, 47, 22, 28]. In practice, we use the e e-prediction parameterization, defined (z( (z, ozEg Z, )) /a,, and train Eş using a mean squared error in e space with sampled according to a cosine schedule [37]. This corresponds to a particular weighting w (>x)) for learning scaled score estimate co co(Z)) -o Vz, log p(z)). where p(z,) is the
true density of z₁ under x ~ p(x) [22, 28, 48). We also train using the v-prediction parameterization for certain models [42].
Sampling We use a variety of diffusion model samplers in this work. One is the discrete time ancestral sampler [22] with sampling variances derived from lower and upper bounds reverse process entropy [46, 22, 37J. To define this sampler, first note that the forward process can be described in reverse as (2,,) 22, x) ((zs; Maje (Z4, x), 22 22 T) (noting
l), where
Ms / = cx h. (as/as)) )z + (1 A Ja,x and Ta (1 -cti-du (3)
Starting at ž ~ ~O, I). the ancestral sampler follows the rule
Z₃ P. (Ze, (z)))
v (6 (o?
(4)
where is standard Gaussian noise, is hyperparameter that controls the stochasticity of the sampler [37], and s, follow a uniformly spaced sequence from I to O.
Another sampler, which we found especially effective with our new method for conditional gener- ation (Section 3.1), is the predictor-corrector sampler [48]. Our version of this sampler alternates between the ancestral sampler step (4) and Langevin correction step of the form
2g f Z (o.e(())) + Voage'
(5)
where 8 is step size which we fix to 0.1 here, and e' another independent sample of standard Gaussian noise. The purpose of the Langevin step is to help the marginal distribution of each n, generated by the sampler to match the true marginal under the forward process starting at x ~ p(x).
In the conditional generation setting, the data x is equipped with a conditioning signal c, which may represent class label, text caption, or other type of conditioning. To train a diffusion model to fit p(x|c), the only modification that needs to be made is to provide c to the model as (0 c). Improvements to sample quality can be obtained in this setting by using classifier-free guidance [20]. This method samples using adjusted model predictions Eg, constructed via
(z c) = (1 + w) () c) wwe)))))
where vo is the guidance strength, 6n 60(2,, c) = of (z: (a c)) is the regular conditional model prediction, and (z; is prediction from an unconditional model jointly trained with the conditional model (if e consists of embedding vectors, unconditional modeling can be represented as c 0). For w. >> 0 this adjustment has the effect of eeehasiiasii the effect of conditioning on the signal which tends to produce samples of lower diversity but higher quality compared to sampling from the regular conditional model [20]. The method can be interpreted as way to guide the samples towards areas where an implicit classifier p<c/2,) has high likelihood, and is an adaptation of the explicit classifier guidance method proposed by [16].

---

Video diffusion models
Our approach to video generation using diffusion models is to use the standard diffusion model formalism described in Section 2 with a neural network architecture suitable for video data. Each of our models trained to jointly model a fixed number of frames at a fixed spatial resolution. To extend sampling to longer sequences of frames higher spatial resolutions, we will repurpose models with a conditioning technique described later in Section 3.1.
In prior work on image modeling, the standard architecture for &o in image diffusion model is U-Net [38, 44], which is a neural network architecture constructed as a spatial downsampling pass followed by spatial upsampling pass with skip connections to the downsampling pass activations. The network built from layers of 2D convolutional residual blocks, for example the style of the Wide ResNet [65], and each such convolutional block is followed by spatial attention block [55, 58, 11]. Conditioning information, such as с and A,, is provided to the network in the form of an embedding vector added into each residual block (we find it helpful for models to process these embedding vectors using several MLP layers before adding).
We propose to extend this image diffusion model architecture to video data, given by block of a fixed number of frames, using a particular type of 3D U-Net [13] that is factorized over space and time. First, we modify the image model architecture by changing each 2D convolution into space-only 3D convolution, for instance, we change each 3x3 convolution into a 1x3x3 convolution (the first axis indexes video frames, the second and third index the spatial height and width). The attention in each spatial attention block remains as attention over space; i.e., the first axis is treated as batch axis. Second, after each spatial attention block, we insert temporal attention block that performs attention over the first axis and treats the spatial axes as batch axes. We use relative position embeddings [45] in each temporal attention block that the network can distinguish ordering of frames in a way that does not require an absolute notion of video time. We visualize the model architecture in Fig. 1.
(2 M,
(2 Ms
(R,
N7.2%
Figure 1: The 3D U-Net architecture for X, in the diffusion model. Each block represents 4D tensor with axes labeled as frames height x width channels, processed in a space-time factorized manner as described in Section 3. The input is noisy video Ze, conditioning c, and the log SNR X The downssapplinguusaapping blocks adjust the spatial input resolution height width by factor of 2 through each of the K blocks. The channel counts are specified using channel multipliers Mi. M2. .... MR. and the upsampling pass has concatenation skip connections to the downsampling pass.
The use of factorized space- time attention is known be a good choice in video transformers for its computational efficiency [2, 5, 21]. An advantage of our factorized space-time architecture, which is unique to our video generation setting, is that is particularly straightforward mask the model to run on independent images rather than video, simply by removing the attention operation inside each time attention block and fixing the attention matrix to exactly match each key and query vector at each video timestep. The utility of doing so is that it allows us to jointly train the model on both video and image generation. We find our experiments that this joint training important for sample quality (Section 4).
3

---

3.1 Reconstruction-guider sampling for improved conditional generation
The videos we consider modeling typically consist of hundreds to thousands of frames, at a frame rate of at least 24 frames per second. To manage the computational requirements of training our models, we only train on small subset of say 16 frames at a time. However, at test time we can generate longer videos by extending our samples. For example, consisting of 16 frames, and then extend it with a second sample x² ~ py px(x""xe") If x consists of frames following x", this allows us to autoregressively extend our sampled videos to arbitrary lengths, which we demonstrate in Section 4.3.3. Alternatively, we could choose x" to represent a video of lower frame rate, and then define x" to be those frames in between the frames of x". This allows one to then to upsample a video temporally, similar to how [34] generate high resolution images through spatial upsampling.
could first generate a video x" ~ pe (x)
Both approaches require one to sample from a conditional model, (x* This conditional model could be trained explicitly, but it can also be derived approximately from our unconditional model po(x) by imputation, which has the advantage of not requiring a separately trained model. For example, [48] present general method for conditional sampling from jointly trained diffusion model (x = xx):: In their approach to sampling from (9(x"(x') the sampling procedure for updating 2 is unchanged from the standard method for sampling from po (z,|zz), with z, [2]. z!] but the samples for z are replaced by exact samples from the forward process, (<), at each iteration. The samples z then have the correct marginal distribution by construction, and the samples zi will conform with z through their effect on the denoising model & z,1). Similarly, we could sample z from q(2%) Xx" zi), which follows the correct conditional distribution in addition to the correct marginal. We will refer to both of these approaches as the replacement method for conditional sampling from diffusion models.
When we tried the replacement method to conditional sampling, we found it to not work well for our video models: Although samples x" looked good in isolation, they were often not coherent with x". This is caused by a fundamental problem with this replacement sampling method. That is, the latents z!: updated in the direction provided by xz(z) Eg [x])) while what is get
needed instead is [,lx""|z,,x") [,[x"|z,, x=] E,,(x|zz,] |z.] + (of (f.)))), V log (x*]2)), where the second term is missing in the replacement method. Assuming perfect denoising model, plugging in this missing term would make conditional sampling exact. Since q(x" (2,) is not available closed form, however, we instead propose to approximate it using& Gaussian of the form q(x" (z,) s (z)) (o? /a II). where & (2.) is a reconstruction of the conditioning data x" provided by our denoising model. Assuming a perfect model, this approximation becomes exact as t - O, and empirically we find it to be good for larger i also. Plugging in the approximation, and adding a weighting factor Wy, our proposed method to conditional sampling is variant of the replacement method with adjusted denoising model, x/. defined by
x"). Writing this in terms of the score of the data distribution,
x²(Z)) = xx((z) W,
V V221ix"
x3 (Z,)
2
The additional gradient term in this expression be interpreted as form of guidance [16, 20] based on the model's 's reconstruction of the conditioning data, and we therefore refer to this method as reconstruction-guiine of guidance, we find that choosing a larger weighting factor, W,y > 1, tends to improve sample quality. We empirically investigate reconstruction guidance in Section 4.3.3, where we find it to work surprisingly well, especially when combined with predictor-corrcctor samplers using Langevin diffusion [48].
sampling,
simply reconstruction guidance. Like with other forms
Reconstruction guidance also extends to the case of spatial interpolation (or super-resolution), in which the mean squared error loss is imposed on a downsampled version of the model prediction, and backpropagation is performed through this downsampling. In this setting, we have low resolution ground truth videos x" (c.g. at the 64x64 spatial resolution), which may be generated from a low resolution model, and we wish to upsample them into high resolution videos (e.g. at the 128x 128 spatial resolution) using an unconditional high resolution diffusion model &y. To accomplish this, we adjust the high resolution model as follows:
X (Z)) = xx()))
W1
Vz. 1|x -x((z)) 1|
(8)
2
4

---

where (z() is our model's reconstruction of the low low-resouttion video from which is obtained by downsampling the high resolution output of the model using differentiable downsampling algorithm such as bilinear interpolation. Note that it is also possible to simultaneously condition on low resolution videos while autoregressively extending samples at the high resolution using the same reconstruction guidance method. In Fig. 2, we show samples of this approach for extending 16x64x64 low resolution samples at frameskip to 64x 12xx8x128 samples at frameskip using a 9x 9x128x diffusion model.
4 Experiments
We report results video diffusion models for unconditional video generation (Section 4.1), conditional video generation (video prediction) (Section 4.2), and text-conditioned video genera- tion (Section 4.3). We evaluate our models using standard metries such FVD [54], FID [19], and IS [43]; details on evaluation are provided below alongside each benchmark. Samples and additional results are provided at https: /evvvvivoddddd cioo-difffiioooo gi thub. io/. Architecture hyperparameters, training details, and compute resources are listed in Appendix A.
4.1 Unconditional video modeling
To demonstrate our approach unconditional generation, we use popular benchmark of Soomro et al. [49] for unconditional modeling of video. The benchmark consists of short clips of people performing one of 101 activities, and originally collected for the purpose of training action recognition models. We model short segments of 16 frames from this dataset, downsampled to spatial resolution of 64x64. In Table I we present perceptual quality scores for videos generated by our model, and we compare against methods from the literature, finding that our method strongly improves upon the previous state-of-the-arr.
We use the data loader provided by TensorFlow Datasets [1] without further processing, and we train on all 13,320 videos. Similar to previous methods, FID and IS, using 10,000 samples generated from our model. C3D internally resizes input data to the 112x112 spatial resolution, so perceptual scores are approximately comparable even when the data is sampled at different resolution originally. As discussed by [64], methods in the literature unfortunately not always consistent in the data preprocessing that is used, which may lead to small differences in reported scores between papers. The Inception Score we calculate for real data (<s 60) is consistent with that reported by [26], who also report a higher real data Inception score of 22 90 for data sampled at the 128x128 resolution, which indicates that our 64x64 model might be at disadvantage compared to works that generate higher resolution. Nevertheless, our model obtains the best perceptual quality metrics that we could find in the literature.
the C3D network [51]² for calculating
Method
Resolution
FID.
IS
MoCoGAN [52]
16x64x64
26998 33
12.42
TGAN-F [26]
16x64x64
8942.63 3.72
13.62
TGAN-ODE [18]
16x64x64
26512 -
15.2
TGAN-F [26]
16x 128
7817 + 10
22.91 + .19
VideoGPT [62]
16x 128x 28
24.69 + 0.30
TGAN-v2 [41]
16x64x64
3431 + 19
26.60 1 0.47
TGAN-v2 [41]
16x128x
3497 + 26
28.87 0.47
DVD-GAN [14]
16xx18x128
32.97 L 1.7
Video Diffusion (ours)
16x64x64
295
1 0.62
real data
16x64x64
60.2
Table 1: Unconditional video modeling results on UCF101.
We use the C3D model implemented at github, com/ http://netttteeee weweeettreeearcctggan [41].
5

---

4.2 Video prediction
common benchmark task for evaluating generative models of video video prediction, where the model is given the first frame(s) of a video and is asked to generate the remainder. Models that do well on this conditional generation task are usually trained explicitly for this conditional setting, for example by being autoregressive across frames. Although our models are instead only trained unconditionally, we can adapt them to the video prediction setting by using the guidance method proposed in section 3.1. Here we evaluate this method on two popular video prediction benchmarks, obtaining state-of-the-art results.
BAIR Robot Pushing We evaluate video prediction performance on BAIR Robot Pushing [17]. a standard benchmark in the video literature consisting of approximately 44000 videos of robot pushing motions at the 64x64 spatial resolution. Methods for this benchmark are conditioned on 1 frame and generate the next 15. Results are listed in Table 2. Following the evaluation protocol of [4] and others. we calculate FVD [54] using the I3D network [8] by comparing 100 x 256 model samples against the 256 examples in the evaluation set.
Kinetics-600 We additionally evaluate video prediction performance on the Kinetics-600 bench- mark [27, 9]. Kinetics-600 contains approximately 400 thousand training videos depicting 600 different activities. We train unconditional models on this dataset at the 64 x 64 resolution and evalu- 50 thousand randomly sampled videos from the test set, where we condition on randomly sampled subsequence of 5 frames and generate the next 11 frames. Like previous works, we calculate FVD and Inception Score using the I3D network [8]. See Table for results. In reported results we sample test videos without replacement, and we use the same randomly selected subsequences for generating model samples and for defining the ground truth, since this results in the lowest bias and variance in the reported FVD metric. However, from personal communication we learned that [33, 14] instead sampled with replacement, and used a different random seed when sampling the ground truth data. We find that this way of evaluating raises the FVD obtained by our model slightly, from 16.2 to 16.9. Inception Score is unaffected.
Table 2: Video prediction BAIR Robot Pushing. FVD]
Table 3: Video prediction on Kinetics-600. Method
Method
FVD4 ISt
DVD-GAN [14]
109.8
Video Transformer [59]
170 + 5
VidcoGPT [62]
103.3
DVD-GAN-FP [14]
69.1 + 0.78
THVD-GAN-FF [33]
103.3
Video VQ-VAE [57]
64.3 + 2.04
Transframer [35]
100
CCVS [31]
55 + 1
CCVS [31]
99
TrIVD-GAN-FF GAN-FP [33]
25.74 1 0.66
12.54
Video'Tranformer [59]
94
Transframer [35]
25,4
FitVid [4]
93.6
NUWA [61]
86.9
Video Diffuusion (ours)
ancestral, 256 steps
18.6
15.39
Video Diffusion (ours)
Langevin, 128 steps
16.2 + 0.34
15.64
ancestral sampler, 512 steps
68.19
Langevin sampler, 256 steps
66.92
4.3 Text-conditioned video generation
The remaining experiments reported on text-conditioned video generation. In this text-conditioned video generation setting, we employ a dataset of 10 million captioned videos, and we condition the diffusion model on captions in the form of BERT-large embeddings [15] processed using attention pooling. We consider two model sizes: a small model for the joint training ablation, and a large model for generating the remaining results (both architectures are described in detail in Appendix A), and we explore the effects of joint video-imag training, classifier free guidance, and our newly proposed reconstruction guidance method for autoregressive extension and simultaneous spatial and temporal super-resolution. We report the following metrics in this section on 4096 samples: the video metric FVD, and the Inception-based image metrics FID and measured by averaging activations across frames (FID/IS-avg) and by measuring the first frame only (FID/IS-first. For FID and FVD, we report two numbers which are measured against the training and validation sets, respectively. For IS, we report two numbers which are averaged scores across split and 10 splits of samples, respectively.

---

Eltmmnate Chficrms Thanel Might Dunny Seowstore
cariuun on cocowey bett
Figure 2: Text-conditioned video samples from a cascade of models. First samples are generated from a 16x64x64 frameskip 4 model. Then those samples are treated as ground truth for simultancous super-resolution and autoregressive extension 62x 128 using 9x128x 128 frameskip model. Both models are conditioned on the text prompt. In this figure, the text prompt, low resolution frames, and high resolution frames are visualized in sequence. See Fig. for more samples.
4.3.1 Joint training video and image modeling
As described in Section 3, of the main advantages of our video architecture is that allows us to easily train the model jointly on video and image generative modeling objectives. To implement this joint training, we concatenate random independent image frames the end of each video sampled from the dataset, and we mask the attention in the temporal attention blocks to prevent mixing information across video frames and each individual image frame. We choose these random independent images from random videos within the same dataset; in future work we plan to explore the effect of choosing images from other larger image-only datasets.
Table 4 reports results for an experiment on text-conditioned 16x64x64 videos, where we consider training additional 0, 4, independent image frames per video. One ments video and image sample quality metrics as more independent image frames are added. Adding independent image frames has the effect of reducing variance of the gradient the expense of some bias for the video modeling objective, and thus it can be seen as a memory optimization to fit more independent examples batch.
clear improve-
Table 4: Improved sample quality due to image-video joint training on text-to-video generation.
Image frames
FVDJ,
FID-avg]
IS-avgī
FID-first..
IS-first
0
202.28/205.42
37.52/37.40
7.91/7.58
41.14/40.87
9.23/8.74
4
68. 1/70..4
18.62/18.42
9.02/8.53
22.54/22.19
10.58/9.91
8 57.84/60.72
15.57/15.44
9.32/8.82
19.25/18.98
10.81/10.12
4.3.2 Efffet classifier-free guidance
Table 5 reports results that verify the effectiveness of classifier-free guidance [20] on text-to-video generation. As expected, there is clear improvement in the Inception Score-like metrics with higher guidance weight, while the FID-like metrics improve and then degrade with increasing guidance weight. Similar findings have been reported on text-to-image generation [36].
7

---

Figure shows the effect of classifier-free guidance [20] on a text-conditioned video model. Similar what was observed in other work that used classifier-free guidance on coxciionit image generation [36] and class-conditioned image generation [20, 16], adding guidance increases the sample fidelity of each individual image and emphases the effect of the conditioning signal.
Figure 3: Example frames from random selection of videos generated by 16x64x64 text- conditioned model. Left: unguided samples, right: guided samples using classifier-fre guidance.
Table 5: Effect of classifier-free guidance text-to-video generation (large models). Sample quality reported for 16x64x64 models trained on frameskip and 4 data. The model was jointly trained on 8 independent image frames per 16-frame video.
Frameskip Guidance weight
FVDJ
FID-avg.
18-avet
FID-first.
IS-firstf
1.0
11.65/43.70
12.49/12.89 39
10.80/10.07
16.42/16.19 19
12. 1.22
50.19/48.79
10,53/10.47
13:22/12.10
13.91/13.75
14.11/13.46
5.0
163.74/160.21
13.34/13.52
14,80/13.46
17.07/16.95
16.40/14.755
56.71/60.30
11.03/10.93
9.40/8.90
16.21/15.96
11.39/10.61
2.0
54,28/51 95
9,39/9.36
11.53/10.75
14.21/14.04
13.81/12.63
185.89/176.82
11.82/11.78
13.73/12.59
16.24/4.62
4.3.3 Autoregressive video extension for longer sequences
In Section 3.1 we proposed the reconstruction guidance method for conditional sampling from diffusion models, an improvement over the replacement method of [48]. In Table 6 we present results on generating longer videos using both techniques, and find that our proposed method indeed improves over the replacement method in terms of perceptual quality scores.
Figure 4 shows the samples of our reconstruction guidance method for conditional sampling compared to the replacement method (Section 3.1) for the purposes of generating long samples in a block- autoregressive manner (Section 4.3.3). The samples from the replacement method clearly show lack of temporal coherence, since frames from different blocks throughout the generated videos appear to uncorrelated samples (conditioned on c). The samples from the reconstruction guidance method, by contrast, are clearly temporally coherent over the course of the entire autoregressive generation process. Figure additionally shows samples of using the reconstruction guidance method to simultaneously condition on low frequency, low resolution videos while autoregressively extending temporally at a high resolution.
Table 6: Generating 64x64x64 videos using autoregressive extension of 16x64x64 models. FiD-avg]
Guidance weight Conditioning method
FVDJ
IS-avgt
FID-first.
fi-stttt
reconstruction guidance
136.22/134.55 451.45/436.1 16
13.77/13.62
10.309.66
16.34/16.46
14.67/13.37
replacement
25.95/25.52
7.0066.75
16.33/16.46
14.67/1 3.34
5.0
reconstruction guidance
133.92/133.04
13.59/13.58
10.31/9.65
16.28/16.53
15.09/13.72
replacement
456.24/441.99
26.05/25.69
7.04/6.78
16.30/16.54
11/1
8

---

Figure 4: Comparing the replacement method (left) vs the reconstruction guidance method (right) for conditioning for block auttreerrreeeei generation of 64 frames from a 16 frame model. Video frames displayed over time from left to right; each row is an independent sample. The replacement method suffers from a lack of temporal coherence, unlike the reconstruction guidance method.
5
Related work
Prior work video generation has usually employed other types of generative models, notably,
autoregressive models, VAES, GANs, and normalizing flows [e.g. 3, 4, 32, 30, 14, 59, 62, 57]. Related work on model classes similar to diffusion models includes [25. 24]. Concurrent work [63] proposes diffusion- predict each individual frame within a RNN temporal autoregressive model. Our video diffusion model, by contrast, jointly models entire videos (blocks of frames) using 3D video architecture with interleaved spatial and temporal attention, and we extend to long sequence lengths by filling in frames or autoregressive temporal extension.
approach to video generation that uses an image diffusion model to
6 Conclusion
We have introduced diffusion models for video modeling, thus bringing recent advances in generative modeling using diffusion models to the video domain. We have shown that with straightforward extensions of conventional U-Net architectures for 2D image modeling 3D space-time, with factorized space-time attention blocks, one can learn effective generative models for video data using the standard formulation of the diffusion model. This includes unconditional models, text-conditioned models, and video prediction models.
We have additionally demonstrated the benefits of joint image-video training and classifier-fres guidance for video diffusion models on both video and image sample quality metrics, and we also introduced new reconstruction-guide. conditional sampling method that outperforms existing replacement or imputation methods for conditional sampling from unconditionally trained models. Our reconstruction guidance method can generate long sequences using either frame interpolation (or temporal super-resolution) or extrapolation in an auto-regressive fashion, and also can perform spatial super-resolution. We look forward to investigating this method wider variety of conditioning settings.
Our goal with this work is to advance research methods in generative modeling, and our methods have the potential to positively impact creative downstream applications. As with prior work in generative modeling, however, our methods have the potential for causing harmful impact and could enhance malicious or unethical uses of generative models, such as fake content generation, harassment, and misinformation spread, and thus we have decided not to release our models. Like all generative models, our models reflect the biases of their training datasets and thus may require curation ensure fair results from sampling. In particular, our text-to-video models inherit the challenges faced by prior work on text-to cott-to-image models, and our future work will involve auditing for forms of social bias, similar to [6, 7, 50, 12] for image-to-text and image labeling models. We see our work as only a starting point for further investigation on video diffusion models and investigation into their societal implications, and we will aim to explore benchmark evaluations for social and cultrall bias in the video generation setting and make the necessary research advances to address them.

---

References
[1] TensorFlow Datasets, a collection ready-to-use datasets. atttps///ww.i www.sooffoweeeeoroooolloooooo org datasets, 2022.
[2] Anurag Arnb, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid, ViVIT: video vision transformer. Proceedings of the IEEE/CVE International Conference Computer Vision, pages 6836-6846, 2021.
[3] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint urXiv: 1252, 2017.
[4] Mohammad Babaeizadch, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. FitVid: Overfitting in pixel-level video prediction. arXiv preprint arXivv2106.. 13195, 2021.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding. arXiv preprint PrXiv:2102.050900 2(3):4, 2021.
[6] Joy Buolamwini and Timnit Gebru. Gender shades: commercial gender classification. In Conference on Fairness, Accountability and Transparency, FAT 2018, 23-24 February 2018, New York, NY, USA. Proceedings of Machine Learning Research, PMLR, 2018
Intersectional accuracy disparities in
[7] Kaylee Burns, Lisa Hendricks, Trevor Darrell, and Anna Rohrbach. Women also snowboare Overcoming bias in captioning models. In European Conference on Computer Vision (ECCV), 2018.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[9] Joao Carreira, Eric Noland, Andras Banki-Horyath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint 180.vv:808..1111 2018.
[10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. WaveGrad: Estimating gradients for waveform generation. International Conference on Learn- ing Representations, 2021.
[11] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An improved autoregressive generative model. In International Conference on Machine Learning, pages 863-871, 2018.
[12] Jaemin Cho, Abhay Zala, and Mohit Bansal. Dall-eval: Probing the reasoning skills and social biases of text-to-image generative transformers. arxiv:2202. 04055, 2022.
[13] Özgün Çişek, Ahmed Abdulkadir, Soeren Lienkamp. Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention, pages 424-432. Springer, 2016.
[14] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint
2019.
[15] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 4171 4171-4186. Association for Computational Linguistics, 2019.
pre-training
[16] Prafulla Dhariwal and Alexander Nichol. Diffusson models beat GANs image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[17] Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, pages 344-356, 2017.
10

---

[18] Cade Gordon and Natalie Parde. Latent neural differential equations for video generation. In NeurIPS 2020 Workshop on Pre-registration in Machine Learning, pages 73-86. PMLR, 2021.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626 6637,6667 2017.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[21] Jonathan Ho. Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv:1912.1 1200, 2019.
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840-6851, 2020.
[23] Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. arXiv preprint arXiv:2106.15282 2021.
[24] Zahra Kadkhodaie and Eero Simoncelli. Stochastic solutions for linear inverse problems using the prior implicit in a denoiser. Advances Neural Information Processing Systems, 34, 2021.
[25] Zahra Kadkhodaie and Eero P Simoncelli. Solving linear inverse problems using the prior implicit in denoiser. arXiv preprint a3X1v:2007.13640, 2020.
[26] Emmanuel Kahembwe and Subramanian Ramamoorthy. Lower dimensional kernels for video discriminators. Neural Networks, 132:506-522 2020.
[27] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 1705. 2017.
[28] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. arXiv preprint 2rXiv:2107.006633 2021.
[29] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representutions, ICLR, 2021.
[30] Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. VideoFlow: flow-based generative model for video. arXiv preprint 1rXiv:1903.01434. 2019.
[31] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. Advances Neural Information Processing Systems, 34, 2021.
[32] Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv: 1804.0153,,,, 2018.
[33] Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation-batt based adversarial video prediction on large-scale data. arXiv preprinl arXiv:2003.04035, 2020.
[34] Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimensional upscaling. In International Conference on Learning Represen- lations, 2019.
[35] Charlie Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint arXiv:2203.094 094944 2022.
[36] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing 2021.
with text-guided diffusion models. arXiv preprint
11

---

[37] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference Machine Learning, ICML, 2021.
[38] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234-241. Springer, 2015.
[39] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi. Palette: Image-to-imagee diffusion models. arXiv preprint 11X.::2111.05866, 2021.
[40] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint wxXvv2214.07636. 2021.
[41] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficien unsupervised training of high-resolution temporal GAN. Interna- lional Journal of Computer Vision, 128( 20):2586666066, 2020.
[42] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representutions, 2021.
[43] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pages 2234 2242, 2016.
[44] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
[45] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv:1803.022 02.155, 2018.
[46] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265, 2015.
[47] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pages 11895-11907, 2019.
[48] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar. Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations, 2021.
[49] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. A dataset of 101 human actions classes from videos in the wild. SOCV-TR-12-01, 2012.
[50] Ryan Steed and Aylin Caliskan. Image representations learned with unsupervised pre-training contain human-like biases. In Proceedings of the 2021 ACM Conference on Fairness, Account- ability, and Transparency, FAccT '21, page 701-713, Association for Computing Machinery, 2021.
[51] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 4489 4497, 2015.
[52] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018
[53] Belinda Tzen and Maxim Raginsky. Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. arXiv preprint weXXv:1905.090000 2019.
12

---

[54] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprini arXiv: 18/2.01777 2018.
[55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, pages 5998 6008, 2017.
[56] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):166111644 2011.
[57] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
[58] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794-7803, 2018.
[59] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In International Conference on Learning Representutions, 2019.
[60] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. arXiv preprint 1rXiv:2112.02475 2021.
[61] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NÜWA: Visual synthesis pre-traine pre-training for neural visual world creation. arXiv preprint arXiv:2111.12417, 2021.
[62] Wilson Yan, Yunzhi Zhang, Picter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10133 2021.
[63] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203.0945: 2022.
[64] Vladyslav Yushchenko, Nikita Araslanov, and Stefan Roth. Markov decision process for video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[65] Sergey Zagoruyko and Nikos Komodakis. 2tfb:1665.07744.2011. 2016.
Wide residual networks.
arXiv preprint
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A] Licenses of existing assets can be found via the citations we provided.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "s8_mHAawBes", "reviewer": "Reviewer_xwAd", "summary": "This paper adapts diffusion models for image generation to generate temporally coherent videos. The adaptation were done by (1) using a 3D U-Net diffusion model architecture to generate a fixed number of video frames; (2) using reconstruction guidance to generate temporally coherent longer videos. This paper claims that the proposed method generates SOTA results on benchmarks for unconditioned video generation and video prediction. The paper also claims that joint training using both images and videos improves the generated video quality. ", "questions": "Given the training hardware used, how long does training take on these tasks? ", "limitations": "Yes.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: \n- The approach is a natural retention from diffusion models for image generation.\n- Results are great both in terms of visual quality and numerical results.\n\nWeaknesses:\n- The claim of joint training using images and videos is better supported with more experiments:\n1. As mentioned in the paper, images are better sampled outside of the video dataset.\n2. In the unconditional video generation task, do joint training help? \n- Video frame interpolation related literatures are better discussed in the sections of the video extension. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lcVnNZPoZqA", "reviewer": "Reviewer_hVqw", "summary": "The paper extends the image-based diffusion model methods into the video domain. It builds upon the 3D U-Net architecture with attentions along the the decoupled space and time dimensions. The proposed video diffusion model also enables text-conditioned video generation. They outperform state-of-the-art methods in different benchmarks for video prediction and unconditional video generation. They also show the joint image-video training and classifier-free guidance for video diffusion models. The proposed diffusion model can also perform on the long sequence via auto-regressive inference.\n", "questions": "* Is the space-time factorization necessary? What would be the performance vs efficiency of using 3D convolution based U-Net compared to the proposed factorized one?\n* How much performance drop will happen without the temporal attention, ie, per-frame method?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The proposed 3D U-Net diffusion model achieves state-of-the-art performances on unconditional video modeling on UCF101, video prediction on BAIR Robot Pushing and Kinetics-600, and text-based video generation tasks.\n2. The proposed joint image and video training is effective.\n3. The paper starts with the background section which gives a good guidance to readers.\n----\n\nWeaknesses\n1. How does the proposed method work on 128x128 resolution results, compared to those of 64x64?\n2. In figure 1, where is the num-frames dimension denoted?\n3. Is the space-time factorization necessary? What would be the performance vs efficiency of using 3D convolution based U-Net compared to the proposed factorized one?\n4. How much performance drop will happen without the temporal attention, ie, per-frame method?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kTRdJcegaEt", "reviewer": "Reviewer_m1dT", "summary": "This paper shows that diffusion models also perform well for video modeling. By choosing a factorized attention module, it can directly extend a 2D U-Net to 3D space-time, allowing joint training with images possible. Results on unconditional text-to-video generation and video prediction show the strong performance of the proposed model. \n\n ", "questions": "1. The major technical contribution seems to come from [39] and the novelty on videos such as factorized attention and co-train are extensively studied by previous works. Nevertheless, combining these components together are worth noting. \n\n2. The writing is more or less like a draft version, it makes people with less prior knowledge very hard to follow. \n\n3. In formula (8), How much extra training time does the reconstruction cost?", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. A straightforward extension of [39] to videos with a factorized encoder, thus allowing joint training with images possible. It shows tricks like classifier-free guidance is also effective in video generation. \n\n2. Ablate different sampling methods in Table 6 shows that the proposed reconstruction-guided conditional sampling method performs the best. \n\n3. Results on UCF101, K600, and BAIR-RP in Table 1, 2, 3 are outstanding. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hAUAZ1g2T7e", "reviewer": "Reviewer_EcMx", "summary": "The paper proposed new diffusion model for video generation. The proposed model use 3D-Unet as a backbone for the diffusion model. Furthermore, the paper introduce novel conditional sampling technique for spatial and temporal video generation. The proposed methods present on unconditional video modeling, video prediction, and text conditioned video generation. Moreover, the proposed method show large improvement comparing to the baselines. ", "questions": "It will be helpful to provide the reason why you choose to use the 3D-Unet. Are there any other architecture that can be also appropriate for this task?\n\nRegarding the dataset that chosen - Can you think about dataset that has more diverse content and interactions? for example, not only video that has repetitive movements. ", "limitations": "The author provide an section regarding potential negative societal impact of their work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper introduce a novel diffusion model for video generation \n- The paper proposed two novel ideas to drive video X^b from sampled video X^a\n- The improvement in the results is relatively high.\n- The paper is well written and easy to follow.\n\n\n\nWeaknesses:\n- Results - looking on the generated video, the resolution is low. Can you trained the model for higher resolution? We all know that diffusion model is capable to generate high fidelity images, so it seems straight forward to use the super resolution approach like in CDM [cascade diffusion model] and get better video.\n- The paper should compare to other previous results. for example, StyleGAN-V [CVPR22], DIGAN [ICLR22].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept