Flexible Diffusion Modeling of Long Videos
Authors: William Harvey
Saeid Naderiparizi
Vaden Masrani
Christian Dietrich Weilbach
Frank Wood
Venue: NeurIPS 2022 Accept (2022)
Keywords: generative modeling
denoising diffusion probabilistic model
video modeling

Abstract: We present a framework for video modeling based on denoising diffusion probabilistic models that produces long-duration video completions in a variety of realistic environments. We introduce a generative model that can at test-time sample any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize a variety of schedules for the order in which frames in a long video are sampled and use selective sparse and long-range conditioning on previously sampled frames.  We demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length.  We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.

Flexible Diffusion Modeling of Long Videos
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, Frank Wood'
Department of Computer Science University of British Columbia
Vancouver, Canada
(wagh, saeidnp, vadmas, weilbach fwood) ubc.
Abstract
We present framework for video modeling based on denoising diffusion prob- abilistic models that produces long-duration video completions in a variety realistic environments. We introduce generative model that can at test-time sam- ple any arbitrary subset of video frames conditioned on any other subset and present an architecture adapted for this purpose. Doing so allows us to efficiently compare and optimize variety of schedules for the order in which frames in long video previously
are sampled and selective sparse and Long-range conditioning sampled frames. We demonstrate improved video modeling over prior work on number of datasets and sample temporally coherent videos over 25 minutes in length. We additionally release a new video modeling dataset and semantically meaningful metrics based on videos generated in the CARLA autonomous driving simulator.
Introduction
Generative modeling of photo-realistic videos is at the frontier of what is possible with deep learning on aurrently-available hardware. Although related work has demonstrated modeling of short photo- realistic videos (e.g. 30 frames [36], 48 frames [6] or 64 frames [16]), generating longer videos that are both coherent and photo-realistic remains an open challenge. major difficulty is scaling: photorealistic image generative models [4, 8] are already close to the memory and processing limits of modern hardware. long video very least a concatenation of many photorealistic frames, implying resource requirements, longerange coherence notwithstanding, that scale with frame count.
Attempting model such long-range coherence makes the problem harder still, especially because in general every frame can have statistical dependencies on other frames arbitrarily far back in the video. Unfortunately fixed-lag autoregressive models impose unrealistic conditional independence assump- tions (the next frame being independent of frames further back in time than the autoregressive lag is problematic for generating videos with long-range coherence). And while deep generative models based on recurrent neural networks (RNN) theoretically impose no such conditional independence assumptions, in practice they must trained over short sequences [12, 26] or with truncated gradi- ents [31]. Despite this, some RNN-based video generative models have demonstrated longer-range coherence, albeit without yet achieving convincing photorealistic video generation [26, 3, 7, 20, 2].
In this work we embrace the fact that finite architectures will always impose conditional indepen- dences. The question ask is: given an explicit limit K on the number of video frames we can jointly model, how can we best allocate these frames to generate video of length N > K? One option to use the previously-dese -exerribd autoregressive model but, if K N/44, we could instead follow Ho et al. [16] by training two models: one which first samples every 4th frame in the video, and another which (in multiple stages) infills the remaining frames conditioned on those. enable
Frank Wood is also affiliated with the Montréal Institute for Learning Algorithms (Mila) and Inverted AL.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

0
5 minutes
25 minutes
Figure 1: A long video (25 minutes, or approximately 15 000 frames) generated by FDM for each of CARLA Town01 and MineRL. conditioned on 500 and 250 prior frames respectively. We show blocks of frames from three points within each video, starting from the final observed frame on the left. Blocks are marked with the time elapsed since the last observation and frames within them are one second apart. We observe no degradation in sample quality even after > 15 000 frames.
efficient exploration of the space of such sampling schemes, propose flexible architecture based on the denoising diffusion probabilistic model (DDPM) framework. This can sample any subset of video frames conditioned on observed values of any other subset of video frames. It therefore lets us explore a wide variety of previously untested sampling schemes while being easily repurposed for different generation tasks such unconditional generation, video completion, and generation of videos of different lengths. Since our model can be flexibly applied to sample any frames given any others we call it Flexible Diffusion Model, FDM.
Contributions (1) At the highest level, we claim to have concurrently developed one of the first denoising diffusion probabilistic model (DDPM)-based video generative models [16, 40J. To do so we augment a previously-used DDPM image architecture [15, 22] with temporal attention mechanism including a novel relative (frame) position encoding network. (2) The principal contribution of this paper, regardless, is "meta-learning" training objective that encourages learning of video generative model that can (a) be flexibly conditioned on any number of frames (up to computational resource constraints) at any time in the past and future and (b) be flexibly marginalized (to achieve this within computational resource constraints).
(3) We demonstrate that our model can be used to efficiently explore the space of resource constrained video generation schemes, leading to improvements over prior work on several long-range video modeling tasks. (4) Finally, we release new autonomous driving video dataset along with a new video generative model performance metric that captures semantics more directly than the visual quality and comparison metrics currently in widespread use.
Sampling long videos
Our goal in this paper is to sample coherent photo-realistic videos with thousands of frames (see Fig. 1). To sample an arbitrarily long video with a generative model that can sample or condition on only small number of frames at once, must this is an autoregressive scheme, an example of which is shown in Fig. 2a for a video completion task. In this example it takes seven stages to sample a complete video, in that must run the generative model's sampling procedure seven times. At each stage three frames are sampled conditioned on the immediately preceding four frames. This scheme is appealing for its simplicity but imposes strong assumption that, given the set of four frames that are conditioned on at a particular stage, all frames that come afterwards are conditionally independent of all frames that came before. This restriction can be partially ameliorated with the sampling scheme shown in Fig. 2b where, in the first three stages, every second frame is sampled and then, in the remaining four stages, the remaining frames are infilled. One way to implement this would be to train two different models operating at the two different temporal resolutions. In the language of Ho al. [16]. who use a similar approach, sampling would be carried out in the first three stages by a "frameskip-2" model and, in the remaining stages, by a "frameskip-1" model. Both this approach and the autoregressive approach are examples of what we call sampling schemes. More generally, we characterize a sampling scheme sequence of tuples V/)) V.)]s each containing vector X, of indices of frames to sample and vector ), of indices of frames to condition on for stages = 1.
sequential procedure. The simplest example of

---

Algorithm 1 Sample video given sampling scheme ((X)..))) For unconditional generation. the imput v can be a tensor of zeros. For conditional generation, the observed input frames should contain their observed values.
I:
procedure SAMPLEVIDEONE 0) for <- I.....5 5 do
2:
3:
y + v('.]
D- Gather frames indexed by V.. P Sample x from the conditional DDPM. Modify frames indexed by X, with their sampled values.
4:
x~ DDPM (y, X, V',
v|X6] +
6: return v
(a) Autoregressive.
(b) Two temporal res.
(c) Long-range (ours).
(d) Hierarchy-2 (ours).
Figure 2: Sampling schemes to complete a video of length N 30 conditioned on the first 10 frames, with access to at most K 7 frames at time. Each stage of the sampling procedure is represented by one row in the figure, going from top to bottom. Within each subfigure, one column represents one frame of the video, from frame of frames marked in blue are sampled conditioned on the (observed or previously sampled) values of frames marked in red; frames marked in gray are ignored; and frames marked in white are yet be sampled. For every sampling scheme, all video frames have been sampled after the final row.
the left frame 30 on the right. At each stage, the values
Algorithm I lays out how such a sampling scheme used to sample a video. If the underlying generative model is trained specifically to model sequences of consecutive frames, sequences of regularly-spaced frames, then the design space for sampling schemes compatible with these models is severely constrained. In this paper we take different approach. We design and train generative model to sample any uubitrarily-chose train it using entirely novel distribution of such tasks. In short, our model is trained to generate frames for any choice of X' and y. The only constraint we impose on our sampling schemes is therefore computational consideration that [X,] + [D& < K for all but, generate meaningful videos, any valid sampling scheme must also satisfy two more constraints: (1) all frames are sampled at least one stage and (2) frames are never conditioned upon before they are sampled.
subset of video frames conditioned on any other subset and
Such a flexible generative model allows us to explore and use sampling schemes like those Fig. 2c and Fig. 2d. We find in our experiments that the best video sampling scheme is dataset dependent. Accordingly, we have developed methodology to optimize such sampling schemes in a dataset dependent way, leading to improved video quality measured by the Fréchet Video Distance [33] among other metrics. We now review conditional DDPMs (Section 3), before discussing the FDM's architecture, the specific task distribution used to train it, and the choice and optimization of sampling schemes in Section 4.
3 A review of conditional denoising diffusion probabilistic models
Denoising diffusion probabilistic models, or DDPMs [28, 15, 22, 30]. a class of generative model for data x, which throughout this paper will take the form of a 4-dimensional tensor representing multiple video frames. We will describe the conditional extension [32], in which the modeled is conditioned on observations DDPMs simulate a diffusion process which transforms x to noise, and generate data by learning the probabilistic inverse of the diffusion process. The diffusion process happens over timesteps 0,, T such that X x is data without noise, xi has a very small amount of noise added, and so on until XT is almost independent of x9 and approximates a random sample from a unit Gaussian. In the diffusion process we consider, the distribution over x, depends only on Xt-1:
(x, X N -xx (1)
Hyperparameters On, OT are chosen all be close to but slightly less than so that the amount of noise added at each step is small. The combination of this diffussin process and data distribution
3

---

Video UNee
X
V KX x
y, XT
K xXX2X 32
KX888KC
y, Xt
Royeet occ Bowntarii
Xx4x4xC
Tarker Blct Am
y, y
xxxxxxx
Sasher Block Arm Upaanse
XX888XC
y, X0
32
Back
SK xEx 6
X,-1
Figure 3: Left: Our DDPM iteratively transforms Gaussian noise XT to video frames Xo (shown with blue borders), conditioning on observed frames y (red borders) every step. Right: The U-net architecture used within each DDPM step. It computes ex(X,, !), with which the Gaussian transition ( (x) is parameterized.
g(xo.y) y) (recalling that x) defines the joint distribution
T
((x.TT y) gxx. y) II (xx(xx1-)..
(2)
t=1
DDPMs work by "inverting" the diffusion process: given values of x, and y neural network is used to parameterize pe (xxx_1/xx,,7 y). an approximation of (xx,_1 This neural network lets us draw samples of Xg by first sampling xy from a unit Gaussian (recall that the diffusion process was chosen so that ((XT) is well approximated by a unit Gaussian), and then iteratively sampling Xt-1 Po Xx, y) for T, T ,,..., The joint distribution of sampled Xp:r given y
T
po(xo:rly) p(xT) II po
where p(xx) is unit Gaussian that does not depend on 0. Training the conditional DDPM therefore involves fitting (a(x (),, to approximate 1xx,_1/xx, for choices of I, Xx, and y.
Several observations have been made recent years which simplify the learning of pa ( y). Sohl-Dickstein al. [28] showed that when 0g is close 1, Pe (x,=1/x,) is approximately Gaus- sian [28]. Furthermore, Ho et al. [15] showed that this Gaussian's 's variance can be modeled well with non-learned function of and that a good estimate of the Gaussian' mean can be obtained from a "denoising model" as follows. Given data Xo and unit Gaussian noise the denoising model (in the form neural network) is fed "noisy" data X; := VåyXo + /1 åÿc and trained to recover via a mean squared error loss. The parameters å; := I ai are chosen ensure that the marginal distribution of x₁ given is (x,, Xo) as derived from Eq. (1). Given weighting function X(t), the denoising loss
(())= Ex(xx.y.e)
cx - y, 0, with x, Vārxa V1 ā,e.
(4)
The mean of Po 101x-11XX,,3 is obtained from the denoising model's output co(x,, y,t) as
Lx, م-مي VI-a, (x (). If the weighting function X(1) is chosen appropriately, optimising Eq. (4)
equivalent optimising lower-bound on the data likelihood under Pa. In practice, simply setting X(t) for all f can produce more visually compelling results in the image domain [15].
In our proposed method, as in Tashiro et al. [32], the shapes of Xy and y sampled from q(-) vary. This because we want train model which can flexibly adapt to e.g. varying numbers of observed frames. To map Eq. (4) to this scenario, note that both and implicitly contain information about which frames in the video they represent (via the index vectors X and y introduced the previous so that interactions between
section). This information is used inside the neural network ep cx(x,, frames can be conditioned on the distance between them (as described in the following section) and also to ensure that the sampled noise vector has the same shape as Xo.

---

Algorithm 2 Sampling training tasks X, )' ~ ul.) given N, K.
1:
X == ():}==1) 0
2:
while True do
3:
?'group
Uniform K)
4:
$group
LogUniform(L (N 11) /ngoupp
5:
Jirrmp
Uniform (0, N (Agrup 1) Sgroup)
6:
Ograpp
Bernoulli(0.3 (0.5)
7:
{ [ggoop + Sgroup -ilié E {0. Vegroup 1}) 12 13
8:
if | [ + [ K then
9:
return set2vector(2)), set2vector())
10:
else if | = 0 or Ogroup = 0 then
11:
X := XUg
12:
else
13:
y y==
Figure 4: Left: Samples from v') with video length NN = 30 and limit K = 10 on the number of sampled indices. Each row shows one sample and columns map frames, with frame I on the left and frame N on the right. Blue and red denote latent and observed frames respectively. All other frames are ignored and shown as white. Right: Pseudocode for drawing these samples. The while loop iterates over a series of regularly-spaced groups of latent variables. Each group is parameterized by: the number of indices in it, Ngroup:' the spacing between indices in it, Sgroup the position of the first frame in it, Tgrup. and an indicator variable for whether this group is observed, Ogroup (which ignored on line 10 if X is empty to ensure that the returned value of X' is never empty). These quantities are sampled in continuous space and then discretized to make a set of integer coordinates on line 7. The process repeats until a group is sampled which, if added number of frames to exceed K. That group is then discarded and X' and y are returned vectors. The FDM's training objective forces to work well for any ()) pair from this broad distribution.
X' or y, will cause the
4 Training procedure and architecture
Training task distribution Different choices of latent and observed indices X' and y can be regarded as defining different conditional generation tasks. In this sense, we aim to learn model which can work well on any task (i.e. any choice of and y) and so we randomly sample these vectors of indices during training. We do with the distribution u(X', V) described in Fig. 4. This provides a broad distribution covering many plausible test-time use cases while still providing sufficient structure to improve learning (see ablation in Section 6 and more details in Appendix C). To cope with constrained computational resources, the distribution is designed such that |X| + [J*] is
upper- bounded by some pre- pre-specified K. Sampling from (Xx. y) in Eq. (4) is then accomplished by randomly selecting both a full training video v and indices N, ~ u(...). We then extract the specified frames x v[X] and y v()) (where we v[x)] denote the concatenation of all frames in v with indices in X' and and v()i similarly).
Architecture DDPM image models [15, 22] typically use U-net architecture [24]. Its distinguish- ing feature is a series of spatial downsampling layers followed by a series of upsampling layers, and these are interspersed with convolutional res-net blocks [14] and spatial attention layers. Since we require an architecture which operates on 4-D video tensors rather than 3-D image tensors we add an extra frame dimension to its input, output and hidden state, resulting in the architecture shown on the right of Fig. 3. We create the input to this architecture as concatenation X, @ y, adding an extra input channel which is all ones for observed frames and all zeros for latent frames. For RGB video, the input shape is therefore (K, image height, image width, 4). Since the output should have the same shape as x, we only return outputs corresponding to the latent frames, giving output shape (I2'), image height, image width, 3). We run all layers from the original model (including convolution, resizing, group normalization, and spatial attention) independently for each of the K frames. To allow communication between the frames, we add a temporal attention layer after each spatial attention layer, described in more detail in the appendix. The spatial attention layer allows each spatial location to attend to all other spatial locations within the same frame, while the temporal attention layer allows each spatial location to attend to the same spatial location across all other frames. This combination of temporal attention layer with spatial attention layer is sometimes
5

---

referred to as factorized attention [32, 16]. We found that, when using this architecture in conjunction with our meta-learning approach, performance could be improved by using a novel form of relative position encoding [27, (8]. This is included in our released source code but we leave its exposition to the supplementary material.
Training batch padding Although the size |x @ J' of index vectors sampled from our training distribution is bounded above by K, it can vary. To fit examples with various sizes of index vectors into the same batch, one option would be to pad them all to length K with zeros and use masks so that the zeros cannot affect the loss. This, however, would waste computation on processing tensors of zeros. We instead use this computation obtain a lower-variance loss estimate by processing additional data with "training batch padding". This means that, for training examples where |x @ y/ < K, we concatenate frames uniformly sampled from a second video to increase the length along the name-dimension to K. Masks are applied to the temporal attention mechanisms so that frames from different videos cannot attend to eachother and the output for each the same as that achieved by processing the videos in different batches.
Sampling schemes Before describing the sampling schemes we explore experimentally, we empha- size that the relative performance of each dataset-dependent and there is no single best choice. A central benefit of FDM is that it can be used at test-time with different sampling schemes without retraining. Our simplest sampling scheme, Autoreg, samples ten consecutives frames at each stage conditioned on the previous ten frames. Long-range is similar to Autoreg but conditions on only the five most recent frames as well as five of the original 36 observed frames. Hierarchy-2 uses a multi-level sampling procedure. In the first level, ten evenly spaced frames spanning the non- non-observed portion of the video are sampled (conditioned on ten observed frames). the second level, groups of consecutive frames are sampled conditioned on the closest past and future frames until all frames have been sampled. Hierarchy-3 adds an intermediate stage where several groups of variables with an intermediate spacing between them are sampled. We include adaptive hierarchy-2. abbreviated Ad. hierarchy-2. as a demonstration of a sampling scheme only possible with a model like FDM. It samples the same frames at each stage as Hierarchy-2 but selects which frames to condition on adaptively at test- best-time with a heuristic aimed at collecting the maximally diverse set of frames, as measured by the pairwise LPIPS distance [41] between them.
Optimizing sampling schemes An appealing alternative to the heuristic sampling schemes de- scribed in the previous paragraph would be to find a sampling scheme that is, in some sense, optimal for given model and video generation/complltion task. While it is unclear how to tractably choose which frames should be sampled at each stage, we suggest that the frames to condition on at each stage can be chosen by greedily optimizing the diffusion model loss which, as mentioned in Section 3, closely related to the data log-likelihood Given fixed sequence of frames to sample at each stage [x,]s= select Vi for each to minimize Eq. (4). This is estimated using set of 100 training videos and by iterating over 10 evenly-spaced values of t (which reduced variance relative to random sampling of t). See the appendix for further details. We create two optimized sampling schemes: one with the same latent indices as Autoreg, and one with the same latent indices as Hierarchy-2. We call the corresponding optimized schemes Opt. autoreg and Opt. hierarchy-2.
CARLA Town01 Dataset
In addition our methodological contributions, we propose new video-modeling dataset and benchmark which provides an interpretable measure of video completion quality. The dataset consists of videos of car driving with a first-person view, produced using the CARLA autonomous driving simulator [9]. All 408 training and 100 test videos (of length 1000 frames and resolution 128 x 128) are produced within single small town, CARLA's Town01. As such, when sufficiently expressive video model is trained on this dataset it memorizes the layout of the town and videas sampled from the model will be recognisable as corresponding to routes travelled within the town. We train a regression model in the form of neural network which maps with high accuracy from any single rendered frame to (ar, y) coordinates representing the car's position. Doing so allows us to plot the routes corresponding to sampled videos (see left of Fig. 5) and compute mimaatically-mannngge quantitative measures of the validity of these routes. Specifically, we compute histograms of speeds, where each speed is estimated by mcasuring the distance between the regressed locations for frames
yet

---

Observed
Sampled
Displayed
Figure 5: Left: Map of the town featured in the CARLA Town01 dataset. We visualize two video completions by FDM by showing coordinates output by our regressor (discussed in Section 5) for each frame. Those corresponding to the initial 36 observed frames are shown in red and those for the 964 sampled frames are shown blue. Right: For each completion, we show observed frames followed by four of the sampled frames (at positions chosen to show the progression with respect visible landmarks and marked by black dots the map). The town's 's landmarks are usually sampled with high-fidelity, which is key to allowing the regressor to produce a coherent trajectory on the left. However there sometimes failures: blue square near the top-right of the map shows where the video model "jumped" to a wrong location for a single frame.
of the initially
spaced ten apart (1 second at the dataset's frame rate). Sampled videos occasionally "jump" between disparate locations in the town, resulting in unrealistically large estimated speeds. To measure the frequency of these events for each method, we compute the percentage our point-speed estimates that exceed threshold of 10m/s (the dataset was generated with maximum simulated speed of 3m/s). We report this metric as the outlier percentage (OP). After filtering out these outliers, we compute the Wasserstein distance (WD) between the resulting empirical distribution and that of the original dataset, giving a measure of how well generated videos match the speed of videos in the dataset. We release the CARLA Town 01 dataset along with code and our trained regression model to allow future comparisons. 2
Experiments
We perform our main comparisons on the video completion task. In keeping with Saxena et al. [26]. we condition on the first 36 frames of each video and sample the remainder. We present results on
three datasets: GQN-Mazes [10], in which videos are 300 frames long; MineRL Navigate [13, 26] (which we will from
refer to simply MineRL), in which videos are 500 frames long, and the CARLA Town01 dataset we release, for which videos are 1000 frames long. We train FDM in all cases with the maximum number of represented frames K 20. We host con-cherry-picckee video samples (both conditional and unconditional) from FDM and all baselines online?.
Comparison of sampling schemes The relative per- formance of different sampling schemes varies sig- nificantly between datasets as shown in Table 1. We report Fréchet Video Distances (FVDs) [33], a mea- sure of how similar sampled completions are the test set, on all datasets. In addition on GQN-Mazes we we report the accuracy metric [26]. which classifies videos based on which rooms are visited and measures
2.0
Ad. Hierarchy-2
1.5
Ground Truth
Density
Long-range
1.0
how often a completion given the same class as the corresponding test video. For CARLA Town01 we re- port the previously described percentage outliers (PO) and Wasserstein distance (WD) metrics.
0.0
Speed (m/ /))
We can broadly consider the aforementioned sampling schemes as either being in the "autoregressive" fam- ily (Autoreg and Long-range) or in the "hierarchical"
Figure 6: Speed distributions measured from sampled and ground-truth dataset videos.
'nttpa://github http://ithub..com//la.i.t wrtppaaagroopplleilllleeeeeeeeee "https://www. cs. ubc. wac..ca//eee ca/"wsgh/fdn
snfffusionmmodeiing
7

---

Table 1: Evaluation video completion with various modes of method along with several baselines from the literature. Error bars denote the standard error computed with random seeds. Higher is better for the accuracy metric [26] and lower is better for all other metrics shown.
GQN-Mazes
MineRL
CARLA Town01
Model
Sampling scheme
FVD
Accuracy
FVD
FVD
WD
OP
CWVAE [26]
CWVAE
837 +
82.6 + 0.5
1573 +
1161
0.666
44.1
TATS [11]
TATS
163 + 2.6
77.0 0.8
807 + 14
329
1.648
42.4
VDM [16]
VDM
66.7 + 1.5
77.8 0.5
271 8.8
169
0.501
16.9
Autoreg
86.4 + 5.2
69.6 1.3
281 + 10
222
0.579
0.51
Long-range
64.5 1.9
77.0 77.01:
267 = 4.0
213
0.653
0.47
FDM (ours)
Hierarchy-2
53.1 1
0.7
275 7.7
120
0.318
3.28
Hierarchy-3
53.7 1.9
83
311
149
0.363
4.53
Ad. hicrarchy-2
55.0 1.4
8.3
316
117
0.311
3.44
family (the remainder). Those in the hierarchical family achieve significantly better FVDs [33] on GQN-Mazes. Our samples in the appendix suggest that this is related to the autoregressive methods "forgetting" the colors of walls after looking away from them for short time. In contrast, for MineRL the autoregressive methods tend to achieve the best FVDs. This may relate to the fact that trajectories in MineRL tend to travel in straight lines through procedurally-genett the number of long-range dependencies. Finally on CARLA Town01 we notice qualitatively different behaviours from our autoregressive and hierarchical sampling schemes. The hierarchical sampling schemes have a tendency to occasionally lose coherence and "jump" to different locations in the town. This is reflected by higher outlier percentages (OP) in Table On the other hand the autoregressive schemes often stay stationary for unrealistically long times at traffic lights. This is reflected in the histogram of speeds in Fig. 6, which has larger peak around zero than the ground truth. The high variance of the sampling scheme's relative performance over different datasets points to strength of our method, which need only be trained once and then used to explore a variety of sampling schemes. Furthermore, we point out that the best FVDs in Table on all datasets were obtained using sampling schemes that could not be implemented using models trained prior work, or over evenly spaced frames.
"worlds" [[13, 26], limiting
Comparison with baselines The related work most relevant to ours is the concurrent work of Ho al. [16], who model 64-frame videos using two trained DDPMs. The first is a "frameskip-4" model trained to generate every fourth frame and the second is a "frameskip-1" model trained on sequences of nine consecutive frames and used to "fill in" the gaps between frames generated the first stage. To compare against this approach, which denote VDM. and a "frameskip-1" model with architectures identical to our own. 4 Since VDM requires two trained DDPMs, we train it for more GPU-hours than FDM despite the fact that FDM is meta-learning over a far broader task distribution. We also compare against TATS [11], which embeds videos into discrete latent space before modelling them with transformers, and the clockwork VAE (CWVAE) [26]. a VAE-based model specifically designed to maintain long-range dependencies within video.
train both a "frameskip-4
Both the diffusion- methods, FDM and VDM, achieve significantly higher FVD scores than TATS and CWVAE This may point toward the utility of diffusion models in general for modeling images and video. Table also makes clear the main benefit of FDM over VDM: although there is no sampling scheme for FDM which always outperforms VDM. there is at least one sampling scheme that outperforms on each dataset. This speaks to the utility of learning a flexible model like FDM that allows different sampling schemes to be experimented with after training.
Optimized sampling schemes As mentioned in Section 4, another advantage of FDM is that it makes possible a model- and dataset-specific optimization procedure to determine on which frames
*The VDM is concurrent work and, at the time of writing, without a code-release. Since we intend this primarily as a comparison against the VDM sampling scheme we do not reimplement their exact architecture and note that there other differences including their approach imputation.

---

Table FVD scores for our sampling schemes with observed indices optimized offline as described in Section 4. We mark with an asterisk (*) the eight numbers which improve on the corresponding non-optimized sampling schemes and highlight bold those that are better than any in Table
GQN-Mazes
MineRL
CARLA Town01
Sampling scheme
FVD
Accuracy
FVD
FVD
WD
OP
Opt. autoreg
53.6 = 1.2
80.2 + 1.2"
257 + 6.8
146"
0.452
0.65
Opt. hierarchy-2
51.1 + 1.3*
84.6 = 0.7"
320 + 7.0
124
0.349
4.11"
to condition. Table 2 shows the results when this procedure is used to create sampling schemes for different datasets. In the first row we show results where the latent frames are fixed to be those of the Autoreg sampling scheme, and in the second row the latent frames are fixed to match those of Hierarchy-2. On two of the three datasets the best results in Table are improved upon, showing the utility of this optimization procedure.
Comparison with training on single task Training network with our distribution over training tasks could be expected to lead to worse performance on a single task than training specifically for that task. To test whether this is the case, we train an ablation of FDM with training tasks exclusively of the type used in our Autoreg sampling scheme, i.e. "predict ten consecutive frames given the previous ten." "n." Tested with the Autoreg sampling scheme, it obtained an FVD of 82.0 on GQN-Mazes and 234 on MineRL. As expected given the specialization to a single task, this is better than when FDM is run with the Autoreg sampling scheme (obtaining FVDs of 86.4 and 281 respectively).
Ablation on training task distribution To test how important our proposed structured training distribution is to FDM's performance, we perform an ablation with different task distribution that samples X' and Y from uniform distributions instead of our proposed structured task distribution We provide full details in the appendix, but report here that switching away form our structured training distribution made the FVD scores worse on all five tested sampling schemes on both GQN-Mazes and MineRI.. The reduction in the average FVD was 31 % on GQN-Mazes and 52% on MineRI. This implies that our structured training distribution has significant positive effect.
7 Related work
Some related work creates conditional models by adapting the sampling procedure of an unconditional DDPM [30, 18, 21, 16]. These approaches require approximations and the more direct approach that we use (explcitly training a conditional DDPM) was shown to have benefits by Tashiro al. [32]. We consider further comparison of these competing approaches to be outside the focus of this work, which on modeling small portion of video frames at a time, essentially performing marginalization in addition to conditioning.
There number of approaches in the literature which use VAEs rather than DDPMs for video mod- elling. Babaeizadeh et al. [2] use a VAE model which predicts frames autoregressively conditioned on global time-invariant latent variable. A related approach by Denton and Fergus [7] also uses VAE with convolutional LSTM architectures in both the encoder and decoder. Unlike Babaeizadeh al. [2] the prior is learned and different latent variable is sampled for each frame. Babaeizadeh et al. [3] use a VAE with one set of latent variables per frame and inter-frame dependencies tracked by two-layer LSTM. Their architecture intentionally overfits to the training data, which when coupled with image augmentations techniques achieves SOTA on various video prediction tasks. Kim al. [20] use a variational RNN [5] with a hierarchical latent space that includes binary indicator variables which specify how the video is divided into a series of subsequences. Both Villegas et al. [35] and Wichers al. [37] target long-term video prediction using a hierarchical variational LSTM architecture, wherein high-level features such as landmarks are predicted first, then decoded into low-level pixel space. The two approaches differ in that Villegas et al. [35] requires ground truth landmark labels, while [37] removes this dependence using unsupervised adversarial approach. Fully GGN--asee video models have also been proposed [1, 6) but generally suffer from "low quality frames low number of frames or both" [1].

---

8
Discussion
Compute Canada We have defined and empirically explored a new method for generating photorealistic videos with long-range coherence that respects and efficiently uses fixed, finite computational resources. Our approach outperforms prior work on long- aog-duration video modeling as measured by quantitative and semantically meaningful metries and opens up several avenues for future research. For one, similar to using DDPMs for image generation, our method is slow to sample from (it takes approximately 16 minutes to generate 300 frame video on a GPU). Ideas for making sampling faster by decreasing the number of integration steps [25, 29, 39] could be applied to our video model.
On a different note, consider the datasets on which our artifact was trained. In each there was policy for generating the sequences of actions that causally led to the crame-to-frame changes in camera pose. In MineRL the video was generated by agents that were trained to explore novel Minecraft worlds to find a goal block approximately 64 meters away [13]. The CARLA data was produced by camera attached to an agent driven by low level proportional following waypoints laid down by a high level planner that was given new, random location goals to drive to intermittently. In both cases our video model had no access either the policy or the specific actions taken by these agents and, so, in a formal sense, our models integrate or marginalize over actions drawn from the stochastic policy used to generate the videos in the first place. Near-term future work could involve adding other modalities (e.g. audio) to FDM as well as explicitly adding actions and rewards, transforming our video generative model into bsson- world model in the reinforcement learning sense [17, 19]. Furthermore, we point out that FDM trained on CARLA TownO1 is in theory capable of creating 100-second videos conditioned on both the first and final frame. Doing so can be interpreted as running a "visual" controller which proposes a path between a current state and a specified goal. Preliminary attempts to run in FDM in this way yielded inconsistent results but we believe that this could be a fruitful direction for further investigation.
-erivvvvvev controller
Acknowledgments
We would like to thank Inverted AI, and especially Alireza Morsali, for generating the CARLA Town01 dataset. We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, and the Intel Parallel Com- puting Centers program. Additional support was provided by UBC's Composites Research Net- work (CRN), and Data Science Institute (DSI). This research was enabled in part by technical support and computational resources provided by WestGrid www.wwestgrid.co. www.ccomputecanaaa.ca and Advanced Research Computing at the University of British Columbia (arc.ubc.ca). WH acknowledges support by the University of British Columbia's 's Four Year Doctoral Fellowship (4YF) program.
References
[1] Nuha Aldausari, Arcot Sowmya, Nadine Mareus, and Gelareh Mohammadi, Video generative adversarial networks: review. ACM Computing Surveys (CSUR), 55(2):1-25, 2022.
[2] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint 1710. 2017.
[3] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint arXiv:2106.1 13195, 2021.
[4] Rewon Child, Very deep vaes generalize autoregressive models and can outperform them on images. arXiv preprint arXiv:2011.106500 2020.
[5] Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28, 2015.
[6] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint 177.0.1907.06571, 2019.
10

---

[7] Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In International conference on machine learning, pages 1174-1183, PMLR, 2018.
[8] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[9] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[10] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (20006694::224-1222
2018.
[11] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.03638, 2022.
[12] Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanctot, and Alex Graves. Memory- efficient backpropagation through time. Advances in Neural Information Processing Systems, 29, 2016.
[13] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv: 13440, 2019.
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arxiv 2015. arXiv preprint arXiv: 2015.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[16] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03455 2022.
[17] Maximilian Igl, Luisa Zintgraf, Tuan Anh Le, Frank Wood, and Shimon Whiteson. Deep varia- tional reinforcement learning for pomdps. In International Conference on Machine Learning. pages 2117-2126. PMLR, 2018.
[18] Zahra Kadkhodaie and Ecro P Simoncelli. Solving linear inverse problems using the prior implicit in a denoiser. arXiv preprint arXiv:2007.13640. 2020.
[19] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, al. Model- based reinforcement learning for atari. arXiv preprint arXiv: 000040 2019.
[20] Taesup Kim, Sungjin Ahn, and Yoshua Bengio. Variational temporal abstraction. Advances in Neural Information Processing Systems, 32, 2019.
[21] Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffusion models. arXiv preprini arXiv:2103. 2021.
[22] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8177 8171. PMLR, 2021.
[23] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,o Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W. W. 2017.
[24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net:
Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and comsuter-assisted intervention, pages 234-241. Springer, 2015.
[25] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512 2022.

---

[26] Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 34, 2021.
[27] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint 1rXiv:1803.02155, 2018.
[28] Jascha Sohl- Dickssicn,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2266222222222222222222222 2265. PMLR, 2015.
[29] liaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[30] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
[31] Corentin Tallec and Yann Ollivier. Unbiasing truncated backpropagation through time. arXiv preprint arXiv: 1005:0770008209 2017.
[32] Yusuke Tashiro, Jiaming Song. Yang Song, and Stefano Ermon. Csdi: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34, 2021.
[33] Thomas Unterthiner, Sjoerd Steenkiste, Karol Kurach, Raphael Marinier, Marein Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 181.001777, 2018.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[35] Ruben Villegas, Dumitru Erhan, Honglak Lee, et al. Hierarchical long-term video prediction without supervision. In International Conference on Machine Learning, pages 6038-6046. PMLR, 2018
[36] Dirk Weissenborn, Osear Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprini (9t.:::9900002344, 2019.
[37] Nevan Wichers, Ruben Villegas, Dumitru Erhan, and Honglak Lee. Hierarchical Long-term Video Prediction without Supervision. June 2018. doi: D0.48550/arxiv.1800.07768
[38] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10033-10041, 2021.
[39] Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference Learning Representations, 2022. URL ttttps://openreiieee net/ ttttt/foommiiddii wo-puumiid-pp//o---0000.
[40] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint 20X1v:2203.09481, 2022.
[41] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ta0h3FKhjei", "reviewer": "Reviewer_wWqf", "summary": "The paper presents a framework for generating long (many minutes, to hours) videos using diffusion models, extending to a series of progressively more complex environments such as Deepmind Lab, Minecraft, and CARLA (a self-driving simulator). As a key contribution, the authors introduce an efficient diffusion model that is trained to generate future frames based on a sparse selection of past frames. The authors investigate a series of selection methods, including various local or strided schedules, as well as a novel learned schedule. \n", "questions": "(1) Regarding my concern about learning long-range dependencies, it might provide a little more insight into the model if the authors were able to evaluate their GQN-Mazes model on the accuracy metric introduced in CW-VAE (https://proceedings.neurips.cc/paper/2021/file/f490d0af974fedf90cb0f1edce8e3dd5-Paper.pdf ). Note that only the Neurips version of the paper mentions it (arxiv was not updated). The relevant accuracy code can be found here: https://gist.github.com/vaibhavsaxena11/97b2d0a195c08ab2ed75cebb7d763799 .\n\n(2) As mentioned before, it’s a little unclear to me what the purpose of the CARLA dataset is. Since it contains only videos of a small town, it should be fairly easy for a model to memorize the town’s layout. If we're aiming to test a model's ability to memorize a scene, I don't think that's particularly difficult or interesting.\n\n(3) I think it would be useful to include some baselines on autoregressive models, such as TATS [1], or HARP [2], which can generate arbitrarily long videos using a sliding window with limited context\n\t\n\n\n\n[1] Ge, Songwei, et al. \"Long video generation with time-agnostic vqgan and time-sensitive transformer.\" arXiv preprint arXiv:2204.03638 (2022).\n\n[2] Seo, Younggyo, et al. \"Autoregressive Latent Video Prediction with High-Fidelity Image Generator.\" (2021).\n\n", "limitations": "Overall, the authors adequately addressed limitation and potential negative societal impacts of their work.\n\n- As mentioned in the paper, diffusion models are slow to sample from, especially in the case of long-horizon video generation in which you may need to generate thousands or tens of thousands of frames.\n- Sampling schemes, even when optimized, are dataset specific and not video specific, so the model is not able to dynamically retrieve context-dependent frames for generation\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well written and clear\n- The authors present a novel diffusion model for videos that is simple and easy to scale.\n- Experiments show high quality, visually consistent results on domains of varying difficulty, from more synthetic environments like DMLab to more realistic environments like CARLA\n\nWeaknesses\n- Novelty is a bit limited on the diffusion side, since it is a fairly direct application of image diffusion models to video\n\n- I’m not too convinced at the ability for the model to accurately learn long-range dependencies, primarily since:\n\n(1) FVD on GQN-Mazes is fairly sensitive to noise and will probably not capture long-range information such as maze layout. \n\n(2) MineRL doesn’t really require long-range information since, as mentioned in the paper, it mainly consists of agents moving forward, \nso only a few past frames are really needed for accurate video generation \n\n(3) CARLA is a pretty small dataset, and it seems like a sufficiently large model can just memorize the town layout\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oLNX6wVuqP_", "reviewer": "Reviewer_twbY", "summary": "This paper proposes a video diffusion model that enables you to sample video by flexible conditioning(depending on your application scenarios). Technical side, the author inject the conditioning information by concatenating it(conditional images) with original images, and feed them into UNet to predict the score/noise of DDPM. The author designs a sampling strategy to enable the pretrained network can be  competent of flexible sampling.  \n\nFor experiments, it mainly compares with Video Diffusion Model(VDM), CWVAE, in video completion task of three datasets(one of them is proposed by themselves). They also did several ablation studies to provide more insight about sampling schemes when testing, and sampling schemes when training. \n\n", "questions": "- More details about the comparison?\n- More explanation about the \"Con\" parts of above session?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pro:\n\n- The paper focuses on the video generation problem by diffusion method, which is a suitable, promising solution  by injecting the inductive-bias of grid-structured data like image, video, audio,etc.\n- Clear illustration of the proposed idea about sampling strategy, training network, etc.\n- The intuition of the method is technically sound in aiming to solve the complex sampling problem in diffusion models.\n\nCon:\n\n- Too few baselines to compare with, the author only compare with VDM a).in different datasets with them (they use UCF, which is way more complex than your dataset choice), b), they didn't release the code, lot's novelty impl of VDM is not included yet, like gradient-method, replacement method(check details from VDM), and also too few details are revealed to make the comparison with VDM not direct enough.\n- Why In table 1, the gap of  loss between VDM and FDM in dataset CARLA Town01 is  so large? Typically, loss is not a good metric here to compare with your method and baselines in Table 1,2. As the model, the hyperparameter, the data preprocessing, and many details can deviate your conclusion from loss comparison. \n- The idea about sampling training task is too simple, there is a lot of human bias  design inside, so I am not convinced by the proposed method especially when the baselines are too few.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4PRo5GJ8UQ-", "reviewer": "Reviewer_sEEi", "summary": "The paper proposes a DDPM based model for video frame prediction. It uses a randomized masking the randomly condition some frames on others allowing for long term predictions. Authors tested different sampling strategies on a newly introduced dataset to demonstrate its capability in generating long videos. The paper compares its result to CWVAE and Video Diffusion Models. ", "questions": "Please refer to weaknesses for my questions. Here is a short list:\n- Is the model overfitting or can it generalize to unseed cases?\n- How does the model perform vs a simpler DDPM method. \n- How does the model compare vs other models is other datasets (CWVAE Table 1 from arxiv has a good list)", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "===== Strengths:\n- The randomized masking algorithm is simple and effective particularly given the capability of limiting it to a fixed number of frames to put an upper bound on computation.\n- The paper is well written, easy to follow and read.\n- The new dataset is effective in demonstrating the capability of the proposed method for predicting long videos in a dynamic setting.\n\n==== Weaknesses:\n- Analysis. There are multiple points that the paper can be improved by adding more analysis. For example, while the proposed masking strategy makes sense, there is no experiment to show how effective it is vs simpler strategies such as just masking every 10 frames. I understand that the proposed method allows for different sampling strategy but it would be great to show how a simple autoregressive + infilling sampling with a simple masking strategy perform on a similar problem. \n\n- Comparisons. The paper only compares its results to other models in Table 1 which is not enough for a fair comparison. I suggest authors to include more comparisons. CWVAE has more dataset numbers, metrics and baseline numbers to compare to. FitVid also has a section on long video predictions (with overfitting) on Human3.6M. I also recommend comparing the model with other video prediction models on shorter videos, given the richer number of models in that domain which can help demonstrating the capability of DDPMs for video prediction.\n\n- Overfitting. Another possible improvement is to condition the model on an out of distribution data e.g. an unseen street. Given the size of the dataset, my main concern is that the model memorized everything and is not capable of generalization which is the main goal. FitVid did similar analysis and found overfitting as the main reason for their capability of generating super long videos.\n\n- Qualitative results. Given the nature of videos, I highly encourage authors to include videos on an anonymous website for better demonstration and comparison.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept