Temporally Consistent Video Transformer for Long-Term Video Prediction
Authors: Wilson Yan
Danijar Hafner
Stephen James
Pieter Abbeel
Venue: Submitted to ICLR 2023 (2023)
Keywords: video generation
video prediction
generative modeling
latent dynamics models

Abstract: Generating long, temporally consistent video remains an open challenge in video generation. Primarily due to computational limitations, most prior methods limit themselves to training on a small subset of frames that are then extended to generate longer videos through a sliding window fashion. Although these techniques may produce sharp videos, they have difficulty retaining long-term temporal consistency due to their limited context length. In this work, we present Temporally Consistent Video Transformer (TECO), a vector-quantized latent dynamics video prediction model that learns compressed representations to efficiently condition on long videos of hundreds of frames during both training and generation. We use a MaskGit prior for dynamics prediction which enables both sharper and faster generations compared to prior work. Our experiments show that TECO outperforms SOTA baselines in a variety of video prediction benchmarks ranging from simple mazes in DMLab, large 3D worlds in Minecraft, and complex real-world videos from Kinetics-600. In addition, to better understand the capabilities of video prediction models in modeling temporal consistency, we introduce several challenging video prediction tasks consisting of agents randomly traversing 3D scenes of varying difficulty. This presents a challenging benchmark for video prediction in partially observable environments where a model must understand what parts of the scenes to re-create versus invent depending on its past observations or generations. An anonymized website with samples can be found at https://sites.google.com/view/iclr23-teco

Under review as a conference paper at ICLR 2023
TEMPORALLY CONSISTENT VIDEO TRANSFORMER FOR LONG-TERM VIDEO PREDICTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Generating long, temporally consistent video remains an open challenge in video generation. Primarily due to computational limitations, most prior methods limit themselves to training on small subset of frames that are then extended to generate longer videos through a sliding window fashion. Although these techniques may produce sharp videos, they have difficulty retaining long-term temporal consistency due to their limited context length. this work, we present Temporally Consistent Video Transformer (TECO), vector-quanized latent dynamics video prediction model that learns compressed representations to efficiently condition on long videos of hundreds of frames during both training and generation. We use a MaskGit prior for dynamics prediction which enables both sharper and faster generations compared prior work. Our experiments show that TECO outperforms SOTA baselines in a variety of video prediction benchmarks ranging from simple mazes DMLab, large 3D worlds in Minecraft, and complex real-world videos from Kinetics-600, In addition, to better understand the capabilities of video prediction models in modeling temporal consistency, we introduce several challenging video prediction tasks consisting of agents randomly traversing 3D scenes of varying difficulty. This presents challenging benchmark for video prediction in partially observable environments where model must understand what parts the scenes to re-create versus invent depending on its past observations generations. Generated videos available on the website: bttps://sitet google.
t= 36
299
Figure TECO generates sharp and consistent video predictions for hundreds of frames on challenging datasets. The figure being conditioned on 36 context frames. From top to bottom, the datasets are are DMLab, Minecraft, Habitat, and Kinetics-600.
shows evenly spaced frames of the 264 frame predictions, after

---

Under review as a conference paper at ICLR 2023
INTRODUCTION
Recent work in video prediction has seen tremendous progress (Ho et al., 2022; Clark et al., 2019; Yan et al., 2021; Le Moing et al., 2021; Ge et al., 2022; Tian et al., 2021; Luc et al., 2020) in producing high-fidelity and diverse samples complex video data. This can largely be attributed to a combination of increased computational resources and more compute efficient high-capacity neural architectures. However, much of this progress has focused on generating short videos, where models can perform well by basing their predictions on only a handful of previous frames.
Video prediction models with short context windows can generate long videos in sliding window fashion. While the resulting videos can look impressive at first sight, they lack temporal consistency. We would like models predict temporally consistent videos- where the same content is generated if camera pans back to a previously observed location. On the other hand, the model should imagine new part of the scene for locations that have not yet been observed, and future predictions should remain consistent to this newly imagined part of the scene.
Prior work has investigated techniques for modeling long-term dependencies, such as temporal hierarchies (Saxena et al., 2021) and strided sampling with frame-wise interpolation (Ge et al., 2022; Hong et al., 2022). Other methods train on sparse sets frames selected out of long videos (Harvey et al., 2022; Skorokhodov et al., 2021; Clark et al., 2019; Saito & Saito, 2018; Yu al., 2022), or model videos via compressed representations (Yan et al., 2021; Rakhimov et al., 2020; Le Moing et al., 2021; Seo et al., 2022; Gupta al., 2022; Walker et al., 2021). Refer to Appendix M for more detailed discussion on related work.
Despite this progress, many methods still have difficulty scaling to datasets with many long- range dependencies. While Clockwork VAE (Saxena et al., 2021) trains on long sequences, it is limited by training time (due recurrent architecture) and difficult scale to more complex data. On the other hand, transformer-based methods over latent spaces (Yan et al., 2021) scale poorly long videos due quadratic complexity in attention, with long videos containing tens of thousands of tokens. Methods that train on subsets of tokens are limited by truncated backpropagation through time (Hutchins et al., 2022; Rae et al., 2019; Dai et al., 2019) or naive temporal operations (Hawthorne et al., 2022).
In this paper, we introduce Temporally Consistent Video Transformer (TECO), a vector-quantized latent dynamics model that effectively models long-term dependencies in a compact representation space using efficient transformers. The key contributions are summarized as follows:
We introduce TECO, an efficient and scalable video prediction model that learns of compressed VQ-latents to allow for efficient training and generation.
We propose several long-length video prediction datasets centered around 3D scenes in DMLab (Beattic et al., 2016), Minecraft (Guss et al., 2019), and Habitat (Szot et al., 2021; Savva et al., 2019) to help better evaluate temporal consistency in video predictions.
We show that TECO has strong performance on variety of difficult video prediction tasks, and is able to leverage long-term temporal context generate high quality videos with consistency.
We provide several ablations providing intuition for why TECO is able to generate more temporally consistency predictions, and how these insights can extend future work in long- term video prediction.
2 PRELIMINARIES
2.1 VQ-GAN
VQ-GAN (Esser et al., 2021; Van Den Oord et al., 2017) is an autoencoder that learns to compress data into a set of discrete latents, consisting of an encoder E, decoder G, codebook C, and discriminator D. Given image # the encoder E maps z to its latent representation h # xw'>>>>> which is quantized by nearest neighbors lookup in codebook of embeddings C K to produce H >w'> The discretized latent is fed through decoder G to
2

---

Under review as a conference paper at ICLR 2023
reconstruct A ttraigh-throughh estimator (Bengio, 2013) is used to maintain gradient flow through the quantization step. The codebook optimizes the following loss:
Lvo sg(h) 11 + 3 h sg(e) (es(e) 11
(I)
where 0.25 hyperparameter, and the corresponding nearest-neighbors embedding from codebook C. For reconstruction, VQ-GAN replaces the original &g loss with perceptual loss (Zhang et al., 2012), LLPIPS. Finally, in order to encourage higher-fidelity samples, patch-level discriminator D is trained to classify between real and reconstructed images, with.
LGAN log D(x) log(1 D()))
(2)
Overall, VQ-GAN optimizes the following combination of losses:
min max LIPIPS + Lvv ALGAN
(3)
E,G.C
Vag CuPiPs lla is an adaptive weight, Gr. is the last decoder layer, å -10-6 and Va, Caax 1|, LLPIPS the exact distance metric described in Zhang al. (2012).
where À =
2.2 MASKGIT
MaskGit (Chang al., 2022) is generative model that models distributions tokens, such produced by a VQ-GAN. Instead of autoregressively modelling the sequence of tokens, MaskGit generates images with competitive sample quality at fraction of the sampling cost by using masked token prediction objective during training. Formally, we denote 2 E " as the discrete latent tokens representing an image. For each training step, we uniformly sample and randomly generate mask m E 1) with N = [HW masked values, where cos (1) Then, MaskGit learns to predict the masked tokens with the following objective
[0,
maskk EEED [logp(> | =m]]-
(4)
During inference, because MaskGit has been trained to model any set of unconditional and conditional probabilities, we can sample any subset of tokens per sampling iteration, from the extreme case of sampling all tokens (independent) to sampling one token a time (autoregressive). Chang et al. (2022) introduces a confidence-based sampling mechanism whereas other work (Lee al., 2022) proposes iterative sample-and-revise approaches.
3 TECO
Generating temporally consistent videos requires training on long videos to correctly learn long- term temporal dependencies between frames. However, computational and memory requirements remain the primary bottleneck in preventing from doing so. We present Temporally Consistent Video Transformer (TECO), video generation model that more efficiently scales training on longer horizon videos.
First, we train VQ-GAN to spatially compress our video data. Shown in prior work (Seo al., 2022), this is an important step for video prediction in a more efficient and scalable manner. However, even in latent space, existing methods are still limited to modeling short sequences of 16-24 frames, which can be attributed to the quadratics costs of transformer layers as sequence length grows. With 256 tokens per frame, 16 frame videos already consist of 4096 tokens, and scaling to longer videos of 100s frames is prohibitively expensive, where resulting videos have tens of thousands of tokens. Therefore, in the following sections, we propose several key design choices to building a more efficient video prediction model.
3.1 VECTOR- QUNTOR--UANTIEED LATENT DYNAMICS
Our proposed framework shown in Figure 2 follows similarly to prior work in latent dynamics models (Hafner et al., 2019; 2020; Saxena et al., 2021), with several key differences in architectural and latent variable design. Let æ1. consist of sequence of video frames encoded using pretrained
3

---

Under review as a conference paper at ICLR 2023
2/2
2
X
Mask Git
Mask Git
Mask Git
Dec
Dec
Dec
Z
Z
X
x L
h
h2
hg
2,
2
Zg
Enc
Enc
Enc
x
X2
(a) VQ-GAN
(b) TECO
Figure 2: The architectural design of TECO. Our proposed method models sequences of videos encoded with pretrained VQ-GAN. We achieve efficient and scalable training and generation on long sequences through several key design choices to maximally compress our representations. We leverage temporal redundancies by encoding frames conditioned on the previous one, and model temporal dependencies in downsampled latent space. For fast sampling, we learn a MaskGit dynamics for the prior.
VQ-GAN. In the following sections, motivate each component for model, with several specific design choices to ensure efficiency and scalability. TECO consists four components:
Encoder:
21 = =
Temporal Transformer: h4 H(ggt)
(5)
Dynamics Prior:
p(z h
Decoder:
p(æ, 24, 14-1)
Encoder Although VQ-GAN exploits spatial redundancies, we can achieve more compressed representations by leveraging temporal redundancy video data. To this, we learn CNN encoder 2 F((t,, which encodes the current frame 2t conditioned on the previous frame by channel-wise concatenating Z;-1, and then quantizes the output using codebook C' to produce 21. We apply the VQ loss defined in Equation (1) per timestep. In addition, we ez-normalize the codebook and embeddings to encourage higher codebook usage (Yu et al., 2021). Conditionally encoding nearby frames lets the model learn smaller latents, and provides general way to take advantage of temporal redundancy. The most common form of temporal redundancy is the large amount of shared bits between neighboring frames, generally only differing small movements, such as slight camera shifts, or objects moving slightly. The first frame is concatenated with zeros and does not quantize 2j to prevent information loss. As we focus on video prediction, there is un-quantized representation of the first frame when computing decoding and dynamics losses. Intuitively, this also does not burden the dynamics model to learn an unconditional prior.
always at least frame to condition on, so we do not need to predict the
Temporal Transformer Compressed, discrete latents are more lossy and tend to require higher spatial resolutions compared to continuous latents. Therefore, before modeling temporal information, we apply single strided convolution to downsample each discrete latent 2t, where visually simpler datasets allow for more downsampling and visually complex datasets require less downsampling. Afterwards, we learn large transformer to model temporal dependencies, and then apply transposed convolution upsample our representation back to the original resolution of 2, In summary, we use the following architecture:
h4 H(zzd) = Conv TonTTraspp
Transformer(onnn(za T
(6)
Decoder The decoder is upsampling CNN that reconstructs f D(2,, h4), where can be interpreted as the posterior of timestep f. and h, the output of the temporal transformer which summarizes information from previous timesteps. 2t and ht concatenated channel-wise before

---

Under review as a conference paper at ICLR 2023
Minecraft
Minecraft
LPIPS (4) over Timestep
LPIPS (4) vs Sampling Time
FitVid
0.50
LPIPS
0.4
0.45
CW-VAE
Perceiver AR
TECO (ours)
FitVid
CW-VAE
Latent FDM
Perceiver AR
0.35
TECO
Latent FDM
100
200
400
500
Timestep
Sampling Time Frame (s)
Figure 3: Quantitative comparisons between TECO and baseline methods in long-horizon temporal consistency (left) and sampling speed (right), Our method is able to remain temporally consistent while still generating sharp samples with fast sampling speed.
being fed into the decoder. Together with the encoder, the decoder optimizes the following cross entropy reconstruction loss
Lroon = - Ei= log p(z 2z, he).
which encourages features to encode relative information between frames since the temporal transformer can aggregate information over time. This allows us to learn more compressed codes that enable more efficient modeling over longer sequences.
Dynamics Prior Lastly, we use a MaskGit (Chang et al., 2022) to model the dynamics prior, p(z, ht). In our experiments, show that using MaskGit prior allows for not just faster but also higher quality sampling compared to an autoregressive prior. During every training iteration, we use the same process prior work to sample random mask m, and optimize
Spiir + Ein p(g 2e 2e ④ m.).
(8)
where he is concatenated channel-wise with masked to predict the masked tokens. During generation, we follow Lee et al. (2022), where we initially generate each frame in chunks of at time and then go through 2 revise rounds of re-generating half the tokens each time.
Training Objective The final objective is the sum of these losses:
LreCo Lvq Lrecon + Cpiior
3.2 DROPLOSS
To train the model efficiently on long videos, propose DropLoss, simple trick to allow for more scalable and efficient training. Due to its architecture design, TECO can separated into two components: (1) learning temporal representations, consisting of the encoder and the temporal transformer, and (2) predicting future frames, consisting of the dynamics prior and decoder. We can increase training efficiency by dropping
random timesteps that are decoded and thus omitted from the reconstruction loss. For example, given a video of frames, we compute h, for all t (I..... T}, and then compute the losses Cprior and Crecon for only 10% of the indices. Because random indices are selected each iteration. the model still needs to learn to accurately predict all timesteps. This reduces training costs significantly because the decoder and dynamics prior require non-trivial computations. DropLoss is applicable to both a wide class of architectures and to tasks beyond video prediction.
EXPERIMENTS
4.1 DATASETS
We introduce three challenging video datasets to better measure long-range consistency in video prediction. We design these benchmarks around 3D environments in DMLab (Beattie et al., 2016), Minecraft (Guss et al., 2019), and Habitat (Savva et al., 2019), with videos of agents randomly traversing different scenes of varying difficulty. These datasets require video prediction models to re-produce observed parts of scenes, and newly generate unobserved parts of the scene. In contrast,
5

---

Under review as a conference paper at ICLR 2023
Table 1: Quantitative evaluation all four datasets. Detailed results in Appendix K.
DMLab
Minecraft
Method
FVD 4
PSNR t
SSIM
LPIPS
FVD +
PSNR
SSIM
LPIPS |
FitVid
176
12.0
0.356
0.491
956
13.0
0.343
0.519
CW-VAE
125
12.6
0.372
0.465
397
13.4
0.338
0.441
Perceiver AR
96
11.2
0.304
0.487
76
13.2
0.323
0.441
Latent FDM
181
17.8
0.588
0.222
167
13.4
0.349
0.429
TECO (ours)
48
21.9
0.703
0.157
116
15.4
0.381
0.340
Habitat
Kinetics-600
Method
FVD 4
PSNR f
SSIM f
LPIPS 4
FVD 4
PSNR
SSIM f
LPIPS 4
Perceiver AR
164
12.8
0.405
0.676
1022
13.4
0.310
0.404
Latent FDM
433
12.5
0.311
0.582
960
13.2
0.334
0.413
TECO (ours)
73
12.8
0.363
0.604
799
13.8
0.341
0.381
many existing video benchmarks do not have strong long-range dependencies, where a model with limited context sufficient. Refer to Appendix N for further details on each dataset.
DMLab DeepMind Lab is simulator that procedurally generates random 3D mazes with random floor and wall textures. We generate 40k action-conditionel 64 64 videos of 300 frames of an agent randomly traversing x 7 mazes by choosing random points in the maze and navigating to them via the shortest path. We train all models for both action- oetion-conditiones and unconditional prediction (by periodically masking out actions) to enable both types of generations. We both modes to evaluate since a video model may generate new parts of scene that do not correlate with the action (e.g. run into wall) which results in out-of-distribution errors. However, action-conditioning is useful with enough conditioned past context, and substantially lowers variance on PSNR, SSIM, and LPIPS evaluations.
This popular game features procedurally generated 3D worlds that contain complex terrain such as hills, forests, rivers, and lakes. We collect 200k action-conditioned videos of length 300 and resolution 128 x 128 in Minecraft's marsh biome. The player iterates between walking forward for random number of steps and randomly rotating left or right, resulting in parts of the scene going out of view and coming back into view later. We train action-conditioned for all models for ease of interpreting and evaluating, though it is generally easy for video models to unconditionally learn these discrete actions.
Minecraft
Habitat Habitat is a simulator for rendering trajectories through seans of real 3D scenes. We compile 1400 indoor scans from HM3D (Ramakrishnan et al., 2021), MatterPort3D (Chang al., 2017), and Gibson (Xia et al., 2018) to generate 200k action-conme citionnnoniiiooeed videos of 300 frames at a resolution of 128 x 128 pixels. We use Habitat's 's in-built path traversal algorithm to construct action trajectories that move our agent between randomly sampled locations. Similar to DMLab, we train all video models to perform both unconditional and
prediction.
Kinetics-600 Kinetics-601 (Carreira & Zisserman, 2017) is a highly complex real-world video dataset, originally proposed for action recognition. The dataset contains 4400k videos of varying length of up to 300 frames. We evaluate our method in the video prediction without actions (as they do exist), generating 80 future frames conditioned 20. In addition, we filter videos shorter than 100 frames, leaving 392k videos that are split for training and evaluation. We use a resolution of 128 x 128 pixels. Although Kinetics-600 does have many long-range dependencies, we evaluate our method on this dataset to show that it can scale to complex, natural video.
4.2 BASELINES
We compare against SOTA baselines selected from several different families of models: latent- variable-based variational models, autoregressive likelihood models, and diffusion models. In addition, for more fair comparisons, as our method. For our diffusion baseline, we follow Rombach et al. (2022) and use pretrained VAE instead of VQ-GAN, Note that we do not have any GANs for baselines, since the best
train all models VQ codes using the same VQ-GAN

---

Under review as a conference paper at ICLR 2023
of our knowledge, there does not exist GAN that trains latent space instead of raw pixels, an important aspect for properly scaling to long video sequences.
FitVid FitVid (Babacizadeh al., 2021) is state-of-the-art variational video prediction model based on CNNs and LSTMs that scales to complex video by leveraging efficient architectural design choices in its encoder and decoder.
CW-VAE (Saxena et al., 2021) is also a variational video prediction model that is designed to better learn long-range dependencies through a hierarchies of latent variables with exponentially slower tick speeds for each new level.
Clockwork VAE
AR baseline over VQ- GAN discrete latents, which has been show to be an effective generative model that can efficiently incorporate long-range sequential dependencies. Conceptually, this baseline is similar HARP (Sco et al., 2022) with a Perceiver AR as the prior instead of a sparse transformer (Child et al., 2019). We choose Perceiver AR over other autoregressive baselines such VideoGPT (Yan et al., 2021) or TATS (Ge et al., 2022) primarily due to the prohibitive costs of transformers when applied to tens of thousands of tokens.
Perceiver AR We Perceiver AR (Hawthome al., 2022)
Latent FDM For our diffusion baseline, we train a Latent FDM model with frame-wiss autoregressive sampling. Although FDM (Harvey et al., 2022) is originally trained on pixel observations, we also train in latent space for a more fair comparison with our method and other baselines, as training on long sequences in pixel space is too expensive. We follow LDM (Rombach et al., 2022) to separately train autoencoder to encode each frame into a set of continuous latents.
4.3 EXPERIMENTAL. SETUP
Training All of our models are trained for 1 million iterations under fixed compute budget (measured in TPU v3 days) allocated for each dataset. Models are trained TPU-v3 instances, ranging from v3-8 to v3-128 TPU pods (similar to 4 V100s to 64 V100s) with training times of roughly 3-5 days. For DMLab, Minecraft, and Habitat we train all models videos, and 100 frames for Kinetics-600. Our VQ-GANs are trained on 8 A5000 GPUs, taking about 2-4 days for each dataset, and downsample all videos 16 x 16 grids of discrete latents per frame regardless of original video resolution. More details on exact hyperparameters and compute budgets for each dataset can be found in Appendix O.
full 300 frames
Evaluation We evaluate our models using a combination of standard video prediction metrics such as PSNR (Huynh-Thu Ghanbari, 2008), SSIM (Wang et al., 2004), LPIPS (Zhang et al., 2012), and FVD (Unterthiner et al., 2019). For DMLab, Minecraft, and Habitat, we measure FVD on 300 frame videos, conditioned on 36 frames (264 predicted frames). For Kinetics-600, we evaluate FVD 100 frame vidcos, conditioned on 20 frames (80 predicted frames). To evaluate temporal consistency, we measure PSNR, SSIM, and LPIPS on video predictions conditioned on 144 frames (156 predicted frames), and action condition for all models. Conditioning the video ensures that the model can observe a large part of the scene, and combined with action- conditioning, the model with femporally-consiserr to the ground truth. Due to this reduced stochasticity, we only sample one prediction for computing PSNR, SSIM, and LPIPS. We compute all metrics over batches of 256 examples, averaged over runs to make 1024 total samples.
large portion of
predictions should generate future frames close
4.4 BENCHMARK RESULTS
DMLab & Minecraft Table shows quantitative results on the DMLab and Minecraft datasets. TECO performs the best across all metrics for both datasets when training on the full 300 frame videos. Figure shows sample trajectories and 3D visualizations of the generated DMLab mazes, where TECO is able to generate more stable and consistent 3D mazes. For both datasets, CW-VAE, FitVid, and Perceiver AR can produce sharp predictions, but do not model horizon context well, with per-frame metrics sharply dropping the future prediction horizon increases as seen in Figure C. Latent FDM has consistent predictions, but high FVD most likely due to FVD being sensitive to high frequency errors.
7

---

Under review as a conference paper at ICLR 2023
Cond
Predictions
GT
TECO (ours)
Latent FDM
Perceiver AR
CI-VAE
FitVid
t=
143 144
299
Figure 4: 3D visualization of predicted trajectories in DMLab for each model, generating 156 frames conditioned on 144. TECO is the only model that retain maze consistency with ground- truth, whereas baselines tend to extend out of the maze create fictitious corridors that did not exist. Video predictions use only the first-person RGB frames. Refer to Appendix N.1 for more details on 3D evaluation. A video corresponding to this figure is available at: https: //sites, google. wwom/viwwwiclttte. teco.
In order to better investigate scaling properties of our models, Figure D.I and Figure D.2 compare TECO and Latent FDM on different training sequence lengths. Intuitively, under a fixed computation budget and batch size, models that train shorter sequence lengths scale larger, with more FLOPs allocated per frame. In general, this reflected in model architectures through computations higher spatial resolutions (e.g. less downsampling). For DMLab, frame metrics, models generally benefit from training on longer videos, where more computation per image has less of effect due to saturation image quality because of relatively simple visual complexity. For Minecraft, we observe that models generally perform best when training with 100 frames of context, which have better per-image sample quality compared to training due to higher downsampling required for longer sequences. Models trained on 300 frames generally have more distortion in predictions compared 100 frames. Theoretically. the compute budget is increased, training on 300 frames would eventually outperform models trained on 100 frames.
that in terms of per-
300 frames
Habitat Table shows results for our Habitat dataset. We only evaluate our strongest baselines, Perceiver AR and Latent FDM due to the need to implement model parallelism. Because of high complexity Habitat videos, all models generally perform equally as bad in per-frame metrics. However, TECO has significantly better FVD. Qualitatively, Latent FDM quickly collapses to blurred predictions with poor sample quality, and Perceiver AR can generate high quality frames, though less temporally consistent than TECO: agents in Habitat videos navigate far points in the scene and back whereas Perceiver AR tends to generate samples where the agent is constantly turning. TECO generates traversals of scene that match the data distribution closely.
8

---

Under review as a conference paper at ICLR 2023
Kinetics-600 Table 1 shows FVD for predicting 80 128 128 frames conditioned on 20 for Kinetics-600. Although Kinetics-600 does not have many long-range dependencies, we found that TECO is able to produce more stable generations that degrade slower by incorporating longer contexts. In contrast, Perceiver AR tends to degrade quickly, with Latent FDM performing in between. Figure K.1 and Table K.4 include further investigations using top-k sampling for Perceiver AR and TECO. Table does not use top-k sampling for fair comparison against Latent FDM. With top-k sampling, Perceiver AR outperforms our method at l 8. However, resulting videos tend to be uninteresting with little to no dynamics movement.
Sampling Speed Figure 3 compares sampling speed for all models. We report sampling speed on Minecraft and observed similar results for the different model sizes used on other datasets. Fit Vid and CW-VAE are both significantly faster that other methods, but have poor sample quality. On the other end, Perceiver AR and Latent FDM can produce high quality samples, but are 20-60x slower than TECO, which has comparably fast sampling speed while retaining high sample quality.
4.5 ABLATIONS
In this section, we perform ablations on various architectural decisions of our model. For simplicity, we evaluate our methods short sequences of 16 frames from Sonething-Sometting-ve (SSv2). We choose SSv2 as it provides insight into scaling our method on complex real-world data more similar Kinetics-600 while being computationally cheaper run.
Table F.1 shows several ablations comparing posterior, prior, and various architectural design choices. We demonstrate that using VQ-latent dynamics with a MaskGit prior proves better compared to alternative formulations for latent dynamics models, such as popular variational methods. addition, we show that conditional encodings learn better representations for video predictions. We also ablate the codebook size, showing that although there exists optimal codebook size, it does not matter too much as along as there are not too many codes, which may make it more difficult for the prior to learn. Lastly, 60% faster training and a minimal increase in FVD. The benefits are greater for longer sequences, and allow video models to better account for long horizon context with little cost in performance.
show the benefits of DropLoss, with up to
Table F.2 shows ablations on scaling different parts of our model, such as the encoder, decoder, temporal transformer, and prior. In general, it is more beneficial to have imbalanced encoder decoder architecture, with more parameters in the decoder. For the temporal transformer, it is more beneficial have larger resolution features (4 x 4), especially for more complex data like SSv2, and less useful for visually simpler datasets such as DMLab or Minecraft. Similarly, having larger width more beneficial than more layers due to increased capacity to represent each frame. Lastly, for scaling the MaskGit prior, more layers is better than larger width networks.
5 DISCUSSION
We introduced TECO, an efficient video prediction model that leverages hundreds of frames of temporal context. Our evaluation demonstrated that TECO accurately incorporates long-range context, outperforming SOTA baselines across wide range of datasets. In addition, several difficult video datasets, which we hope make it easier to evaluate temporal consistency in future video prediction models. We identify several limitations as directions for future work:
introduce
Although we show that PSNR, SSIM, and LPIPS can be reliable metrics to measure consistency when video models are properly conditioned, there remains room for better evaluation metrics that provide reliable signal as the prediction horizon grows, since new parts of scene that generated are unlikely to correlate with ground truth.
Our focus was on learning compressed tokens and an expressive prior, which we combined with a simple full attention transformer as the sequence model. Leveraging prior work on efficient sequence models (Choromanski et al., 2020; Wang et al., 2020; Zhai et al., 2021: Gu al., 2021; Hawthorne et al., 2022) would likely allow for further scaling-
We trained all models top of pretrained VQ-GAN codes reduce the data dimensionality. This compression step lets us train on longer sequences at a cost of reconstruction error, which causes noticeable artifacts in Kinetics-6000 such as corrupted text and incoherent faces. Although TECO can train directly on pixels, a & loss results in slightly blurry predictions. Training directly pixels with diffusion GAN losses would be promising.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY
We provide several resources in order to aim for better reproducilbity. We include anonymized code in the supplementary materials for our models, baselines, and datasets. In addition, Appendix O details hyperparameters and compute requirements for all models.
REFERENCES
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv 2017.
Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint
1eXiv:2106.13195 2021.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprint Vr11v::612.03801, 2016.
Yoshua Bengio. Estimating or propagating gradients through stochastic neurons. arXiv preprint arXiv:1305.2968 2013.
Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chun Wang, Timo Aila, Jaakko Lehtinen, Ming-Yu Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. arXiv preprint (xXiv:2206.03429, 2022.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Angel Chang, Angela Dai. Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint 1oXiv:1709.0610 2017.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315- 1125,, 2022.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint
2019.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009.1473 2020.
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2019.
Zihang Dai. Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 2019.
Emily Denton and Rob Fergus. Stochastic video generation with learned prior. arXiv preprint arXiv:1802 180..07677 2018.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (0006699):1204112110 2018
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873- 12883, 2021.
10

---

Under review as a conference paper at ICLR 2023
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with ameeaggnstic vqgan and time-sensitive transformer. arXiv preprint arXiv:2204.0366 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672 26800, 2014.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 1894, 2022.
William Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: large-scale dataset of minecraft demonstrations. arXiv preprint arXiv: 1907. 13440, 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learming, pp. 2555-2565. PMLR, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020.
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXiv: 1495, 2022.
Curtis Hawthorne, Andrew Jaegle, Cätälina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Malinowski, Sander Dieleman, Oriol Vinyals, Matthew Borvinick, Ian Simon, et al. General-purpose, long-context autoregressive modeling with perceiver ar. arXiv preprint arXiv:2202.07765, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006. 11399, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv.220..03458, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205 2022.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprini arXiv:2206.0766 2022.
DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers. arXiv preprint arXiv:2203.07855. 2022.
Quan Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. Electronics letters, 44(13):800-80) 2008
Nal Kalchbrenner, Aäron Oord, Karen Simonyan, Ivo Danihelka, Oriol Vinyals. Alex Graves, and Koray Kavukcuoglu. Video pixel networks. In International Conference on Machine Learning, pp. 1771-1779. PMLR, 2017.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, 2013.
Manoj Kumar, Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, and Durk Kingma. Videoflow: A flow-based generative model for video. arXiv preprint 1003.0/903.01333 2(5):3, 2019.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis. Advances in Neural Information Processing Systems, 34, 2021.
11

---

Under review as a conference paper at ICLR 2023
Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv preprint arXiv:1804.01523, 2018.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Draft-and-revise: Effective image generation with contextual rq-transformer. arXiv preprint SiV:22006004552, 2022.
Wonkwang Lee, Whie Jung, Han Zhang, Ting Chen, Jing Yu Koh, Thomas Huang, Hyungsuk Yoon, Honglak Lee, and Seunghoon Hong. Revisiting hicrarchical approach for persistent long-term video prediction. In International Conference Learning Representations, 2021. URL https: / //openreview.. httae//ooouumnddd.LLLALM...
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformation based adversarial video prediction on large-scale data. arXiv preprint 1tti::00030040355, 2020.
Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. arXiv e-prints, pp. arXiv-2204, 2022.
Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint urXiv:1911.0550) 2019.
Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprini arXiv:2006. 10704, 2020.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westhury, Angel X Chang, al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. arXiv preprint 1iX.::22099002388. 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Masaki Saito and Shunta Saito. Tganv2: Efficient training of large models for video generation with 2018.
multiple subsampling layers. arXiv preprint
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
Vaibhav Saxena, Jimmy Ba, and Danijar Hafner. Clockwork variational autoencoders. Advances in Neural Information Processing Systems, 34:29246-29257, 2021.
Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, and Pieter Abbeel. Harp: Autoregressive latent video prediction with high-fidelity image generator. International Conference on Image Processing, 2022.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. arXiv preprint 12, 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Modeling arbitrary probability distributions using nonequilibrium thermodynamics. Unpublished Manuscript, 2014.
Andrew Szot, Alex Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In Advances in Neural Information Processing Systems (NeurlPS). 2021.
12

---

Under review as a conference paper at ICLR 2023
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng. Dimitris N Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis. arXiv preprint arXiv:2104.15699, 2021.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1526-1535, 2018.
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Lc, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. Advances Neural Information Processing Systems, 32:81-91, 2019.
Vikram Voleti, Alexia for prediction, generation, and interpolation. arXiv preprint arXiv:2205.098: 2022.
and Christopher Pal. Masked conditional video diffusion
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.01950, 2021.
Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04766. 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. arXiv preprint /rXiv:1906.022 2019.
Chao-Yure Wu, Yanghao Li. Karttikeya Mangalam. Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augnnn augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13587-13597, 2022.
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9068-9079, 2018.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2 10157, 2021.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector Vector-quantized image modeling with improved vggan. arXiv preprint 1 10.04627, 2021.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Wae Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint arXiv:2202.10571, 2022.
Shuangfei Zhai, Walter Talbott, Nitish Srivastava, Chen Huang, Hanlin Goh, Ruixiang Zhang, and Josh Susskind. An attention free transformer. arXiv preprint arXi::2105.14103 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2012.
13

---

Under review as a conference paper at ICLR 2023
A SAMPLING PROCESS
Given sequence of conditioning frames, 01,-.., Ot, we encode each frame using the pretrained VQ- GAN produce ,1,..., It, and then the conditional encoder compute 21y..., 2z. In order to generate the next frame, we use the temporal transformer to compute he, and feed it into the MaskGit dynamics prior to predict 21+1- Let 2t+1 ê and feed it through the temporal tranformer and MaskGit to predict 4t+2. We repeat this process until the entire trajectory is predicted, 24+1..... &y. In order decode back into frames, we first decode into the VQ-GAN latents, and then decode to RGB using the VQ-GAN decoder. Note that generation can be completely done in latent space, and rendering back RGB can be done in parallel over time once the latents for all timesteps are computed.
14

---

Under review as a conference paper at ICLR 2023
B SAMPLES
B.1 DMLAB
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
CW-VAE
FitVid
73
144
163
182
202
221
240
260
279
299
Figure B.1: 156 frames generated conditioned on 144 (action-cooditioonel)
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
CW-VAE
FitVid
1= 0
17
35
36
68
101
134
167
200
233
266
299
Figure B.2: 264 frames generated conditioned on 36 (no setion-condittonnnn))
15

---

Under review as a conference paper at ICLR 2023
TECO (ours)
Latent FDM
Perceiver-AR
CW-VAE
FitVid
Figure B.3: 3D visualizations of the resulting generated DMLab mazes
16

---

Under review as a conference paper at ICLR 2023
B.2 MINECRAFT
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Percelver-AR
CW-VAE
FitVid
1=
73
143
144
163
202
221
240
260
279
299
Figure B.4: 156 frames generated conditioned on 144 (action-comditioone))
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
CW-VAE
FitVid
0
17
101
134
167
200
233
268
299
Figure B.5: 264 frames generated conditioned on 36 (action-conditooned)
17

---

Under review as a conference paper at ICLR 2023
B.3 HABITAT
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
t= 0
73
143
144
163
182
202
221
240
280
279
299
Figure B.6: 156 frames generated conditioned on 144 (action-conditioned)
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
t=
68
134
167
266
299
Figure B.7: 264 frames generated conditioned on 36 (no action-condittoningg
18

---

Under review as a conference paper at ICLR 2023
B.4 KINETICS-600
Cond
Predicted Frames
TECO (ours)
Latent FDM
Perceiver-AR
t= 0
9
19
20
29
39
49
59
69
79
89
Figure B.8: 80 frames generated conditioned on (no top-k sampling)
Cond
Predicted Frames
GT
TECO (ours)
Latent FDM
Perceiver-AR
29
39
49
79
89
99
Figure B.9: 80 frames generated conditioned on 20 (with top-k sampling)
19

---

Under review as a conference paper at ICLR 2023
C
PERFORMANCE VERSUS HORIZON
PSNR Over Time (1)
SSIM Over Time (1)
LPIPS Over Time (4)
30
0.6
0.0
PSN
20
0.6
0.4
0.2
0.0
50
100
150
50
100
150
50
100
150
Tirwstap
Tirestap
Tinwslop
TECO
Filvid
(a) DMLah
PSNR Time
SSIM Over Time
LPIPS Time
22.5
0.5
0.5
17.5
0.4
12.5
10.0
Timestap
Timestap
Perceiver AR
TECO lours)
Latent FDM
CWVVAE
Fivid
(b) Minecraft
PSNR Over Time (1)
SSIM Over Time (1)
LPIPS Over Time (4)
22.5
20.0
15.0
0.4
12.5
50
100
150
50
100
150
50
100
150
Timestop
Timestop
Timestop
TECO
FDM
Percehrer
(c) Habitat
Figure C.I: All plots shows PSNR, SSIM, and LPIPS 150 predicted frames conditioned on 144 frames. The 144 conditioned frames are not shown on the graphs and timestep 0 corresponds to the first predicted frame
Figure C.1 shows PSNR, SSIM, and LPIPS as a function of prediction horizon for each dataset. Generally, each plot reflected the corresponding aggregated metrics in Table 1. For DMLab, TECO shows much better temporal consistency for the full trajectory, with Latent FDM coming in second. CW-VAE is able retain some consistency but drops fairly quickly. Lastly, FitVid and Perceiver AR lose consistency very quickly. We see similar trend in Minecraft, with Latent FDM coming closer matching TECO, For Habitat, all methods generally have trouble producing consistent predictions, primarily due to the difficulty of the environment.
20

---

Under review as a conference paper at ICLR 2023
D PERFORMANCE VERSUS TRAINING SEQUENCE LENGTH
FVD (4) Train Sequence Length
LPIPS (4) Train Sequence Length
0.5
TECO (ours)
TECO (aurs) Latent FDM Perceiver-AR CW-VAE FitVid
Latent FDM
Perceiver-AR CW-VAE Fitidd
0.4
FVD
LPIPS
0.3
100
100
150
200
300
100
150
200
250
300
Train Sequence Length (Fewer FLOPs Per Frame)
Train Sequence Length (Fewer FLOPs Per Frame)
PSNR (T) vs Train Sequence Length
SSIM (1) vs Train Sequence Length
22.5
(ours)
0.7
20.0
Latent FDM
Perceiver-AR
0.6
17.5
CW-VAE
SSIM
TECO (ours) Latent FDM Perceiver-AR CW-VAE FitVid
Rividd
0.5
15.0
0.4
12.5
0.3
50
100
150
200
250
300
50
100
150
200
250
300
Train Sequence Length
Train Sequence Length
(Fewer FLOPs Per Frame)
(Fewer FLOPs Per frame)
Figure D.1: DMLab
FVD (4) vs Train Sequence Length
LPIPS (4) vs Train Sequence Length
1000
TECO (ours)
0.50
TECO (ours)
800
Latent
Latent FDM
Percelver-AR
Perceiver-AR
CW-VAE
0.45
CW-VAE
FD
FitVid
400
200
0.35
100
150
200
250
300
150
200
250
Train Sequence Length (Fewer FLOPs Per Frame)
Train Sequence Length (Fewer FLOPs Frame)
PSNR (t) vs Train Sequence Length
SSIM (t) vs Train Sequence Length
TECO (ours)
SSIM
TECO (ours) Latent FDM
Latent FDM
Perceiver-AR
Perceiver-AR
CW-VAE
CW-VAE
FitVid
FivVd
0.32
50
100
150
200
250
300
50
100
150
200
250
300
Train Sequence Length (Fewer FLOPs Per Frame)
Train Sequence Length (Fewer FLOPs Per Frame)
Figure D.2: Minecraft
21

---

Under review as a conference paper at ICLR 2023
Figure D.1 and Figure D.2 show plots comparing performance with training models different sequence lengths. Under a fixed compute budget and batch size, training on shorter videos enables us to scale to larger models. This can also be interpreted model capacity or FLOPs allocated per image. In general, training on shorter videos enables higher quality frames (per-image) but at of worse temporal consistency due reduced context length. We in DMLab, in that TECO is able to better scale on longer sequences, and correspondingly benefits from it. Latent FDM has trouble when training on full sequences. We hypothesize that this may be due to diffusion models being less amenable towards downsamples, it it needs to model and predict noise. Minecraft, we see the best performance at around 50-100 training frames, where model has higher fidelity image predictions, and also has sufficient context.
very clear trend
E SAMPLING
FVD (4) vs Sampling Time
Minecraft
1000
Fitvid
LPIPS (4) vs Sampling Time
Fitvid
800
0.50
FV
LPIPS
0.45
CW-VAE
Perceiver AR
400
CW-VAE
0.40
200
Perceiver AR
0.35
Latent FDM
TECO
Latent FDM
TECO
4
8
Sampling Time Frame (5)
Sampling Time Frame (s)
Sampling Time per Frame (ms)
TECO (ours)
186
Latent FDM
3606
Perceiver-AR
8443
CW-VAE
0.062
FitVid
0.074
F ABLATIONS
DropLoss Rate
FVD
Train Step (ms)
Posteriors
FVD
0.8
187
125
VQ (+ MaskGit prior) (ours)
189
0.6
186
143
OneHot (+ MaskGit prior)
199
0.4
184
155
OneHot (+ Block AR prior)
209
0.2
184
167
OneHot (+ Independent prior)
228
0.0
182
182
Argmax (+ MaskGit prior)
336
(a) DropLoss Rates
(b) Posteriors
Number Codes
FVD
Dynamics Prior
FVD
Conditional Encoding
FVD
64
191
MaskGit (ours)
189
Yes (ours)
189
256
195
Independent
220
No
208
1024
186
Autoregressive
207
4096
200
(d) Conditional Encoding
(c) Prior Networks
(e) VQ Codebook Size
Table F.1: Ablations comparing alternative prior, posterior, and codebook designs
22

---

Under review as a conference paper at ICLR 2023
FVD
FVD
FVD
Size
2 2
4
4
Layers
Width
2 2
4
Layers
Width
2 x 2
4 x
4
Base
204
189
8
768
204
189
768
204
189
Small Enc
214
191
8
384
260
196
384
228
193
Small Dec
232
198
2
768
216
202
768
228
201
(a) Encoder and Decoder
(b) Temporal Transformer
(c) MaskGit Prior
Table F.2: Ablations on scaling different parts of TECO.
FVD (1)
PSNR (t)
SSIM (f)
LPIPS ())
Train Step Time (ms)
TECO (ours)
48
21.9
0.703
0.157
151
MaskGit
950
19.3
0.605
0.274
167
Autoregressive
44
20.1
0.640
0.197
267
Table F3: DMLab dataset comparisons against similar model as TECO without latent dynamics, and Maskgit AR model VQ-GAN tokens directly.
Table F.3 shows comparisons between TECO and alternative architectures that not use latent dynamics. Architecturally, MaskGit and Autoregressive are very similar to TECO, with few small changes: (1) there is no CNN decoder and (2) MaskGit and AR directly predict the VQ-GAN latents (as opposed to the learned VQ latents in TECO). In terms of training time, MaskGit and AR are little slower since they operate on 16 x 16 latents instead of x 8 latents for TECO. In addition, conditioning for the AR model is done using cross attention,
channel-wise concatenation does not work well due to unidirectioal masking. Both models without latent dynamics have worse temporal consistency,
well overall sample quality. We hypothesize that TECO has better temporal consistency due to weak bottlenecking of latent representation, as lot of time can be / video statistics. MaskGit shows very high FVD
spent modeling likelihood of imperceptible image due to a tendency to collapse in later frames of prediction, which FVD is sensitive to.
23

---

Under review as a conference paper at ICLR 2023
G
METRICS DURING TRAINING
during training
LPIPS during training
0.35
0.30
50
LPIPS
45
40
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
10
Training Iteration
1e6
Training Iteration
1e6
Figure G.I: Comparing FVD and LPIPS evaluation metrics over the course of training. FVD tends to saturate carlier (200k) while LPIPS keeps on improving up until 1M iterations.
Figure G.1 shows plots of FVD (over chunks of generatd 16 frame video) and LPIPS during training, evaluated at saved model checkpoints every 50k iterations over 1M iterations. We can see that although FVD (measuring frame fidelity) tends saturate early during training (at around 200k iterations), the long-term consistency metric (LPIPS) continues to improve until the end of training. We hypothesize that this may be due the model first learning the "casier bits" more local in time, and then learning Long-horizer bits once the easier bits have been learned.
H PROGRESSION FROM EXISTING WORK
Time per Training Step (s)
Model
Architecture
FVDL
PSNR1
SSIM
LPIPS
VideoGPT
3D VQ-VAE
0.881
156
11.1
0.296
0.468
TATS
Autoregressive (time + space)
3D VQ-VAE
Phenaki
0.905
725
11.0
0.202
0.474
MaskGit (time + space)
2D VQ-VAE
TECO w/o
CNN encoder
0.169
950
19.3
0.605
0.274
latent dynamics
Autoregressive (time)
MaskGit (space)
2D VQ-VAE
CNN encoder
TECO (ours)
Autoregressive (time)
0.131
48
21.9
0.703
0.157
MaskGit (latent)
CNN decoder (space)
Table H.l: We iteratively apply architectural modifications starting from existing work up to TECO
Table H.1 shows the progressive improvement from existing work (TATS, Phenaki) and how TECO able scale far better on all metrics with proposed architectural improvements.
24

---

Under review as a conference paper at ICLR 2023
1
HIGH QUALITY SPATIO-TEMPORAL COMPRESSION
Model
Dataset
FVD
DMLab
54
TATS
Minecraft
226
DMLab
TECO
Minecraft
53
Table L1: Reconstruction FVD comparing TATS Video VQGAN to TECO
Table 1.1 compares reconstruction FVD between TECO and TATS. At the same compression rate (same number of discrete codes). TECO learns far better spatio-temporal codes that TATS, with more of a different on more visually complex scenes (Minecraft vs DMLab).
J TRADE-OFF BETWEEN FIDELITY AND LEARNING LONG-RANGE DEPENDENCIES
Downsample Resolution
FVD.
PSNR
SSIM
LPIPS
1 X 1
44
20.4
0.666
0.170
2 x 2
18.6
0.597
0.221
4 4 4
33
17.7
0.578
0.242
Table J.1: Comparing different input resolutions to the temporal transformer
Latent FDM Arch
FVDJ
PSNR
SSIM
LPIPS.
More downsampling + lower resolution computations
181
17.8
0.588
0.222
Less downsample + higher resolution computations
94
15.6
0.501
0.277
Table J.2: Comparing different Latent FDM architectures with more computation at different resolutions
Table J.1 and Table J.2 show a trade-off between fidelity (frame or image quality) and temporal consistency (long-range dependencies) for video prediction architectures (both TECO, and Latent FDM).
25

---

Under review as a conference paper at ICLR 2023
K
FULL EXPERIMENTAL RESULTS
TPU-v3 Days
Params
FVD 4
PSNR f
SSIM
LPIPS
TECO (ours)
32
169M
27.5 1.77
22.4 + 0.368
0.709 0.0119
0.155 + 0.00958
Latent FDM
32
31M
= 1
17.8 = 0.111
0.588 0.00433
0.222 + 0.00493
Perceiver-AR
32
30M
96.3
11.2 0.00381
0.487 + 0.00123
CW-VAE
32
111M
125 + 7.95 7.95
12.6 +:6::0000 0.0585
0.372 + 0.000330
0.465 + 0.00156
FitVid
32
165M
176 + 4.86
12.0 +: 0.0126
0.356 + 0.00171
0.491 + 0.00108
Table K.1: DMLab
TPU-v3 Days
Params
FVD 4
PSNR
SSIM 1
LPIPS 1
TECO (ours)
80
274M
116 5.08
15.4 + 0.0603
0.381 + 0.00192
0.340 0.00264
Latent FDM
80
33M
167 6.26
13.4 + 0.0904
0.349 0.00337
0.429 + 0.00284
Perceiver-AR
80
166M
76.3
13.2 - 0.0711
0.323 0.00336
0.441 0.00207
CW-VAE
80
140M
397 = 15.5
13.4 0.0610
0.338 = 0.00274
0.441 = 0.00367
FitVid
80
176M
956 + 15.8
13.0 0.00895
0.343 0.00380
0.519 0.00367
Table K.2: Minecraft
TPU-v3 Days
Params
FVD L
PSNR
SSIM t
LPIPS 1
ТECO (ours)
275
386M
76.3 + 1.72
12.8 + 0.0139
0.363 + 0.00122
0.604 + 0.00451
Latent FDM
275
87M
433 + 2.67
12.5 1 0.0121
0.311 + 0.000829
0.582 0.000492
Perceiver-AR
275
200M
164 + 12.6
12.8 + 0.0423
0.405 + 0.00248
0.676 + 0.00282
Table K.3: Habitat
TPU-v3 Days
Params
FVD
TPU-v3 Days
Params
FVD 1
TECO (ours)
640
1.09B
649 16.5
TECO (ours)
640
1.09B
799 + 23.4
Latent FDM
640
831M
960 + 52.7
Latent FDM
640
831M
960 + 52.7
Perceiver-AR
640
1.06B
607 6.98
Perceiver-AR
640
1.06B
1022 = 32.4
(a) Using top-k sampling for Perceiver AR and TECO
(b) No top-k sampling
Table K.4: Kinetics
FVD (4) vs Top
1000
900
TECO (ours) Perceiver AR
PVD
800
Latent FDM
700
600
2
32
128
2048
8192
Top
Figure K.1: FVD on Kinetics-600 with different top-k values for Perceiver-AR and TECO
26

---

Under review as a conference paper at ICLR 2023
L
SCALING RESULTS
TPU-v3
Train Seq Len
Params
FVD
PSNR t
SSIM
LPIPS +
Days
300
169M
48.2 2.02
21.9 = 0.368
0.703 + 0.0114
0.157 =
200
169M
59.7 2.29
19.9 0.186
0.628 = 0.00821
0.187
TECO (ours)
32
100
86M
63.9 + 7.84
15.4 0.199
0.476 + 0.00745
0.322 0.00992
50
195M
52.7 + 6.23
13.9 0.0311
0.418 + 0.000659
0.383 0.000302
300
31M
181 2.20
1
0.588 + 0.00453
0.222 + 0.00493
200
62M
66.4
1717
0.561 = 0.00623
0.253 0.00550
Latent FDM
32
100
80M
55.6 1.36
15.5 0.233
0.468 = 0.00776
0.336 0.00051
50
110M
68.3 3.19
14.0 + 0.0445
0.414 0.424
0.385 0.00151
Table L.1: DM Lab scaling
TPU-v3
Train Seq Len
Params
FVD
PSNR 1
SSIM A
LPIPS
Days
300
274M
116 5.08
15.4 + 0.0603
0.381 + 0.00192
0.340 + 0.00264
200
261M
109.5 1.46 1.46
15.4 = 0.0906
0.379 + 0.00263
0.343 + 0.00148
TECO (ours)
80
100
257M
85.1 = 4.09
15.7 0.0516
0.385 + 0.00244
0.325 + 0.00121
50
140M
80.7 = 1.42
14.8 = 0.0404
0.369 + 0.00197
0.360 + 0.00133
300
33M
167 = 6.26
13.4 + 0.0904
0.349 + 0.00327
0.429 + 0.00284
200
80M
1
15.0 = 0.0701
0.384 1 0.00320
0.366 0.00311
Latent FDM
80
100
69M
4.40
= 4.40
15.1 = 0.0866
0.390 = 0.00281
0.358 - 0.00250
50
186M
85.6 = 2.25
14.8 = 0.0578
0.378 = 0.00144
0.372 = 0.000966
Table L.2: Minecraft scaling
27

---

Under review as a conference paper at ICLR 2023
M RELATED WORK
Video Generation Prior video generation methods can be divided into a few classes of models: variational models, exact likelihood models, and GANs. SV2P (Babaeizadeh al., 2017), SVP (Denton & Fergus, 2018), SVG (Villegas et al., 2019), and FitVid Babaeizadeh et al. (2021) are variational video generation methods models videos through stochastic latent dynamics, optimized using the ELBO (Kingma& & Welling, 2013) objective extended in time. SAVP (Lee et al., 2018) adds an adversarial (Goodfellow et al., 2014) loss to encourage more realistic and high-fidelity generation quality. Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2014) have recently emerged as powerful class of variational generative models which learn to iteratively denoise an initial noise sample to generate high-quality images. There have been several recent works that extend diffusion models to video, through temporal attention (Ho et al., 2022; Harvey et al., 2022), 3D convolutions (Höppe al., 2022), or channel stacking (Voleti et al., 2022). Unlike variational models, autoregressive models (AR) and flows (Kumar et al., 2019) model videos by optimizing exact likelihood. Video Pixel Networks (Kalchbrenner et al., 2017) and Subscale Video Transformers (Weissenborn et al., 2019) autoregressively model each pixel. For more compute efficient training, some prior methods (Yan et al., 2021; Le Moing et al., 2021; Seo et al., 2022; Rakhimov et al., 2020; Walker et al., 2021) propose to learn an AR model in a spatio-temporalyy compressed latent space of a discrete autoencoder, which has shown to be orders of magniudess more efficient compared to pixel-based methods. Instead of a VQ-GAN, Le Moing et al. (2021),
learns a frame conditional autoencoder through flow mechanism. Lastly, GANs (Goodfellow et al., 2014) offer alternative method to training video models. MoCoGAN (Tulyakov et al., 2018) generates videos by disentangling style and motion. MoCoGAN-HD (Tian et al., 2021) can efficiently extend to larger resolutions by learning to navigate the latent space of pretrained image generator. TGAN22 (Saito & Saito, 2018), DVD-GAN (Clark et al., 2019), StyleGAN- V (Skorokhodov et al., 2021), and THIVD-GAN (Luc et al., 2020) introduce various methods to scale to complex video, such as proposing sparse training, or more efficient discriminator design.
The main focus of this work lies with video prediction, specific interpretation of conditional video generation. Most prior methods are trained autoregressive in time, so they can be easily extended video prediction. Video Diffusion, although trained unconditionally proposes reconstruction guidance for prediction. GANs generally require training a separate model for video prediction. However, some methods such as MoCoGAN-HD and DI-GAN can approximate frame conditioning by inverting the generator to compute corresponding latent for a frame.
Long-Horizon Video Generation CW-VAE (Saxena al., 2021) learns a hierarchy of stochastic latents to better model long term temporal dynamics, and is able to generate videos with long-term consistency for hundreds of frames. TATS (Ge et al., 2022) extends VideoGPT which allows for sampling of arbitrarily long videos using sliding window. In addition, TATs and Cog Video (Hong et al., 2022) propose strided sampling as a simple method to incorporate longer horizon contexts. StyleGAN-V (Skorokhodov et al., 2021) and DI-GAN (Yu et al., 2022) learn continuous-time representations for videos which allow for sampling of arbitrary long videos as well. Brooks et al. (2022) proposes efficient video GAN architecture that is able to generate high resolution videos of 128 frames on complex video data for dynamic scenes and horseback riding. FDM (Harvey et al., 2022) proposes diffusion model that is trained to be able flexibly condition a wide range of sampled frames to better incorporate context of arbitrarily long videos. Lee et al. (2021) is able to leverage hierarchical prediction framework using semantic segmentations to generate long videos.
Long-Horizon Video Understanding Outside of generative modeling, prior work such as McMViT (Wu et al., 2022) and Vis4mer (Mohaiminul Islam & Bertasius, 2022) introduce architectures for modeling long-horizon dependencies in videos.
28

---

Under review as a conference paper at ICLR 2023
N DATASET DETAILS
N.1 DMLAB
We generate random 7 mazes split into four quadrants, with each quadrant containing a random combination of wall and floor textures. We generate 40k trajectories of 300 frames, each 64 x 64 images. Actions in this environment consist of 20 left turn, 20 right turn, and walk forward. In order to maximally traverse the maze, we code agent that traverses the furthest unvisited point in the maze, with some added noise for stochasticity. Since the maze grid, we can easily hard-code navigation policy to move to any specified point in the maze.
For 3D visualizations, we also collect depth, camera intrinsics and camera extrinsics (pose) for each timestep. Given this information, reconstruct the maze as a 3D pointcloud. Note that since videos are generated only using RGB as input, they do not have groundtruth depth and pose. Therefore, that are used during evaluation. Specifically, we train a depth estimator to map from RGB frame to depth, and pose estimator that takes in adjacent RGB frames and predicts the relative change in orientation. During evaluation, we are given an initial ground truth orientation that we apply sequentially to predicted frames.
project RGB points into 3D coordinate space and
train depth and pose estimators
Although the GQN Mazes (Eslami et al., 2018) already exists as a video prediction dataset, it is difficult properly measure temporal consistency. The 3D scenes are relatively simple, and it does not have actions to help reduce stochasticity in using metrics such as PSNR, SSIM, and LPIPS. As a result, FVD is the reliable metric used in GQN Mazes, but tends be sensitive to noise in video predictions. In addition, we perform 3D visualizations using our dataset that are not possible with GQN Mazes.
N.2 MINECRAFT
We generate 200k trajectories (each of different Minecraft world) of 300 128 x 128 frames in the Minecraft marsh biome. We hardcode an agent to randomly traverse the surroundings by taking left, right, and forward actions with different probabilities. In addition, we let the agent constantly jump, which we found to help traverse simple hills, and prevent itself from drowning. We specifically chose the marsh biome, for consistent generation. Forest and jungle biomes tend to be too dense for any meanginfully clear consistency, as all surroundings look nearly identical. On the other hand, plains biomes had the opposite issue where the surroundings were completely flat. Mountain biomes were too hilly and difficult to traverse.
contains hilly turns with sparse collections of trees that act as clear landmarks
We opt to introduce alternative to the MineRL Navigate (Guss et al., 2019) since this dataset primarily consists human demonstrations of people navigating to specific points. This means that trajectories usually follow relatively straight line, there are not many long-term dependencies in this dataset, as only a few past frames of context are necessary for prediction.
N.3 HABITAT
Habitat is a 3D simulator that can render realistic trajectories in scans of 3D scenes. We compile roughly 1400 3D scans from HM3D (Ramakrishnan et al., 2021). MatterPort3D (Chang et al., 2017) and Gibson (Xia et al., 2018), and generate total of 200k trajectories of 300 128 x 128 frames. We use the in-built path traversal algorithm provided in Habitat to construct action trajectories that allow our agent to move between randomly sampled locations in the 3D scene. Similar to Minecraft and DMLab, the agent action space consists of left turn, right turn, and move forward.
29

---

Under review as a conference paper at ICLR 2023
O HYPERPARAMETERS
0.1 VQ-GAN & VAE
DMLab Minecraft
Habitat Kinetics-600
GPU Days
16
32
Resolution
64 / 128
128
Batch Size
64
64
LR
x 10
3 x 10
Num Res Blocks
Attention Resolutions
16
Channel Mult
1,2,2,2
1,2,3,4
Base Channels
128
128
Latent Size (VQ-GAN)
16 16
16 16
Embedding Dim (VQ-GAN)
256
256
Codebook Size (VQ-GAN)
1024
8192
Latent Size (VAE)
16 X 16 x 4
16 x 16 x
0.2 TECO
Hyperparameters
DMLab
Minecraft
Habitat
Kinetics-600
TPU-v3 Days
80
275
640
Params
169M
274M
386M
1.09B
Resolution
64
128
128
128
Batch Size
32
32
32
32
Sequence Length
300
300
300
100
LR
10-4
10-4
1
10-4
10
LR Schedule
cosine
cosine
cosine
cosine
Warmup Steps
10k
10k
10k
10k
Total Training Steps
1M
1M
IM
1M
DropLoss Rate
0.9
0.9
0.9
0.9
Encoder
Depths
256, 512
256, 512
256, 512
256, 512
Blocks
2
4
4
8
Size
1024
1024
1024
1024
Codebook
Embedding Dim
32
32
32
32
Decoder
Depths
256, 512
256, 512
256, 512
256, 512
Blocks
8
10
Downsample Factor
8
8
Hidden Dim
1024
1024
1024
1536
Temporal
Feedforward Dim
4096
4096
4096
6144
Transformer
Heads
16
16
24
Layers
8
12
8
24
Dropout
0
Mask Schedule
cosine
cosine
cosine
cosine
Hidden Dim
512
768
1024
1024
Feedforward Dim
2048
3072
4096
4096
MaskGit
Heads
8
12
16
16
Layers
8
6
16
24
Dropout
0
0
30

---

Under review as a conference paper at ICLR 2023
Train Sequence Length
(Fewer FLOPs per Frame)
Hyperparameters
300
200
100
TPU-v3 Days
32
32
32
32
Params
169M
169M
86M
195M
Resolution
64
64
64
64
Batch Size
32
32
32
LR
1 10-4
1 x 10 10-4
1 x 10-1
1 X 10-4
LR Schedule
cosine
cosine
cosine
cosine
Warmup Steps
10k
10k
10k
10k
Total Training Steps
1M
1M
1M
1M
DropLoss Rate
0.9
0.85
0.85
0.85
Encoder
Depths
256, 512
256, 512
256, 512
256, 512
Blocks
2
2
2
2
Size
1024
1024
1024
1024
Codebook
Embedding Dim
32
32
32
Decoder
Depths
256, 512 4
256, 512 4
256, 512 4
256, 512
Blocks
Downsample Factor
8
Hidden Dim
1024
1024
512
1024
Temporal
Feedforward Dim
4096
4096
2048
4096
Transformer
Heads
16
16
8
16
Layers
8
8
Dropout
0
0
Mask Schedule
cosine
cosine
cosine
cosine
Hidden Dim
512
512
512
768
Feedforward Dim
2048
2048
2048
3072
MaskGit
Heads
8
8
8
12
Layers
8
Dropout
0
0
0
Table O.1: Hyperparameters for scaling TECO on DMLab
31

---

Under review as a conference paper at ICLR 2023
Train Sequence Length
(Fewer FLOPs per Frame)
Hyperparameters
300
200
100
TPU-v3 Days
80
80
80
80
Params
274M
261M
257M
140M
Resolution
128
128
128
128
Batch Size
32
32
LR
1 10
1 x 10-1
1 x 10-1
1 X 10-4
LR Schedule
cosine
cosine
cosine
cosine
Warmup Steps
10k
10k
10k
10k
Total Training Steps
1M
IM
1M
1M
DropLoss Rate
0.9
0.85
0.25
0.25
Encoder
Depths
256, 512
256, 512
256, 512
256, 512
Blocks
4
4
4
4
Size
1024
1024
1024
1024
Codebook
Embedding Dim
32
32
Decoder
Depths
256, 512
256, 512
256, 512
256, 512
Blocks
Downsample Factor
Hidden Dim
1024
1024
1024
512
Temporal
Feedforward Dim
4096
4096
4096
2048
Transformer
Heads
16
16
16
Layers
12
12
Dropout
0
Mask Schedule
cosine
cosine
cosine
cosine
Hidden Dim
768
768
768
768
Feedforward Dim
3072
3072
3072
3072
MaskGit
Heads
12
12
12
12
Layers
Dropout
0
0
Table O.2: Hyperparameters for scaling TECO on Minecraft
0.3 LATENT FDM
Hyperparameters
DMLaa
Minecraft
Habitat
Kinetics-600
TPU-v3 Days
32
80
275
640
Params
31M
33M
87M
831M
Resolution
64
128
128
128
Batch Size
32
32
32
LR
x 10
1 x 10
x 10
10
LR Schedule
cosine
cosine
cosine
cosine
Optimizer
Adam
Adam
Adam
Adam
Warmup Steps
10k
10k
10k
10k
Total Training Steps
IM
IM
IM
1M
Base Channels
128
128
128
256
Num Res Blocks
1,1,1,2
1,1,2,2
1,2,2,4
2,2,2,2
Head Dim
64
64
64
64
Attention Resolutions
4,2
4,2
4.2
8,4,2
Dropout
0
0
0
0
Channel Mult
1,1,1,2
1,2,2,2
1,2,2,4
1,2,3,8
Table 0.3: Hyperparameters for Latent FDM
32

---

Under review as a conference paper at ICLR 2023
Train Sequence Length
(Fewer FLOPs per Frame)
Hyperparameters
300
200
100
TPU-v3 Days
32
32
32
32
Params
31M
62M
80M
110M
Resolution
64
64
64
64
Batch Size
32
32
32
LR
1 x 10
1 x 10-1
1 x 10-4
1 X 10-4
LR Schedule
cosine Adam
cosine
cosine Adam
cosine
Optimizer
Adam
Adam
Warmup Steps
10k
10k
10k
10k
Total Training Steps
1M
IM
1M
1M
Base Channels
128
128
128
192
Num Res Blocks
1,1,1,2
1,1,2,2,4
2,2,2,2
3,3,3,3
Head Dim
64
64
64
64
Attention Resolutions
4.2
4.1
4.2
8,4,2
Dropout
0
0
Channel Mult
1,1,1,2
1,1,2,2,4
1,2,3,4
1,2,3,4
Table O.4: Hyperparameters for scaling Latent FDM on DMLab
Train Sequence Length
(Fewer FLOPs per Frame)
Hyperparameters
300
200
100
50
TPU-v3 Days
80
80
80
80
Params
33M
80M
69M
186M
Resolution
128
128
128
128
Batch Size
32
LR
1 10
1 10
10
x 10
LR Schedule
cosine
cosine
cosine
cosine
Optimizer
Adam
Adam
Adam
Adam
Warmup Steps
10k
10k
10k
10k
Total Training Steps
1M
1M
1M
1M
Base Channels
128
128
128
192
Num Res Blocks
1,1,2,2
2,2,2,2
3,3,3,3
2,2,2,2
Head Dim
64
64
64
64
Attention Resolutions
4,2
4,2
8,4,2
8,4,2
Dropout
0
0
0
Channel Mult
1,2,2,2
1,2,3,4
1,2,2,3
1,2,3,4
Table 0.5: Hyperparameters for scaling Latent FDM on Minecraft
33

---

Under review as a conference paper at ICLR 2023
0.4 CW-VAE
Hyperparameters
DMLab
Minecraft
TPU-v3 Days
32
80
Params
11
140M
Resolution
64
128
Batch Size
32
32
LR
10-4
x 10-4
LR Schedule
cosine
cosine
Optimizer
Adam
Adam
Warmup Steps
10k
10k
Total Training Steps
1M
1M
Kernels
4,4.4
4,4,4
Encoder
Filters
256,512,1024
256,512,1024
Depths
256,512
256,512
Decoder
Blocks
4
8
Levels
3
Abs Factor
6
6
Enc Dense Layers
3
Enc Dense Embed
1024
1024
Dynamics
Cell Stoch Size
128
256
Cell Deter Size
1024
1024
Cell Embed Size
1024
1024
Cell Min Stddev
0.001
0.001
Table O.6: Hyperparameters for CW-VAE
0.5 FITVID
Hyperparameters
DMLab
Minecraft
TPU-v3 Days
32
80
Params
165M
176M
Resolution
64
128
Batch Size
32
32
LR
x 10-4
1
14
LR Schedule
cosine
cosine
Optimizer
Adam
Adam
Warmup Steps
10k
10k
Total Training Steps
1M
1M
Dim
256
256
RNN Size
512
768
Dim
64
128
Filters
128, 128,255226
128, 12,,266555
Table 0.7: Hyperparameters for FitVid
34


Reviews:
[{"review_id": "VBdKFF_kw3n", "reviewer": "Reviewer_JK8Y", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method for long-term (up to 300 frames) video prediction. The technical method includes \n1) encoding latent code from consecutive frames, \n2) a temporal transformer with autoregressive sampling,\n3) a MaskGit for decoding the dynamic prior, and \n4) a CNN decoder to decode the frame latent code.\nTo validate the effectiveness of the proposed approach, the paper creates and evaluates several datasets, e.g., DMLab, Minecraft, Habitat, and Kinetics-600. The results show that the proposed method outperforms the state-of-the-art video prediction algorithms. ", "summary_of_the_review": "I think the paper demonstrates great results in long-term video generation. The evaluation is strong. But I am not sure if the technical novelty is sufficient for conferences like ICLR. As discussed earlier, the main source of efficiency for modeling long videos is 1) encoding consecutive frames and 2) downsampling before using a temporal transformer. I thus rate the paper as marginally above the acceptance threshold at this point.", "strength_and_weaknesses": "Strength:\n+ The paper is well-written. The exposition is generally clear. \n+ The method is technically sound. \n+ The evaluation is thorough and convincing. The paper tested two long-term video prediction tasks (DMLab, Minecraft) and showcases generation results on complex natural video (Kinetics-600). The ablation study in Section 4.5 validates the contributions of various technical components. \n\nWeakness: \n- The novelty of the work is somewhat limited. The core difference over prior work on video generation is to first compress the frame latent tokens using an encoder to map concatenated consecutive frame latent code into a new \"Vector-Quantized latent dynamics\". The main efficiency comes from the strided convolution for downsampling discrete latent z_t and a transposed convolution for upsampling after modeling with a transformer. The resolution downsampling and upsampling operations for transforms have been extensively used in methods like VQVAE and VQGAN. \n\n- In Figure 2 shows the Decoder takes h_1 and z_2 as input. Yet, this contradicts Equation (5) where the decoder models p(x_t | z_t, h_t). Which one is correct?\n\n- Section 3.1 presents the core method. I can follow it up with the temporal transform part since they are common operations. But I did not get the motivations and rationale for designing MaskGit iterative decoding for z_t and a CNN for decoding h_t and z_t to frame latent code x_t. The paper only describes what was done, but does not provide reasons. In particular, I think it would make the exposition clearer if the paper shows a simple baseline method (e.g., from frame latent x_t, run a temporal transformer to predict h_t, and then decode h_t back to x_t) so that the readers understand the core differences and modifications are. \n\n- It would be helpful to show how the sequences of frame tokens are generated in the prediction process. It's hard to imagine what the autoregressive steps are with h_t, z_t, and x_t. I can only guess how it might work but it would very helpful if the paper can show it explicitly. For example, during autoregressive sampling, do we use the decoded \\hat(z_t) as input to the decoder? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DYl2ZxhSoju", "reviewer": "Reviewer_HpHv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose TECO (Temporal Consistent Video Transformer) for long-term video prediction. To make video prediction on long videos feasible, TECO trains on pretrained VQ-GAN codes, using transformer for temporal modeling, and on the top is a decoder (with an expressive prior). Experiments are done on various benchmarks (some are generated from game engine or embodied-AI simulation, e.g., Habitat, one realistic benchmark is also included, e.g., Kinetics-600). Various baselines are also provided. The results show that TECO performs better than other baselines on multiple video prediction metrics. TECO also provides faster sampling while maintain temporal coherence. Written presentation is fair and readable. ", "summary_of_the_review": "Overall, the current submission has moderate novelty and fair experimental results. I currently rate this paper with a borderline rating and would love to hear the authors/other reviewers to discuss about the significance of this work.", "strength_and_weaknesses": "# Strength\n- The proposed TECO has better performance when compared with baselines on standard video prediction metrics.\n- Experiments on various benchmarks, baselines are provided.\n\n# Weakness\n- It is unclear how significant those improvements (metrics) will render on predicted frames. It would be interesting to conduct a user study to test if those improvements make any difference.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7tkaeKb6psX", "reviewer": "Reviewer_zBcT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method, TECO, for the task of video prediction. TECO learns compressed representations to effectively and efficiently condition on long-form videos by modeling long-term dependencies.In order to achieve that, they leverage the MaskGit prior for dynamic prediction and propose a few new architectures based on VQ-GAN. As last, they also introduce several new long-form video prediction benchmarks in this work.\n", "summary_of_the_review": " \nOverall, I would vote weak reject at this point due to some unclear points mentioned in the above section. But I tend to increase my rating after seeing the feedback from authors. As is mentioned in this paper, the long-form video topics have not been well-explored in the community and I believe this paper is able to make contributions in this area.\n", "strength_and_weaknesses": "This paper is well-written. The discussion on the previous video prediction works is mostly well done. Their main contribution is extending the short video generation to the long-form video generation while maintaining the temporal consistency. Thus, the motivation is well justified. \n \nThey show a quantitative comparison between TECO and other baseline methods in long temporal consistency and sampling speed, demonstrating TECO could achieve competitive results with fast sampling speed. \n \nBesides, this paper introduces three challenging video benchmarks to better measure the long-range consistency in video prediction, which I believe could be a significant contribution to the community. \n \nAt last, this paper demonstrates promising performance in these challenging datasets and outperforms the previous state-of-the-art methods.\n \nHowever, there are also a few points that need to be further explained and addressed.\nIn Figure 2, it seems that the $\\hat x_t = D(z_{t}, h_{t-1})$, which is different from the text in the paper. It seems the prediction of the future time-step from the current time-step include the feature embedding from the future? To make it clear, will be good to show how $\\hat x_t $ is computed.  Also, I am not sure how the output of MaskGit ($\\hat z_t$) is leveraged in the equation. It will be good to generate notation for $\\hat z_t$.\n \nIn Encoder, it is not well explained how the proposed algorithm could leverage/benefit from the temporal redundancy in video data. Also the temporal redundancy should be datasets-dependent. There should be some justifications showing there are temporal redundancies with different levels in each dataset.\n \nIn addition to these, there are a few minor points:\nIt will be good to show the formulation of $L_{LPIPS}$ in Eq 3\nfirst row below Eq(6): the decoder is a unsampling -> is an unsampling..\nIn Figure 3 right, what is the timestep for this figure? As the LPIPS of each method is dynamically changed with timestep (left figure).\nIn the main paper, some explanation refer the result in the Appendix, which is not suggested.\n \nSuggestion:\nIn terms of modeling the long-term dependencies, there are a few recent works that are not included in the related work. For example, the MeMViT[1] and ViS4mer[2]. Both of them focus on capturing long-term dependencies in an efficient and effective way. Would be good to at least mention them in the related work.\n\n[1]Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition, CVPR 2022\n\n\n[2] Long movie clip classification with state-space video models, ECCV 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-KQe5OOUVr8", "reviewer": "Reviewer_7Nbj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to improve the discrete sequence model for long-term video prediction. The proposed method operates on (patch-wise) quantized representation of videos extracted by VQ-GAN, and models the temporal dynamics through Transformers. To handle the quadratic complexity of Transformers thus enabling training with long sequences, the authors proposed to incorporate an additional CNN-based encoder and decoder and model the latent dynamics with smaller tokens using temporal and bidirectional Transformers (MaskGit). The proposed method is evaluated on three challenging video benchmarks and demonstrated compelling long-term prediction performance over baselines.", "summary_of_the_review": "I believe that the paper tackles the interesting and important problem of long-term video prediction, but the technical and empirical significance of each part is unclear from the current draft. It would also be great to have additional experiments to analyze the performance in terms of long-term consistency. ", "strength_and_weaknesses": "Strength:\n- The paper is generally well-written and easy to follow.\n- The ideas presented in the paper are sound. The extensive experiment results support that training with longer sequences by the proposed method is indeed helpful in generating long videos better than the baselines in terms of general fidelity. \n\nWeakness:\n1. It is unclear from the paper which factors play a major role in improving the efficiency of video prediction. Based on my understanding, the major improvements are from the spatial downsampling in both the encoder and temporal Transformer, while the model retains the same quadratic complexity as the prior works. Despite the practicality, this is a simple trick that is also applicable to other Transformers, and having it as a major component for enabling long-term prediction limits the significance of the work. The same arguments also apply to DropLoss, given that there are also many hierarchical Transformers that apply the prediction at multiple scales (i.e., I feel that DropLoss is a simplified version of a hierarchical Transformer where the loss is coarsely defined over time). \n\n2. To demonstrate the significance of each component in improving efficiency, it would be great to have ablation studies on complexity (e.g., FLOPS and peak memory) together with fidelity. Also, it would be great to have a comparison with baselines in terms of complexity to demonstrate the full advantages. \n\n3. Since one of the main goals of this work is to improve the long-term consistency in prediction, the evaluation should also clearly demonstrate that part. Current protocols are measuring general fidelity of the predicted frames, and are not specifically focused on long-term consistency. One way to focus on the latter is to condition all video prediction models on ground-truth action trajectories and measure the loss on the part of the trajectories revisiting the observed places. This will isolate how well the model can reconstruct the known places from how good the fidelity of the predicted frames is.\n\n4. When predicting the next frame \\hat{x}_t, the decoder takes both the temporal encoding of the Transformer (h_t) and the prior (z_t). It creates a trivial autoencoding path through the prior (x_t → z_t → x_t), which makes it unclear how the model will learn to leverage the temporal encoding h_t.\n\n5. Figure 4 is not quite intuitive to interpret. Based on my understanding, the authors intended to show that TECO is the only method that preserves the size of the environment (which is fixed and bounded), which serves as evidence that it understands the long-term dependency. However, since they all rely on different action sequences, it is hard to isolate its impact. I feel that it would be more intuitive to condition all generative models with the same ground-truth action sequence and show the reconstructed map. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject