Phenaki: Variable Length Video Generation from Open Domain Textual Descriptions
Authors: Ruben Villegas
Mohammad Babaeizadeh
Pieter-Jan Kindermans
Hernan Moraldo
Han Zhang
Mohammad Taghi Saffar
Santiago Castro
Julius Kunze
Dumitru Erhan
Venue: ICLR 2023 poster (2023)
Keywords: generative models
video generation
video prediction
text to video

Abstract: We present Phenaki, a model capable of realistic video synthesis given a sequence of textual prompts. Generating videos from text is particularly challenging due to the computational cost, limited quantities of high quality text-video data and variable length of videos. To address these issues, we introduce a new causal model for learning video representation which compresses the video to a small discrete tokens representation. This tokenizer is auto-regressive in time, which allows it to work with video representations of different length. 
To generate video tokens from text we are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs as well as a smaller number of video-text examples can result in generalization beyond what is available in the video datasets. Compared to the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text or story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from time variable prompts.

Published as a conference paper at ICLR 2023
PHENAKI: VARIABLE LENGTH VIDEO GENERATION FROM OPEN DOMAIN TEXTUAL DESCRIPTIONS
Ruben Villegast
Mohammad Babacizadch
Pieter-Jan Kindermans?
Google Brain rubville@googlee.oo
Google Brain
Google Brain
mbzégoogle.
påkindezdyoogiee com
Hernan Moraldo
Han Zhang
Mohammad Taghi Saffar
Google Brain hmoraldc@google..co
Google Brain
Google Brain msaffarge faafffarggogll..
changhan&googlc.cooom
Santiago Castro" University of Michigan
Julius Kunze
Dumitru Erhan
University College London
Google Brain dumi dumitru@google.o com
sacastro@umi miichh. edu
Mulluskunze@gall com
ABSTRACT
We present Phenaki, model capable of realistic video synthesis, given sequence of textual prompts. Generating videos from text particularly challenging due the computational cost, limited quantities of high quality text-video data and vari- able length of videos. To address these issues, we introduce new model for learning video representation which compresses the video to a small representa- tion of discrete tokens. This tokenizer uses causal attention in time, which allows it to work with variable-length videos. To generate video tokens from text are using a bidirectional masked transformer conditioned on pre-computed text tokens. The generated video tokens are subsequently de-tokenized to create the actual video. To address data issues, we demonstrate how joint training on a large corpus of image-text pairs well smaller number of video-text examples result in generalization beyond what is available in the video datasets. Compared the previous video generation methods, Phenaki can generate arbitrary long videos conditioned on a sequence of prompts (i.e. time variable text story) in open domain. To the best of our knowledge, this is the first time a paper studies generating videos from open domain time variable prompts. In addition, com- pared the per-frame baselines, the proposed video encoder-decoder computes fewer tokens per video but results in better spatio-temporal consistency.
INTRODUCTION
It is now possible to generate realistic high resolution images given a description B 651 but generating high quality videos from text remains challenging. In essence, videos are just sequence of images, but this does not mean that generating long coherent video is easy. In practice, is a significantly harder task because there much less high quality data available and the computational requirements are much more severe [ For image generation, there are datasets with billions of image-text pairs (such as LAION-5B 45 and JFT4B (67] while the text-video datasets substantially smaller e.g. WebVid 4 with ~ 10M videos, which is not enough given the higher complexity of open domain videos. As for computation, training current state-of-the- art image generation models is already pushing the state-of-the-ar computational capabilities (65), leaving little to no room for generating videos, particularly videos variable length.
39 36
42
To make the matters worse, a complete description of a video (except for short clips), and instead, a generated video must be conditioned sequence of prompts, or a story, which narrates what happens over time. Ideally, a video generation model must be able to generate videas of arbitrary length, all the while having the capability of conditioning the generated frames at time t on prompts at time t that can vary over time. Such capability can clearly distinguish the video from "moving image" and open up the way
argue that single short text prompt is not sufficient provide
Equal contribution. Intern Google Brain while working this project.

---

Published as a conference paper at ICLR 2023
1st prompt: "A photorealistic teddy bear is swimming in the ocean at San Francisco"
2nd prompt: "The teddy bear goes under water"
3rd prompt: "The teddy bear keeps swimming under water colorful fishes"
4rd prompt: "A panda bear is swimming under water"
Time variable text (i.e. story) conditional video generation. The entire figure is tinuous video generated auto-regressively. We start by generating the video conditioned on the first prompt and then after couple of frames we change the prompt the next one. Each row contains selected number of frames (from left to right in order) while the model was conditioned on that par- ticular prompt. The model manages preserve the temporal coherence of the video while adopting to the new prompt, usually taking the shortest path for the adaption (notice the morphing of the teddy bear to the panda). Note that the generated video has complex visual features such as reflections, occlusions, interactions and scene transitions. Full video is available at phenaki.github.io
Figure 1.
to real-world creative applications in art, design and content creation. the best our knowledge, story based conditional video generation in open domain has never been explored before and this is the first paper to take early steps towards that goal. A traditional deep learning approach simply learning this task from data is not possible, since there is story-based dataset to learn from. Instead, to achieve this we rely on a model that is designed specifically with this capability in mind.
In this paper, introduce Phenaki, text video model trained both text video and text to image data that can:
Generate temporally coherent and diverse videos conditioned on open domain prompts even when the prompt is a new composition of concepts (Fig. B. The videos can be long (minutes) even though the model is trained on 1.4 seconds videos (at 8 fps).
Generate videos conditioned on story (i.e. a sequence prompts), e.g. Fig. and Fig.
To enable these capabilities, we could not rely on current video encoders, because they either can only decode fixed size videos they encode frames independently. Hence, we introduce C-ViViT, novel encoder-decode architecture that:
Exploits temporal redundancy in videos improve reconstruction quality over per frame model while compressing the number of video tokens by 40% or more.
Allows encoding and decoding of variable length videos given its causal structure.
2

---

Published as a conference paper at ICLR 2023
C-ViViT Encoder
Training Transformer
Video Generation
Po Ist Prompt
X
1:2
Xat 1:20
Totere
X
Po "Prompp"
TIX
Pasch End
Parch End
Passh Erd
CASNT Encocer
Pred
Tolere
1:2
2/1-1:24
Transkenrer
Trerakerne
Tuanskaer
Frocne Past
Py TNext Prompt"
Rancom Masiing
Tolent
T 21:-1:25
Canoe Trandoorrr
Caucal Transtrmar
Caucal Traedomer
Traestmer
Predided Future Tokeet
Zo
1:2
21
21-1:22
2122
82
1:21
10
1:21
8...
Token
Masked/Empty Token
Transformer
Linear Embedding
Frozen Model
Operation
The architecture of Phenaki. Left: C-ViViT encoder architecture. The embeddings of images and video patches from raw frames x are processed by a spatial and then a causal transformer (auto-regressive in time) to generate video tokens z. Center: MaskGiT is trained to reconstruct masked tokens z predicted by a frozen C-ViVIT encoder and conditioned on T5X tokens of a given prompt Pa- Right: How Phenaki generating the future tokens. The prompt can change over time to enable variabe prompt (i.e. story) conditional generation. The subscripts represent time (i.e. frame number).
Figure 2.
generate arbitrary long videos by freezing the past token and
THE PHENAKI MODEL
and text to video 60 encoder-decoder model which compresses videos to discrete embeddings (i.e. tokens) and a transformer model to translate text embeddings video tokens. To get the text embeddings, Phenaki uses a pre-trained language model, T5-XXL AT. We will discuss each one of these components in the following subsections.
Inspired by the previous work in auto-regressive text to image 38 65
59 (22,
Phenaki is designed with two main components (see Figure
2.1 ENCODER-DECON DECODER VIDEO MODEL: C-VIVIT
One of the primary challenges for generating vidco from text, is to get a compressed representation of videos. Previous work text to video either per-frame image encoders 12
60
63
such
as VQ-GAN [4] or fixed length video encoders 58 such as VideoVQVAE (55]
The former
allows for generating videos of arbitrary length, however in practice, the videos have be short because the encoder does not compress the videos in time and the tokens are highly redundant in consecutive frames. The latter is generate variable length videos. In Phenaki, our goal is to generate videos of variable length while keeping the number of video tokens minimum so they can be modeled with transformer within current computational limitations. To do so, we introduce C-ViViT VIVIT I with additional architectural changes for video generation, which can compress the videos in temporal and spatial dimensions, while staying auto-regressive in time. This capability allows for generating videos arbitrary length auto-regressively.
efficient in the number of tokens but does allow
causal variation of
Encoder architecture: As illustrated in Figure 2 start with video sequence of l,z frames with a resolution of w, x htp and C₂ channels: E +1x Xw, Xc,, This sequence will be compressed into a token representation of size (t, + 1) x w, x h2 where the first w, x h2 tokens represent the first frame independently from the rest of the video, and the remaining tokens represent spatio-temporal video tokens that auto-regressivell depend on previous frames. To do so, we extract non-overlapping image patches of size Wp hp X cp from the first frame and video patches of size ip x Wp x hp x Cp from the rest of the video. We typically use all channels at once such that the number of patches equals the number of video tokens tr = is f, w. m,,
us and hr he Each of h,
these patches is flattened and linearly projected into d, dimensional space. We combine the spatial
3

---

Published as a conference paper at ICLR 2023
dimensions to have tensor of shape (tz (1)+1))xu *h, xd, where the spatial and temporal dimensions are separated. Then multiple transformer layers are applied along the spatial dimensions with all- to-all attention. This is followed by multiple transformer layers over the temporal dimension with causal attention such that each spatial token only observes spatial tokens from previous frames in an auto-regressive manner. The effect of this is that the first frame can be completely independently encoded. This opens up the possibility of text to image training to be embedded naturally into our video model. The second advantage that we can condition the video generation process on number of starting frames. The resulting patch embeddings of shape iz X wz h2 x d, are then tokenized into learned codewords c3 by vector quantization. The codebook learning will be discussed later together with the losses.
Decoder architecture: The C-ViVIT decoder is simply an upside down version of the encoder. First tokens are transformed into embeddings. This is followed by the temporal transformer, then the spatial transformer. After the output of the spatial transformer, we apply a single linear projection without activation to map the tokens back to pixel space.
Quantization and Losses: To learn a discrete latent space, we quantize our encoder outputs into the entries of a learned codebook via the vector quantization (VQ) objective in VQVAEs (5].
Lve ||s((z) z)) e1i2 + B||z 2g(e)]
where sg(x)) x, and dz sg(x)) 0 is the stop-gradient operator, ,3 is the commitment loss weight, and e is a codebook vector from codebook E. The index to the codebook vector closest to z is adopt the factorized and fz-
found by argmin, z E, 1 In addition to the VQ objective, to improve codebook usage and reconstruction quality.
34 normalized codes from VQGAN
To train our model, perceptual loss Lyp by using the 13D network [ as feature extractor, and adversarial loss Lad, with StyleGAN architecture 25
combination of L2 loss, image perceptual loss Tip
24
68 video
As training objective, we use the following
L = Lve + 0.1 X Lad + 0.1 x LIP + 1.0 x Lvp + 1.0 x Lz.
Novelty over the VIVIT architecture: While our proposed C-ViVIT architecture is inspired by the factorized encoder in ViVIT (. we modify their architecture enable self-supervised learn- ing from unlabeled videos. We first remove the [CLS] tokens in the spatial and the temporal transformers. Next, we apply temporal transformer for all spatial tokens computed by the spatial en- coder, in contrast to single run of the temporal transformer over the [CLS] tokens in ViViT. Most importantly, the ViVIT encoder requires a fixed length video input due the all-to-all attention in time. Therefore, we apply causal attention instead such that our CiViVIT encoder becomes auto- regressive and allows for variable number of input frames which are necessary learn from image datasets, and auto-regressively extrapolate video or single frames into the future.
2.2 TEXT-TO-VIDER GENERATION WITH BIDIRECTIONAL. TRANSFORMERS
In this stage, the text-to-video task can be formulated as a sequence-to-sequeeee sequence problem to predict video tokens given the paired text embeddings. Most of recent methods 38 transformer model for these
adopt tasks. In their models, they use an auto-regressive transformer which predicts the image video tokens sequentially given the encoded text features. As a result, the sampling time scales linearly with the sequence length, even when caching is used. This becomes impractical for long video sequence generation.
60
Masked bidirectional transformer: In this work, we aim to reduce the sampling time by having a small and fixed sampling step disregarding different video sequence lengths. Inspired by previous work for image generation [10, we use bidirectional transformer since it can predict different video tokens simultaneously. For training step i, we first sample a mask ratio % from 0 to I and randomly replace 17 N tokens with the special token [MASK], where N is the video sequence length. Then we learn the model parameters by minimizing the cross entropy loss on those masked tokens given the encoded text embeddings and unmasked video tokens. During inference, label all of the video tokens as the special token [MASK]. Then, at each inference step, we predict all the masked (unknown) video tokens in parallel conditioned on the text embeddings and unmasked
first
4

---

Published as a conference paper at ICLR 2023
(predicted) video tokens. We keep ratio B; of the predicted tokens at sampling step i and the remaining tokens are re-masked and re- re-predicted in the next step.
As discussed in MaskGIT [Q) the masking schedule 7/ and sampling schedule B, have signifi- cant effect on the samples quality therefore we follow the same strategies. Compared to an auto- regressive transformer, the number of sampling steps is an order-of-magnitude smaller (typically we use values in the range of 12 to 48). Generally speaking, more sampling steps improves the quality.
Losses and training strategies: Given a pre-trained C- C-ViVIT, videos are encoded into codebook ids a of shape (tz + 1) x w; x h- which are flattened into a long vector using the raster ordering from (4]) We then model the text-conditional video token distribution using Masked Visual Token Modeling (MVTM) [TO]
Linas = = - V.M p(a, a p),
(3)
where a represents the masked version of a, m, is a binary variable indicating whether a, is masked not, N is the number of video tokens, and p is the text condition embedding. In addition to the MVTM objective, we train using classifier-free guidance by dropping the text condition 10%
of the time during training 20 651. Finally, we dynamically adjust the MVTM objective during training to allow the use of image and video datasets as a single large dataset. We achieve this by only applying the masking ratio and objective on the first w; x h2 tokens if only single frame given or over all video tokens if full video is given. This mixed image and video dataset training strategy allows our models to learn concepts only present in image datasets, and transfer them to concepts present video datasets (e.g., the pencil drawing styled video of the panda in Figure B
Inference and auto-regressive generation of long videos: At inference time, we sample videos tokens by the same iterative process used in [T0] with classifier-free guidance scale À to control alignment between the generation and the text condition. Once the first t + 1 frames are generated in latent space, we can extrapolate additional frames auto-regressively by re-encoding the last K generated frames in the last generated video using C-ViViT initializing MaskGIT with the tokens computed by our C- ViV encoder, and proceed generate the remaining video tokens conditioned on text input. During video extrapolation, the text condition can be the same or a different one which enables our model to dynamically create visual transitions between the previous and current text condition visual content, effective generating visual story an described by the input text. Phenaki does not perform generation purely in latent space due to the design choices we made to take advantage of text-image and text-video datasets during training. The first wa x h2 tokens must be single image (space only) tokens and the next t₂ X w, X h2 tokens must be video (space-time) tokens. Therefore, MaskGIT will always expect this input arrangement during generation.
EXPERIMENTS
To evaluate Phenaki, we test it on the following tasks: 1) text conditional video generation, 2) text- image conditional video generation, 3) open domain time variable text conditional video generation (i.e.) story mode, 4) video quantization and 5) image conditional video generation a.k.a. video prediction. To the hest of our knowledge, 3) open domain time variable text conditional video generation has not been explored in prior work. Given the dynamic nature of videos, we highly encourage readers to visit phenakiipkki.ppppm github.io to check the generated videos. The website also includes qualitative comparisons to a subset of the prompts from the Cog paper [2]. While the focus is on the text to video generation tasks, it is remarkable that Phenaki is still competitive on the more traditional video tasks despite not being developed explicitly for these tasks.
3.1 TEXT CONDITIONAL VIDEO GENERATION
Currently there is established benchmark for evaluating text video methods. This makes com- paring Phenaki to recent methods such as NUWA 601, Cog Video 22]. NUWAA-nfinity (59) and video diffusion models [1] difficult.
Unless specified otherwise, we train a 1.8B parameter Phenaki model on a corpus of ~15 text- video pairs at 8 FPS mixed with 550 text-images plus ~400M pairs of LATON-400M 45 (more
5

---

Published as a conference paper at ICLR 2023
Table 1. Text to vidco compar-
Table 2. Text to vidco and text image results highlighting the importance image datasels in video models. Text-to-image eval- uation is done on ~<10K images of LAION-40OM 45
isons Kinetics-400 26
FID
FID
Method
Image
Video
Data Split
Text to Video
Text to Image
T2V 29
82.13
14.65
SC [5]
33.51
7.34
Vid% / Img%
CLIP 1
FID
FVD 4
CLIP
HID [
TFGAN S
31.76
7.19
100% / 0%
0.298
19.2
168.9
0.240
53.9
NUWA
28.46
7.05
80% / 20%
0.303
21.4
198.4
0.289
29.4
Phenaki [0-Shot]
37.74
3.84
50% / 50%
0.302
21.4
239.7
0.287
30.5
details in Appendix B.3). The model used in the visualisations in this paper was trained for million steps at batch size of 512, which took less than 5 days. In this setup 80% of the training data came from the video dataset and each image dataset contributed 10%.
Qualitative evaluation: Samples from this model can be seen in Figure S and additional samples are provided at actors and the background dynamics the videos. The appearance of the actors and the video style can be adjusted by the text prompt as well (e.g. a regular video, cartoon or a pencil drawing).
We observe that there is a high degree of control over both the
On phenaki. iithub.io we provide examples from prompts that were provided in the Cog Video 22 demo. Since there are substantial differences between these methods it is hard to compare them on an equal footing. As an example, there are massive differences in scale: 9B parameters for Cog Video and 1.8B for our model. Additionally, the training data is different. Finally, we do not know how representative the prompts in the Cog Video demo are for the general performance of the Cog Video.
Quantative comparison: The NUWA 600 paper provided qualitative evaluation on Kinetics- 400. Since the NUWA model is only 0.9B parameters we also use a model of the same size. Our model was trained on 50% video and 50% image data in this experiment. The NUWA model fine- tuned on Kinetics but the Phenaki model is not: it is evaluated in a zero shot setting. The results in Table show that Phenaki achieves comparable generation quality, in a zero-shot setting, compared to previous text to video methods that were actually trained or finetuned on this dataset.
On the importance of joint text-to-image and text-to-video training While there are some text- video datasets, text-image datasets dominate the internet in terms of quality and quantity 34 Con- sequently, there is simply not enough video data available to cover all the concepts present in text- image datasets. For example using only our video data, concepts such as pencil drawings different painting styles cannot be learned. To be able to learn a model that can combine video dynamics with these additional concepts we have to combine training on image and video data. In Table y we evaluate the performance of using different ratios of video and images. We start with data splits of only video, and vary the ratio of image and video datasets up to using 50% image and 50% video datasets. In our results, we find that there is a trade-off in performance between models trained with only video video (i.e., significantly better FVD), and models trained with more image data (i.e.. better text-video and text-image aligmment, and significantly better FID in image datasets), On phenaki. wiwnukiigttuubioo. we show samples from different models side by side where this trade-off between control over the content and the quality of the dynamics can be seen. We believe that the trade- off between concepts and dynamics will be improved as the quality and size of text-video datasets increases in the future.
3.2 TEXT- CONDITIONAL VIDEO GENERATION
Given that Phenaki can be conditioned on both still images and text, an interesting setup is to animate existing images given a text prompt. For this experiment, we use the same model from Section 3.1 but conditioned on unseen pictures (captured with our phones from local subjects) and a related prompt. As can be seen in Figure the the model can generate coherent videos starting from the given images, while following the given prompts.
3.3 VISUAL STORY TELLING BY DYNAMIC TEXT INPUTS
A notable and useful feature of Phenaki is that it rego-regressive in time. This allows for generating long videos, while the prompt changes over time. Time variable prompts can be thought of as story;

---

Published as a conference paper at ICLR 2023
Prompt: 'HD Video: A really cute panda washing dishes with yellow glowes garden'
Prompt: "An astronaut riding horse with sunsel in the background
Prompt: "A happy panda wearing red boxiigg gloves and standing brandenburg gate fireworks in the background"
Prompt: astronaut diving coral many fishes."
Prompt: "Pencil drawing: Panda listening music with headphones knitting sweater while sitting on the couch"
Prompt: 'A cartoon an astronaut high fiving brown bear."
Text conditional video generation. Each row shows selected frames from a video gen- erated given the prompt. The model is trained on mix of images and videos. The video dataset does not include any stylized videos such as pencil drawings, however, the image dataset does. The model generalize from still images to videos. This figure also demonstrate the capability of the model generating new unseen compositions. Full videos are available at phenaki. github.io
Figure 3.
Given Image
Prompt: "Camera quickly into the eye the cat"
Given Image
Prompt: white touches the camera paw'
Given Image
Prompt: "A white cat yawns loudly"
Figure 4. Animating images conditioned on a prompt. Each row demonstrates multiple frames of generated video conditioned on a given first frame as well as given text prompt. The first frames are new (captured by author's s phone) and not observed during the training. The model animates the given image while following the prompt. Full videos are available at prrenaki.githuh.i.
narration of the entire video where each prompt corresponds to scene from the video. This allows for creating dynamically changing scenes. To the best our knowledge, this paper is the first work to generate such videos. An example of this can be seen in Fig. and on wwm.phhn.aiggmmm thenaak.githuub.io github.io The way it works is that we generate a video with the first prompt and then extend it in time by conditioning a possibly prompt and the last N, typically previously generated frames.

---

Published as a conference paper at ICLR 2023
Table 3. Video reconstruction results on Moments-in-Time The number of tokens is computed for 10 frames with the exception of C-ViViT which is for 11, due the isolated initial frame.
Method
FID J
FVD |
Number of Tokens
Conv VQ-GAN IE!
7.5
306.1
2560
Conv VQ-GAN Video loss
13.7
346.5
2560
VIT VQ-GAN
3.4
166.6
2560
VIT VQ-GAN Video loss
3.8
173.1
2560
C-ViVIT VQ-GAN (Ours)
4.5
65.78
1536
Table 4. Video prediction Kineties-600
While Phenaki not designed for video prediction it achieves com- parable results with SOTA video prediction models.
9
Table 5. Video prediction BAIR 1B
Method
FVD
DVD-GAN I
109.8
Method
FVD 4
VideoGPT
61
103.3
Video Transformer
170.0 + 5.00
TrVDD-GAN
103.3
Video 22
109.2
Transframer
35
100.0
DVD-GAN- III
69.1 + 0.78
HARP 63
99.3
Video VQ- VAE 55
64.3 = 2.04
CCVS
32
99.0
CCVS 32
55.0 + 1.00
Video Transformer
57
94.0
THVD-GAN-FF
25.7 + 0.66
FitVid
93.6
Transframer
25.4
MCVD
89.5
RaMViD 23
16.5
NUWA 60
86.9
Video Diffusion 21
16.2 + 0.34
RaMViD
23
84.2
Phenaki (Ours)
36.4
Phenaki (Ours)
97.0
3.4 VIDEO ENCODING
To evaluate the video encoding and reconstruction performance of C-ViViT we use the Moments- in-Time (MiT) [33] dataset. MiT contains ~802K training, ~33K validation and ~67K test videos at 25 FPS. The MiT dataset, in contrast to other publicly available video datasets, a high quality balanced dataset with high coverage and density of verbs depicting moments of few seconds [3] We compare C-ViViT against per-frame image based encoder-decoders that have been used as video quantizers for conditional video generation
50
22 58
ViT 64 and convolutional
VQ-GAN 4.] The experimental details can be found in the Appendix B.1
As demonstrated Table we evaluate the video reconstruction quality using FID and FVD [50]. Both FID and FVD compare the distribution of generated vidcos (or images) to the ground truth distribution. The FID ignores temporal coherency, while the FVD measures how well the spatio-temporal dynamics of the videos are reconstructed. Results in Table show that per- frame image based methods slightly outperform our video method (indicated by marginally higher FID of C-ViViT ). however, they do poorly modeling the spatio-temporal dynamies in video (sig- nificantly lower FVD C-ViViT ). This expected as C-ViVIT has spatio-temporal connections between patches in each frame, allowing space and time to be modeled together. In addition, C- ViVIT compresses the video into fewer tokens per video compared to the image based baselines. This is erucial the number of tokens drastically impacts the computational cost of the transformer in downstream tasks. Furthermore, C-ViViT tokens are auto-regressive time which enables vari- able length videos to be modeled with the same encoder which important for video extrapolation conditioned on previously generated frames.
3.5 IMAGE CONDITIONAL VIDEO GENERATION A.K.A VIDEO PREDICTION
To evaluate the learnt video representation of C-ViViT beyond reconstruction, we test it on the task of condecoond video generation, also commonly known as video prediction 3 In this ex- periment, we test Phenaki on BAIR Robot Pushing benchmark [[3] where the task is to generate 15 frames conditioned on a given single frame. For open domain videos, we test Phenaki on Kinetics- 600 9 where the task is to predict 11 frames given 5 frames. More details about these experiments can found in Appendix B.2 Tables and S show the results of these experiments. Note that Phenaki is not specifically designed for video prediction, therefore, it lacks components such as skip connections in U-Nets which are known to improve the performance for video prediction meth- ods 122 52 B. Nevertheless,
these benchmarks with SOTA video prediction methods. Overall, these experiments show that Phenaki is strong at modeling dynamics of the videos which is required for generating coherent videos from text.
method is competitive
8

---

Published as a conference paper at ICLR 2023
4 RELATED WORKS
This paper is closely related to auto-regressive methods for text conditioned image and video genera- tion. DALL-E [38] translates text tokens to discrete image embeddings learnt using a VQVAE [5]]. Parti 65 has similar architecture but can generate higher quality images by predicting tokens from ViT VIT-VQGAN [4]]] using 20B parameters transformer. Similar architectures have been used for generating videos as well. GODIVA [58] uses transformer to map text tokens to video tokens from image based VQVAE. Given the large number of tokens from multiple frames, GODIVA relied on a local-attention mechanism. Similarly, NUWA 60 and NUWA-Infinity (59] both employ auto-regressive architectures to generate videos and images from text. NUWA generates fixed size outputs, while NUWA-Infinity introduces a second layer of auto-regressive computation to support variable size videos. Likewise, CogVideo generation is the scarcity of good text-video data and tried to leverage pre- pre-trained text to images models to generate high quality video.
Video [22] argues the main reason behind low quality video
While Phenaki sticks to the same architecture principles, it has major differences with previous work. Most notably, NUWA, NUWA-Infinity and CogVideo treat videos a sequence of independent images. This can lead to poor modeling of dynamics and generate motion artifacts. To combat this, NUWA-infinity used the previous frame during decoding to combat this. In Phenaki, we go further and treat videos as a temporal sequence of images which substantially decreases the number of video tokens given the redundancy in video generation, and results in a much lower training cost. The auto-regressive nature of the Phenaki also allows us to effectively condition on previous frames and generates longer videos as detailed in Section
Diffusion models are another class of models which recently have been used for conditional and unconditional video generation, which we call VDM (2I]. In VDM, authors proposed replacing the conventional U-Net architectures for 2D image modeling with 3D space-time model to run the diffusion process directly on pixels. While this approach provides an effective formulation for modeling videos, it is limited to fixed size videos. To address this issue, VDM provides an auto- regressive extension, which allows the model to generate longer videos but is typically impractical due to high sampling time of diffusion models.
Text conditional vidco generation is a relatively new field of research, nonetheless, image condi- tional video generation, commonly known video prediction, and unconditional video generation have been studied more comprehensively. These papers include deterministic methods using a com- bination of recurrent and convolutional networks 400
4x
variational based stochastic meth-
ods 2 52 3 models (5 61
and more recently by learning a discrete representation [55] 37 35
auto-regressive
diffusion models (53 18
52
PK] flow based models 2 and finally adver-
Iir These works mostly consider limited domain (e.g. robotic videos, video games and small datasets) prediction/generation or short fixed size clips. Section and Section C provide comparison with some of these models.
sarial based methods S5 40
B1 660 16 46
5 CONCLUSION
We introduced Phenaki, a model which is capable of generating variable length videos conditioned on a sequence of open domain text prompts. Phenaki uses C-ViViT as video encoder. C-ViViT is a new model which provides temporal-spatial compression while being ruto-regressive in time. The C-ViVIT model is a crucial part of Phenaki that allows it to generate variable length videos. We demonstrate how joint training on images and videos can improve the generation quality, and diver- sity, given the existence of much larger image-text dataset with order of magnitude more samples. The Phenaki model achieves good performance on video prediction, it can be used as to generate long videos conditioned on a text prompt. Additionally it is able to condition on both text and starting frame. Finally, Phenaki is not limited to generating a video depicting a single concept or caption. It is actually able to generate longer coherent video stories based on sequence of text prompts. The more complex narratives it can visualize demonstrate how this can become a great creative tool for story telling.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
While we have not explored potential downstream applications of the generative models described in this work, we believe Phenaki can have a positive impact in a variety of creative settings. In general, many of the samples from the model will not perfectly correspond to the input caption or the user's intent; however, the end-user is likely to gain considerable time savings even if only one of the generated samples aligns with their intent. We thus foresee Phenaki being useful in eventually empowering users to accelerate their creativity, especially since the model can so quickly generate videos. Phenaki and similar models will be part of an ever-broad toolset for artists and non-artists alike, providing new and exciting ways to express creativity.
The flip-side of this acceleration and ease-of-use the potential for harmful impact, as with many of the prior or concurrent work in generative modeling. An easy-to-use system like Phenaki can be repurposed for generating maliciously fake content and enable spreading of such content much easier. While the quality of the videos generated by Phenaki is not yet indistinguishable from real videos, getting to that bar for specific set of samples within the realm of possibility, even today. This can be particularly harmful if Phenaki is to be used to generate vidcos of someone without their consent and knowledge.
Parti 655 and others, Phenaki is trained on a collection of datasets that is known to encode a number of undesirable biases. LAION-40OM [45] specifically has a variety of issues regarding violence, pomography, gore [6) While our primary image and video datasets have minimal traits like this, we did incorporate LAION-400M into our training and observed better results. In currently training version of Phenaki, we use set of datasets that minimizes such problems.
Like DALLE-2 39), Imagen
22
Another potential issue when training generative models of any sorts is that public datasets of im- agery and videos may contain examples whose ownership status is unclear or impossible to verify. The Phenaki model not an exception, as part of its training set uses the LAION-40M set of im- ages. There is an ongoing public debate about how generative modeling research should proceed in the presence of potentially copyrighted works in the training set. Given this debate, we are taking a conservative stance by not releasing the trained checkpoints.
Taken together, these issues contribute to our decision not release the underlying models, code, data or interactive demo at this time. Before we can do that, we want to focus our efforts better understanding of data, prompt and output filtering. We would also like to more explicitly measure the biases encoded in the outputs of Phenaki, so that we can further mitigate them actively, either in the data, models or pre/post- pre/post-processing steps.
REFERENCES
[1] Anurag Arna,, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
[2] Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. ICLR, 2018.
[3] Mohammad Babaeizadeh, Mohammad Taghi Saffar, Suraj Nair, Sergey Levine, Chelsea Finn, and Dumitru Erhan. Fitvid: Overfitting in pixel-level video prediction. arXiv preprint 21f.:22006.19999 2020.
[4] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[5] Yogesh Balaji. Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Con- ditional gan with discriminative filter generation for text-to-video synthesis. IJCAI, 2019.
[6] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe, Multimodal datasets: misog- yny, pornography, and malignant stereotypes. arXiv preprint arXiv:2 2021.
10

---

Published as a conference paper at ICLR 2023
[7] Tim Brooks, Janne Hellsten, Miika Aittala, Ting-Chur Wang, Timo Aila, Jaakko Lehtinen, Ming- Yu Liu, Alexei A Efros, and Tero Karras. Generating long videos of dynamic scenes. arXiv preprint arXiv:2206.03442 2022.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017.
[9] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600, 2018.
[10] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. arXiv preprint arXiv:2202.00 04200, 2022.
[11] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets. arXiv preprint arXiv: (e0p::00070077777 2019.
[12] Emily Denton and Rob Fergus. Stochastic video generation with learned prior. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1174-1183, 2018.
[13] Frederik Ebert, Chelsea Finn, Alex X. Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections, 2017.
[14] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
[15] Chelsea Finn, lan Goodfellow, and Sergey Levine. Unsupervised learning for physical inter- action through video prediction. In Advances in neural information processing systems, pages 64-72, 2016.
[16] Songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnosio vqgan and time-sensitive trans- former. arXiv preprint arXiv:2204.03638, 2022.
[17] Sonam Gupta, Arti Keshari, and Sukhendu Das. Rv-gan: Recurrent gan for unconditional video generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 2024-2033, 2022.
[18] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos. arXiv preprint arXh::2205.11495 2022.
[19] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2021.
[21] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint 2xXiv:2204.03455 2022.
[22] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint urXiv:2205. 15868, 2022.
[23] Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprint 22X1:22006006666 2022.
[24] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. arXiv preprint arXiv:1603.0811 2016.
[25] JTero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
11

---

Published as a conference paper at ICLR 2023
[26] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and An- drew Zisserman. The kinetics human action video dataset, 2017.
[27] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
[28] Manoj Kumar, Mohammad Babacizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Lau- rent Dinh, and Durk Kingma. Videoflow: flow-based generative model for video. arXiv preprint arXiv: 1903.01434,, 2019.
[29] Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. In AAAI, 2018.
[30] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In ICLR, 2019.
[31] Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cas- sirer, and Karen Simonyan. Transformation-baseed adversarial video prediction on large-scale data. arXiv preprint arXiv:2003.0440 2019.
[32] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. CCVS: Context-aware controllable video synthesis. NeurlPS, 2021.
[33] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
[34] Arsha Nagrani, Paul Hongsuck Seo, Bryan Andrew Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. In ECCV, 2022.
[35] Charlic Nash, João Carreira, Jacob Walker, lain Barr, Andrew Jaegle, Mateusz Malinowski, and Peter Battaglia. Transframer: Arbitrary frame prediction with generative models. arXiv preprint N203:22003.0444,, 2019.
[36] Alex Nichol, Prafulla Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 12.1041,, 2021.
Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc-
[37] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev. Latent video transformer. arXiv preprint arXiv:2006. 10704, 2020.
[38] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25. 2022.
[40] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos. arXiv preprint arXiv: 1XXi::442.6604 2014.
[41] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scal- ing up models and data with t5x and seqio. arXiv preprint 2rX3v:2203.17188 2022.
[42] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv: 1487, 2022.
12

---

Published as a conference paper at ICLR 2023
[43] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. In Proceedings of the IEEE international conference on computer vision, pages 2830-2833 2017.
[44] Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, gener- ate densely: Memory-efficient unsupervised training of high-resolution temporal gan. Interna- 2020.
tional Journal of Computer Vision, 128 (28(10)2558-220)
[45] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
[46] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3626-3636, 2022.
[47] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 1rXiv:121..04000 2012.
[48] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using lstms. In International Conference on Machine Learning, 2015.
[49] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018.
[50] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & chal- lenges. arXiv preprint urXiv:
2018
[51] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurlPS, 2018
[52] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoc V Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. In Ad- vances in Neural Information Processing Systems, pages 81-91, 2019.
[53] Vikram Voleti, Alexia Seficoeur-Martnnen and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.08 09853, 2022.
[54] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dy- namics. arXiv preprint arXiv:1609.02612, 2016.
[55] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint 10X.v:2103.01950, 2019.
[56] Yunbo Wang, Mingsheng Long, Jianmin Wang, Zhifeng Gao, and Philip Yu. Predrnn: Re- current neural networks for predictive learning using spatiotemporal Istms. Advances in neural information processing systems, 30, 2017.
[57] Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video mod- els. In ICLR, 2020.
[58] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li. Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
[59] Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis. arXiv preprint arXiv:2207.09814 2022.
[60] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NÜwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022.
13

---

Published as a conference paper at ICLR 2023
[61] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXi:21004.0057, 2019.
[62] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. arXiv preprint arXiv:2203. 09481, 2022.
[63] Fangchen Liu Stephen James Pieter Abbeel Younggyo Seo, Kimin Lee. Harp: Autoregressive latent video prediction with high-fidelity image generator. arXiv preprint arXiv:2209.07143, 2022.
[64] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. ICLR, 2022.
[65] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Va- sudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[66] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awarc implicit generative adversarial networks. arXiv preprint 2rXiv:2202.10577 2022.
[67] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 12104-12113, 2022.
[68] Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. CVPR, 2018.
14


Reviews:
[{"review_id": "lL2erRFpr7", "reviewer": "Reviewer_E9Bs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article presents, Phenaki, an approach to text-to-video modelling and synthesis based on an autoregresserve transfomer-based encoder-decoder model. \n\n# Stage 1. C-ViViT:\nFrames from a given video are first linearly projected into patch-embeddings then processed by a spatial transformer; the resulting output of the spatial transformer is further processed across time with a causal (temporal) transformer. \n\n# Stage 2. MaskGIT\nThe outputs of the causal transformer are project to a discrete latent space via a learned codebook. A bidirectional transformer is trained via a self-supervised reconstruction mechanism (masking) to reconstruct sequences of the tokens output by the causal transformer. This is done by taking the discretized outputs of stage 1 and randomly masking (according to some procedure) parts/subsets of these causal tokens. The bidirectional transformer is trained to undo the masking operation and reconstruct the original unmasked token sequence. The bidirectional transformer also conditions this reconstruction process on text information (ie. prompt) provided by consuming embeddings provided by a pretrained language model.\n\n# Stage 3: Text driven Video Generation\nOnce all these pieces come together, generation occurs via providing empty/blank tokens and a text prompt to the bidirectional transformer. The bidirectional transformer outputs a set of tokens that align with those in the codebook, these tokens are projected back to the image space by an image decoder that is the inverse of the ViViT procedure and architecture. Auto-regressive generation can be achieved by shifting the tokens generated by the bidirectional transformer forward by one time-step, appending an empty/blank token for the new time step and providing that to the bidirectional transformer along with the text prompt.\n", "summary_of_the_review": "Overall this is a great piece of work and a significant contribution to the field. If the authors remedy the issues i raised as weaknesses then I will increase my score. But as is, I support the publication of this work at ICLR.", "strength_and_weaknesses": "## Strengths:\n1. The chosen approach allows for unprecedented flexibility and performance in conditional video generation tasks.\n\n2. It is a simple extension of existing architectures from other domains (video classification, image generation)\n\n3. A good awareness of the ethical considerations with regard to this kind of research and its potential misuses.\n\n\n## Weaknesses: \n1.  **Technical Novelty**: The technical novelity in this article is quite limited. Phenaki is essentially a merger of MaskGIT and ViViT (and T5), the ensuing capabilities and advantages of this model come from the capabilities of the base models being used together. The technical novelity in the paper is limited to intelligent architectural choices. Of note is the addition of text conditioning to the bidirectional vision transformer and the change towards a causal transformer for better temporal modelling of video, allowing for unconstrained video generation.\n\n2. **Code (and Demo) is not available**: The model itself is not dangerous but its uses could be. I would agree with keeping the trained models closed source since they could immediately be misused by the lay individual. But keeping the code closed source does not benefit the field as a whole and prevents researchers, reviewers and other academics from carrying out accurate experiments, bechmarks and comparisons. There are already several attempts at reproducing the model already. It would be interesting to see what the ethical reviewers have to say with regard to this argument. For the sake of accurate benchmarking and comparisons, I would urge the authors to open source their code since the stated reasons for keeping it closed source (i.e. \"understanding of data, prompt and output filtering\") are not impacted by this.\n\n3. **Clarity**: Figure 2 is incomplete and the explanation of encoder-decoder model unclear.  It can be argued that the decoding process is one of the most important procedures in this work yet it is barely covered. It is unclear whether decoding and generation primarily happens in the latent space? are the outputs from previous timesteps decoded and then re-encoded and fed into the bidirectional transformer? can the entire decoding process for several timesteps purely occur in the latent space? \n\n4. **Missing Sections** : The computational requirements and cost of training such models is of increasing interest to the wider community. There are references throughout the paper to computational cost and efficient but this topic is never fully explored. It would be good to get some idea of computational requirements, from the equipment used for the experiments to (tpu/gpu hrs/days, flips), to wall-clock inference speed and time (per frame, time-step, etc). This is not in the article or its associated supplementary material.\n\n5. **T5X**: There is mention of discussing each of the model components in the paper in the last line of the first paragraph in section 2. But T5X is not elaborated on. What is T5X? Is this the original T5 model repackaged? Is it the T5-XXL (4.6B) model?\n\n6. **LAION**: Mentions of the dangers of the LAION-400M dataset are provided without reference. The work that raised this issues should be directly referenced within that paragraph (ie. [Multimodal datasets](https://arxiv.org/abs/2110.01963)).\n\n7. **UCF-101** : The standard UCF-101 benchmark is missing and should be included. It is a tractable and accessible benchmark that allows for better comparison to prior art within the video generation field. I encourage the authors to have a look at Table 1.d in the [TATS](https://arxiv.org/abs/2204.03638) paper. The model presented in this work should be evaluated on base level video generation ability and class-conditional video generation capability. Whether the text prompt is a sentence or a word/action should not present a challenge but such benchmarks are useful for evaluating base level capabilities of such models. In the case of using pre-trained models, it should be highlighted that the model is pre-trained on a range of other datasets rather than the standard practice of training on only the benchmark dataset. It would also be useful to know of its base-level long horizon [video prediction](https://wilson1yan.github.io/teco/) capabilities and dynamics.\n\n\n\n### Missing References:\nThe field of video generation has existed for some time now and was long established within the GAN world. References to this prior art are missing. These include but are not limited to: \n\n- [Long Videos of Dynamic Scenes](https://arxiv.org/abs/2206.03429)\n- [RV-GAN](https://openaccess.thecvf.com/content/CVPR2022W/WiCV/html/Gupta_RV-GAN_Recurrent_GAN_for_Unconditional_Video_Generation_CVPRW_2022_paper.html) - CVPR-W 2022\n- [TATS](https://songweige.github.io/projects/tats/index.html) - ECCV 2022\n- [StyleGAN-V](https://openaccess.thecvf.com/content/CVPR2022/html/Skorokhodov_StyleGAN-V_A_Continuous_Video_Generator_With_the_Price_Image_Quality_CVPR_2022_paper.html) - CVPR 2022  \n- [LDVD-GAN](https://www.sciencedirect.com/science/article/pii/S0893608020303397) - Neural Networks 2020\n- [DIGAN](https://arxiv.org/abs/2202.10571)\n\nI would recommend the authors have a look at Section 2, paragraph 2, of the [DiGAN](https://arxiv.org/abs/2202.10571) paper covering prior video generation work.\n\n**Concurrent work** such as [Imagen](https://arxiv.org/abs/2210.02303) and [Make-a-video](https://arxiv.org/abs/2209.14792) should also be refenced in the revised article.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NQreSILw0X", "reviewer": "Reviewer_2JKT", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new video generation method from given text prompts, called Phenaki. The proposed method address four tasks such as text conditional video generation, text-image conditional video generation, time-variable text-conditioned video generation, video prediction, and image conditional video generation. For achieving this, the authors advanced ViViT to C-ViViT and designed Transformer-based decoder. They compare their method with multiple state-of-the art methods including NUWA, NUWA-Infinity, and CogVideo. The results look promising. \n", "summary_of_the_review": "Overall, I like this paper. Although there are some room for improvement (please see the weakness), I decided to give accept as score considering the meaningful contributions of this paper.  ", "strength_and_weaknesses": "### Strength\n- Video generation from text prompts is a challenging and innovative research topic. In particular, time-variable video generation from text condition is a very meaningful achievement.\n- The proposed method including C-ViViT and Transformer-based decoding seems a little novel. C-ViViT looks beyond simple extension of ViViT.\n- The authors provide the results of five meaningful tasks, which are very promising and impressive.\n- The paper is easy to read and clear.\n\n### Weakness\nOverall, I like this paper. But there are some room for improvements.\n- [Major] Despite its impressive results, more ablation studies or analyses on key hyperparameters are required. For example, the number of video tokens seem have influence on the quality and compuational cost (both training and inference). The trade-off analysis is helpful. The results on maksing ratio and predicted token ratio can help to understand the model properties. \n- [Major] In addition to quality metrics, the results on inference time can enhance the contributions. \n- [Minor] How is the performance on few shot or fine-tuning of Phenaki in Table 1?\n- [Minor] How long can the model consistently or stably generate videos for prompt sequences when generating time-variable generation?\n- [Minor] Why did the author use 50%/50% setup for Table 1 instead of 80/20 setup even if 80/20 is better than 50/50?\n- [Minor] TATS [Ge et al. 2022] needs to be added in related work. The comparison is not required because it is published in ECCV 2022. Also, because the authors present image conditional video generation, some video generation works need to be added in the related work such as StyleGAN-V [Skorokhodov et al. 2022] and DIGAN [Yu et al. 2022].\n\n### References\n- Ge et al. [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://arxiv.org/abs/2204.03638). ECCV 2022.\n- Skorokhodov et al. [StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2](https://arxiv.org/abs/2112.14683). CVPR 2022.\n- Yu et al. [Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks](https://arxiv.org/abs/2202.10571). ICLR 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DwEOVBFpuR", "reviewer": "Reviewer_Qmqe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies text-to-video generation. It proposes an architecture based on the VQGAN framework. For the encoder-decoder part, a novel ViT-like model is introduced to encode both videos and images into discrete tokens, which allows joint training on the text-image dataset. For the sequence modeling part, a MaskGiT-like model is trained to predict video tokens. The authors show the effectiveness of the proposed framework as well as joint training on image / text conditional generation with qualitative and quantitative evaluation. ", "summary_of_the_review": "This paper shows improved results on open-domain text-to-video generation using VQGAN-like framework. It shows impressive results when conditioning on a variable length of open domain texts. However, the paper misses important baselines and contains some unconvincing yet tempting conclusions. If the authors can resolve these concerns and add the corresponding experiments and discussions, I would like to change my current rating.", "strength_and_weaknesses": "[Strengths]\n\n1. The idea to extend ViViT to C-ViViT for encoding videos and images into discrete tokens is reasonable and novel.\n2. Adapting the MaskGiT-like framework to video VQGAN models is novel.\n3. The ablation study of the image-video joint training is valuable to the community.\n4. The quality of the generated short and long videos based on texts is much better than the previous arts.\n\n[Weaknesses]\n\nThe paper misses an important baseline [1] which also shows results in generating videos based on variable prompts. [1] shares a similar goal of generating long videos with a similar video VQGAN architecture. Therefore, I think it is important to discuss and compare this method, especially on the following points:\n\n1. [1] also leverages a VQGAN framework for video generation by extending the auto-encoder part with 3D convolutions. It would be necessary to compare with it in Table 3 on video reconstruction. Currently, only 2D convolution and per-frame ViT models are compared. It is not clear that C-ViViT works better than 3D convolution.\n2. [1] also show video results conditioned on variable length of text prompts, which means the argument that this paper is the first to study this problem is not true. It is more reasonable to say this is the first to study generating videos from variable lengths of **open-domain** text prompts, since I believe [1] only shows results on a limited domain. \n\nIncluding the above discussion should make the paper more complete and a stronger submission. In addition to the concern above, some of the experimental results in the paper are not convincing:\n\n1. The authors claim that the video dataset does not contain any stylized videos like pencil drawings which makes such videos impossible to be learned solely from the video dataset. However, in the shown qualitative results, some difficult prompts like \"astronaut riding a horse\" can still be generated on Mars. Can the authors show examples of the \"pencil drawing styles\" with a model trained on video datasets only?\n\n2. One advantage of using MaskGIT claimed in the paper is the speed of sampling longer sequences. However, there are no experiments showing that, and no ablation on the hypermeters $\\gamma$, $\\beta$, and sampling steps which are claimed to be important in the paper. \n\n[1] Long video generation with time-agnostic vqgan and time-sensitive transformer. ECCV 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2-Vx16OKWLZ", "reviewer": "Reviewer_xbMS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces masked visual token modeling (MVTM) for video generation by extending ViT-VQGAN and MaskGIT into the video domain. The modified ViT-VQGAN tokenizer for video has two blocks with spatial- and temporal attention. To mitigate the nature of fixed code resolution in image (or video) tokenizer, it modifies full attention in the temporal blocks into causal attention. MaskGIT-like masked token modeling learns the distribution of videos represented as the sequence of tokens. The novelty of this work lies in the joint leaning on image and video datasets. Examples in the paper demonstrate that those combination generalize well on ", "summary_of_the_review": "This paper introduces a straightforward extension of masked visual token modeling for the video domain. By designing proper settings for video, the proposed method can generate high-quality, diverse-length videos from the various text description. The current version of the paper makes me confused about some details of the works, However, I believe that the overall work is quite fascinating for future research on video synthesis.", "strength_and_weaknesses": "### Strength\n**appropriate modifications of previous works for video generation**\n\nThis paper modifies ViT-VQGAN and MaskGIT for video generation. These modifications seem to be plausible to process the video data. In the aspect of masked visual token modeling for video, this approach is fancy and new although implemented architectures are borrow from the previous works\n\n**joint learning of image and video datasets**\n\nThis work demonstrate how to use both image and video datasets for video generation. I think that this idea is crucial for future research on video sythesis. I hope that there will be more analysis on this idea.\n\n### Weaknesses\n\n**validity of the statement (novelty over the ViViT architecture)**\n\nI think that the descriptions in the last 4 sentences in **Novelty over the ViViT architecture** are wrong. The main statement here insists that causal attention is needed to support arbitrary temporal length. As I understand, C-ViViT architecture with all-to-all attention (I understand bidirectional attention) still supports encoding and decoding for arbitrary temporal length. (If it is not the case the bidirectional transformer can not model the masked token sequences.) If my conjecture is right, this work should discuss the difference between all-to-all and causal attention. For example, comparing the reconstruction performance differences between two attentions.\n\nIn this aspect, long video generation is not related to this video tokenizer, it is a matter of auto-regressive generation using bidirectional transformer inspired from MaskGIT. These confusing descriptions appear several times on the paper. Do I understand wrong about the proposed method?\n\n**analysis on joint learning of image and video datasets**\n\nIt is not a weakness but suggestion. If possible, analyze the vocabularies for image and video dataset. For example, we can use this analysis to generate video with the text with out-of-vocaburary in video dataset and in-vocaburary in image dataset and check the quality of this video.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster