Seer: Language Instructed Video Prediction with Latent Diffusion Models
Authors: Xianfan Gu
Chuan Wen
Weirui Ye
Jiaming Song
Yang Gao
Venue: ICLR 2024 poster (2024)
Keywords: video prediction
multi-modality generation
diffusion model
language-vision model
Primary area: generative models

Abstract: Imagining the future trajectory is the key for robots to make sound planning and successfully reach their goals. Therefore, text-conditioned video prediction (TVP) is an essential task to facilitate general robot policy learning.
To tackle this task and empower robots with the ability to foresee the future, we propose a sample and computation-efficient model, named Seer, by inflating the pretrained text-to-image (T2I) stable diffusion models along the temporal axis. We enhance the U-Net and language conditioning model by incorporating computation-efficient spatial-temporal attention. Furthermore, we introduce a novel Frame Sequential Text Decomposer module that dissects a sentence's global instruction into temporally aligned sub-instructions, ensuring precise integration into each frame of generation. Our framework allows us to effectively leverage the extensive prior knowledge embedded in pretrained T2I models across the frames. 
With the adaptable-designed architecture, Seer makes it possible to generate high-fidelity, coherent, and instruction-aligned video frames by fine-tuning a few layers on a small amount of data. The experimental results on Something Something V2 (SSv2), Bridgedata and EpicKitchens-100 datasets demonstrate our superior video prediction performance with around 480-GPU hours versus CogVideo with over 12,480-GPU hours: achieving the 31\% FVD improvement compared to the current SOTA model on SSv2 and 83.7\% average preference in the human evaluation. Our project is available at https://seervideodiffusion.github.io/

Published as a conference paper at ICLR 2024
SEER: LANGUAGE INSTRUCTED VIDEO PREDICTION WITH LATENT DIFFUSION MODELS
Xianfan Gu' Chuan Wen Weirui Ye 1,2,3 Jiaming Song+
Yang Gao 423,
IIS. Tsinghua University SShnggai Artificial Intelligence Laboratory "Shnnghai Qi Zhi Institute *NVIDIA
guxfesqz ac {cwen20, ywr20, gaw.arggi1s/emmllss.smo.... j slamins@nvvddia.com
ABSTRACT
Imagining the future trajectory is the key for robots make sound planning and successfully reach their goals. Therefore, text-conditioned video prediction (TVP) is essential task to facilitate general robot policy learning. To tackle this task and empower robots with the ability to foresee the future, we propose a sample and computation-efficcent model, named Seer, by inflating the pretrained text-to- image (T2I) stable diffusion models along the temporal axis. We enhance the U-Net and language conditioning model by incorporating computation-efficien spatial-tempont attention. Furthermore, we introduce a novel Frame Sequen- tial Text Decomposer module that dissects a sentence's global instruction into temporally aligned sub-instructions, ensuring precise integration into each frame generation. Our framework allows us to effectively leverage extensive prior knowledge embedded in pretrained T2I models across the frames. With the adaptable-desinee architecture, Seer makes possible to generate high-fidelity. coherent, and instruction-aligned video frames by fine-tuning few layers small amount of data. The experimental results on Something Something V2 (SSv2), Bridgedata and EpicKitchens 100 datasets demonstrate superior video prediction performance with around 480-GPU hours versus Video with over 12,480-GPU hours: achieving the 31% FVD improvement compared the cur- rent SOTA model on SSv2 and 83.7% average preference in the human evaluation. Our project available htttts:: seee/eeeeevieee /ttp:///eeeeeeeeeeeeeeeeee.e fuuiion..f github..hh..... 1o/
INTRODUCTION
"removing oeer con, revenling betaus"
Text-conditioned Video Prediction (TVP), task that generates future video frames con- ditioned on few frames and language in- structions, crucial for diverse downstream tasks requiring instruction alignment and tem- poral consistency from an initial environment. For instance, in video editing, TVP empowers the generation diverse temporal movements from an input video clip, guided by range of language instructions. Importantly, TVP can selectively extend video segments while pre- serving temporal consistency from the input video. TVP also plays an important role in the scenario of robot learning. TVP samples coherent future frames with aligned motion trajectories based on the initial state of a robot, providing task-level visual guidance for long-horizon planning. This over- comes the challenge for robot align abstract language instructions with long-horizon operations. Consequently, learning a TVP model is fundamental task to achieve both temporal consistency in the transition distribution of the initial state and alignment between task-level language instructions and video motion, facilitating the development of video foundation models.
ref.
Seer
"witiig o Powerpot novt to
Seer is efficient video diffusion model that uses natural language instructions and reference frames (ref.) predict multiple varia- tions of future frames.
Figure 1:
"Corresponding Author.

---

Published as a conference paper at ICLR 2024
pon-mmieed conditiornal
Pre-tramedd 121
Safart
Offiusion Process
Spetial Conv2D
latent
infiated 3D U-Net
Inflated
D
SANVT: ArtS
Attmi
SHOTT
Aria
Altan
Conv3D
Allssoo
Scaled Autorggrasion Window
Temporal Atteniton (SAWTT-fctn)
"spinning cube"
G.ip
1o-kod Decoerscoe
Sep-aermctio
BMD Vestruction enibediiing
(a]Pipeline
Frozen
(b) Inflated 3D modules
Figure 2: (a) Seer's ' pipeline includes an Inflated 3D U-Net for diffusion and Frame Sequential Text Transformer for text conditioning. (b) Our Inflated 3D U-Net expands the pre-trained 2D Conv kernel to 3D kernels and connects with the Sealed Autoregressive Window Temporal Attention layer.
Despite its potential benefits, text-to-video prediction (TVP) is a challenging task because it requires deep understanding of the initial frames, the natural language instruction, and the grounding be- tween language and images, while predicting based upon all the information above. In contrast to traditional text-to-video generation tasks (Wu al. 202T; 2022a Hong et al. 2023 Ho al. 20226 Singer et al., 2022), which do not explicitly condition on initial frames, TVP requires a model to synthesize predictions based on the given initial frames and textual instructions. Merely generating few prototypical videos corresponding to the input text is no longer viable solution in TVP: the task necessitates a more detailed comprehension of temporal movement. Besides, the existing text- to-video generation task usually aims to generate short horizon video clips with text specifying the general content, such as "a person is sking", " while our aim in the TVP task is to use the text as a task descriptor, such as "tipping a beer can",, as shown in Figure
Specifically, there are mainly three problems limiting the performance of the TVP task: 1) Require- ment for large-scale labeled text-video datasets and expensive computational cost: learning to capture the correspondence between two different modalities is non-trivial and needs large amounts of supervised text-video pairs and excessive computation overhead for training. 2) Low fidelity of generated frames: the frames generated by the models are usually blurry and cannot clearly display the background and objects specified in the reference frames. 3) Lack of arneigr instruction for each frame in the task-level videos: the goals specified by text instructions are usually in the task level, making difficult to understand the progress and generate the corresponding frame in each timestep only conditioned on global text embedding. To address these issues, we propose Seer: a TVP method capable of generating task-level videos according to the text guidance with high data and training efficiency.
Motivated by the recent progress on generative models (Rombach al. 2022 Ramesh al. 2022 we propose to leverage text-to-image (T2I) latent diffusion models (Rombach et al.) 2022) for the TVP tasks. T2I models are pretrained with billions text-image pairs crawled from the inter- net (Schuhmann et al. 2021). They have acquired rich prior knowledge and thus are able generate high-quality images corresponding to the text descriptions. Therefore, inheriting such prior knowl- edge by inflating T2I model along the temporal axis and fine-tuning it with small vixeo dataset is an appealing solution for TVP tasks, which relieves the requirement for extensive labeled data and computational overhead, i.e., Problem 1.
Since the T2I models contain two modalities: image and language, we propose to inflate these two parts to generate high-quality video frames and tine-grained text instruction embeddings for each timestep respectively. For the visual model, we extend the 2D latent diffusion model (Rombach et al.) 2022 to data and computation-efficient 3D network to model spatial dependencies and the temporal dynamics simultaneously. which is called Inflated 3D U-Net. By taking advantage of joint modeling of spatial and temporal dimensions, as well as autoregressive generation, we successfully synthesize coherent and high-fidelity frames, which alleviates Problem 2. As for the language module, in contrast to existing approaches (Zhou et al. [022] Wu et al., 2022b) that encode one text embedding for the whole video with text encoder, we propose to decompose the single text instruction into fine-grained guidance embeddings for each time step. We achieve the automatic decomposition by Frame Sequential Text (FSText) Decomposer based on the causal attention mechanism. By temporally splitting the instruction into different phases, Seer improves the guidance embeddings for each frame and thus enables task-level video generation (Problem 3).
We conduct extensive experiments on Something-Somethhng V2 (Goyal et al. 2017), Bridge Data (Ebert et al. (02T) and Pitic-Kitchens-1000 (Damen et al. (02T datasets. We outperform all the
2

---

Published as a conference paper at ICLR 2024
baselines, such as MCVD (Voleti et al. 2022). TATS (Ge et al. 2022) and Tume-A- Video (TAV) (Wu et al., 2022b), and achieve state-of-the-ee art performance in terms of FVD and KVD. Especially we improve FVD from 163 to 113 on maaethiing-Smethiing V2, compared to the SOTA TAV. Compare to over 480 hours with 13 x 8 A 100 GPUs in CogVideo (Hong et al.) 2023), the experiments show the high efficiency of method: 120 hours with 4 RTX 3090 GPUs The ablation studies illus- trate the effectiveness of our computation-efficien FSText Decomposer. Furthermore, our method supports video manipulation by modifying the text instructions, and we demonstrate our superior generation quality through a human evaluation study, showing more than 70% preference over TAV and around 90% preference over TATS and MCVD.
vidco-inflated model with the newly proposed
RELATED WORK
2.1 TEXT-TO-IMAGE GENERATION
Since Scott Reed al. (Reed et al. 2016) firstly set up the T2I generation task and proposed GAN-based method, this multi-modal generation task has attracted the attention of the computer vision community. DALL-E (Ramesh et al., 202T) makes breakthrough by modeling the T21 generation task as a sequence-fo-sequence translation task with a VQ- VAE (Van Den Oord et al.) 2017) and Transformer (Vaswani al. (0017). Since then, many variants have been proposed with an improved image tokenizer (Yu et al. domain-specific knowledge (Gatni al. Probabilistic Models (DDPM) (Ho et al. T2I generation tasks (Nichol et al. 2022, Saharia al.) 2022; Ramesh al., 2022). Specifically, GLIDE (Nichol et al., 2022) proposes classifier-free guidance for T2I diffusion models to improve image quality. For a better alignment between text. and image, DAILL-E 2 (Ramesh et al.) 2022) proposed denoise CLIP (Radford et al., (2021) image embedding conditioned on CLIP text em- bedding, which integrated high-level semantic information. To reduce the computation cost of the denoising process in pixel space, Latent Diffusion Model (LDM) employs VAE (Kingma & Welling 2014) to operate in the latent space. Seer takes advantage of the prior language-vision knowledge of pretrained LDM and inflates it along the time axis.
2022a). hierarchical Transformers (Ding et al. 2022) or
2022). With the recent progress of Denoising Diflusion
2020), the diffusion models have been widely used for
2.2 TEXT-TO-VIDEE GENERATION
In contrast to the huge success of Text-to-Image (T2I) generation, Text-to-Video (T2V) generation is still underexplored due to the limitation of the large text-video data annotation and computing resources. Inspired by the various variants of T2I generation, recent T2V studies have attempted to explore compatible variants for video generation modeling. GODIVA (Wu et al. 202T) first proposes VQ-VAE based rugo-segressive model with fince-dimensioall sparse attention for T2V generation. NÚWA Wu et al. 2022a) further improves it by designing a 3D encoder with 3D nearby attention and achieves competitive performance multi-task generation. Unlike the single frame-rate T2V approaches trained from scratch on large-scale text-video datasets, Cog Video (Hong et al. (023) proposes -ult hierarchical model for T2V generation. This approach leverages the pre-trained module of T2I Cog View-2 (Ding et al. 2022).
Motivated by the remarkable progress of T21 diffusion models (Ramesh al. 2022 Saharia al.) 2022 Rombach et al., 2022). Make-A- (Singer et al. 2022). Magic Video (Zhou et al. 2022). Tune-A- A-Video (Wu et al. (002b)) and Imagen Video (Ho al.
2022a) transfer the 2D diffusion models to 3D models by incorporating temporal modules in T2V generation. In contrast to Imagen Video, all other three methods utilize the prior knowledge of T2I pre-trained model. Similarly, we use the pre-trained weight of the 2D T2I diffusion model in our 3D T2V model. Varying from the aforementioned methods, our method Seer utilizes autoregressive attention on both spatial and tem- poral spaces to generate high-fidelity and coherent video frames. And Seer is able to handle the task- level video prediction by decomposing the language condition into fine- mne-grained sub- instruction.
3 PRELIMINARIES
Denoising Diffusion Probabilistic Models with classifier-free guidance: Diffusson models are probabilistic models that approximate the data distribution by iteratively adding noise and denoising
3

---

Published as a conference paper at ICLR 2024
through forward/teverse Gaussian Diffusion Process (Ho al. 2020 Song al. 2021 The forward process applies noise at each time step t E O.... T to the data distribution Xo. creating noisy sample x, where X, Vā.xp vI ā,E (e _N I)). and ā, is the accumulation of the noise schedule ao: defined by ā₁ II ag. To denoise images, the diffusion process uses a reparameterized variant of Gaussian noise prediction eg Ex(x,, () targeting Gaussian noise e. The reverse process p(x;-1|x;) of the Markov Chain generates new samples from Gaussian noise, which is approximated by Bayes' theorem as (xx Xu), where is derived from the forward process Xo
Va. (x: V1 (x,6 t)).
Classifier-free guidance Ho & Salimans 2022 introduced for conditional diffusion models to generate images without requiring an extra image classifier. A conditional model with a parame- terized reverse process - Xx, c) uses a conditional identifier through ex t, c). To predict an unconditional score, the conditional identifier is replaced with a null token Ô and denoted as co(x,, f, c Ø). Classifier-free guidance can then be approximated as a linear combination of conditional and unconditional predictions:
Eo(x;, 1, c) = (1 + w))c i, c) weo(x,, 0),
where w is the guidance scale. Text-video and ext-imagee-asse diffusion models (Rombach et al., 2022 Saharia et al. [202) Nichol al., [202]] Ho et al.) [022b] [Singer et al. 2022) use DDPM with classifier-free guidance. This diffusion method can be adapted to various tasks with flexibility.
Latent Diffusion Models: Compared with image diffusion, video diffusion has significantly higher computation costs because it needs to process multiple frames. Recent works have explored the computation-efficie version of diffusion modeling, such as latent diffusion model (LDM) (Rom-) bach et al. 2022). LDM proposes the VAE-based latent diffusion, including a KL-regularized au- toencoder for encoding/decoding latent representation e(x), ; and diffusion model to operate on the latent space z, For the conditional generation, LDM introduces domain-specine encoder Ty the projection of condition y for various modality generations. Thus, the objective of LDM is:
LLDM = E, [...()))
||e (z t, (y
4 METHODOLOGGY
In this paper, we aim to explore an efficient diffusion method to predict coherent video frames guided by language instructions, which requires learning parse natural language, understand the scene, and ground the language and scene together. However, it is challenging to directly apply conventional video diffusion models for TVP due the following problems: (1) The limited labeled text-video data and computational resources. (2) Low fidelity of frame generation. (3) Lack of fine- grained instruction for each frame in the task-level videos.
4.1 OVERVIEW OF SEER
Motivated by the robust generative capabilities of text-to-image (T2I) diffusion models, we lever- age the prior knowledge implied in pretrained T2I models by inflating the 2D U-Net (Rombach) al. 2022) and incorporating temporally consistent layers. However, the inflated video diffusion model guided solely by coarse global language instruction tends to generate irrelevant T2I outcomes and fails maintain temporal coherency between video frames. To address this limitation and provide precise and controllable guidance for our inflated model, we introduce a novel temporal decomposition component for language instruction, this component decomposes global instruction as temporally aligned sub-instruction for delicate laskl guidance, which significantly enhances the fidelity and coherency of predicted video.
Our Seer method comprises two main components: the video diffusion and the language condition- ing modules. We propose to enhance these two components to facilitate high-fidelity frame synthesis and the temporal alignment of text instructions, respectively. Specifically, as shown in Figure 2 (a), we utilize two pathways to implement the conditional diffusion process guided by reference frames and language: 1) We incorporate the spatial-temporal module discussed in Section 4.2 into the In- flated 3D U-Net. This integration enables the propagation of contextual information from reference frames to future frames within the spatial-temporal space, allowing for coherent motion prediction based on the reference frames. 2) To plan continuous motion with fine- ane-ggrined language guidance, introduce Frame Sequential Text (FSText) Decomposer in Section 4.3) This module transforms
4

---

Published as a conference paper at ICLR 2024
global language instructions into Subsequently, we inject these wraee-wise subinstruction tokens into the intermediate latent space of the video frames at each time step. With this design, merely train the spatial-temporal layers and FSText module from scratch while freezing the remaining pretrained modules within our 31) inflated U-Net. These two modules are jointly trained by the diffusion objective, where fo is FSText decomposer, T is the frozen CLIP text encoder, and is the input text:
sub-instructions that are synchronized with video.
Liiffusson -e [|le co(z,, (o((()))))]
(3)
4.2 DATA & COMPUTATION 3D NETWORK
To design a computation-effiiin
visual backhone for our video diffusion model, we refer to some
relevant works on lifting 2D 3D video modeling (Carreira & Zisserman 2017) and efficient at- tention computation (Liu et al. 2021 2022). In general, we leverage the latent diffusson model (LDMs) pretrained on T2I tasks build a text-video model. Our inflated 3D U-Net consists of two principal components as illustrated in Figure 2 (b): 1) The 3D spatial layers, where we draw inspiration from I3D (Carreira & Zisserman 2017) and enhance the 2D convolution kernel from (3 x 3) to a 3D counterpart (1 x 3 3) with added video frames axis from the pre- trained 2D modules, consisting of series
2D ResNet blocks and Spatial Attention Blocks. 2) The temporal layers, play crucial role in our visual backbone for propagating contextual information from the reference frame's image prior across the temporal sequence. We investigated various temporal attention and incorporated them into our 3D U-Net architecture. Our empirical observations indicate that bi-directional temporal attention tends to disregard guidance from ref- erence frames, and both bi-directional and directed tem- poral attention struggle to capture dependencies among spatial regions, as discussed in Section [5.5] To ad- dress these limitations while reducing complexity, we employ efficient approach that builds upon the con- cept of window attention (Liu et al. 202I) in 3D space: the implementation of local window attention in an dimen- establish fixed lo-
Bi-directonnal
Directed Temporal Attention
Temporal
Autoregressvee
Scaled Autorignessive
Spatial- Pemiill-emmrrll Attertion
Window Temporal Attentton
Figure 3: Variants of temporal attention,
only the blue tokens attend to the current token in the red box. Red dashed arrows indicate the direction of attention. And the orange boxes indicate the local win- dow region (2 window in this case).
autoregressive manner across spatial-tempont
sions. As illustrated in Figure cal windows for each spatial region with a window size of relative to the global frame sequence Within this framework, we compute self-attention using a causal mask, considering both local spatial and global temporal dimensions within the 3D space. This effectively constrains pixel propagation from the future camporal-spatial sequence.
Finally, We maintain the acquired knowledge from the 2D modules by freezing all pretrained weights and exclusively training the spatiotemporal attention layers during fine-tuning. Overall, through a combination of frozen pre-trained spatial layers and lightweight spatiotemporal layers, our inflated 3D U-Net not only retains crucial knowledge but also enhances fine-tuning efficiency.
4.3 FRAME SEQUENTIAL TEXT DECOMPOSER
For the language conditioning module, since our 3D inflated U-Net is built upon a pretrained text- to-image model, we noticed that using a text-to-image prior alongside a global instruction tends to provide strong semantic guidance. which can override the scene in reference frames, deviat- ing from the intended guidance for prediction based on the existing scenes. To address the above limitation and better capture long-term dependencies from both text and reference frames, we in- troduce the Frame Sequential Text (FSText) Decomposer. This novel approach decomposes the global instruction into fine-grained sub-instructions, aligning with each frame. We further explore the interpretability of sub-instruction embeddings in Section 5.6 To derive a sequence of tempo- rally aligned sub-instruction embeddings from the global instruction generated by the CLIP text encoder Radford al. (2021), we employ transformer- based temporal network designed to ful- fill three essential properties for meaningful uub-instructions: 1) Contextual aggregation, which ensures that the inner tokens of each sub-instruction aggregate contextual information within the
5

---

Published as a conference paper at ICLR 2024
Init. weights optimization (Identity)
2. diffusion process (Cross-Atten.)
global instruction embedding
with
prompts
1000 --- 1l
Wi
sequence!! !
sub-instruction embedding
Bemaable tokens
fhoee
Ro Reemmes
FSTaat
W
Teats Crows-eet
Decomposer
Text Sog Lengeh
Diflusion Process
SuB Inttocttion
frames
Frame Sequential lest Decomposer |FSleet Decomposer
frames
embraiskog
Figure 4: Frame Sequential Text Decomposer is shown in (a). We start by initializing the weight of the network to project identity vectors from CLIP text tokens. We then optimize the generated text tokens via the diffusion process (b), where frame-individual cross-attention is denoted by "fic-attn."
sentence. 2) Semantic inheritance, the semantic information of these sub-instructions is inher- ited directly from the global instruction 3) Temporal consistency ensures alignment between the sub-instructions and the time sequence, thereby facilitating the generation of temporally consistent video. Based on these properties, our network consists of three key components: a) To achieve the property of contextual aggregation, we employ Text-Sequential Attention, akin to BERT, a bidirectional self-attention layer (Devlin et al. 2019) to capture global dependencies among dif- ferent positions within text sentences, b) To ensure semantic inheritance, we use Cross-Attention responsible for projecting the global instruction's 's textual sequence onto the inner tokens of each sub-instruction this component ensures that all sub-instruction contain essential global instruction signals for guiding video frame generation. e) To maintain temporal consistency, we adopt tem- poral Attention, a directed attention layer to capture temporal dependencies along the frame axis, which enhances temporal consistency among the generated sub-instructions throughout the video.
Specifically, shown in Figure [ we start with global CLIP text embedding, denoted as (1, C)), where / signi- fies the text sentence length and C' is the channel size, we initialize learnable tokens with shape (n, 1,,,C. C.) where n denotes the number of frames. The tokens are fed into the text sequential attention layer to perform self-attention along the 7 axis. Subsequently, the cross-attention layer employs these learnable tokens as queries and the global text embedding as keys and values, resulting in a one-to- multiple projection from the global text into time steps. This yields (n, I, C') tokens for n frame containing task instruction information. Finally, the tempo- ral attention layer conducts directed attention along the n axis for each token in the textual sequence, transforming the macro-instruction progress into frame-specific guidance.
global text prompt IV
"takingglss from dest"
giobal embechting
step2 cross otter
Aaamoallee takans
In XI XX
stepd: tomp-ottie
stepad Test-sen-attinn
Figure 5: The FSText attention of sub- instruction tokens.
After getting n sub-instruction embeddings corresponding each frame, the next step is to inject this guidance into the diffusion process, which is commonly completed by a cross-attention layer. As shown Figure 4 (b), different from the existing works (Zhou et al.) 2022 Wu et al. 2022b) that calculate the cross-attention between the global instruction embedding and n. frames. In our cross-attention layer, where cross-attentiin is calculated separately between visual latent vectors and sub-instruction embeddings for each frame, and the results from all frames are then concatenated, an attention mechanism we refer to as frame-individual cross-attention (fi--ati).
Initialization We find initialization is critical to FSText decomposer. Especially, the random ini- tialization fails to approximate the distribution of text embeddings in the pretrained '21 model and results in poor performance. To guarantee the sub-instruction embeddings become a close approx- imation of the CLIP text embedding, we employ an initialization strategy by enforcing the FSText decomposer to be an identity function (Note that this initialization step is completed before the diffusion process. We ablate this design in Section 5.5). It can be achieved by this objective:
Lidentitt = [Ifo(T(y)) (
44
EXPERIMENTS
In this section, we evaluate Seer on the text-conditioned video prediction task. We compare against various recent methods and conduct ablation studies on the techniques presented in Section

---

Published as a conference paper at ICLR 2024
5.1 DATASETS
We conduct experiments on three text-video datasets: Something Something-V2 (SSv2) Goyal ee al. 2017), which contains videos of human daily behaviors with language instructions, Bridge- Data (Ebert et al. (02T) that is rendered by a photo-realistic kitchen simulator with text prompts, and EpicKitchens-IOl (Damen al. 2021) (Epic100), which collects human daily activities in the kitchen in egocentric vision with mulg-language narrations. For SSv2, we follow (Soomro et al.) 2012) to evaluate the first 2048 samples during evaluation to save testing time. For BridgeData, we split the dataset into an 80% training set and 20% validation set for evaluation. To reduce complexity, we downsample each video clip to 12 frames for SSv2 and Epic and 16 frames for BridgeData during training/evaluation. Besides, We provide the zero-shot evaluation on EGO4D (Grauman jet al., 2022) dataset in section D.2 Moreover, we also included an additional evaluation on the
UCF-101 dataset (Soomro et al. 2012 in Appendix B.2
5.2
IMPLEMENTATION DETAILS
Gallwring dlpipe Pac
Poino
We use the pre-trained weights of Stable Diffusion- v1.5 (Rombach et al., 2022) initialize the VAE, ResNet Blocks and Spatial Cross Attention layers of the 3D U-Net. We freeze both the pre-trained VAE, and only fine-tune the SAWT-Atten layers in our 3D U-Net. To fine-tune the FSText Decomposer, we ini- tialized it as the identity function of the CLIP text embedding, as described in Section 4.3 We train the models with image resolution of 256 x 256 on Something Something- V2 for 200k training steps, EpicKitchens- 100 and BridgeData for 80k training steps. In the evaluation stage, we speed up the sam- pling process with the fast sampler DDIM (Song) let al. 2020) and conditional guidance of 7.5 for 30 timesteps. See more details in Appendix C
Seer
(Daza)
See
ValedFrssoo
(
VileoTreion
Tune-
Time-A-Vidor
Ref Troms
Wintang ieer
Seer (Ouns
(Our))
VidzoFusce
VideoPaacion
Tuno-A-Viden
Tume-A-Videe
Figure 6: Visualization of Text-conditioned Vidco Prediction with the original (a) and manually modified (b) text prompts SSv2.
5.3 EVALUATION SETTINGS
Baselines. We compare Seer with seven publicly released baselines for video generation (1) con- ditional video diffusion methods: Tune-A- VVideo (Wu et al., 20226), Masked Conditional Video Dif-
fusion (MCVD) Voleti et al. 2022). Video Probabilistic Diffusion Models (PVDM) et al.) 2023)
and VideoFusion et al. 2023); (2) autoregressive-basee transformer method: Time-Agnostic
VQGAN and Time-Sensiine Sensitive Transformer (TATS) et al. 2022) and Make It Move (MAGE)
ct al.
2022b); and (3) CNN-based encoder-decoder: VP Gao et al. 2022).
Machine Evaluation. We evaluate the text-conditioned video prediction of several baseline meth- ods on Something Something- V2 (SSv2) (with 2 reference frames), Bridgedata (with reference frame) and Sppic-Kitchens-100 100 (0pic100) (with reference frame). Additionally, we conduct sev- eral ablation studies of our proposed modules on SSv2. We report the Fréchet Video Distance (FVD) and Kernel Video Distance (KVD) metrics in our evaluation. FVD and KVD are calculated with the Kinetics-400 pre-trained I3D model (Carreira & Zisserman (017). We evaluate FVD and KVD on 2,048 SSv2 samples, 5,558 Bridgedata samples and 9,342 100 samples in the validation sets. For FVD metrics, we follow the evaluation code of VideoGPT (Yan et al. 2021). We further eval- uate the caas-connditooee video prediction of our method on the UCF- 101 dataset and present the comparison results in Appendix B.2
Human Evaluation. Besides evaluating the models on the standard validation sets, we also manually modify the text prompts to provide richer testing results, called text-conditioned video manipulation. Because of the absence of ground-truth frames, conducted a human evaluation of conditiodd video manipula-
Ritse Tano- Videz
Sour 1425
Greil MACVD
60%
75%
20%
Figure 7: Human evaluation results. Preference percentage for TVM task SSv2. tion (TVM) using 99 video clips from the validation set of SSv2. We manually modified partial
7

---

Published as a conference paper at ICLR 2024
Table 1: Text-conditioned video prediction (TVP) results Something-Sometting V2 (SSv2),
Bridgedata (Bridge), and Epic-Kitchens (Epic100). We report both FVD and KVD metrics.
SSv2
Bridge
EpicI00
Method
Pre.-weight Text Resolution
FVD4
KVD
FVDU
KVD
FVD]
KVDJ
TATS
Ge et al. 2022
video
No 128 x 128
428.1
2177
1253
6213
920.0
5065
MCVD
Volet et al.
2022
No
No 256 x 256
1407
3.80
1427
2.50
4804
5.17
Si
Gao et al. 2022
No
No 64 x 64
537.2
0.61
681.6
0.73
1991
1.34
MAGE
Hu et al. 2022b
video
Yes 128 128
1201.8
1.64
2605
3.19
1358
1.61
PVDM
Yu al. 20223
No
No 256 256
502.4
61.08
490.4
122.4
482.3
104.8
VideoFusion Luo ct al. 20233
txt-vidco
Yes 256 256
163.2
0.20
501.2
1.45
349.9
1.79
Tunc-A-Viden (Wu ct al.
2022h
ixt-img
Yes 256 x 256
291.4
0.91
515.7
2.01
365.0
1.98
Seer (Ours)
txt-img
Yes 256 256
112.9
0.12
246.3
0.55
271.4
1.40
text prompts and generated 99 predicted videos for each method. Then, we invited 54 anonymous evaluators to rate the quality of the prediction, with a higher priority placed on the semantic contents in the videos and an intermediate priority placed on the fidelity of the video frames. We report the overall preference choices among the 99 video clips. More details are introduced in Appendix B
5.4 MAIN RESULTS
Quantitative Results. Table presents the text-condtioned video prediction results on Something Something- V2 (SSv2), BridgeData and Epic-kitchens- -00 (Epic100). Seer achieves the best perfor- mance among all baselines, with the lowest Fréchet Video Distance (FVD) of 12.9 and Kinematic Distance (KVD) of 0.12 SSv2, the lowest FVD of 246.3 and KVD of 0.55 BridgeData, and the lowest FVD of 271.4 in Epic 100. Notably, Seer, MAGE, VideoFusion and Tune-A- Vidde all incor- porate text conditioning, and the results highlight Seer's superior text-video alignment performance.
The results of the human evaluation in the rext-conditioned video manipulation experiment are shown in Figure 7 Our proposed Seer outperforms the other baselines in terms of both semantic content and fidelity of video, with preference rate of at least 72.2% in comparison. This indicates that Seer effective in generating high-quality video clips that are faithful the input text prompts.
Qualitative Results. O the text- conditioned video prediction and manipulation performance of Seer, VideoFusion and Tune-A- Video on Something Something-Ve (SSv2). Seer performs better in handling the temporal dynam- ics of the video and achieving more precise text- video alignment in video manipulation. For in- stance, consider the task of "turning the camera left" Seer seamlessly generates coherent move- ment while preserving the background during the camera view adjustment. In contrast, both VideoFusion and Tune-A-Video exhibit semantic move- ment but fail to maintain temporal consistency in the video background. Additionally, Seer can imagine hidden objects by utilizing its text-to-image diffusion prior. This flexibility allows Seer to effectively address occlusion in video prediction. In the "tearing paper" sample, Seer accurately predicts that a man hidden behind the paper and generates coherent frames including the man's 's face. Figure 8 compares Seer, VideoFusion and Tune-A-Video TVP performance on Bridgedata, illustrating that Seer achieves better text-video alignment of instructed behavior and target objects in future frames, and predicts more coherent video with higher fidelity.
GI
Seet (Cure
Seer
(Dom
Videoiluar
Viceo
Viien
Figure 8: Visualization of Text--onditionee Video Prediction on Bridgedata.
5.5 ABLATION STUDY
In this section, we evaluate the effect of different components of our method in the TVP task on the SSv2 dataset. We also evaluate the zero-shot TVP task of different models on the EGO4D dataset.
Table 2: Ablation study of temporal attention
temp. attn. bi-direct.
FVD.
KVDJ
258.2
0.56
Temporal Attention. As shown in Table 2 studies the effective- ness of different types of temporal attention. Our scaled autore- gressive window temporal attention (win-auto.) outperforms au- toregressive spatial-temporal attention (autoreg.), bi-directional temporal attention (bi-direct.) and directed temporal attention (directed.), resulting in the lowest FVD and KVD scores. We also find that "directed." further improves video prediction performance compared to "bi-direct." because utilizes the inductive bias of sequential generation.
directed.
222.3
0.40
autorey.
200.1
0.30
win-auto.
112.9
0.12
8

---

Published as a conference paper at ICLR 2024
ESText Decomposer. Table 3 compares different weight initial- ization strategies of FSText decomposer. The results show that using identity initialization described in Section yields higher prediction quality compared with random initialization. This find- ing demonstrates that identity initialization is necessary for the temporal-text projection of FSText decomposer. See additional ablation results in Appendix B.5
Table 3: Init. weight ablation
results of FSText init. weight
FVD.
KVDJ
random
367,9
0.75
identity(Ours)
112.9
0.12
Fine-tune Setting. We compare various fine- tuning settings of 3D Inflated U-Net in Table 4 Our default setting involves fine-tuning both FS- Text decomposer (FSText.) and scaled autore- gressive window temporal attention (SAWT-Aut.) layers (temp-attn),, while freezing the remaining crussttemp-attn. modules in 3D U-Net. For the "temp-attn." " setting, we only finetune the SAWT-Attn. layers and freeze all other components. In the "cross+temp-atti spatial-cros attn. and SAWT-Attn. layers. We observe that our default setting achieves the highest quality of video prediction among all these settings. Based on default setting, further fine- "cross+temp-att""" causes the performance of Seer to drop a lot. These results suggest that the optimization of the FSText decomposer is strongly guided by the frozen conditional diffusion prior,
Table 4: Ablation study of Fine-tune settings
fine-tune temp-atm.
FSText.
FVD.
KVDJ
328.2
1.26
crosstlemp-atin
249.9
0.73
(mmp-att.(Ours))
112.9
0.12
1807
5.12
setting, jointly update the parameters of
fremm
frumz
Furre
5.6
VISUAL. ANALYSIS OF INSTRUCTION EMBEDDING
Jub-
To assess the impact of sub-instruction guidance on frame gen- eration and unveil the implicit semantic information contained within a single sub-instruction at specific time steps, we com- pare text-conditioned video prediction (TVP) results condi- tioned on default frame-wise sub-instructions with those con- ditioned on the constant clone of sub-instruction from the third or twelfth frame along the video axis, as depicted in Fig- ure 9 Unlike the default frame-wise sub-instruction guidance, the sub- inbsisttuuctin clone from the twelfth frame tends to pro- duce a transition from the reference frame to the video's ter- mination state without temporal coherence. In contrast, the sub-instruction clone from the third frame tends to guide the motion until an intermediate state of the video, indicating that temporal sub-instructions provide proximate semantic guidance for motion at each time step. These findings underscore that sub-instructions align with the temporal sequence of the global instruction, offering fine-grat fine-grained guidance across multiple steps. See more results in the Appendix D.3
Instrctioo
Juh-
12 Jaam
Toe poop www
Figure 9: TVP results conditioned on frame-wise sub-instruction and constant clone of sub-instruction.
5.7 ZERO- SHOT EVALLATION
Table 5: Zero-shot text-conditioned video prediction on EGO4D
To assess the model's generalizability on an unseen dataset, we conducted comparative analysis between our proposed Seer model and the current statc-of-the-an bascline Vidco- Fusion. Both models were fine-tuned on the Something Something- V2 dataset and evaluated on the EGO4D dataset, with the results summarized in Table [ Notably, Seer method outperformed VideoFusion, evidenced by its superior performance in terms of FVD and KVD metrics. Drawing insights from these results, we can conclude that our default setup demonstrates strong generaliz- ability while effectively adapting to the EGO4D dataset with higher-quality video generation.
temp. attn. FVDJ
KVD4
VideoFusion 618.6
1.85
Seer(Ours)
301.7
0.55
6 CONCLUSION
In this paper, we propose Seer, a sample and computation-effccien model, for the challenging text- conditioned video prediction (TVP) task. We design a data and computation-efficient video network with Frame Sequential Text (FSText) Decomposer to inflate the pretrained text-to-image (T2I) stable diffusion models along the temporal axis. With the rich prior knowledge contained in pretrained T2I models and the well-designed architecture. Seer successfully generates high-quality videos by only tie-ttnnng the SAWT-Attn and FSText Decomposer, which significantly reduces the data and computation costs. The experiments illustrate our superior performance all the recent models.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
The main implementations of our proposed method are in Section 5.2 and 5.1 In addition, the settings the experiments and hyper-parameters we choose are in Appendix C And the implemen- tation details are in Appendix B.
ACKNOWLEDGEMENT
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), This work is also supported by the National Key R&D Program of China (2022ZD0161700).
REFERENCES
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2017.
Yilun Dai, Mengjiao Yang. Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint 2xXiv:2302.00111, 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Transactions Pattern Anal- ysis and Machine Intelligence (TPAMI), 43(11):4125 4141, 2021. doi: 10.1009TPAMM22022
2991965.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4188 Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/N19-160 1423. URL www.ac/lantolooyy.o.oooo
1423
Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. Cogview2: Faster and better text-to-image generation via hierarchical transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https: pettps:/oopena /ttps/////eeeeeeee.v.oo net www.r//wwweee/oooeii--------
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danilidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109. 13396, 2021.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make- a-scene: Scene-based text-to-image generation with human priors. In Computer Essin-ECCH 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV, pp. 89-106. Springer, 2022.
Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Li. Simvp: Simpler yet better video predic- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 3170-3180, June 2022.
Songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 102-118. Springer, 2022.
10

---

Published as a conference paper at ICLR 2024
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Aueeiar-Freeiag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang. Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu. Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price. Paola Ruiz, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijic Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawa- har, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torre- sani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19992- June 2022.
Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion mod- cls for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.132.. 2022.
Jonathan Ho and Tim Salimans. arXiv:2207.12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
Neural Information Processing Systems, 33:6840-688 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang. Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 10.0233,, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint 2204.3220,. 2022h.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. International Conference on Learning Representations, 2023.
Tobias Höppe, Arash Mehrjou, Stefan Bauer, Didrik Nielsen, and Andrea Dittadi. Diffusion models for video prediction and infilling. arXiv preprint urXiv:2206.07666 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022a. URL /ttptts///ppeeeeiieee
id=nZevKeeFYfD
Yaosi Hu, Chong Luo, and Zhenzhong Chen. Make it move: Controllable image-to-video generation with text descriptions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019- 3026, 2020.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3202-3211, 2022.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10209- 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp- 16784-16804, PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint SxX::2204.061255 2022.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pp. 1060-1069. PMLR, 2016.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. 1487, 2022.
Masaki Saito, Shunta Saito, Masanori Koyama, and Sosuke Kobayashi. Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan. International Journal of Computer Vision, 128 (00(1011) 11111-111:25822201 2606, 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02111 2021.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
Uriel Singer, Adam Polyak, Thomas Hayes. Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 2022.
12

---

Published as a conference paper at ICLR 2024
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3626-3636 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1212.0402 2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 4491944497720 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference computer vision and pattem recognition, pp. 1526-1535, 2018.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Vikram Voleti, Alexia Martinnaa, and Christopher Pal. Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint Stti::2205009853.3 2022.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and i Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2/04. 14806, 2021.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pp- 720-736. Springer, 2022a.
Jay Zhangjic Wu, Yixiao Ge, Xintao Wang. Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11555, 2022b.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv: 104. 10157, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022a.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. arXiv preprint arXiv:2212.05199, 2022b.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aww aware implicit generative adversarial networks. arXiv preprint 22Xiv:2202.10571, 2022c.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
13

---

Published as a conference paper at ICLR 2024
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference robot learning. pp. 1094-1100. PMLR, 2020.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv. Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11016 2022.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
B ADDITIONAL EXPERIMENTAL RESULTS
B.I IMPLEMENTATION DETAILS OF BASELINES
We compare three baselines in our paper. For Tune-A-Video, to ensure a fair comparison, we use the pre-trained weight of Stable Diffusion-vl. (same as our model) to initialize the UNet and we fine- tune the model with an image resolution of 256 x 256 on the training sets of Something Something- V2 (SSv2), Bridgedata and Epic- Bitcc-KtchenssIuee 100 for 200k training steps. Similarly, we further fine- tune pre-trained VideoFusion on Something Something- V2, Spiic-Kicchens-000 100, and BridgeData for 200k training steps. For MCVD, SimVP and PVDM, we train the model with an image resolution of 256 256, 64 x 64, 256 256 respectively on the training sets of SSv2, Spic-Kitchens 100, and Bridgedata for 300k training steps. For TATS and MAGE, we fine-tune the pre-trained UCF-101 model with an image resolution of 128 x 128 on the training sets of SSv2, Epic- Kitchens- 100, and Bridgedata for 300k training steps.
B.2 EVALUATION DETAILS AND RESULTS OF UCF-101
Most prior text-conditioned video generation methods (Hong et al., 2023 Ho et al. 2022b Singer et al. 2022, Zhou et al.] 2022) evaluate their performance on the UCF-101 (Soomro et al. 2012) benchmark. However, since proposed method, Seer, is designed for text-conditioned video prediction (TVP) on task-level video datasets, the UCF-101 benchmark, which evaluates class- conditioned video prediction on random short-horizon video clips, is not an ideal evaluation bench- mark for TVP. Nonetheless, in order to fairly compare these baselines, we still evaluate the class- conditioned video prediction performance of Seer on UCF-101.
Settings Specifically, we fine-tune our model with video resolution of 16 x 256 256 on UCF- 101. Following the evaluation protocols of (Hong et al.] 2023), Seer predicts the videos condi- tioned on 5 reference frames during fine-tuning and inference stage. We report FVD and Inception score (IS) metrics on the UCF-101 dataset (Soomro et al., 2012). The IS is calculated by a C3D (ran et 2015) that is pre-trained on the Sports- IM dataset (Karpathy et al. 2014) and fine-tuned on UCF101, We follow the evaluation code of TGAN-v2 (Saito et al.) 2020) to calculate IS metric. Following (Hong et al., 2023 Ho al.. 2022b Singer et al. 2002)) we evaluate the FVD metric with 2,048 samples and IS metric with 100k samples in the validation set of UCF-101.
Results Table 6 presents the class-conditioned video prediction results on UCF-101, demonstrat- ing that Seer outperforms Video (Hong al. (023) and Magic (Zhou et 2022). but falls short of Make-A-Video (Singer et al. 2022). Make-A-Video employs unlabelled video pre- training on temporal layers and achieves the best performance among all other methods. While Make-A- A-le-AA-ee A-Video shows superior performance on FVD and IS, Seer has the potential to further im- prove its generation performance by addressing the following two limitations. First, Seer has not been pre-trained on video data. Second, Seer obtains latent vectors via a pre-trained 2D VAE, which has not been fine-tuned on UCF-101 and limits the video generation quality of Seer (with 259.4 FVD and 68. 16 IS reconstruction quality). However, as we focus the text-conditioned video prediction task, addressing the above limitations on UCF-101 is out of the scope of this paper.
B.3 EVALUATION RESULTS OF SAMPLING STEPS
Sampling steps:
To assess the influence of sampling steps on the quality of video predictions across varying sequence lengths, we conducted an evaluation on both 12-frame and 16-frame video pre- dictions using a series of DDIM sampling steps (10, 20, 30, 40, 50, 60 DDIM steps). All generated outputs were sampled utilizing 12-frame SSv2 fine-tuned model. The comparative results pre- sented in Figure T0 Notably, the 16-frame curve exhibits a more rapid decline from DDIM steps 10 to 20 compared to the 12-frame curve. As both curves progress beyond DDIM step 30, they tend to stabilize, showing marginal gains. These findings collectively underscore that increasing
15

---

Published as a conference paper at ICLR 2024
Table Class-conditioned video prediction performance on UCF-101 we evaluate the Seer on the UCF-101 with 16-frames-loor videos. Ex.data indicates that the model has been pre-trained or fine-tuned on extra datasets. Method
Ex.data
Cond.
Resolution
FVD4
1St
MoCoGAN-HD Tulyakov et al. 2018
No
Class.
256 x 256
700
33.95
VideoGPT Yan et al. 2021
No
No
128 x 128
24.69
RaMViD Höppe et al. 2022
No
No
128 x 128
21.71
StyleGAN-V (Skorokhodov
2022
No
No
128 128
23.944003
DIGAN
2022c
No
No
27
32.70
TGANv2 Satto of al.
2020
No
Class.
128 x 128
1431.0
20.60:047
VDM (Ho et al. 2022bb
No
No
64 x 64
57.80
TATS-base (Ge et al.
2022
No
Class.
128 128
278
79.28
MCVD (Voleti
2022
No
No
64 64
1143.0
LVDM (He al. 2022)
No
256 x 256
372
27:1
MAGVIT-B Yuu et al. 2022b
No
Class.
128 x 128
159
83.55
VideoFusion (Luo al. 202.5
txI-video
Class. 128 x T28
173
80.03
CogVideo HHong ct al. 2023
txt-img & txt-video
Class.
160 160
626
50.46
Make- A- Viden (Singer al. 2022
txt-img video
Class. 256 256
81.25
82.55
Magic Video Zhou et al. Seer(Ours)
2022
ixtimgg & txt-video
Class.
699
txt-mgg
Class. 256 x 256
260.7
57.74
pre-trained VAE*
256 256
259.4
68.16
we evaluate the reconstruction quality of pre-trained 2D VAE in this table, the pre-trained 2D VAE is initialized with the pre-trained weight from Stable Diffusion-v1.5 without extra fine-tuning.
772.8
1000
2.5
EVD
600
448.2
355.4
359.3
KVD
1.5
1.11
400
280.7
0.73
0.63
0.47
0.48
200
129.8
112.9
101.3
101.8
105.2
0.5
0.16
0.08
0.08
0.06
10
30
50
10
50
60
DDIM steps
DDIM steps
12-frame 16-frame
12-frame 16-frame
Figure 10: Evaluation results sampling 12-frame video and 16-frame video using 12-frame Seer model with DDIM sampling steps ranging from 10 to 60 on the Something-Something Something V2 dataset.
DDIM sampling steps notably enhances video quality for longer sequences (10 30 DDIM steps). However, the quality improvements of longer videos diminish as DDIM steps exceed 30.
Video length: Furthermore, present the qualitative outcomes of the 16-frame video in Figure 12 comparison with the results of the 12-frame video Figure 23 reveals that both the 16-frame and 12-frame videos can effectively capture task-described motion, as demonstrated in the example of "moving pen." However, the 16-frame video exhibits gradual loss of appearance information from the reference frame and a decline in temporal consistency along the temporal axis with the increasing sequence length. Given that 16-frame videos are beyond the anticipated sequence length for a 12-frame model, enhancing the generation quality of the 16-frame video could be achieved through the training of a dedicated 16-frame Seer model.
addition, we explore the impact of fast sampling on generation results during evaluation by comparing Seer with Tune-A-Vided, We apply series of DDIM sampling steps (10, 20, 30, 40, 50 DDIM steps), as shown in Figure Seer consistently outperformed Tune- A-Video in terms of both FVD and KVD, with improvements observed from 20 DDIM steps to 50 DDIM steps. Particularly noteworthy is Seer's 's advantage in video quality (280.7 FVD and 0.73 KVD) compared to Tune-A- Video (419.3 FVD and 1.5 KVD) when using only 10 DDIM steps, demonstrating Seer's ability to sample high-fidelity videos efficiently with minimal denoising steps.
Methods comparison:
16

---

Published as a conference paper at ICLR 2024
800
2.5
1.5
700
600
500
400
261
254.2
X20
0.91
300
280.7
0.84
0.75
200
112.9
100
0.12
0.08
50
DDIM stops
DDIM steps
Seer
Tune-A-Video
Seer
Tune-A-Video
Figure 11: Evaluation results of Seer and Tune-A- A-Vido with DDIM sampling steps ranging from 10 to 50 on the Something -Smethhig V2 dataset.
Ref. Frame
tearing piece of paper into two pieces
Ref. Frame
moving pen up
Figure 12: Visualization of 16-frame text-conditioned video prediction sampled by 12-frame Seer
on SSv2.
B.4 COMPUTATION EFFICIENCY OF SEER
To enhance the computational efficiency of Seer, both the inclusion of frozen 2D layers and the thoughtful design of temporal layers contribute significantly. A comprehensive evaluation of the computational cost for different tempo- ral layer types, conducted GPU, is presented in Table 7 Remarkably, in comparison to plain autoregressive spatial-temporal attention (autoreg.), both bi-direecioonal/direeeed temporal attention (bi-direet/ddrrected)) and SAWT-Atter (win-auto.) substantially reduce computational overhead. Furthermore, SAWT- Atten demonstrates superior generation quality compared to bi-directional and directed temporal attentions, as evidenced by the ablation results in Section 5.5 of this paper. In addition, a compar-
Table 7: Training time of the variants of temporal attention on 16-frame video
temp. attn. bi-direct.
(sec./iter.)
single 24GB NVIDIA 3090
2.35
directed.
2.35
autoreg.
5.50
wOn-a)to.(Ours)
2.40
Table 8: Training time (time.) and GPU mem- Table 9: Training time (time.) and GPU mem- ory (Mem.) consumption of the models (16- ory (Mem.) consumption of the models (16- frame)
frame)( 90% GPU memory usage)
model
2D. frozen
time. (iter/iter.)
Mem.
(GB)
model
2D. frozen
(sec./iter.)
time.
Mem.
(GB)
Seer (Ours)
0.75
24.7
Seer (Ours)
Yes
3.10
72.9
Seer
No
0.96
39.2
Seer
No
6.89
75.8
VideoFusion.
No
1.07
45.0
VideoFusion.
No
7.63
78.7
ison involving our ablated setting and the baseline method VideoFusion (uoo et al. 2023), which shares a network design based on the Stable Diffusion U-Net, is presented. In our proposed Seer setting, the 2D spatial layers of the U-Net are frozen during fine-tuning, while the ablated setting maintains all 2D spatial layers trainable. GPU memory consumption and training time for different models were assessed running with single 80GB NVIDIA A800 GPU. These results are available in Table g and highlight the proposed Seer setting with frozen 2D layers exhibiting approximately half the GPU memory consumption (24.7GB) compared to Seer without frozen 2D layers (39.2GB) and VideoFusion (45.0BB). To ensure a fair comparison of training speed among various settings on the single A800 GPU, we conducted additional assessments models with GPU memory usage exceeding 90%, detailed in Table The results in the table reveal notable advantage in com-
17

---

Published as a conference paper at ICLR 2024
Table 11: Components in FSText Decomposer
Table 10: Layer depth in FSText. (SSv2).
(SSv2). Our settings are marked in gray
layers.
FVD1
KVDJ
Temp. Seq.
FVDL
KVDJ
238.6
0.51
125.8
0.13
229.7
0.23
127.7
0.14
8(Ours)
112.9
0.12
112.9
0.12
Table 12: Ablation study of temporal attention Table 13: Ablation study of Fine-tune settings (BridgeData)
(BridgeData)
temp, attn.
FVDL
KVDJ
fine-ture tun-tune temp-astn. crosstlemp-atm.
FSText.
FVD
KVDJ
bi-direct.
284.5
0.71
410.7
0.97
directed.
258.0
0.64
319.9
1.01
autoreg.
261.5
0.83
temp-attn. (Ours)
246.3
0.55
win-autt.(Ours)
246.3
0.55
crossttemp-attnn
2058.4
9.43
putation efficiency for the proposed Seer setting with frozen 2D layers, exhibiting reduced training time (3.10 sec/iter) compared to Seer without frozen 2D layers (6.89 sec/iter) and VideoFusion (7.63 sec/iter). These findings firmly establish the enhanced computational efficiency of Seer relative to baseline models.
B.5 ADDITIONAL ABLATION RESULTS
FSText layer depth In this section, we additionally investigate the impact of FSText Decom- poser's layer depth in Table TO Our default setting (8-layer FSText Decomposer) outperforms shal- lower models (2-layer and 4-layer) in terms of FVD. Though the 4-layer model shows marginal advantage over the 8-layer model in terms of KVD, our experiments indicate that the 8-layer FSText Decomposer shows remarkable advantage on FVD metrics and exhibits robustness in text-video alignment. Therefore, we adopt the 8-layer FSText Decomposer as the default setting for Seer.
FSText componenets To evaluate the impact of individual attention layers within the FSText de- composer, conducted an ablation study on the FSText component, as presented in Table 8. In this study, we ablate the temporal attention layer, labeled as "Temp.." and the text-sequential atten- tion layer, denoted "Seq.." within the FSText network. To maintain consistency in model size across different settings, we replaced the ablated component with Cross-Attention layer. The re- sults, shown in Table IT highlight the superiority of our proposed setting, which integrates both eoxt-sequential-atteniiie
layers and temporal attention layers. Our proposed setting outperforms the other two settings, underscoring the significant attributes of Textt-Squential-AAIe Atteenioo layers and Temporal Attention layers to capture text contextual information and model temporal dependencies, collectively enhancing the overall performance of the FSText decomposer.
Additional ablation on BridgeData To validate the robustness and consistency of Seer ablation results across different datasets, we conducted additional experiments on the BridgeData dataset, extending our analysis from the Something Something- V2 (SSv2) dataset. The corresponding abla- tion studies Seer fine-tuning settings presented in Table [ while the temporal layer settings are detailed Table 12 These results mirror the ablation outcomes reported in Table 4 and Table for the SSv2 dataset. Notably, the consistent improvement observed in both Seer fine-tuning and temporal layer ablation across different datasets, as demonstrated in Table 13 and Table [2 on the BridgeData dataset, demonstrates the robustness of the Seer component design.
Qualitative results of fine-tuning ablation We conduct qualitative analysis of various fine-tune settings. We provide additional visualizations Fine-tune Setting ablation in Section 5.5 the main paper. Figure 13 shows the results of different settings. Among these settings, "femp+FSTexr" stands out as it preserves a higher-level temporal consistency in video prediction starting from reference frames and also delivers superior text- bost-bsedd video motion compared to the other fine-tune settings.
default setting
B.6 EVALUATION OF POLICY LEARNING ON SEER
To investigate whether Seer can help policy learning, we choose the UniPi Dai et al. 2023) as baseline in the simulated roboties environment Meta-Worka (Yu et al. 2020). which gener-
18

---

Published as a conference paper at ICLR 2024
pouring water into cup
GroundTruth
"cross. +temp.
"temp."
"temp+FSTer"
tilting book with package on it until falls off
GroundTruth
"cross. +temp."
"temp."
"temp+FSText."
tilting book with measuring tape on until it falls off
GroundTruth
*temp..
"temp."
"temp=FSText."
Figure 13: Additional qualitative results of fine-tuning ablation. "temp+FSSe FSeex." is default setting.
ates videos from the initial state and infers actions from adjacent frames via a pre-trained inverse-dynamics model. Specifically, we distill policy model from the videos generated by the Seer and the labeled actions from the pretrained inverse-dynamics model. We choose 3 tasks and use 10 in-domain videos for each task to fine-tune the Seer. And make comparisons between a) Policy a: Distilled policy from the dataset generated by the Fine-tuned Seer model (1000 generated videos for each task). b) Policy b: Distilled policy from the 10 in-domain videos for each task. The results are shown in Table [4] where we find that compared to utilizing the given 10 in-domain videos generate policy, fine-tuning the Seer with them and generating more videos can be better because it can acquire more scalable data, which is of comparable quality. Therefore, the videos generated by Seer can help policy learning simulated robotics tasks way. The visualization of Seer within a robot simulation environment is presented in Section D.5
Table 14: Success rate of distilled policy tasks
policy a policy
button- prenonn-rrsss--toooeeeee
0.45
0.4
drawer-close-22 0.1
0.0
drawer-open-v2
0.05
0.0
19

---

Published as a conference paper at ICLR 2024
C IMPLEMENTATION DETAILS
C.I FINE-TUNING AND SAMPLING
In this section, list the hyperparameters, fine-tuning details, sampling details, and hardware information of our model in Table 15
C.2 ARCHITECTURE INFORMATION
In this section, we list the hyperparameters 3D U-Net in Table I6 and hyperparameters of FSText Decomposer in Table 17
Table 15: Hyperparameters and details of Fine Tuning/Inference
param.
value
optim.
Adam W
Adam-8; 81
0.9
Adam-8 B2
0.99
Adam-e
1e-8
weight decay
le-
Ir end Ir Ir sche. noise sche. train hatch size grad, acc. warmup steps resolution train. steps train. hardware
1.024e -4
0.0
cosine
cosine
1/GPU
2
10k
256 x 256
200k
RTX 3090
val. batch size
2/GPU
sampler
DDIM
sampling steps
30
guidance scale
7.5
Table 16: Hyperparameters of 3D U-Net
hyperparam.
value
input/output channels
Base channels
320 1 1,2,4,4
Channel multipliers
3D Downsample blocks
4
Upsample blocks
Number layers (per block)
Modules of layer
3D ResnetBlock Spatial-cross Atten. SAWT-Atten. Down//pp 3D ResnetBlock
Dimension of atten. heads activation function
8
SiLU 768
Dimension of cross-atten,
20

---

Published as a conference paper at ICLR 2024
Table 17: Hyperparameters of FSText Decomposer
hyperparam.
value
Icarnable tokens channels
768
output channels
768
Base channels
768
Number of layers
8
Modules of layer
Seq-cross Atten.
Feedforward
Directed temporal Atten.
Feedforward
Number of atten. heads Dimension of cross-atten.
8
768
D VISUALIZATION
D.I ADDITIONAL QUALITATIVE RESULTS
We provide additional visualization Something-Something v2 (SSv2) of our text-conditioned video prediction in Figure 22 and text-conditioned video prediction/maniiulation results Fig-
ure 23 Additionally. provide the visualization BridgeData of text-conditioned video predic- tion in Figure 244and and text-conditioned video prediction/manipuaation in Figure 25 We also provide the visualization results of text-conditioned video prediction Epic-Kitchens-- 100 in Figure 26
Ref. Frame
pretending to pick a tennisball up
Ref. Frame
moving pen up
Figure 14: Additional visualization results of 12-frame Text-conditioned video prediction (reference
frame=1) SSv2 dataset.
D.2 GENERALIZABILITY
OF SEER ON DOWNSTREAM TASKS
To thoroughly assess Seer's adaptability on downstream tasks such video manipula- tion on BrtageData, we conducted an inves- tigation into diverse fine-tuning strategies this dataset. Our default setting proceeded fine-tune only the temporal block of 3D U-Net, freezing the FSText decomposer on mixed dataset including both the pre-training video dataset and the task-specific downstream dataset. Additionally, we compare our default setting with LORA modules (Hu et al. 2022a).
Gure Oraut At miterrr medid
Pertual Inipioos nevel mede (ivi)
Broggndhtt
The visualization of comparison with different fine-tune settings BridgeData (given text instructions "pick up black mug". where "black" text prompt is unseen in BridgeData). Compare to other two settings, our default setting successfully recognizes "black mug" and gener- ates RGB frame with higher fidelity
Figure 15:
which are integrated into each temporal layer the UNet architecture and throughout the entire FS- Text decomposer. As depicted in Figure 15, both proposed setting and the LORA setting have effectively identified the previously unseen "black mug" in the training set of BridgeData. Besides, proposed setting maintains higher background consistency from the reference frame (frame 0) and generates future frames with superior fidelity when compared to the LORA setting. We also provide additional visualization in Appendix D.4
21

---

Published as a conference paper at ICLR 2024
D.3 ADDITIONAL VISUALIZATION OF IN--INSTUCTTIO EMBEDDING
We present additional visualizations comparing frame-specific sub-instructions to those constant clones from the third twelfth frame along the temporal axis (Figure T6). These visualizations reveal that each frame's s sub-instruction represents the motion at its corresponding time step. For example, in the case of "pushing iphone adapter from left right" the default sub-instruction guides the entire instructed behavior with continuous temporal motion, while the sub-instruction embedding from the third frame directs the motion to halfway point in the case of pushing the iphone adapter to right. In contrast, utilizing the sub-instruction embedding from the twelfth frame leads to frames that are disconnected from the reference frame's s scene, resulting in the absence of transition between frames. This observation demonstrates that sub-instructions closely follow the temporal sequence of the global instruction, providing precise guidance across multiple steps.
D.4 ADDITIONAL VISUALIZATION OF VIDEO MANIPULATION WITH UNSEEN OBJECTS AND ZERO-SHOT VIDEO MANIPULATION
We also include visualizations of video manipulation with unseen objects in BridgeData and zero- shot generation on EGO4D. These experiments involved fine-tuning the SSv2 video model on mixed dataset consisting of SSv2 and BridgeData. The fine-tuning process lasted for 800k itera- tions, employing learning rate of 4. 1.096e-5 Subsequently, the model's 's performance was evaluated on Bridgedata (as shown in Figure a7]a) and the EGO4D dataset (illustrated on the right side of b)),, where we sampled 15 frames with one reference frame. the evaluation on Bridge- Data, Seer demonstrated the ability to recognize previously unseen objects such "red plate" and "cabbage" and successfully performed a "pick up" motion, leveraging prior knowledge learned from SSv2. In certain scenarios, such as picking up the plate, where Seer has not encountered the spe- cific action during training, the generated video depicted some limitations the interaction between the robotic and the plate. Regarding the evaluation on EGO4D, although Seer had not been fine-tuned on this dataset, exhibited the capability to accurately identify objects in the EGO4D environment, such as "laptop" and "book," and execute actions based on prior knowledge acquired from observing human activities. However, Seer still faced challenges in predicting future frames based on the understanding of the scene the reference frame. For instance, in the case of "closing a book," Seer tended to generate a hand outside the camera view instead of manipulating the object with the main body, such as the hand holding the book, within the scene.
22

---

Published as a conference paper at ICLR 2024
Text prompt:
pushing iphone adupter from left to right
Groundtruth
instruction
sub- instruction
3 frame
sub- instruction
12 frome
Text prompt: taking glass from desk
Groundtruth
Instruction
sub- instruction
ot 3 frame
sub- instruction
12 frome
Text prompt:
moving pen up
Groundtruth
sub- instruction
instruction
3 frome
sub- instruction 12 frame
Figure 16: Additional visualization results of Seer's Text-conditioned video prediction conditioned on frame-win wise sub-instruction and duplicate sub-instruction the third/twelfth frame.
frame o (ww)
frume
frams
Frams 34
Frams (ue]
frame
frame
frums 14
Those lqpteee
aed sioer
"all ap ashhoge
"sous Loce
BridgeData
Figure 17: Additional visualization of Seer generalizability evaluation on unseen dataset EGO4D (b) and BridgeData (a), where "cabbage" and "red plate" unseen objects in the training set of BridgeData.
23

---

Published as a conference paper at ICLR 2024
D.5 VISUALIZATION OF ROBOT SIMULATION ENVIRONMENT
To investigate the applicability of Seer in robot environments, we assess its performance on robot simulation datasets, including Meta-World (Yu al.) 2020), CLIPort (Shridhar et al. 2022). and RLBench (James et al. 2020). Beginning with the SSnn-tine Seer model, we further fine-tune on mixed dataset comprising Something Something- V2 (SSv2), Bridgedata, 236 MetaWorld video clips, 785 CliPort video clips, and over 2000 video clips from RLBench for 80,000 steps, using a learning rate of 4e-5. Throughout the fine-tuning process, all 2D layers of the 3D inflated U-Net and the FSText decomposer remain frozen. Figure 18 presents visualizations of Seer these robot simulation datasets. In these visualizations, Seer successfully generates coherent videos with continuous motion trajectories aligned with language instructions. In the CLIPort video clips, Seer accurately detects small target objects in complex scenes and places them in the correct positions as instructed by the task descriptors. Moreover, in the RLBench videos, Seer demonstrates the ability to perform multi-step tasks such as "stacking a pyramid with the boxes." These observations highlight Seer's adaptability to multi-task environment, encompassing human video, robot manipulation, and simulation scenarios, while maintaining robust temporal alignment with task descriptors. We conduct an assessment of Seer performance on policy learning, and further analysis is presented in Section B.6
Frante
open green duseer
Ref Fram:
put the green is gny
Bel
opan the doer nariing the handle
Ref
yellow block cyen red Mocks
(a1) Meta-Weeld
(b) CLIPort
Ref From
posaa dowa the red butuon
Bef Frans
close the for the botile
Ref Framz
pick dhe grozn biz Soo the red box tuble
Ref. Fare
stack a pyeamid with the boxes the same color
(a2) Meta-World
(c) RLBenc
Figure 18: Text-conditioned video prediction of Seer (conditioned on first frame) on Meta- World
(al-2), CLIPort (b), and RLBench (c) datasets
D.6 LONG VIDEO PREDICTION
To evaluate Seer's ability to generate extended video frames during the inference stage, we con- ducted a qualitative test on long video prediction using 12-frame SSv2 fine-tuned Seer model. Our video sampling approach involves the sequential generation of video clips, where the first 12 frames are conditioned on 2 reference frames, and the last two frames of the first clip serve as the condition for the subsequent clip. The concatenation of these clips results in a long video. A cru- cial element for aligning long video frames with language is the presence of a prolonged sequence of sub-instructions. To achieve this, we employ two text conditioning strategies for expanding the sub-instruction embedding along the frame axis, enabling the iterative sampling of longer frames.
The first strategy involves interpolating sub-instruction embedding along the frame axis, while the second strategy entails repeating the sub-instruction embedding from the first video clip to guide the second clip. In Figure 19 we compare these two strategies and observe that direct interpolation (Fig- ure [9 (a)) tends to degrade overall generation quality, introducing unexpected noise. Conversely, utilizing the same sub-instn iob-instruction (Figure 19 (b)) maintain coherent motion, persisting until the
target object is no longer present or the current motion is in a terminated state. Although this strategy facilitates the generation of coherent movements, it is distinct from simple upsampling of video
24

---

Published as a conference paper at ICLR 2024
clips along the frame axis. It is noteworthy that all results were obtained using the 12-frame Seer model.
Intuitively, appending additional upsampler network could potentially enhance Seer's ability to expand video frames, which causes extra computational costs. Observing the results of the evalua- tion, we believe that expanding the frame length of generated subinstruction embeddings during the training stage represents a promising direction for enabling Seer to generate longer video frames in multi-steps without incurring additional computational overhead.
(a) Interpolate sub-instruction embedding
Ref. Frame
moving pen
(b) Repeat sub-instruction embedding
Ref. Frame
moving up
Frame 1hth
Ref. Frame
pretending to pick tennisball up
Frame 1 th
Frame
|[th-- 12th
Ref. Frame
tearing piece of paper into two pieces
Frame
the th
Frame 11hh 12th
Figure 19: 22-frame video prediction conditioned on reference frames using 12-frame Seer model on SSv2. (a) Interpolation of 12-frame subinstruction embeddings to 22-frame sequence. (b) Repetition of 12-frame subinstruction embeddings for the second video clip prediction from the eleventh frame to the twenty-second frame.
25

---

Published as a conference paper at ICLR 2024
D.7 FAILURE CASE
In this section, we present instances where Seer encounters challenges in handling environmental motion in human-generated videos. The generated videos highlight situations such "dropping a card in front of a coin" and "book falling like a rock" (refer to Figure 20). where Seer suc- cessfully predicts task-descriptive motions like "dropping" and "falling" and correctly identifies text-described objects such as "card" and "book." However, the generated future frames fall short in capturing appearance consistencies, such as the color of the card in the previous frame and the cover of the book in the reference environment. In the scenario of "pouring red wine into glass." Seer tends to generate a wine glass based on its knowledge of pouring red wine but overlooks the transition distribution from the reference environment.
Notably, in the Epic-Kitchens-100 100 dataset, where scene transitions are prevalent, Seer exhibits preference for predicting camera pose movements and generating novel views of the environment, reflecting its imaginative capabilities. However, these outcomes extend beyond the scope of Seer's primary objective, which is to learn human behavior. Consequently, addressing challenges such as filtering out irrelevant background information, including camera pose and object occlusion, and refining Seer's awareness of temporal motion becomes imperative for its adaptation to learning from internet videos.
Ref. Frame
dropping a card in front of a coin
Ref. Frame
pouring red wine into a glass
Ref. Frame
book falling like rock
Ref.
put the cap on the peanut butter
Ref.
continue chopping onion
Ref.
flip chips
Figure 20: Seer failure cases on Something Something V2 (row 1,2.3) and Epic- BeiccKitchens--. 100 (row 4,5,6)
26

---

Published as a conference paper at ICLR 2024
E HUMAN EVALUATION DETAILS
To evaluate the quality of video predictions according to human preferences, we conducted a hu- man evaluation with 99 video clips on the validation set of the Something-Sometting V2 dataset (SSv2), the evaluation process involved 54 anonymous evaluators, To eliminate biases towards spe- eific baselines, we randomly selected 20 questions for each evaluator. Each single-choice question consisted of ground-truth -rutt video as reference, manually modified text instruction, and two video prediction results generated by Seer and another baseline method. The evaluators were required to choose the video clip that is more consistent with the text instruction and has higher fidelity from the two options. ensure the clarity of the questions, we provided an example explain the options in each questionnaire. Moreover, strong text-based motions as their first preference and the fidelity the generated video as their second preference. For reference, Figure provides screenshot of an example questionnaire.
recommended that evaluators prioritize video predictions with
In total, we collected 342 responses for the Seer vs. TATS comparison. 363 responses for the Seer Tune-A-Video comparison, and 357 responses for the Seer vs. MCVD comparison. And the results in the main paper Figure 7 are calculated based on the collected questionnaires.
Text-conditioned Video Prediction
Please review text prompt and the reference video and choose the video result that is closer to text description from the two options presented, nether optoon accurately matches the text description, please choose the result
more similar reference video, regardiess resolution. example, the resolution low but the generated result closely matches the text description, is recommended select that option.
Example:
According following description, right filming language description exactly, but your language description.
generated "tuminn
the given reference may match
given
Reference frame
(A)
Figure 21: Screenshot of questionnaire example shown to human evaluators.
27

---

Published as a conference paper at ICLR 2024
pushing iphone adppter from to right
covarring saitt shateer with towel
dropping,
6odine
folding paper
continees spiening
LE
folöing wool
putting into bowt
nitingg quarte
and small
mwayy
taking book
spinning cain so continus spinning
Figure 22: Text-conditioned video prediction of Seer SSv2.
28

---

Published as a conference paper at ICLR 2024
moving pen down
turniny: tumera left while filming wall muunted
meed
tuminy the while filming wall mounsed
poshing that Balls off offittee
picking scissors
preiending pick a tennisball up
past
dnolendiingeee picking tennisholl up
removing beer revealing glass bahind
putting flowerpot next been con
moving bowd
moving bowl right
putting
putting a green apple on shated surface
mani
neetending to jfjice up
postending to picking box of
Figure 23: Text-conditioned video prediction/mamiiulaaaii of Seer SSv2, where "pred." refers to prediction, "mani." refers to manipulation.
29

---

Published as a conference paper at ICLR 2024
put pot stove
Neor
pul broccoli in pot or pan
flip pot upright sink
put fork from basket to tray
green squish pot
Secr
flip pot upright sink
turn lineet front right
pui on stove
Seet
Figure 24: Text-conditioned video prediction of Seer on BridgeData.
30

---

Published as a conference paper at ICLR 2024
put banuna on plate
peed.
pot com on plate
upricht
pred.
pick up eggplant
mum
up groen mun
pred.
pick up glass cup
vertical
pead.
pick up from
mani
pick up bowl and put in small box
pred.
close small box Clap
mami
Text-conditioned video prediction/maniipulaioon of Seer on BridgeData, where "pred." refers to prediction, "mani." refers to manipulation.
Figure 25:
31

---

Published as a conference paper at ICLR 2024
continue chopping onion
GT
TAV
VF
Seer
still wiping bowl with rag
G1
TAV
VF
Seer
cupboard
GT
Seer
open tap
GT
Seer
stir pasta
GT
Seer
cut polato
GT
Seer
Figure 26: Text-conditioned video prediction (conditioned first frame) Epic- Siii-Kitchens-1000 TAV refers to Tune-A-Vides VF indicates VideoFusion.
32


Reviews:
[{"review_id": "ziEmR1oxxd", "reviewer": "Reviewer_shhp", "summary": "The authors propose a novel text-conditioned video prediction \"Seer\". Seer leverages pretrained text-to-image stable diffusion model, extending it along the temporal axis to generate video. The key contributions of the authors are a novel Frame Sequential Text Decomposer module for frame-aligning sub instructions and a computation-efficient spatial-temporal attention to improve efficiency of temporal attention. FSText dissects global textual instructions into temporally aligned sub-instructions for precise integration into video frames, enabling the effective use of prior knowledge embedded in pretrained T2I models. Seer's architecture allows for high-fidelity, coherent video generation that aligns with textual instructions with only fine-tuning of a few layers on a small dataset. The model demonstrates superior video prediction performance on 3 egocentric task datasets, achieving significant improvements in Frechet and Kernel video distances and parameter efficiency compared to previous models.", "questions": "In Sec 1. Introduction *'existing text-conditioned video generation task (use) text specifying the general content, such as “a person is skiing”, while our aim in the TVP task is to use the text as a “task descriptor” in robotics, such as “tipping a beer can”'*. Could the authors please explain how does using text as a task descriptor differ from text specifying general content? Does the VLM model encode the intent of text in the former, and if so are there significant differences across the two representations (e.g. CKA value)? How does it relate to the frame aligned sub-instructions in the FSText decomposer?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose and combine several ways to make text conditional video generation computationally tractable: reusing LDMs pre-trained on text to image tasks, and introducing a novel spatiotemporal attention that combines the benefits of bidirectional and directed temporal attention. I think the proposed spatiotemporal attention method is quite well thought and explained, and the empirical results and ablations suggest that it is effective for various task based video prediction tasks.", "weaknesses": "1. The design choices behind FSText are explained by the authors in Sec 4.3, but the overly complicated design choices could limit the utility of SEER outside of highly specific stepwise instruction videos. Did the authors perform any ablation studies that justified the various design choices in FSText (e.g. using cross-attention, using directed temporal attention, starting with CLIP text embedding etc.) \n\n2. For the ablations, could these be reported on more than one dataset? Results on one particular task in the SSv2 are quite convincing, but I am wondering if their could be dataset biases that seem to justify these design choices. For example, the BridgeData is a robotic task dataset versus human, so maybe that could be a good candidate to ensure results are not cherry picked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tcp0EkTTql", "reviewer": "Reviewer_xJcW", "summary": "The authors propose a novel method for Text conditioned Video Prediction (TVP) that utilizes pretrained CLIP embeddings for language and inflates pretrained UNets from static image generation models to produce a language and image-conditioned next-frame prediction model. They perform experiments on something-something-v2, the Berkeley Bridge dataset, and Epic-Kitchens data demonstrating improved video prediction in comparison to competing methods on FVD and KVD. They also perform human studies and report that their model shows an improvement in the prediction task in comparison to existing baselines.", "questions": "1. How does sequence length of the video along with number of steps of prediction affect performance?\n2. What is the inductive bias that forces the Text Decomposer to produce coherent sub-instructions? Its unclear why it would segment the instructions into substeps on its own.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The method utilizes pretrained models thereby requiring less data than training from scratch. \n3. The experiments are thorough and performed on a variety of benchmarks\n4. The authors also perform human studies to validate the quantitative claims made in the paper", "weaknesses": "1. Improve the motivation for the paper: The first few paragraphs of the introduction focus on robot learning, but the paper completely eschews this domain. It would be useful to motivate the paper from a different perspective that does not involve robotics. I appreciate that there are experiments conducted on the bridge dataset but I still believe this is primarily not a robotics paper as there are no resultant policies. \n2. Difficulty in comparison to text-to-video generation: Playing devil’s advocate here, one could argue that performing text-to-video generation involves learning both the initial state distribution along with the transition distribution, while prediction involves only learning the transition distribution. Therefore it is unclear which is harder and it would be useful to avoid making such claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qz7ywAgAb1", "reviewer": "Reviewer_zbgX", "summary": "The authors propose Seer, a video prediction model created from extending a pretrained text-to-image StableDiffusion model and finetuning additional elements.  When performing a denoised output, the per-frame convolutions are maintained from a pretrained text-to-image StableDiffusion model, however their outputs are subsequently updated by a temporal attention layer that can condition on reference/initialization frames as well as past temporal frames.  This serves to visually learn temporal consistency.  Secondly, the authors propose learning and using per-frame sub-instruction embeddings as conditioning information.  Ultimately this helps to guide each frame's generation with respect to sub-portions of the input instruction.  The authors perform experimentation on text-video datasets, and thorough ablations over different design choices of their approach.", "questions": "1. Why were 2 reference frames used for Something Something-v2, but not for Bridgedata or EpicKitchens100?  Is there something special about SSv2, or were the generated results simply empirically worse for 1-reference frame SSv2 (and why might this be the case for this particular dataset)?\n2. Seer was demonstrated for 12 and 16 frame generations, but how does such a model perform for long-form generation?  Can the final frame(s) of an output sequence be used as a reference to generate subsequent frames using the same text prompt to create a longer video?  Or would the downsampling assumptions used prevent that from being meaningful (since it might generate 12 frames spanning an entire video all at once)?  It would strengthen the work to showcase and evaluate longer-form video generation.\n3. Are there ways to probe, textually, the subinstructions learned to understand what they are encoding?\n4. The paper makes repeated reference to \"initializing\" the subinstruction embeddings to be CLIP global embeddings.  However, at the end of Section 4 on Page 6, a loss equation is provided.  Is this an additional objective that is utilized during optimization of the FSText decomposer during finetuning, or are the subinstructions explicitly initialized to the global CLIP embeddings before performing self-attention, cross-attention, and temporal attention?  This distinction is important.\n5. Given that the global CLIP embedding is used for initialization, as well as for cross-attention, there does not seem to be any additional textual or contextual data to utilize for subinstructions.  Then, subinstructions seem to be generated purely as a combination of information captured in the specific words of the global instruction; to this reviewer, the FSText decomposer therefore acts as a form of attention network over existing the global embedding.  Can the authors highlight how this is conceptually different, given that the information generated in each subinstruction is simply a subset of the global instruction?  Furthermore, this approach seems highly dependent on the initial global instruction utilized.  Are there other choices for the text embedding that could be ablated over?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "There are multiple works that have proposed generating video conditioned on text, particularly how to do so leveraging models that are pretrained to do text-to-image generation.  A large contribution of this work, in this reviewer's opinion, is the thorough exploration into what design decisions are important for the text-video generation task while making limited adjustments to a pretrained text-to-image model.  In other words, what are the important considerations that enable text-to-image models to cheaply extend to modeling videos.  The design decisions (such as what type of temporal attention is useful) are interesting insights.  This paper also focuses on natural (or photo-realistically simulated) videos, which is appreciated.  This reviewer makes the following additional, specific keyword comments:\n\n- Originality: this work is not particularly original, in that there are many efforts that try to extend a pretrained text-to-image diffusion model with minimal adjustments to perform text-to-video generation.  The architecture design decisions to extend static image diffusion models were reused from prior work, with additional novel components in the form of temporal window attention and per-frame subinstruction conditioning.\n- Quality: the quality of the work was high due to the experimentation on natural-looking videos, and the ablation insights.\n- Clarity: the paper was not the most easy to read, with certain design decisions described using unnecessarily confusing prose.  It often took multiple reads to understand what should have been an easy to describe design decision.  Furthermore, the figures were small (to the point of illegibility on printed paper) and often not particularly intuitive to understand.  I recommend the authors highlight the components of their method with larger and more intuitive figures (beyond arrows and flow-charts).  For example, in Figure 4, the output of part (a) is sub-instructions, but there is also an arrow that leads back with a \"step 1\" label, confusing the reader on how the flow of computation should be interpreted.  There is a similar issue in Figure 5, where deciphering the steps and how the arrows should be interpreted are naturally confusing (there are arrows pointing up as well as down, and all steps are shown simultaneously).  Perhaps such figures that depict multiple steps can be split into larger visuals with each step independently shown for clarity.\n- Significance: this work is of mediocre significance; it builds off of existing text-to-image models and past text-to-video approaches, but adds additional insight into what might be important to consider for future efforts along this direction (such as improving temporal consistency and fine-grained conditioning).", "weaknesses": "This reviewer believes that learning text-grounded subinstructions for conditioning is an interesting and fruitful direction forward for solving fine-grained video generation, particularly for long-form video generation.  The authors therefore make useful initial efforts towards this effort.  However, the authors generate subinstructions by 1) initializing subinstructions using as the global embeddings and 2) performing cross attention with the global embeddings.  Therefore, it appears that subsets of the same information within a global embedding is selected for each subinstruction embedding, similar to performing attention on the global embedding.  Is it possible for a model to learn proper subgoals/subinstructions in such a way, when it is constrained as a manipulation of global information.  Intuitively, for a global instruction of \"cook a pasta dinner\", subinstructions might naturally be about \"boiling water\" and \"heating up sauce\", but such information may not inherently be extractable from a global embedding of \"cook a pasta dinner\".  This reviewer would be more convinced if the global text was expanded to include intermediate info, such as by thinking \"step by step\" via a pretrained LLM, which is then converted into subinstructions per timestep.  The current formulation does not seem conceptually convincing beyond being a slightly more advanced attention head, which may be inherently limiting as an approach.  Additional investigations into what the subinstructions may be encoding in terms of textual information would be helpful.\n\nThe second weakness is with the length of video generation.  The authors state that they only generate 12 frames for SSv2 and Epic100, and 16 frames for BridgeData.  Let us focus on SSv2 for the sake of example, but the point generally holds true.  SSv2 videos are stated to have a [FPS of 12](https://developer.qualcomm.com/software/ai-datasets/something-something).  Then, essentially the Seer model finetuned on SSv2 is only able to generate 1 second of video.  Put in this context, can enough happen within one second where finegrained details such as subinstructions are necessary.  On the other hand, if the authors are downsampling videoclips by sampling frames from the entire video equally spaced out by time, then essentially Seer is only learning to produce intermediate snapshots of a given natural video.  Only for very short sequences would the result of Seer then be treated as a smooth, natural video.\n\nText-Video Prediction (TVP) was motivated by the authors as a way to help robots follow human instructions (specifically, how to map abstract language instructions to policy actions).  Indeed, the authors motivate it by stating that humans employ an imagining-planning approach, where future trajectories are imagined in their minds.  This reviewer agrees with the authors, but notes that an additional step needs to be performed to ground generated text-conditioned videos to actual policies; given the motivations, it would be stronger to demonstrate an example (even synthetic) where the videos generated by Seer was able to help policy learning either in a robot, simulated environment, or game.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VKlvNBYKQs", "reviewer": "Reviewer_KPNS", "summary": "This paper introduces the Seer model, a Language-Instructed Video Prediction with Latent Diffusion approach, for the text-conditioned video prediction (TVP) task. It enhances the U-Net architecture by incorporating a spatial-temporal attention module and introducing a Frame Sequential Text (FSText) Decomposer. This paper conducts experiments on three different datasets, demonstrating the effectiveness of the proposed method.", "questions": "Please refer to weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The authors propose a Frame Sequential Text Decomposer, which extends the capabilities of pretrained text-to-image (T2I) diffusion models along the temporal axis.\n- Seer achieves superior performance on three datasets: Something Something V2 (SSv2), Bridgedata, and EpicKitchens-100.", "weaknesses": "- While the paper repeatedly claims to have designed a sample and computation-efficient model, the only comparison provided in the appendix is in Table 8, focusing on temporal attention. It would be beneficial to include more computational comparisons to validate this claim.\n- The writing could be improved for better clarity. Additionally, the positioning of figures could be optimized for easier readability; consider rearranging Figures 3 and 4.\n- It would better to enhance the paper to include analysis of failure cases and generated video examples for a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)