GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Authors: Mingzhen Sun
Weining Wang
Zihan Qin
Jiahui Sun
Sihan Chen
Jing Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Generation
Video Autoencoder
Diffusion Probabilistic Model
TL;DR: This work presents a novel non-autoregressive method GLOBER for video generation tasks, which generates coherent videos with global guidance and maximum flexibility.

Abstract: Video generation necessitates both global coherence and local realism. This work presents a novel non-autoregressive method GLOBER, which first generates global features to obtain comprehensive global guidance and then synthesizes video frames based on the global features to generate coherent videos. Specifically, we propose a video auto-encoder, where a video encoder encodes videos into global features, and a video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner. To achieve maximum flexibility, our video decoder perceives temporal information through normalized frame indexes, which enables it to synthesize arbitrary sub video clips with predetermined starting and ending frame indexes. Moreover, a novel adversarial loss is introduced to improve the global coherence and local realism between the synthesized video frames. Finally, we employ a diffusion-based video generator to fit the global features outputted by the video encoder for video generation. Extensive experimental results demonstrate the effectiveness and efficiency of our proposed method, and new state-of-the-art results have been achieved on multiple benchmarks.

GLOBER: Coherent Non-autoregressive Video Generation via GLOBal Guided Video DecodER
Mingzhen Sun 1,2
Weining Wang Jiahui Sun 1.2 Sihan Chen 1.2
Zihan Qin 1,2
Jing Liu 1,2,*
Institute of Automation, Chinese Academy of Sciences (CASIA)
Sccooo of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
summingahen222008a.. ac weining. wangônlpr ia. qinzihan20210ia.88 sunjiahui190man ucas. cn sihan. chen@nlpr aa. jliutnlpr.. cn
Abstract
Video generation necessitates both global coherence and local realism. This work presents a novel non oon-autoregressivv method GLOBER, which first generates global features to obtain comprehensive global guidance and then synthesizes video frames based on the global features to generate coherent videos. Specifically, we propose video auto-encoder, where video encoder encodes videos into global features, and video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner. To achieve maximum flexibility, our video decoder perceives temporal information through normalized frame indexes, which enables it to synthesize arbitrary sub video clips with predetermined starting and ending frame indexes. Moreover, a novel adversarial loss is introduced improve the global coherence and local realism between the synthesized video frames. Finally, we employ a diffusion- based video generator to fit the global features outputted by the video encoder for video generation. Extensive experimental results demonstrate the effectiveness and efficiency of our proposed method and new sfate-of-the-ar results have been achieved on multiple benchmarks.
Introduction
When producing real video, it is customary to establish the overall information (global guidance), such as scene layout character actions and appearances, before filming the details (local character- istics) that draw up each video frame. The global guidance ensures coherent storyline throughout the produced video, and the local characteristics provide the necessary details for each video frame. Similar to real video production, generative models for the video generation task must synthesize videos with coherent global storylines and realistic local characteristics. However, due to limited computational resources and the potential infinite number of video frames, how to achieve global coherence while maintaining local realism remains a significant challenge for video generation tasks.
Inspired by the remarkable performance of diffusion probabilistic models 21,23, researchers have developed variety of diffusion-baaie based methods for video generation. When generating multiple video frames, strategies of existing methods can be devided into two categories: autoregression and interpolation strategies. As illustrated Fig. ()), the autoregression strategy 4 5
first generates
an initial video clip, and then employs the last few generated frames as conditions to synthesize subsequent video frames. The interpolation strategy [6] 77, depicted in in Fig. bb). generates keyframes first and then interpolates adjacent keyframes iteratively. Both strategies utilize generated video frames as conditions to guide the generation of subsequent video frames, enabling subsequent
Our codes have been released in ftsttp:://gichub..om/rvaa www.iva.oiiiiieee www.mm/ssssma.....
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Autoregression
(b) Interpolation
(c) Non-autoregression
Iteration #1
Iteration
(ours)
Generative Model
Generatve Model
Iteration #1
X2-
x
X39
X3
6: "A person playing hula hoop"
Iteration #2
Iteration #2
Generative Modill
Xx3
Xx3
X23
Generaatvv Mode
ravve Mode!
Global Video Feature
Itaation #5
Iteration
Xg
Genentive Model
1
Cerverative Modil
4
lg
X4-
Decnder of VideoAE
3
Xxy
X
X
X1
Xx
Xxi
X4.
Xxi
Xy²
X21
Xx
x42
X
Figure 1: Three strategies for multi- video generation. (a) Autoregression strategy first generates the starting video frames and then autoregressively predicts subsequent video frames. This approach is prone to accumulating errors [6], such as the fading of the hula hoop. (b) Interpolation strategy generates video keyframes first and then iteratively interpolates adjacent keyframes. This approach result in suboptimal video consistency since it is unaware of the global content. (c) Our proposed non-autoregression strategy first generates a global feature to provide global guidance, and then synthesizes local characteristics of video frames in non-autoregressivv manner. e names the condition input of video description. x; and I denote the i-th video frame and its index.
global storylines be predicted from the context of the given frames. However, since the number of conditional video frames is limited by available computational resources and is generally small, these strategies have relatively poor capacity to provide global guidance, resulting in suboptimal video consistency. In addition, the generation of local characteristics of subsequent frames refers to previous single frames, which can lead error accumulation and distorted local characteristics [6].
In this paper, we present a novel non-autoregression method GLOBER, which first generates 2D global features serve as global guidance, and then synthesizes local characteristics of video frames based on global features to obtain coherent video generation, as illustrated in Fig. [c). To be specific, propose video auto-encoder that involves video encoder and diffusion-based powerful video decoder. The video encoder encodes each input video into a 2D global feature. The video decoder decodes the storyline from the generated global feature and synthesizes necessary local characteristics for video frames. To achieve maximum flexibility, our video decoder is designed to involve no temporal modules and thus decode multiple video frames In particular, normalized frame indexes are integrated with generated global features to provide temporal information for the video decoder. this way, we can obtain arbitrary sub video clips with predetermined starting and ending frame indexes. To leverage the success of image generation models, we initialize the video decoder with pretrained image diffusion model g. Then the video auto-encoder can be trained in a self-supervised manner with the target of video reconstruction. Furthermore, we propose a Coherence and Realism Adversarial (CRA) loss improve the global coherence and local realism in the decoded video frames. For video generation, another diffusion model is used as the video generator to generate global features by fitting the output video encoder.
antruurrrrrreeivvee manner.
Our contributions are summarized as follows: (1) We propose a novel non-autee con-autoregression strategy for video generation, which provides global guidance while ensuring realistic local characteristies. (2) We introduce powerful video decoder that allows for parallel and flexible synthesis of video frames. (3) We propose a novel CRA loss to improve global coherence and local realism. (4) Experiments show that our proposed method obtains new state-of-the-art results on multiple benchmarks.
2 Related Work
Current video generation models can be divided into three categories: transformer-based methods that model discretized video signals, generative adversarial networks (GAN) and diffusion probabilistic models (DPM). The first category encodes videos to discrete video tokens and then models these
tokens with transformers |9: 10
I3 [4, MOSO [2) decomposes video scenes, objects,

---

and their motions to cover video prediction, generation, and interpolation tasks in an autoregressive manner. TATS [13 follows the interpolation strategy, and introduces time-agnostic VQGAN and time-sensitive hierarchical transformer to capture long temporal dependencies
excel at generating videos in specific domains. MoCoGAN 17 GAN-base methods 15 16
proposes to decompose video content and motion by dividing the latent space. DIGAN [S explores video encoding with implicit neural representations. StyleGAN-V [16] extends the image generation model StyleGAN [8 to the video generation task ron-autoregressiveel. However, StyleGAN-V focuses on the division of content and motion noises and ignores the importance of global guidance. As result, its content input (i.e., randomly sampled noise) contains limited information, whereas our global features can provide more valuable instructions.
DPM is a recently emerging method for vision generation work that applies DPM video generation by replacing all 2D convolutions with 3D convolutions. VIDM [22 generates videos in a frame-wise autoregression manner with two individual diffusion models. Both VDM and VIDM employ the autoregression strategy to obtain multiple video frames. In contrast, NUWA-XL [6] adopts the interpolation strategy but requires significant computational resources support parallel inference. VideoFusion [9]] targets dividing the shared and residual video signals of different video frames and modeling them respectively. It employs non-temporal diffusion model synthesize video frames based on their indexes within the pixel space. PVDM 23 utilizes 3D-to-20 2D projection to encode video into three 2D latent features for efficient video generation. In PVDM, the video decoder reconstructs video frames with fixed length (i.e. 16 frames) and fixed interval (i.e. predefined FPS). LVDM [24] encodes videos using 3D CNN through both spatial and temporal downsampling. Compared with previous methods, our method encompass the following advantages. Firstly, GLOBER can take advantage of the powerful generative capability of pretrained image diffusion models (e.g. stable diffusion) to synthesize reconstructed video frames, thus requiring much smaller dimension of latent features to represent videos. Secondly, GLOBER more flexible than prior works when decoding video frames from video latents. The video decoder in GLOBER can decode arbitrary video frames without length or interval limitations by taking the normalized indexes of target video frames as inputs. Thirdly, GLOBER is more efficient when training for video generation and synthesizing long videos. Fourthly, GLOBER obtains new state-of- state-of-the-ant the-hht-rr results on multiple benchmarks.
5 20 21
VDM 4 is the first
3 Method
In this section, we present our proposed method in details. The overall framework of our method is depicted in Fig. 2 A video sample is represented as x E xF xW xC' where F denotes the number of video frames, H the height, W is the width, and C is the number of channels. The i-th video frame is denoted as E HXXX
3.1 Video Auto-Encoder
The video auto-encoder comprises video encoder and video decoder. To reduce the computational complexity involved in modeling videos, an auxiliary image auto-encoder, which is known as KL- VAE [8] and has been validated previous studies (. is employed to encode each video frame individually into a low-resolutie
feature. The video encoder takes video keyframes as input and encodes them into 2D global features. Then the video decoder is used synthesize each video frame based on the corresponding frame index and the global feature.
are widely used models that reduce search space for vision generation. For high-resolution image generation, VAEs typically encode each image into a latent feature and then decode back to the original input image TO, 27
Frame Encoding VAES 225
To reduce the spatial details of videos in similar way, we employ a pretrained KL-VAE g to encode each
29
video frame individually. Specifically, the KL- VAE downsamples each video frame Z; by a factor of
frames obtaining a frame latent feature 2, E R xW' xc' where H' and W' are Jreame and respectively, and C" represents the number of feature channels.
Freame
Video Encoding As illustrated in Fig. 2 the video encoder is composed of an embedding feature ev. input layer, downsample module with downsample factor of fuideos mapping module, and
3

---

TVVEE
KLVAE
Video
Video Decoder
Global Feabure
Encoder
X
i i<j
noise
Video Generator
CRr
Video Discriminator
Video Encoder
Video Decoder
Video Discriminator
Global
4
Feature
Residhal Block
KLEVAE
Index Embed
T Temporal Concatenate
Spatial Conv 3x3
Full A ttention
7em
Downsample Module
Zero Conv
Downsample Conv
Mapping M Module
Upsample Module
Spatio T Temporal Module
Output Module
D Downsample
Concat & Pool
Module
Global
Linear
Feature
Middle Module
D((x,,x)))
The overall framework of proposed method GLOBER. During training, the video encoder and decoder are optimized jointly, with the video encoder encoding vidcos into global features, which used by the video decoder synthesize two randomly sampled video frames based on their corresponding frame indexes. We do not draw up the processing of timesteps and video descriptions in the video decoder for conciseness. The synthesized video frames evaluated by the video discriminator for global coherence and local realism. Then the video generator is trained to generate global features by fitting the outputs of the video encoder. During generation, the video generator generates a novel global feature, which is then decoded by the video decoder to synthesize video frames in non-autoregressive manner.
Figure 2:
an output module. The embedding feature E RRH Q2''xW"xC" xC' is randomly initialized and optimized jointly with the entire model. It should be noted that the temporal dimensions the embedding feature and each frame feature are 1, which are omitted here for brevity. Since content redundancy exists between adjacent video frames [12], we select keyframes from each input video at equal intervals and encode them individually using KL-VAE. This process produces the corresponding frame features zq1 Agg1 Agg where Qs denotes the index of the k-th selected keyframe. Then, concatenate the embedding feature e, with keyframe features along the temporal dimension, obtaining the input feature jep Zqg : =qp] E R(K+ +(++))x11 xw xC' for the video encoder.
The input layer employs a simple spatial convolution with kernel size of 3 to expand the available channel maps from C' to D, where D represents the number of new channel maps. After the input layer, the downsample module processes these keyframe features to capture spatial and temporal information, while the mapping module follows. Specifically, the downsample module is composed of residual layer, a spatial attention layer, a temporal attention layer, and a downsample convolution. while the mapping module follows similar structure but replaces the downsample convolution with a temporal split operation. Finally, the output module comprises two spatial convolutions, a group normalization layer, and SiLU activation function. It takes only the embedding part of the transformed feature as input and produces the mean and standard deviation (std) features of the global feature. Then the global feature can be sampled using the following equation:
Umean + Cstd *
where n is sampled from an isotropic Gauss distribution N (( 1), Uneon, Uetd E R xC" are the mean and std features, H" and 4 are Tundes of channels. Considering that going to model the global features for generation using
w H' and Turdes respectively, and C" is the number

---

diffusion- baseed model, which will be specified in Sec. 3.2 an additional kl loss g is employed to force the distribution of global features towards an isotropic Gauss distribution:
Lk = ( + Dedd - log v -
(2)
Video Decoding We view the synthesis of video frames conditional diffusion-based generation task. As illustrated in Fig. 2 we employ UNet [8] as the backbone of the video decoder, which can be structured into the downsample, mapping, and upsample modules. Each module starts with a residual block and a spatial attention layer, while the downsample and upsample modules additionally include downsample and upsample convolutions respectively. Following [30], each frame feature 2, (i.e. 2,) is corrupted by 'I' steps during the forward diffusion process using the transition kernel:
a (2 v1- 11--8 3,,,-- ,
<< No vāi2, (1 G.)) II
(4)
where {Be E (0, 10,1) a set of hyper-parameters, 0t 1 - 3g and ā; = II ag Based on Eq. (4). we obtain the corrupted feature 2f directly given the timestep t as follows:
2 + (1 āz)m,
(5)
where n, is a noise feature sampled from an isotropic Gauss distribution NN0, I). The reverse diffusion process q(2;="1/ |24,2 = has a traceable distribution:
q(
- 11 =9) =
22),
(6)
where 24)
Vaa, -a,
6. - -1-a- nc),
ne ~N I), and Br
1-a,-1 I=a; Br.
Given that the added noise n2 in the t-th step is the only unknown term in Eq. (6). train the UNet to predict n, from 2 condition on the timestep t, the global feature v, the frame index i, and the video description if exists. To achieve this, a timestep and index embedding layers, additional modules, and a text encoder are employed to process these condition inputs. In particular, the timestep embedding layer first obtain the sinusoidal position encoding [1] of the diffusion timestep and then passes it through two linear functions with SiLU activation interposed between them. Following previous works 30 32,, the timestep embedding is integrated with intermediate features by the residual block in each UNet module. The index embedding layer embeds the frame index in a similar way with the timestep embedding layer. The additional modules (i.e. downsample, mapping, and upsample modules) are utilized to incorporate the index embedding with the global feature and to extract multi-scale representations of the corresponding video frame. These representations are then added to the outputs of UNet modules after corvolonvoluiion. When encoding video descriptions into text embeddings, a pretrained model, namely CLIP 3 is utilized as the text encoder. During attention calculation, these text embeddings are concatenated with flattened frame features to provide cross-modal instructions. Finally, the L2 distance between the predicted noise n(24, t, i, v, c) and the added noise nı, is calculated as the training target:
Lree ||,(,t, nell2
where ||+1|a denotes the calculation of the L2 distance. The total training loss is:
(8) where λ and A2 are syper-parametery and LG is specified in the following paragraph. During generation, the video decoder synthesizes video frames given content features, frame indexes and video descriptions (if exist) by denoising noise features with multiple steps as in 30 34
L= Lree + >1 + C2
Coherence and Realism Adversarial Loss (CRA) loss to improve global coherence and local realism of synthesized video frames. To reduce computation complexity,
We propose novel Coherence and Realism Adversarial
randomly synthesize two video frames with indexes i and j. where 0 < i < < F. using the video decoder. The synthesized video frame ã, can be decoded by KL-VAE from the frame feature ž, which is obtained directly in each training step with the following formulation:
(1 (+.) n t, v.
99)
Var
5

---

where n(2f; t, i, v, is the predicted noise of n4. Then the CRA loss is calculated given the synthe- sized and real video frames using a video discriminator as depicted in Fig. 2
In our case, we expect the discriminator to provide supervision on global coherence and local realism of synthesized video frames. To ensure the effectiveness of the CRA loss, we formulate it based on the following two guiding principles: (1) the real samples selected for the discriminator must exhibit the desired correlations, whereas the fake samples must deviate from the expected patterns and are often challenging to distinguish; (2) the real samples selected for the video auto-encoder (i.e. the generator) should serve as the fake samples for the discriminator for adversarial training. In particular, to make the discriminator aware of local realism, we select (Iz. 2T) as the real sample and (z,,,y), (Tg; Zz); (ā,, 3z) as fake samples according to the first principle. This is because the latter samples contain at least one synthesized video frame and violate the target pattern of realism. Furthermore, to ensure that the discriminator is aware of the global coherence, we utilize samples that violate temporal relationships like (x,, ar)) and (Ty. F.) as fake samples of the discriminator. Then, according the second principle, Tz). (ri, Tzy) [y)) are chosen as real samples of the video auto-encoder, since these samples contain at least one synthesized video frames. The CRA loss can be formulated as C for the discriminator and L for the generator:
CD =log(1 - ((,,, + logD logD + (z), x,)) + (000))))))7)))))))))))))))000),) 2.))
1ogD)
(10)
CG =log(1 - D( ( + log(11 D( (((2,, +
(((,,,
As illustrated in Fig. 2 the discriminator is built on several spatio-temporal modules that consist of residual blocks and spatio-temporal full attentions. Considering that traditional attention is invariant to the order of input features, two position embeddings ep and ei are employed and added to the previous video frame z; and the subsequent video frame zy respectively with 0 < i < j F. These position embeddings are randomly initialized and optimized at the same time with the discriminator.
3.2 Generative Model
As shown in Fig. [] we obtain any desired video clip by feeding the frame indexes, global feature, and corresponding video description (if available) to the video decoder. Since the global feature is the only unknown term, can train conditional generative model to generate global feature based on the video description. Considering that global feature are 2D) features, the generative model can be relieved from the burden of modeling intricate video details.
In practice, global features typically have small spatial resolution, and we utilize a transformer-based diffusion model, DiT [35), to generate them. Specificcally, each 2D global feature v E R# xW"xc xc is flattened to a ID feature with length H" x W Then the diffusion and reverse diffusion processes of vy are similar to the procedures outlined in Eq. (3) and Eq. (6). except that the only condition is the video description (if exists) in the training and generation processes.
4 Experiments
In this section, we first introduce the experimental setups in Sec. 4.1
Following this, in Sec. 4.2
and Sec. [4.3] we compare the quantitative and qualitative performance of our method and prior methods on four challenging benchmarks: Sky Time-lapse (36). TaiChi-HD (37), UCF-101 [38]. and Webvid-10M (3). Finally, Sec. 4.4 presents the results of ablation studies conducted to analyze the necessity of the CRA loss, and Sec. 4.5 explores the influence of global feature shape on the generation performance.
4.1 Experimental Setups
All experiments are implemented using PyTorch [40] and conducted on 8 NVIDIA A100 GPUs, with 16-precision adopted for fast training. During the training of the video auto-encoder, pretrained KL- KL-VAEs [8] were utilized to encode each video frame z; into a latent feature 2,, with a downsample factor of fframe = 8 for 256 resolution and frrme = 4 for 1282 resolution. The latent features were then of resolution 32 The video encoder subsequently encoded the latent features of video keyframes, extracted at fixed frame indexes of [0, 5, 10, 15] for 16-frame videos, with downsample

---

Table Quantitative comparison against prior methods on UCF-101, Sky Time-lapse and TaiChi-HD datasets for video generation. denotes the conditon of video descriptions.
(a) Sky Time-lapse w/o
(c) Webvid-
UCF-101 w/c
Methods
FVD
Methods
FVD
CLIP
Methods
FVD
Resolution 256
Resolution
256
MoCoGAN- HD 41
164.1
VideoCrafters
759.3
0.2981
Resolution 128
VideoGPT
42
222.7
LVDM (2
455.5
0.2751
TGANx2 44
1209
StyleGAN-V
DIGAN L5
16
83.1
79.5
MoxelScope"
VideoFactory
43
242.4
414.1
0,3070
0.3000
TATS [3]
DIGAN [6]
465
332
GLOBER (ouns)
234.8
0.2816
MMVG 45
328
GLOBER (ours)
78.1
CogVidco Vidco 9
305
(d) UCF-101 w/o c
VIDM 22
263
(b) TaiChi-HD w/o e
VideoFusion
19
Methods
FVD
GLOBER (ours)
151.5
Methods
FVD
Resolution 128
Resolution 256
Resolution 128a
DIGAN T5
577
DIGAN [S]
471.9
SyteGGAN-V [16]
143.5
TATS L3
420
Make A------iiie A-Vidoo 46
367.2
DIGAN 15
128.1
VIDM 22
GLOBER (ours)
168.9
GLOBER (ours)
124.2
GLOBER (ours)
239.5
Table 2: Quantitative comparison against prior methods on UCF-101 for unconditional video genera- tion with 256 resolution.
Method
CogVideo 9
MagicViden
LVDM
ModelScope
VideoI.DM
Zero-Shot
V 641.8
/
FVD
701.6
699.0
6339.
550.6
Method
VideoCrafters
VideoFactory
3k]
VideoGPT
4p
MoCoGAN I7
StyleGAN-V
16
Zero-Shot
X 2880.6
X
X 1431.0
FVD
516.2
410.0
2886.8
Method
Video 9
LVDM
24
PVDM 26
GLOBER (ours)
Zero-Shot
X 372
FVD
626
343.6
252.7
factor of fvideo 2 and a number of output channels of C" 16, resulting in global features of shape 16 x 16 x 16. The dimension of C" is 4 for 256x256 resolution and 3 for 128x resolution. We first train the video auto-encoder as well as the discriminator jointly until convergence, and then train DiT with the parameters of the video auto-encoder being fixed. The video auto-encoder was trained with batch size of 40 per GPU for 80K, 40K, and 40K steps on the UCF101, TaiChi- HD, and Sky Time-lapse datasets, respectively. The loss weight X and X2 are set as 1e-6 and 0.1, respectively. When training the video generator, a Transformer-based diffusion model, DiT 35 was used as the backbone. The batch size was set to 32 per GPU, and the number of training iterations was 200K, 100K, and 100K for the UCF101, 101, TaiChi-HD, and Sky Time-lapse datasets, respectively. When generating videos, we sample each global feature with the number of DDIM steps being 50 and the unconditional guidance scale being 9.0 expect otherwise specified. Then the video decoder synthesizes target video frames parallelly with the number of DDIM steps being 50 and the unconditional guidance scale being 3.0 for the Sky Time-lapss and TaiChi-HD datasets and 6.0 for the UCF101 dataset.
4.2 Quantitative Comparison
Generation Quality Table and Table 2 reports the results of our model trained on the Sky Time-lapse, TaiChi-HD, UCF-101, and Webvid-10Mg datasets for 16-frame video generation in both unconditional and conditional settings. As shown in Table [[a), Table [[bb, and Table [[c), our method achieves comparable performance with prior state-of-the-art models on the Sky Tame-lapse. TDiChi-HD, and Webvid-10M, datasets, respectively. To comprehensively compare the performance

---

Table 3: Comparison of sampling time/memory using different methods for generating multiple video frames with resolution of 2562, batch size of 1, diffusion steps of default settings, and comparable GPU memory on v100 GPU. represents the number of video frames. AR denotes autoregression, IP denotes interpolation and NAR denotes non-autoregression.
Method
VIDM 22
VDM
LVDM
PVDM 23
VideoFusion
TATS 13
ModelScope
GLOBER
4
24
19
(ours)
NAR Video Encoding
X
X
X
/
/
V
/
NUM NUM
12288
8192
4096
16384
4096
98304
65536
32768
131072
4096
Diffusion Steps
100
100
.00
50
.00
50+50
F = 16
192s/20G
125s/11G
75s/9G
22s/7G
6e/16G
31s/6G
6s/7G
32
375s/20G
234s/11G
141s/13G
39s/9G
26s/16G
48s/8G
1ls/1IG
F = 64
771s20G
329s/11G
288w/20G
76s/13G
65s/16G
82s/12G
21s/19G
on the more challenging UCF-101 dataset, we conduct experiments both unconditional and conditional video generation resolutions of 128 and 256 For unconditional video generation, we use the fixed textual prompt "A video of person" as the video description, while for conditional video generation, we use the class name of each video as the video description. Our proposed method significantly outperforms previous state-of-the-e tfrtthe-rrt models, as shown in TTb)) (able([e) and T The superior performance of our method can be attriibuted to two factors. Firstly, we explicitly separate the generation of global guidance from the synthesis of frame-wise local characteristics. As global features are 2D features with relatively small spatial resolution, our generative model can model them effectively and efficiently without imposing high computational burden. Secondly, the synthesis of frame details can refer to the global feature, which provides global guidance such as scene layout, object appearance, and overall behaviours, making it easier for our method achieve global-coherent and local-realistic video generation.
Generation Speed We quantitatively compare the generation speed among various video gener- ation methods and report the results in Table Our GLOBER method demonstrates remarkable efficiency in generating video frames, thanks to its utilization of the non-autoregression strategy. In contrast, prior methods such as VIDM and VDM, which follow the auto-regression strategy, maintain unchanged memory but require significantly more time for generation. VideoFusion also adopts non-autoregressive strategy for generating multiple video frames. However, VideoFusion remains slower than our GLOBER, since VideoFusion directly models frame pixels, which brings huge computational burden when generating multiple video frames. TATS employs the interpola- tion strategy by first generating video keyframes and then interpolating frames between adjacent keyframes. However, it involves autoregressive generationg of video keyframes and the interpolation process is not parallelly implemented, thus being slower than our GLOBER. Notably, GPU memory of VideoFusion and our GLOBER increases with the number of video frames due to the parallel synthesis of all video frames.
4.3 Qualitative Comparison
As depicted in Figure 3 we conduct a qualitative comparison of our method with previous approaches on the UCF-101, Sky Time-lapse, and TaiChi-HD datasets. Samples of prior methods are obtained The UCF-101 dataset records 101 human actions and is the most challenging and diverse dataset. When applied this dataset, GAN-based methods such DIGAN and StyleGAN- V generate video samples that lack distinctiveness. In contrast, TATS, which is built on Transformer and utilizes the interpolation strategy, generates video samples that are more identifiable. In comparison to TATS, diffusion- methods like VideoFusion [19] and VIDM (22) produce samples with more pronounced appearances. However, VideoFusion generates slightly blurred object appearances, and VIDM generates overly trivial object motions. Conversely, our GLOBER generates samples that exhibit both distinct appearances and conspicuous movements. On the Sky Time-lapse dataset, samples generated by DIGAN, StyleGAN-V, and TATS display trivial motions and simplistic objects. VideoFusion and VIDM generate samples with enhanced details and more discernible boundaries, while their motion remains somewhat negligible. In contrast, our GLOBER generates video samples with more dynamic movements and significantly richer visual details. Similarly, on the TaiChi- HD dataset, the human appearances generated by DIGAN, StyleGANVV and TATS are noticeably
from 22 19
8

---

DIGAN
SkyWGANNY
TATS
Video
VIOI
GLOBER (ound)
Figure 3: Quantitative comparison against previous methods on the UCF-101 (left). Sky Time- lapse (middle) and TaiChi-HD (right) datasets. Our showcased samples on the UCF-101 datasets are produced using the video descriptions "Apply Lipstick", "Soccer Juggling", and "Pull Ups", respectively. Samples on the Sky Tame-lapse and TaiChi-HD datasets are generated using fixed video descriptions being "A time-lapse video of sky" and "Tai chi", respectively.
Table 4: Ablation study the CRA loss.
Datasets
loss
rFVD
w/o
106.7
UCF-101
w/
69.7
w/o
84.3
Sky Time-lapse
w/
63.5
w/o
91.3
Visualization samples synthesized with or without the CRA loss by the video auto-encoder.
Figure 4:
TaiChi-HD
w/
60.1
distorted. While VideoFusion and VIDM achieve improved human appearances, their movements remain trivial. In contrast, samples generated by our GLOBER exhibit significant movements and distinct object appearances.
4.4 Ablation Study on the CRA Loss
To investigate the effectiveness of our proposed CRA loss, we conducted an ablation study where the CRA loss was removed during the training of the video auto-encoder. The quantitative results are presented Table 4 which unequivocally demonstrate the effectiveness of our CRA loss all three benchmarks: T01, Sky Time-lapse, and TaiChi-HD, with remarkable reduction in FVD scores by 37.0, 20.8, and 31.2, respectively. In addition, we qualitatively compare videos synthesized by models with without the CRA loss in Fig. 4 which clearly shows that videos generated using the CRA loss exhibit better video consistency and more distinct local details, demonstrating the necessity of our proposed CRA loss. The effectiveness of our CRA loss can be attributed to two key factors. Firstly, by penalizing samples that violate temporal relationships, the CRA loss enhances the overall coherence of the synthesized video frames. Secondly, by utilizing pairwise video frames as inputs for

---

the discriminator, the CRA loss can identify and penalize video frames that exhibit inconsistent or distorted local characteristics, which further enhances the realism of the synthesized videos.
4.5 Impact on the Global Feature Shapes
We experiment the effect of different shapes of the global features the quality of synthesized video frames. As reported in Table [ videos generated using global features with shape of 8 x 8 x 64 exhibit lower quality than those generated using global features with shape of 16 x 16 x 16. And the use of global features with a shape of 16 x 16 32 brings further improvement of 39.2 in FVD score. It may be due to two reasons. Firstly, our experiments are conducted on the UCF-101 dataset, which contains complex motions that require more channels to be effectively represented. Secondly, by utilizing the KL loss to constrain the distribution of global features toward an isotropic Gaussian, becomes easier for the generative model to fit the distribution of the global features even with a number of channels, thus increasing the number of channels could bring improved performance.
Table 5: Sensitivity analysis on the shape of global features on the UCF-101 datasets.
H" W" x C"
FVDrec +
FVD +
8 8 64
276.1
725.1
16 16 x 16
189.7
560.4
16 x 16 x 32
184.9
521.2
Conclusion and Limitations
This paper introduces GLOBER, novel diffusion biffusion-based video generation method that emphasizes the significance global guidance multi-frame video generation. Our method offers three distinct advantages. Firstly, it alleviates the computation burden of modeling video generation by replacing video signals with 2D global features. Secondly, utilizes a non-autoregressive generation strategy, enabling the efficient synthesis of multiple video frames and surpassing the performance of prior methods in terms of efficiency. Lastly, by incorporating global features as guidance, our method generates videos with enhanced coherence and realism, achieving new state-of-the-a art results on multiple benchmarks. Nevertheless, our research exhibits several limitations. Firstly, we find GLOBER difficult to process videos with frequent scene changes, such transitions have the potential to disrupt the video coherence. Furthermore, we have not explored the performance of GLOBER in open-domain video generation tasks due computational resource constraints. Future works are encouraged to solve the above issues.
Acknowledgements
This work was supported by the National Key Research and Development Program of China 200.2022 18801), National NaturalScience Foundation of China (U21B2043.62102416 62206279, 62102419).
References
[1] C. Saharia, W. Chan, S. Saxena, I.. I.i, J. Whang, F. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., "Photorealistic text-to-image diffusion models with deep language understanding." Advances in Neural Information Processing Systems, vol. 35, pp. 36 479-36 494, 2022.
[2] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditions image generation with clip latents," arXiv preprint arXiv:2204.06125 2022.
[3] L. Zhang and M. Agrawala, "Adding conditional control to text-to-image diffusion models." arXiv preprint vmXiv:2302.05533 2023.
[4] Ho. Salimans, A. A. Gritsenko, Chan, M. Norouzi, and D. Fleet, "Video diffusion models." Advances in Neural Information Processing Systems, 2022.
[5] V. Voleti, A. Aalicoeu--aartneee and C. Pal, "Masked conditional video diffusion for prediction, genera- tion, and interpolation," Advances in Neural Information Processing Systems, 2022.
[6] S. Yin, C. Wu, H. Yang. J. Wang, X. Wang, M. Ni. Z. Yang. L. Li. S. Liu, F. Yang et al., "Nuwa-xl: Diffusion over diffusion for extremely long video generation," arXiv preprint 2rXiv:2303.12344 2023.
10

---

[7] A. Blattmann. R. Rombach, H. Ling. T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, "Align your latents: High-resolution video synthesis with latent diffusion models," arXiv preprini arXiv:2304.08818, 2023.
[8] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, HHigh-resoluuee image synthesis with latent diffusion models." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 10684-10695.
19] Hong, M. Ding, W. Zheng, X. Liu, and J. Tang, "Cogvideo: Large-scale pretraining for text-to-video generation via transformers," arXiv preprini 2xXi::2205./8865 2022.
[10] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan, "Nüwa: Visual synthesis pre-training for neural visual world creation." in European Conference on Computer Vision, 2022, pp. 720-736.
[11] J. Liu, W. Wang, S. Chen, X. Zhu, and J. Liu, "Sounding video generator: A unified framework for text-guided sounding video generation." IEEE Transactions on Multimedia, pp. 1-13, 2023.
[12] M. Sun, W. Wang, X. Zhu, and J. Liu, "Moso: Decomposing motion, scene and object for video prediction." 2023.
[13] S. Ge, T. Hayes, H. Yang. X. Yin, G. Pang, D. Jacobs, J.-B, Huang, and D. Parikh, "Long video generation with time-agnostic vqgan and time-sensilive transformer," European Conference on Computer Vision, 2022. pp. 102-118.
[14] A. Gupta, S. Tian, Y. Zhang, J. Wu, R. Martin-Martia, and L. Fei-Fei, "Maskvit: Masked visual pre- pre-training for video prediction." arXiv preprint arXiv:2206. 18994, 2022.
[15] S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J. Ha, and J. Shin, "Generating vidcos with dynamics-aware implicit generative adversarial networks," in International Conference Learning Representations, 2022.
[16] I. Skorokhodov, S. Tulyakov, and M. Elhoseiny, "Stylegan-v; A continuous video generator with the price, image quality and perks of stylegan2," " in Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition. 2022, pp. 3626-3636.
[17] S. Tulyakov, M.-Y. Liu, X. Yang, and Kautz, "Mocogan: Decomposing motion and content video generation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp.
[18] T. Karras, S. Laine, and T. Aila, "A stylc-based generator architecture for generative adversarial networks," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2019, pp. 4401-4410.
[19] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang. Y. Shen, D. Zhao, J. Zhou, and T. Tan, "Decomposed diffusion models for high-qualit video generation," 2023.
[20] Harvey, S. Naderiparizi, V. Masrani, C. Weilbach, and Wood, "Flexible diffusion modeling of long videos," arXiv preprint ar arxiv:2205.1 1495, 2022.
[21] P. Esser, J. Chiu, P. Arighehchian, J. Granskog, and A. Germanidis, "Structure and content content-guided video synthesis with diffusion models," arXiv preprint arXiv:2302.03001, 2023.
[22] K. Mci and V. M. Patel, "Vidm: Video implicit diffusion models," arXiv preprint arXi::2212.0025:: 2022.
[23] S. Yu, K. Sohn, S. Kim, and J. Shin, "Video probabilistic diffusion models in projected latent space," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp.
18 4564456866 18--18846 466.
[24] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q Chen, "Latent video diffusion models for high-fidelity vidco generation with arbitrary lengths," arXiv preprint arXiv:221 2022.
[25] A. den Oord, O. Vinyals, and k. kavukcuoglu, "Neural discrete representation learning." " in Advances in Neural Information Processing Systems, 2017, pp. 6306-6315.
[26] A. Razavi, A. van den Oord, and O. Vinyals, "Generating diverse high-fidelity images with vq-vae-2," in Advances in Neural Information Processing Systems, 2019, pp. 14 1866-14876.
[27] P. Esser. R. Rombach, and B. Ommer, "Taming transformers for high-resolution image synthesis." in Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 12 873 12 883.
11

---

[28] M. Ding, Z. Yang. W. Hong, W. Zheng, C. Zhou, D. Yin, J. Lin, X. Zou, Z. Shao, H. Yang, and J. Tang. "Cogview: Mastering text-to-image generation via transformers," in Advances in Neural Information Processing Systems, 2021, pp. 19 922-19835.
[29] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, "Zero-shot text-to-image generation,"" in International Conference on Machine Learning, 2021, pp. 8821-8831.
[30] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," " Advances in Neural Information Processing Systems, vol. 33. pp. 6840 6851, 2020.
[3]] A. Vaswani. N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in nearal information prucessing systems, vol. 30, 2017.
[32] A. Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in International Conference on Machine Learning, 2021. pp. 8162-8177
[33] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark et al., "Learning transferable visual models from natural language supervision." in International Conference on Machine Learning, 2021, pp. 8748-8763.
[34] Song. C. Meng, and S. Ermon, "Denoising diffusion implicit models," International Conference on Learning Represemtations, 2021.
[35] Peebles and Xie, "Scalable diffusion models with transformers," " arXiv preprint arXiv:2212.097455 2022.
[36] Zhang, C. Xu, L. Liu, M. Wang. X. Wu, Y. Liu, and Y. Jiang, "Dtvnet: Dynamic time-lapse video generation via single still image," in European Conference on Computer Vision. 300-315.
Springer. 2020, pp.
[37] A. Siarohin, S. Lathuilière, S. Tulyakov, F. Ricci, and N. Sebe, "First order motion model for image animation," Advances in Neural Information Processing Systems, vol. 32, 2019.
[38] K. Soomro, A. R. Zamir, and M. Shah, "UCFI01: A dataset of 101 human actions classes from videos in the wild," CoRR, vol. abs/1212.0402, 2012.
[39] M. Bain, A. Nagrani, G. Varol, and A. Zisserman, "Frozen in time: A joint vidco and image encoder for end-to-end retrieval," in IEEE International Conference Computer Vision, 2021.
[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradhury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., "Pytorch: An imperative style, high-performance deep learning library," Advances in neural information processing systems, vol. 32, 2019.
[41] Y. Tian, J. Ren, M. Chai, K. Olszewski, X. Peng, D. Metaxas, and S. Tulyakov, "A good image generator is what you need for high-resolution video synthesis." arXiv preprint arXiv:2104. 15069, 2021.
[42] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas, "Videogpt: Video generation using vq-vae and transformers," arXiv preprint X00: 100.11000.111..... 20251,2021.
[43] W. Wang, H. Yang, Z. Tuo, H. He, Zhu, J. Fu, and Liu, "Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation," " arXiv preprint arXiv:2305. 10874, 2023.
[44] Y. He, T. Yang, Y. Zhang, Y. Shan, and Q Chen, "Latent video diffusion models for high-fidelity vidco generation with arbitrary lengths," arXiv preprint 1.Xxv:2211.1322111 2022.
[45] T.J.. Fu, I.. Yu, N. Zhang, C.-Y. Fu, J.-C. Su, W. Y. Wang, and S. Bell, "Tell me what happened: Unifying text-guided video completion via multimodal masked video generation," " arXiv preprint 1.Xx:22211112224 2022.
[46] U. Singer, A. Polyak, T. Hayes, X. Yin, An, S. Zhang, Hu, H. Yang. O. Ashual, O. Gafni et al., "Make-a-video: Text-to-video generation without text-video data," arXiv preprint arXiv:2209. 14792, 2022.
[47] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, "Magievideo: Efficient video generation with latent diffusion models." arXiv preprint arXiv:2211. 2022.
12

---

A
Appendix
B Broader Impact
The goal this work to advance research on video generation methods. Our method has the potential to facilitate the workflow of film production and animation, exhibiting positive influence on creative video applications. Since our method trained mainly on domain-specific datasets, the potential deleterious consequences of exploiting misinformation or producing fake videos, seem to be insignificant. Nevertheless, it remains crucial to apply an abundance of caution and implement strict and secure regulations.
model for malicious purposes, such as spreading
(a) Sky Time-lapse
(b) UCF-101
Figure 5: Genetated long videos with 128 frames the Sky Time-lapse and UCF-101 datasets (4 frames skipped).
C Experimental Results on Long Video Generation Tasks
We obtain new state-of-the-art results on the SkyTimelapse and UCF-101 datasets for long video generation tasks. All experiments are conducted without conditional inputs. The quantitative results
13

---

reported in Table MoCoGAN, McCoGAN-HD, DIGAN, and StyleGAN-V V are GAN-based methods, which dominate the field of vision generation until 2022. Based diffusion probabilistic models, VIDM outperforms these GAN-based methods by large margin. However, VIDM employs the auto-regression generation strategy to generate long videos, which lacks global guidance and suffers from error accumulation. Our method, GLOBER, outperforms VIDM significantly due to its incorporation global features and non-autoregressionn generation strategy. We present several video samples in Fig. which demonstrate that our method can generate long videos of remarkable quality.
Table 6: Quantitative Results of FVD comparison on the SkyTimelapse and UCF-101 datasets for 128 frame long video generation.
Method
UCF-101
Sky Time-lapse
MoCoGAN [CVPR18]
3679.0
575.9
+StyleGAN2 backbone
2311.3
272.8
MoCoGAN-HD [ICLR21]
2606.5
878.1
DIGAN [ICLR22]
2293.7
196.7
StyleGAN-V [CVPR22]
1773.4
197.0
VIDM [AAAI23]
1531.9
140.9
GLOBER (ours)
1177.4
125.5
D
More Qualitative Results
We present more qualitative results the UCF-101, Sky Time-lapse, and TaiChi-HD datasets in the link: httpss//iva mzsun. gi tit. io/GLBB
E
Sensitivity Analysis of Unconditional Guidance Scale
We investigate the effectiveness of the unconditional guidance scale y that is used when employing class-condition constraints. Table presents the influence of the FVD score of videos condition- ally decoded by the video decoder on the UCF-101 256 benchmark. Table presents the influence of µu on the FVD score of videos conditionally sampled by the video generator the UCF-101 256 dataset. is evident that appropriate selection of the unconditional guidance scale is important in ensuring the quality of videos decoded sampled with class conditions.
Table 7: Sensitivity analysis of the unconditional guidance scale μ for video reconstruction on the UCF-101 dataset.
3
12
15
FVD
211.4
114.3
106.7
133.2
281.1
670.8
Table 8: Sensitivity analysis of the unconditional guidance scale for video generation the UCF-101 dataset.
3
12
15
FVD
575.6
173.0
172.6
171.5
168.9
224.3
F
Quality of frame features generated by Equ. (9)
When adopting the CRA loss, we directly obtain the synthesized frame feature 2, according to Equ. (9). Considering that diffusion models typically perform T denoising steps to synthesize realistic sample, we explore the rationality of this design choice. As depicted in Fig. 6 the quality deterioration in estimated video frames is acceptable for most samples. It is reasonable since the diffusion process under the guidance of global features, which contain sufficient local and global information.
14

---

Ground Truth
T=200
T=400
T=600
T=800
T=900
Figure 6: Video frames estimated by Equ. (9) with different timesteps on the UCF-101 dataset with 256 x 256 resolution.
G Necessity to consider "keyframes" for extracting global features from given video
It's time- and computation- consuming to process all video frames for extracting global features, especially when encode long videos such as 128. As depicted in Fig. 7 when the number of input video frames increasing, the maximum training batchsize dramaticly decreases and the required training time boosts. Thus is necessary keyframes rather than all video frames to obtain better training efficiency.
80
4000
Maximum training batch size
3500
60
3000
2500
40
Trining time(s)
30
1500
20
1000
10
500
120
input video frames
of input video frames
Figure 7: Comparison of the maximum training batch size and required time using different number of input video frames.
H More Efficiency Comparison
We compare the generation efficiency of our GLOBER with prior works, e.g. PVDM, with comparable settings as reported in Table. 99) Results with are taken from PVDM and measured with a single NVIDIA 3090ti 24GB GPU. The rest are evaluated on single NVIDIA 3090 24GB GPU by us due to lack of 3090ti. LVDM has not released models and scripts for 128-frame video generation. Limited by memory, our GLOBER decodes a 128-frame video by parallelly and no-overlapped
15

---

decoding every 32 video frames. DiT is the video generator of GLOBER. The results demonstrate that our GLOBER is more efficient than PVDM and LVDM when training for video generation and synthesizing long videos.
Table 9: Maximum training batch size and required inference time (in seconds) for different methods to synthesize a 256x256 resolution video.
Method
Train Batch Size
Inference Time (16
Interence Time (128 frames)
frames)
TATS
84.8
434
VideoGPT*
139
N/A
VDM
LVDM
N/A
PVDM-I,*
2
20.4
166
GLOBER (ours)
4
21.4
145.7
GLOBER (DiT only)
8
3.57
3.57
I Ablation Study on the KL loss
Following { the KL loss, i.e. Equ. (2), punish the distribution of latent features towards standard normal distribution to avoid high shift and high variance. We conduct an ablation study on the variance prediction and the KL. loss. As reported in Tā) removing variance prediction brings improvements on rFVD, but deteriorates FVD significantly since the video decoder is no longer robust disturbance of generated features. Adding variance prediction improves the generation performance to some extent. The performance further boosts after employing the KL loss since the KL loss brings neglectable decrease on rFVD but can effectively facilitate DiT to model the distribution of global features using the diffusion theory.
Table 10: Ablation study variance prediction and the KL loss. All experiments conducted on the TaiChiHD dataset for 16-frame video generation with 256 resolution. Autoencoders are trained for 1500 epochs (15h) and DiTs trained for 2000 epochs (14h).
Design of the video anto-en
rFVD
FVD
Determinstic (w/a variance)
773.4
variance prediction KI. loss) variance prediction KL. loss
71.5
$49.5
75.3
332.7
J Settings of Hyper Parameters
The detailed settings of model hyper parameters are presented in Table 11
16

---

Table 11: Hyper-parameters of the video auto-encoder and the quantitative results on video recon- struction. Experimental settings on the UCF-101 dataset are the same for both conditional and unconditional video generation except given video descriptions.
UCF-101 256
Sky Tame-lapse 256²
TaiChi-HD 1282
Batch Size Learning Rate
40
32
32
1e-5
le-4
5e-5
KL- VAE
f frame
8
4
Video Encoder
foideo
2
Input Shape
32
Input Channels
4
Output Channels
16
Model Channels
320
Num Res. Blocks
2
Num Head Channels
64
Attention Resolutions
[16, 8]
Channel Multiplies
[1,2]
Video Decoder (UNet)
Input Shape
32
Input Channels
4
Output Channels
4
Model Channels
320
Num Res. Blocks
2
Num Head
8
Attention Resolutions
[32, 16, 8]
Channel Multiplies
[1, 2, 4, 4]
Video Generator (DiT)
Input Shape
16
16
16
Input Channels
16
16
16
Model Channels
1152
1024
1024
Num Head
16
Depth
28
20
20
Mip Ratio
4
17


Reviews:
[{"review_id": "vNevNdJkb8", "reviewer": "Reviewer_4poX", "summary": "The study introduces a unique non-autoregressive approach called GLOBER. This method initially generates global features, offering a thorough global guidance, which then synthesizes video frames using these global features to produce cohesive videos. Furthermore, the study suggests a coherence and realism adversarial loss to improve the quality of the videos.", "questions": "Please refer weakness", "limitations": "The authors have discussed limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The suggested non-autoregressive technique is simple and effective.\n\nThe empirical tests and ablation studies conducted are adequate", "weaknesses": "For Table 2, the absence of some contemporary methods such as modelscope implies that the assertion regarding inference time and GPU memory may not be as robust as claimed.\n\nWhat strategies are in place to ensure that the distribution of global features produced by DiT during inference aligns with the features acquired by the video encoder during the training phase?\n\nI'm intrigued to find out whether this non-autoregressive approach is effective with lengthy videos, for example, those consisting of 128 or 256 frames. If it's not, the benefits of this method could be significantly reduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R1.1 More efficiency comparison with contemporary methods.\n[Author] We present more efficiency and quality comparsion with contemporary methods like Modelscope, LVDM, and PVDM in our global rebuttal **[Common Question 1]**, please view it for more details.\n\n\n### R1.2 Align the distribution of global features.\n[Author] We first train the video auto-encoder as well as the discriminator jointly until convergence, and then train DiT with the parameters of the video auto-encoder being fixed to ensure the alignment. As specified in L53-54 and L191-194, DiT is an independent generative model and is optimized after we finish training the video auto-encoder. The training setups for the video auto-encoder and DiT are also separately presented in Section 4.1.\n\n### R1.3 The effectiveness of GLOBER on synthesizing long videos.\n[Author] Given that 128 is a commonly used length when testing long video generation, we have presented quantitative and qualitative results of 128-frame video generation on the UCF-101 and Sky Time-lapse datasets with 256x256 resolution in the appendix, which demonstrate the effectiveness of our GLOBER on long video generation. Samples are also visualized in the link of the A.3 section in the appendix.\n\n"}, {"review_id": "fdtlY0Ofxe", "reviewer": "Reviewer_xuBL", "summary": "In this paper, the author studies the text-to-video task and proposes a method called GLOBER. The proposed method first generates a global guidance feature, then the video frames are generated through a diffusion model that takes the frame index as a condition. An adversarial loss is also proposed to improve global coherence and local realism.", "questions": "Please refer to the weakness part.", "limitations": "Please refer to the weakness part.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The overall presentation of the proposed method is clear and easy to follow.\n2. The author conducts experiments on three widely used datasets (e.g., Sky Time-lapse, TaiChi-HD and UCF-101).", "weaknesses": "1. As the author claims their method is capable of generating videos from text. It would be great if the author could compare their methods with SOTA open-sourced methods (DAMO-text2video, VideoCrafters, CogVideo and VideoFactory) on WebVid-10M.\n2. The author should also consider comparing their methods with PVDM.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R2.1 Experiments on WebVid-10M.\n[Author] We present quality comparsion with ModelScope (DAMO-text2video)[1], VideoCrafters[2], LVDM[3], and VideoFactory[4] on WebVid-10M in our global rebuttal **[Common Question 1]**, please view it for more details. We find the open-sourced code of CogVideo does not support parallel generation given different input descripitons and requires ~10 minutes to synthesize a video, which is time-consuming thus we only add comparison with CogVideo on the UCF-101 dataset.\n\n[1] VideoFusion: Decomposed diffusion models for high-quality video generation. 2023. CVPR.\n[2] Inference with codes from https://github.com/VideoCrafter/VideoCrafter\n[3] Latent video diffusion models for high-fidelity long video generation. 2022. ARXIV.\n[4] VideoFactory: Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generation. 2023. ARXIV.\n\n### R2.2 Comparison with PVDM.\nThanks for bringing this to our attention, we discuss the differences between PVDM and our GLOBER, and then analyze the advantages and disadvantages of our GLOBER compared to PVDM as follows (all reported results of GLOBER are evaluated using previous checkpoints), which we will add to the paper later.\n\n#### Differences\n- Motivation and auto-encoder: To address the computation- and memory-inefficiency problem of video generation, PVDM utilizes 3D-to-2D projection to encode a video into three 2D latent features, and reconstructs videos in a deterministic manner, where the video decoder directly decodes local characteristics of reconstructed frames from the latent features. Different from PVDM, our GLOBER focuses on providing global guidance for long-video generation and encodes a video into a single latent feature. Thus GLOBER utilizes the powerful generation capability of the pretrained image diffusion model to synthesize local characteristics of reconstructed images, which significantly reduces the total dimension of flattened latent features.\n\n- Video generator and generation strategy: The video generator of PVDM has to fit three 2D latents simultaneously for each video, and modeling long video generation in an autoregressive manner (i.e. generating following 16 frames given previous 16 generated frames). However, the video generator of GLOBER only needs to model one 2D latent for each video and employs a non-autoregressive generation strategy to synthesize long videos (i.e. generating 128 frames with one sampling).\n\n#### Advantages of GLOBER compared to PVDM:\n-\tGLOBER can take advantage of the powerful generative capability of pretrained image diffusion models (e.g. stable diffusion) to synthesize local characteristics of reconstructed video frames under the guidance of global features, thus requiring a much smaller dimension of latent features to represent videos as demonstrated in Table R2.2.1.\n\nTable R2.2.1 Comparison on the dimension of flattened latent features for representing a T-frame video with HxW resolution. LVDM encodes videos using 3D CNN with the spatial and temporal downsampling rates being 8 and 4 respectively, obtaining latents with shape (T//4, H//8, W//8, 3). PVDM utilizes 3D-to-2D projection to encode a video into three 2D latents with shape (T, H//d, 4), (T, W//d, 4) and (H//d, W//d, 4) respectively, with d=8 in default. GLOBER extracts video global information and encodes an input video into a latent feature with shape (H//16, W//16, 16).\n| （T, H, W, C） |  LVDM  | PVDM | GLOBER(ours) |\n| :---: | :---: | :---: | :---: |\n| (16, 256, 256, 3) | 12288 | 8192 | **4096** |\n| (128, 256, 256, 3) | 98304 | 65536 | **4096** |\n\n- GLOBER is more flexible than PVDM when decoding video frames from video latents. In PVDM, the video decoder reconstructs video frames with fixed length (i.e. 16 frames) and fixed interval (i.e. predefined FPS). However, the video decoder in GLOBER can decode arbitrary video frames without any length or interval limitation by taking the normalized indexes of target video frames as inputs.\n\n- GLOBER is more efficient than PVDM when training for video generation and synthesizing long videos as demonstrated in Table Q1.2 in **[Common Question 1]**. \n\n- As reported in Table. R2.2.2, GLOBER obtains better performance than PVDM on UCF-101 for 16-frame video generation and on SkyTimelapse for 128-frame video generation. It is reasonable since by initializing from pretrained image diffusion model, the video decoder in GLOBER becomes much more powerful than that in PVDM, thus enhancing the generation of complex videos like UCF-101. For the small domain dataset SkyTimelapse, the global guidance provided by our GLOBER is able to improve the global coherence of synthesized long videos, thus obtaining better FVD.\n\nTable R2.2.2 Quantitative comparison for video generation with the resolution of 256x256. N/M-s for PVDM names N DDIM steps for genration the initial video clip and M DDIM steps for synthesing following video clips. N/M-s for GLOBER means N DDIM steps for the generation of global features and M DDIM steps for decoding video frames.\n| |  |  UCF-101 |  | SkyTimelapse |\n| :---: | :---: | :---: | :---: | :---: |\n| | FVD16 | Total Sampling Steps &#124; | FVD128 | Total Sampling Steps |\n| StyleGAN-V | 1431.0 | - | 197.0 | - |\n| PVDM-S; 100/20-s | 457.4 | 100 | 159.9 | 240 |\n| PVDM-L; 200/200-s | 398.9 | 200 | 137.2 | 1600 |\n| PVDM-L; 400/400-s | 343.6 | 400 | 125.2 | 3200 |\n| GLOBER (ours); 50/50-s | 252.7 | 100 | 125.5 | 100 |\n| GLOBER (ours); 100/100-s| **248.9** | 200 | **122.4** | 200 |\n\n#### Disadvantages of GLOBER compared to PVDM:\n- PVDM poses stronger constraints on the correlations of adjacent video frames than GLOBER, thus obtaining better performance than GLOBER in short-video generation on simple domain dataset like SkyTimelapse and long-video generation on multi-action datasets like UCF-101."}, {"review_id": "KA9NpkBPTW", "reviewer": "Reviewer_JPZt", "summary": "This work introduces a novel non-autoregressive method, GLOBER, that first generates global features for comprehensive global guidance and then synthesizes video frames based on these global features to produce coherent videos. The authors propose a video auto-encoder to encode videos into global features and a video decoder to decode the global features and synthesize video frames in a non-autoregressive manner. Notably, the video decoder uses normalized frame indexes to perceive temporal information, allowing it to synthesize any video clips with predetermined frame indexes. The authors also introduce a unique adversarial loss to enhance global coherence and local realism of the synthesized video frames. Finally, a diffusion-based video generator is employed to fit the global features produced by the video encoder for video generation. The effectiveness and efficiency of the proposed method are demonstrated through extensive experiments, and it sets new state-of-the-art results on multiple benchmarks.", "questions": "Please refer to the weaknesses.", "limitations": "In this work, the authors mentioned limitations and broader impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The inclusion of Coherence and Realism Adversarial Loss is a novel approach compared to previous diffusion-based architectures.\n2. Extensive experiments have been performed on various benchmarks, all demonstrating the significance of GLOBER.", "weaknesses": "1. The authors identify VideoFusion as the most closely related work due to its use of non-autoregressive generation. However, there are other public models, such as ModelScope Text-to-Video, that use non-autoregressive generation in the latent space similarly to the authors' work. I suggest that the authors compare their work to these models as well.\n2. It appears that GLOBER outperforms VideoFusion in all tasks, which generates videos in the pixel space. This superiority seems to result from the CRA loss proposed by the authors. Therefore, a direct comparison between GLOBER (without CRA loss) and VideoFusion would be intuitive. However, inconsistent results in Table 1 and 3 make this comparison unfeasible. Could the authors explain this inconsistency and provide justifications for GLOBER's superiority over VideoFusion, aside from the CRA loss?\n3. My interpretation of Equation 9 suggests it's an estimation of the frame feature. However, this estimation might not be accurate because, like in DDPM (or DDIM), one could perform T steps of reverse denoising to generate images. What is the quality difference between these two types of images? I assume that the frame feature generated by Equation 9 will be of lower quality.\n4. In Table 2, why does GLOBER use 50+50 diffusion steps?\n5. The optimization objective of Equation 2 in the Video Encoding section appears to be derived from the Variational Autoencoder. Could the authors provide a justification for this design? My understanding is that video encoding trains a dataset-dependent distribution to be sampled as z_t.\n6. In Line 127, \"Gauss distribution\" appears to be misspelled.\n7. The authors seem to have overlooked specifying the dimension of C'.\n\nMy score could be revised upward if my concerns are adequately addressed.\n\n**Reference:**\n\n[1] ModelScope Text-to-Video Technical Report, arXiv. (Model: **https://modelscope.cn/models/damo/text-to-video-synthesis/summary**).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R3.1 More comparison with contemporary methods.\nWe present more efficiency and quality comparsion with contemporary methods like Modelscope, LVDM, and PVDM in our global rebuttal **[Common Question 1]**, please view it for more details.\n\n### R3.2 \n#### R3.2.1 Inconsistency of Table 1 and Table 3. \n[Author] Table 1 reports the generation performance that involves both the video auto-encoder and DiT, while Table 3 measures the reconstruction FVD that only involves the video auto-encoder as specified in L276-277. We will modify the FVD in Table 3 to FVDrec to avoid confusion in the paper later.\n#### R3.2.2 Comparison of VideoFusion and GLOBER without CRA is unfair. \n[Author] Different from VideoFusion, which naturally incorporates constraints on the local consistency of frame-wise characteristics by decomposing video components, our GLOBER requires the CRA loss to obtain local constraints by punishing video frames that violate such consistency. Thus it may be unfair for our GLOBER to compare with VideoFusion without the CRA loss.\n\n### R3.3 Quality difference between two types of images is acceptable.\n[Author] As depicted in Fig. R3.3.1 **[in the PDF file]**, the quality deterioration in estimated video frames is acceptable for most samples. It is reasonable since the diffusion process is under the guidance of global features, which contain sufficient local and global information.\n\n### R3.4 Why we use 50+50 diffusion steps in GLOBER.\n[Author] As specified in L220-224, 50+50 diffusion steps are the default settings for all our experiments. Such setups follow the setting of stable diffusion, which uses 50 ddim steps in default, and obtains well efficiency and generation quality as reported in our experiments.\n\n### R3.5 The design of Equation 2.\n[Author] Our target is exactly to train global features to conform to a dataset-dependent distribution similar to z_t. In fact, the distribution of z_t is also punished towards a standard normal distribution during training by employing the KL loss (Equation 2) to avoid high shift and high variance, as specified in [1]. The formulation of the KL loss is derived from the KL divergence between two multi-variable Gauss distributions (one is the distribution outputted by the video encoder and another is the standard Gauss distribution), and the KL-penalty is slight with a small loss weight of 1e-6 as specified in L217 following [1], thus the final distribution of global features is still dataset-dependent. The effectiveness of the KL loss has been proven in [1]. For better understanding, we conduct an ablation study on the variance prediction and the KL loss. As reported in Table R3.5.1, removing variance prediction brings improvements on FVDrec, but deteriorates FVDgen significantly since the video decoder is no longer robust to disturbance of generated features. Adding variance prediction improves the generation performance to some extent. The performance further boosts after employing the KL loss since the KL loss brings neglectable decrease on FVDrec but can effectively facilitate DiT to model the distribution of global features using the diffusion theory.\n\nTable R3.5.1 Ablation study on variance prediction and the KL loss. All experiments are conducted on the TaiChiHD dataset for 16-frame video generation with 256^2 resolution. Autoencoders are trained for 1500 epochs (15h) and DiTs are trained for 2000 epochs (14h).\n| Design of the video auto-encoder | FVDrec | FVDgen |\n| :-: | :-: | :-: |\n| Determinstic (w/o variance) | 68.9 | 773.4 |\n| +variance prediction (w/o KL loss) | 71.5 | 549.5 |\n| +variance prediction + KL loss | 75.3 | 332.7 |\n\n[1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models”.\n\n### R3.6 Spelling Error.\n[Author] Thank you for bringing this to our attention. We will rectify the spelling error in revision.\n\n### R3.7 The dimension of C'.\n[Author] The dimension of C’ is 4 for 256x256 resolution and 3 for 128x128 resolution. We will add explanations in the revision.\n"}, {"review_id": "Dk2FKQ0Ja4", "reviewer": "Reviewer_P49b", "summary": "This paper proposes a new video generation framework based on extracting the global features of the video and conditional diffusion model to predict frame features, leading to the frame. The paper argues the proposed method outperforms prior video generation methods on various benchmarks, including UCF-101, Taichi-HD, and SkyTimelapse. ", "questions": "- I guess the proposed method may show a worse performance if the targeting video length becomes large because the quality of global features might have a limitation and the decoder that synthesizes a frame in frame-index conditioned manner has a limited capacity. What is the (empirical) maximum length for high-quality modeling with this framework? ", "limitations": "The paper adequately addresses the limitations in Conclusion section. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Compared with the prior video generation methods, the proposed method considers non-autoregressive approach for generating the video, which can improve the efficiency in inference time.\n- The proposed method shows better performance compared with prior works. ", "weaknesses": "- The overall framework is quite complex, including so many notations, and a bit difficult to follow. For instance, Why $I_j$ is put into the video decoder model as well as $I_j$ in Figure 2? Moreover, are KL-VAE and video encode/decoder, discriminator jointly trained or not? What is the intuition of letting the video decoder network as a conditional diffusion model instead of letting simple 2D CNNs? Why do we need to consider \"keyframes\" for extracting global features from a given video? Does DiT for modeling global features is trained in post-hoc manner after the training of the entire framework? \n- The paper misses an efficiency comparison with recent latent video diffusion models to improve the efficiency in training and efficiency: e.g., LVDM [He et al., 2023] and PVDM. Compared with these frameworks, what is the advantage and disadvantages of the method?\n- Typo: Specificcally -> Specifically in L197. \n\n---\n[He et al., 2023] Latent Video Diffusion Models for High-Fidelity Long Video Generation   \n[Yu et al., 2023] Video Probabilistic Diffusion Models in Projected Latent Space, CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### R4.1 To facilitate following our framework.\n[Author] To faciliate following, we will release our codes and checkpoints as stated in the footnote of the paper.\n\n#### R4.1.1 Why Ij is put into the video decoder model as well as Ii in Figure 2?\n[Author] In Figure 2, we depict the training procedure of our framework. As specified in L164-166 and L186-188, the video discriminator takes paired frames <xi,xj> as inputs during training. Thus Ij should be inputted into the video decoder as well as Ii to obtain corresponding frames xi and xj.\n\n#### R4.1.2 Are KL-VAE and video encode/decoder, discriminator jointly trained or not? \n[Author] They are not joint trained. As specified in L102-104, KL-VAE is pretrained and fixed, and the video encoder/decoder as well as the video discriminator are trained jointly.\n\n#### R4.1.3 What is the intuition of letting the video decoder network as a conditional diffusion model instead of letting simple 2D CNNs? \n[Author] Our video auto-decoder utilizes much fewer latents to encode an input video compared to previous methods as presented in the Table Q1.3 (in **[common question 1]**). To this end, we have to rely on the powerful generation capability of the conditional diffusion model to help reconstruct local characteristics of video frames under the guidance of video global features. Moreover, training consumption can be reduced by initializing the conditional diffusion model with parameters of successful image diffusion model as specified in L49-50.\n\n#### R4.1.4 Necessity to consider \"keyframes\" for extracting global features from a given video.\n[Author] It’s time- and computation- consuming to process all video frames for extracting global features, especially when we encode long videos such as 128. As depicted in Fig. R4.1.4 **[in the PDF file]**, when the number of input video frames increasing, the maximum training batchsize dramaticly decreases and the required training time boosts. Thus it is necessary to use keyframes rather than all video frames to obtain better training efficiency.\n\n#### R4.1.5 Does DiT is trained in post-hoc manner?\n[Author] Yes, as specified in L191-194, DiT is separately trained to fit the distribution of global features for video generation.\n\n### R4.2 Comparison with LVDM and PVDM.\nWe find that PVDM has not provided generation codes and model checkpoints, and LVDM only released checkpoints and scripts for short-video generation. Thus, we follow the experimental setups in PVDM to compare the effeciency among PVDM, LVDM, and our GLOBER. The results are reported in Table Q1.2 (in [common question 1]). We discuss the advantages and disadvantages of our method as follows:\n\n**Advantages:**\n\n- GLOBER can take advantage of the powerful generative capability of pretrained image diffusion models (e.g. stable diffusion) to synthesize reconstructed video frames, thus requiring a much smaller dimension of latent features to represent videos as demonstrated in Table Q1.3 (in [common question 1]).\n- GLOBER is more flexible than PVDM and LVDM when decoding video frames from video latents. The video decoder in GLOBER can decode arbitrary video frames without length or interval limitations by taking the normalized indexes of target video frames as inputs.\n- GLOBER is more efficient than PVDM and LVDM when training for video generation and synthesizing long videos as demonstrated in Table Q1.2 (in [common question 1]). \n- As reported in Table. R4.2.1, GLOBER obtains better performance than PVDM and LVDM on UCF-101 for 16-frame video generation and on SkyTimelapse for 128-frame video generation. \n\nTable 4.2.1 Quantitative comparison for video generation with the resolution of 256^2. N/M-s for PVDM names N DDIM steps for genration the initial video clip and M DDIM steps for synthesing following video clips. N/M-s for GLOBER means N DDIM steps for the generation of global features and M DDIM steps for decoding video frames.\n|||UCF-101||SkyTimelapse|\n|:-:|:-:|:-:|:-:|:-:|\n|| FVD16 | Total Sampling Steps | FVD128 | Total Sampling Steps |\n|StyleGAN-V | 1431.0 | - | 197.0 | - |\n|LVDM | 372 | - | 185.0 | - |\n|PVDM-S; 100/20-s | 457.4 | 100 | 159.9 | 240 |\n|PVDM-L; 200/200-s | 398.9 | 200 | 137.2 | 1600 |\n|PVDM-L;400/400-s|343.6|400|125.2|3200|\n|GLOBER(ours); 50/50-s|252.7|100|125.5|100|\n|GLOBER(ours);100/100-s|**248.9**|200|**122.4**|200|\n\n**Disadvantages of GLOBER:**\n\n- PVDM and LVDM poses stronger constraints on the correlations of adjacent video frames than GLOBER, and thus obtains better performance than GLOBER on short-video generation on simple domain dataset like SkyTimelapse and long-video generation on multi-motion datasets like UCF-101.\n\n### R4.3 Spelling error.\n[Author] Thanks, we will fix the spelling error in the revision.\n\n### R4.4 Explore the capacity of global features and the video decoder.\n[Author] To explore the empirical maximum length of our method, we first calculate the distribution of video length in the SkyTimelapse and TaiChiHD datasets. As depicted in Fig. R4.4.1(a) **[in the PDF file]**, TaiChiHD contains much more long videos than SkyTimelapse, thus we conduct analysis experiments on the TaiChiHD dataset. We train our video auto-encoder to model L-frame videos with (EXP1) L=16, FPS=16, 1s; (EXP2) L=64, FPS=16, 4s; (EXP3) L=128, FPS=16, 8s; (EXP4) L=256, FPS=16, 16s; (EXP5) L=1024, FPS=32, 32s. Particularly, EXPi loads the last checkpoint of EXPi-1 and is trained for 1500 epochs (15h). As depicted in Fig. R4.4.1(b), the performances of different models are comparable when the video length is no more than 512, while significantly performance drop can be seen when the video length being 1024 due to the increasement of video information. Moreover, we find that frame interpolation (FI) can help our model obtain longer videos with comparable FVD scores. In conclusion, our global features can well-capture a video with 16 seconds, and our video decoder can decode videos with FPS being at least 32.\n"}]

Author rebuttal:
We sincerely thank all reviewers for your valuable reviews and suggestions. We have carefully replied to each question and we are welcome for more discussions!!

### [Common Question 1] More comparison with contemporary methods.
### Efficiency comparison:
[Author] We add comparisons with contemporary methods such as modelscope and LVDM in Table Q1.1, where all settings are the same as that in Table 2. We also compare with the recent method PVDM as reported in Table Q1.2 using a 3090 GPU due to lacking 3090ti GPUs. Our method obtains outstanding efficiency for the following two reasons: 
- 1) We adopt a non-autoregressive strategy to synthesize all video frames with only one sampling while most methods employ either the autoregression or interpolation strategy, which requires multiple sampling to create a long video. 
- 2) We use a latent vector (i.e. global feature) with the least number of elements to represent an input video as demonstrated in Table Q1.3 Notably, many methods (including VIDM, VDM, and VideoFusion) model video generation at the frame level, thus being time-consuming.

Table Q1.1 Comparison of sampling time/memory using different methods for generating multiple video frames with 256x256 resolution, 1 batch size, default diffusion steps, and comparable GPU memory on a v100 GPU. F represents the number of video frames. 
|         | VIDM | VDM| LVDM | Modelscope | VideoFusion | TATS | GLOBER(ours) |
| :-------: |  :-------: |  :-------: |  :-------: |  :-------: |  :-------: |  :-------: |   :-------: | 
| DDIM Steps | 100     | 100    | 50    | 50 | 50 | N/A | 50+50 |
| F=16   | 192s/20G | 125s/11G | 75s/9G | 31s/6G | 22s/7G | **6s**/16G | **6s**/7G |
| F=32   | 375s/20G | 234s/11G | 141s/13G | 48s/8G | 39s/9G | 26s/16G | **11s**/11G |
| F=64   | 771s/20G | 329s/11G | 288s/20G | 82s/12G | 76s/13G | 65s/16G | **21s**/19G |

Table Q1.2 Maximum training batch size and required inference time (in seconds) for different methods to synthesize a 256x256 resolution video. Results with * are taken from PVDM and measured with a single NVIDIA 3090ti 24GB GPU. The rest are evaluated on a single NVIDIA 3090 24GB GPU by us due to lack of 3090ti. LVDM has not released models and scripts for 128-frame video generation. Limited by memory, our GLOBER decodes a 128-frame video by parallelly and no-overlapped decoding every 32 video frames. DiT is the video generator of GLOBER.
|               | Train Batch Size | Inference Time(16-frame) |  Inference Time(128-frame) |
|  :-------: |  :-------: |  :-------: |   :-------: | 
| TATS* | - | 84.8 | 434 |
| VideoGPT* | - | 139 | N/A |
| VDM* | - | 113 | N/A |
| LVDM | - | 98 | N/A|
|PVDM-L*| 2 | 20.4 | 166 |
|GLOBER(ours) | 4 | 21.4 | 145.7 |
|GLOBER(DiT only) | **8** | **3.57** | **3.57** |

Table Q1.3 Model design of different methods and comparison of the number of elements used to represent a T-frame video with HxW resolution. LVDM encodes videos using 3D CNN with the spatial and temporal downsampling rates being 8 and 4 respectively, obtaining latent features with shape (T//4, H//8, W//8, 3). PVDM utilizes 3D-to-2D projection to encode a video into three 2D latent features with shape (T, H//d, 4), (T, W//d, 4) and (H//d, W//d, 4) respectively, with d=8 in default. TATS uses 3D VQGAN to encode videos with the spatial and temporal downsampling rates being 8 and 4 respectively, obtaining discrete video latent feature with shape (T//4, H//8, W//8). Modelscope adopts KL-VAE to encode videos by frame, obtaining video latent feature with shape (T, H//16, W//16, 4). GLOBER encodes video global information into a latent feature of shape (H//16, W//16, 16), which is time-independent and can be flexibly decoded into video frames with arbitrary FPS.
|  | VIDM | VDM | LVDM | PVDM | VideoFusion | TATS | Modelscope | GLOBER (ours) |
|  :--: |  :--: |  :--: |   :--: |  :--: |  :--: |  :--: |   :--: |    :--: | 
| Non-arutoregression | &#10008; | &#10008; | &#10008; | &#10008; | &#10004; | &#10008; | &#10004; | &#10004; |
| Video Encoding         |  &#10008; | &#10008; | &#10004; | &#10004; | &#10008; | &#10004; | &#10004; | &#10004; |
| (16, 256, 256, 3)       | - | - | 12288 | 8192 | - | **4096** | 16384 | **4096** |
| (128, 256, 256, 3)     | - | - | 98304 | 65536 | - | 32768 | 131072 | **4096** |

### Quality Comparison
We add comparisons with contemporary methods on UCF-101 in Table Q1.4 and train our method on the webvid-10M dataset for 5 epochs to compare with other SOTA methods as reported in Table Q1.5. Our GLOBER significantly outperforms other methods on the UCF-101 dataset and obtains comparable performance on the Webvid-10M dataset. It can be attributed to two main reasons:
- 1. By initializing our video decoder with pretrained Stable Diffusion, we inherit its powerful ability to synthesize high-quality video frames. 
- 2. The reduction in the element number of video latents makes it easier for our video generator to fit the distribution of video latents for the video generation task.

Table Q1.4 Quantitative comparison for video generation with the resolution of 256x256 on UCF-101. 
| Method | Zero-shot | FVD |
| :---: | :---: | :---: |
| CogVideo | &#10004; | 701.6 |
| MagicVideo|  &#10004; | 699.0 |
| LVDM | &#10004; | 641.8 |
| ModelScope | &#10004; | 639.9 |
| Video LDM | &#10004; | 550.6 |
| VideoCrafters | &#10004; | 516.2 |
| VideoFactory | &#10004; | 410.0 |
| VideoGPT | &#10008; | 2880.6 |
| MoCoGAN | &#10008; | 2886.8 |
| StyleGAN-V | &#10008; | 1431.0 |
| CogVideo | &#10008; | 626 |
| LVDM | &#10008; | 372 |
| PVDM | &#10008; | 343.6 |
| GLOBER (ours) | &#10008; | **252.7** |

Table Q1.5 Quantitative comparison for video generation with the resolution of 256x256 on Webvid-10M.
| Method | FVD | CLIPSIM |
| :---: | :---: | :---: |
| VideoCrafters | 759.30 | 0.2981 |
| LVDM | 455.53 | 0.2751 |
| ModelScope | 414.11 | 0.3000 |
| VideoFactory | 292.35 | **0.3070** |
| GLOBER (ours) | **234.84** | 0.2816 |

Meta-review: The paper received in-depth reviews, in which they highlighted the overall presentations, extensive experiments, results superior to prior baselines and improved inference time. The latter is particularly interesting as recent methods tend to require a lot of compute to generate content. At the same time they also raised a lot of questions. During the discussion period the authors did a good job in addressing most of the concerns leading to increased scores overall. Reviewer P49b provided additional concerns and remained not convinced about the paper. While their concerns have merit, the AC believes that the manuscript is an interesting piece of work and, hence, recommends acceptance.
Decision: Accept (poster)