SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice, global features typically have a small spatial resolution, and we utilize a transformer-based diffusion model, DiT [35], to generate them. Specifically, each 2D global feature  $v \in R^{H'' \times W'' \times C}$  is flattened to a 1D feature with length  $H'' \times W''$ . Then the diffusion and reverse diffusion processes of  $v_{l}$  are similar to the procedures outlined in Eq. (3) and Eq. (6), except that the only condition is the video description (if exists) in the training and generation processes.
In this section, we first introduce the experimental setups in Sec. 4.1. Following this, in Sec. 4.2 and Sec. 4.3, we compare the quantitative and qualitative performance of our method and prior methods on four challenging benchmarks: Sky Time-lapse [36], TaiChi-HD [37], UCF-101 [38], and Webvid-10M [39]. Finally, Sec. 4.4 presents the results of ablation studies conducted to analyze the necessity of the CRA loss, and Sec. 4.5 explores the influence of global feature shape on the generation performance.
4.1 Experimental Setups
All experiments are implemented using PyTorch [40] and conducted on 8 NVIDIA A100 GPUs, with 16-precision adopted for fast training. During the training of the video auto-encoder, pretrained KL-VAEs [8] were utilized to encode each video frame  $x_{i}$  into a latent feature  $z_{i}$ , with a downsample factor of  $f_{frame} = 8$  for  $256^{2}$  resolution and  $f_{frame} = 4$  for  $128^{2}$  resolution. The latent features were then of resolution  $32^{2}$ . The video encoder subsequently encoded the latent features of video keyframes, extracted at fixed frame indexes of [0, 5, 10, 15] for 16-frame videos, with a downsample
Table 1: Quantitative comparison against prior methods on UCF-101, Sky Time-lapse and TaiChi-HD datasets for video generation.  $c$  denotes the conditon of video descriptions.
(a) Sky Time-lapse w/o c
Methods	FVD↓
Resolution 2562
MoCoGAN-HD [41]	164.1
VideoGPT [42]	222.7
DIGAN [15]	83.1
StyleGAN-V [16]	79.5
GLOBER (ours)	78.1
(b) TaiChi-HD w/o c
Methods	FVD
Resolution 1282
SytleGAN-V [16]	143.5
DIGAN [15]	128.1
GLOBER (ours)	124.2
(c) Webvid-10M w/c
Methods	FVD	CLIP
Resolution	2562
VideoCrafters2	759.3	0.2981
LVDM [24]	455.5	0.2751
ModelScope1	414.1	0.3000
VideoFactory [43]	292.4	0.3070
GLOBER (ours)	234.8	0.2816
(d) UCF-101 w/o c
Methods	FVD
Resolution 1282
DIGAN [15]	577
TATS [13]	420
VIDM [22]	263
GLOBER (ours)	239.5
(e) UCF-101 w/c
Methods	FVD
Resolution 1282
TGANv2 [44]	1209
DIGAN [15]	465
TATS [13]	332
MMVG [45]	328
CogVideo [9]	305
VIDM [22]	263
VideoFusion [19]	173
GLOBER (ours)	151.5
Resolution 2562
DIGAN [15]	471.9
Make-A-Video [46]	367.2
GLOBER (ours)	168.9
Table 2: Quantitative comparison against prior methods on UCF-101 for unconditional video generation with  $256^{2}$  resolution.
Method	CogVideo [9]	MagicVideo [47]	LVDM [24]	ModelScope1	VideoLDM [7]
Zero-Shot FVD	✓ 701.6	✓ 699.0	✓ 641.8	✓ 639.9	✓ 550.6
Method	VideoCrafters2	VideoFactory [43]	VideoGPT [42]	MoCoGAN [17]	StyleGAN-V [16]
Zero-Shot FVD	✓ 516.2	✓ 410.0	X 2880.6	X 2886.8	X 1431.0
Method	CogVideo [9]	LVDM [24]	PVDM [23]	GLOBER (ours)
Zero-Shot FVD	X 626	X 372	X 343.6	X 252.7
factor of  $f_{video} = 2$  and a number of output channels of  $C'' = 16$ , resulting in global features of shape  $16 \times 16 \times 16$ . The dimension of  $C'$  is 4 for 256x256 resolution and 3 for 128x128 resolution. We first train the video auto-encoder as well as the discriminator jointly until convergence, and then train DiT with the parameters of the video auto-encoder being fixed. The video auto-encoder was trained with a batch size of 40 per GPU for 80K, 40K, and 40K steps on the UCF101, TaiChi-HD, and Sky Time-lapse datasets, respectively. The loss weight  $\lambda_1$  and  $\lambda_2$  are set as 1e-6 and 0.1, respectively. When training the video generator, a Transformer-based diffusion model, DiT [35], was used as the backbone. The batch size was set to 32 per GPU, and the number of training iterations was 200K, 100K, and 100K for the UCF101, TaiChi-HD, and Sky Time-lapse datasets, respectively. When generating videos, we sample each global feature with the number of DDIM steps being 50 and the unconditional guidance scale being 9.0 expect otherwise specified. Then the video decoder synthesizes target video frames parallelly with the number of DDIM steps being 50 and the unconditional guidance scale being 3.0 for the Sky Time-lapse and TaiChi-HD datasets and 6.0 for the UCF101 dataset.
4.2 Quantitative Comparison
Generation Quality Table 1 and Table 2 reports the results of our model trained on the Sky Time-lapse, TaiChi-HD, UCF-101, and Webvid-10M datasets for 16-frame video generation in both unconditional and conditional settings. As shown in Table 1(a), Table 1(b), and Table 1(c), our method achieves comparable performance with prior state-of-the-art models on the Sky Time-lapse, TaiChi-HD, and Webvid-10M datasets, respectively. To comprehensively compare the performance
<https://github.com/modelscope/modelscope>
$^{2}$ https://github.com/AILab-CVC/VideoCrafter
Table 3: Comparison of sampling time/memory using different methods for generating multiple video frames with resolution of  $256^{2}$ , batch size of 1, diffusion steps of default settings, and comparable GPU memory on a v100 GPU.  $F$  represents the number of video frames. AR denotes autoregression, IP denotes interpolation and NAR denotes non-autoregression.
Method	VIDM [22]	VDM [4]	LVDM [24]	PVDM [23]	VideoFusion [19]	TATS [13]	ModelScope	GLOBER (ours)
NAR	✘	✘	✘	✘	✓	✘	✓	✓
Video Encoding	✘	✘	✓	✓	✘	✓	✓	✓
NUMF=16	-	-	12288	8192	-	4096	16384	4096
NUMF=128	-	-	98304	65536	-	32768	131072	4096
Diffusion Steps	100	100	50	-	50	-	50	50+50
F=16	192s/20G	125s/11G	75s/9G	-	22s/7G	6s/16G	31s/6G	6s/7G
F=32	375s/20G	234s/11G	141s/13G	-	39s/9G	26s/16G	48s/8G	11s/11G
F=64	771s/20G	329s/11G	288s/20G	-	76s/13G	65s/16G	82s/12G	21s/19G
on the more challenging UCF-101 dataset, we conduct experiments on both unconditional and conditional video generation at resolutions of  $128^2$  and  $256^2$ . For unconditional video generation, we use the fixed textual prompt "A video of a person" as the video description, while for conditional video generation, we use the class name of each video as the video description. Our proposed method significantly outperforms previous state-of-the-art models, as shown in Table 1(d), Table 1(e), and Table 2. The superior performance of our method can be attributed to two factors. Firstly, we explicitly separate the generation of global guidance from the synthesis of frame-wise local characteristics. As global features are 2D features with relatively small spatial resolution, our generative model can model them effectively and efficiently without imposing a high computational burden. Secondly, the synthesis of frame details can refer to the global feature, which provides global guidance such as scene layout, object appearance, and overall behaviours, making it easier for our method to achieve global-coherent and local-realistic video generation.
Generation Speed We quantitatively compare the generation speed among various video generation methods and report the results in Table 3. Our GLOBER method demonstrates remarkable efficiency in generating video frames, thanks to its utilization of the non-autoregression strategy. In contrast, prior methods such as VIDM and VDM, which follow the auto-regression strategy, maintain unchanged memory but require significantly more time for generation. VideoFusion also adopts a non-autoregressive strategy for generating multiple video frames. However, VideoFusion remains slower than our GLOBER, since VideoFusion directly models frame pixels, which brings huge computational burden when generating multiple video frames. TATS employs the interpolation strategy by first generating video keyframes and then interpolating 3 frames between adjacent keyframes. However, it involves autoregressive generation of video keyframes and the interpolation process is not parallelly implemented, thus being slower than our GLOBER. Notably, GPU memory of VideoFusion and our GLOBER increases with the number of video frames due to the parallel synthesis of all video frames.
4.3 Qualitative Comparison
As depicted in Figure 3, we conduct a qualitative comparison of our method with previous approaches on the UCF-101, Sky Time-lapse, and TaiChi-HD datasets. Samples of prior methods are obtained from [22; 19]. The UCF-101 dataset records 101 human actions and is the most challenging and diverse dataset. When applied to this dataset, GAN-based methods such as DIGAN and StyleGAN-V generate video samples that lack distinctiveness. In contrast, TATS, which is built on Transformer and utilizes the interpolation strategy, generates video samples that are more identifiable. In comparison to TATS, diffusion-based methods like VideoFusion [19] and VIDM [22] produce samples with more pronounced appearances. However, VideoFusion generates slightly blurred object appearances, and VIDM generates overly trivial object motions. Conversely, our GLOBER generates samples that exhibit both distinct appearances and conspicuous movements. On the Sky Time-lapse dataset, samples generated by DIGAN, StyleGAN-V, and TATS display trivial motions and simplistic objects. VideoFusion and VIDM generate samples with enhanced details and more discernible boundaries, while their motion remains somewhat negligible. In contrast, our GLOBER generates video samples with more dynamic movements and significantly richer visual details. Similarly, on the TaiChi-HD dataset, the human appearances generated by DIGAN, StyleGAN-V, and TATS are noticeably
Figure 3: Quantitative comparison against previous methods on the UCF-101 (left), Sky Time-lapse (middle) and TaiChi-HD (right) datasets. Our showcased samples on the UCF-101 datasets are produced using the video descriptions "Apply Lipstick", "Soccer Juggling", and "Pull Ups", respectively. Samples on the Sky Time-lapse and TaiChi-HD datasets are generated using fixed video descriptions being "A time-lapse video of sky" and "Tai chi", respectively.
Figure 4: Visualization of samples synthesized with or without the CRA loss by the video auto-encoder.
Table 4: Ablation study on the CRA loss.
Datasets	CRA loss	rFVD
UCF-101	w/o	106.7
w/	69.7
Sky Time-lapse	w/o	84.3
w/	63.5
TaiChi-HD	w/o	91.3
w/	60.1
distorted. While VideoFusion and VIDM achieve improved human appearances, their movements remain trivial. In contrast, samples generated by our GLOBER exhibit significant movements and distinct object appearances.
4.4 Ablation Study on the CRA Loss
To investigate the effectiveness of our proposed CRA loss, we conducted an ablation study where the CRA loss was removed during the training of the video auto-encoder. The quantitative results are presented in Table 4, which unequivocally demonstrate the effectiveness of our CRA loss on all three benchmarks: UCF-101, Sky Time-lapse, and TaiChi-HD, with a remarkable reduction in FVD scores by 37.0, 20.8, and 31.2, respectively. In addition, we qualitatively compare videos synthesized by models with or without the CRA loss in Fig. 4, which clearly shows that videos generated using the CRA loss exhibit better video consistency and more distinct local details, demonstrating the necessity of our proposed CRA loss. The effectiveness of our CRA loss can be attributed to two key factors. Firstly, by penalizing samples that violate temporal relationships, the CRA loss enhances the overall coherence of the synthesized video frames. Secondly, by utilizing pairwise video frames as inputs for
the discriminator, the CRA loss can identify and penalize video frames that exhibit inconsistent or distorted local characteristics, which further enhances the realism of the synthesized videos.
4.5 Impact on the Global Feature Shapes
We experiment the effect of different shapes of the global features on the quality of synthesized video frames. As reported in Table 5, videos generated using global features with a shape of  $8 \times 8 \times 64$  exhibit lower quality than those generated using global features with a shape of  $16 \times 16 \times 16$ . And the use of global features with a shape of  $16 \times 16 \times 32$  brings a further improvement of 39.2 in FVD score. It may be
Table 5: Sensitivity analysis on the shape of global features on the UCF-101 datasets.
H'' × W'' × C''	FVDrec ↓	FVDgen ↓
8 × 8 × 64	276.1	725.1
16 × 16 × 16	189.7	560.4
16 × 16 × 32	184.9	521.2
due to two reasons. Firstly, our experiments are conducted on the UCF-101 dataset, which contains complex motions that require more channels to be effectively represented. Secondly, by utilizing the KL loss to constrain the distribution of global features toward an isotropic Gaussian, it becomes easier for the generative model to fit the distribution of the global features even with a number of channels, thus increasing the number of channels could bring improved performance.
This paper introduces GLOBER, a novel diffusion-based video generation method that emphasizes the significance of global guidance in multi-frame video generation. Our method offers three distinct advantages. Firstly, it alleviates the computation burden of modeling video generation by replacing 3D video signals with 2D global features. Secondly, it utilizes a non-autoregressive generation strategy, enabling the efficient synthesis of multiple video frames and surpassing the performance of prior methods in terms of efficiency. Lastly, by incorporating global features as guidance, our method generates videos with enhanced coherence and realism, achieving new state-of-the-art results on multiple benchmarks. Nevertheless, our research exhibits several limitations. Firstly, we find GLOBER difficult to process videos with frequent scene changes, as such transitions have the potential to disrupt the video coherence. Furthermore, we have not explored the performance of GLOBER in open-domain video generation tasks due to computational resource constraints. Future works are encouraged to solve the above issues.
This work was supported by the National Key Research and Development Program of China (No.2022ZD0118801), National Natural Science Foundation of China (U21B2043,62102416, 62206279, 62102419).
[1] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans et al., "Photorealistic text-to-image diffusion models with deep language understanding," Advances in Neural Information Processing Systems, vol. 35, pp. 36479-36494, 2022.
[2] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” arXiv preprint arXiv:2204.06125, 2022.
[3] L. Zhang and M. Agrawala, "Adding conditional control to text-to-image diffusion models," arXiv preprint arXiv:2302.05543, 2023.
[4] J. Ho, T. Salimans, A. A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," in Advances in Neural Information Processing Systems, 2022.
[5] V. Voleti, A. Jolicoeur-Martineau, and C. Pal, "Masked conditional video diffusion for prediction, generation, and interpolation," Advances in Neural Information Processing Systems, 2022.
[6] S. Yin, C. Wu, H. Yang, J. Wang, X. Wang, M. Ni, Z. Yang, L. Li, S. Liu, F. Yang et al., "Nuwa-xl: Diffusion over diffusion for extremely long video generation," arXiv preprint arXiv:2303.12346, 2023.
The goal of this work is to advance research on video generation methods. Our method has the potential to facilitate the workflow of film production and animation, exhibiting a positive influence on creative video applications. Since our method is trained mainly on domain-specific datasets, the potential deleterious consequences of exploiting our model for malicious purposes, such as spreading misinformation or producing fake videos, seem to be insignificant. Nevertheless, it remains crucial to apply an abundance of caution and implement strict and secure regulations.
(a) Sky Time-lapse
(b) UCF-101
Figure 5: Genetated long videos with 128 frames on the Sky Time-lapse and UCF-101 datasets (4 frames skipped).
C Experimental Results on Long Video Generation Tasks
We obtain new state-of-the-art results on the SkyTimelapse and UCF-101 datasets for long video generation tasks. All experiments are conducted without conditional inputs. The quantitative results
are reported in Table 6. MoCoGAN, MoCoGAN-HD, DIGAN, and StyleGAN-V are GAN-based methods, which dominate the field of vision generation until 2022. Based on diffusion probabilistic models, VIDM outperforms these GAN-based methods by a large margin. However, VIDM employs the auto-regression generation strategy to generate long videos, which lacks global guidance and suffers from error accumulation. Our method, GLOBER, outperforms VIDM significantly due to its incorporation of global features and non-autoregression generation strategy. We present several video samples in Fig. 5, which demonstrate that our method can generate long videos of remarkable quality.
Table 6: Quantitative Results of FVD comparison on the SkyTimelapse and UCF-101 datasets for 128-frame long video generation.
Method	UCF-101	Sky Time-lapse
MoCoGAN [CVPR18]	3679.0	575.9
+StyleGAN2 backbone	2311.3	272.8
MoCoGAN-HD [ICLR21]	2606.5	878.1
DIGAN [ICLR22]	2293.7	196.7
StyleGAN-V [CVPR22]	1773.4	197.0
VIDM [AAAI23]	1531.9	140.9
GLOBER (ours)	1177.4	125.5
D More Qualitative Results
We present more qualitative results on the UCF-101, Sky Time-lapse, and TaiChi-HD datasets in the link: https://iva-mzsun.github.io/GLOBER.
E Sensitivity Analysis of Unconditional Guidance Scale
We investigate the effectiveness of the unconditional guidance scale  $\mu$  that is used when employing class-condition constraints. Table 7 presents the influence of  $\mu$  on the FVD score of videos conditionally decoded by the video decoder on the UCF-101  $256^{2}$  benchmark. Table 8 presents the influence of  $\mu$  on the FVD score of videos conditionally sampled by the video generator on the UCF-101  $256^{2}$  dataset. It is evident that appropriate selection of the unconditional guidance scale is important in ensuring the quality of videos decoded or sampled with class conditions.
Table 7: Sensitivity analysis of the unconditional guidance scale  $\mu$  for video reconstruction on the UCF-101 dataset.
μ	0	3	6	9	12	15
FVD	211.4	114.3	106.7	133.2	281.1	670.8
Table 8: Sensitivity analysis of the unconditional guidance scale  $\mu$  for video generation on the UCF-101 dataset.
μ	0	3	6	9	12	15
FVD	575.6	173.0	172.6	171.5	168.9	224.3
F Quality of frame features generated by Equ. (9)
When adopting the CRA loss, we directly obtain the synthesized frame feature  $\bar{z}_i$  according to Equ. (9). Considering that diffusion models typically perform T denoising steps to synthesize a realistic sample, we explore the rationality of this design choice. As depicted in Fig. 6, the quality deterioration in estimated video frames is acceptable for most samples. It is reasonable since the diffusion process is under the guidance of global features, which contain sufficient local and global information.
Figure 6: Video frames estimated by Equ. (9) with different timesteps on the UCF-101 dataset with  $256 \times 256$  resolution.
G Necessity to consider "keyframes" for extracting global features from a given video
It's time- and computation- consuming to process all video frames for extracting global features, especially when we encode long videos such as 128. As depicted in Fig. 7, when the number of input video frames increasing, the maximum training batchsize dramatically decreases and the required training time boosts. Thus it is necessary to use keyframes rather than all video frames to obtain better training efficiency.
Figure 7: Comparison of the maximum training batch size and required time using different number of input video frames.
H More Efficiency Comparison
We compare the generation efficiency of our GLOBER with prior works, e.g. PVDM, with comparable settings as reported in Table. 9. Results with  $*$  are taken from PVDM and measured with a single NVIDIA 3090ti 24GB GPU. The rest are evaluated on a single NVIDIA 3090 24GB GPU by us due to lack of 3090ti. LVDM has not released models and scripts for 128-frame video generation. Limited by memory, our GLOBER decodes a 128-frame video by parallelly and no-overlapped
decoding every 32 video frames. DiT is the video generator of GLOBER. The results demonstrate that our GLOBER is more efficient than PVDM and LVDM when training for video generation and synthesizing long videos.
Table 9: Maximum training batch size and required inference time (in seconds) for different methods to synthesize a 256x256 resolution video.
Method	Train Batch Size	Inference Time (16 frames)	Interference Time (128 frames)
TATS*	-	84.8	434
VideoGPT*	-	139	N/A
VDM*	-	113	N/A
LVDM	-	98	N/A
PVDM-L*	2	20.4	166
GLOBER (ours)	4	21.4	145.7
GLOBER (DiT only)	8	3.57	3.57
I Ablation Study on the KL loss
Following [8], the KL loss, i.e. Equ. (2), can punish the distribution of latent features towards a standard normal distribution to avoid high shift and high variance. We conduct an ablation study on the variance prediction and the KL loss. As reported in Table 10, removing variance prediction brings improvements on rFVD, but deteriorates FVD significantly since the video decoder is no longer robust to disturbance of generated features. Adding variance prediction improves the generation performance to some extent. The performance further boosts after employing the KL loss since the KL loss brings negligible decrease on rFVD but can effectively facilitate DiT to model the distribution of global features using the diffusion theory.
Table 10: Ablation study on variance prediction and the KL loss. All experiments are conducted on the TaiChiHD dataset for 16-frame video generation with  $256^2$  resolution. Autoencoders are trained for 1500 epochs (15h) and DiTs are trained for 2000 epochs (14h).
Design of the video auto-encoder	rFVD	FVD
Deterministic (w/o variance)	68.9	773.4
+ variance prediction (w/o KL loss)	71.5	549.5
+ variance prediction + KL loss	75.3	332.7
J Settings of Hyper Parameters
The detailed settings of model hyper parameters are presented in Table 11.
Table 11: Hyper-parameters of the video auto-encoder and the quantitative results on video reconstruction. Experimental settings on the UCF-101 dataset are the same for both conditional and unconditional video generation except given video descriptions.
UCF-101 2562	Sky Time-lapse 2562	TaiChi-HD 1282
Batch Size	40	32	32
Learning Rate	1e-5	1e-4	5e-5
KL-VAE
fframe	8	8	4
Video Encoder
fvideo	2
Input Shape	32
Input Channels	4
Output Channels	16
Model Channels	320
Num Res. Blocks	2
Num Head Channels	64
Attention Resolutions	[16, 8]
Channel Multiplies	[1, 2]
Video Decoder (UNet)
Input Shape	32
Input Channels	4
Output Channels	4
Model Channels	320
Num Res. Blocks	2
Num Head	8
Attention Resolutions	[32, 16, 8]
Channel Multiplies	[1, 2, 4, 4]
Video Generator (DiT)
Input Shape	16	16	16
Input Channels	16	16	16
Model Channels	1152	1024	1024
Num Head	16	16	16
Depth	28	20	20
Mlp Ratio	4	4	4