SlowGuess/NeurIPS_2023_OCR huggingface dataset:
GLOBER: Coherent Non-autoregressive Video Generation via GLOBAL Guided Video Decoder
Mingzhen Sun 1,2
Jiahui Sun 1,2
Weining Wang
Sihan Chen 1,2
Zihan Qin 1,2
Jing Liu 1,2,\*
$^{1}$ Institute of Automation, Chinese Academy of Sciences (CASIA)
$^{2}$ School of Artificial Intelligence, University of Chinese Academy of Sciences (UCAS)
sunmingzhen2020@ia.ac.cn weining.wang@nlpr.ia.ac.cn qinzihan2021@ia.ac.cn
sunjiahui19@mails.ucas.ac.cn sihan.chen@nlpr.ia.ac.cn jliu@nlpr.ia.ac.cn
Video generation necessitates both global coherence and local realism. This work presents a novel non-autoregressive method GLOBER, which first generates global features to obtain comprehensive global guidance and then synthesizes video frames based on the global features to generate coherent videos. Specifically, we propose a video auto-encoder, where a video encoder encodes videos into global features, and a video decoder, built on a diffusion model, decodes the global features and synthesizes video frames in a non-autoregressive manner. To achieve maximum flexibility, our video decoder perceives temporal information through normalized frame indexes, which enables it to synthesize arbitrary sub video clips with predetermined starting and ending frame indexes. Moreover, a novel adversarial loss is introduced to improve the global coherence and local realism between the synthesized video frames. Finally, we employ a diffusion-based video generator to fit the global features outputted by the video encoder for video generation. Extensive experimental results demonstrate the effectiveness and efficiency of our proposed method<sup>1</sup>, and new state-of-the-art results have been achieved on multiple benchmarks.
When producing a real video, it is customary to establish the overall information (global guidance), such as scene layout or character actions and appearances, before filming the details (local characteristics) that draw up each video frame. The global guidance ensures a coherent storyline throughout the produced video, and the local characteristics provide the necessary details for each video frame. Similar to real video production, generative models for the video generation task must synthesize videos with coherent global storylines and realistic local characteristics. However, due to limited computational resources and the potential infinite number of video frames, how to achieve global coherence while maintaining local realism remains a significant challenge for video generation tasks.
Inspired by the remarkable performance of diffusion probabilistic models [1; 2; 3], researchers have developed a variety of diffusion-based methods for video generation. When generating multiple video frames, strategies of existing methods can be devised into two categories: autoregression and interpolation strategies. As illustrated in Fig. 1(a), the autoregression strategy [4; 5] first generates an initial video clip, and then employs the last few generated frames as conditions to synthesize subsequent video frames. The interpolation strategy [6; 7], depicted in in Fig. 1(b), generates keyframes first and then interpolates adjacent keyframes iteratively. Both strategies utilize generated video frames as conditions to guide the generation of subsequent video frames, enabling subsequent
<sup>1</sup>Our codes have been released in https://github.com/iva-mzsun/GLOBER
Figure 1: Three strategies for multi-frame video generation. (a) Autoregression strategy first generates the starting video frames and then autoregressively predicts subsequent video frames. This approach is prone to accumulating errors [6], such as the fading of the hula hoop. (b) Interpolation strategy generates video keyframes first and then iteratively interpolates adjacent keyframes. This approach can result in suboptimal video consistency since it is unaware of the global content. (c) Our proposed non-autoregression strategy first generates a global feature to provide global guidance, and then synthesizes local characteristics of video frames in a non-autoregressive manner.  $c$  names the condition input of video description.  $x_{i}$  and  $I_{i}$  denote the  $i$ -th video frame and its index.
global storylines to be predicted from the context of the given frames. However, since the number of conditional video frames is limited by available computational resources and is generally small, these strategies have a relatively poor capacity to provide global guidance, resulting in suboptimal video consistency. In addition, the generation of local characteristics of subsequent frames refers to previous single frames, which can lead to error accumulation and distorted local characteristics [6].
In this paper, we present a novel non-autoregression method GLOBER, which first generates 2D global features to serve as global guidance, and then synthesizes local characteristics of video frames based on global features to obtain coherent video generation, as illustrated in Fig. 1(c). To be specific, we propose a video auto-encoder that involves a video encoder and a diffusion-based powerful video decoder. The video encoder encodes each input video into a 2D global feature. The video decoder decodes the storyline from the generated global feature and synthesizes necessary local characteristics for video frames. To achieve maximum flexibility, our video decoder is designed to involve no temporal modules and thus decode multiple video frames in a non-autoregressive manner. In particular, normalized frame indexes are integrated with generated global features to provide temporal information for the video decoder. In this way, we can obtain arbitrary sub video clips with predetermined starting and ending frame indexes. To leverage the success of image generation models, we initialize the video decoder with a pretrained image diffusion model [8]. Then the video auto-encoder can be trained in a self-supervised manner with the target of video reconstruction. Furthermore, we propose a Coherence and Realism Adversarial (CRA) loss to improve the global coherence and local realism in the decoded video frames. For video generation, another diffusion model is used as the video generator to generate global features by fitting the output of video encoder.
Our contributions are summarized as follows: (1) We propose a novel non-autoregression strategy for video generation, which provides global guidance while ensuring realistic local characteristics. (2) We introduce a powerful video decoder that allows for parallel and flexible synthesis of video frames. (3) We propose a novel CRA loss to improve global coherence and local realism. (4) Experiments show that our proposed method obtains new state-of-the-art results on multiple benchmarks.
Current video generation models can be divided into three categories: transformer-based methods that model discretized video signals, generative adversarial networks (GAN) and diffusion probabilistic models (DPM). The first category encodes videos to discrete video tokens and then models these tokens with transformers [9; 10; 11; 12; 13; 14]. MOSO [12] decomposes video scenes, objects,
and their motions to cover video prediction, generation, and interpolation tasks in an autoregressive manner. TATS [13] follows the interpolation strategy, and introduces a time-agnostic VQGAN and a time-sensitive hierarchical transformer to capture long temporal dependencies
GAN-base methods [15; 16; 17] excel at generating videos in specific domains. MoCoGAN [17] proposes to decompose video content and motion by dividing the latent space. DIGAN [15] explores video encoding with implicit neural representations. StyleGAN-V [16] extends the image generation model StyleGAN [18] to the video generation task non-autoregressively. However, StyleGAN-V focuses on the division of content and motion noises and ignores the importance of global guidance. As a result, its content input (i.e., randomly sampled noise) contains limited information, whereas our global features can provide more valuable instructions.
DPM is a recently emerging method for vision generation [4; 19; 5; 20; 21]. VDM [4] is the first work that applies DPM to video generation by replacing all 2D convolutions with 3D convolutions. VIDM [22] generates videos in a frame-wise autoregression manner with two individual diffusion models. Both VDM and VIDM employ the autoregression strategy to obtain multiple video frames. In contrast, NUWA-XL [6] adopts the interpolation strategy but requires significant computational resources to support parallel inference. VideoFusion [19] targets at dividing the shared and residual video signals of different video frames and modeling them respectively. It employs a non-temporal diffusion model to synthesize video frames based on their indexes within the pixel space. PVDM [23] utilizes 3D-to-2D projection to encode a video into three 2D latent features for efficient video generation. In PVDM, the video decoder reconstructs video frames with fixed length (i.e. 16 frames) and fixed interval (i.e. predefined FPS). LVDM [24] encodes videos using 3D CNN through both spatial and temporal downsampling. Compared with previous methods, our method encompasses the following advantages. Firstly, GLOBER can take advantage of the powerful generative capability of pretrained image diffusion models (e.g. stable diffusion) to synthesize reconstructed video frames, thus requiring a much smaller dimension of latent features to represent videos. Secondly, GLOBER is more flexible than prior works when decoding video frames from video latents. The video decoder in GLOBER can decode arbitrary video frames without length or interval limitations by taking the normalized indexes of target video frames as inputs. Thirdly, GLOBER is more efficient when training for video generation and synthesizing long videos. Fourthly, GLOBER obtains new state-of-the-art results on multiple benchmarks.
In this section, we present our proposed method in details. The overall framework of our method is depicted in Fig. 2. A video sample is represented as  $x \in R^{F \times H \times W \times C}$ , where  $F$  denotes the number of video frames,  $H$  is the height,  $W$  is the width, and  $C$  is the number of channels. The  $i$ -th video frame is denoted as  $x_{i} \in R^{H \times W \times C}$ .
3.1 Video Auto-Encoder
The video auto-encoder comprises a video encoder and a video decoder. To reduce the computational complexity involved in modeling videos, an auxiliary image auto-encoder, which is known as KL-VAE [8] and has been validated in previous studies [8; 6], is employed to encode each video frame individually into a low-resolution feature. The video encoder takes video keyframes as input and encodes them into 2D global features. Then the video decoder is used to synthesize each video frame based on the corresponding frame index and the global feature.
Frame Encoding VAEs [25; 26; 27; 8] are widely used models that reduce search space for vision generation. For high-resolution image generation, VAEs typically encode each image into a latent feature and then decode it back to the original input image [10; 27; 28; 29]. To reduce the spatial details of videos in a similar way, we employ a pretrained KL-VAE [8] to encode each video frame individually. Specifically, the KL-VAE downsamples each video frame  $x_{i}$  by a factor of  $f_{frame}$ , obtaining a frame latent feature  $z_{i} \in R^{H' \times W' \times C'}$ , where  $H'$  and  $W'$  are  $\frac{H}{f_{frame}}$  and  $\frac{W}{f_{frame}}$ , respectively, and  $C'$  represents the number of feature channels.
Video Encoding As illustrated in Fig. 2, the video encoder is composed of an embedding feature  $e_v$ , an input layer, a downsample module with a downsample factor of  $f_{video}$ , a mapping module, and
Figure 2: The overall framework of our proposed method GLOBER. During training, the video encoder and decoder are optimized jointly, with the video encoder encoding videos into global features, which are used by the video decoder to synthesize two randomly sampled video frames based on their corresponding frame indexes. We do not draw up the processing of timesteps and video descriptions in the video decoder for conciseness. The synthesized video frames are evaluated by the video discriminator for global coherence and local realism. Then the video generator is trained to generate global features by fitting the outputs of the video encoder. During generation, the video generator generates a novel global feature, which is then decoded by the video decoder to synthesize video frames in a non-autoregressive manner.
an output module. The embedding feature  $e_v \in R^{H' \times W' \times C'}$  is randomly initialized and optimized jointly with the entire model. It should be noted that the temporal dimensions of the embedding feature and each frame feature are 1, which are omitted here for brevity. Since content redundancy exists between adjacent video frames [12], we select  $K$  keyframes from each input video at equal intervals and encode them individually using KL-VAE. This process produces the corresponding frame features  $z_{q_1}, z_{q_2}, \ldots, z_{q_K}$ , where  $q_k$  denotes the index of the  $k$ -th selected keyframe. Then, we concatenate the embedding feature  $e_v$  with keyframe features along the temporal dimension, obtaining the input feature  $[e_v : z_{q_1} : \ldots : z_{q_K}] \in R^{(K+1) \times H' \times W' \times C'}$  for the video encoder.
The input layer employs a simple 2D spatial convolution with a kernel size of 3 to expand the available channel maps from  $C'$  to  $D$ , where  $D$  represents the number of new channel maps. After the input layer, the downsample module processes these keyframe features to capture spatial and temporal information, while the mapping module follows. Specifically, the downsample module is composed of a residual layer, a spatial attention layer, a temporal attention layer, and a downsample convolution, while the mapping module follows a similar structure but replaces the downsample convolution with a temporal split operation. Finally, the output module comprises two spatial convolutions, a group normalization layer, and a SiLU activation function. It takes only the embedding part of the transformed feature as input and produces the mean and standard deviation (std) features of the global feature. Then the global feature can be sampled using the following equation:
$$
v = v _ {\text {m e a n}} + v _ {\text {s t d}} * n \tag {1}
$$
where  $n$  is sampled from an isotropic Gauss distribution  $\mathcal{N}(0, \mathbf{I})$ ,  $v_{mean}$ ,  $v_{std} \in R^{H'' \times W'' \times C''}$  are the mean and std features,  $H''$  and  $W''$  are  $\frac{H'}{f_{video}}$  and  $\frac{W'}{f_{video}}$  respectively, and  $C''$  is the number of channels. Considering that we are going to model the global features for generation using a
diffusion-based model, which will be specified in Sec. 3.2, an additional kl loss [8] is employed to force the distribution of global features towards an isotropic Gauss distribution:
$$
\mathcal {L} _ {k l} = \frac {1}{2} \left(v _ {\text {m e a n}} ^ {2} + v _ {\text {s t d}} ^ {2} - \log v _ {\text {s t d}} ^ {2} - 1\right) \tag {2}
$$
Video Decoding We view the synthesis of video frames as a conditional diffusion-based generation task. As illustrated in Fig. 2, we employ UNet [8] as the backbone of the video decoder, which can be structured into the downsample, mapping, and upsample modules. Each module starts with a residual block and a spatial attention layer, while the downsample and upsample modules additionally include downsample and upsample convolutions respectively. Following [30], each frame feature  $z_{i}^{0}$  (i.e.  $z_{i}$ ) is corrupted by  $T$  steps during the forward diffusion process using the transition kernel:
$$
q \left(z _ {i} ^ {t} \mid z _ {i} ^ {t - 1}\right) = \mathcal {N} \left(z _ {i} ^ {t}; \sqrt {1 - \beta_ {t}} z _ {i} ^ {t - 1}, \beta_ {t} \mathbf {I}\right) \tag {3}
$$
$$
q \left(z _ {i} ^ {t} \mid z _ {i} ^ {0}\right) = \mathcal {N} \left(z _ {i} ^ {t}; \sqrt {\bar {\alpha} _ {t}} z _ {i} ^ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right) \tag {4}
$$
where  $\{\beta_t\in (0,1)\}_{t = 1}^T$  is a set of hyper-parameters,  $\alpha_{t} = 1 - \beta_{t}$  and  $\bar{\alpha}_t = \prod_{i = 1}^t\alpha_i$ . Based on Eq. (4), we can obtain the corrupted feature  $z_{i}^{t}$  directly given the timestep  $t$  as follows:
$$
z _ {i} ^ {t} = \sqrt {\bar {\alpha} _ {t}} z _ {i} ^ {0} + (1 - \bar {\alpha} _ {t}) n _ {t} \tag {5}
$$
where  $n_t$  is a noise feature sampled from an isotropic Gauss distribution  $\mathcal{N}(0, \mathbf{I})$ . The reverse diffusion process  $q(z_i^{t-1} | z_i^t, z_i^0)$  has a traceable distribution:
$$
q \left(z _ {i} ^ {t - 1} \mid z _ {i} ^ {t}, z _ {i} ^ {0}\right) = \mathcal {N} \left(z _ {i} ^ {t - 1} \mid \tilde {\mu} _ {t} \left(z _ {i} ^ {t}, z _ {i} ^ {0}\right), \tilde {\beta} _ {t} \mathbf {I}\right) \tag {6}
$$
where  $\tilde{\mu}_t(z_i^t,z_i^0) = \frac{1}{\sqrt{\alpha_t}} (z_i^t -\frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} n_t),n_t\sim \mathcal{N}(0,\mathbf{I}),$  and  $\tilde{\beta}_{t} = \frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_{t}}\beta_{t}$
Given that the added noise  $n_t$  in the  $t$ -th step is the only unknown term in Eq. (6), we train the UNet to predict  $n_t$  from  $z_i^t$  condition on the timestep  $t$ , the global feature  $v$ , the frame index  $i$ , and the video description  $c$  if exists. To achieve this, a timestep and an index embedding layers, additional modules, and a text encoder are employed to process these condition inputs. In particular, the timestep embedding layer first obtains the sinusoidal position encoding [31] of the diffusion timestep  $t$  and then passes it through two linear functions with a SiLU activation interposed between them. Following previous works [30; 32], the timestep embedding is integrated with intermediate features by the residual block in each UNet module. The index embedding layer embeds the frame index in a similar way with the timestep embedding layer. The additional modules (i.e. downsample, mapping, and upsample modules) are utilized to incorporate the index embedding with the global feature and to extract multi-scale representations of the corresponding video frame. These representations are then added to the outputs of UNet modules after zero-convolution. When encoding video descriptions into text embeddings, a pretrained model, namely CLIP [33], is utilized as the text encoder. During attention calculation, these text embeddings are concatenated with flattened frame features to provide cross-modal instructions. Finally, the L2 distance between the predicted noise  $n(z_i^t, t, i, v, c)$  and the added noise  $n_t$  is calculated as the training target:
$$
\mathcal {L} _ {r e c} = \left\| n \left(z _ {i} ^ {t}, t, i, v, c\right) - n _ {t} \right\| _ {2} \tag {7}
$$
where  $\| * \|_2$  denotes the calculation of the L2 distance. The total training loss is:
$$
\mathcal {L} = \mathcal {L} _ {\text {r e c}} + \lambda_ {1} \mathcal {L} _ {k l} + \lambda_ {2} \mathcal {L} _ {\text {c r a}} ^ {G} \tag {8}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are hyper-parameters and  $\mathcal{L}_{\text{tra}}^{G}$  is specified in the following paragraph. During generation, the video decoder synthesizes video frames given content features, frame indexes and video descriptions (if exist) by denoising noise features with multiple steps as in [30; 34].
Coherence and Realism Adversarial Loss We propose a novel Coherence and Realism Adversarial (CRA) loss to improve global coherence and local realism of synthesized video frames. To reduce computation complexity, we randomly synthesize two video frames with indexes  $i$  and  $j$ , where  $0 \leq i < j \leq F$ , using the video decoder. The synthesized video frame  $\bar{x}_i$  can be decoded by KL-VAE from the frame feature  $\bar{z}_i$ , which is obtained directly in each training step with the following formulation:
$$
\bar {z} _ {i} ^ {0} = \frac {1}{\sqrt {\bar {\alpha} _ {t}}} z _ {i} ^ {t} - \frac {(1 - \bar {\alpha} _ {t})}{\sqrt {\bar {\alpha} _ {t}}} n \left(z _ {i} ^ {t}, t, i, v, c\right) \tag {9}
$$
where  $n(z_i^t, t, i, v, c)$  is the predicted noise of  $n_t$ . Then the CRA loss is calculated given the synthesized and real video frames using a video discriminator as depicted in Fig. 2.
In our case, we expect the discriminator to provide supervision on global coherence and local realism of synthesized video frames. To ensure the effectiveness of the CRA loss, we formulate it based on the following two guiding principles: (1) the real samples selected for the discriminator must exhibit the desired correlations, whereas the fake samples must deviate from the expected patterns and are often challenging to distinguish; (2) the real samples selected for the video auto-encoder (i.e. the generator) should serve as the fake samples for the discriminator for adversarial training. In particular, to make the discriminator aware of local realism, we select  $\langle x_i,x_j\rangle$  as the real sample and  $\langle x_i,\bar{x}_j\rangle$ ,  $\langle \bar{x}_i,x_j\rangle$ ,  $\langle \bar{x}_i,\bar{x}_j\rangle$  as fake samples according to the first principle. This is because the latter samples contain at least one synthesized video frame and violate the target pattern of realism. Furthermore, to ensure that the discriminator is aware of the global coherence, we utilize samples that violate temporal relationships like  $\langle x_j,x_i\rangle$  and  $\langle \bar{x}_j,\bar{x}_i\rangle$  as fake samples of the discriminator. Then, according to the second principle,  $\{\langle \bar{x}_i,\bar{x}_j\rangle$ ,  $\langle x_i,\bar{x}_j\rangle$ ,  $\langle \bar{x}_i,x_j\rangle\}$  are chosen as real samples of the video auto-encoder, since these samples contain at least one synthesized video frames. The CRA loss can be formulated as  $\mathcal{L}_{cra}^{D}$  for the discriminator and  $\mathcal{L}_{cra}^{G}$  for the generator:
$$
\begin{array}{l} \mathcal {L} _ {c r a} ^ {D} = \log (1 - \mathcal {D} (\langle x _ {i}, x _ {j} \rangle)) + \log \mathcal {D} (\langle x _ {i}, \bar {x} _ {j} \rangle) + \log \mathcal {D} (\langle \bar {x} _ {i}, x _ {j} \rangle) \\ + \log \mathcal {D} \left(\left\langle \bar {x} _ {i}, \bar {x} _ {j} \right\rangle\right) + \log \mathcal {D} \left(\left\langle x _ {j}, x _ {i} \right\rangle\right) + \log \mathcal {D} \left(\left\langle \bar {x} _ {j}, \bar {x} _ {i} \right\rangle\right) \tag {10} \\ \end{array}
$$
$$
\mathcal {L} _ {c r a} ^ {G} = \log (1 - \mathcal {D} (\langle \bar {x} _ {i}, \bar {x} _ {j} \rangle)) + \log (1 - \mathcal {D} (\langle \bar {x} _ {i}, x _ {j} \rangle)) + \log (1 - \mathcal {D} (\langle x _ {i}, \bar {x} _ {j} \rangle))
$$
As illustrated in Fig. 2, the discriminator is built on several spatio-temporal modules that consist of residual blocks and spatio-temporal full attentions. Considering that traditional attention is invariant to the order of input features, two position embeddings  $e_0$  and  $e_1$  are employed and added to the previous video frame  $x_i$  and the subsequent video frame  $x_j$  respectively with  $0 \leq i < j \leq F$ . These position embeddings are randomly initialized and optimized at the same time with the discriminator.
3.2 Generative Model
As shown in Fig. 2, we can obtain any desired video clip by feeding the frame indexes, global feature, and corresponding video description (if available) to the video decoder. Since the global feature is the only unknown term, we can train a conditional generative model to generate a global feature based on the video description. Considering that global feature are 2D features, the generative model can be relieved from the burden of modeling intricate video details.