NUWA-XL: Diffusion over Diffusion for eXtremely Long Video Generation
Year: 2023

Abstract:
In this paper, we propose NUWA-XL, a novel Diffusion over Diffusion architecture for eXtremely Long video generation. Most current work generates long videos segment by segment sequentially, which normally leads to the gap between training on short videos and inferring long videos, and the sequential generation is inefficient. Instead, our approach adopts a “coarse-to-fine” process, in which the video can be generated in parallel at the same granularity. A global diffusion model is applied to generate the keyframes across the entire time range, and then local diffusion models recursively fill in the content between nearby frames. This simple yet effective strategy allows us to directly train on long videos (3376 frames) to reduce the training-inference gap and makes it possible to generate all segments in parallel. To evaluate our model, we build FlintstonesHD dataset, a new benchmark for long video generation. Experiments show that our model not only generates high-quality long videos with both global and local coherence, but also decreases the average inference time from 7.55min to 26s (by 94.26%) at the same hardware setting when generating 1024 frames. The homepage link is https://msra-nuwa.azurewebsites.net/

heading: 1 Introduction
text: Recently, visual synthesis has attracted a great deal of interest in the field of generative models. Existing works have demonstrated the ability to generate high-quality images (Ramesh et al., 2021; Saharia et al., 2022; Rombach et al., 2022) and short videos (e.g., 4 seconds (Wu et al., 2022b), 5 seconds (Singer et al., 2022), 5.3 seconds (Ho et al., 2022a)). However, videos in real applications are often much longer than 5 seconds. A film typically lasts more than 90 minutes. A cartoon is usually 30 minutes long. Even for “short” video applications
∗Both authors contributed equally to this research. † Corresponding author.
like TikTok, the recommended video length is 21 to 34 seconds. Longer video generation is becoming increasingly important as the demand for engaging visual content continues to grow.
However, scaling to generate long videos has a significant challenge as it requires a large amount of computation resources. To overcome this challenge, most current approaches use the “Autoregressive over X” architecture, where “X” denotes any generative models capable of generating short video clips, including Autoregressive Models like Phenaki (Villegas et al., 2022), TATS (Ge et al., 2022), NUWA-Infinity (Wu et al., 2022a); Diffusion Models like MCVD (Voleti et al., 2022), FDM (Harvey et al., 2022), LVDM (He et al., 2022). The main idea behind these approaches is to train the model on short video clips and then use it to generate long videos by a sliding window during inference. “Autoregressive over X” architecture not only greatly reduces the computational burden, but also relaxes the data requirements for long videos, as only short videos are needed for training.
Unfortunately, the “Autoregressive over X” architecture, while being a resource-sufficient solution to generate long videos, also introduces new challenges: 1) Firstly, training on short videos but forcing it to infer long videos leads to an enormous training-inference gap. It can result in unrealistic shot change and long-term incoherence in generated long videos, since the model has no opportunity to learn such patterns from long videos. For example, Phenaki (Villegas et al., 2022) and TATS (Ge et al., 2022) are trained on less than 16 frames, while generating as many as 1024 frames when applied to long video generation. 2) Secondly, due to the dependency limitation of the sliding window, the inference process can not be done in parallel and thus takes a much longer time. For example, TATS (Ge et al., 2022) takes 7.5 minutes to generate 1024 frames, while Phenaki (Villegas et al., 2022) takes 4.1 minutes.
1309
To address the above issues, we propose NUWAXL, a “Diffusion over Diffusion” architecture to generate long videos in a “coarse-to-fine” process, as shown in Fig. 1. In detail, a global diffusion model first generates L keyframes based on L prompts which forms a “coarse” storyline of the video. The first local diffusion model is then applied to L prompts and the adjacent keyframes, treated as the first and the last frames, to complete the middle L− 2 frames resulting in L+(L− 1)× (L− 2) ≈ L2 “fine” frames in total. By iteratively applying the local diffusion to fill in the middle frames, the length of the video will increase exponentially, leading to an extremely long video. For example, NUWA-XL with m depth and L local diffusion length is capable of generating a long video with the size of O(Lm). The advantages of such a “coarse-to-fine” scheme are three-fold: 1) Firstly, such a hierarchical architecture enables the model to train directly on long videos and thus eliminating the training-inference gap; 2) Secondly, it naturally supports parallel inference and thereby can significantly speed up long video generation; 3) Thirdly, as the length of the video can be extended exponentially w.r.t. the depth m, our model can be easily extended to longer videos. Our key contributions are listed in the following:
• We propose NUWA-XL, a “Diffusion over Diffusion” architecture by viewing long video
generation as a novel “coarse-to-fine” process.
• To the best of our knowledge, NUWA-XL is the first model directly trained on long videos (3376 frames), which closes the traininginference gap in long video generation.
• NUWA-XL enables parallel inference, which significantly speeds up long video generation. Concretely, NUWA-XL speeds up inference by 94.26% when generating 1024 frames.
• We build FlintstonesHD, a new dataset to validate the effectiveness of our model and provide a benchmark for long video generation.

heading: 2 Related Work
text: Image and Short Video Generation Image Generation has made many progresses, auto-regressive methods (Ramesh et al., 2021; Ding et al., 2021; Yu et al., 2022; Ding et al., 2022) leverage VQVAE to tokenize the images into discrete tokens and use Transformers (Vaswani et al., 2017) to model the dependency between tokens. DDPM (Ho et al., 2020) presents high-quality image synthesis results. LDM (Rombach et al., 2022) performs a diffusion process on latent space, showing significant efficiency and quality improvements.
Similar advances have been witnessed in video generation, (Vondrick et al., 2016; Saito et al.,
2017; Pan et al., 2017; Li et al., 2018; Tulyakov et al., 2018) extend GAN to video generation. Syncdraw (Mittal et al., 2017) uses a recurrent VAE to automatically generate videos. GODIVA (Wu et al., 2021) proposes a three-dimensional sparse attention to map text tokens to video tokens. VideoGPT (Yan et al., 2021) adapts Transformerbased image generation models to video generation with minimal modifications. NUWA (Wu et al., 2022b) with 3D Nearby Attention extends GODIVA (Wu et al., 2021) to various generation tasks in a unified representation. Cogvideo (Hong et al., 2022) leverages a frozen T2I model (Ding et al., 2022) by adding additional temporal attention modules. More recently, diffusion methods (Ho et al., 2022b; Singer et al., 2022; Ho et al., 2022a) have also been applied to video generation. Among them, VDM (Ho et al., 2022b) replaces the typical 2D U-Net for modeling images with a 3D U-Net. Make-a-video (Singer et al., 2022) successfully extends a diffusion-based T2I model to T2V without text-video pairs. Imagen Video (Ho et al., 2022a) leverages a cascade of video diffusion models to text-conditional video generation.
Different from these works, which concentrate on short video generation, we aim to address the challenges associated with long video generation.
Long Video Generation To address the high computational demand in long video generation, most existing works leverage the “Autoregressive over X” architecture, where “X” denotes any generative models capable of generating short video clips. With “X” being an autoregressive model, NUWA-Infinity (Wu et al., 2022a) introduces autoregressive over auto-regressive model, with a local autoregressive to generate patches and a global autoregressive to model the consistency between different patches. TATS (Ge et al., 2022) presents a time-agnostic VQGAN and time-sensitive transformer model, trained only on clips with tens of frames but can infer thousands of frames using a sliding window mechanism. Phenaki (Villegas et al., 2022) with C-ViViT as encoder and MaskGiT (Chang et al., 2022) as backbone generates variable-length videos conditioned on a sequence of open domain text prompts. With “X” being diffusion models, MCVD (Voleti et al., 2022) trains the model to solve multiple video generation tasks by randomly and independently masking all the past or future frames. FDM (Harvey et al., 2022) presents a DDPMs-based framework that
produces long-duration video completions in a variety of realistic environments.
Different from existing “Autoregressive over X” models trained on short clips, we propose NUWAXL, a Diffusion over Diffusion model directly trained on long videos to eliminate the traininginference gap. Besides, NUWA-XL enables parallel inference to speed up long video generation

heading: 3 Method

heading: 3.1 Temporal KLVAE (T-KLVAE)
text: Training and sampling diffusion models directly on pixels are computationally costly, KLVAE (Rombach et al., 2022) compresses an original image into a low-dimensional latent representation where the diffusion process can be performed to alleviate this issue. To leverage external knowledge from the pretrained image KLVAE and transfer it to videos, we propose Temporal KLVAE(T-KLVAE) by adding external temporal convolution and attention layers while keeping the original spatial modules intact.
Given a batch of video v ∈ Rb×L×C×H×W with b batch size, L frames, C channels, H height, W width, we first view it as L independent images and encode them with the pre-trained KLVAE spatial convolution. To further model temporal information, we add a temporal convolution after each spatial convolution. To keep the original pretrained knowledge intact, the temporal convolution is initialized as an identity function which guarantees the output to be exactly the same as the original KLVAE. Concretely, the convolution weight W conv1d ∈ Rcout×cin×k is first set to zero where cout denotes the out channel, cin denotes the in channel and is equal to cout, k denotes the temporal kernel size. Then, for each output channel i, the middle of the kernel size (k − 1)//2 of the corresponding input channel i is set to 1:
W conv1d[i, i, (k − 1)//2] = 1 (1)
Similarly, we add a temporal attention after the original spatial attention, and initialize the weights W att_out in the out projection layer into zero:
W att_out = 0 (2)
For the T-KLVAE decoder D, we use the same initialization strategy. The training objective of TKLVAE is the same as the image KLVAE. Finally , we get a latent code x0 ∈ Rb×L×c×h×w, a compact representation of the original video v.

heading: 3.2 Mask Temporal Diffusion (MTD)
text: In this section, we introduce Mask Temporal Diffusion (MTD) as a basic diffusion model for our proposed Diffusion over Diffusion architecture. For global diffusion, only L prompts are used as inputs which form a “coarse” storyline of the video, however, for the local diffusion, the inputs consist of not only L prompts but also the first and last frames. Our proposed MTD which can accept input conditions with or without first and last frames, supports both global diffusion and local diffusion. In the following, we first introduce the overall pipeline of MTD and then dive into an UpBlock as an example to introduce how we fuse different input conditions.
Input L prompts, we first encode them by a CLIP Text Encoder to get the prompt embedding p ∈ Rb×L×lp×dp where b is batch size, lp is the number of tokens, dp is the prompt embedding dimension. The randomly sampled diffusion timestep t ∼ U(1, T ) is embedded to timestep embedding t ∈ Rc. The video v0 ∈ Rb×L×C×H×W with L frames is encoded by T-KLVAE to get a representation x0 ∈ Rb×L×c×h×w. According to the prede-
fined diffusion process:
q (xt|xt−1) = N (xt; √ αt xt−1, (1− αt) I) (3)
x0 is corrupted by:
xt = √ ᾱt x0 + √ (1− ᾱt)ϵ ϵ ∼ N (0, I) (4)
where ϵ ∈ Rb×L×c×h×w is noise, xt ∈ Rb×L×c×h×w is the t-th intermediate state in diffusion process, αt, ᾱt is hyperparameters in diffusion model.
For the global diffusion model, the visual conditions vc0 are all-zero. However, for the local diffusion models, vc0 ∈ Rb×L×C×H×W are obtained by masking the middle L − 2 frames in v0. vc0 is also encoded by T-KLVAE to get a representation xc0 ∈ Rb×L×c×h×w. Finally, the xt, p, t, xc0 are fed into a Mask 3D-UNet ϵθ (·). Then, the model is trained to minimize the distance between the output of the Mask 3D-UNet ϵθ (xt, p, t, x c 0) ∈ Rb×L×c×h×w and ϵ.
Lθ = ||ϵ− ϵθ (xt, p, t, xc0)||22 (5)
The Mask 3D-UNet is composed of multi-Scale DownBlocks and UpBlocks with skip connection,
while the xc0 is downsampled to the corresponding resolution with a cascade of convolution layers and fed to the corresponding DownBlock and UpBlock.
To better understand how Mask 3D-UNet works, we dive into the last UpBlock and show the details in Fig. 3. The UpBlock takes hidden states hin, skip connection s, timestep embedding t, visual condition xc0 and prompts embedding p as inputs and output hidden state hout. It is noteworthy that for global diffusion, xc0 does not contain valid information as there are no frames provided as conditions, however, for local diffusion, xc0 contains encoded information from the first and last frames.
The input skip connection s ∈ Rb×L×cskip×h×w is first concatenated to the input hidden state hin ∈ Rb×L×cin×h×w.
h := [s;hin] (6)
where the hidden state h ∈ Rb×L×(cskip+cin)×h×w is then convoluted to target number of channels h ∈ Rb×L×c×h×w. The timestep embedding t ∈ Rc is then added to h in channel dimension c.
h := h+ t (7)
Similar to Sec. 3.1, to leverage external knowledge from the pre-trained text-to-image model, factorized convolution and attention are introduced with spatial layers initialized from pre-trained weights and temporal layers initialized as an identity function.
For spatial convolution, the length dimension L here is treated as batch-size h ∈ R(b×L)×c×h×w.
For temporal convolution, the hidden state is reshaped to h ∈ R(b×hw)×c×L with spatial axis hw treated as batch-size.
h := SpatialConv (h) (8)
h := TemporalConv (h) (9)
Then, h is conditioned on xc0 ∈ Rb×L×c×h×w and xm0 ∈ Rb×L×1×h×w where xm0 is a binary mask to indicate which frames are treated as conditions. They are first transferred to scale wc, wm and shift bc, bm via zero-initialized convolution layers and then injected to h via linear projection.
h := wc · h+ bc + h (10) h := wm · h+ bm + h (11)
After that, a stack of Spatial Self-Attention (SA), Prompt Cross-Attention (PA), and Temporal SelfAttention (TA) are applied to h.
For the Spatial Self-Attention (SA), the hidden state h ∈ Rb×L×c×h×w is reshaped to h ∈ R(b×L)×hw×c with length dimension L treated as batch-size.
QSA = hWSAQ ;K SA = hWSAK ;V SA = hWSAV (12)
Q̃SA = Selfattn(QSA,KSA, V SA) (13)
where WSAQ ,W SA K ,W SA V ∈ Rc×din are parameters to be learned. For the Prompt Cross-Attention (PA), the prompt embedding p ∈ Rb×L×lp×dp is reshaped to p ∈ R(b×L)×lp×dp with length dimension L treated as batch-size.
QPA = hWPAQ ;K PA = pWPAK ;V PA = pWPAV (14)
Q̃PA = Crossattn(QPA,KPA, V PA) (15)
where QPA ∈ R(b×L)×hw×din , KPA ∈ R(b×L)×lp×din , V PA ∈ R(b×L)×lp×din are query, key and value, respectively. WPAQ ∈ Rc×din , WPAK ∈ Rdp×din and WPAV ∈ Rdp×din are parameters to be learned.
The Temporal Self-Attention (TA) is exactly the same as Spatial Self-Attention (SA) except that spatial axis hw is treated as batch-size and temporal length L is treated as sequence length.
Finally, the hidden state h is upsampled to target resolution hout ∈ Rb×L×c×hout×wout via spatial convolution. Similarly, other blocks in Mask 3DUNet leverage the same structure to deal with the corresponding inputs.

heading: 3.3 Diffusion over Diffusion Architecture
text: In the following, we first introduce the inference process of MTD, then we illustrate how to generate a long video via Diffusion over Diffusion Architecture in a novel “coarse-to-fine” process.
In inference phase, given the L prompts p and visual condition vc0, x0 is sampled from a pure noise xT by MTD. Concretely, for each timestep t = T, T − 1, . . . , 1, the intermediate state xt in diffusion process is updated by
xt−1 = 1√ αt
( xt −
1− αt√ (1− ᾱt) ϵθ (xt, p, t, x c 0)
)
+ (1− ᾱt−1)βt
1− ᾱt · ϵ (16)
where ϵ ∼ N (0, I), p and t are embedded prompts and timestep, xc0 is encoded v c 0. αt, ᾱt, βt are hyperparameters in MTD. Finally, the sampled latent code x0 will be decoded to video pixels v0 by T-KLVAE. For simplicity, the iterative generation process of MTD is noted as
v0 = Diffusion(p, v c 0) (17)
When generating long videos, given the L prompts p1 with large intervals, the L keyframes are first generated through a global diffusion model.
v01 = GlobalDiffusion(p1, v c 01) (18)
where vc01 is all-zero as there are no frames provided as visual conditions. The temporally sparse keyframes v01 form the “coarse” storyline of the video.
Then, the adjacent keyframes in v01 are treated as the first and the last frames in visual condition vc02. The middle L − 2 frames are generated by feeding p2, vc02 into the first local diffusion model where p2 are L prompts with smaller time intervals.
v02 = LocalDiffusion(p2, v c 02) (19)
Similarly, vc03 is obtained from adjacent frames in v02, p3 are L prompts with even smaller time intervals. The p3 and vc03 are fed into the second local diffusion model.
v03 = LocalDiffusion(p3, v c 03) (20)
Compared to frames in v01, the frames in v02 and v03 are increasingly “fine” with stronger consistency and more details.
By iteratively applying the local diffusion to complete the middle frames, our model with m depth is capable of generating extremely long video with the length of O(Lm). Meanwhile, such a hierarchical architecture enables us to directly train on temporally sparsely sampled frames in long videos (3376 frames) to eliminate the training-inference gap. After sampling the L keyframes by global diffusion, the local diffusions can be performed in parallel to accelerate the inference speed.

heading: 4 Experiments

heading: 4.1 FlintstonesHD Dataset
text: Existing annotated video datasets have greatly promoted the development of video generation. However, the current video datasets still pose a great challenge to long video generation. First, the length of these videos is relatively short, and there is an enormous distribution gap between short videos and long videos such as shot change and long-term dependency. Second, the relatively low resolution limits the quality of the generated video. Third, most of the annotations are coarse descriptions of the content of the video clips, and it is difficult to illustrate the details of the movement.
To address the above issues, we build FlintstonesHD dataset, a densely annotated long video dataset, providing a benchmark for long video generation. We first obtain the original Flintstones cartoon which contains 166 episodes with an average of 38000 frames of 1440×1080 resolution. To support long video generation based on the story and capture the details of the movement, we leverage the image captioning model GIT2 (Wang et al., 2022) to generate dense captions for each frame in the dataset first and manually filter some errors in the generated results.

heading: 4.2 Metrics
text: Avg-FID Fréchet Inception Distance(FID) (Heusel et al., 2017), a metric used to evaluate image generation, is introduced to calculate the average quality of generated frames.
Block-FVD Fréchet Video Distance (FVD) (Unterthiner et al., 2018) is widely used to evaluate the quality of the generated video. In this paper, we propose Block FVD for long video generation, which splits a long video into several short clips to calculate the average FVD of all clips. For simplicity, we name it B-FVD-X where X denotes the length of the short clips.

heading: 4.3 Quantitative Results

heading: 4.3.1 Comparison with the state-of-the-arts
text: We compare NUWA-XL on FlintstonesHD with the state-of-the-art models in Tab. 1. Here, we report FID, B-FVD-16, and inference time. For “Autoregressive over X (AR over X)” architecture, due to error accumulation, the average quality of generated frames (Avg-FID) declines as the video length increases. However, for NUWA-XL, where the frames are not generated sequentially, the quality does not decline with video length. Meanwhile, compared to “AR over X” which is trained only on short videos, NUWA-XL is capable of generating higher quality long videos. As the video length grows, the quality of generated segments (BFVD-16) of NUWA-XL declines more slowly as NUWA-XL has learned the patterns of long videos.
Besides, because of parallelization, NUWA-XL significantly improves the inference speed by 85.09% when generating 256 frames and by 94.26% when generating 1024 frames.

heading: 4.3.2 Ablation study
text: KLVAE Tab. 2a shows the comparison of different KLVAE settings. KLVAE means treating the video as independent images and reconstructing them independently. T-KLVAE-R means the introduced temporal layers are randomly initialized. Compared to KLVAE, we find the newly introduced temporal layers can significantly increase the ability of video reconstruction. Compared to T-KLVAE-R, the slightly better FID and FVD in T-KLVAE illustrate the effectiveness of identity initialization.
MTD Tab. 2b shows the comparison of different global/local diffusion settings. MI (Multi-scale Injection) means whether visual conditions are injected to multi-scale DownBlocks and UpBlocks in Mask 3D-UNet or only injected to the Downblock and UpBlock with the highest scale. SI (Symmetry Injection) means whether the visual condition is injected into both DownBlocks and UpBlocks or it is only injected into UpBlocks. Comparing MTD w/o MS and MTD w/o S, multi-scale injection is significant for long video generation. Compared to MTD w/o S, the slightly better FID and FVD in MTD show the effectiveness of symmetry injection.
Depth of Diffusion over Diffusion Tab. 2c shows the comparison of B-FVD-16 of different NUWA-XL depth m with local diffusion length L fixed to 16. When generating 16 frames, NUWAXL with different depths achieves comparable results. However, as the depth increases, NUWA-XL can produce videos that are increasingly longer while still maintaining relatively high quality.
Length in Diffusion over Diffusion Tab. 2d shows the comparison of B-FVD-16 of diffusion local length L with NUWA-XL depth m fixed to 3. In comparison, when generating videos with the same length, as the local diffusion length increases, NUWA-XL can generate higher-quality videos.

heading: 4.4 Qualitative results
text: Fig. 4 provides a qualitative comparison between AR over Diffusion and Diffusion over Diffusion for long video generation on FlintstonesHD. As introduced in Sec. 1, when generating long videos, “Autoregressive over X” architecture trained only on short videos will lead to long-term incoherence (between frame 22 and frame 1688) and unrealistic shot change (from frame 17 to frame 20) since the model has no opportunity to learn the distribution of long videos. However, by training directly on long videos, NUWA-XL successfully models the distribution of long videos and generates long videos with long-term coherence and realistic shot change.

heading: 5 Conclusion
text: We propose NUWA-XL, a “Diffusion over Diffusion” architecture by viewing long video generation as a novel “coarse-to-fine” process. To the best of our knowledge, NUWA-XL is the first model directly trained on long videos (3376 frames), closing the training-inference gap in long video generation. Additionally, NUWA-XL allows for parallel inference, greatly increasing the speed of long video generation by 94.26% when generating 1024 frames. We further build FlintstonesHD, a new dataset to validate the effectiveness of our model and provide a benchmark for long video generation.
Limitations
Although our proposed NUWA-XL improves the quality of long video generation and accelerates the inference speed, there are still several limitations: First, due to the unavailability of open-domain long videos (such as movies, and TV shows), we only validate the effectiveness of NUWA-XL on public available cartoon Flintstones. We are actively building an open-domain long video dataset and have achieved some phased results, we plan to extend NUWA-XL to open-domain in future work. Second, direct training on long videos reduces the training-inference gap but poses a great challenge to data. Third, although NUWA-XL can accelerate the inference speed, this part of the gain requires reasonable GPU resources to support parallel inference.
Ethics Statement
This research is done in alignment with Microsoft’s responsible AI principles.

heading: Acknowledgements
text: We’d like to thank Yu Liu, Jieyu Xiao, and Scarlett Li for the discussion of the potential cartoon scenarios. We’d also like to thank Yang Ou and Bella Guo for the design of the homepage. We’d also like to thank Yan Xia, Ting Song, and Tiantian Xue for the implementation of the homepage.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
line 531 limitations
3 A2. Did you discuss any potential risks of your work? line 547 Ethics Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? abstract line 001; introduction line 107
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We are actively building an open-domain long video dataset and have achieved some phased results, we plan to extend NUWA-XL to open-domain in future work. Second, direct training on long videos reduces the training-inference gap but poses a great challenge to data. Third, although NUWA-XL can accelerate the inference speed, this part of the gain requires reasonable GPU resources to support parallel inference