OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation
Authors: Kepan Nan
Rui Xie
Penghao Zhou
Tiehan Fan
Zhenheng Yang
Zhijie Chen
Xiang Li
Jian Yang
Ying Tai
Venue: ICLR 2025 Poster (2025)
Keywords: Text-Video Dataset
Video Generation
Primary area: datasets and benchmarks

Abstract: Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges: 1) Lacking a precise open sourced high-quality dataset. The previously popular video datasets, e.g.WebVid-10M and Panda-70M, overly emphasized large scale, resulting in the inclusion of many low-quality videos and
short, imprecise captions. Therefore, it is challenging but crucial to collect a precise high-quality dataset while maintaining a scale of millions for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of making full use of semantic information from text tokens. To address these issues, we introduce OpenVid-1M, a precise high-quality dataset with expressive captions. This open-scenario dataset contains over 1 million text-video pairs, facilitating research on T2V generation. Furthermore, we curate 433K 1080p videos from OpenVid-1M to create OpenVidHD-0.4M, advancing high-definition video generation. Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens. Extensive experiments and ablation studies verify the superiority of OpenVid-1M over previous datasets and the effectiveness of our MVDiT.

Published as a conference paper at ICLR 2025
OP:NVID-IM: A LARGE-SCALE HIGH-QUALITY DATASET FOR TEXT-TO-VIDEO GENERATION
Kepan Nan' Rui Xie Penghao Zhou?* Tiehan Fan'
Zhenheng Yang² Zhijie Chen² Xiang Li Jian Yang' Ying Tai"
1 State Key Laboratory for Novel Software Technology, Nanjing University
2 ByteDance Nankai University
ABSTRACT
Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges: 1) Lacking a precise open sourced high-quality dataset. The previously popular video datasets, e.g. Web VOM and Panda-70M, overly emphasized large scale, resulting in the inclusion of many low-quality videos and short, imprecise captions. Therefore, it is challenging but crucial to collect precise high-quality dataset while maintaining scale of millions for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of making full use of semantic information from text tokens. To address these issues, we introduce OpenVid-1M, precise high-quality dataset with expressive captions. This open scenasoc dataset contains over I million text- video pairs, facilitating research on T2V generation. Furthermore, we curate 433K 1080p videos from OpenVide venviid-1i 1M to create OpenVidHD cHD-O.4M, advancing high- definition video generation. Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens. Extensive experiments and ablation studies verify the superiority of OpenVid-11 1M over previous datasets and the effectiveness of our MVDiT. Project webpage is available at https
//nju-pca ab. github. pttpto/ectss//ooooooooooscccsscs oppevvid
INTRODUCTION
Text-to-video (T2V) generation, which aims to create video sequence based on the condition of
a text describing the video, is an emerging visual understanding task. Thanks to the significant
advancements of large multi-modality model Sora (Brooks et al. 2024). T2V generation has recently
garnered significant attention. For example, based on DiT (Peebles Xie 2023). OpenSora
OpenSoraPlan (Lab & etc. (0244) and recent works (Wang al.
2023c Lu et al. (023)) utilize the
collected million-scale text-video datasets to reproduce Sora. However, these diffusion models (Ma)
et al. 20224a) Wang et al.) 2023a Lu et al. 2023) Chen et al.) 2023c; Wang al., (023c) still face two critical challenges: 1) Lacking precise high-quality video dataser. Previously popular video datasets, such as WebVid-IOM and Panda-70M. overly emphasized large scale, resulting in the inclusion of many low-quality videos and short, imprecise captions. Therefore, collecting a precise high-quality text-to-video dataset while maintaining scale of millions is challenging but crucial for T2V generation. 2) Ignoring to fully utilize textual information. Recent T2V methods have focused on vision transformer (e.g., STDiT in OpenSora), using a simple cross attention module, which falls short of making full use of semantic information from text tokens.
In this work, we curate a precise high-quality dataset named OpenVid-1M, which comprises over million
video clips, all with resolutions of at least 512x512, accompanied by detailed captions. As shown in Figure I our OpenVid- 1M has several characteristics: 1) Superior quantity: Compared specific-scenario datasets like UCF- 101, which are typically tailored for particular contexts with limited video clips, our OpenVid- 1M stands out as a million-level dataset designed for open aeenarios,enhancing model generalization and enabling video generation across diverse scenes.
Equal contributions. i Corresponding author: yingtaii iingtainnju... https///gihhhhcc//haaaiccc////s
www.iinuuuuuu........ edu. on

---

Published as a conference paper at ICLR 2025
UCF-101: Biking.
The video captures driving convertible sports car on sunny day- He wearing red plaid shirt, top down, and the man is holding the steering wheel with both hands. The car moving on road with clear sky and green hills in the background. The appears to be enjoying the drive, and the overall atmosphare of the video is chearful and relaxed.
Webvid-10M: Apples the garden.
glasses,
Panda-70M: A woman is cutting vegetables on wooden table.
Existing text-to-video datasets
Open 'idd-MM (Ours)
Figure Comparison of OpenVid-1M to the existing text-to-video datasets. Specific-scenario datasets like UCF-101 contain low rasolution videos with simple captions (categories), WebVid-IOM contains low-quality videos with watermarks and Panda-70M contains many flickering (or still) and blurry videos along with imprecise captions. In contrast, OpenVid-1M contains million high-quality video clips coupled with expressive and precise captions (we highlight nouns in green, verbs in blue, and easily overlooked details in purple).
2) Superior in visual quality: OpenVid-1 1M is strictly selected from the aspects of aesthetics, temporal consistency, motion difference, and clarity assessment. We also curate OpenVidHD-O.41. to advance research in high-definition video generation. Specifically, OpenVid 1M far outstrips the commonly- used WebVid- 1aMMidd-MMM (Bain al., 2021) in both resolution and video quality, WebVid- 10M includes low-quality, watermarked videos. Meanwhile, Panda-70M (Chen et al. 2024b) contains many videos with low aesthetics, static, flickering, excessively dynamic or poor clarity, whereas OpenVid-1M is curated to ensure high-quality visuals across various aspects. 3) Expressive in caption: Specific- scenario datasets like UCF-101 use category labels as captions, while datasets like WebVid-100 ViM and Panda-70M often have short, imprecise captions. In contrast, our OpenVid Vi-MMM provides expressive captions, enabling the generation of rich, coherent video content through the multimodal model LLaVA-v1.6-34B (Liu et al.) (2224).
To address the second challenge, we propose a novel Multi-modal Video Diffusion Transformer (MVDIT). Unlike previous DiT architectures (Lab & etc. [024] Ma et al. (024a) that focus on modeling the visual content, our MVDiT features a parallel visual-text architecture to mine both structure information from visual tokens and semantic information from text tokens to improve the video quality. MVDiT extracts visual and text tokens, combines them into a multi-modal feature, and enhances token interaction through a self-attention module. Then, multi-modal temporal-attention module ensures semantic and structural consistency, while a multi-head cross-attention module integrates text semantics into visuals.
Our contributions are threefold: 1) We introduce OpenVid-1M, -M,,, a million-level high-quality dataset with expressive captions for facilitating video generation. 2) We validate OpenVid-1M T2V task with two models, iee. STDIT and our proposed MVDiT, which makes full use of semantic information from text tokens improve visual quality. 3) We further demonstrate the superiority of OpenVid-11 1-MM on video restoration task.
RELATED WORK
Text-to-video Datasets. Existing text-to-video training datasets can be categorized into two classes:
Specific-scenario and open-scenario. Specific-scenario datasets (Yu et al. 2023a Yuan al. 2024 Rossier et al. 2019 Soomro et al. 2012 Xiong al. 2018 Siarohin et al. 2019) typically consist of
a limited number of text-video pairs collected for specific contexts. For example, UCF-101 (Soomro
et al. 2012) is a action recognition dataset which contains 101 classes and 13,320 videos in total.
Taich- HD Siarohin et al.) 2019) contains 2,668 videos recording a single person performing Taichi. ChronaMagic (Yuan et al." 2024) comprises 2,265 high-quality time-lapse videos with accompanying
2

---

Published as a conference paper at ICLR 2025
Table 1: Data processing pipeline. The first three steps can be processed in parallel to enhance processing efficiency, while the subsequent steps are processed sequentially.
Pipeline
Tool
Computation
Processing Time (hours)
Remark
Resources
Aesthetics score
LAION Aesthetics Predictor
A180
320
Get high aesthetics score set 8
Temporal consistcacy
CLIP Radford et atal. 2021 UniM
48 A100
173
Obtain moderate consistency set Sy
Motion difference
48 A100
59
Obtain moderate amplitude of motion Set Syr
Intersection of quali-
Intersection
Obtain intersection: Si = SA n Sr SM
fiod videos
Clarity assessment
DOVER Technical ea al.
AA100
clear high quality video set Split multi-sceue videos: S Defector(S)
Clip extraction
Cascaded Cut Detector BBaittmann
A110
etal
2023
Video caption
1.6--4b (Luu ct 2023a
8A10
46
Obtain long captions for the videos
text descriptions. As pioneering open-scenario T2V dataset, WebVid-1OM (Bain al. 202 comprises 10.7 million text-video pairs with total of 52K video hours. Panda-70M (Chen et al. 2024b) collects 70 million high-resolution and semantically coherent video samples. Recently, Intern Vid (Wang et al. (223d)) proposes a scalable approach for autonomously constructing a video- text dataset using large language models, resulting in 234 million video clips with text descriptions. However, Web Vid-10M contains low-quality videos with watermarks, Panda-70M contains lots of static, flickering, low-clarity videos along with short captions, while Intern Vid primarily focuses on video understanding tasks. In contrast, our OpenVid-1M comprises over million high-quality in-the-wild video clips, with 433K in 1080p resolution, accompanied by expressive captions.
Text-to-videe Models. Current text-to-video generation methods can be divided into:
UNet (Khacha-
tryan et al.
2023 Ge et 2023 Wang et al. 2023a Chen al., 2023a
2024a Yu al.
2023b]
Zeng et al.
2023), and DiT based methods (Ma et al. 2024a Chen et al.
2023c Lu et al. 2023).
UNet based methods have been widely studied. Modelscope (Wang et al. 2023a) introduces a spatio-
temporal block and a multi-frame training strategy to enhance iext-to-video synthesis, achieving State-of-the-Ar (SOTA) results. VidcoCrafter (Chen et al.) 2024a) investigates the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain high-quality video model. DiT has demonstrated significant superiority and efficiency in image generation task (Chen) al. 2023b; 2024ct Zhao al. 2024) and has recently attracted considerable attention in the field of video generation. Sora (Brooks et al. 2024) revolutionizes video generation. Latte (Ma et al.) 2024a) employs simple and general video Transformer the backbone to generate videos. Recently, OpenSoral trained based on a pretrained T2I model (Chen et al. 2023b) and a large text-to-video dataset, aims to reproduce Sora. In contrast the previous DiT structures, we propose novel MVDiT that features a parallel visual-text architecture, mining both structure information from visual tokens and semantic information from text tokens improve the video quality.
CURATING OpenVid-1M
This section outlines the date processing steps in Table T OpenVid-1M is curated from ChronoMagic, CelebvHQ (Zhu et al., 2022), Open-Sora-plan (Lab & etc. 2024 and Panda Since Panda is much larger than the others, here we primarily describe the filtering details on our downloaded P0nda-50M.
Aesthetics Score. Visual aesthetics are crucial for video content satisfaction and pleasure. To enhance text-to-video generation, we filter out videos with low aesthetics scores using the LAION Aesthetics Predictor. This results in subset SA with the top 20% highest-scoring videos from Panda-50M. For the other three datascts, we select the top 90% to form subset S'A.
Temporal Consistency. Video clips with temporal consistency are crucial for training. We use CLIP (Radford et al. [021] to extract visual features and measure temporal consistency by analyzing cosine similarity between adjacent frames. Clips with high scores (nearly static) and low scores (frequent flickering) are filtered out, yielding a suitable subset, Sr, from Panda-50M.. For the other datasets with good temporal consistency, filtering is performed.
Motion Difference. We employ UniMatch (Xu et al. 2023) to assess optical flow as a motion difference score, selecting videos with smooth movement, since temporal consistency alone is insufficient to filter out high-speed objects that still maintain consistency. Videos with high flow scores, indicating rapid motion, are unsuitable for training. We filter out clips with the highest and
Since can only download 50M, we refer to this version Panda-50M in this work.
3

---

Published as a conference paper at ICLR 2025
Pands OUrs-OU-M
Panda
Ours-O-4M
Percemage 1N1
40
Pencenage
15
28
10-
Holleiig Aidens
8:9
(al Video aesthetics distribttonn
Video temporal consistency distribution
80
60
Randa: 50M
Ours-0
Ours 0.4M
50-
Ours 0.4M
50
Percentage
20
C0 Static Videox
300
10
Eacesslivll Epsaans Aieecs
20
40
60
80
100
120
140
150
%
50
100
150
250
250
200
video mocion distribttion
Caption length distribution
Figure 2: Comparisons on video statisties between OpenVid-1M and Panda-50M.
Indker Scarne
20.00 By
Fortrat
5.0e-23.x
Seai and Stlws
27
Natural Scesey
08:-38 E
Vencoo
20k-6.t
832
Viseo cierity dhatribulter
Video Categories
Video Danatarrs
Figure 3: Left:
Clarity distribution of OpenVid-1M. We also present 4 samples to visualize the clarity differences. Samples outlined in green contour are blurry with low clarity scores, while those outlined in red contour are clearer with high clarity. Middle & Right: OpenVid- 1M contains diverse distributions of video category and duration.
lowest scores in Panda-50M to create subset Sas. For the other three datasets, we derive subset, Sy, without applying the remaining processing steps. Instead, we directly calculate the intersection of SS', and S/ to obtain S' S'A S i.e., Ours-0.4M illustrated in Figure 2
Clarity Assessment. High-clarity videos are essential for T2V generation. Since Panda-50M contains many blurry clips, we filter those with very low clarity, as shown in Figure 3 We calculate the intersection of the three sets from Panda to obtain S₁ = S O ST n SM. resulting in aesthetically pleasing, stable videos with smooth movement. Using the DOVER (Wu et al., 2023) model, estimate the DOVER-Technices score for each clip in S₁ and retain high-clarity videos with clean textures. Finally, we select the top 30% of clips with the highest scores to form the video set S.
Clip Extraction. Beyond the aforementioned steps, some video clips may contain multiple scenes, thus we introduce the Cascaded Cut Detector (Blattmann al. (023) to split multi-scene clips in S to achieve clip extraction, ensuring each contains only one scene. After clip extraction, we obtain S from Panda-50M, iee., Ours-0.6h -0.6 illustrated in Figure
Video Caption. As highlighted in Sora technical report, detailed captions greatly benefit video generation. After obtaining the video clip set, we recaption them using large multimodal model, LLaVA- (Liu et al. 2023a), to create expressive descriptions. Since CelebvHQ lacks captions, we also provide captions for its video clips. Figure 2 d) compares the length of our prompts with those in Panda-50M, showing our expressive prompts offer a significant advantage by providing richer semantic information. We compile our high-quality dataset, OpenVid- 1M (i.e., Ours-0.6M Ours- 0.4M). Additionally, we meticulously select 1080p videos from OpenVid 1M to construct OpenVidHD- 0. AM, advancing the High-Deffnition (HD) video generation within the community.
4

---

Published as a conference paper at ICLR 2025
Table 2: Comparisons with previous text-to-video datasets. Our OpenVid-1M is million-level,
high-quulity and open-scenario video dataset for training high-fidelity text-to-video models.
Dataset
Scenario
Vidco clips
Average longth (scconds)
Duration (hours)
Resolution
Caption
UCFI0I
Action
13K
7.2
2.7
220
N/A
Taichi
Human
3K
256
N/A
SkyTimelapse
Sky
35K
640x360
N/A
FaceForensicst*
Diverse
N/A
WehVid
Open
T0M
18.7
52k
396x333
Short
Intern
Open
234M
11.7
760.3K
Diverse
Short
ChronoMagic
Metamorphic
2K
Diverse
Long
CelebvHQ
Portrait
35K
6.6
65
512x512
N/A
OpenSoraPlan-Va
Open
400K
24.5
274
512x512
Long
Panda
Open
70M
8.5
166k
Diverse
Short
OpenVid-1M (Ours)
Open
IM
7.2
2.1k
Diverse
Long
OpenVidHln 0.4M (Ours)
Open
433K
9.6
1.2k
1920x
Long
4 DATA PROCESSING AND STATISTICAL COMPARISON
Data Processing Differences against SVD (Blattmann et al. 2023). Our data processing pipeline draws inspiration from the SVD pipeline, yet several distinctions exist: 1) Vixual quality evaluation: Both SVD and our OpenVid-1M utilize an aesthetic predictor to retain highly aesthetic videos. Additionally, OpenVid-11 1M integrates the recent model DOVER (Wu et al. (2023) to assess the video clarity, preserving high-quality videos with clean textures. 2) Motion evaluation: SVD utilizes the traditional Farneback optical flow method and RAFT (Teed & Deng 2020) to estimate optical flows. In contrast, OpenVid-1M adopts more efficient UniMatch (Xu et al. (223) to achieve better optical flows, addressing not only static videos but also those with fast movements. 3) Time consistency evaluation: SVD employs clip extraction solely to prevent sudden video changes, whereas OpenVid- 1M additionally removes flicker videos. 4) Processing efficiency: SVD initially extracts video clips and then filters from a large pool, while OpenVid-11 1-1M first selects high-quality videos and then extracts clips, significantly enhancing processing efficiency. Finally, OpenVid-11 1M will be made publicly available, while the training dataset in SVD is not.
Comparison with Panda-50M. The statistical comparisons between OpenVid i and Panda-50M are illustrated in Figure?? 2 1) Video Aesthetics Distribution: Our subsets Ours-0.6M and Ours-0.4M
exhibit higher aesthetics scores compared to Panda-50M, suggesting superior visual quality. 2) Video Motion Distribution: Our subsets display higher proportion of videos with moderate motion, implying smoother and more consistent motion. Conversely, Panda-50M appears to contain numerous videos with flickering and statie scenes. 3) Video Temporal Consistency Distribution: Our subsets exhibit a more balanced distribution of moderate temporal consistency values, whereas Panda-50M includes videos with either static excessively dynamic motion. 4) Caption Length Distribution: Our subsets feature significantly longer captions than Panda-50M, providing richer semantic information. Overall, OpenVid-11 1M demonstrates superior quality and descriptive richness, particularly in aesthetics, motion, temporal consistency, caption length and clarity as well.
Comparisons with Other Text-to-video Datasets. We compare our OpenVid- 1M and Open dHD- 0.4M to several previous datasets in Table We also present video categories and durations statistics of OpenVid- 1M in Figure 3 As shown, OpenVid-1M is a million-scale, high-quality and open- scenario video dataset designed for training high-fidelity text-to-video models. Specifically, OpenVid- 1M consists of 1,019,957 clips, averaging 7.2 seconds each, with a total video length of 2,051 hours. Compared to previous million-level datasets, WebVid-1OM contains low-quality videos with watermarks and Panda- 70M contains many still, flickering, or blurry videos along with short captions. In contrast, our OpenVid-11 1M contains high-quality, clean videos with dense and expressive captions generated by the large multimodal model LLaVA-v1.6-344 Additionally, compared to previous high-quality datasets that are usually designed for specific scenarios with limited video clips, our OpenVid- a large-scale dataset for open scenarios, including portraits, scenic views, cityscapes, metamorphic content, etc.
5 METHOD
Inspired by MMDiT (Esser et al. 2024), we propose a Multi-modal Video Diffusion Transformer (MVDIT) architecture. Shown Higure 4 its architecture diverges from prior methods (Lab & etc.
5

---

Published as a conference paper at ICLR 2025
2024 Ma et al. 2024a) by emphasizing parallel visual-text structure for mining both structure from visual tokens and semantic from text tokens. Each MVDiT layer encompasses four steps: Initial extraction of visual and linguistic features, integration of novel Multi-Modal Temporal-Attention module for improved temporal consistency, facilitation of interaction via Multi- Self-Attention and Huad Cross-Attention modules, and subsequent forwarding to the final feedforward layer.
5.1 FEATURE EXTRACTION
Given a video clip, we adopt pre-trained variational au- toencoder encode input video clip into features in latent space. After being corrupted by noise, the obtained video latent is input into a 3D patch embedder to model the tem- poral information. Then, add positional encodings and flatten patches of the noised video latent to a patch encoding sequence X E Rx HW Following Chen et al. (2023b), We input the text prompts into the T5 large language model (Raffel et al.] 2020) for conditional feature extraction. Then, we embed the text encoding to match the channel dimension of the visual tokens to obtain the text tokens Y RCxL where L represents the length of the text tokens. Finally, we take the text and noised visual tokens as input of MVDiT.
Suaind
Nowwi Restamad
Touree Featsorard
Melli ieed Cros Aitettion
+
Mais Alecal
Turapant Ansetice
5.2 MULTI-MODAL VIDEO DIFFUSION TRANSFORMER
Multi-Modal Self-Attention Module. We design a Multi- Modal Self- Self-AAtention (MMSA) module. Text tokens are repeated by T times along the temporal dimension to gener- ate Y E R XCXLL We adopt adaptive layer normalization both in branch and visual branch encode timestep information into the model. Then, we concatenate the visual tokens with text tokens to generate the multi-modal feature F Cx*CCx(W++L which is input into the MMSA mod- ule containing Self Attentoonn Layer (SAL):
besch
MulL-Mooll
sanict
Naad
Toi
Toooe
Figure Overview of MVDIT with parallel visual-text architecture. Con- catenation is indicated by C and split is indicated by a
SAL SAL((Conea(AAaaLNXXX AdaLN( t.))
(1)
AdaLN(XX t1) = 7 LayerNorm(X)) + 3.. (2)
The self-attention operation is conducted to promote the interaction between visual tokens and text tokens in each frame, which can be implemented easily with matrix multiplication. Notably, since each video frame is paired with a unique text prompt, the text tokens vary across frames after the SAL, where they receive structural information from different frames. Then, we split the visual tokens and text tokens from the enhanced multi-modal features. Following Chen et al. (2023b), we also regress dimension- wise scaling parameter a, which applied before residual connections within the Transformer block. It can be formulated as follows:
XA Y SAL = Split(F SAL); X X x SAL: Y Y Y
Multi-Modal Temporal-Attention Module. After obtaining the enhanced visual features and text features, we build a Multi-Modal Temporal-Attenn Attentioo (MMTA) module on the top of the MMSA to efficiently capture temporal information. Unlike the temporal attention used in previous methods (Lab & etc. 2024) Ma et al., (024a)), we consider capturing temporal information from both the text features and the visual features. Specifically, we concatenate the tokens from two branches obtain the multi-modal feature F' (HWW+++) We then input F' into the MMTA module, where Temporal-Attenn Attention Layer (TAL) is used to conduct communication along the temporal dimension:
XLAL, Y
t2), AdaLN(Y t2)))),
X = X"
S
Y
Y
o Y! SAL
(5)

---

Published as a conference paper at ICLR 2025
20
Diceee 159
05
ChRS
1200
GPG GPUs
0.955
1000
$6990
10
ViDeoCnittrr
5:965
Show-1
35
A
OpL
20
20
20
40
50
70
80
05 20 30 40 35 65 70
Staps (5)
0.0
OGPUs
Figure 5: Left: Comparison with SOTA T2V models VQAT, GPU type and resolution. The color of the middle dot in each circle indicates GPU type, and circle diameter represents video resolution. Middle: Curves of FVD with different number of GPUs. More GPUs accelerates the model's convergence. Right: Curves of clip temporal score and warping error. Our T2V model typically starts stabilize at 35K steps and achieves temporal consistency around 50K steps.
This design enables the model to learn both structural temporal consistency from visual information and semantic temporal consistency from textual information. For simplicity, temporal positional embedding is omitted.
Module. While the MMSA module merges tokens from both modal- ities for attention, T2V still requires an explicit process to embed semantic information from text tokens into visual tokens. The absence of semantic information may impair video generation per- formance. Therefore, we developed a Multi-Head Cross-Attention (MHCA) module, in which a Cross-Attention Layer (CAL) is introduced to facilitate direct communication between text and visual tokens. Specifically, we take the flattened visual tokens X E HB''xxxHW as Query and text tokens Y! IR X Key and Value, and input them into a cross-attention layer:
Multi-HHad
X X CAL(X +X*
(6)
Afterward, both the visual and text tokens are passed through a feedforward layer. Since a single MVDIT layer updates both token types, this process can be repeated iteratively to enhance video gen- eration performance. After N iterations, the final visual feature is used to predict noise and covariance at time l. Our MVDIT is inspired by MMDiT, whose effectiveness has been thoroughly validated. Importantly, our work is the first to emphasize a parallel visual-text structure for extracting structural information from visual tokens and semantic information from text tokens in T2V generation.
6 EXPERIMENTS
6.1 EXPERIMENTAL SETTINGS
Datasets and Evaluation Metrics. We adopt proposed OpenVid- 1i train our MVDiT. OpenVi dHDD 0.4M is used further for HD video generation. WebVid- 10M and Panda-50M are adopted for dataset comparisons. We evaluate model on public benchmark in Liu et al. (2023b). which evaluates text-to-video generation model based on visual quality, text-video alignment and temporal consis- tency. Specifically, we adopt aesthetic score (VQA and technical score(VQAr) for video quality assessment. We evaluate the alignment of input text and generated video in two aspects, including image-video consistency (SD_so (SD_score) and text-text consistency (Blip_bleu). We also evaluate temporal consistency of generated video with warping error and semantic consistency "Clip_temp score..
Implementation Details. We use Adam (Kingma & Ba 2014) as optimizer, and the learning rates is set to 2e - 5. We sample video clips containing 16 frames at 3-frame intervals in each iteration. We adopt random horizontal flips and random crop to augment the clips during the training stage. All experiments are conducted on NVIDIA 1100 80G GPUs. We adopt PixArt-a (Chen et al. (023b) for weight initialization and employ T5 model as the text encoder. The training process starts with 256x250 256 models, whose weights are then used to train 512 models, and these in turn serve as pretrained weights for 1024x 1024 models. Starting with now-resolution training equips the model with coarse-gre graneed modeling capabilities, while subsequent high- reghuutooottion finetuning enhances its ability to capture fine details. This staged approach reduces both computational cost and overall training time compared directly starting with high-resolution training.
7

---

Published as a conference paper at ICLR 2025
Table 3: Comparison with state-of-the-art text-to-video generation methods. The best results marked in bold, while the second best ones are underscored.
Method
Resolatios
VQAut
VQAyT
Blip_heett
SD_scoret
Clip_ecsp_caee
Waspiag_crooe
532
Vinse255
42.50
22.38
68.15
99.57
0.0059
Shows
526x1000
WebVid IIM
23.19
44:34
23.34
59:77
0.067
513
Self cullected- IIM
20.04
1562
23.00
67.65
99/66
0.1700
Latic Maut
512x513
Saff collacted- 1 10K
55.46
46.55
22.39
68.05
00.59
0.0035
VideoCrutter LTer drole P0-3y
1024x:576
Wes 1084; Laion 0000
66.18
5895
22.17
08.73
99:76
0,0295
Modelscope
Wang 11 20230
1280x720
Self collected Billions
40.06
1293
22.54
67:93
99:74
0.162
1088
Unbnown Self collected-7.. IM
59:00
64.55
21:14
08.57
99.97
0
OpenScruPlun-V V1
Lib
2ic 2004
640
23.25
65.46
19.03
69.21
99.97
0.001
CogVideeX 3B
+400
Self collected 35M
25.12
76,36
24.31
68.91
99.79
0,077
O
1024x:102
OpanVad-18
73.46
68.58
23.45
68.04
99.37
0,0032
Table 4: Comparisons with previous representative text-to-video training datasets. The STDiT model used in OpenSora is adopted and kept the same for all of the cases. For fair comparison, training iterations are selected at the same step (50K) for fair comparison. All models with 256 resolution are adequately trained on 32 A100 GPUs for least 14 days to reach 50K iterations.
Resolation
Training Data
VQA.T
VQArt
bip_b
SD score
Clip pemp__emporrr"
Warping_
256x
WebVid 10M (Dain et al 2021
13.40
13.34
23.45
67.64
99.62
0.0138
256x 256
PUnda-S0M ( Cheiee ml
202441
12.08
9.60
24.06
67.47
99.60
0.0200
OpanVid (O((O))
17.78
12.98
24.93
67.77
99.75
0.0134
1024x1024
Web (4x Super-resolution)
69.26
65.74
23.15
67.60
99.64
0.0137
1024x1024
Punda-5UM Super-acsolttion
67.44
0.0163
1024x1024
Panda-50M-HD
13:48
42.89
21.78
68.43
99.84
0.0136
1024x1024
OpenVadil--o 0.44 (Ours)
73.46
68.58
23.45
68.04
99.87
0,0052
6.2 COMPARISON WITH STATE-OF-THE-ART MODELS
In this section, we evaluate our method's performance and compare it with other models. For each model, we employ consistent set of 700 prompts from Liu al. (2023b) to generate videos. Metrics from Liu et al. (2023b) is used to evaluate the quality of generated videos.
Quantitative Evaluation. The comparison between our method and others is summarized in Table and Figure 5) Our model achieves the highest VQA (73.46%) and the second best VQA (68.58%), indicating superior video aesthetics and clarity. Additionally, it achieves the second best hip__emp_sceee score (99.87%). demonstrating its good ability on temporal consistency. Overall, our model shows robust performance across various metrics while using less training data, demonstrating the superiority of our OpenVid-1M, highlighting its effectiveness in text-to-video generation tasks.
The comparison between OpenVid-11 1M and previous representative text-to-video training datasets is listed in Table We adopt STDIT model used in OpenSora for all of the cases. In 256 x 256 resolution, the model trained with our OpenVid-1N 1M generates the best scores across all metrics except VQAT. This is reasonable, the low resolution of the videos prevents showcasing the high quality of OpenVid-1M. The similar conclusion can be found in 1024 x 1024 resolution results, indicating the superiority of OpenVid-1M4 MM in generating high-quality videos. Moreover, our OpenVidHD-O. 1.DD-...MM 0.4M can be directly used to train high-definition (e.g.. 1024 x 1024) videos, whereas Web Vid- 10M cannot and Panda-50M has not yet undergone resolution- and quality-level filtering. To compare results at 1024 x 1024 resolution, we use x4 video super-resolution to generate 1024 x 1024 videos from models trained on WebVid-10h and Panda-50M. Clearly, training with our OpenVicHD-O..A dHD-0.4M yields better scores than combining other datasets with super-resolution. We also manually select 1080P videos from Panda-50M to create Panda-50M-HD for a fairer comparison. We can see that the model trained on our Open Vid- 1M demonstrates superior performance, while the model trained on Panda-50M. 50D performs poorly. This discrepancy may be attributed to the low-quality videos in Panda-50M-HD (e.g., low aesthetics and clarity, nearly static scenes, and frequent flickering) problem our data processing pipeline effectively avoids.
Qualitative Evaluation. Visual comparisons are shown in Figure 6 The first column demonstrates that our model generates clearer, more aesthetically pleasing and more detailed videos due to our high-resolution OpenVid-1M. In the second example, our model demonstrates a strong ability on prompt understanding, accurately depicting the "android" and 'surrounded by colorful Easter eggs' from the text. In the third column, unrealistic dust appears in front of the in videos from Lavie and VideoCrafter, while our model better captures the 'kicking up dust', highlighting its superior motion quality. We emphasize our method's ability to generate clearer and more aesthetically pleasing videos
8

---

Published as a conference paper at ICLR 2025
Lovie
O. v1.1
Video Oro
Pika
Cura
paranoul undoaa fresking out junping Jnto the decouse nurronnded by coloraal Easter
tricersncpo charging down
popplp mack licknng up dur back road.
oggs.
Figure 6: Visual comparison of different T2V generation models. Please zoom in for more details.
Table 5: Quantitative comparison of models trained on different datasets for video restoration.
Method
Training Dataset
Dataset Size
PSNR
SSIM
LPIPS
DOVERT
Enarp
Upscale-A A-sdco
(CVPR 2024)
WebVid. YouTube
370K 130K
23.43
0.6195
0.2731
0.4863
0.00532 0.00283
Ours
OpenVidHD-O.44
23.49
0.7165
0,2015
0.5351
LR
Upscale-A- --iieo
Ours
Figure 7: Visual comparison of restoring low-resolution (LR) video from UDM10 TTao et al. 2017).
compared to closed-source commercial product Pika, which are trained on much larger datasets and with more computational resources. We present higher resolution versions in Figure [3] Figure [4] and Figure [5 for clearer comparison. Figure presents a comprehensive analysis the proposed model's performance against SOTA T2V models across various metrics.
Video Restoration. We further demonstrate the superiority of OpenVidHD-O. on the video restora- tion task. As shown in Table 5 used OpenVidHD-O.Wm 4M to synthesize 130K training samples to train a video restoration model 12VGen-XL. for arbitrary-resolution super-resolution, compar- ing it with the SOTA video restoration method, Upscale-A- A-alle-A-Viees (Zhou et al. 2024). The results show that our model outperforms across metrics (both in fidelity and perception), demonstrating that OpenVidHD- 0..0..44 4M significantly improves performance, even without task task-specific design opti- mizations, due to its high quality. The visual comparison in Figure ( shows that the model trained with @pHn-VidDD-O produces clearer textures and more accurate structures.
6.3 ABLATION STUDY
Ablations on Resolution, Architectures and Training Data. Results are depicted in Table 6 We draw the following conclusions: 1) Higher resolution leads to better metric scores. 2) The proposed MVDiT further improves both VQA and VQAr compared to STDiT, indicating higher video quality and greater diversity. 3) More high-quality training data results in better metric scores.
Ablations on MVDIT. As shown in Table 7 we conduct ablations MHCA module and scaling parameter a on MVDIT-256. From the results, we can draw the following conclusions: MHCA boosts video quality and alignment, and parameter improves video quality and convergence. Notably, we observed that removing a causes the loss to decrease very slowly, indicating that a accelerates training, consistent with the findings reported in Peeebles & ( (2023). Please note that after removing MMTA, the model unable to generate videos and instead produces multiple unrelated images, completely failing to meet the requirements for video generation.

---

Published as a conference paper at ICLR 2025
Table 6: Ablations different resolutions, architectures and training data. For models trained with 256x256 resolution, training iterations are selected at the similar steps for fair comparison. 'Pretrained Weight' means initializing with corresponding pretrained model, e.g., 'MVDIT-256'
indicates that the MVDIT model with 256 256 resolution is used as the pretrained weight.
Model
Resolution
Training Data
Pretrained Weight
VQAxT
VQAyT
Blip_bleu?
SD sDune
Clip_ Cip__empp_cong
Warping_enee earor
STDIT
256x256
Ours-0.4M
PixAn-
11.11
12.46
24.85
67.96
99.81
0.0105
512x512
Ours-B.4M
STDIT-256
MVDIT
256
Ours-0.4M
PixArt-
22.39
14.15
23:72
67.73
99:71
0.0091
MVDIT
256x256
Open/idd12 1N
PisAn-o
21.87
14.57
21,01
67.61
99.75
0.0081
MVDIT
512x512
OpenVid-1N
MVDIT-256
66.65
63.96
24.14
68.31
99,83
0.0008
Table 7: Ablation studies on the effectiveness of modules in MVDIT.
Setting
VQA.t
VQArt
Blip_bleur"
SD SD_scoret
Warping
w/o MHCA
13.9
12.35
19.74
67.58
99.73
0.01
w/o a MVDiT
3.16
3.55
14.38
66.94
99.01
0.0561
22.39
14.15
23.72
67.73
99.71
0.0091
Table 8: Evaluation of human preference on captions over 1, 117 samples and volunteers.
Omission
Hallucinations
Distortion
Temporal Mismatch
Mean
Preference
Panda's short captions Our generated long captions
3.18
4.29
3.84
3,53
3.71
19.25%
4.53
4.08
4.28
3,99
4.22
80.75%
Human Preference on Captions. In Table we compare Panda's 's short captions with our generated long captions on 1, 117 validation samples, evaluated by 10 volunteers. Volunteers assessed captions across four criteria: (1) Omission: missing key elements, (2) Hallucinations: imagined elements, (3) Distortion: accuracy of attributes like color and size, and (4) Temporal mismatch: accuracy of event sequences. Each pair was rated from 1 to 5 (higher is better), and preferences were recorded. The results show: (I) The long captions provide richer descriptions, particularly in element accuracy and temporal events. (2) Though some hallucinations occur, accurate descriptions dominate. (3) Both captions can still be improved in modeling motion. (4) Long captions are strongly preferred overall.
7 CONCLUSION
In this work, we propose OpenVid-1K, 1MM a novel precise high-quality datasets for text-to-video gen- eration. Comprising over million high-resolution video clips paired with expressive language descriptions, this dataset aims to facilitate the creation of visually compelling videos. To ensure the inclusion of high-quality clips. we designed an automated pipeline that prioritizes aesthetics, tempo- ral consistency, and fluid motion. Through clarity assessment and clip extraction, each video clip contains single clear scene. Additionally, we curate OpenVidHD-We 0.4M, subset of OpenVid- Vid-1N for advancing high-definition video generation. Furthermore, we propose novel Multi- Video Diffusion Transformer (MVDiT) capable of achieving superior visually compelling videos, making full use of our powerful dataset. Extensive experiments and ablatíve analyses affirm the efficacy of OpenVid- compared to prior famous datasets, including Web Vid-I0M and Panda-50M.
I Limitations. Despite advancements in T2V generation, our model, like previous SOTA models, also faces limitations in modeling the physical world. sometimes struggles with intricate dynamics and motions of natural scenes, Icading to unrealistic vidcos. We believe that with more high-quality training data, our model could be further scaled up and enhanced to handle such limitation.
8 ACKNOWLEDGEMENTS
This work was supported by Natural Science Foundation of China: No. 62406135, Natural Science Foundation of Jiangsu Province: BK20241198, the Gusu Innovation and Entrepreneur Leading Talents: No. ZXL2024362, and the AI & Al for Science Project of Nanjing University: No. 14380007.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738. 2021.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024.
ceopenaii.cm/ert wtt.e/oco///eeeeee research/ video- gerooreeeeiiionnn modelss as world simulators
Haoxin Chen, Menghan Xia, Yingqing He. Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint wwXiv:2401.090477 2024a.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic texl-to-image synthesis, 2023b.
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:2312.04557, 2023c.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Ying Lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. arXiv preprint 2xXiv:2402.19479 2024b.
Zhennan Chen, Yajie Li, Haofan Wang, Zhibo Chen, Zhengkai Jiang, Jun Li, Qian Wang, Jian Yang, and Ying Tai. Region-aware text-to-image generation via hard binding and soft refinement. arXiv preprint arXiv:241 1.065588, 2024c.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206, 2024.
songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 199930222441 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11954-15966 2023.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X.:: 2014.
PKU- Yuan Lab and Tuzhan Al etc. Open-sora-plan, apr 2024. URL httttht:///oi.org//// 5281/ 10948109.
Haotian Liu, Chunyuan Li. Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023a.

---

Published as a conference paper at ICLR 2025
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
-lavavvl.gitauul http://.../oooo//ooo//o/0
Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang. Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint urXiv:2310. 1440, 2023b.
Haoyu Lu, Guoxing Yang, Nanyi Fei, Yuqi Huo, Zhiwu Lu, Ping Luo, and Mingyu Ding. Vdt: General-purpose video diffusion transformers via mask modeling. In The Twelfth International Conference on Learning Representations, 2023.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint (TXi::240100004.. 2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15762-15772, 2024b.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 20233.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1 1-6, 2020.
Andreas Rossler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. Faseforensics+#: Learning detect manipulated facial images. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 1-11, 2019.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 1tt.:222200440 2202402,2220 2012.
Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. Detail-revcaling deep video super- resolution. In Proceedings of the IEEE international conference computer vision, pp. 4472-4480. 2017.
Zachary Teed and Jia Deng. Raft: Recurrent all pairs field transforms for optical flow. In European Conference Computer Vision. Springer, 2020.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.0657/, 2023a.
Xiang* Wang, Hangjie* Yuan, Shiwei* Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. 2023b.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu. Peiqing Yang, al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15103 2023c.
Yi Wang, Yinan He. Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen. Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06440 2023d.
12

---

Published as a conference paper at ICLR 2025
Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou, Annan Wang, Wenxiu Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In IEEE/CVF International Conference on Computer Vision. IEEE, 2023.
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2364-2373, 2018.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2023.
Zhuoyi Yang, Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint S00.::2200.000777777 2024.
Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. Celebv-text: lcage-scale facial text-video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14805-14814, 2023a.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18456-18466, 2023b.
Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijic Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo. Magictime: Time-lapse video generation models as metamorphic simulators. arXiv preprint arXiv:2404.05000 2024.
Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li. Make pixels dance: High-dynamic video generation. arXiv preprint arXiv:2311.1090 2023.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023.
Wangbo Zhao, Yizeng Han, Jiasheng Tang, Kai Wang, Yibing Song, Gao Huang, Fan Wang, and Yang You. Dynamic diffusion transformer. arXiv preprint arXiv:2410.034566 2024.
Shangchen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale-a-video: Temsstrllconsistentt diffusion model for real-world video super-resolution. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2535-2545, 2024.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy. Celebv-hq: large-scale video facial attributes dataset. In European conference on computer vision, pp. 650-667. Springer, 2022.
13

---

Published as a conference paper at ICLR 2025
A MORE IMPLEMENTATION DETAILS
A.1 DATA PROCESSING PIPELINE
Aesthetics and Clarity Assessment. We adopted the LAION Aesthetic Predictor and DOVER (Wu al., 2023) to separately assess aesthetic and clarity scores due their fast inference speeds and alignment with human preferences. These qualities make them efficient and well-suited for integration into our pipeline for processing million-level video data.
Temporal Consistency. Extracting CLIP representations has proven effective for computing cosine similarity between images. We calculate the CLIP similarity (Radford et al.) (021) between every two adjacent frames in the video and take the average as an indicator of the temporal consistency, measuring the coherence and consistency of the video frames.
Motion Difference. To measure motion amplitude, we utilize UniMatch (Xu al. 2023), , pretrained state-of-the-an optical flow estimation method that is both efficient and accurate. We calculate the flow score between adjacent frames of the video, taking squared average of the predicted values to represent motion dynamics, where higher values indicate stronger motion effects.
Clip Extraction. Our observations reveal that fade-ins and fade-outs between consecutive scenes often go undetected when using single cut detector with a fixed threshold. To address this, we employ cascade of three detectors (Blattmann al. 2023). each operating at different thresholds. This approach effectively captures sudden changes, fade-ins, inse-ins,,a and fade-outs in videos.
Filtering Ratios. We randomly sampled subset from the collected raw data and processed it through our data processing pipeline. A panel evaluators was then tasked with assessing these video subsets, determining whether the videos at each processing step met our requirements. Based on their preferences, we derived score thresholds and filtering ratios for each step after multiple evaluations. Figure 8 provides visualizations of the videos with varying clarity, aesthetic, motion, and temporal consistency scores computed by our pipeline.
A.2 DIFFERENCES BETWEEN MVDIT AND MMDIT
Multi-Modal Self-Attention Module. We design a Multi-Modal Self-Attention (MMSA) module based the self-attention module of MMDiT. To handle video data, repeat the text tokens T times and then concatenate the text tokens with video frame tokens using the same method as MMDiT. The self-attention operation is conducted along spatial and within the same frame. This provides simple yet effective adaptation of MMDiT to video data input.
Multi-Modal aappoorl--Ateettiinn Module. Since MMDiT lacks the ability to generate videos, we introduce a novel Mudti-Modal Temporal-Attente (MMTA) module on top the MMSA module to efficiently capture temporal information in video data. To retain the advantages of the duall-banch structure in MMSA, we employ a similar approach in MMTA, incorporating a temporal attention layer to facilitate communication along the temporal dimension.
Multi-Heas Cross-Attention Module. Since the absence of semantic information may impair video generation performance, explicitly embedding semantic information from text tokens into visual tokens is helpful. To address this, we introduce a novel cross attention layer enable direct communication between text and visual tokens.
B ABLATIONS ON DATA PROCESSING STEPS
We discuss effectiveness of each data processing step in OpenVid-1M, shown in Table 1) Tem- poral sereening improves Clip temp_ sccpssscreeeeeee and warping error, enhancing temporal consistency. 2) Screening for aesthetics, temporal and motion boosts VQA VQAT, and Slip_bleu suggesting better aesthetics and text understanding in generated videos. 3) Screening for clarity significantly improves VQAA and VQAT. Combining all four steps yields the highest scores in most metries.
14

---

Published as a conference paper at ICLR 2025
R15
111
24
(b)
(c)
Figure 8: Visualizations of the videos with varying (a) clarity, (b) aesthetic, (c) motion, and (d)
temporal consistency scores.
Table 9: Ablation studies on the effectiveness of each data processing step. The number of training data (0.6M), training iterations (50K) and resolution (256x256) each setting are kept the same.
Settings
Aesthetics
Temporal
Motion
Clurity
VQAxT
VQArt
Blip_bect bleut
_c_scoref
_emp_sccre??
Warping_errorl
20.40
10.90
23.31
67.57
99.73
0.0113
16.78
9.39
23.91
67.44
99.58
0.0217
20.32
11,42
24.43
67.62
99.71
0.0123
30.26
14.05
23.43
67.66
99.81
0.0081
C ACCELERATION FOR HD VIDEO GENERATION
Diffusion models, though powerful, often suffer from high computational costs and slow inference, especially for high-definition video generation. This is due to the sequential denoising process and attention computation, which has an (L*)) complexity based on token length L. Inspired by Ma al. 2024b), we observed significant temporal consistency in attention values between consecutive steps of the reverse denoising steps (Figure 99. revealing redundancy. These values can be cached and reused to accelerate denoising without retraining. Specifically, at timestep f, attention values are computed normally. At l 1, cached values for Self-Attention, Temporal-Attention, Cross-Attention. and Feedforward layers are reused, repeating this process every two steps. As shown in Figure g this method achieves up to 1.7x speedup at 1024 resolution, with minimal quality impact. This indicates that diffusion models trained on OpenVidHD-0 4M can be accelerated efficiently without compromising quality.
D EXAMPLES OF OpenVid-1M DATASET
In Figure 10 we visualize some samples from our OpenVid-1M. We randomly select samples with 512 512 and 1920 x 1080 resolution, respectively. With well designed data processing pipeline, 0penVid-1M demonstrates superior quality and descriptive richness, particularly in aesthet- ics, motion, temporal consistency, caption length and clarity as well.
15

---

Published as conference paper at ICLR 2025
Self-Attention
Temporal Attention
Timestap
Timestap
1.7x speedup
Cross-Aitention
Feedforward
Tinestep
Triestep
"a hible girl stands in park holding out her hand she foods a fluck of birds flutering ber.""
"adog pogples boor.
Timeslep
Timeslep
Figure 9: Left: Similarity for different attention values at different timesteps. Right: We compare the generation quality between accelerated model and original model at 1024 resolution.
The sidea fistuer a aue daré hair and brard worritg red stirasda Madd balany nicrophone the word Foor af fnaidle rill scresn auaiong colerii anhertrenent...
brotiisaing oriil sine nes countise of nourt The sas descosiing bekisd Mo recks, cumte varns pulácc zior contrum busstiffllly with slop wxsts risic fiutly, crstt agaiisl jisgesi reck...
standlng
vostle
fhdf white
olmuans, snitmpp) nua the the laks: 1he lake encirced
girafle. The giratl with long, reavliigg inwands Aurver facing
buk
nenords right ride of the some bead twrad sligkly smanndh
The vidno expsaer segiestle broupy of entrerateend aesustii rauge ot supset the siñeo ls Aooue tranysil with for natanal feoagy of landecupe mexere segle frves disnerr allwwing the vieser rolr coppe of avoetui raage Tho raloss video are vilawe sncl vie pric and seunge book savst ontraniing beuntifoll upsior aur,, smanall altuning poriape sstaee's grandur...
Mls counter calened wambbr fratt, regntablzz, and buttles chat sls sle conlingg hullnge Jie oudoor lavi and ove and plast sarroeding the bit anoa. The ronar be cuidle of cooking
An proporing food rundor Eischon
varinur ingresionts
handllog lone and catting board.
Figure 10: Examples of OpenVid- dsataset.
E MORE TEXT-TO-VIDEO EXAMPLES
We present more visual results of our model. As depicted in Figure the first column illustrates our model's proficiency in generating aesthetically pleasing content with a painting style. The second column showcases the superior text alignment of the videos generated by our model, accurately depicting 'crashed down' from the text. The third and fourth columns highlight our model's ability to produce intricate dynamics and motions, e.g., 'motorcycle race" and 'gallop across'.
VIDEO DURATIONS COMPARISON WITH OTHER DATASETS
We present video durations comparison between our OpenVid d-1M and other million level text-to-video datasets in Figure 12 Specifically, OpenVid-1M consists of 1.1199577 clips, averaging 7.2 seconds each, with a total video length of 2,051 hours. Compared to previous million-level datasets, WebVid- 10M contains low-quality videos with watermarks and Panda-70M contains many still, flickering, blurry videos along with short captions. contrast, our OpenVid-1e contains high-quality, clean videos with dense and expressive captions.
16

---

Published as a conference paper at ICLR 2025
"Unicorn sliding on a rainbow." "
"A snow avalanche crashed down mountain peak, causing destruction and mayhem."
"A motorcycle race through the city streets night." "
"Three horses gallop across wide open field, tails and manes flying in the wind."
Figure 11: More text-to-video showcases.
10.0s-20.5
10.0e-2..4
200
202
>205
5.pp-00.0
12%
23%
215
-0.0
21%
70
3.0e-5.tx
Durations
Randa-70M
Durations Vibbvis-10M
Figure 12: Comparisons on video durations with previous million level text-to-video datasets.
17

---

Published as a conference paper at ICLR 2025
Lavie
O.S.P. V1.1
Crafter
Video
Pika
Ours
triceratops charging down hill.
Figure 13: Visual comparison of different T2V generation models. Our model generates clearer,
more aesthetically pleasing and more detailed videos due to our high-resolution OpenVid-1M.
18

---

Published as a conference paper at ICLR 2025
Lavie
O.S.P. V1.1 V1.1
Crafter
Video
Pika
Ours
"a paranoid android freaking out and jumping into the air because it is
surrounded by colorful Easter eggs."
Figure 14: Visual comparison of different T2V generation models. Our model demonstrates strong ability on prompt understanding, accurately depicting the 'android" and 'surrounded by colorful Easter eggs' from the text.
19

---

Published as a conference paper at ICLR 2025
Lavie
O.S.P. V1.1
Video Crafter
Pika
Ours
"a pickup truck kicking up dust on a back road."
Figure 15: Visual comparison of different T2V generation models. Our model better captures the "kicking up dust', highlighting its superior motion quality.
20


Reviews:
[{"review_id": "Uhj2vZsWPQ", "reviewer": "Reviewer_zA3J", "summary": "This paper proposes a high-quality large-scale T2V dataset namely OpenVid-1M, as well as a new T2V model architecture – the MVDIT. \n\nAs main contribution, OpenVid-1M is born out of several open sourced dataset such as Panda, CelebvHQ, etc. and filtered with carefully designed pipeline followed by recaption. \n\nAs secondary contribution, the proposed MVDIT can be considered as a straight extension of MMDIT, where video and text token are jointly feed to 3 successive attention modules. The author(s) claim such design can mine structure information from visual feature and semantic information form text feature, and verify it through experiments.", "questions": "1.\tIn Figure 3 Right, it is noticed that 37% clips are less than 3s. Considering mainstream T2V  systems are more than 5s, is it valuable to keep such short clips? Can you perform an ablation study showing the impact of filtering out clips shorter than 5 seconds on model performance?\n2.\tIs the Self-Attention Module along spatial and within the same frame? I guess so. Please make it clearer. Furthermore, have you tried full 3D attention( Open Sora Plan v1.2) with self attention module?\n3.\tIn table 3, there are some competitive models appearing before submission DDL of ICLR, such as CogvideoX-5B and OpenSoraPlanV1.2. Can you add them in the SOTA list?\n4.\tIn table 4, to compare model trained by OpenVidHD with the one 4x by Panda 50M is not fair. Is it possible to select 1020P from Panda 50M to form Panda 50MHD for fair comparison?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Meticulously designed data process pipeline, which producing relatively higher quality comparing to previous datasets. There is no doubt that a publicly available million level dataset with high quality is critical for video generation task.\n\n\n2. The architecture designing of MVDIT is reasonable. Text tokens are repeated by T times to fitting the frames, which makes it natural to equally treat visual and semantic information in the self-attention and temporal-attention modules. \n\n\n3. Superior generation results among popular open sourced T2V systems. BTW, is it possible to make the trained model released?", "weaknesses": "1.\tAs a dataset paper. The proposed OpenVid-1M is somewhat weak. First, it is in fact a downstream collection of several publicly available datasets, which doesn’t provide extra videos. (Are you considering collecting new video data?)  Second，in contrast with carefully designed filtering operations, it is too crude to directly use raw LLAVA model as captioner without any comparison, since video caption is extremely important for T2V task. It is suggested to try sophisticated commercial LMMs such as GPT 4V and Gemini, or to finetune task aware open source models.\n\n2.\tThe introduced MVDIT is greatly inspired by MMDIT. It can be seen as a naturally extension to T2V task of MMDIT, which largely limit its technic novelty. It is notable in Figure 4 that this work adds Temporal-Attention and Cross-Attention layers besides Self-Attention in MMDIT,  so can you take an empirical ablation study to verify their effectiveness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tIJuK2CID8", "reviewer": "Reviewer_e1N1", "summary": "This paper introduce OpenVid-1M dataset, a new precise high-quality datasets for T2V generation. This dataset consists of about 1M videos, all with resolutions of at least 512x512, accompanied by detailed and long captions, facilitating the creation of visually compelling videos. An automated filtering and annotation pipeline is proposed to ensure high-quality of the dataset. Additionally, a new Multi-modal Video Diffusion Transformer (MVDiT) method is proposed to incorporate multi-modal information for better visual quality. Extensive experimental results verify the effectiveness of the proposed dataset and method.", "questions": "Refer to Weaknesses for more details.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tA dataset is proposed, comprising over 1 million high-resolution video clips paired with expressive language descriptions, this dataset aims to facilitate the creation of visually compelling videos. \n2.\tAn automated filtering and annotation pipeline is proposed to ensure the quality of videos. \n3.\tExtensive experimental results verify the effectiveness of the proposed method.", "weaknesses": "1.\tThe proposed automatic data cleaning pipeline seems to be a pipeline that many previous methods have commonly used in SD-3 and SVD, lacking a certain novelty.\n2.\tLack of ablation study for the proposed method, such as the effectiveness of scaling parameter α and Multi-Modal Temporal-Attention Module.\n3.\tThe video shown in Figure 6 and Figure 8 takes up very little space, making it difficult to see the details clearly. Increasing the size of the video frames or providing higher resolution versions in an appendix would be helpful.\n4.\tThe section on Acceleration for HD Video Generation seems redundant and is not the method proposed in this work. It can be placed in the appendix, leaving room for more qualitative text-to-video results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niSbF4AkiZ", "reviewer": "Reviewer_Y42k", "summary": "This paper proposes a large-scale high-quality dataset for text-to-video generation, named OpenVid-1M. Compared to existing related datasets, OpenVid-1M has the edge in high-quality videos and expressive captions. OpenVid-1M is curated from ChronoMagic, CelebvHQ, Open-Sora-plan and Panda by controlling aesthetic score, temporal consistency, motion difference, clarity assessment, clip extraction and video caption. Besides, the authors follow MMDiT and design a Multi-modal Video Diffusion Transformer (MVDiT) architecture for text-to-video generation.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors descrive the advantages of the proposed dataset clearly.\n2. There are sufficient experiments to evaluate the dataset and model.", "weaknesses": "1. Since the proposed MVDiT follows MMDiT, the authors should compare the differences between the two in detail through text descriptions or figures. For example, what modules does MVDiT retain, remove, or add from MMDiT? How do these changes more effectively cope with video data?\n2. As a work on text-to-video generation, there is no project or demo webpage to showcase the dataset or model performance. It is difficult to judge the overall quality of the video from a few frames captured from the video, so I would like to know if the author has plans to make the demo website public.\n3. It would be better to compare the video duration distribution between OpenVid-1M and other datasets in the form of Figure 3 Right, since the video duration can influence the quality of video generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zNji7Hl5Zl", "reviewer": "Reviewer_Kgx2", "summary": "This paper addresses two key issues in the text-to-video field: the lack of high-quality datasets and poor textual representations, proposing the 1M high-quality dataset OpenVid-1M. MVDiT is introduced to validate the effectiveness of the dataset.", "questions": "1. Why choose the top 20% of Panda-50M while selecting the top 90% from other datasets?\n\n2. How is optical flow used to filter videos? Why is temporal consistency used to discard the highest and lowest, while optical flow is not?\n\n3. In Table 3, is the comparison across different resolutions fair? Could super-resolution impact the quality of the original videos?\n\n4. Is the dataset publicly available?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper resolves the critical issue of the lack of high-quality datasets in the text-to-video field, significantly impacting the fine-tuning and training of pre-trained text-to-video models, especially smaller models.\n\n2. The article presents two datasets of different scales: 1M and 0.4M, with the 1M dataset having a resolution greater than 512.\n\n3. The dataset’s text is highly detailed, making it suitable for future transformer or DiT-based video training models that require long text inputs.\n\n4. The article selects high-quality data from multiple datasets and various models, demonstrating significant effort.\n\n5. Experiments training the same model on different datasets indicate that this dataset is indeed of high quality and can enhance the model's output performance.\n\n6. The article has a clear motivation, with writing that is clear and easy to understand.", "weaknesses": "I tend to rate the paper between 6 to 8 points, but I'm currently giving it a score of 6. Despite its rich experiments and significant contributions, there are still the following issues:\n\n1. For the ICLR conference, this paper lacks explanatory work, such as a clear justification for each step of the dataset filtering process. Specifically, it does not adequately explain why certain models were chosen or the rationale behind the selected filtering ratios.\n\n2. OpenVid-1M only filters and integrates existing datasets and does not include any new high-quality videos. Models trained on this dataset do not learn new knowledge.\n\n3. Using LLaVA to generate captions for videos merely indicates that the captions are longer and does not guarantee improved accuracy or richness of the descriptions compared to the originals. Models trained with such captions primarily transfer some knowledge from LLaVA rather than gaining new knowledge to achieve performance breakthroughs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)