OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generation

## Abstract

Abstract Text-to-video (T2V) generation has recently garnered significant attention thanks to the large multi-modality model Sora. However, T2V generation still faces two important challenges:
1) Lacking a precise open sourced high-quality dataset.
The previous popular video datasets, e . g .WebVid-10M and Panda-70M, are either with low quality or too large for most research institutions.
Therefore, it is challenging but crucial to collect a precise high-quality text-video pairs for T2V generation.
2) Ignoring to fully utilize textual information.
Recent T2V methods have focused on vision transformers, using a simple cross attention module for video generation, which falls short of thoroughly extracting semantic information from text prompt.
To address these issues, we introduce 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 \mathtt{OpenVid} typewriter_OpenVid - 𝟷 ⁢ 𝙼 1 𝙼 \mathtt{1M} typewriter_1 typewriter_M , a precise high-quality dataset with expressive captions.
This open-scenario dataset contains over 1 million text-video pairs, facilitating research on T2V generation.
Furthermore, we curate 433K 1080p videos from 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 \mathtt{OpenVid} typewriter_OpenVid - 𝟷 ⁢ 𝙼 1 𝙼 \mathtt{1M} typewriter_1 typewriter_M to create 𝙾𝚙𝚎𝚗𝚅𝚒𝚍𝙷𝙳 𝙾𝚙𝚎𝚗𝚅𝚒𝚍𝙷𝙳 \mathtt{OpenVidHD} typewriter_OpenVidHD - 0.4 ⁢ 𝙼 0.4 𝙼 \mathtt{0.4M} typewriter_0.4 typewriter_M , advancing high-definition video generation.
Additionally, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of mining both structure information from visual tokens and semantic information from text tokens.
Extensive experiments and ablation studies verify the superiority of 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 \mathtt{OpenVid} typewriter_OpenVid - 𝟷 ⁢ 𝙼 1 𝙼 \mathtt{1M} typewriter_1 typewriter_M over previous datasets and the effectiveness of our MVDiT.

## 1 Introduction

Text-to-video (T2V) generation, which aims to create a video sequence based on the condition of a text describing the video, is an emerging visual understanding task.
Thanks to the significant advancements of large multi-modality model Sora , T2V generation has recently garnered significant attention.
For example, based on DiT , OpenSora(^1^11https://github.com/hpcaitech/Open-Sora), OpenSoraPlan  and recent works ; utilize the collected million-scale text-video datasets to reproduce Sora.
However, these diffusion models ; ; ; ; still faces two critical challenges:
1) Lacking a precise high-quality dataset.
The previous popular video datasets, WebVid-10M and Panda-70M, are either with low quality or too large for most research institutions.
Therefore, it is challenging but crucial to collect a precise high-quality text-video pairs for T2V generation.
2) Ignoring to fully utilize textual information.
Recent T2V methods have focused on vision transformer (e.g., STDiT in OpenSora), using a simple cross attention module, which falls short of making full use of semantic information from text tokens.

In this work, we curate a new precise high-quality dataset for text-to-video generation named $\mathtt{OpenVid}$-$\mathtt{1M}$, which comprises over 1 million in-the-wild video clips, all with resolutions of at least 512$\times$512, accompanied by detailed captions (see Figure 1).
Our $\mathtt{OpenVid}$-$\mathtt{1M}$ has several characteristics:
1) Precise in quantity: Compared to WebVid-10m and Panda-70m, our $\mathtt{OpenVid}$-$\mathtt{1M}$ curates 1M video clips, so that most research institutes can download our dataset to facilitate text-to-video generation.
2) Superior in quality: $\mathtt{OpenVid}$-$\mathtt{1M}$ is strictly selected from the aspects of aesthetics, temporal consistency, motion difference, and clarity assessment, along with expressive captions.
3) High-definition in resolution: $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ is specially curated to advance research in high-definition video generation, which is the mainstream form of video presentation in the real world.
Specifically, $\mathtt{OpenVid}$-$\mathtt{1M}$ far outstrips the commonly-used WebVid-10M  in both resolution and video quality, attributable to the inclusion of low-quality videos with watermarks within WebVid-10M.
Meanwhile, Panda-70M  contains lots of low aesthetics, static, flickering, excessively dynamic or low clarity videos along with brief captions, while our $\mathtt{OpenVid}$-$\mathtt{1M}$ features high-quality videos coupled with expressive captions, achieved through the expansive multimodal model LLaVA-v1.6-34b .
Moreover, in contrast to previous high-quality datasets ; typically tailored for specific scenarios with limited video clips, our $\mathtt{OpenVid}$-$\mathtt{1M}$ stands out as a large-scale dataset designed for open scenarios, including portrait, scenic, city, metamorphic, animals etc.

To address the second challenge, we propose a Multi-modal Video Diffusion Transformer (MVDiT),
mining both structure information from visual tokens and semantic information from text tokens to improve the video quality.
Unlike previous DiT architectures ; that focus on modeling the visual content, our MVDiT features a parallel visual-text architecture to enhance the consistency between the text and generated video.
First, our model extracts visual tokens from video clips and text tokens from prompts, then concatenates them to generate the multi-modal feature.
This is processed by a multi-modal self-attention module to enhance interaction between tokens.
To capture temporal information, we propose a multi-modal temporal-attention module, which ensures semantic-structural temporal consistency and improves text-video alignment.
Additionally, a multi-head cross-attention module directly links text and visual tokens, integrating semantic information from text into visuals.

Our contribution can be summarized in three points:
1) We introduce $\mathtt{OpenVid}$-$\mathtt{1M}$, a precise high-quality dataset with expressive captions for facilitating T2V generation.
2) We verify the proposed $\mathtt{OpenVid}$-$\mathtt{1M}$dataset with two models, i.e. STDiT and our proposed MVDiT, which makes full use of semantic information from text tokens to improve quality.
3) Extensive experimental results have demonstrated the superiority of our proposed $\mathtt{OpenVid}$-$\mathtt{1M}$ and effectiveness of our MVDiT.

## 2 Related Work

Text-to-video Training Datasets.
Existing text-to-video training datasets can be categorized into two classes: Specific-scenario and open-scenario.
Specific-scenario datasets ; ; ; ; ;
typically consist of a limited number of text-video pairs collected for specific contexts.
For example, UCF-101  is a action recognition dataset which contains 101
classes and 13,320 videos in total.
Taichi-HD  contains 2,668 videos recording a single person performing
Taichi.
ChronaMagic  comprises 2,265 high-quality time-lapse videos with accompanying text descriptions.
Compared to previous specific-scenario datasets, our $\mathtt{OpenVid}$-$\mathtt{1M}$ is a million-level, high-quality dataset for open scenarios, including portrait, scenic, city, metamorphic, animals etc.
As a pioneering open-scenario T2V dataset, WebVid-10M  comprises 10.7 million text-video pairs with a total of 52K video hours. Panda-70M  collects 70 million high-resolution and semantically coherent video samples.
InternVid  proposes a scalable approach for autonomously constructing a video-text dataset using large language models, yielding
234 million video clips accompanied by text descriptions for basic video understanding tasks such as recognition and retrieval, as well as T2V generation.
However, WebVid-10M  contains low-quality videos with watermarks and Panda-70M  contains lots of static, flickering, low-clarity videos along with short captions.
Despite containing over 200 million samples, InternVid  primarily focuses on video understanding tasks. Meanwhile, during the data curation process, InternVid prioritizes aesthetics over dynamic consistency and clarity, with a resolution capped at 720p, leading to the inclusion of low-quality video clips.
In contrast, our $\mathtt{OpenVid}$-$\mathtt{1M}$ comprises over 1 million high-quality in-the-wild video clips, with 433K in 1080p resolution, accompanied by detailed captions. This precise T2V dataset boasts superior video quality and expressive captions, tailored for T2V tasks.

Text-to-video Models.
Current text-to-video generation methods can be divided into two categories from architecture design: UNet based methods ; ; ; ; ; ; , and DiT based methods ; ; .
UNet based methods have been widely studied using various forms of UNet.
Modelscope  introduces a spatio-temporal block and a multi-frame training strategy to enhance text-to-video synthesis, achieving state-of-the-art results.
VideoCrafter  investigates the feasibility of leveraging low-quality videos and synthesized high-quality images to obtain a high-quality video model.
DiT based video duffusion models have recently garnered significant attention.
Sora  revolutionizes video generation with significant advancements.
Latte  employs a simple and general video Transformer as the backbone to generate videos.
Recently, OpenSora${}^{\ref{opensora}}$, trained based on a pretrained T2I model  and a large text-to-video dataset, aims to reproduce Sora.
In contrast to the previous DiT structures, we propose a novel MVDiT that features a parallel visual-text architecture to enhance the interaction between text and visual tokens, thereby fully mining both structure information from visual tokens and semantic information from text tokens.

## 3 Curating 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 \mathtt{OpenVid} typewriter_OpenVid - 𝟷 ⁢ 𝙼 1 𝙼 \mathtt{1M} typewriter_1 typewriter_M

This section outlines the date processing steps as detailed in Table 1.
$\mathtt{OpenVid}$-$\mathtt{1M}$ is curated from ChronoMagic, CelebvHQ , Open-Sora-plan and Panda(^2^22Since we can only download 50M, we refer to this version Panda-50M in this work.).
Since Panda is much larger than the other datasets, here we primarily describe the filtering details on our downloaded Panda-50M.

Aesthetics Score.
Visual aesthetics are crucial for video content satisfaction and pleasure.
To enhance text-to-video generation, we filter out videos with low aesthetics scores using the LAION Aesthetics Predictor.
This results in a subset $S_{A}$ with the top 20$\%$ highest-scoring videos from Panda-50M.
For the other three datasets, we select the top 90$\%$ to form subset $S_{A}^{\prime}$.

Temporal Consistency.
Compared to image generation, text-to-video tasks are more challenging but offer opportunities to enhance quality by leveraging information from nearby video frames.
They require high visual quality and temporal consistency, making the problem complex.
Therefore, video clips with temporal consistency are crucial for training.
We use CLIP  to extract visual features and measure temporal consistency by studying cosine similarity between adjacent frames.
We filter out video clips with high consistency scores (almost static) and those with low scores (frequent flickering), obtaining a suitable subset $S_{T}$ from Panda-50M.
For the other datasets with good temporal consistency, no filtering is performed.

**Table 1: Data processing pipeline. The first three steps can be processed in parallel to enhance processing efficiency, while the subsequent steps are processed sequentially.**
| Pipeline | Tool | Remark |
| --- | --- | --- |
| Aesthetics score | LAION Aesthetics Predictor | Get high aesthetics score set $S_{A}$ |
| Temporal consistency | CLIP | Obtain moderate consistency set $S_{T}$ |
| Motion difference | UniMatch | Obtain moderate amplitude of motion Set $S_{M}$ |
| Intersection of qualified videos | Intersection | Obtain intersection: $S_{I}$ = $S_{A}\cap S_{T}\cap S_{M}$ |
| Clarity assessment | DOVER-Technical | Obtain clear and high-quality video set $S$ |
| Clip extraction | Cascaded Cut Detector | Split multi-scene videos: $\widetilde{S}$ = $Detector(S)$ |
| Video caption | LLaVA-v1.6-34b | Obtain long captions for the videos |

Motion Difference.
We introduce UniMatch  to evaluate optical flow score as the motion difference score to select videos with smooth movement, as the temporal consistency score alone is insufficient to filter out videos with high-speed objects that still maintain temporal consistency.
Video clips with high flow scores, indicating high-speed objects, are unsuitable for generation training.
We retain video clips with moderate optical flow scores to obtain a subset $S_{M}$ with smooth movement from Panda-50M.
For the other three datasets, we obtain a subset $S_{M}^{\prime}$.
We do not apply the rest of processing procedures to the other three datasets.
Instead, we directly calculate the intersection of $S_{A}^{\prime}$ and $S_{M}^{\prime}$ to obtain $S^{\prime}=S_{A}^{\prime}\cap S_{M}^{\prime}$, i.e., Ours-0.4M illustrated in Figure 3.

Figure: Figure 2: Video clarity distribution of $\mathtt{OpenVid}$-$\mathtt{1M}$. We also present 4 face samples to visualize the clarity differences. Faces outlined in green contour are blurry with low clarity scores, while those outlined in red contour are clearer with high clarity.
Refer to caption: x1.png

Clarity Assessment.
High-quality videos are crucial for T2V generation.
Since Panda-50M contains many blurry clips, we filter those with very low clarity, as shown in Figure 2.
Specifically, we calculate the intersection of the three sets from Panda to obtain $S_{I}=S_{A}\cap S_{T}\cap S_{M}$, resulting in aesthetically pleasing and stable videos with smooth movement.
Using the recent model DOVER , we estimate DOVER-Technical score for each clip in $S_{I}$, retaining high-quality videos with clean textures.
Finally, we keep clear, high-quality clips with the top $30\%$ highest scores  to form video set $S$.

Clip Extraction.
Beyond the aforementioned steps, some video clips may contain multiple scenes, thus we introduce the Cascaded Cut Detector  to split multi-scene clips in $S$ to achieve clip extraction, ensuring each contains only one scene.
After clip extraction, we obtain $\widetilde{S}$ from Panda-50M, i.e., Ours-0.6M illustrated in Figure 3.

Video Caption.
After obtaining the video clip set, we re-caption them using the large multimodal model, LLaVA-v1.6-34b , to create expressive descriptions.
As CelebvHQ lacks captions, we provide captions for its video clips too.
Figure 3(d) briefly compares the length of our prompts with those in Panda-50M.
Our dense prompts provide a significant advantage by offering richer semantic information.
We compile our high-quality dataset, $\mathtt{OpenVid}$-$\mathtt{1M}$ (i.e., Ours-0.6M $+$ Ours-0.4M).
Additionally, we meticulously select 1080p videos from $\mathtt{OpenVid}$-$\mathtt{1M}$ to construct $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$, aiming to advance the high-definition text-to-video generation within the community.

Figure: Figure 3: Comparisons on video statistics between $\mathtt{OpenVid}$-$\mathtt{1M}$ and Panda-50M.
Refer to caption: extracted/5770161/Figures/dataset_distribution.png

## 4 Data Processing and Statistical Comparison

Data Processing Differences against SVD.
Our data processing pipeline draws inspiration primarily from the SVD pipeline, yet several distinctions exist:
1) Visual quality evaluation: Both SVD and our $\mathtt{OpenVid}$-$\mathtt{1M}$ utilize an aesthetic predictor to retain highly aesthetic videos.
Additionally, $\mathtt{OpenVid}$-$\mathtt{1M}$ integrates the recent model DOVER  to assess the video clarity, preserving high-quality videos with clean textures.
2) Motion evaluation:
SVD utilizes the traditional Farneback optical flow method and RAFT  to estimate optical flows.
In contrast, $\mathtt{OpenVid}$-$\mathtt{1M}$ adopts a more efficient UniMatch  to achieve better optical flows, addressing not only static videos but also those with fast movements.
3) Time consistency evaluation: SVD employs clip extraction solely to prevent sudden video changes, whereas $\mathtt{OpenVid}$-$\mathtt{1M}$ supplements this by employing CLIP  to remove flicker videos.
4) Processing efficiency: SVD initially extracts video clips and then filters from a large pool, while $\mathtt{OpenVid}$-$\mathtt{1M}$ first selects high-quality videos and then extracts clips, significantly enhancing processing efficiency.
Finally, $\mathtt{OpenVid}$-$\mathtt{1M}$ will be made publicly available, while the training dataset in SVD is not.

Comparison with Panda-50M.
The statistical comparisons between $\mathtt{OpenVid}$-$\mathtt{1M}$ and Panda-50M are illustrated in Figure 3.
1) Video Aesthetics Distribution:
Our subsets Ours-0.6M and Ours-0.4M exhibit higher aesthetics scores compared to Panda-50M, suggesting superior visual quality.
2) Video Motion Distribution:
Our subsets display a higher proportion of videos with moderate motion, implying smoother and more consistent motion.
Conversely, Panda-50M appears to contain numerous videos with flickering and static scenes.
3) Video Temporal Consistency Distribution:

Figure: Figure 4: Overview of MVDiT with parallel visual-text architecture. Concatenation is indicated by ⓒ and split is indicated by ⓢ.
Refer to caption: x2.png

Our subsets exhibit a more balanced distribution of moderate temporal consistency values, whereas Panda-50M includes videos with either static or excessively dynamic motion.
4) Caption Length Distribution:
Our subsets feature significantly longer captions than Panda-50M, providing richer semantic information.
Overall, $\mathtt{OpenVid}$-$\mathtt{1M}$ demonstrates superior quality and descriptive richness, particularly in aesthetics, motion, temporal consistency, caption length and clarity as well.

## 5 Method

Inspired by MMDiT , we propose a Multi-modal Video Diffusion Transformer (MVDiT) architecture.
Shown in Figure 4, its architecture diverges from prior methods ; by emphasizing a parallel visual-text structure for mining both structure from visual tokens and semantic from text tokens.
Each MVDiT layer encompasses four steps: Initial extraction of visual and linguistic features, integration of a novel Multi-Modal Temporal-Attention module for improved temporal consistency, facilitation of interaction via Multi-Modal Self-Attention and Multi-Head Cross-Attention modules, and subsequent forwarding to the final feedforward layer.

### 5.1 Feature Extraction

Given a video clip, we adopt a pre-trained variational
autoencoder to encode input video clip into features in latent space. After being corrupted by noise, the obtained video latent is input into a 3D patch embedder to model the temporal information. Then, we add positional encodings and flatten patches of the noised video latent to a patch encoding sequence $\mathbf{X}\in\mathbb{R}^{T\times C\times HW}$. Following , We input the text prompts into the T5 large language model for conditional
feature extraction.
Then, we embed the text encoding to match the channel dimension of the visual tokens to obtain the text tokens $\hat{\mathbf{Y}}\in\mathbb{R}^{C\times L}$, where $L$ represents the length of the text tokens.
Finally, we take the text tokens and noised visual tokens as input of our MVDiT. During training, both the text encoder and visual encoder are frozen.

### 5.2 Multi-modal Video Diffusion Transformer

Multi-Modal Self-Attention Module.
We design a Multi-Modal Self-Attention (MMSA) module.
Text tokens $\hat{\mathbf{Y}}$ are repeated by $T$ times along the temporal dimension to generate $\mathbf{Y}\in\mathbb{R}^{T\times C\times L}$.
We adopt adaptive layer normalization both in text branch and visual branch to encode timestep information
into the model. Then, we concatenate the visual tokens with text tokens to generate the multi-modal feature $\mathbf{F}^{s}\in\mathbb{R}^{T\times C\times(HW+L)}$, which is input into the MMSA module containing a Self-Attention Layer (SAL). This can be formulated as:

$$ $\mathbf{F}^{s}_{\text{SAL}}=\text{SAL}(\text{Concat}(\text{AdaLN}(\mathbf{X}, \mathbf{t}_{1}),\text{AdaLN}(\mathbf{Y},\mathbf{t}_{1})))$ (1) $$

$$ $\text{AdaLN}(\mathbf{X},\mathbf{t}_{1}))=\gamma^{1}_{1}\text{LayerNorm}( \mathbf{X})+\beta^{1}_{1}.$ (2) $$

The self-attention operation is conducted to promote the interaction between visual tokens and text tokens in each frame, which can be implemented easily with matrix multiplication. Then, we split the visual tokens and text tokens from the enhanced multi-modal features. Following , we also regress dimension-wise scaling parameter $\alpha$, which is applied before residual connections within the Transformer block. It can be formulated as follows:

$$ $\mathbf{X}^{s}_{\text{SAL}},\mathbf{Y}^{s}_{\text{SAL}}=\text{Split}(\mathbf{F }^{s}_{\text{SAL}})$ (3) $$

$$ $\mathbf{X}^{s}=\mathbf{X}+\alpha^{1}_{1}\mathbf{X}^{s}_{\text{SAL}},\mathbf{Y} ^{s}=\mathbf{Y}+\alpha^{2}_{1}\mathbf{Y}^{s}_{\text{SAL}}.$ (4) $$

Multi-Modal Temporal-Attention Module. After obtaining the enhanced visual features and text features, we build a Multi-Modal Temporal-Attention(MMTA) module on the top of the MMSA to efficiently capture temporal information. Unlike the temporal attention used in previous methods ; , we consider capturing temporal information from both the text features and the visual features. Specifically, we concatenate the tokens from two branches to obtain the multi-modal feature $\mathbf{F}^{t}\in\mathbb{R}^{T\times C\times(HW+L)}$. We then input $\mathbf{F}^{t}$ into the MMTA module, where a temporal-attention layer (TAL) is used to conduct communication along the temporal dimension. This design enables the model to learn semantic-structural temporal consistency and further improve the video quality:

$$ $\mathbf{X}^{t}_{\text{TAL}},\mathbf{Y}^{t}_{\text{TAL}}=\text{Split}(\text{TAL }(\text{Concat}(\text{AdaLN}(\mathbf{X}^{s},\mathbf{t}_{2}),\text{AdaLN}( \mathbf{Y}^{s},\mathbf{t}_{2}))))$ (5) $$

$$ $\mathbf{X}^{t}=\mathbf{X}^{s}+\alpha^{1}_{2}\mathbf{X}^{t}_{\text{SAL}}, \mathbf{Y}^{t}=\mathbf{Y}^{s}+\alpha^{2}_{2}\mathbf{Y}^{t}_{\text{SAL}}.$ (6) $$

Here, temporal positional embedding is omitted for symbol simplicity. After the MMTA module, we obtain the visual tokens and text tokens with temporal coherent content.

Multi-Head
Cross-Attention Module. Although the MMSA module joins the tokens of the two modalities for the attention operation, T2V still requires an explicit process to insert semantic information from the text tokens into the visual tokens. The lack of semantic information also may hinder performance of video generation. Hence, we adopt a Cross-Attention Layer (CAL) to build communication between text tokens and visual tokens directly. To be specific, we take the flattened visual tokens
$\mathbf{X}^{t}\in\mathbb{R}^{T\times C\times HW}$ as Query and text tokens $\mathbf{Y}^{t}\in\mathbb{R}^{T\times C\times L}$ as Key and Value, and input them into a cross-attention layer. This can be formulated as:

$$ $\mathbf{X}^{c}=\text{CAL}(\mathbf{X}^{t},\mathbf{Y}^{t})+\mathbf{X}^{t}.$ (7) $$

After that, both the visual tokens and the text tokens are input into feedforward layer. Since one MVDiT layer can update both the visual tokens and text tokens, we can iteratively perform this process and achieve better video generation performance. After N iterations, the
final visual feature is used to predict the noise and covariance at time $t$.

## 6 Experiments

### 6.1 Experimental Settings

Datasets and Evaluation Metrics.
We adopt proposed $\mathtt{OpenVid}$-$\mathtt{1M}$ to train our MVDiT. $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ is used further for HD video generation. WebVid-10M and Panda-50M are adopted for dataset comparisons.
We evaluate our model on public benchmark in , which evaluates text-to-video generation model based on visual quality, text-video alignment and temporal consistency.
Specifically, we adopt aesthetic score (VQAA) and technical score(VQAT) for video quality assessment. We evaluate the alignment of input text and generated
video in two aspects, including image-video consistency (SD_score) and text-text consistency (Blip_bleu). We also evaluate temporal consistency of generated video with warping error and semantic consistency (Clip_temp_score).

Implementation Details.
We adopt PixArt-$\alpha$ for weight initialization and employ T5 model as the text encoder.
We use Adam  as our optimizer, and the learning rates is set to $2e-5$.
We sample video clips containing 16 frames at 3-frame intervals in each iteration.
Before feeding the video clips into model, we adopt random horizontal flips and random crop to augment the clips during the training stage.
All experiments are conducted on NVIDIA A100 80G GPUs.

Figure: Figure 5: Left: Comparison with Sota T2V models on VQAT, GPU type and resolution. The color of the middle dot in each circle indicates GPU type, and circle diameter represents video resolution. Middle: Curves of FVD with different number of GPUs. More GPUs accelerates the model’s convergence. Right: Curves of clip temporal score and warping error. Our T2V model typically starts to stabilize at 35K steps and achieves temporal consistency around 50K steps.
Refer to caption: extracted/5770161/Figures/gpu_comparison_FVD_score_error.png

### 6.2 Comparison with State-of-the-Art T2V Models

In this section, we evaluate our method’s performance and compare it with other models. For each model, we employ a consistent set of 700 prompts from to generate videos. Metrics from is used to evaluate the quality of generated videos.

Quantitative Evaluation.
The comparison between our method and others is summarized in Table 2 and Figure 5.
Our model achieves the highest VQAA (73.46%) and VQAT (68.58%), indicating superior video aesthetics and clarity.
Additionally, it achieves the second best Blip_bleu (23.45%), demonstrating its good ability on prompt understanding.
Overall, our model shows robust performance across various metrics, demonstrating the superiority of our $\mathtt{OpenVid}$-$\mathtt{1M}$, highlighting its effectiveness in text-to-video generation tasks.

The comparison between $\mathtt{OpenVid}$-$\mathtt{1M}$ and previous representative text-to-video training datasets is listed in Table 3. We adopt STDiT model used in OpenSora for all of the cases. In $256\times 256$ resolution, the model trained with our $\mathtt{OpenVid}$-$\mathtt{1M}$ generates the best scores across all metrics except $\rm VQA_{T}$.
This is reasonable, as the low resolution of the videos prevents showcasing the high quality of $\mathtt{OpenVid}$-$\mathtt{1M}$.
The similar conclusion can be found in $1024\times 1024$ resolution results, indicating the superiority of $\mathtt{OpenVid}$-$\mathtt{1M}$ in generating high-quality videos. Moreover, our $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ can be directly used to train high-definition (e.g., $1024\times 1024$) videos, whereas WebVid-10M cannot and Panda-50M has not yet undergone resolution- and quality-level filtering.
To compare results at $1024\times 1024$ resolution, we use $\times 4$ video super-resolution to generate $1024\times 1024$ videos from models trained on WebVid-10M and Panda-50M.
Clearly, training with our $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ yields better scores than combining other datasets with super-resolution.

**Table 2: Comparison with state-of-the-art text-to-video generation methods. The best results are marked in bold, while the second best ones are underscored.**
| Method | Resolution | Training Data | VQAA$\uparrow$ | VQAT$\uparrow$ | Blip_bleu$\uparrow$ | SD_score$\uparrow$ | Clip_temp_score$\uparrow$ | Warping_error$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Lavie | 512$\times$320 | Vimeo25M | 63.77 | 42.59 | 22.38 | 68.18 | 99.57 | 0.0089 |
| Show-1 | 576$\times$320 | WebVid-10M | 23.19 | 44.24 | 23.24 | 68.42 | 99.77 | 0.0067 |
| OpenSora-V1.1 | 512$\times$512 | Self collected-10M | 22.04 | 23.62 | 23.60 | 67.66 | 99.66 | 0.0170 |
| OpenSoraPlan-V1.1 | 512$\times$512 | Self collected-4.8M | 51.16 | 58.19 | 23.21 | 68.43 | 99.95 | 0.0026 |
| Latte | 512$\times$512 | Self collected-330K | 55.46 | 48.93 | 22.39 | 68.06 | 99.59 | 0.0203 |
| VideoCrafter | 1024$\times$576 | WebVid-10M; Laion-600M | 66.18 | 58.93 | 22.17 | 68.73 | 99.78 | 0.0295 |
| Modelscope | 1280$\times$720 | Self collected-Billions | 40.06 | 32.93 | 22.54 | 67.93 | 99.74 | 0.0162 |
| Pika | 1088$\times$612 | Unknown | 59.09 | 64.96 | 21.14 | 68.57 | 99.97 | 0.0006 |
| Ours | 1024$\times$1024 | $\mathtt{OpenVid}$-$\mathtt{1M}$ | 73.46 | 68.58 | 23.45 | 68.04 | 99.87 | 0.0052 |

**Table 3: Comparisons with previous representative text-to-video training datasets. The STDiT model used in OpenSora is adopted and kept the same for all of the cases. For fair comparison, training iterations are selected at the same step (50K) for fair comparison. All models with 256$\times$256 resolution are trained on 32 A100 GPUs for at least 14 days to reach 50K iterations.**
| Resolution | Training Data | VQAA$\uparrow$ | VQAT$\uparrow$ | Blip_bleu$\uparrow$ | SD_score$\uparrow$ | Clip_temp_score$\uparrow$ | Warping_error$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 256$\times$256 | WebVid-10M | 13.40 | 13.34 | 23.45 | 67.64 | 99.62 | 0.0138 |
| 256$\times$256 | Panda-50M | 17.08 | 9.60 | 24.06 | 67.47 | 99.60 | 0.0200 |
| 256$\times$256 | $\mathtt{OpenVid}$-$\mathtt{1M}$ (Ours) | 17.78 | 12.98 | 24.93 | 67.77 | 99.75 | 0.0134 |
| 1024$\times$1024 | WebVid-10M (4$\times$ Super-resolution) | 69.26 | 65.74 | 23.15 | 67.60 | 99.64 | 0.0137 |
| 1024$\times$1024 | Panda-50M (4$\times$ Super-resolution) | 63.25 | 53.21 | 23.60 | 67.44 | 99.57 | 0.0163 |
| 1024$\times$1024 | $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ (Ours) | 73.46 | 68.58 | 23.45 | 68.04 | 99.87 | 0.0052 |

**Table 4: Ablations on different resolutions, architectures and training data. For models trained with 256$\times$256 resolution, training iterations are selected at the similar steps for fair comparison. ‘Pretrained Weight’ means initializing with a corresponding pretrained model, e.g., ‘MVDiT-256’ indicates that the MVDiT model with 256$\times$256 resolution is used as the pretrained weight.**
| Model | Resolution | Training Data | Pretrained Weight | VQAA$\uparrow$ | VQAT$\uparrow$ | Blip_bleu$\uparrow$ | SD_score$\uparrow$ | Clip_temp_score$\uparrow$ | Warping_error$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| STDiT | 256$\times$256 | Ours-0.4M | PixArt-$\alpha$ | 11.11 | 12.46 | 24.55 | 67.96 | 99.81 | 0.0105 |
| STDiT | 512$\times$512 | Ours-0.4M | STDiT-256 | 65.15 | 59.57 | 23.73 | 68.24 | 99.80 | 0.0089 |
| MVDiT | 256$\times$256 | Ours-0.4M | PixArt-$\alpha$ | 22.39 | 14.15 | 23.72 | 67.73 | 99.71 | 0.0091 |
| MVDiT | 256$\times$256 | $\mathtt{OpenVid}$-$\mathtt{1M}$ | PixArt-$\alpha$ | 24.87 | 14.57 | 24.01 | 67.64 | 99.75 | 0.0081 |
| MVDiT | 512$\times$512 | $\mathtt{OpenVid}$-$\mathtt{1M}$ | MVDiT-256 | 66.65 | 63.96 | 24.14 | 68.31 | 99.83 | 0.0008 |

**Table 5: Ablation studies on the effectiveness of each data processing step. The number of training data (0.6M), training iterations (50K) and resolution (256$\times$256) for each setting are kept the same.**
| Settings |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Aesthetics | Temporal | Motion | Clarity | VQAA$\uparrow$ | VQAT$\uparrow$ | Blip_bleu$\uparrow$ | SD_score$\uparrow$ | Clip_temp_score$\uparrow$ | Warping_error$\downarrow$ |  |  |
| ✔ |  |  |  | 19.48 | 10.39 | 24.07 | 67.61 | 99.70 | 0.0137 |  |  |
|  | ✔ |  |  | 20.40 | 10.90 | 23.31 | 67.57 | 99.73 | 0.0113 |  |  |
|  |  | ✔ |  | 16.78 | 9.39 | 23.91 | 67.44 | 99.58 | 0.0217 |  |  |
| ✔ | ✔ | ✔ |  | 20.32 | 11.42 | 24.43 | 67.62 | 99.71 | 0.0123 |  |  |
| ✔ | ✔ | ✔ | ✔ | 30.26 | 14.05 | 23.43 | 67.66 | 99.81 | 0.0081 |  |  |

Figure: Figure 6: Visual comparison of different text-to-video generation models. Best viewed with Acrobat Reader. Click the images to play the video clips. Please zoom in for more details.

Qualitative Evaluation.
Visual comparisons are shown in Figure 6.
The first and second columns demonstrate that our model generates clearer and more aesthetically pleasing videos due to our high-resolution $\mathtt{OpenVid}$-$\mathtt{1M}$.
In the third example, our model demonstrates a strong ability on prompt understanding, accurately depicting the ‘pumpkin car’ and ‘candle in it’ from the text.
The fourth column shows our model better capturing the ‘river flowing’, highlighting its superior motion quality. Figure 5 presents a comprehensive analysis of the proposed model’s performance compared to state-of-the-art T2V models across various metrics.

### 6.3 Ablation Study

The ablation results on resolution, model architectures and training data are depicted in Table 4.
We can draw the following conclusions:
1) Higher resolution leads to better metric scores.
2)
The proposed MVDiT further improves both VQAA and VQAT compared to STDiT, indicating higher video quality and greater diversity.
3) More high-quality training data results in better metric scores.

We then discuss effectiveness of each data processing step in $\mathtt{OpenVid}$-$\mathtt{1M}$, shown in Table 5:
1)
Temporal screening improves Clip_temp_score and warping_error, enhancing temporal consistency.
2)
Screening for aesthetics, temporal and motion boosts VQAA, VQAT, and Blip_bleu, suggesting better aesthetics and text understanding in generated videos.
3)
Screening for clarity significantly improves VQAA and VQAT.
4) Combining all four steps yields the highest scores in most metrics.

## 7 Conclusion

In this work, we propose $\mathtt{OpenVid}$-$\mathtt{1M}$, a novel precise high-quality datasets for text-to-video generation.
Comprising over 1 million high-resolution video clips paired with expressive language descriptions, this dataset aims to facilitate the creation of visually compelling videos.
To ensure the inclusion of high-quality clips, we designed an automated pipeline that prioritizes aesthetics, temporal consistency, and fluid motion.
Through clarity assessment and clip extraction, each video clip contains a single clear scene. Additionally, we curate $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$, a subset of $\mathtt{OpenVid}$-$\mathtt{1M}$for advancing high-definition video generation.
Furthermore, we propose a novel Multi-modal Video Diffusion Transformer (MVDiT) capable of achieving superior visually compelling videos, making full use of our powerful dataset.
Extensive experiments and ablative analyses affirm the efficacy of $\mathtt{OpenVid}$-$\mathtt{1M}$compared to prior famous datasets, including WebVid-10M and Panda-50M.

Limitations.
Despite advancements in high-definition T2V generation, our model, like previous Sota models, also faces limitations in modeling the physical world.
It sometimes struggles with intricate dynamics and motions of natural scenes, leading to unrealistic videos.
Additionally, our MVDiT cannot yet model long-duration dynamics, posing challenges for practical applications.

## Appendix A Appendix

### A.1 Data Processing Differences against OpenSora

1) OpenSora operates sequentially, while $\mathtt{OpenVid}$-$\mathtt{1M}$ embraces concurrent multitasking to enhance processing efficiency.
2) OpenSora prioritizes scene clip extraction, while $\mathtt{OpenVid}$-$\mathtt{1M}$ first curates aesthetically pleasing, dynamically consistent and clear videos before executing clip extraction to reduce complexity.
3) OpenSora discards videos with aesthetic scores below 5.0, while $\mathtt{OpenVid}$-$\mathtt{1M}$ retains only the top 20$\%$ based on aesthetic scores to ensure high-quality video clips.

### A.2 Comparisons with Other Text-to-video Datasets.

We compare our $\mathtt{OpenVid}$-$\mathtt{1M}$ and $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ to previous datasets in Table 6. As described in Table 6, our $\mathtt{OpenVid}$-$\mathtt{1M}$ is a million-level, high-quality and open-scenario video dataset for training high-fidelity text-to-video models. Compared to previous million-level datasets, WebVid-10M contains low-quality videos with watermarks and Panda-70M contains many still, flickering, or blurry videos along with short captions. In contrast, our $\mathtt{OpenVid}$-$\mathtt{1M}$ contains high-quality, clean videos with dense and expressive captions generated by the large multimodal model LLaVA-v1.6-34b. Additionally, compared to previous high-quality datasets that are usually designed for specific scenarios with limited video clips, our $\mathtt{OpenVid}$-$\mathtt{1M}$ is a large-scale dataset for open scenarios, including portraits, scenic views, cityscapes, metamorphic content, etc.

**Table 6: Comparisons with previous text-to-video datasets. Our $\mathtt{OpenVid}$-$\mathtt{1M}$ is a million-level, high-quality and open-scenario video dataset for training high-fidelity text-to-video models.**
| Dataset | Scenario | Video clips | Aesthetics and Clarity | Resolution | Caption |
| --- | --- | --- | --- | --- | --- |
| UCF101 | Action | 13K | Low | 320$\times$240 | N/A |
| Taichi-HD | Human | 3K | Low | 256$\times$256 | N/A |
| SkyTimelapse | Sky | 35K | Medium | 640$\times$360 | N/A |
| FaceForensics++ | Face | 1K | Medium | Diverse | N/A |
| WebVid | Open | 10M | Low | 596$\times$336 | Short |
| ChronoMagic | Metamorphic | 2K | High | Diverse | Long |
| CelebvHQ | Portrait | 35K | High | 512$\times$512 | N/A |
| OpenSoraPlan-V1.0 | Open | 400K | Medium | 512$\times$512 | Long |
| Panda | Open | 70M | Medium | Diverse | Short |
| $\mathtt{OpenVid}$-$\mathtt{1M}$ (Ours) | Open | 1M | High | Diverse | Long |
| $\mathtt{OpenVidHD}$-$\mathtt{0.4M}$ (Ours) | Open | 433K | High (1080p) | 1920$\times$1080 | Long |

Figure: Figure 7: Visual comparison of open-source academic text-to-video generation models. Best viewed with Acrobat Reader. Click the images to play the video clips. Please zoom in for more details.

Figure: Figure 8: Visual comparison of closed-source commercial text-to-video generation models. Best viewed with Acrobat Reader. Click the images to play the video clips. Please zoom in for more details.

### A.3 More Visual Comparison with State-of-the-art Methods

We present more visual comparison results of our model against open-source academic and closed-source commercial text-to-video generation models in Figure 7 and Figure 8, respectively.

Comparison with open-source academic models.
As depicted in Figure 7, the first column showcases the superior text alignment of the videos generated by our model, accurately depicting “hailstones pelt" from the text.
The second column highlights our model’s ability to produce more natural-looking videos, e.g., with more realistic facial expressions.
The third column illustrates our model’s proficiency in generating aesthetically pleasing content with the accurate oil painting style.

Comparison with closed-source commercial products.
To comprehensively evaluate our dataset and model, we compare it with closed-source commercial products (i.e., Gen2 and Pika), as shown in Figure 8.
The first column shows superior text alignment ability of our model, precisely depicting the ‘coffee mug’ and ‘floating in the sky’.
The second column features a pig video, where our model achieves the most natural appearance.
The third column demonstrates our method’s ability to generate clearer and more aesthetically pleasing videos compared to Gen2 and Pika.

Figure: Figure 9: Examples of $\mathtt{OpenVid}$-$\mathtt{1M}$ dsataset. We show the expressive caption of the last video (lower right corner) as an example: ‘In the video, a woman is seen preparing food in a colorful outdoor kitchen. She is wearing a blue shirt and is standing behind a counter filled with various ingredients and utensils. The counter is adorned with a variety of fruits, vegetables, and bottles, suggesting that she is in the process of cooking or baking. The outdoor setting is lush and green, with trees and plants surrounding the kitchen area. The woman appears to be in the middle of her cooking process, as she is seen handling a bowl and a cutting board. The overall style of the video is vibrant and lively, capturing the essence of a home-cooked meal being prepared in a beautiful outdoor setting’. Best viewed with Acrobat Reader. Click the images to play the video clips. Please refer to our dataset for the original high-quality videos. Videos here are compressed.

### A.4 Examples of 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 𝙾𝚙𝚎𝚗𝚅𝚒𝚍 \mathtt{OpenVid} typewriter_OpenVid - 𝟷 ⁢ 𝙼 1 𝙼 \mathtt{1M} typewriter_1 typewriter_M dataset

In Figure 9, we visualize some samples from our $\mathtt{OpenVid}$-$\mathtt{1M}$. We randomly select samples with $512\times 512$ and $1920\times 1080$ resolution, respectively. With well designed data processing pipeline, $\mathtt{OpenVid}$-$\mathtt{1M}$ demonstrates superior quality and descriptive richness, particularly in aesthetics, motion, temporal consistency, caption length and clarity as well.

### A.5 Hosting, licensing, and maintenance plan

Our $\mathtt{OpenVid}$-$\mathtt{1M}$, which can be rapidly and efficiently
downloaded, was created to build a new and high-quality dataset for high-definition text-to-video generation to facilitate research on T2V generation. $\mathtt{OpenVid}$-$\mathtt{1M}$ is curated from Panda, ChronoMagic, Open-Sora-plan and CelebvHQ datasets.
Here we provide the [URL](https://huggingface.co/datasets/nkp37/OpenVid-1M) to website where our $\mathtt{OpenVid}$-$\mathtt{1M}$ dataset can be publicly viewed and downloaded.
The website is hosted on the widely-used website hosting service Hugging Face.
We provide introductions and links to download the datasets, as well as usage guides. Our $\mathtt{OpenVid}$-$\mathtt{1M}$ is released under CC-BY-4.0 license.
As described above, the video samples are collected from publicly available datasets.
Users must follow the related licenses mentioned in the previous datasets ([Panda](https://github.com/snap-research/Panda-70M/tree/main?tab=readme-ov-file#license-of-panda-70m), [ChronoMagic](https://github.com/PKU-YuanGroup/MagicTime?tab=readme-ov-file#-license), [Open-Sora-plan](https://github.com/PKU-YuanGroup/Open-Sora-Plan?tab=readme-ov-file#-license), CelebvHQ (Unknow)) to use these video samples.
For ongoing maintenance, we will continue to refine our $\mathtt{OpenVid}$-$\mathtt{1M}$ by filtering out videos with unsatisfactory quality.
Furthermore, we will regularly update our $\mathtt{OpenVid}$-$\mathtt{1M}$ to provide a larger and more precise version, ensuring the datasets remain accurate, up-to-date, and useful for the research community.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.	In Figure 3 Right, it is noticed that 37% clips are less than 3s. Considering mainstream T2V  systems are more than 5s, is it valuable to keep such short clips? Can you perform an ablation study showing the impact of filtering out clips shorter than 5 seconds on model performance?
2.	Is the Self-Attention Module along spatial and within the same frame? I guess so. Please make it clearer. Furthermore, have you tried full 3D attention( Open Sora Plan v1.2) with self attention module?
3.	In table 3, there are some competitive models appearing before submission DDL of ICLR, such as CogvideoX-5B and OpenSoraPlanV1.2. Can you add them in the SOTA list?
4.	In table 4, to compare model trained by OpenVidHD with the one 4x by Panda 50M is not fair. Is it possible to select 1020P from Panda 50M to form Panda 50MHD for fair comparison?
soundness: 3
strengths: 1. Meticulously designed data process pipeline, which producing relatively higher quality comparing to previous datasets. There is no doubt that a publicly available million level dataset with high quality is critical for video generation task.


2. The architecture designing of MVDIT is reasonable. Text tokens are repeated by T times to fitting the frames, which makes it natural to equally treat visual and semantic information in the self-attention and temporal-attention modules. 


3. Superior generation results among popular open sourced T2V systems. BTW, is it possible to make the trained model released?
summary: This paper proposes a high-quality large-scale T2V dataset namely OpenVid-1M, as well as a new T2V model architecture – the MVDIT. 

As main contribution, OpenVid-1M is born out of several open sourced dataset such as Panda, CelebvHQ, etc. and filtered with carefully designed pipeline followed by recaption. 

As secondary contribution, the proposed MVDIT can be considered as a straight extension of MMDIT, where video and text token are jointly feed to 3 successive attention modules. The author(s) claim such design can mine structure information from visual feature and semantic information form text feature, and verify it through experiments.
weaknesses: 1.	As a dataset paper. The proposed OpenVid-1M is somewhat weak. First, it is in fact a downstream collection of several publicly available datasets, which doesn’t provide extra videos. (Are you considering collecting new video data?)  Second，in contrast with carefully designed filtering operations, it is too crude to directly use raw LLAVA model as captioner without any comparison, since video caption is extremely important for T2V task. It is suggested to try sophisticated commercial LMMs such as GPT 4V and Gemini, or to finetune task aware open source models.

2.	The introduced MVDIT is greatly inspired by MMDIT. It can be seen as a naturally extension to T2V task of MMDIT, which largely limit its technic novelty. It is notable in Figure 4 that this work adds Temporal-Attention and Cross-Attention layers besides Self-Attention in MMDIT,  so can you take an empirical ablation study to verify their effectiveness?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Refer to Weaknesses for more details.
soundness: 3
strengths: 1.	A dataset is proposed, comprising over 1 million high-resolution video clips paired with expressive language descriptions, this dataset aims to facilitate the creation of visually compelling videos. 
2.	An automated filtering and annotation pipeline is proposed to ensure the quality of videos. 
3.	Extensive experimental results verify the effectiveness of the proposed method.
summary: This paper introduce OpenVid-1M dataset, a new precise high-quality datasets for T2V generation. This dataset consists of about 1M videos, all with resolutions of at least 512x512, accompanied by detailed and long captions, facilitating the creation of visually compelling videos. An automated filtering and annotation pipeline is proposed to ensure high-quality of the dataset. Additionally, a new Multi-modal Video Diffusion Transformer (MVDiT) method is proposed to incorporate multi-modal information for better visual quality. Extensive experimental results verify the effectiveness of the proposed dataset and method.
weaknesses: 1.	The proposed automatic data cleaning pipeline seems to be a pipeline that many previous methods have commonly used in SD-3 and SVD, lacking a certain novelty.
2.	Lack of ablation study for the proposed method, such as the effectiveness of scaling parameter α and Multi-Modal Temporal-Attention Module.
3.	The video shown in Figure 6 and Figure 8 takes up very little space, making it difficult to see the details clearly. Increasing the size of the video frames or providing higher resolution versions in an appendix would be helpful.
4.	The section on Acceleration for HD Video Generation seems redundant and is not the method proposed in this work. It can be placed in the appendix, leaving room for more qualitative text-to-video results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. The authors descrive the advantages of the proposed dataset clearly.
2. There are sufficient experiments to evaluate the dataset and model.
summary: This paper proposes a large-scale high-quality dataset for text-to-video generation, named OpenVid-1M. Compared to existing related datasets, OpenVid-1M has the edge in high-quality videos and expressive captions. OpenVid-1M is curated from ChronoMagic, CelebvHQ, Open-Sora-plan and Panda by controlling aesthetic score, temporal consistency, motion difference, clarity assessment, clip extraction and video caption. Besides, the authors follow MMDiT and design a Multi-modal Video Diffusion Transformer (MVDiT) architecture for text-to-video generation.
weaknesses: 1. Since the proposed MVDiT follows MMDiT, the authors should compare the differences between the two in detail through text descriptions or figures. For example, what modules does MVDiT retain, remove, or add from MMDiT? How do these changes more effectively cope with video data?
2. As a work on text-to-video generation, there is no project or demo webpage to showcase the dataset or model performance. It is difficult to judge the overall quality of the video from a few frames captured from the video, so I would like to know if the author has plans to make the demo website public.
3. It would be better to compare the video duration distribution between OpenVid-1M and other datasets in the form of Figure 3 Right, since the video duration can influence the quality of video generation.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Why choose the top 20% of Panda-50M while selecting the top 90% from other datasets?

2. How is optical flow used to filter videos? Why is temporal consistency used to discard the highest and lowest, while optical flow is not?

3. In Table 3, is the comparison across different resolutions fair? Could super-resolution impact the quality of the original videos?

4. Is the dataset publicly available?
soundness: 3
strengths: 1. The paper resolves the critical issue of the lack of high-quality datasets in the text-to-video field, significantly impacting the fine-tuning and training of pre-trained text-to-video models, especially smaller models.

2. The article presents two datasets of different scales: 1M and 0.4M, with the 1M dataset having a resolution greater than 512.

3. The dataset’s text is highly detailed, making it suitable for future transformer or DiT-based video training models that require long text inputs.

4. The article selects high-quality data from multiple datasets and various models, demonstrating significant effort.

5. Experiments training the same model on different datasets indicate that this dataset is indeed of high quality and can enhance the model's output performance.

6. The article has a clear motivation, with writing that is clear and easy to understand.
summary: This paper addresses two key issues in the text-to-video field: the lack of high-quality datasets and poor textual representations, proposing the 1M high-quality dataset OpenVid-1M. MVDiT is introduced to validate the effectiveness of the dataset.
weaknesses: I tend to rate the paper between 6 to 8 points, but I'm currently giving it a score of 6. Despite its rich experiments and significant contributions, there are still the following issues:

1. For the ICLR conference, this paper lacks explanatory work, such as a clear justification for each step of the dataset filtering process. Specifically, it does not adequately explain why certain models were chosen or the rationale behind the selected filtering ratios.

2. OpenVid-1M only filters and integrates existing datasets and does not include any new high-quality videos. Models trained on this dataset do not learn new knowledge.

3. Using LLaVA to generate captions for videos merely indicates that the captions are longer and does not guarantee improved accuracy or richness of the descriptions compared to the originals. Models trained with such captions primarily transfer some knowledge from LLaVA rather than gaining new knowledge to achieve performance breakthroughs.

Meta-review: This paper proposes a high-quality large-scale text-to-video (T2V) dataset called OpenVid-1M, alongside a T2V  model MVDiT based on MMDiT. Reviewers acknowledged the contributions of the dataset, the model design, good results and the open source. The initial major concerns include no new videos added to existing data, and  the lack of ablations on video captioning models and model components. The rebuttal addressed most of these concerns, and the four reviewers unanimously provided positive scores (8866). Therefore, the AC recommends acceptance.

Decision: Accept (Poster)