GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Authors: Zhun Mou
Bin Xia
Zhengchao Huang
Wenming Yang
Jiaya Jia
Venue: ICML 2025 poster (2025)
Keywords: Text-to-Video Generation
Evaluation
MLLMs
Primary area: deep_learning
TL;DR: We introduce GRADEO-Instruct and GRADEO, a dataset and model for evaluating AI-generated videos through multi-step reasoning, aligning better with human evaluations.

Abstract: Recent great advances in video generation models have demonstrated their potential to produce high-quality videos, bringing challenges to effective evaluation. Unlike human evaluation, existing automated evaluation metrics lack high-level semantic understanding and reasoning capabilities for video, thus making them infeasible and unexplainable. To fill this gap, we curate **GRADEO-Instruct**, a multi-dimensional T2V evaluation instruction tuning dataset, including 3.3k videos from over 10 existing video generation models and multi-step reasoning assessments converted by 16k human annotations. We then introduce **GRADEO**, one of the first specifically designed video evaluation models, which **grades** AI-generated **videos** for explainable scores and assessments through multi-step reasoning. Experiments show that our method aligns better with human evaluations than existing methods. Furthermore, our benchmarking reveals that current video generation models struggle to produce content that aligns with human reasoning and complex real-world scenarios. The models, datasets, and codes will be released soon.

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Zhun Mou'
Bin Xia
Zhengchao Huang'
Wenming Yang'
Jiaya Jia
Abstract
Question: Can you evaluate the creativity of this video (Rate it in 1-5, means low creafivity, 5 means high creativity)
Recent great advances in video generation mod- els have demonstrated their potential to produce high-quality videos, bringing challenges to effec- tive evaluation. Unlike human evaluation, ex- isting automated evaluation metrics lack high- level semantic understanding and reasoning ca- pabilities for video, thus making them infeasible and unexplainable. To fill this gap, we curate GRADEO-Instruct, multi-dimensional T2V evaluation instruction tuning dataset, including 3.3k videos from over 10 existing video genera- tion models and multi-step reasoning assessments converted by 16k human annotations. We then introduce GRADEO, one of the first specifically designed video evaluation models, which grades Al-generated videos for explainable scores and assessments through multi-step reasoning. Exper- iments show that our method aligns better with human evaluations than existing methods. Fur- thermore, benchmarking reveals that current video generation models struggle to produce con- tent that aligns with human reasoning and com- plex real-world scenarios.
Widdii video portrays an interns. featuring contrantation
Casersttd Video
10)
lange robots with another cantra robet.
4
exhibitt by bleading dagres compet-tive themes is stuully captiwsting way.
Exisling Metrics Directly Give Score (Scaled to 1-37
Overview
Descriptinn tion
Analysia
Assesement
GRADEO
Figure Traditional evaluation methods, limited by small datasets and model parameters, suffer from three key issues: (1) inability to accurately understand video content, (2) lack of explainability with only score outputs, and (3) focus on low-level features like video quality, neglecting high- level aspects such as rationality, safety and creativity, We propose GRADEO, novel approach that leverages human- like reasoning for comprehensive video evaluation, enabling accurate and interpretable assessments.
completely wrong videos. (Huang al., 2024; Liu al., 2024b;c; Kou et al., 2024; He et al., 2024; Bansal et al., 2024) Thus, there is a significant demand for automated frameworks that can objectively, accurately, and efficiently evaluate T2V models across diverse test sets, facilitating continuous performance and quality improvements.
1. Introduction
Recent breakthroughs in Text-to-Videe (T2V) generation models (Zhang et al., 2024a; Guo et al., 2024; Wang al., 2024b; Zheng et al., 2024; OpenAI, 2024) have led to remarkable advancements in the field, demonstrating the potential to generate long-duration and realistic videos. However, existing models still generate hallucinated con- tent, videos that do not align with real-world scenarios, or
Early T2V generation evaluation method (Chivileva et al., 2023) introduces the T2V-CL framework, which focuses on image naturalness and text similarity but oversimplifies the evaluation. It also highlights the limitations of commonly used video quality metrics, including FVD (Unterthiner et al., 2018), CLIPSim (Wu et al., 2021), and IS (Salimans et al., 2016), and emphasizes that these metrics fail to elim- inate the need for human assessment. While human pref- erence remains the gold standard, it is not scalable to meet the growing demand for Al- Al-generated video content. To automatically and comprehensively evaluate Al-generated videos, EvalCrafter (Liu et al., 2024b) and Vbench (Huang et al., 2024) propose detailed evaluation frameworks with hierarchical dimensions and automatic metrics, validated by human evaluations. On the other hand, some studies (Wu
'Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China 'CSE department, The Chinese Uni- versity of Hong Kong, Hong Kong,China Depaatment of Com- puter Science and Engineering. The Hong Kong University of Science and Technology, Hong Kong, China. Correspondence to: Wenming Yang <yang. wenningennnngee @ sz, www.inguaacuu.cn.......
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

GRADEO: Towards Human-Like Evaluation for Text-to-Videe Video Generation via Multi-Step Reasoning
et al., 2024; Zhang al., 2024e; Qu al., 2024) attempt to model human evaluation using large-scale datasets that mimic human scores, aiming for unified and simplified evaluation method.
tion models, and asked human evaluators to score them ac- cording to the evaluation criteria for each dimension, while also annotating the rationale. Using GPT-4o, we converted the human-annotatee rationale and scores into formatted responses. Finally, we obtained a 3.3k video evaluation instruction tuning dataset GRADEO-Instruct Training on this, we develop GRADEO which exhibits superior correlation with human assessments, thus acting as a more effective model for evaluating T2V models. We create prompt suite for the seven dimensions, each containing 100 text prompts for video generation, and then evaluate several popular video generation models.
Nevertheless, critical challenges persist in the evaluation of text-to-video generation: First, metrics in multi-dimensional evaluation frameworks often rely on pre-trained mod- els (Caron et al., 2021; Radford et al., 2021; Wu et al., 2023a) trained on large-scale real-world image and video datasets. While these models capture general visual features and reflect human preferences for real-world content, they struggle to represent the distributions of generated videos, limiting their ability to assess generated content accurately. Second, traditional non-LLLMs evaluation methods, con- strained by limited datasets and model parameters, lack the capability to accurately comprehend video content. As result, they primarily focus on low-level perceptual aspects like visual quality and fail to evaluate high-level semantic dimensions, such as toxicity, alignment with real-world sce- narios, and creativity. Several research (Bansal et al., 2024; Meng et al., 2024; Miao et al., 2024) focus on evaluating the physical commonsense or safety, but their methods are not scalable and lack comprehensive evaluation. Third, existing metrics, including SOTA method VideoScore (Kou et al., 2024; Lin et al., 2024b; He et al., 2024), provide only scores (cf. Figure 1) without offering explanations for the rationale behind the scoring, resulting in poor interpretability. Fourth, Multimodal Large Language Models (MLLMs) (OpenAI, 2023; Liu al., 2023; Team al., 2023), despite their poten- tial comprehend video content evaluate videos based on given instructions, are not specifically trained for video evaluation and lack alignment with human preferences. As result, they struggle to provide accurate and comprehensive assessments of generated videos.
To summarize, our work provides contributions as follows:
We curate diverse dataset called GRADEO-Instruct one of the first instruction tuning datasets that fo- cuses on simulating human evaluation of Al-generated videos.
We develop a novel comprehensive and automated video evaluation model, GRADEO, based on our dataset. It can assess videos via multi-step reasoning, demonstrating superior alignment with human evalu- ation. Moreover, compared to previous methods, our approach provides score rationales, enhancing model interpretability and human understanding of its deci- sions.
We benchmark recent T2V generation models, provid- ing new findings and insights that contribute to the advancement of the video generation field.
2. Related Work
Text-to-Video Generative Models. In recent years, dif- fusion models (Sohl-Dickstein al., 2015; Song et al., 2021; Ho et al., 2020) have emerged as the primary meth- ods for text-to-video generation. The early video diffu- sion model VDM (Ho et al., 2022b) extends the architec- ture to a 3D U-Net, enabling joint training on both image and video data. Imagen Video (Ho et al., 2022a). Video LDM (Blattmann al., 2023), LaVie (Wang et al., 2024b), and Show-1 (Zhang et al., 2024a) leverage multi-stage mod- els for high-resolution video generation. Some training-fre methods (Khachatryan et al., 2023; Zhang et al., 2024d) modify only the inference process to generate videos but face issues with frame inconsistency. AnimateDiff (Guo et al., 2024) trains a plug-and-pla: motion module that can be integrated into any text-to-image model, enabling broad community applications. After the emergence of Sora (OpenAI, 2024), many models pre-trained on large- scale video datasets have been made open-source, includ- ing Open-sora (Zheng et al., 2024), Open-sora-plan (Lin et al., 2024a), and Mochi-1 (Team, 2024). Despite their im- pressive generation performance, closed-souree commercial
To more accurately understand the content of generated videos and provide more interpretable evaluations across diverse dimensions of generated videos, especially high- level semantic dimensions, we propose a unified and inter- pretable evaluation method for Al-generated videos that sim- ulates human-like assessment called GRADEO: Generates Reasoning- based Assessments, considering multiple diver- sity dimensions for evaluating text-to-viDEO models based on video semantic understanding. To accomplish this goal, we first introduce data construction pipeline that collects triples
from human annotators to create video-instruction data, which teaches MI.LMs rea- soningly assessing specific aspects of generative videos. Specifically, we selecte seven dimensions, ranging from low-level visual quality perception to high-level semantic understanding and visual reasoning, and meticulously de- sign more granular key aspects and evaluation criteria for each dimension. Then, we collected generated videos from several open-source and closed-souree T2V video genera-
2

---

GRADEO: Towards Human-Like Evaluation for Text-to- Video Generation via Multi-Step Reasoning
Dataset Construction
Evaluation Process
Creatisity
Prompt Collection
Safety
You are an expert in evaluating AT-generated videos. Dimension: quality, Criteria: ...0-5)
Vittench
Webvid
EvstCrafferr
VidProl
Ponudll wiih
Rationality
Alignment
Consistency
Description Oweeee
.. I will begin with detailed description - conduct en andlyste... .. aspecti to such as clarity, artifacls, texturw/detal, and natursiness provide final assessment score
Ausihelic
Quality
Final Noemanent
Video Collection
shows a park-ilke setting with green grese and frees, There are Geveral signs with Text, but the hedl not clear and appears to be distorted or blurred..
R
Show
Plica
The blurring and distertion of the text on the signs make it rearly impossible to read the presence of noticsable or artifacts evident.. derupt the harmony between the background and the main subjects- These Flaws collective impact the viewing experienca, making below overoge
CogVideo
Analyaa
Human Annotation
Generated Video
You are an expert in evaluating At-generated ted videos. Dimension: rationality, Criteria: ...(1-5) Prompt: seeing snow for first time
Score
GRADEO
Theme: Emational Responses
Mew NiS
will first andlyze the given prompt infer logical Minteen ONeNee
Bigung
du culcomes based on human commoo sense conduct a failed onalyss .. provide on assessment score...
Reason
Aad
Finl Nessement
When a child sees snow for the first time, what con we logicolly expectt Snow a new and Intrgging phenomenon for someone unfumiliar with ih... The child's seyss night widen warder, and their Face could break into Joyful grin..
LLMs Summary
Upon revicwing the video, we observe the child's reaction to teeing snow for the first time This aligns well with our expectation of a navel sersory experience provoking wonder .. the child's Face breaks into a joyful gris, reflecting the happiness and excitement These visual cues are consistent with the inferred emolional outcooes...
Descrrpotion
Anetyyls
3
Overviee
Analyye
GRADEO-
Generoted Video
Instruct
Figure 2: An overview of GRADEO. D)Data Construction Pipeline. First, we source (prompt, video) data, and collect human annotations. Then, we convert them to instruction tuning datasets. b)Evaluation Process Pipeline. GRADEO generates assessment score after multi-step reasoning.
models (OpenAI, 2024; Runway, 2024; Lab, 2024; Kling, 2024; Zhipu, 2024) exhibit significantly stronger perfor- mance. However, these models fail to align with human evaluations in areas such as simulating real-world scenarios and ensuring safety. To address these limitations, we incor- porate multimodal reasoning large models to assess these dimensions more effectively.
ol (Xu et al., 2024; Zhao et al., 2024) integrate CoT into the model training process, utilizing organized tags to enable a systematic and structured reasoning process. Inspired by these work, we aim to explore the potential of reasoning MLLMs in text-to-video evaluation.
Text-to-Vidd Generation Benchmarks. With text-to- video models show exceptional performance on high res- olution video generation, there is increasing interest in evaluating text-to-video models. Fréchet Video Distance (FVD) (Unterthiner et al., 2018), Inception Score (IS) (Sali- mans et al., 2016) and CLIP-Similarity (Radford et al., 2021) are most commonly automatic video quality assessment met- rics. However, there are concerns about correlation between them and the evaluation of humans. VBench (Huang et al., 2024) introduces multi- dimensional and nulti-categorical benchmark suite for evaluating video generation model per- formance. EvalCrafter (Liu et al., 2024b) construct their metrics by human aligmment. Some recent studies(Bannal et al., 2024: Sun et al., 2024: Meng et al., 2024) prompt MLLMS to assess video quality or other performance di- mension of generative models. VideoScore (He et al., 2024) shows models that were not trained on video evaluation task did not correlate well with human assessments. Evaluation Agent (Zhang et al., 2024b) introduces a dynamic, agent- based framework with hierarchical assessments but raises fairness concerns in evaluating diverse models while lacks
Multimodal LLMs Reasoning. Built upon Large Lan- guage Models (LLMs) (OpenAl, 2022; Achiam et al., 2023; Bai et al., 2023; Touvron et al., 2023a;b) and vision en- coders (Radford et al., 2021; He et al., 2022), Multimodal Large Language Models (MLLMs) (OpenAI, 2023; Liu et al., 2023; Team al., 2023) exhibit artificial general intelligence, excelling in open-world vision tasks such as understanding, instruction following, and conversational abilities. In-Context Learning (ICL) (Brown et al., 2020) enable LLMS excellent reasoning capability by provid- ing samples and context. Additionally, Chain-of-Though (CoT) (Wei et al., 2022) further strengthens multi-step rea- soning by generating intermediate reasoning steps to solve complex problems. Extensions of CoT, such Multimodal- CoT (Zhang et al., 2023), VCoT (Rose et al., 2023), and Visual CoT (Shao et al., 2024), explore visual reasoning in Multimodal Large Language Models (MLLMs). Recent works (Hao et al., 2024; Wang et al., 2024c) apply CoT to the video understanding domain. I.Lava-ool CooT and Marco-
3

---

GRADEO: Towards Human-Like Evaluation for Text-to-Videe Generation via Multi-Step Reasoning
human annotations. We propose a comprehensive bench- mark suite including seven dimensions from video low-level perception to high-level video semantic understanding and common sense reasoning. Also, we introduce an automatic model for fine-grained evaluating text-to-video models.
Gwn
0.5
1-Port
SoAvenge
Ratio
S.Perlect
0.3
3. GRADEO
0.1
In this section, we introduce our framework to teach MLLMs to evaluate Al-generated videos. In Sec. 3.1, we first elaborate seven evaluation dimensions, as well as each dimension's 's key aspects we focus on. We then detail the process of dataset construction pipeline, and propose our GRADEO-Instruct dataset which we plan to release in Sec. 3.2. Finally, we describe how to build GRADEO and how the evaluator works in the evaluation process (See Sec. 3.3).
0.0
Quality
Aesthetic Cossidency/lggumee Retionality
Safery
Crentivity
Human score distribution for datasets across di- mensions in GRADEO-Instruce
Figure 3:
ing and sampling from large-scale text-to-video pretrain- ing dataset WebVid (Bain et al., 2021), real user- geer-geneatted prompt dataset VidProM (Wang & Yang. 2024), and existing open-source text-to-video benchmark datasets (Liu et al., 2024b; Huang et al., 2024; Sun al., 2024; Feng al., 2024; Liu et al., 2024c; Kou et al., 2024). Additionally, the Safety dimension prompts are curated from specialized safety-focueed dataset SafeSora (Dai et al., 2024). Inspired by recent studies (Bansal et al., 2024; Fu et al., 2024; Meng et al., 2024) on evaluating text-to-visual generation based on physical laws, we define thematic categories (e.g.. Physical Laws, Emotional Responses, Weather Conditions, Chemical Laws, more themes and prompt examples in Appendix A.2) that encompass human commonsense knowledge and manually craft multiple examples for each category in Ratio- nality dimension. These examples guide the GPT-assisted generation of additional prompts. For Creativity dimension, we collect prompts from the video generation community to ensure diverse and imaginative inputs, while also generating additional creative prompts based community examples. At the same time, we include simpler, less creative prompts (e.g., a bird flying in the sky, a child playing with a ball) to maintain a balanced dataset.
3.1. GRADEO Evaluation Dimensions
Table I: Key Aspects of Evaluation Dimensions.
Dimension
Key Aspects
Quality
clarity, artifacts, naturalness, testure and detaal
Aesthttic
content, composition, color, lighting
Consistency
suhjort, background
Aligament
object, count, color, style, spatial, temporal, action, camera
Rationality
human
Safery
illegality. hommfuluess offensieeness, discrimination, misiafomuution, sansitivity
Creativity
divensity, narrativity monotomy
Existing methods (Liu et al., 2024b; Huang et al., 2024; Kou et al., 2024) typically focus on assessing low-level dimensions, primarily video quality, with relatively less attention given to higher-level semantic evaluations. In addition to retaining the primary dimensions of assessment from previous methods, we include evaluations on whether the video complies with human common sense, whether it is safe, and whether it possesses creativity. As a result, we develop a bottom-up benchmark that spans from low-level visual perception to high-level visual understanding and reasoning. As shown in Table 1, we also define key aspects for each dimension with fine granularity. Further details are provided in Appendix A.I.
Video Collection For the dimensions of Quality, Aes- thetics, Consistency, Alignment, and Rationality, we leverage range of open-source models, including Open- Sora (Zheng et al.,, 2024), VideoCrafter-1 (Chen al., 2023), VideoCrafter-2 (Chen et al., 2024a), Latte (Ma et al., 2024), La (Wang et al., 2024b), and ZeroScope-576w (Cer- spense, 2024), to generate videos. Moreover, we collect pre-generated videos from the VidProM dataset, ensuring a diverse set of examples. For the Safety dimension, we sam- ple malicious videos from the SafeSora dataset and integrate benign videos from VidProM. For the Creativity dimension, we use the Kling 1.0 and Kling 1.5 models (Kling, 2024) to generate videos with varying creativity levels, enhancing the diversity of the dataset. Finally, We filter out videos with unsuitable dynamics to ensure they meet the required motion quality, and crop those with watermarks to elimi- nate model-specific characteristics. Details are shown in
3.2. GRADEO-Instruct Dataset Construction
We first introduce the dimensional prompt sources, followed by the model used to generate the video. Next, we obtain the triples (video, rationale, score) from human annotations, and finally, we convert this data into a CoT instruction tuning dataset, GRADEO-Instruct.
Prompt Collection To construct a comprehensive prompt dataset for text-to-video evaluation, we ensured broad cov- erage across the dimensions of Quality. Aesthetics, Con- sistency, and Aligument. This was achieved by collect-
4

---

GRADEO: Towards Human-Like Evaluation for Wext-to-Video Generation via Multi-Step Reasoning
Appendix A.2.
Referring to Figure 2, GRADEO divides the an- swer into four parts: <Overview>, <Descriptions, <Analysis>, <Ässessment>, In <Overview> stage, based on the questions and the evaluation criteria for each dimension, it plans the reasoning steps to reach the final score assessment. For most dimension, it generates video description in inscriptions poses the prompt text into several key aspects for alignment dimension, and engages in human common-sense reason- ing for rationality in this stage. Then, in <Analysis> incorporating the information obtained previously and the video itself, it conducts a thorough analysis and thoughtful reasoning in accordance with the assessment criteria for this dimension. <Assessments stage, it assesses the final score based on the previous reasoning process.
Human Annotation We establish evaluation criteria for each dimension as outlined in Sec. 3.1. Five annotators as- sess scores for each dimension, which is rated on a 1-5 scale. Inspired by the slow thinking process in human evaluation (cf. Figure 1), which involves multiple steps of reasoning and analysis, for ensuring the accuracy and interpretability of video as- sessments. In addition to assigning scores (cf. Figure 3), annotators must provide rationales for their assessments, ensuring a thorough evaluation. To maintain consistency and reliability, we require that more than 50% of annotators agree on the score, with a maximum 2-point difference be- tween scores. For each dimension, annotators are instructed to focus on key aspects and provide detailed rationales.
Instead, it decom-
recognize that this approach is essential
To evaluation text-to-video model G, outside the dataset GRADEO-Instruct, we generate 100 prompts for each dimension we represent the symbols as (Pi} E D, totaling 700 prompts, as the benchmark prompt suite GRADEO- BENCH. See Appendix D.1 for further elaboration.
Extend CoT Response To create an instruction tuning dataset that enables multimodal large models to learn human evaluation of Al-generated videos, we define an assessing ohain-offThhughh process with four steps: <Overview>, <Descrip on>, <Analysiss and <Assessment>> The overview step involves planning the overall reasoning process for the subsequent stages. The description step provides detailed descriptions of the video content. The analysis step emulates the slow thinking and reasoning process in human evaluation, systematically elab- orating the rationale behind the scores. Finally, the assessment step generates the evaluation score based on the reasoning and analysis. We designed a Gss-tssisted CoT response construction pipeline, where we feed three keyframes from the video, evaluation criteria, and human annotators' assessment rationales into GPT-4o (Hurst et al., 2024) to generate CoT data consisting of four steps. If the generated score in the evaluation step does not align with the human-assigned score, we discard the CoT data and regenerate it. For chains with correct scores, we manually verified that the reasoning steps were consistent with human logic.
4. Experiments
4.1. Experimental Setup
Datasets and Metrics. To thoroughly assess the corre- lation between model predictions and human evaluations, we perform experimental comparisons on both the test set and the constructed pairwise dataset. We extract 340 sam- ples from the collected dataset GRADEO to serve as the test set. On this dataset, we employ Spearman's rank correlation coefficient (SROCC), Pearson's 's linear correlation coefficient (PLCC), and Mean Absolute Er- ror (Á) test metrics, calculating each dimension sepa- rately. SROCC evaluates rank correlation, while PLCC as- sesses the linear relationship, with higher values indicating stronger alignment with human evaluations, and a lower A signifying a closer proximity between predicted and actual scores. To further validate the effectiveness of the model, we collect paired human preference data from open-source datasets. T2VQA-DB (Kou et al., 2024) is a Text-to- Video quality assessment Database, which contains 10000 gener- ation videos by different T2V models using 1000 prompts. GenAl-Benche Video (Li et al., 2024) collected 1.6k prompts from professional designers, generated videos by four T2V models, and hired three trained human annotators to rate each video for how well these videos align with prompts. TVGE (Wu al., 2024) collected 2.5k videos generated by five generation models and human ratings on the quality and alignment perspectives. For T2VQA-DB, GenAl-Bench- Video, TVGE-Quality and TVGE-Alignment, we sample 200 prompts randomly, and for each prompt, select two videos, with a total of 1600 videos for pairwise compar- isons. For each pair, one video has a higher score than
3.3. GRADEO Evaluation Process
Fine-tuning on train split dataset GRADEO- we collet in Sec. 3.2, we develop a MILM evaluator called GRADEO. The formulation of the loss function as fol- lows:
N-1
L(0) = Σ
logp( R. R2;;) VS,
(1)
where Q is question containing dimension instruction and assessment criteria (See Appendix C) V is the video being assessed, R is the rationale and reasoning process, S is final assessment score, N is the length of whole answer.
5

---

GRADEO: Towards Human-Like Evaluation for Text-to- Video Generation via Multi-Step Reasoning
Table 2: The correlation between Metries/MLLMss and human evaluation. MLLMs baselines are divided into Image MLLMs and Video MLLMs, with our method demonstrating superior performance in aligning with human evaluations in terms of Spearman's 's rank-order correlation coefficient (Parocc), Pearson's 's linear correlation coefficient (Ppice), and Mean
Absolute Error (A).
Model
Quality
Assihetic (143
A (fae
Consituntty
Aligument Tww.eTWaygysa (T
Ratiomality
Safity
Crustvvity P
Automaate Monics
PIQE (N
00531-00001110 .123
0.071/-0.2500 123
00145-0.142/11 ..445
0.145/000441.
0,255/1.446
BJ008/-0.1544
0.361/0.30VV 201
BRISQUE (btin 2012)
0.7H 0099)119
1U023/ 3277121100 .099
DLBES (U0851) 772
400055 0,18800888188
0,234/ 729
0.385N9.22N1
CLI Score (Hesel et.ll. 2021)
0.1849008011000 (,700
0,364R 577/1.213 211
0.2769.177/144 452
0,203/0 (53/1 .004
0.8818006511.1 1100
0,531/0 1,80
0,3960 342/3324//110 1020
LLAVA-1.5- (Lis 2023)
0.544/ 595
0.25300. 194
0.405--0 589
0.27590.55211.0 .041
0.493/0 2,001,400
0.607/0 566859958
1J 6-18 20350
1100
434-21
0.1773-0/99011
GLM-4x (GL
0.54380.42900.
B.12//0 505-01
0.3618)222/11 A88
0.333/
0.58090-45501
0.554/ 220
0.4450035401.22
0.462N) 3711
1,02500 641211
0.3140)-29M0775)
1.505
0,33590 (00/1
0,373A0-231/1
1220077720
0.4E1A)2004111 9332
0,41500
Vaiee MIZAts
1... (Chen 20246)
0.24220: 145
0,293/0 302/09913
0.32280.3001 215
0.563/..477/1100
0.52800.26 38300..61/1122
Video (2hhng et cl., 30246)
0.56500 44100855
0.250 MB0
0.200100012807795
895
0.354N0 15701115511102000
0.002/9008-11 9402
0.52300 291111177
Qwes2-VIL-1B (Ware et sk., 2024a)
0.59000 42400442400544
8.668/0 599/1
0.401X0.3 1000.622
0,2331-0 023/1 000
0,46180-22511 060
0,5500 244/2.244442222
0.60600 59311
Genini- 1.5 Flads (Taa at. 2023)
0611005100.82
(.663/0 00470
0.3220,000000700
1636/ 048/1 .465
0,011-000531110
0,348/0 255/132555/139
0.026-000851 375
Genitii (Teass al.. 2023)
0.5580-4540..14
0.58900556/1.555
0.4010022880.856
D.438/0 283/1
0.25880.22211.44 .499
0/8190044/1.00
0.7920.721/0772
MalesSonow I. (Hl et al., 2024)
0:3655-0 10600.789
0.57550 1060 545
01484/1 28 64.649
0.29280070000000
Oais
GRADKO
0.763907159044
0.7177L7799133511
BLOIBAL 618/1.99
W.W.V.555.554
0.747/07762/1366
0.791M.7900.542
the other. The accuracy of preference judgments, i.e., cor- rectly identifying the higher-scoring video, is used as the evaluation metric.
downstream tasks. To enhance the model's ability to as- sess generated videos in a manner similar to human evalu- ations, while preserving its original Insmuctionnfoollooee instraction-follovinng and reasoning capabilities, we apply the LoRA fine-tuning method (Hu et al., 2021). The learning rate is set to 1 x 10 and the model is trained for 10 epochs on 4 RTX 3090 (24G) GPUs
Baselines. On the test dataset, we primarily seleete current state-of-the-an art image and video multimodal large language models. On the pairwise dataset, we additionally selecte some automatic evaluation metrics as baselines. 1) Image MLLMs: 1...5 (Liu et al., 2023). LLa (Liu et al., 2024a), GLM-4v (GLM et al., 2024), GPT-4-Turbo, GPTT40 (Achiam et al.. 2023). 2) Video MLLMs: Intern VL- 2.5 (Chen et al., 2024b), LLaVA-NeXT- Video (Zhang et al., 2024c), Owen2-VL-77B (Wang et al., 2024a), Gemini-1.5- Flash, Gemini-1.5-P P.o (Team et al., 2023), VideoScore- 1.1 (He et al., 2024). 3)Automatic Metries: PIQE (N et al., 2015), BRISQUE (Mittal et al., 2012), CLIP-Score (Hes- sel et al., 2021), ImageReward (Xu et al., 2023), HPS- 2.1 (Wu et al.. 2023b). For MLLMs, with the exception of VideoScore, the input text for the other models are con- sistent with those of GRADEO, comprising task-specific instructions and dimensional evaluation criteria. For the dimensions of alignment and rationality, additional text prompts are incorporated to further guide the evaluation process. For automatic metrics, we use keyframes as in- put to calculate the scores, which are then scaled to a 1-5 range to align with the evaluation criteria. For VideoScore, we map visual quality, temporal consistency, text-to- vidvideo alignment, and factual consistency to the four dimensions of quality, consistency, alignment, and rationality, respectively. The scores from 1 to 4 are then scaled to a 1-5 range and rounded to the nearest level. More details are shown in Appendix C.
Table 3: Pairwise comparison accuracy. Our model demonstrates superior accuracy across all datasets, outper- forming other models consistently.
Medal
12VQA.08 Valoo Fabday
Gendl Valar
Video Prckanss
Tou Alganaand
TYGE
Video Quality
Ascosunic Mericr
BRISOCTE (Minal 2012
0.340
0.39
0.295
CUP Sume (Heoed et al. 2001
0.370
9.455
0,99)
0.035
IrogrRoweti- 100 Poner 2005)
0.880
8.60
0730
0.95
HPSS-21 ctal, 30226)
0.413
0.500
(000)
0025
12aVa- oi N0M4 ET M. MiLMara al, 2030)
0.155
6.40
0.94)
01300
8.500
B/135
077)
CPT-4
04550
0.500
0335
84Y)
Guezai- 1.5 (llaa 2023)
6320
062)
GRADEO
0,11
0.700
0.790
0344
4.2. Experimental Results
Quantitative Results We present a comparison of various state-of-the-art MILMs and our model in terms of human correlation on the test set in Table 2, and show the pairwise results on four Al-generated video assessment datasets in Table 3. Based on the data presented in these two Table, our model GRADEO demonstrates the highest alignment with human assessment across all evaluation dimensions. We ob- serve that video MLLMs do not outperform image MLLMs in this evaluation task. In comparison to open-source mod- els, fosed-souree models show higher consistency with human evaluations. For evaluation dimensions involving high-level semantics, only MLLMs are effective. Auto- mated evaluation metrics can perceive videos but cannot understand their content, let alone perform reasoning. As a result, they are ineffective in evaluation dimensions that re-
Training Details. We
adopt Qwen2-VL-7B (Wang et al., 2024a) as our base model, which has proven its perception and understanding capabilities for multimodal data, par- ticularly videos, across various open-source datasets and
6

---

GRADEO: Towards Human- Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
You are on expert in erclucting Al-genaroted videos. Your fast to asess the specific dimension of the video:
To assesa the video effectivell, I will bagin with a comprehersive and thorough description of its content: Fallowing this I will conduct a detsiled, criterin-ssacffic andlyss, excluating the video againat the provided rahing criherie. finolly, will synthesize ze these insights to defermine and present the findl asssssment score.
The video features an open book with handwriften hext on wooden table. The bosk's cover wern and aged, suggeling if has bean wall-used over lime.
creativity. and provide score for this dimention.
Promph: An old, leather-bound book an wooden table. Roting cnteno:...
Generated Video
Firsll, the video's visual elements stroightforward and
The open book with handwritfen imaginative visual techniiques The elaments that would clevate the
text familiar sight, lacking any sense age listrry, not Introduce
whlle
-lovo-1.6-7b
Liove-NeXT-Vie -V
video's crea
Based on the image proviced, would rafe the creativity of the video as
Secendly, the
dipect of video minimal. The book itself does convey any story relying solely the visual presentation narrative further diminshes video's creative impact. considering overall imagination hion
Bosed on the images provided, it is not possible to
through its
evaluste the creativity of on Al-generched video the
image shows physical deject, specifically an open book, rother than video.
The visual elements and nerrotive techrqques are not overly ardinary, but there is room for improsement in herms of depth and novelty, The content is not dull, but it does not fully break away from the rorm.
evident that the video fails spark viswer's lack intovalive storyfelling visual expression that the basic expectations
To ordiuste the creahviity of video, one would need to watch the video and andlyce its visual elements, narrative techriques, and overall content.
creative So, dull against the rating ciiteria, vides clearly does exhibit the characteristics unique imeginative higher ratiig.
high tachniques, engaging night the have any
level creatistty It that
elements, innovative
qualities? While
touch of
Gemini-1. Gemini-1.5-Flashh
outhennicity and iestory, it does compensate for the averall monatonous and common nature siseo.
falls away
making diffcult any
would give the video o creativity score of 2. The video shows static shat of on antique book. While
sigreficont novelty.
the video's crastivity significontly linitad, oiigning it closely the raling criteris.
the boot itself vouully intoresting due to its age and leather airding, the video locks any crestive steual elements or norretive techniques. There's no movement, edi ediiting or special effects.
Gemini-L.5-Pro
Creativity:
Rating Score:
It's sinply a protonged still image. The content is not thought-provoknng provoking beyend the potenticl intorest in the book's age and possible historical sigrificance. There's attump! fell story convey message beyord simple visual presentation of the oeject. While the jubject nather could be used creativaly. this video does do ac.
I'm unable to evaluste the creativity of the video based on images. If you can descrioe the video's eiements and namrative or provid more context, Idd be happy to help assess its creativity within the given criteria.
GPT-40
engaging or thought
This video does not fully nost the criheris for creativity it prosents atatic image of leather-buurd book on tabls While book may wokke curiosity about itss contents, there isrit much dynamic visual starytelling or innovative techniquas involved. Thus, while may be considered slightly original, it dossr't push the bourdories creativily sigrificonily Score:
InternVL-2.5-8b
Figure 4: Qualitative Results for Creativity Dimension. MLLMs baselines may refuse to provide a score or give a score before presenting the evaluation reasoning. In contrast, our approach generates detailed reasoning steps prior to assigning final score, resulting in higher consistency with human assessments.
quire semantic understanding. On the other hand, MILMs, due to their ability to follow instructions and understand visual content, can score according to evaluation standards. Their understanding and reasoning capabilities help ensure consistency with human assessments.
soning steps in the chain. We exclude the final scoring component from the CoT and input the remaining reasoning steps, along with the evaluation prompt, into the base model, which was found to improve overall alignment with human evaluations. The absence of <Descripp iittiiiiin impairs the understanding of the original video content, resulting in a substantial decline in overall evaluation metrics. Without <Overview>, the model struggles to establish a logical reasoning steps for evaluation, significantly reducing its precision. We also experiment by removing all reasoning steps and retaining only the score as the model's output. However, this approach neither enhances consistency with human evaluations nor maintains interpretability. We further introduce a simple CNN-based video scoring model trained from scratch for comparison (See more details in Appendix C.2). As shown in Table 4, this baseline performs signif- icantly worse than our LLM--bsed based method, highlighting that large language models leverage their rich pre-trained knowledge to understand video content.
Qualitative Results In addition to quantitative results, we also present qualitative results in Figure 4, 5. It can be observed that our model leverages multi-step reasoning ca- pabilities to achieve interpretable score evaluations across multiple dimensions. We have found that some MLLMs oc- casionally refuse to answer or provide very brief responses, which often leads to evaluation failures errors. LLava-1.5- 7b and Gemini-1.5 only output scores in a large number of examples, which is insufficient to meet our goal of provid- ing a more interpretable evaluation using MLLMs. GPT-4o refuses to evaluate in some examples, which also affects the evaluation results. Models like Llava-NeXT- 7b typically respond with score first, which leads the eval- uation rationale being generated based on that score, rather than utilizing reasoning capabilities to improve the evalu- ation results. For video Rationality evaluation, MI.LMs without instruction tuning exhibit a higher susceptibility to prompt-induced hallucinations, where generated responses are disproportionately influenced by the input prompts rather than grounded in the visual content.
Table 4: Ablation study on the impact of multi-step reason- ing and CoT integrating fine-tuning.
Human Correlation
Fine-luning Descriiptio>
<overview>
CsT
x
x
0.51240.339100 1,006
0.572AL.3221 .023
x
x
0 0.5284).4240.66 42400.42
4.3. Ablation Study
0 0.58814462200611
0.9929.647:00422
We perform an ablation study on both the integration of fine-tuning and the completeness of rea-
Basic CNN Video Scoring Model
0 0.302/0, 185
7

---

GRADEO: Towards Human- Like Evaluation for Text-to- Videoo Generation via Multi-Step Reasoning
You are on expert in evalusting Al-generohed videcs. Your fast axeess the spacific oi of the videot
To assesa the viden, I will first onslyse the given prompt, rarely watered lown" and infer itss logical outcomes based on human common sense, particularty focusing on the theme of biological lowe. Next, will conduct det alled, or ietrrnsspaaffic onalysis of the
The prompt "A renety wetered lown" inherently suggests acenorio where the lown receives minimal
retionality, and provide store for this dimension.
water, leading fo a lock of greenery and posalbly dry
Prompt: rarety watered lown The thems of human cormon saroe: Biological Laws Roting critente..
patches This rosted in biological lawa, particularty the need for water to sustoin plant life. One might ask.
"What would o lown look like it's rorely wateredf The
Generated Video
video content relation to these inferred
effects. Findlly I will provide on casesoment score based on how well the
netural consequance of insufficient water. THis allgno
with patches of brown or yellow grase, reflecting the
expeched outiome would be lawn that appsors dry.
Llavo-1.675
Midva-NeXT-Video--1
video algge with reol-worid expactations and commen serse ressening
between water and plant health.
with common sence knowladge about the onsipionsip
Based the prompt "A nonely watered lown"
the provided criteria,
theme common reflected
rationality of the video fallows real-world expectation telions and
Upon exemining the video, it becomes evident that the lown consishently green and lush, which directly contradicts the inferred effect nenely wate wahered lown. The video does not depict signa dryness brown potichs, which typically asaociated with insufficient forfiing Ok, the lown looks healthy thy and vibrent, but this confredicts the expected outcome based on the prompt. Thin, fals represent
"Bological
To the rafionality of the videa. 1
would cansider the following:
logical consistency
prompt suggests condition,
knowledge reasoring, with lapses. example, the prampt "A watered lawn" watering load dry usheal thy suggeats
coreidering the theme of tielogitll lowe, the watering how doee this align with resl werid expectationat Clsarly, it dons not The video's depiction lui,, lown does not reflect the no nafural sutcome ronely wahared lown, leading to noticcable devistion from
of
rarely wills
2. expected effectt expechs fifons.
lown
dry, discolared, grass.
watered
sonse ressoning.
patantially develop patches
video
that appears
fion. However, the video how
Rating Score:
cry, discolared, posiibly
does provide any information
greas.
Information provided,
Gemini-L.5-Fipp
Hligh resionaliity video
ronely
The video shows lown that ospears dry and somewhat sporse, which consistent with a rorely watened
prompt aligna
logical inconsisheney. Addilionally.
lown. The color is slightly dul, yellowist-grees suggesling lack of masture. There are no abvious
eepactations logical heney
the theme "Bulogical Lowe"
signs of lushness healthyy growth.
common knowledge and reasoring, cloeely expactations showing poor condition lack watering.
directly related the prompt,
Therefore, would rate the rationality of the video as 4: High retionality. The vida'ss depiction dllgns
minor lapans.
ranely watered
well with the real world expectation of a narely watered lown. while the viusl qudlity ion't extrenely
which
iopan
high-definihen, the averall visual reprasentation of the dryness and leae- finel health of the grass is sufficient and plauible. There are no noticeable incorsistancies or illogical elements
logical consistency.
Rationality Score: 4 Ratianality Assessment: The video presents lown that appears to be dry and has patches of brown, which aligns with the description of raraly watered lawn, However, lown still appears be mostly grean, which may not perfectly match the expacted effect of watering. Overall, the video is acceptable in terms of rationality, with a minor lapse in fully representing the effects of infrequent watering
Gemini-1.5-Pro
Figure 5: Qualitative Results for Rationality Dimension. MLLMS baselines either fail to assess the rationality of the video accurately, or are misled by the prompt and generate hallucinations. Our model, by faithfully adhering to the video content and reasoning according to the prompt, effectively compares the observed scenes with real-world expectations.
5. Benchmarking recent T2V models
with science fiction or fantasy themes are rarely represented in the training datasets of these models. Furthermore, the low scores in the Rationality dimension highlight the diffi- culty of generating vidcos that conform to real-world prin- ciples, underscoring the inherent challenges in producing high-quality, semantically meaningful content aligned with human conceptual standards.
5.1. Benchmark Setup
We adopt 8 open-source SOTA Text-to-Video models, in- cluding Hotshot-XL (Mullan et al., 2023), Lavie (Wang et al., 2024b), Latte (Ma et al., 2024), ModelScope (Wang et al., 2023), ZeroScope (Cerspense, 2024), VideoCrafter- 1 (Chen et al., 2023), VideoCrafter-2 (Chen et al., 2024a), Open-Sora (Zheng al., 2024). We evaluate models GRADEO-BENCE using GRADEO, and the scores ob- tained for each dimension are averaged and scaled to a percentage. Further details are provided in Appendix D.
5.3. Insights
High-quality video generation and high alignment are prerequisites for producing videos that are consistent with real-world scenarios. In our designed rationality prompts, most prompts depict real-world scenarios that can be visualized through human common sense. In these sce- narios, events involve actions performed by humans ob- jects, which align with common sense. Reasoning about the consequences of these events constitutes commonsense reasoning. To achieve high rationality scores, a model must first construct a realistic scenario, generate the key elements to perform activities, and subsequently produce the conse- quences of these activities. However, current models fail to construct such scenarios for certain prompts, are unable to generate realistic and high-quality key elements, and there- fore cannot lead to accurate behavioral outcomes.
5.2. Benchmark Results
Table 5 presents the evaluation results of T2V models across seven dimensions. Among these models, VideoCrafter2 demonstrates the best overall performance, while the rel- atively outdated ModelScope model exhibits the weakest results. Scores for the Consistency dimension are notably high, likely because this dimension emphasizes the absence of distortions in the subject and background rather than fine- grained pixel-level details. Given the short-duration videos produced by current models, significant distortions are less likely to occur. Hotshot-XL. which generates I-second GIFs at 8 FPS, achieves the highest score in this dimension, likely due the simplicity of maintaining consistency in such short outputs. However, beyond Consistency and Align- ment, most models perform poorly in other dimensions, particularly Creativity. This is to be expected, as videos
Effective generation of key elements and logical coher- ence in video timelines is essential for creating videos that accurately reflect the narrative described in the text prompt. When the text prompt is longer, it often
8

---

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Table 5: Benchmarking results of recent T2V models on GRADEO- B-BENCH for each dimension. All scores are scaled from 0 to 100, and the average value across each dimension is taken as the final score. Higher score indicates better performance for a dimension.
Combitency
T2V Models
Quality
Aesthetic
Consistency
Aligument
Rationality
Safety
Creativity
Hotshot-Xi XI.
42.2
58.2
70.2
62.5
48.0
47.0
27.8
Aligamant
Lavie
44.0
62.8
67.0
63.5
42.5
48.0
33.2
Latte
40.8
56.0
67.2
62.2
38.5
44.0
31,5
Links
ModelSoope
25.8
48.5
56.5
56.2
37.2
49.8
26.0
Quality
Medechopp
ZeroScupe
38.5
54.5
65.5
62.0
41.5
44.5
30.5
Zeroiooe
Drifheed
VidcoCrafter-1
31.5
54.2
60.0
36.5
39.0
45.2
34.0
VideoCrafter-2
48.8
48.8
Open Sera
Open-Sora
32.2
51.2
65.0
60.2
39,8
44.5
28.5
Crestivit
contains richer scene elements, forming coherent story narrative. The model needs to effectively generate the key elements described in the prompt while ensuring that the temporal and spatial order of the overall elements is logical. However, videos generated by existing models exhibit sev- eral issues, such as elements blending together, resulting in overly chaotic scenes; grand but overly simplistic settings lacking details; and elements appearing over time but with no logical coherence in the video.
ments demonstrate that GRADEO achieves high alignment with human scores, surpassing sfate-of-the-art models like GPT-4o, Gemini 1.5 Pro, and VideoScore. Through bench- marking current T2V models on our prompt suite, we reveal that existing models struggle to generate videos that adhere to real-world knowledge and human commonsense. We hope our insights inspire further exploration into building more reliable and human-alignel video evaluation systems.
A Acknowledgements
Models with strong instruction-follovin capabilities and extensive training data are more prone to gener- ating unsafe content. Through our evaluation of several advanced video generation models, we observe that models with high cnstuucioonffolowvii fidelity tend to more strictly adhere to prompt specifications, even when such prompts imply or explicitly request unsafe, unethical, or harmful content. This suggests that as instruction tuning and model capabilities improve, the potential risks associated with mis- use or adversarial prompting also increase. Notably, these models often lack robust safety constraints alignment mechanisms to reject or sanitize unsafe inputs, making them susceptible to generating content that violates ethical or safety norms. This highlights the urgent need for future research on adversarial robustness, jailbreak prevention, and the development of safety-aware training frameworks in video generation models.
We would like to thank the anonymous reviewers for their insightful comments and suggestions, which helped im- prove the quality of this paper. This work was partly sup- ported by the National Natural Science Foundation of China (Nos.623 1530100062271223 tions for the Development of Strategic Emerging Industries of Shenzhen (No1K/2220033002204770000
and the Special Founda-
Impact Statement
This paper presents novel evaluator and benchmark for evaluating video generation models. The development of GRADEO- and GRADEO fills a crucial gap in assessing Al Al-generated video content, establishing foun- dation for future research in this field. By introducing new evaluation strategies, we aim to influence future benchmark- ing practices, which may promote the development of T2V models. Our evaluation framework can help uncover the po- tential of current video generation models to produce unsafe, unethical, or harmful content. However, the advancement of T2V models also brings potential negative impacts, in- cluding the risk of misuse to create realistic but mislcading video content, such as deepfakes, which could contribute to misinformation and societal harm. We therefore urge the research community to prioritize the development of robust video generation detection systems to identify and mitigate the spread of harmful synthetic content. At the same time, we emphasize the importance of incorporating safety-aware design principles into the development of video generation models to minimize the risk of misuse from the outset.
6. Conclusion
Our work represents meaningful step towards improving video evaluation methodologies for Al-generated content. By proposing GRA GRADEO, a video evaluation model trained on human-labeled data and capable of performing multi-step reasoning, we bridge the gap between machine assessments and human evaluations. From low-level quality metries to high-level semantic reasoning, we introduce comprehen- sive evaluation framework encompassing seven dimensions with well-defined criteria. Our systematic dataset construc- tion pipeline guides MLLMs to assess generative videos through structured multi-stage stage reasoning. Extensive experi-
9

---

GRADEO: Towards Human-Like Evaluation for Text-to-Vidde Video Generation via Multi-Step Reasoning
References
Chen, Z., Wu, J.. Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q. Zhu, X., Lu, I.., et al. Internvi: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024b.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S.,
Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F,, Hui, B., Ji, L.,, Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P.. Wang. S., Wang, W. Wu, S., Xu, B.. Xu, J., Yang. A., Yang, H., Yang, J., Yang, S., Yao, Y., Yu, B., Yuan, H.. Yuan, Z., Zhang, J., Zhang, X., Zhang, Y., Zhang, Z., Zhou, C. Zhou, J., Zhou, X., and Zhu, T. Qwen technical report, 2023.
Chivileva, I., Lynch, P., Ward, T. E., and Smeaton, A. F. Measuring the quality of text-to-video model outputs: Metrics and dataset. arXiv preprint arXiv:2309.08M0 0,00,, 2023.
Dai, J., Chen, T., Wang, X., Yang. Z., Chen, T., Ji, J., and Yang, Y. Safesora: Towards safety alignment of text2video generation via human preference dataset. arXiv preprint arXiv:2406. 1 14477, 2024.
Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Feng, W., J., Saxon, M., Fu, T.j.. Chen, W., and Wang, W. Y. Te-bench: Benchmarking temporal compositional- ity in text-tovvideo and image-to-vides generation. arXiv preprint arXiv:2406 08656, 2024.
Bansal, H., Lin, Z., Xie, T., Zong, Z., Yarom, M., Bitton, Y., Jiang, C., Sun, Y., Chang, K.-W., and Grover, A. Videophy: Evaluating physical commonsense for video generation. arXiv preprint arXiv:2406.03522 03.500000 2024.
Fu, X., He, M., Lu, Y. Wang, W. Y., and Roth, D. Commonsense 12i challenge: Can text-to-image genera- tion models understand commonsense? arXiv preprint M.Kiv:2406.05444 2024.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., Yu, H., Wang, H., Sun, J., Zhang, J., Cheng, J., Gui, J., Tang, J., Zhang, J., Li, J., Zhao, L., Wu, L.,, Zhong, L., Liu, M., Huang, M., Zhang. P., Zheng, Q.. Lu, R., Duan, S., Zhang, S., Cao. S., Yang, S., Tam, W. L., Zhao, W., Liu, X., Xia, X., Zhang, X., Gu, X., Lv, X., Liu, X., Liu, X., Yang, X., Song, X., Zhang, X., An, Y., Xu, Y., Niu, Y., Yang, Y., Li, Y., Bai, Y., Dong, Y., Qi, Z., Wang, Z., Yang. Z., Du, Z., Hou, Z., and Wang, Z. Chatglm: A family of large language models from glm- to glm-4 all tools, 2024.
Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S. W., Fidler, S., and Kreis, K. Align your latents: High- resolution video synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference on Computer 2023.
Vision and Pattern Recognition, pp.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Guo, Y., Yang. C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image image diffusion models without specific tuning. In The Twelfth International Conference on Learning Representations, 2024.
Caron, M., Touvron, H.. Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Hao, F., Shengqiong, W., Wei, J., Hanwang. Z., Meishan, Z., Mong-Li, L., and Wynne, H. Video-of-thought. Step- by-step video reasoning from perception to cognition. In Proceeding of the ICML, 2024.
Cerspense. Zeroscope. https: http:/hugggnggggfcccc co/ cerapense httpeennseeexcccppeee_eee v2__2_2_660 576,, 2024.
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q. Wang, X., Weng, C., and Shan, Y. VidcocrafterI: Open diffusion models for high-quality video generation, 2023.
He, K., Chen, X., Xie. S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 16000~16009, 2022.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng. C., and Shan, Y. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024a.
He, X., Jiang, D., Zhang, G., Ku, M., Soni, A.. Siu, S., Chen, H., Chandra, A., Jiang, Z.,, Arulraj. A., et al. Videoscore:
10

---

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Building automatic metrics simulate fine-grained hu- man feedback for video generation. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 2105-2122 2024.
Li, B., Lin, Z.,, Pathak, D., Li, J., Fei, Y., Wu, K., Ling, T., Xia, X., Zhang, P., Neubig, G., et al. Genai-bench: Evaluating and improving compositional text-to-visual generation, arXiv preprint arXiv:2406. 13743, 2024.
Hessel. J., Holtzman, A., Forbes, M., Bras, R. L.. and Choi, Y. Clipscore: A reference- free evaluation metric for im- age captioning. arXiv preprint arXiv:2104.087/6, 2021.
Lin, B., Ge, Y., Cheng, X., Li, Z., Zhu, B., Wang, S., He, X., Ye, Y., Yuan, S., Chen, L.. et al. Open-sora plan: Open-source large video generation model. arXiv preprint arXiv:2412.0013// 2024a.
Ho. J., Jain, A., and Abbeel,
P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-688 6801, 2020.
Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint urXiv:2404.013 2024b.
Ho, J., Chan, W. Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210002303, 2022a.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. arXiv preprint arXiv:2304.080 08485, 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge.
Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-866 8646, 2022b.
github. 2024a,
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang, Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
Liu, Y., Li, L., Ren, Gao, R., Li, S., Chen, S., Sun, X., and Hou, L. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-vider generation. Advances in Neural Information Processing Systems, 36, 2024c.
Huang, Z., He, Y., Yu, Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Ma, X., Wang. Y., Jia, G., Chen, X., Liu, Z., Li, Y.-F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.00 0030488 2024.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Meng, F., Liao, J., Tan, X., Shao, W., Lu, Q. Zhang, K., Cheng, Y., Li, D., Qiao, Y., and Luo, P To- wards world simulator: Crafting physical commonsense- based benchmark for video generation. arXiv preprint arXiv:2410.05363, 2024.
Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang. Z., Navasardyan, S., and Shi, H. Text2video- zero: Text-to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15954-15964, 2023.
Miao, Y., Zhu, Y., Dong, Y., Yu, L., Zhu, J., and Gao, X.-S. T2vsafetybench: Evaluating the safety of text-to-video generative models. arXiv preprint arXiv:2407.05965. 2024.
Kling, K. Kling. tittps:/kkl:.
com/ 2024.
Mittal, A., Moorthy, A. K., and Bovik, A. C. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing, 21(12)4695-4408. 2012. doi: 00.1099TI12201222244555
Kou, T., Liu, X., Zhang, Z., Li, C., Wu, H., Min, X., Zhai G., and Liu, N. Subjective-aligned dataset and metric for text-to-video quality assessment. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7793-7802, 2024.
Mullan, J., Crawbuck, D., and Sastry, A. Hotshot- xl. https: / /ggttub.b../// cww/hot chot com/cotthottooo
Lab, P. Pika. https://piie art// art//2024. 2024.
hotshot-x1, 2023

---

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
N, P. Bh, M. C., Channappayya, S. S., and Medasani, S. S. Blind image quality evaluation using perception based features. In 2015 Twenty First National Conference on Communications (NCC), pp. 1-6, 2015, doi: 10.1109/ NCC.2015..084843.
Team, G. Mochi 1. https://github. genmoat genmoai/modell /models, 2024.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu,
J., Sorieut, R., Schalkwyk, J., Dai, M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint 2rXiv:2312.118000 2023.
OpenAI. Chatgpt. chatgpt, 2022.
https://opene compblgg//
Touvron, H.,, Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. Llama: Open and efficient foundation language models, 2023a.
OpenAL. Gpt-4w(ision) system card. https://openal: com/ com/research/ gpt- 4v-
Seert--v-yystte--oe system- card, 2023.
OpenAL. Sora. https: /openai..aaaaaa//e//e wom/sorra////.
Qu, B., Liang, X., Sun, S., and Gao, W. Exploring aige video quality: A focus on visual harmony, video-text consistency and domain distribution gap. arXiv preprint 0rXiv:2404.13573. 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y.. Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L.., Ferrer, C. C.,, Chen, M., Cucurull, G., Esiobu, D., Fernandes, J. Fu, J., Fu, W. Fuller, B.. Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog. I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, X.,
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp.
PMLR, 2021.
Rose, D., Himakunthala, V., Ouyang, A., He, R., Mei, A., Lu, Y., Saxon, M., Sonar, C., Mirza, D., and Wang, W. Y. Visual chain of thought: bridging logical gaps with multi- modal infillings, arXiv preprint arXiv:2305.02311 2023.
Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, Ilama Open foundation and fine-tuned chat models, 2023b.
Runway, Runway gen-2. https: /tte:///runeeee http://wwwwwwmmmmm/mmm com/ gesearch/gen-a 2024.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric challenges. arXiv preprint 1rXiv:1812.01711: 2018.
Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V.,
Radford, A., and Chen, X. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Wang. J., Yuan, H., Chen, D., Zhang, Y., Wang, X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint arXiv:2308..m 06571, 2023.
Shao, H., Qian, S., Xiao, H., Song, G., Zong. Z.,, Wang, L., Liu, Y., and Li, H. Visual cot: Unleashing chain- of-thouuht reasoning in multi-modal language models. arXiv preprint arXiv:2403. 16999, 2024.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing languaee model's perception of the world at any resolution. arXiv preprint arXiv:2409. 2024a.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2265. PMLR, 2015.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Wang, W. and Yang, Y. Vidprom: A million-scale real prompt-gallery dataset for text-to-video diffusion models. 2024.
Sun, K., Huang, K., Liu, X., Wu, Y. Xu, Z., Li, Z., and Liu, X. T2y-compbench: A comprehensive benchmark for compositional text-to-video generation. arXiv preprint arXiv:2407. /4505, 2024.
Wang. Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High- quality video generation with cascaded latent diffusion models. IJC,, 2024b.
12

---

GRADEO: Towards Human-Like Evaluation for Text-to-Vidde Generation via Multi-Step Reasoning
Wang, Y., Zeng. Y., Zheng, J., Xing. X., Xu, J., and Xu, X. Videocot: video chain-of-thought dataset with active annotation tool. arXiv preprint urXiv:2407.05333 2024c.
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., and Smola, A. Multimodal chain-of- reasoning in lan- guage models. arXiv preprint arXiv:2302.0000 00000000022,,.. 2023.
Wei. J., Wang, X.. Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Zhang, Z., Sun, W., Li, X., Jia, J., Min, X., Zhang, Z.,, Li, C.. Chen, Z., Wang, P., Sun, F,, Jui, S., and Zhai, G. Benchmarking multi-dimensional aige video quality assessment: A dataset and unified model. arXiv preprint arXiv:2407.21400 2024e.
Wu, C., Huang, L.., Zhang, Q. Li, B., Ji, L.,, Yang, F,, Sapiro, G., and Duan, N. Godiva: Generating open- domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Zhao, Y., Yin, H., Zeng, B., Wang, H., Shi, T., Lyu, C., Wang. L., Luo, W., and Zhang, K. Marco-ol: Towards open reasoning models for open-ended solutions. arXiv preprint 1r 144005 2024.
Wu, H., Zhang, E., Liao, L., Chen, C., Hou, H., Wang, A., Sun, W. S., Yan, Q., and Lin, W. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In International Conference on Computer Vision (ICCV), 2023a.
Zheng, Z., Peng, X., Yang. T., Shen, C., Li, S., Liu, H.,
Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all. https://gitts com/hpes tetceehhhhp SOran- 2024.
Zhipu. Qing ying. https: /tte:://chaate chwtglaattggmmmmmm en/video, 2024.
Wu, J. Z., Fang, G., Wu, H., Wang, X., Ge, Y., Cun, X., Zhang, D. J., Liu, J.,W., Gu, Y., Zhao, R., et al. Towards a better metric for text-to-video generation. arXiv preprint arXiv:2401.077 00.70111,,7,, 2024.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.094 1, 2023b.
Xu, G., Jin, P., Hao, L., Song, Y., Sun, L., and Yuan, L. Llava-ol: Let vision language models reason step-by- step. arXiv preprint 1..iv:24/1.144444 2024.
Xu, J., Liu, X., Wu, Y.. Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Zhang, D. J., Wu, J. J., Liu, J.-W., Zhao, R., Ran, I.,, Gu, Y., Gao, D., and Shou, M. Z. Show-l: Marrying pixel and latent diffusion models for text-to-video generation. IJCV, 2024a.
Zhang, F. Tian, S., Huang, Z., Qiao, Y., and Liu, Z. Evaluation agent: Efficient and promptable evaluation framework for visual generative models. arXiv preprint MrXiv:2412.09645, 2024b.
Zhang, Y., Li. B., Liu, h. Lee, Y. j.. Gui, L., Fu, D., Feng. J. Liu, Z., and Li, C. Llava-next: A strong zero-shot video understanding model. https:
wtatttuu..../bl....e io lo/blog/ 2024-0 01-4-04-300 30avlavv next veeettiiideo,//0 2024c.
Zhang, Y., Wei, Y., Jiang, D., ZHANG, x., Zuo, W., and Tian, Q. Controlvideo: Training- free controllable text-to- video generation. The Twelfth International Confer- ence on Learning Representations, 2024d.
13

---

GRADEO: Towards Human-Like Evaluation for Mext-to-Video Generation via Multi-Step Reasoning
A. More Details on Dataset
A.1. Dimensions Definitions
Quality Dimension. evaluates the overall visual fidelity Al-generated videos by examining key aspects such as clarity, artifacts, naturalness, texture and detail, and temporal consistency. (1) Clarity focuses on whether the video is sharp and free from blurriness, ensuring that objects and features are well-defined and easy discern. (2) Artifacts refers visual anomalies in Al-generated videos, such distortions, motion artifacts, unnatural shadows. (3) Naturalness assesses the plausibility of object shapes, movements, and the harmony of the scene, ensuring alignment with human intuition about the real world. (4) Texture and detail measure the richness and accuracy of surface characteristics, identifying issues like overly smooth surfaces or repetitive patterns. (Examples see Figure 6)
Pear
Blur
Artifacts
Uninteresting Content
Advanced Composition
Lack of Detail
Low Naturalness
Attractive Colors
Excellent Lighting
(a) Qulaity Examples
(b) Aesthetic Examples
Figure 6: Examples that align with the definitions quality and aesthetic dimensions.
Aesthetic Dimension. evaluates the overall artistic and visual appeal of Al-generated videos by considering content, composition, color, and lighting key aspects. (1) Content assesses the relevance and engagement level of the subject matter, ensuring that it captures interest and aligns with the video's purpose. (2) Composition evaluates the arrangement of visual elements, focusing on balance, focal points, and intentionality to create cohesive and visually pleasing layout. (3) Color examines the vibrancy, harmony, and appropriateness the palette, ensuring it enhances rather than detracts from the visual experience. (4) Lighting considers how illumination contributes to mood, depth, and clarity, ensuring it supports the overall aesthetic quality of the video. Together, these aspects determine the video's ability to deliver visually engaging and artistically refined experience. (Examples see Figure 6)
(a) Single subject elements distortion.
(b) Confusion and distortion of multiple subject elements.
(c) Complete distortion of the subject and background.
(d) Strong temporal consistency.
Figure 7: Examples of temporal consistency dimension: (a), (b), and (c) exhibit low consistency with distortions, while (d)
shows high consistency.
14

---

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
Consisteney Dimension. evaluates the stability and coherence of Al-generated videos, focusing on the continuity of main subjects and background elements across frames. High consisteney ensures that the subjects and scenes maintain their identity throughout the video. Low consistency introduces distortions, warping, or fluctuations. (Examples see Figure 7)
Alignment Dimension. evaluates the degree to which Al-generated videos correspond the provided prompt across key aspects: objects, count, color, style, spatial, temporal, action, and camera. Object alignment examines whether the depicted subjects match those described in the prompt, while Count ensures the correct number of elements is represented. Color checks for consistency with the colors of Object, and Style assesses whether the video reflects the specific style. Spatial verifies the arrangement and positioning of elements, while Temporal alignment ensures that the sequence and timing of events match the description. Action focuses on the accuracy behaviors or movements, and Camera alignment evaluates whether the angles and camera movements align with the described perspective. Together, these aspects determine how effectively the video realizes the intended prompt, ensuring clear and faithful representation.
Rationality Dimension. evaluates the degree to which Al-generated videos align with real-world expectations and adhere to logical consistency based on common sense. This dimension focuses on the plausibility of events, interactions, and the behavior of objects and characters, ensuring they correspond to human intuition and knowledge of the physical world. (Examples see Figure 8)
<hysical Lawss
A pot of boiling water on cold day.
(a) A pot of boiling water should produce steam, so the video demonstrates high rationality.
<Weather Conditions> A blizzard hitting town.
(b) A blizzard hitting reduces visibility, with buildings covered snow, making the video rational.
<Chemica! Lawso Phenolphthalein indicator changing color in basic solution.
(c) Phenolphthalein turns pink in alkaline solutions, so this video is irrational.
<Blological Laws> A wheat field in the spring.
(d) The wheat fields in spring are green, and only mature wheat turns yellow, so this video content is unrealistic.
Figure 8: Examples of rationality dimension: Whether it aligns with human common sense and the real world is the core focus of the rationality dimension.
Safety Dimension. evaluates the potential risks harmful elements present in Al-generated videos, ensuring they adhere to safety standards and avoid controversial sensitive content. High safety indicates the absence of such risks, presenting content that is universally acceptable and contextually appropriate. Lower safety scores reflect the presence of elements that could raise concerns, such as provocative, graphic, or sensitive material, even if not overtly harmful.
Creativity Dimension. evaluates the originality and imaginative quality of Al-generated videos, focusing on the novelty of visual elements and narrative techniques. Exceptional creativity is marked by unique, thought-provoking, and innovative content that captivates the viewer and breaks away from conventional norms. Average creativity reflects a mix of standard techniques with occasional innovative elements. Lower creativity indicates a lack of novelty, with predictable or monotonous visuals and narratives.
A.2. Data Collection Details
Prompts. For the Rationality dimension, we primarily generate corresponding prompts based on commonsense knowledge from different themes. In the Table 6, we present some example prompts, each representing commonsense event related
15

---

GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
its respective theme. For the creativity dimension, created some routine prompts with little creativity, while also collecting creative prompts from the community. Using these, we leveraged GPT-4o generate additional creative pSempts.(See Table 7)
Theme
Exumples
Percent
Theme
Examples
Percent
An uncapped water bottle turning upside down
A child seeiigg snow for the first time
Physical Laws
pebble the
16.33
Emotional Responses
athlete crossing finish line first
11.75
A sundac univuched for several hours
A soldier returning home
Someone standing when superior enters
A peasock ultracting mule
person recelving gift
Behaviors
eating gorilla
7.97
A lighthouse during daytime
A monkey cating a bonana
A heavy rainstorm city
Daily Items
A bicycle with a broken chain
7.37
Weather Conditions
A cloudless sky during the day
6.77
hourglass finishing counting
ninbow after shower
A chunyy uree in winler
Ptsnoolpaalskin indicator changing color in basic solution
Hiological
pine
drought
5:38
Chemical
Rlench ruming stained fabric while
4
A wheut field during harvest
Calcium curbonate reacting with hydrochloric acid
Saturn's rings through telescope
A coffee grinder crushing beans
Astronomical Phenomena
Sumspoes visible on sun
4.98
Mechanical Operations
A bulldocer pushing soil
1.78
Gamma-re ray burst distant galaxy
A sowing machine stiiching fabric
A balloon with
A peciutticaan checking child's knee-jerk reaction
Material Properties
A metal rod heated in fire
3.19
Other
A dragon dance in a Chinese New Year parade
15.74
glass window by
A glass window by
Table 6: Themes of commonsense knowledge and the corresponding percentage of data samples.
Level
Examples
(1) starry night sky with full moon.
(2) A bird flying in the sky.
Low Creativity
(3) A child playing with ball.
(4) driving on road.
(I) A medieval castle floating in the sky, held aloft by colossal, golden chains
that stretch into the clouds. The towers made of shimmering glass and enchanted marble, and the courtyard is filled with glowing flowers that bloom in time with the sound of distant music, A group of knights silver armor ride ethereal, winged horses through the clouds, their swords crackling with magical energy. (2) A hidden cove, where the sea communicates through bioluminescent patterns on their skin. As they dance under the moonlight, their bodies light up in sync with the rhythm of the waves, creating mesmerizing spectacle that attracts sea creatures from the depths. (3) a forgotten temple deep in the jungle, a massive tree grows through the ancient stone structure, its roots wrapping around towering pillars. At the base of the tree, enormous crystal floats in midair, casting rainbow reflections on the moss moss-covered walls, A group explorers with glowing tattoos approaches, their steps silent as the ground beneath them pulses with ancient energy.
the jungle, home tribe that
High Creativity
Table 7: Examples of prompts with varying levels of creativity.
Videos. During the video collection process, we used a series of video generation models to generate results, with the model parameters used during generation shown in the Table 9, Except for OpenSora, which runs inference on RTX 3090 GPUs, and Kling, which generates videos on the web, all other models perform inference on a single RTX 3090 GPU. After generation, we employed two distinct methods filter videos based on their motion intensity. The first method involves calculating the normalized video motion strength using optical flow intensity, denoted as D Dlow- The second method assesses the motion strength based on the Structural Similarity Index Measure (SSIM), denoted as D.sm- By utilizing these two metrics, Drow and Dysim we were able to effectively filter out videos with excessively high or low motion intensities, thereby enhancing the quality and relevance of our dataset for subsequent analysis. The formulas for these calculations are given by:
h (F (..
D How
Σ
(2)
S
16

---

GRADEO: Towards Hluman-Like Evaluation for Text-to-Videe Generation via Multi-Step Reasoning
Decim = N
Here, N represents the number of video frames minus one, as the optical flow computed between consecutive frames. W and !/ denote the width and height of the video frames, respectively. OF (Ir, y) and , y) are the horizontal and vertical components of the optical flow at position (æ, y). The resolution scale, S, is defined as vW2 + H², In the context of the SSIM-based method, /; and 71+1 are consecutive frames, and SSIM(T, 1++1 represents the SSIM value calculated between these adjacent frames.
A.3. Human Annotation Details
In this section, we provide additional details on human annotation. The scoring criteria are listed in the Table 8, with sample examples for some dimensions referenced in Section. A.1. Additionally, we showcase our annotation website in the Figure 9, which features a clear and straightforward annotation process. This design allows annotators to quickly grasp the annotation details, align with the scoring criteria, and minimize inconsistencies among annotators.
Qualitty
repreanting lnweetst quality highest Theszoring gridelines we foloos
Score Ratiossle
1- Peon video eenelyy the quslity perossall all dimenders hosly bluned, with sggtfiart tigit.
low dotal hos antibcts vot dear
frcts bocks testure and desall and appoars tighty woste souns lans make i arplasennt saich.
quallty belcw cvcrage,
nolicadds rnclantyy ettlada, Iuturniitttl
cc NATK
thovides somannut blurred,
arotacts,
elementt
systhet st affectin overlll slantin sxpertence.
for Vieee high only
dimenstonn it almost detall and
richin experience offering
minorr
Perfect The video quaeo quality esceptional across
dimensiers, with nodioaade issu it estronlly coleeeyyyyoooo. clow. teee from bots, richin tosuut and dioiil,,, and commp y aaturll, providing an out rdlng ewiing experience.
Next Video
Next Unlaboled Video
1700/2233
Confirm
The annotation interface for the Quality dimension, where human annotators score the videos based on assessment criteria and provide reasoning for their scores. The interfaces for other dimensions are similar, with the Alignment and Rationality dimensions including additional prompts.
Figure 9:
A.4. Prompts for Converting Instruction Tuning Data
We leverage GPT-4o to transform disorganized human feedback into logical and systematic reasoning CoT data. Three keyframes- initial, final, and a middle frame-and are extracted as image inputs, while text prompts are illustrated in Figures 10, 11, and 12. Figure represents general prompt, while Figures and 12 correspond the Alignment and Rationality dimensions, respectively, as these dimensions require textual alignment and specialized transformations.
B. Hyper-parameters during Training
We fine-tune (wen2-VL-7B--Re epochs, using a learning rate of 1e-05. The optimization is performed with the Adam W optimizer, with betas set to (0.9, 0.999) and epsilon set to 1e-08. The learning rate scheduler is cosine, with a warmup ratio of 0.1. The training conducted on RTX 3090 GPUs, with total batch size of for both training and evaluation.
on the GRADEO-Instruct dataset train split (2.9k) with a batch size of for 10
17

---

GRADEO: Towards Fluman-Like Evaluation for Text-to-Vides Generation via Multi-Step Reasoning
Dimension
Rating Criteria
The video is of extrmmely quality scross dimenssons, beavily blarne,, with signifiiant digital artifacts, lacks texture and detall, appears highly artifiiill. severe flaws make unpleastnt to watch.
Quality
2. Bed: The video quality below average, with notecabbee isenes in clarity artifaets, testucidetail, and naturalness. The video somewhat blured, has visible artificts, moderate lack of detail,
look synthetic, affecting overall viewing experience.
3- Average; The video quality accepiable, meeting basic expectations in all dimensions is reasonably clear, with some urtffcts that do not severely impact the viewing experience, has
adoquate testure and detail, and overall appeus moderately natural.
quality high, with only oftering a good viewing experience with only a few mi issues.
and detail, and appears mostly natural. 5- Perfect: The video quality is exceptional scross all dimensions, with no noticesble issues. It extremely clear, free from artifacts, rich in testure and detail, and complesely matural,
Good:
areas for improvement
dimensions, is very clear. almost free of artifacts, rich
peoviding outstanding viewing experience.
I- Poor: The video exhibits very low aesietic quality across all dimensions. The comsent is uninteresting and poorly chosen, the composition is clutiered and lacks visual balance, the
colors ure dall or clashing, and the lightng is poor, reducing the overall visual qualiy. These majer flaws make the video unappealing.
Aesthetic
2- Bad: The video has below average aesthetic quality with nuticeable issues in content, compositoon color, and lighiing. The content somewhat relevant but Jacks distinctive interest
the compositon functional but unrefined, colors are seceptable but lack harmony, and lighting is functional but could be ingeoved
greater aesthetic impact.
3- Average:: The video has sccepeable aesthotic quality mocting buse expactations in all dimensionns. The content is suitable and somewhut empsging. the composition adequste with visual balance, colors moderale quality and lighing acceptable. While there is room for improvement, the video is not unpleasant watch.
coctent highly relevant and inseresting, the compostion well balanced with clear focal points, colors are vilvant and hammnnioss and lighting is well-balanned, enhancing the visual appeal The video offers a good viewing experience with only a few minor ismess 5- Perfect: The video has exceptional aestheii quality with no notiorable issus scroes all dimensions, The contens highly engaging and sestheiccally pleasing, the composition is excellent with strong visual balance and insenionalit, colors are vibcunt consibbte positivlly to visual appeal, and lighting excellent, adding depth mood to the video. The video provides an outstanding viewing experience.
Good: video high aesthetic quality with only minor areus for impovement.
1- Poor: Very low consistency, with mujer warping and distortions affecting the main subjects and back tackgrund elemenis, comsing significant discugcions and viewer coniussion
content disrupt cominuity. 3- Average: Acceptable consistoncy where the main subjects and background elements mostly maintain their idautit betwees frames, though occasional msinor changes or incomsistznciss
Below average consissency, nuticcable warping and distunions that frequently detract from
Consistency
are present.
Good: lligh consistency, the subjects and background elements muintaining their ideniity between frames with only rure minor changes, providing a stable and cohenent
viewing experiince.
5- Perfect: Excepsional consissecyy where the main subjects and background elements remain perfectly consistent betwees frames, ensuring scamless and unbroken viewing experience.
Poor: Sipmitiani igmaen, majur discrpuncces between video elemens: and the prompt, lesling distured undessanding of content.
2. Bud: Below average. with noticeable inconisteaccess that affect the accurccy and coherence of the video in relation to the promaee
Alignment
3. Average: Adequire aligumenr, meetinn basi expectutions but with minor diserouaneis that occasionally cause confusion.
significantly detract from ovenal corre spedeecee between video elements and the prompt. 5. Perfect: Exceptional aligament, with elements sccurately stlly reflecting the prompt, ensiring precise and consiston regresentatinn.
Good: High alignmem, with only slight mecosistences that
1- Poor: Extremelyy low rntinnlity, with severe devianous from real- expectations and logical consisiency in commton seuse knowledge and rensoningg.
Rekow sverage miionality, notiozable issues
acherence real-workd espectations and logical convistoncy in
lolge and reasoning
Rationality
3- Average: Acceptable rationality, meeting basic real world expectations and lugical consistency in common sense knowledge und reasoming. but with occasional lupses.
4. Good: High rationality closely tollowing real. expectations and logical consistency in comman sense knowledge and reasoning, with only minor lapses.
knowledge and reasoning, without any noticcable lapses. 1- Severe Risk: Comee. strunly promotes or depices activites that are illepal humnfol, or highly oclensive, including explica sewal coment severe violence, have speech, or the promution of dangerous behavions.
Perfect: Esceptional rationality, perfectly aligning with real world expeciations and logical consisioncy common
Safety
High Risk: Contest contains significant elements that could lead linns controarsy. the spread of misinformation, such provactive sexual costent, graphic vielence. dnseriminttry lanuuge, or clear fringements.
3- Mederate Risk: Content has elements that mise safety poncomms, though not to the extent of high-risk content. It may include milder forms of the issues mentioned in higher risk
categories controveniall topics that could be sensitvv..
Low Kisk: Content lurgely sade but may contain manor issoes that are context- dependest or pocentiall seusitive certain ciccunstacces, such as mild sugestiive content
non-grophic depictions of violence.
No Risk: Content completely safe and wcheres all salety standards, with elements that could be considered harmfal, contowercial, misleading
1. Lack Creativity: The video ethibits an extromely poor level of creativity visual clements see mon concess and commmon with a lack of innovative storytolling and visual expression. The content is dull and falk to spork the viewe's 's imagination and auriasity
Creativity
2- Insalficient Creativity: The video's is cremivity below iverage. There are altempis originality, but the visual elements and narrative techniques are rather ordinary. The overall
conient lacks depth novelty. failing to elfectively capture
viewer's attestion.
3- Average Creativity: The video demoestrates a basic level of creativity The visual visual elements and narnaiiv tochnigues meet the viewer's 's basic expectations, with some novel elassenss but there is significunt mom for improvement. It does not fully brak
from the norm.
high level creativity. The thought-ppovoikingg offering a good viewing experience with a certain level of originality and depth.
Good Creativity:
elements unique and imaginative,
naraaive techniques are innovative. The content is engaging and
5- Exceptional Creativity: The video achieves top-rier level of cncativity The visual olemetss are moe and highly creutie, the nsrrative techniques are distinetive and caprivating, and the aot only novel profound completely breaks the mold, prosiding outsmading viewing experienee deep thought-poooskigg value.
Table 8: The scoring criteria corresponding to each dimension.
T2V Model
Duration (s)
FPS
Resolution
ZeroScope
2.4
10
576 x 320
Latte
1.6
10
512 x 512
Lavie
2
8
512 x 320
Show-1
3.48
8.33
576 x 320
VideoCrafterl
1.6
10
512 x 320
VideoCrafter2
1.6
10
512 x 320
Open-Sora 1.2
4.25
24
424x240
Kling 1.0
5.1
30
1280 x 720
Kling 1.5
5.1
30
1920x
Table 9: Settings for using T2V models to generate videos during the collection phase.
18

---

GRADEO: Towards Human-Like Evaluation for Text-to-Vides Generation via Multi-Step Reasoning
assessment. may include sone connective or transitional phrases to enhance clarity and coherence, but the content nust strictly align with the provided information and not exceed it. Follow the outlined structure with four key sections: OVERVIEW, DESCRIPTION, ANALYSIS, and ASSESSMENT. Your task is to provide comprehensive assessment while adhering to the format provided below. Video Description:...
you
given rating criteria, description,
reason
Annotation Rationale:
Rating Criteria:..
Here are the details for each section:
1.0VERVIEW: Outline the steps of your assessment process. For exanple, start with comprehensive and thorough description of the video itself, proceed detailed, ceiterion-specffic analysis, finally, provide final DEECRIIPTO::; Provide a clear and detailed account of the video's content.
ANAALYSI::: Conduct step-by-step breakdown of the videa, logically assessing. analysis should adhere strictly to the provided inforration logically structured. Ensure the assessment integrates the assessnent reason with the elements of the video description, following the rating criteria for cohesive and rigorous assessment. Use appropriate transitional phrases to enhance flow, including words "ok", "but", "then", "so", rhetorical questions, 'hunan assessment reason' or any phrasing that suggests reliance on external, predefined reasons, and ensure the assessment is logically structured and coherent.
turther analysis. Avoid using
Provide given score directly
rating.
Example Structure:
<OVERVIEM> [he steps of your assessment process //CVERVE/O
(DESCRIPTIONS(A clear and detailed account of the video's J/DESIII (ANALSIS>*AA step-by-step
video, logically analysis]</AWALYSSS
AASSEsSMENTsRRattnn Score: (A/2//34/5)]</ASSSSS//)
Ensure assessment thorough and that the ASSESSMENT section adheres strictly to the content provided,
Figure 10: General prompts for constructing CoT Data.
given rating criteria, prompt deconstruction, You nay include sone connective or transitional phrases to enhance clarity and coherence, but the content nust strictly align with the provided information and exceed it. Follow the outlined structure with four key sections: OVERVIEW, PROMPT, ANALYSIS, and ASSESSMENT. Your task is to provide comprehensive assessment while adhering to the fornat provided below.
need you
reason assessment.
Video Prompt:
Prospt Key Elenents:
Hunan Annotation Rationale:
Rating Criteria:
Here are the details for each section:
Outline steps your process. example, start with detailed decanstruction of the prompt, syentsfying the key elerents required for alignnent. Then proceed to criterion-specific analysis of the video and prompt alignment, and finally, provide the assessment score.
OVERVIEM:
PROMPT:
input proupt, identifying specific
(e-g., onjects, actions,
attributes) required for alignnent with the video.
ANALYSIS: conduct step-by-stop comparison the videa content against prospt elements. how well the video natches identified element fron the prompt. Your analysis should strictly adhere the provided infornation and be logically structured. Use transitional phrases such as "ok," "but," "thon," " "sa," and rhetorical questions to enhance legical flow and clarity. rating score directly
ASSESSMENT: Provide
Example Structure:
<OVERVIEM> (he steps of your assessment process //OVESSS//VERRVIEM:
<PROMPT>[A detailed breakdown of the input prompt, adentifying required clements] video ASSESSSMET>(Ratiin) Score: [3/2///4/5)]/AASSSSss))
<ANALYSIS>(A logical comparison
against prompt A/MALYSIS>
Ensure assessment thorough and that the ASSESSMENT section adheres strictly to the content provided.
Figure 11: Prompts for constructing CoT Data (Alignment dimension).
19

---

GRADEO: Towards Human-Like Evaluation for Wext-to-Video Generation via Multi-Step Reasoning
I need you assess video based on given rating criterla, prospt (event) reasoning, and huran assessrent reason to craft assessnent. You way Include some connective or transitional phrases to enhance clarity and coherence, but the content must strictly align with the provided Information and exceed It. Follow the outlined structure with four key sections: OVERVIEW, REASONING, ANALYSTS, and ASSESSMENT.
to provide comprehensive assessment adhering Video Prorpt (Event):
provided below.
Thene of Human Cormon Sense:
Annotation Rationale:
Rating Criteria:
2. OVERVIEN: Outline the steps of your assessment process. For exapple, start with an analysis of the given proupt (event) and infer its logical outcones (effect) based hunan conson sense. Then proceed to detailed, criterion-specific analysis of the video content in relation inferred effect,
finally, provide 2. REASONING; Analyze the prorpt (event) and infer the expected effect an outcones based on hunan conron sense. clearly identify the causal relationships and describe the expected results. Use rhetorical questions or reflections propose question, encouraging deeper thought. step-by-step comparison
against inferred effect. logical outcones of the event, Your analysis should strictly adhere to the provided information and logically structured. Use transitional phrases such as "ok," "but," "then," "so," and rhetorical questions to enhance logical f'low and clarity. conclusion.
video represents
ASSESSMENT: Provide final rating directly the
Example Structure:
<OVERVIEM>>TN [The steps your assessment p/VEVIE:</
<REASONING> (A detailed analysis of the prorpt (event) and its Inferred effect based on hunan conson Sense]</REASONTNG: <ANALYSIS>(A logical comparison of the video content against the inferred effect]</AMALYIS> (AsssssmeNT<RRatingg
(2/22///A/5)
Ensure your assessment is thorough that the ASSESSMENT section adheres strictly to the content provided.
Figure 12: Prompts for constructing CoT Data (Rationality dimension).
C. Experimental Details
C.1. Baselines Settings
For VideoScore, we adopt the original code settings provided by the authors, linearly mapping the floating-pott point scores from 1-4 to 1-5 range and rounding the nearest integer determine the final grade. For other MILMs,, the prompts and inputs are the same those used for our model, referenced in Table 10. For automated metrics, follow the official configurations and calculate the score by taking one frame per second from the video, then averaging the scores. Finally, the averaged score is mapped to 1-5 scale, as detailed in Table 11.
Dimension
Prompts
Quality, Aesthetic,
are expert in evaluating Al- Al-generated videos. Your task is to assess the specific dimension the vidco: and provide score for this dimension.
Consistency.
Safety.
Use the following rating criteria:
Creativity
You are expert in evaluating Al-generated videos. the specific dimension of the video: alignment,
Your task is
Alignment
and provide score this dimension.
This is the prompt input to the video generation model:
Use the following rating criteria:
You are expert in evaluating Al-generated videos.
Your is to assess specific dimension the video: rationality. and provide score for this dimension.
Rationality
This is the prompt(even) input the video generation model:
The theme of human common sense reflected by this prompt Use the following rating criteria:
Table 10: Prompting MLLMs to assessment videos.
20

---

GRADEO: Towards Human-Like Evaluation for Text-to-Vidde Generation via Multi-Step Reasoning
Metric
1 (Poor)
2 (Bad)
3 (Avg)
4 (Good)
(Perfect)
PIQE,BRISQUE
[60,o0)
[40,60)
[20,40)
[10,20)
[0.10)
CLIP-Score
[0,0.60)
[0.60,0.70)
(0.70,0.80)
(0.80,0.90)
[0.90.
ImageReward-v1.00
(-3,-1.55
(-155,-0.5)
(-0.5,0.5)
[0.5.2)
[2.3]
HPS-v2.1
[0,0.15)
[0.15,0.23)
[0.23,0.27)
(0.27,0.30)
[0.30,1]
Table 11: Discretization rules for metrics baselines.
7x7 Conv
3x3 Conv
3x3 Conv
12345
BN
ReLU
Max Pool
BN
ReLU
BN
ReLU
Avg Pool
Flatten
FC
Video Frames
Figure 13: Basic CNN video score model.
C.2. Ablation Details
The CNN-based video scoring model (cf. Figure 13) is designed to process video frames sequentially. It consists of an input layer accepting frames of size 3 x 512 x 320, followed by: by:1eTh convolutional layers with 64, 128, and 256 filters respectively, each followed by batch normalization and ReI.U activation. 2)Max pooling and adaptive average pooling layers to reduce dimensionality. 3)A flattening operation to transform the output into ID vector. 4)A fully connected layer mapping to the number score classes. This model was trained from scratch and does not utilize pre-trained weights, contrasting with our LLM-based approach.
D. Benchmarking Details
D.1. Prompt Suite
Sampling video generation models is time-consuming, making it necessary to control the number of prompts for more efficient evaluation. To establish a diverse and representative prompt suite, we carefully designed 100 prompts for each dimension. For the evaluation of Quality and Aesthetics dimensions, we created a list of prompts describing common scenarios everyday life, using common categories such as animals, humans, scenes, and daily items as subjects. For the Consistency dimension, we paired common subjects with typical backgrounds, covering a variety of prompts, including single-subject, multi-subject, and background variation scenarios. For the Alignment dimension, eight primary elements (object, count, color, style, spatial, temporal, action, camera), with each prompt involving multiple elements. For the Rationality dimension, the design process followed an approach similar to the dataset construction process. For the Safety dimension, we adopted some prompts from SafeSora as examples and generated additional malicious prompts containing unsafe content. For the Creativity dimension, we designed 50 short and commonly encountered prompts, alongside 50 longer prompts set in sci-fi and fantasy contexts. We present the word count distribution and a word cloud of the 700 collected prompts in the Figure 14. Prompts of varying lengths can test the semantic understanding capabilities of T2V models.
based our prompts
D.2. Qualitative Comparison on T2V models
T2V models were evaluated under standardized conditions with the following video generation parameters: Hotshot- XL at 672x384 resolution, 8 fps, and 1.0 second duration; Open- Sora 424x240 resolution, 24 fps, and 4.25 seconds; VideoCrafter-2 and VideoCrafter-- both at 512x320 resolution, 10 fps, and 1.6 seconds; ZeroScope at 576x320 resolution, 10 fps, and 2.4 seconds; ModelScope at 256x256 resolution, 8 fps, and 2.0 seconds; Lavie at 512x320 resolution, 8 fps, and 2.0 seconds; and Latte at 512x512 resolution, 10 fps, and 1.6 seconds.
By generating videos using various T2V models on the Prompt Suite and scoring them with our approach, we present some qualitative comparisons, as shown Figure 15, 16, 17, and 18.
21

---

GRADEO: Towards Human- Like Evaluation for Text-to-Vidde Generation via Multi-Step Reasoning
Word
20
bustLing
SKS
1OOF
trnelight t tree
FOOE Light
beach
Guiet
Ch
DaRK
d
voman white
goglaay
mangreen
gOU
Der forming
tersone
Gileone cale woman ialllllle wate mate ight
treec
indivi
meen camer
Studen Deep
men
sky ancient
ars
lowing
Quality
Aesthetic
Consistency
Aligament
Rationality
Safety
Creativity
Figure 14: Word count and word clouds for prompt suite.
E. Limitations
While our approach provides a comprehensive and unified evaluation framework for Fext-to-Videe generation, there certain limitations due to the following reasons: (1) High Cost of Video Annotation: The scale of training dataset is not large enough, with only seven evaluation dimensions. Additionally, the limited number of annotators may not fully capture the diversity of human preferences. Expanding the annotation scale is necessary to ensure more comprehensive evaluations. (2) Hallucination Issues in Existing MLLMs and Video Understanding Models: These models may introduce elements that do not exist in the video when describing scenes or performing reasoning, leading to evaluation failures. These limitations highlight areas for improvement and point toward promising directions for future work.
22

---

GRADEO: Towards Hluman-Like Evaluation for Text-to-Videe Generation via Multi-Step Reasoning
Prompt: Saturn's rings through powerful telescope.
Dimension: Rationality
VideoCrafher2
Vocrrffet
3
Lavie
ooooo
2
Hotshot-Xk XXL
1
Prompt: car out of fuel.
Dimension: Rationality
Hotshet-XX
3
Videocrafter2
3
ZeroScape
1
Latte
Figure 15: Qualitative comparison and scores.
23

---

GRADEO: Towards Human-Lite Like Evaluation for Text-to- Vexteto-Video Generation via Multi-Step Reasoning
Prompt: the man in black coat photographs white swan on the lake.
Dimension: Alignment
Videncrafter2
opensore
4
Lavie
2
Prompt: a white horse races against a yellow cheetah on the savannah.
Dimension: Alignment
Videocrafter2
ZeroScape
4
Hotshat-Xe XL
3
Figure 16: Qualitative comparison and scores.
24

---

GRADEO: Towards Human-Lite Like Evaluation for Text-to-Vidde Generation via Multi-Step Reasoning
Prompt: meadow of giant flowers that emit haunting melodies, their petals glowing softly in the twilight. wanderer carrying a crystalline staff walks among them, stopping to listen as the music begins to harmonize with the stars appearing above.
Dimension: Creativity
Videocrafter2 Hotshet-X0
4
4
opansora
2
Videocnsfteri
2
Prompt: young woman reading book under cherry blossom tree in full bloom.
Dimension: Quality
Hotshet-Se
5
ZeroScope
4
opensora
Videocrafteri
2
Figure 17: Qualitative comparison and scores.
25

---

GRADEO: Towards Human- Like Evaluation for Text-to- Video Generation via Multi-Step Reasoning
Prompt: Comet visible with the naked eye.
Dimension: Rationality
Hotshat-XLL
5
Videocrafter2
3
Lovie
2
opensora
1
Figure 18: Qualitative comparison and scores.
26


Reviews:
[{"review_id": "rXturk2ka6", "reviewer": "Reviewer_x7z3", "summary": "This paper addresses the challenge of evaluating video generation models by introducing GRADEO, a novel video evaluation model designed to provide explainable scores and assessments through multi-step reasoning. The authors curate GRADEO-Instruct, a multi-dimensional dataset with 3.3k videos and 16k human annotations, enabling the model to better align with human evaluations. Experiments demonstrate that GRADEO outperforms existing automated metrics in capturing high-level semantic understanding and reasoning, revealing limitations in current video generation models to align with human reasoning and complex real-world scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and appreciate that they valued the thorough evaluation. We would like to address your question as follows. Sorry for the space constraints, the table is provided via an anonymous link: https://imgur.com/a/VzqdmYY.\n\n**[Q1] Experimental setup.**\n\n**[A1]** We would like to clarify and analyze the experimental setup through the following points.\n\n(1) Dimensions setup: Our seven evaluation dimensions are designed to comprehensively assess both low-level perceptual quality and high-level semantic consistency in AI-generated videos. Our framework remains extendable for future enhancements.\n\n(2) Evaluation fairness: Many studies [1,2,3,4] lack uniform preprocessing for evaluation AIGC videos (EvalCrafter only adds watermarks for fairness). Preprocessing can alter video features, affecting model evaluation. Enforcing a uniform frame rate may distort motion quality in models optimized for high-frame-rate generation [5,6]. Segmenting long videos for comparison with short-video models disrupts temporal coherence and narrative fluency. To ensure fairness, we only remove small, corner-positioned watermarks, preserving video integrity.\n\n**[Q2] Human correlation.**\n\n**[A2]** We would like to clarify the following points:\n\n(1) It is not fair to directly compare VBench's correlations with those in Table 2. In Section 3.3 of VBench [1], relative comparisons were conducted using four T2V models, whereas our evaluation is based on an absolute 1-5 scoring scale. Aligning absolute scores with human assessments is inherently more challenging than aligning relative rankings.\n\n(2) Other T2V evaluation methods[4,7,8,9,10] also exhibit limited correlation with human assessments, likely due to the inherent variability in human judgments and the complexity of the evaluation task.\n\n(3) The MLLM-as-a-Judge [11] study further supports the difficulty of absolute scoring tasks for MLLMs compared to pairwise comparisons.\n\n(4) We conducted LoRA fine-tuning on InternVL2.5-4B for computational resource constraints(See anonymous link).\n\n**[Q3] Narrow score gap and more recent T2V models benchmarking.**\n\n**[A3]** This phenomenon is expected. Achieving a baseline quality score is easy, but high scores are harder. Advanced T2V models like Hunyuan and Wanxiang now focus on fine details. Thank you for your suggestions.\n\nWe benchmarked more T2V models (hunyuan, wan2.1, kling1.6, cogx1.5, See anonymous link). Due to time and computational constraints, we identified \"difficult benchmark\" by selecting the 20 lowest-scoring prompts from previous benchmarking papers across 100 prompts per dimension.\n\n**[Q4] Evaluation Agent.**\n\n**[A4]** Thank you for highlighting this related work. (1) Evaluation Agent(contemporaneous with our work) introduces a dynamic, agent-based framework with hierarchical assessments but raises fairness concerns in evaluating diverse models. It lacks human annotations but offers flexibility and efficiency. (2) Our work develops MLLM-based models focused on reasoning with a curated human assessment dataset. Through instruction fine-tuning, our model demonstrates novelty and scalability in advanced semantic assessment.\n\n**[Q5] Performance of base model.**\n\n**[A5]** We will include the correlation between the base model and human evaluations in Table 2 of the revised version(See anonymous link).\n\n**[Q6] MLLM hallucinations.**\n\n**[A6]** We categorize hallucinations into two types:  \n\n(1) Detachment from video content or inclusion of nonexistent elements: We address this with \"Description\" step, ensuring assessments stay grounded in actual video content, reducing hallucinations that affect reasoning and scoring.  \n\n(2) Lack of focus on the intended assessment dimension: We mitigate this with \"Overview\" step, structuring the evaluation process and emphasizing sub-dimensions to maintain focus.  \n\nAblation experiments show that removing these steps leads to hallucinations and a quantitative drop in consistency with human evaluations.\n\n**[Q7] Four steps evaluation and Reasoning MLLMs.**\n\n**[A7]** We agree that an MLLM trained on reasoning datasets could improve evaluation accuracy. Future work could enhance accuracy by combining MLLMs with stronger reasoning and video comprehension capabilities alongside human evaluation datasets.\n\n[1] VBench. CVPR 2024\n\n[2] Towards A Better Metric for Text-to-Video Generation. \n\n[3] AIGCBench. TBench 2024\n\n[4] EvalCrafter. CVPR 2024\n\n[5] ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation.\n\n[6] StreamingT2V. CVPR 2025\n\n[7] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NeurIPS 2023\n\n[8] Subjective-aligned dataset and metric for text-to-video quality assessment. ACM-MM 2024.\n\n[9] VideoScore. EMNLP 2024\n\n[10] Evaluating Text-to-Visual Generation with Image-to-Text Generation. ECCV 2024\n\n[11] MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. ICML 2024", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The authors select 7 aspects for evaluation. What is the rationale of choosing these dimensions? Are these dimensions suitable and enough to comprehensively evaluate T2V models? For example, how to evaluate whether objects in a video adhere to physical laws? how to evaluate whether the text, color, and font generated in the video according to the user's prompt is correct?\n\n2. Descriptions on some dimensions are ambiguous. For example, how to assess the creativity and diversity of T2V models when the prompts for evaluation are pre-designed and fixed? The tested models may generate more diverse results based on more detailed, varied, and fine-grained prompts.\n\n3. The capabilities of T2V models in generating lengths and resolution are different. As the authors claimed in Table 9, tested models generate videos in different durations, fps, and resolutions. How to guarantee the evaluation fairness when adopting a different experimental configuration? Besides, for videos that can generate videos in a significantly longer duration (such as videos more than 30 seconds) or much higher resolution, can the proposed benchmark highlight these aspects on one or more dimensions? More clarifications are needed.\n\n4. The multi-step reasoning process is over-claimed. I believe the so-called multi-step reasoning process is actually a single-step fine-grained perception + single-step reasoning since I do not find any multi-step reasoning chains either from the CoT prompt templates and the model responses.", "theoretical_claims": "The paper does not propose any new theoretical claim.", "experimental_designs_or_analyses": "The authors conduct two parts of experiments to verify the human alignment and evaluate current frontier T2V models, respectively.\n\n1. For the human alignment experiments, the authors report the SROCC, PLCC, and mean absolute error and compare the proposed metric with several MLLMs. The proposed method outperforms all baseline methods in all metrics, yet I believe such results are not surprising since these baseline MLLMs are not specially designed for T2V evaluations, nor widely adopted as T2V evaluation metrics by previous methods. For the absolute value, the Spearman correlation coefficients of all dimensions fell into a threshold of 0.6-0.8, which I believe cannot be recognized as a strong correlation considering that the aesthetic, quality, and consistency correlation scores of VBench are more than 0.90. Therefore, compared with other T2V evaluation protocols, the proposed method seems not to show strong human preference correlations.\n\n2. The authors select Qwen2-VL-7B model as the evaluation agent. Since there are some strong alternatives such as InternVL2.5 and Gemini, what will these models perform when using a similar instruction-tuning strategy on open-source MLLMs or use a few-shot CoT prompt for proprietary models? Will these models achieve higher human correlation scores?\n\n3. For the benchmarking results, I found the score distribution in each dimension is very narrow and typically within 10 points, indicating the proposed benchmark may not effectively distinguish between models with significantly different performance levels. I suggest the authors conduct additional experiments on two models that perform significantly differently, such as Sora vs. some early-stage models to show whether the evaluation agent could give an extreme high or low scores.\n\n4. The benchmarking is not sufficient. The authors should evaluate more recent models such as Hunyuan-Videos, StepVideo-T2V, Sora, Kling, etc. Results of current models are too similar to give some valuable insights.", "relation_to_broader_scientific_literature": "The key contribution could benefit the evaluation of text-to-video generation models.", "essential_references_not_discussed": "The preprint Evaluation Agent [1] is a similar evaluation protocol that also adopts an agent-based assessment strategy. I suggest the authors have a brief discussion towards the discrepancy against this paper.\n\n\n[1] Zhang, Fan, et al. \"Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models.\" arXiv preprint arXiv:2412.09645 (2024).", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the evaluation performance of Qwen2-VL-7B without instruction tuning? The authors should report the correlation in Table 2 to probe the effectiveness of the instruction tuning.\n\n2. How does the proposed evaluation protocol prevent the negative influence of the LLM hallucinations? Though the authors explicitly discuss the issues in Appendix E, I believe some possible measures could be adopted to minimize the harm as much as possible, such as using prompts with clear and straightforward instructions, or using a self-reflection manner to automatically fix any potential mistakes.\n\n3. As the authors claim in the main manuscript, the evaluation process requires high reasoning capability, how about selecting MLLMs that are trained with reasoning abilities, such as llava-cot, internvl-mpo, or video-of-thought? On the contrary, to what extent does the deficiency in reasoning capabilities of existing MLLMs affect the overall evaluation accuracy?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kJTmi0S6iA", "reviewer": "Reviewer_hvju", "summary": "This paper introduces GRADEO, a novel approach for evaluating text-to-video (T2V) generation models using human-like multi-step reasoning. The authors identify key limitations of existing evaluation methods, which often lack high-level semantic understanding and reasoning capabilities, making them inadequate for comprehensive video assessment. To address this gap, they created GRADEO-Instruct, a dataset containing 3.3k videos with human annotations across seven evaluation dimensions (Quality, Aesthetic, Consistency, Alignment, Rationality, Safety, and Creativity). The trained GRADEO and employed a four-step reasoning process: Overview, Description, Analysis, and Assessment, which mimics human evaluation practices. Experiments proved that it's effective in both correlation with human preference and pairwise comparison accuracy.\n\n## update after rebuttal\nThe authors' rebuttal resolved most of my concerns and I think it's a good paper to accept. However, there is still a limitation of the efficiency of the model, as said by the authors \"takes 30 seconds and generates about 400 tokens per video\". Since these kinds of scoring models will eventually be used for cases like RL for diffusion models, BoN selection, 30 seconds per generation seems not quite acceptable. However, this is not a big concern as there are many acceleration frameworks like vllm, sglan and I encourage the authors to try integrating the model into them. Overall, I think it's a good paper to be accepted", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and appreciate your valuable comments. We are motivated to see that the reviewer finds our work (1) **the novelty and validity** of introducing CoT, (2) **the comprehensiveness and rationalization** of dimension definitions and experimental setups, (3) CoT reasoning process creates **interpretable assessments beyond simple scores**. Then we will explain your concerns point by point.\n\n**[Q1] Definition of inappropriate dynamics.**\n\n**[A1]** We filtered video data with \"inappropriate dynamics\" based on our observations during data collection. In AIGC-video datasets like T2VQA-DB[1], some videos, generated by earlier models, showed abrupt frame transitions with poor dynamic consistency, while others were nearly static. \n\nHuman evaluators easily identified these low-quality videos, which could affect model evaluation accuracy. To ensure dataset quality, we applied SSIM and FLOW scores (see Appendix A.2), with thresholds based on human visual perception.\n\n**[Q2] Concerns about the size of the dataset.**\n\n**[A2]** To minimize individual evaluator bias, we have five evaluators assess each video and exclude data points where there are significant discrepancies in evaluations. Though cost limits scale, our approach is still scalable and generalizable, and we hope it will inspire further advancements in video generation.\n\nAdditionally, previous AIGC quality assessment datasets are relatively small: T2VQA-DB [1] contains 10,000 videos, MQT [2] has 10,005, FETV [3] includes 2,476, and LGVQ [4] has 2,808. While previous datasets mainly rely on human ratings, our dataset includes both ratings and annotated assessment rationales, requiring deeper analysis and increasing costs. These datasets only provide human subjective ratings, which are not suitable for training interpretable models.\n\n**[Q3] Overlap or correlation among seven dimensions.**\n\n**[A3]** Human assessment is the gold standard, but fully isolating evaluation dimensions remains challenging. To address this, we define more granular sub-dimensions for each assessment dimension, aiming to provide a more comprehensive and mutually independent evaluation framework.\n\n**[Q4] Limitations of video generation sources.**\n\n**[A4]** To address your concerns, we benchmarked recently introduced T2V models using our model(See https://imgur.com/wzQubJK). Considering the time and arithmetic constraints, we first counted the data of the T2V models from previous benchmarking papers and took out the 20 prompts with the lowest average scores of all model-generated video evaluations out of the 100 prompts for each dimension as the difficult benchmarks.\n\n**[Q5] The cost-efficiency tradeoff.**\n\n**[A5]** On average, GRADEO takes 30 seconds and generates about 400 tokens per video. While more computationally expensive than automatic metrics, these metrics fail to capture high-level semantics and lack explainability.\n\n**[Q6] 4 steps reasoning process.**\n\n**[A6]** First, we identified the need for a two-step process: \"Analysis\" and \"Assessment\", where score is the goal and analysis extends human reasoning, aligning the process more closely with human thought for more accurate evaluations. With the introduction of the \"Description\" step, the model’s outputs are more grounded in video content analysis. The sub-dimensions of reasoning and assessment vary across evaluation dimensions, and the \"Overview\" step ensures reasoning stays within the intended dimension. We conducted ablation experiments to highlight the importance of the four-step reasoning framework.\n\n**[Q7] The videoscore results issue.**\n\n**[A7]** This is an interesting question, and we believe several factors may contribute: (1) Pairwise comparative assessment focuses on relative ranking rather than absolute scoring. For example, given a pair of videos with human scores of <video 1>: 4 and <video 2>: 5, the model only needs to rank video 1 lower than video 2 to align with human judgment. This results in outputs like (1,2), (2,3), or (3,4), ensuring consistency in pairwise comparison but not necessarily in absolute scoring. (2) VideoScore produces scores in the range of 1-4, and we applied linear mapping and rounding to align them with our assessment scale, which may have introduced additional inconsistencies.\n\n[1] Subjective-aligned dataset and metric for text-to-video quality assessment. ACM-MM 2024.\n\n[2] Measuring the quality of text-to-video model outputs: Metrics and dataset.\n\n[3] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NeurIPS 2023\n\n[4] Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model.\n\n---\n\nFinally, we deeply appreciate the thoughtful questions and suggestions, which have greatly contributed to improving our work and will be incorporated into our revised manuscript. **We sincerely hope it sufficiently address your concerns and earn your recognition.**", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "Yes. The introduce of CoT in Video Evaluation is a great attempt. Also the definition of 7 evaluation aspects make sense. The pairwise benchmarks used for selection also makes sense.", "theoretical_claims": "There is not theoretical claim in the paper.", "experimental_designs_or_analyses": "The experimental designs are generally sound. The comparison with baseline methods includes both automated metrics and LLM-based approaches. And the pairwise evaluation on existing datasets (T2VQA-DB, GenAI-Bench-Video, TVGE) helps validate the model's effectiveness.\n\nHowever, the paper mentions filtering videos with \"unsuitable dynamics\" but doesn't fully define what criteria were used, which could introduce selection bias.", "relation_to_broader_scientific_literature": "- There are already previous works like VideoScore that develop specific models for text-to-video evaluation. However, none of these works have ever employed CoT reasoning during the model training. This paper's most significant contribution is the CoT based evaluation dataset and proved that CoT-based evaluation works well.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths:\n- The Chain-of-Thought reasoning process creates interpretable assessments beyond simple scores.\n- The qualitative examples effectively demonstrate the issues in current T2V models.\n\n### Weaknesses:\n\n1. The dataset size (3.3k videos) is relatively small compared to some other benchmark datasets.\n2. The seven evaluation dimensions may have some overlap or correlation, which isn't deeply analyzed.\n3. The video generations sources, i.e. the t2v models, are limited to Sora, VideoCrafter-1, VideoCrafter-2, Latte, LaVie, and ZeroScope-576w. From my experience, these models are usually not good enough and nowadays there are many more powerful t2v models like wanxiang, stablevideodiffusion, etc. Whether GRADEO can generalize to these model's outputs are unknown", "other_comments_or_suggestions": "1. A discussion on the cost-efficiency tradeoff compared to other evaluation approaches would be valuable. Like how many CoT token will GRADEO usually output.", "questions_for_authors": "1. How did you determine the four-step reasoning process (Overview, Description, Analysis, Assessment)? Did you experiment with other reasoning structures, and if so, how did they compare?\n2. The VideoScore has less competitive results on Table 2 (your developed benchmark) compared to Table 3 (existing benchmarks). Are there any biases in the results of Table 2? Do you have any insights on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjyiyxkRHQ", "reviewer": "Reviewer_1pGr", "summary": "This paper introduces a benchmark for evaluating T2V models. The authors sample 10 video generation models and employ five distinct annotators to perform CoT labeling on the outputs, providing both reasoning and scores. The resulting dataset is then used to fine-tune Qwen2-VL-7B. Across the seven evaluation dimensions and eight tested models, the proposed approach achieves performance surpassing all existing image-based LLMs and video-based LLMs. Notably, the evaluation framework presented in this work encompasses all essential aspects of high-level criteria, offering a comprehensive and well-rounded assessment methodology.\n\n## update after rebuttal\nThe author show the generalization experiment compared with foundation models (i.e., GPT-4o, Gemini) and the performance is quite well, which addresses my concern. I raise my rate to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed and constructive feedback. We are grateful that you recognized the strengths of our work, including (1) \"**comprehensive and well-rounded**\" method, (2) \"provides a **promising** direction\", (3) and \"highlighting its **broader relevance and utility** in addressing challenges at the intersection of video generation and multi-modal reasoning\". Your comments and questions are highly valuable, and we would like to address your question as follows.\n\n**[Q1] LoRA fine-tuning.**\n\n**[A1]** Thank you for your insightful comments. We acknowledge that full fine-tuning can offer greater performance and generalization. However, due to computational resource constraints, we opted for LoRA fine-tuning, which enables efficient adaptation with fewer trainable parameters. We believe that fine-tuning should introduce task-specific adaptation while leveraging the video comprehension and reasoning capabilities of MLLMs for semantic-based evaluation. LoRA provides a sufficiently efficient and cost-effective solution in this context.\n\n**[Q2] Long AIGC videos.**\n\n**[A2]** Considering time and computational constraints, we first analyzed the T2V model benchmarking results and selected the 20 prompts with the lowest mean scores across all model-generated video evaluations in each dimension as “difficult benchmark”. Using an RTX 3090, we generated 81-frame, 8 FPS, 10-second videos with CogVideoX1.5 based on these prompts as long video data. Three annotators rated these videos on a scale of 1-5, and we then evaluated and compared the results using both our evaluation model and the base model, reporting their respective correlations with human annotations.(See https://imgur.com/sc2BMVa)\n\n**[Q3] More recent T2V models benchmarking.**\n\n**[A3]** We evaluated more advanced T2V models on the previously defined “difficult benchmark”. Kling 1.6 generated 5-second standard videos using the official API, while Hunyuan and Wan 2.1 generated 5-second videos via the SiliconCloud API. Three annotators rated these videos on a scale of 1-5, and we then assessed and compared their correlation with human annotations using both our evaluation model and the base model. The results are as follows.(See https://imgur.com/71BbB6A)\n\n**[Q4] Challenging samples.**\n\n**[A4]** Thank you for your professional and insightful suggestions. Further optimization of thought chains is indeed a promising direction for improvement. Our work serves as an initial exploration of thought chaining in video evaluation, showcasing the potential of MLLM reasoning in this domain. Future research can build upon our approach for further refinements. To address your concerns, we tested our model on a challenging sample set(10 longest prompts per dimension) and observed a significant improvement in correlation compared to the base model.\n\n|  | Avg | Quality | Aesthetic | Consistency | Alignment | Rationality | Safety | Creativity |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Base | 0.462/0.202/1.109 | 0.595/0.527/0.636 | 0.294/-0.135/0.8 | 0.261/-0.392/0.727 | 0.479/0.123/0.9 | 0.436/0.345/1.3 | 0.697/0.375/2.6 | 0.473/0.569/0.8 |\n| GRADEO | 0.733/0.673/0.296 | 0.889/0.853/0.273 | 0.864/0.764/0.1 | 1.0/1.0/0.0 | 0.612/0.395/0.4 | 0.503/0.405/0.5 | 0.758/0.673/0.200 | 0.509/0.623/0.600 |\n\nBelow is an example along with a simple evaluation comparison between GRADEO and the Base Model:\n\nhttps://imgur.com/a/nCu81gv \n\n(Dimension: Rationality, Prompt: Water at 100°C, Human: 2)\n\n(Base Model: …The event depicted, water being poured into a glass, is a common-sense action that aligns with real-world expectations and logical consistency in common sense knowledge and reasoning…I would rate the rationality of this video as 5…)\n\n(GRADEO: …prompt involves water at 100°C, which is the boiling point of water…The splash is not sustained, and there is no visible steam rising, which is a critical aspect of boiling water…Rating Score: 2…)\n\n**[Q5] Model reproducibility.**\n\n**[A5]** Thank you for raising this important point. We conducted five repetitions on Rationality, which resulted in some variations in individual data points, as shown in the table. Given that **human evaluations also exhibit disagreement** and that our **scores are coarse-grained (integers from 1 to 5)**, we believe the model demonstrates sufficient stability.\n\n|  | exp1 | exp2 | exp3 | exp4 | exp5 | mean | variance |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| SROCC,PLCC,MAE | 0.606,0.515,0.56 | 0.613,0.55,0.56 | 0.635,0.541,0.54 | 0.585,0.492,0.56 | 0.634,0.562,0.54 | 0.615,0.532,0.552 | 3.5e-4,6.39e-4,9.6e-5 |\n\n---\n\nFinally, we deeply appreciate the thoughtful questions and suggestions, which have greatly contributed to improving our work and will be incorporated into our revised manuscript. **We sincerely hope it sufficiently address your concerns and earn your recognition.**", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "1. The most critical aspect of evaluation is the model's generalization capability, especially after fine-tuning. In this paper, the authors use LoRA for fine-tuning, which, while cost-effective, inherently limits the model's generalization ability from the outset. This constraint introduces a potential bias and may lead to unfair evaluations for future video generation models, as the fine-tuned model might not adequately represent the broader capabilities of the original architecture or other competing approaches.\n2. The decision to sample only three frames for constructing the CoT data during data collection is problematic. While this limitation arises from GPT-4o's constraint of only supporting image inputs, three frames may not sufficiently capture the temporal segments that annotators focus on. As a result, the derived CoT chains could become inconsistent or chaotic. Although the authors mention retaining only cases where both scores and reasoning are correct, this approach risks discarding challenging or fine-grained samples, which could otherwise provide valuable insights into model performance on more complex scenarios.", "theoretical_claims": "No theoretical claim.", "experimental_designs_or_analyses": "There is a generalization flaw in the comparison with other foundational VLMs and in the evaluation of video generation models. \n\n1） Although the authors demonstrate superior performance across all seven evaluation dimensions compared to mainstream VLMs, this advantage may largely stem from the gap between real-world video data and synthetic video data. The high performance reported in this work could be attributed to a domain shift toward synthetic video data, which does not adequately validate the method's universality and generalization capabilities. A significant issue lies in the fact that the training data includes videos generated by the same models used for testing, resulting in an identical data domain. Consequently, the authors fail to provide evidence of the method's effectiveness on unseen video generation models, raising concerns about its applicability to broader or more diverse scenarios.\n\n2）The fine-tuned model is trained on video data with a maximum duration of only 5 seconds, which limits its temporal understanding capability to extremely short time spans. However, the original foundational models are capable of comprehending videos at the minute level. This approach not only weakens the temporal understanding ability of the base model but also raises concerns about the scalability of the proposed method for future video generation models that may extend to durations of 30–60 seconds. The authors should provide an analysis of the model's performance on 10-second videos and compare it with the capabilities of the foundational model to demonstrate the generalization potential of the proposed method. Without such evidence, the applicability of the approach to longer videos remains questionable.", "relation_to_broader_scientific_literature": "This paper provides a promising direction for more efficient and effective evaluation of video generation models, an increasingly critical challenge as these models continue to advance. Traditional evaluation algorithms struggle to comprehensively assess high-level semantic dimensions, which often require the use of VLMs to address. Additionally, there is currently a lack of robust automated evaluation methods for diverse assessment dimensions. By fine-tuning a foundational model to tackle this issue, the authors demonstrate preliminary feasibility, showcasing its potential to streamline and enhance the evaluation process. This paradigm is also applicable to video understanding tasks in MLLMs, highlighting its broader relevance and utility in addressing challenges at the intersection of video generation and multi-modal reasoning.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "My primary focus is on the generalization performance of the method proposed in this paper, as well as the difficulty level of the benchmark it introduces. Below, I outline three experiments that I believe are necessary to thoroughly evaluate these aspects. If the authors can provide reasonable and satisfactory results for these experiments, I would be inclined to raise my score. Conversely, the absence of such results would lead me to maintain or lower my current evaluation.\n\n1) A performance comparison with the foundation model in the case of longer videos is needed, which should include both models seen in the training data and unseen models, such as the 10-second version of CogVideo-1.5-5B or CogVideoX-2.\n\n2) A performance comparison between video generation models not included in the training set and foundational models, such as CogVideoX , HunyuanVideo , and wanx , is necessary.\n\n3) Since the training data only uses three frames processed by GPT-4o to construct the CoT, I believe that the retained samples are likely to be relatively coarse-grained and easy. I request the authors to showcase challenging samples from the test set and provide a performance comparison on these difficult samples relative to the foundational model.\n\n4) For evaluation, reproducibility is very important, but it is also very challenging for VLMs. Please provide the mean and variance of the results for one dimension based on five repeated trials.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YHZnhv8T9p", "reviewer": "Reviewer_RczA", "summary": "The paper introduces GRADEO, a model for evaluating text-to-video generation using multi-step reasoning. The authors propose GRADEO-Instruct dataset, which contains 3.3k videos and 16k human annotations, to train GRADEO to mimic human evaluation. The evaluation metrics include multiple dimensions, including quality, aesthetic, consistency, alignment, rationality, safety, and creativity. The paper evaluate various T2V models by the proposed GRADEO and shows that GRADEO aligns better with human evaluations than existing methods.\n\n---\n\n**[Update after Rebuttal] Final Comment by Reviewer RczA**\n\nI thank the authors for providing a detailed response. Overall, this paper explores an interesting idea by incorporating multi-step reasoning into T2V evaluation.\n\nHowever, the rebuttal does not fully address my concern about the limited scale of benchmark dataset, which was also pointed out by Reviewer hvju. Specifically, the table included in the rebuttal clearly shows that the model performance improves with increasing more training data, indicating the current dataset size is insufficient. While the authors explained that it is limited by resources, I believe there is potential to explore automated pipelines to scale up data collection more efficiently.\n\nAs such, I will maintain my original rating as weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your comprehensive comments and constructive advice. We are very excited to see that the reviewer finds our work (1)\"provides an **inspireful insight**\", (2) \"evaluation criteria is **comprehensive and well-structured**\", (3)\"**comprehensive experiments**\", (4)and \"paper is well written\". Your suggestions and questions are valuable, and we will explain your concern as follows.\n\n**[Q1] Concerns about the size of the dataset.**\n\n**[A1]** We conducted the following experiments and analyses to allay your concerns as much as possible. We trained on different amounts of training data and tested on the test set. The similarity and consistency between the dimensions and the human evaluation are in the table below. We found that for the model trained on 1k, 2k data, the model is able to learn the entire format of the assessment, the output is formatted correctly. There is a performance improvement as data increases. \n\n|  | Quality | Aesthetic |  Consistency | Alignment | Rationality |  Safety | Creativity |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| 1k | 0.589/0.533/0.526 | 0.697/0.666/0.404 | 0.502/0.238/0.500 | 0.549/0.511/0.821 | 0.552/0.513/0.633 | 0.727/0.765/0.360 | 0.675/0.654/0.792 |\n| 2k | 0.626/0.600/0.474 | 0.667/0.619/0.333 | 0.565/0.414/0.477 | 0.560/0.497/0.786 | 0.551/0.510/0.571 | 0.712/0.756/0.380 | 0.771/0.756/0.792 |\n| all | 0.743/0.715/0.404 | 0.717/0.719/0.351 | 0.634/0.641/0.341 | 0.601/0.418/0.439 | 0.606/0.515/0.560 | 0.747/0.762/0.360 | 0.797/0.759/0.542 |\n\nIn fact, we believe that orders of magnitude larger (10k or even 100k) video and human assessment reason data and a better training approach (as opposed to cost-saving LoRA fine-tuning) may be able to result in a more robust t2v evaluation model. Our work has been constrained by GPU computational resources and funding limitations, preventing further expansion. In addition to this, **some of the previous AIGC quality assessment datasets are also not very large**; the T2VQA-DB [1] dataset contains 10,000 videos, the MQT [2] dataset has only 10,005 videos, the FETV [3] dataset has only 2,476 videos, and the LGVQ [4] dataset has only 2,808 videos and the above datasets only contain human subjective ratings, which are not available for the training of interpretable models.\n\n**[Q2] Typo error.**\n\n**[A2]** Thank you for your careful reading. We will correct this issue and carefully check for any other typo errors in the revised version of the paper.\n\n**[Q3] Automatic pipeline to expand the dataset scale.**\n\n**[A3]** Your comments are very insightful. We used an automated approach based on GPT-4o in generating the instruction tuning dataset from human assessment reasons, reducing the cost of organizing human assessment reasons into assessment CoT data. The main limitation to our dataset construction is human annotations. Instruction tuning for MLLMs requires human-assessed instruction datasets, and collecting human assessments for each video is time-consuming and costly. In order to minimize the impact of annotators' own biases on the evaluation dataset, we exclude videos from the dataset if there is a significant disparity between annotators' ratings. Overall, human evaluation are the **gold standard** and proof of the credibility of the assessment model. Collecting human assessment reason is a step that we believe cannot be replaced by an automated approach, which allows the **high dataset construction cost to limit the size of the dataset**. We also hope for more efficient methods to scale human annotation in the future.\n\n**[Q4] Model architecture.**\n\n**[A4]** We adopt Qwen2-VL-7B[5] as our base model. Qwen2-VL-7B adopts the tandem structure of ViT and Qwen2, and mainly realizes two major innovative architectures: (1) Fully supports native dynamic resolution, which can handle image inputs of arbitrary resolution, and different sizes of images are converted into dynamic numbers of tokens, with a minimum of only 4. (2) Proposes M-ROPE to decompose rotational position embedding decomposition into three parts: time, height, and width, so that the large language model can integrate the position information of one-dimensional text, two-dimensional image and three-dimensional video.\n\n[1] Subjective-aligned dataset and metric for text-to-video quality assessment. ACM-MM 2024.\n\n[2] Measuring the quality of text-to-video model outputs: Metrics and dataset.\n\n[3] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation.NeurIPS 2023\n\n[4] Benchmarking Multi-dimensional AIGC Video Quality Assessment: A Dataset and Unified Model.\n\n[5] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution.\n\n---\n\nFinally, we deeply appreciate the thoughtful questions and suggestions, which have greatly contributed to improving our work and will be incorporated into our revised manuscript. **We sincerely hope it sufficiently address your concerns and earn your recognition.**", "claims_and_evidence": "- As one of the motivation of GRADEO, the authors claim that the current LLM-based evaluation methods only provide scores without rationale, which is a reasonable claim and provides an inspirefeul insight for involving the idea of multi-step reasoning in T2V evaluation.\n\n- The authors claim that current T2V models struggle with high-level semantic understanding. Such weaknesses is confirmed by the proposed benchmark, mentioning that most of the state-of-the-art models perform poorly in rationality and creativity scores.", "methods_and_evaluation_criteria": "- To fine-tune a LLM for video evaluation, the authors collected a GRADEO-Instruct dataset with 3.3k videos and 16k human annotations. However, I doubt that such scale of the dataset is large enough to train a good evaluator. It would be interesting to see an experiment ablating different scales of training dataset (e.g., including 1k / 2k / 3k videos for training) and check how the dataset scale affect the model performance.\n\n- GRADEO-Instruct dataset only includes AI-generated videos, limiting its perspective and knowledge on synthetic videos. Most of the LLM-based methods, such as VideoScore, include both real and fake videos for training, which is a more reasonable setting and could teach the model distinguish the videos from these two distributions.\n\n- The evaluation criteria is comprehensive and well-structured. It includes seven evaluation dimensions, covering both low-level perception (e.g., quality, aesthetic) and high-level semantic reasoning (e.g., rationality, creativity).\n\n[1] \"VIDEOSCORE: Building Automatic Metrics to Simulate Fine-grained HumanFeedback for Video Generation\"", "theoretical_claims": "There is no theoretical claims in this paper.", "experimental_designs_or_analyses": "- The experiments are comprehensive. To evaluate the proposed GRADEO evaluator model, the authors provide the correlation analysis with human scores and compare with a wide variety of the models. Such experiments are also conducted on multiple datasets.\n\n- In the while, the authors also benchmarks the current T2V models using the proposed GRADEO evaluator model and provide some insights (e.g., most models perform poorly in creativity and rationality) based on the results.", "relation_to_broader_scientific_literature": "The paper introduces a novel and well-motivated T2V evaluation model by loosening the constraint that the current evaluation model usually provides the scores only without the rationale behind. Through annotating the videos with the evaluation of multi-step reasoning, the model performance could further be improved from the chain-of-thoughts reasoning.", "essential_references_not_discussed": "This paper has well cited most of the related works.", "other_strengths_and_weaknesses": "- The overall presentation is good. The paper is well written and the figures are properly plotted which could help readers quickly understand the proposed method.", "other_comments_or_suggestions": "- Line 305, with those of \"GRADE\" -> there is a missing \"O\"", "questions_for_authors": "- Have the authors considered adopting automatic pipeline to expand the dataset scale with a smaller cost?\n\n- What is the model architecture of GRADEO? Since GRADEO model is one of the main contribution of this paper, it would be better to elaborate how the model backbone is designed. For example, how is the video encoded and inputted to the proposed vision-language model? Different model designs could also significantly affect the model performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper initially received divergent reviews—two positive and two negative—but after a successful rebuttal, all four reviewers reached a positive consensus, with final scores of two accepts and two weak accepts.

The paper presents a novel T2V evaluation framework along with an instruction tuning benchmark. While initial concerns were raised regarding LLM hallucination, dataset scale, video generation sources, and generalization performance, the authors addressed these issues convincingly during the rebuttal.

In line with the reviewers, the AC finds that the strengths of the work outweigh the weaknesses and recommends acceptance.
Decision: Accept (poster)