Evaluation of Text-to-Video Generation Models: A Dynamics Perspective
Authors: Mingxiang Liao
Hannan Lu
Qixiang Ye
Wangmeng Zuo
Fang Wan
Tianyu Wang
Yuzhong Zhao
Jingdong Wang
Xinyu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Video generation model
dynamics evalution
Primary area: evaluation

Abstract: Comprehensive and constructive evaluation protocols play an important role when developing sophisticated text-to-video (T2V) generation models. Existing evaluation protocols primarily focus on temporal consistency and content continuity, yet largely ignore dynamics of video content. Such dynamics is an essential dimension measuring the visual vividness and the honesty of video content to text prompts. In this study, we propose an effective evaluation protocol, termed DEVIL, which centers on the dynamics dimension to evaluate T2V generation models, as well as improving existing evaluation metrics. In practice, we define a set of dynamics scores corresponding to multiple temporal granularities, and a new benchmark of text prompts under multiple dynamics grades. Upon the text prompt benchmark, we assess the generation capacity of T2V models, characterized by metrics of dynamics ranges and T2V alignment. Moreover, we analyze the relevance of existing metrics to dynamics metrics, improving them from the perspective of dynamics. Experiments show that DEVIL evaluation metrics enjoy up to about 90\% consistency with human ratings, demonstrating the potential to advance T2V generation models.

Evaluation of Text-to-Vides Generation Models: A Dynamics Perspective
Mingxiang Liao!*
Hannan Lu
Xinyu Zhang 3.4
Fang Wan
Tianyu Wang'
Yuzhong Zhao' Wangmeng Zuo?
Qixiang Ye
Jingdong Wang'
Uniivrsityy Chinese Academy of Sciences
Harbin Institute Technology
"The University of Adelaide "Baidu Inc.
[liaominxxiang20, wangtianyu21, zhaoyuzhong20 @mails. ucassacc.o
(wanfang, qxye] @ auuassaccccm (luhannan, wmzzu| @ @hit.edu.cn
xinyu.zhang02 @ adelaide. aelaaddeeduu.au, wangjingdong baidu.com
Abstract
Comprehensive and constructive evaluation protocols play an important role when developing sophisticated text-to-video (T2V) generation models. Existing evalua- tion protocols primarily focus on temporal consistency and content continuity, yet largely ignore dynamics of video content. Such dynamics an essential dimension measuring the visual vividness and the honesty of video content to text prompts. In this study, we propose an effective evaluation protocol, termed DEVIL, which centers the dynamics dimension to evaluate T2V generation models, well improving existing evaluation metries. practice, we define set of dynamics scores corresponding to multiple temporal granularities, and new benchmark of text prompts under multiple dynamics grades. Upon the text prompt benchmark, assess the generation capacity of T2V models, characterized by metrics of dynamics ranges and T2V alignment. Moreover, we analyze the relevance existing metrics dynamics metrics, improving them from the perspective of dynamics. Experiments show that DEVIL evaluation metrics enjoy up to about 90% consistency with human ratings, demonstrating the potential to advance T2V generation models. Project page: (2veval. gitvvaa.ithub.oo/EEEV
Introduction
With rapid progress of video generation technology, the demand of comprehensively evaluating model performance continues to grow. Recent benchmarks [27 24 have included various metrics, e.g., generation quality, video-text alignment degree, and video content continuity, to evaluate T2V generation models. Despite of the great efforts made, essential characteristic of video, i.c., dynamics, remains ignored.
Dynamics is crucial dimension when evaluating video generation models for the following two reasons: (i) In practical applications, prompts, e.g., dramatic text prompts result in videos of high dynamics. (ii) In real-world scenarios, dynamics are negatively relevant to commonly used evaluation metrics, as observed by recent benchmark studies [24] 27). This allows T2V models to 'cheat' by generating low-dynamic video content in many cases to achieve high scores upon these metrics.
expected that generated video content honest text
In this study, we introduce DEVIL, comprehensive evaluation protocol, which assesses T2V generation models from a perspective of dynamics. DEVIL treats dynamics primary dimension for T2V model evaluation, as well as enhancing the completeness of existing metrics. To fulfill
Equal contribution. Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Dynamies
Inter-
frame
Naturalness
Scene/Pattern Dynamics
Quality
Infer-
segment
Global Dynamics
Aesthetic
Smoothness
Video
Count
Level
45
Dynamies Score
(a)
(b)
Figure 1: Evaluation of video dynamics. (a) Illustration of dynamics at multiple temporal granularities. (b) Video quality distribution w.r.t. dynamic scores. (Best viewed in color)
this purpose, we first establish benchmark incorporating text prompts under multiple dynamics grades. The text prompts are collected from commonly used datasets 7 46 and categorized using Large Language Model (LLM), GPT-4 [30], with further manual refinement. We then define a set of dynamics scores, which are aggregated into two dynamics metrics to reveal the temporal characteristics of generated videos. In addition, we conduct user study synthesize the proposed metrics into an overall dynamics score, facilitating a comprehensive evaluation of dynamics capacity, characterized by dynamics ranges and T2V alignment.
To enhance the completeness of existing evaluation metrics, we introduce bi-variate analysis strategy. additional dimension to examine the distribution of existing
Specifically, we use dynamics metrics, such as aesthetics, continuity, and consistency. Through bi-variate analysis, we identify those metries negatively related to dynamics, and update them by incorporating the dynamics factor. We also introduce a metric to evaluate the naturalness based on a multimodal large language model (MLLM), e.g.. Gemini- 1.5 Pro
With the proposed DEVIL protocol, we evaluate state-of-the-art T2V models and commonly used benchmarks and find the following problems. (i) Existing generation models typically generate slow-motion videos, as most videos in existing benchmarks are of low dynamics. (ii)The text prompts in commonly used T2V benchmarks can not reflect the degrees of video dynamics. If such prompts are used to generate videos, they cause poor T2V alignment w.r.l. dynamics. (ii)) By experiments, we observe that the naturalness of generated video decreases with video dynamics, which implies that the capability of simulating real-world scenarios remains be elaborated.
The contributions of this study are summarized as follows,
We propose novel evaluation protocol, termed DEVIL, which benchmarks T2V generation models by integrating dynamics metries. Together with existing evaluation metrics, DEVIL builds more comprehensive evaluation protocol.
We establish a text prompt benchmark w.r.t. dynamics grades and propose a set of metrics to evaluate video dynamics across temporal granularities, facilitating the assessment of dynamics range and T2V alignment.
Extensive evaluation of existing T2V generation models allows us to thoroughly analyze the capabilities of T2V models through the proposed protocol and benchmarks. The results would inspire sophisticated T2V generation models.
2 Related Work
2.1 Text-to-Vich Generation Model
As recent breakthrough artificial intelligence, diffusion models have pushed video generation technology to height. Earlier studies [2 21 explored the 3D U-Net and cascaded models for

---

T2V Generation
Dynamics Scores
Dynamics Evaluation
DEVIL Text Prompts
Dynamic Leveli High, matechernnng Aiph spaiid..
Ootica Flos Stnewch
Aaccertal Synonic
Strcchud bynanis Scare
Human Alignment COne- Time
Centrellabllit
Dynomics
Dynamics Ronge
Iitter sogment
Piica I runway Gen-2
rudiciiity
Apericiicty
VideoCrafrer2
Video level
PVR
Teapord Entragy
Tempord Semortc Synonics
Figure 2: Flowchart to calculate dynamics metrics based on dynamics scores and text prompts.
employed latent diffusion models to efficiently manage the diffusion process within compressed latent space. Following these studies, a variety of approaches [40] IT, 26 43 16 42 47
diffusion within pixel space. Recent solutions
29 25 updated and improved this paradigm. Building on these advancements, subsequent methods further explored generating videos of higher quality and extended duration. The Videocrafter approach [4] pursued high-quality video generation through disentangling spatial and temporal learning and tuning spatial modules using high-quality images. In a similar way, commercial models such as Pika [4] and GEN-2 2 demonstrated substantial improvements, showcasing videos with exceptional visual clarity. For longer video generation, Gen-L-Vides 33 aggregated short clips generated by base T2V models using temporal co-denoising enhance continuity. Freenoise 31 extended pre-trained T2V models through rescheduling noise for longer-duration video inference. StreamingT2V [99 enhanced long-term content consistency by integrating short-term and long-term memory blocks.
The rapid development of T2V models poses a growing demand for quality evaluation protocols. Unfortunately, existing protocols primarily focus on temporal consistency and content continuity, yet largely ignore temporal dynamics. This hinders the exploitation video content vividness and the honesty of video content to text prompts.
2.2 Evaluation Protocol
Early evaluation protocols B5 primarily relied on class labels T2V models. For example, they commonly used video clips from the UCF-101 dataset and human-annotated video captions from the MSR-VTT [46) dataset the evaluation data. For
specific assessment, FETV (2 assigned fine-grained category labels to prompts and calculated the CLIP-SIM score for each category.
However, conventional quality assessment metrics such Inception Score (IS) 34 Fréchet Inception Distance (FID) (20), Frechet Video Distance (FVD) [37], and CLIP-SIM typically operate on a single dimension while can not provide comprehensive evaluation. When addressing the limitation, EvalCrafter 32] expanded both the prompt scale and the number of evaluation metrics so that the text-video alignment degree and the quality of generated videos can be better evaluated. Additionally, VBench 24 proposed a multi dineniiiiinnionnal, multi-category evaluation suite that not only considered the diversity of prompts but also encompassed variety of assessment metrics.
Despite of the evolution of evaluation metrics, we argue an essential characteristic of video, i.e., dynamics, remains ignored. In this study, we introduce the dynamics dimension to evaluate T2V generation models, as well as enhance the completeness of existing metrics.
3 Dynamics Evaluation Protocol
As shown in Fig. 2 we first establish benchmark incorporating text prompts under multiple dynamics grades. The text prompts are collected from commonly used datasets 7 46 and categorized to dynamics grades using GPT-4 [30] and human refinement. We then define set of dynamics
3

---

Table 1: Formulations of dynamics scores at different temporal granularities.
Granularity
Dynamics scores
Formulation
Optical Flow Strength
Dofs N-1 FLOW(())
Inter-frame
Structural Dynamics Score
D = 1 Zi=1 (s,M fitz)
Perceptual Dynamics Score
D pd
N-T Di=1 PHASHD (f,,
Patch-level Aperiodicity
D pa
HW En. ([F...E))))
Inter-segment
Global Aperiodicity
D Dga =
[rN] E!'N Ejje SIM
Temporal Entropy
Die H(f1, ʃ2, SN((1) N
Video
Temporal Semantic Diversity
Disd
||F F112
scores corresponding to multiple temporal granularities, to reveal the video characteristics at multiple temporal levels. We finally conduct a user study to aggregate the proposed dynamics into overall dynamics metrics about ranges of dynamics and T2V alignment.
3.1 Text Prompt Benchmark
Building the benchmark includes a coarse categorization step and a post-processing step. In the coarse categorization step, the GPT-4 model is used to categorize approximately 50k text prompts collected from existing benchmarks into five grades. These benchmarks include VidProm [41),
WebVid (8) MSR-VTT 446, Didemo [8]] etc. The five dynamics grades summarized follows:
Static video: Video content is nearly stationary. Example: A man is laying on the ground
Staric
Low
Medium
Tmally extr eemel y long sarply poo
High
Very High
Fapidly LoUS
Low dynamics: Video content has slow and slight changes. Example: male fencer adjusts his epee mask and prepares duel with his sparring partner in slow motion.
400
Sudden
narrowll
Medium dynamics: Noticeable activity and changes, but relatively smooth overall. Example: Tilt up of shirtless sportsman doing pull-ups bars during cross-training workout at gym.
back Pa
awayy
Vieench
(4)
(b)
Figure 3: (a) Distribution of dynamics grades for text prompts from DEVIL, Vbench [24], and Eval-
High dynamies: Fast actions and changes. Example: A runner explodes out of the starting blocks, racing down the track.
Crafter (27). (b) Word cloud of the text prompt
benchmark of DEVIL.
Very high dynamics: Extremely rapid and frequent video content changes. Éxample: A medieval siege with catapults launching, walls breaking, soldiers charging, and arrows raining down.
In the post-processing step, we recruit six human annotators to refine the dynamics grades following same criterion. Finally, we sample 800 text prompts at different dynamics grades for a uniform distribution along the grades.
Fig. shows the statistics of the DEVIL benchmark, which contains approximately 800 text prompts, and each dynamics grade includes 19 object categories and scene categorics. Unless otherwise specified, all experiments in this paper conducted on the DEVIL benchmark.
3.2 Dynamics Scores
As illustrated in Fig. video dynamics can classified into three categories based on their temporal granularities: (i) fnter-frame dynamics, which describes the variations between successive frames. The dynamics score at this level reflects rapid and prominent content variations. (ii) Inter-segment dynamics, which refers to the changes between video segments which contains K video frames.

---

Define on middle-level, this score captures middle-speed transitions and motion patterns. (iii) Video-level dynamics, which encompasses the overall content diversity and the frequency of changes throughout the video.
(i) Inter-frame Dynamics Score. This is further cat- egorized to optical flow strength, structural dynamics and perceptual dynamics.
Optical flow strength. We first employ RAFT 49 to estimate the optical flow [9 33) for each video frame. The mean optical flow magnitudes of each frame are averaged to calculate the optical flow strength of this frame. Averaging the optical flow strength values of all video frames, have the optical flow strength Dofs of the video, as
(a) Inter-frame Dynamies
N-1 Dofs N Σ FLOW(F))
(b) Inter Inier-seginent Dynamics
(I)
where FLOW calculate the mean optical flow strength values of frame ft.
(c) Video-level Dynamics
Structural dynamics score. While optical flow ex- cels in capturing motion, less effective when detecting structural dynamics such as lighting condi- tions. To capture such information, we calculate the average structural similarity index (SSIM) between consecutive frames from all frame pairs to quantify inter-franc structural variations of the video,
Figure 4: Video dynamics at different tempo- ral granularities: (a) Inter-frame Dynamics, (b) Inter-segment Dynamics, and (c) Video- level Dynamics.
D = 1- N
Σ SSIM(J,,(++))))))
(2)
Perceptual dynamics. The human visual system is sensitive to changes in low-frequency regions of video frames. To reflect this characteristic, we introduce perceptual dynamics metric that measures the difference between the perceptual hashes (8 of consecutive frames. The perceptual distance D Da is defined as the mean all frame pairs, as
N-
D pd
N
where PHASHD denotes the Hamming distance between the perceptual hash of fi and fi+1. (ii) Inter-segment Dynamics Score. This is further categorized into patch-level aperiodicity and global aperiodicity, which measure the dynamics between video segments.
Patch-level aperiodicity. We first calculate inter-segment dynamics the patch level using the auto- correlation factor [1], to evaluate the scene and temporal pattern dynamics. The auto-correlation factor measures the feature similarity of time series, revealing periodicity and changing trends of features. Given features at position (h, w) across N frames, w the auto-correlation factor of the features is defined
k k
ACF(
Σ N K k=
SIM FN-k+s,h.w)
where SIM represents the cosine similarity between two feature vectors. The minimal segment length Ko is empirically set to N/88, as most generated videos contain more than 8 frames. H and W are the height and width of the feature map, respectively. With auto-correlation factors of all patches, we define the patch-level aperiodicity of the video, as
1
Σ JC
D pa = 1-
(5)
HW
h.a
5

---

Global aperiodicity. In addition to patch-level dynamics, we employ global aperiodicity metric measure the diversity of patterns between video segments. Specifically, we divide the video into segments. Each segment has a length rN,, where r is proportion factor, empirically set to 0.25. We use ViCLIP 44 to extract the spatial-temporal features for each segment. The features are denoted as { 1=1 We then calculate the similarity of these features to assess the variation in spatial-temporal patterns across segments, as
IeN] Σ [=1 j.
D ga
=
Σ
FI, Fj).
[rN]
(iii) Video-level Dynamics. The dynamics of whole video sequence defined upon the temporal entropy and temporal semantic dynamics.
Temporal entropy. To evaluate the dynamics the video level, we first measure the temporal information of each video. The temporal information is defined as the conditional entropy of the entire video sequence given the first frame
Die H fxlfi).
To estimate the conditional entropy Die, we employ the video encoding toolbox FFmpeg [1]
Temporal Semantic Dynamics. Beyond low-level dynamics, we further introduce a semantic diversity score assess high-level dynamics across the whole video. The semantic diversity computed to reflect semantic-level dynamics and defined as the variance of DINO [2]
Diad is features
(F.) of each frame,
I Σ
D Drad
|. FTP
(8)
where F == N EE P; denotes the mean feature vector of all frames.
3.3 Human Aligned Dynamics Scores
To establish reliable and robust assessment, we introduce human alignment module, Fig. 2 to refine the empirically defined dynamcis scores. It utilizes human ratings to provide ground-truth, based on which we fit linear regression model each temporal level,
D f Linear,, 8, (D Ded: D Dpd);
(9)
D. Linear (Dpa; Dga),
(10)
D. Lincar,, (Die, Disd).
(11)
where 0,, 0,, 02 respectively denote the model parameters of linear regression at each scale. The overall dynamics score of the video then defined as the average aligned dynamics scores from all three levels, as
1 DD,
D
(12)
Through this learnable human alignment procedure, the empirically defined dynamics scores more consistent with human perception, as validated in Sec. 5.1
3.4 Dynamics Metrics
After calculating the aligned dynamics scores of all generated videos at inter-frame inter-segment, and video levels, we combine these scores together to obtain the following two evaluation metrics.
(i) Dynamics Range. The metric evaluates model's capability to generate videos with vivid dynamics. larger dynamics range implies higher dynamics capability. In detail, determine the dynamics range Mronge by identifying the extremes of the dynamic scores over the benchmark, while excluding the top and bottom 1% scores mitigate the influence of outliers. This is formulated as
Myange Q0.99 Q0.01.
(13)
Appendix presents the model weights and typical values for various dynamics scores.

---

where Q0.99 and Q0.01 denote the 99th and 1st percentile values of the dynamics scores for videos generated with the DEVIL benchmark, respectively. This metric reflects a realistic spread of dynamics. excluding atypical extremes.
(i) Dynamics Controllability. Let P()) p E [1,5] respectively denote the ground dynamics grades of prompt i and j. and D D the predicted dynamics scores by prompt and j. For p(3) > PO), we should have D > D so that the dynamics scores of the generated videos are consistent with the dynamics grades of text prompts. Accordingly, we can calculate the dynamics controllability metric by
Mcontrol
1
E I(D - -D -p(1)) jo3A
(14)
T.
where [Z'] is the total prompt number and Ti denotes the number of prompts at dynamics grade (().
4 Improving Existing Metrics with Dynamics
As observed by our experiments, existing metrics have negative relevance video dynamics. To identify these metrics, we calculate the correlation, e.g.. Pearson and Kendal correlation coefficients, between dynamics scores and existing metrics, Table 11 These metrics include naturalness, motion smoothness, subject con- sistency, and background consistency. Under these metrics, models might 'cheat' for high-quality scores by generating low-dynamic videos.
Table 2: Correlation hetween the dynamics
metric with the existing metrics including Naturalness (Nat), Visual Quality [45] (VQ),
Motion Smoothness (MS) (24). Subject
Consistency(SC) (2 and Background
Consistency(BC) correlation, and 'KC' denotes Kendall's correlation.
24 "PC' denotes Pearson's
We improve the identified metrics by incorporating our proposed human human-aligned dynamics score D. In specific, we propose equally divide the human aumann-lignned dynam- ics score into L = 13 intervals. Within each interval, we calculate the mean metric values. The mean values of the T. intervals are further averaged as the improved metrics. Upon the improved metrics, have high score, the generated videos should spread all dynamics intervals, which implies a large dynamics range.
Evaluation Metries
PC
KC
Naturalness (Nat)
-51.8
-44.2
Visual Quality (VQ)
-24.8
-18.6
Motion Smoothness (MS)
-64.0
-54.6
Subject Consistency (SC)
-88.9
-74.9
Background Consistency (BC)
-79.4
-61.4
Naturalness. In addition to the improved metrics, we introduce the Naturalness metric, which reflects how much the generated videos are like camera-captured ones. This is done by using the MLLM, i.c.. Gemini-1.5 1.5 Pro [[]. to calculate a naturalness score for each video. The scores are categorized into five grades: "Almost Real" (100 points), "Slightly Unrealistic" (75 points). "Moderately Unrealistic" (50 points), "Noticeably Unrealistic" (25 points), and "Completely Fictitious" (0 points). The overall naturalness is then determined by averaging the scores of all videos. For evaluation, we invited five users to rate the naturalness of the generated videos and then perform correlation analysis between human ratings and model scores. A high correlation (larger correlation coefficients) indicates the plausibility of the naturalness metric.
5
Experiment
5.1 Human Alignment Assessment
To evaluate the plausibility of the proposed dynamics metrics and the naturalness metric, we conduct the following human alignment experiments.
Ground-truth Annotation. We first generate videos using six state-of-the-art (SOTA) T2V mod-
els, including GEN-2 (2]. Pika [4) VideoCrafter2 [14), Open-Sora 23), StreamingT2V [19 and
Lree 31 and DEVIL text prompts. For the generated videos, we collect human evaluated
3 Pleesse refer to Appendix D for the details of the improved metrics.
7

---

Table 4: Evaluation of dynamics across text-to-video models multiple temporal levels. Metrics include inter-frame (M (MFage)) inter-segment (Mygge)) and video-level (M, (agge)) dynamics range and overall dynamics range (ange)) also shown. Dynamics ranges and dynamies controllability (M are from 0 to 100, where higher scores indicate better performance.
Dynamics Ranges
Dynamics Control.
T2V-Models
Maange
Mfange
Ma
Mrange
Moontrol
GEN-2 2
18.8
49.7
21.4
27.8
80.8
Pika 4
36.2
56.3
29.1
36.8
72.2
VideoCrafter2 14
38.9
43.2
17.0
29.4
56.6
OpenSora 23
65.2
80.1
36.4
55.3
61.7
Streaming
59.8
80.0
69.9
61.4
64.0
FreeNoise-Lavie
67.6
71.7
66.3
63.44
58.2
VideoCrafterl
13
62.0
60.3
28.5
44.6
63.7
Hotshot-XI.
52.2
56.8
17.6
36.1
59.0
Show-1 48
55.5
62.4
37.0
45.0
74.2
ModelScope 40
72.1
78.1
40.3
56.0
63.7
ZeroScope
24.9
46.8
19.3
28.5
66.4
dynamics and naturalness the ground-truth. Six persons are recruited to assess each video's grade of dynamics under three temporal levels (Frame, Segment, and Video). For each temporal level, evaluators are required to rate the grade of dynamics from "static" to "very high dynamics".. To guide the annotation process, provide specific prompts for each temporal level. We conduct between- group correlation analyses using Pearson's 's correlation, Kendall's 's correlation, and the win ratio to evaluate the consistency of dynamics scores with respect to human ratings.
The evaluation of the naturalness metric follows the same process, where a higher human assigned grade indicates greater degree of naturalness.
Table Human alignment by correlation between dynamics scores and human rat- ings on the proposed DEVIL benchmark. Video generation is based text prompts in DEVIL. "PC" denotes Pearson's ' correla- tion, "KC" Kendall's correlation, and "WR" the win ratio.
Evalution. We calculate dynamics grades and natu- ralness for generated videos the proposed DEVIL benchmark. For dynamics scores at multiple temporal levels, integrate them using the linear regression model defined by Eq. 12] For each linear regres- sion model, it takes the human evaluation results ground-truths, trained upon 75% the randomly se- lected videos and tests the remaining 25% videos. During testing, the human alignment performance reflected by the correlation e.g., Pearson and Kendall correlation coefficients and win ratio, between pre- dicted and human-evaluated dynamics grades. The win ratio involves comparing each video against oth- ers with different grades of dynamics. For instance, a video rated as "high dynamics" by evaluators should score lower in dynamics than any video rated "Very high dynamics" but higher than those rated as "static".
Scores
PC1
KC 1
WR
Safe
93.1
89.9
79.2
S.d
91.7
88.0
78.1
Inter-frame
Syd
96.4
93.2
86.1
Sy
96.5
93.5
86.5
Spo
95.1
94.3
87.0
Inter-segment
Sy
94.6
93.0
85.6
S.
95.8
94.8
87.7
Sie
96.4
93.7
83.5
Video level
Stad S.
97.7
96.4
90.5
98.0
97.2
91.4
Naturalness
79.0
75.5
52.4
Table 3 shows the assessment results of the six T2V models. It can be seen that the dynamics metrics and the naturalness metric exhibit strong alignment with human evaluation. The improved metrics (Dy, D.. D., defined in Sec. alignment with human evaluations.
3.3 further enhance the
5.2 Influence of Frame Rate.
'Please refer to Appendix G for details

---

Table 6: Evaluation of existing metrics and improved metrics. These metrics include Motion Smoothness (MS), Background Consistency(BC), Subject Consistency(SCC, and Naturalness (Nat).
Existing Metries
Improved Metrics
Model
MS
BC
SC
Nat
MS
BC
SC
Nat
GEN-2 2
99.4
97.2
95.6
81.6
57.9
55.6
53.3
38.0
Pika
99.4
96.5
93.2
69.0
57.7
55.8
53.1
32.4
VideoCrafter2
14
97.7
97.4
95.5
70.8
48.1
47.7
46.5
33.1
OpenSora
23
95.4
94.3
88.7
63.6
69.9
69.5
63.1
37.1
Streaming T2V
19
94.9
91.0
85.2
55.2
70.7
68.0
61.2
33.2
FreeNoise-LLavie
95.5
94.1
90.1
73,4
77.4
68.1
76.3
51.4
VideoCrafter
13
95.8
95.3
92.8
75.4
61.3
60.7
57.9
39.7
Hotshot-XL
97.0
95.5
93.4
85.6
55.7
54.7
52.1
45.1
Show-1 48
97.1
94.4
91.5
74.4
62.9
61.8
58.6
43.0
ModelScope
40
95.8
93.5
89.3
71.6
70.5
68.2
62.6
44.9
ZeroScope 5
98.2
94.8
89.3
74.6
49.1
46.8
45.6
34.2
Table demonstrates how frame rate influences the correlation between dynamics scores and human eval- uations. Experiments indicate that dynamics scores maintain a high correlation (>0.9) with hu- ratings across various frame rates. To mitigate the impact of frame rate variations on dynamics, we standardized the frame rate of each video FPS.
Table 5: Influence of frame rate on the consis- tency of dynamics scores with human evalua- tion (measured by Pearson's correlation).
Dynamies
4FPS
8FPS
16FPS
Origin
Inter-frame
0.952
0.950
0.946
0.951
Inter-Segm Video
0.952
0.954
0.954
0.953
0.967
0.967
0.967
0.967
5.3
Computation Efficiency
Our dynamics metrics offer high computational efficiency, achieving around 10 frames per second on single NVIDIA A100 GPU, and are scalable to multiple GPUs.
5.4
Evaluation of Video Dynamics
We evaluate the dynamics range M Mrange, and dy- namies controllability Mcontrol of T2V models the proposed DEVIL benchmark. We also evalu- dynamics ranges at different temporal scales: inter-segment dynamics rangeM/ /ange inter-segment dynamics range M Mrange> and video level dynam- ics range M range. The results are shown in Ta- ble The GEN-2 (2 and Pika [4] models score high in dynamics controllability but low in range due to their generation of low-dynamic videos. Con- versely, the FreeNoise- Lavie method (31] attains high range but low controllability, suggesting it produces videos with dynamics that do not align well with text prompts.
Video Quantity Density
3.0
2.0
1.0
0.0
0.0
0.2
0.4
0.8
1.0
Dynamics Score
Figure 5: Video quantity density w.r.f. dy- namics score of the WebVid-2M dataset.
5.5 Improved Evaluation Metrics
As shown in Table 6 the existing metrics exhibit an obvious negative correlation when embedded with dynamics, indicating that these models can achieve high scores on these metries by generating low-dynamic videos rather than high-quality content.
5.6 Insights from Video Dynamics Analysis
Biased Dynamics Distribution of Existing Dataset. The distribution dynamics of the video datasets (such as WebVid2M [8) is biased. The statistical result is shown in Fig. 5 It can be seen that most of the videos have small dynamics score (S 20). The limited number of videos with high dynamics scores restricts the model's ability to generate dynamics-rich vidcos which are common in

---

Video Quantity
Background Consistency
Motion Smoothness
Naturalness
GEN-2
GEN-2
VC-2
VC-2
Pika
Pila
EN
FN OS
OS
ST2V
ST2V
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Dynamics Score
Dynamles Score
Dynamies Score
Dynamles Score
Distribution video quantity along dynumics score.
(b) Distribution of quality scores along dynamics score.
Figure 6: Distributions of video quantity and quality scores along the dynamics score for various video generation models including: GEN-2 (2). Pika [4) VideoCratter2(VC-22) [14], Open-Sora(OS) [23] TtreamingT2V [19] and FreeNoise-Lavieffl
[31]. Subplot (a) shows video quantity distribution. Subplots (b) display the distribution of quality score of generated videos in terms of Background
Consistency, Motion Smoothness, and Naturalness, respectively. All videos are generated based on our text prompt benchmark.
practical applications. Therefore, existing datasets should be expanded in terms of dynamics, and the proposed metrics can provide guidance for this expansion.
Video Bias of Existing Datasets. We used the dynamics controllability metric evaluate two popular datasets, WebVid2M [8] and MSR-VTT [46), by using the ground-truth text prompts and videos. Unfortunately, they respectively achieve alignment scores of 36.31 and 52.98. The poor performance indicates that the two datasets can not provide sufficient informmation/guddann while training the video generation models. To train better video generation models, the text prompt of these datasets requires to be elaborated on aspects of dynamics.
Prompt- Video
Limited Real- World Simulation Ability of Existing Methods. As shown in Fig. 6 performed statistical analysis of frequency, visual quality, motion smoothness and naturalness metric scores for SOTA methods based on the distribution of dynamics score. When the dynamics score is low, videos generated by these SOTA models perform well across the four metrics mentioned. As the dynamics score rises, these metrics, particularly naturalness, tend to decrease significantly. This decline may be due to the models' focus on optimizing the generation of simple, slow-motion content, with dynamics considered in the evaluation metries.
Conclusion
We proposed DEVIL, a comprehensive and constructive evaluation protocol for T2V generation models. In the protocol, we defined set of dynamics metrics corresponding to multiple temporal granularities, and a new benchmark of text prompts under multiple levels of dynamics. Based on the distribution of dynamics scores over the benchmark, we assessed the generation capacity of T2V models, characterized by dynamic ranges and degree of T2V alignment. Experiments show that DEVIL enjoys 90% consistency with human evaluation results, demonstrating the potential to be powerful tool for advancing T2V generation models.
Limitations. At present, the grades of dynamics remain limited, which should be improved to more fine-grained grades. Furthermore, only limited number of T2V models are evaluated using the proposed protocol. A more comprehensive evaluation of T2V models should be done in future work.
7 Acknowledgment
This work was supported by National Natural Science Foundation of China (NSFC) under Grant 62472402 and 62225208, and the Fundamental Research Funds for the Central Universities. This work was also supported by the Centre for Augmented Reasoning, an initiative by the Department of Education, Australian Govermment.
10

---

References
[1] Gemini.
com/ 2024. Accessed: 2024-05-21.
[2] Gen-2. http://reeerddd.munaammm.oooo/g 2024. Accessed: 2024-05-21.
[3] Hotshot-xl. [ttppp///wuuggggeeee.coooooocoo/ 2024. Accessed: 2024-
05-21.
[4] Pika labs. https://pika.ee art, 2024. Accessed: 2024-05-21.
[5] Zeroscope. 2024-05-21.
http:///ww......oooooooo//////oa 2024. Accessed:
[6] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803-5812, 2017.
[7] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1728-1738, 2021.
[8] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1738. 2021.
[9] Simon Baker, Daniel Scharstein, James P Lewis, Stefan Roth, Michael Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International journal of computer vision, 92:1-31, 2011.
[10] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolutie video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[11] George EP Box, Gwilym Jenkins, Gregory C Reinsel, and Greta M Ljung. analysis: forecasting and control. John Wiley & Sons, 2015.
Time series
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In IEEE ICCV, pages
2021.
[13] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.194 2023.
[14] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-qualit video diffusion models. arXiv preprint (ww.::::001000044. 2024.
[15] FFmpeg Developers. Ffippg: A complete, cross-platform solution to record, convert and stream audio and video, 2024. URL https://fffpeg org/ Accessed: 2024-05-21.
[16] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv.2311.10709 2023.
[17] Richard W Hamming. Error detecting and error correcting codes. The Bell system technical journal, 1972):147-160, 1950.
[18] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell, Localizing moments in video with temporal language. Empirical Methods Natural Language Processing (EMNLP), 2018.

---

[19] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tade- vosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint 2rXiv:2403.14777 2024.
[20] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[21] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.03303, 2022.
[22] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 118633-86466 2022.
[23] Technoooy Inc. Open-sora: Democratizing efficient video production for all. https 7/github comm/hpcaitecch/pene..e.
2023.
[24] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. arXiv preprint arXiv:2311.17982 2023.
[25] Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dongmei Zhang. Keep the structure: A latent shift-reduce parser for semantic parsing. In IJCAI, pages 3864-3870, 2021.
[26] Han Lin, Jaemin Cho, Abhay Zala, and Mohit Bansal. Ctrl-adapter: An efficient and versatile framework for adapting diverse controls to any diffusion model. arXiv preprint arXiv:2404. 09967, 2024.
[27] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models. arXiv preprint arXiv:2310.11444 2023.
[28] Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation. Advances in Neural Information Processing Systems, 36, 2024.
[29] Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9117-9125, 2023.
[30] OpenAI. Chatgpt: A large language model. attps://www.oom hwwp/wwwwooo Accessed: 2024-05-21.
com/ chatgpt
2023.
[31] Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free free longer video diffusion via noise rescheduling. arXiv preprint arXiv:2310.15169 2023.
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-8766 PMLR, 2021.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[34] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
12

---

[35] Danying Su, Zhiqiang Su, Jiaye Wang, Shanshan Yang, and Jing Ma. Ucf-101, a novel omi/htra2 inhibitor, protects against cerebral ischemia/reperfusion injury in rats. The Anatomical Record: Advances in Integrative Anatomy and Evolutionary Biology: Advances in Integrative Anatomy and Evolutionary Biology, 292(6):854 861, 2009.
[36] Shangkun Sun, Jiaming Liu, Thomas Li, Huaxia Li, Guoqing Liu, and Wei Gao. Stream- flow: Streamlined multi-frame optical flow estimation for video sequences. arXiv preprint arXiv:2311.17099 2023.
[37] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly. Fvd: A new metric for video generation. 2019.
[38] Ramarathnam Venkatesan, S-M Koon, Mariusz H Jakubowski, and Pierre Moulin. Robust image hashing. In Proceedings 2000 International Conference on Image Processing (Cat. No. 00CH37101), volume pages 664 666. IEEE, 2000.
[39] Fu-Yun Wang. Wenshuo Chen, Guanglu Song, Han-Jia Ye, Yu Liu, and Hongsheng Li. Gen-l-video: Multi-text to long video generation via temporal co-denoising, arXiv preprint arXiv:2305.18264, 2023.
[40] Jiumiu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06577, 2023.
[41] Wenhao Wang and Yi Yang. Vidprom: million-scale real prompt-gallery dataset for text-to- video diffusion models. arXiv preprini M.Ni::2403.06098, 2024.
[42] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Videofactory: Swap attention spatiotemporal diffusions for toxt-to-vided generation. arXiv preprint arXiv:2305. 10874, 2023.
[43] Yaohui Wang, Xinyuan Chen, Xin Ma. Shangchen Zhou Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309. 15103, 2023.
[44] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.0666 00.00992..0....... 2023.
[45] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin. Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In International Conference on Computer Vision (ICCV), 2023.
[46] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vit: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[47] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18456-18466) 2023.
[48] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818. 2023.
[49] Tianjun Zhang, Shishir G Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E Gonzalez. Raft: Adapting language model to domain specific rag. arXiv preprint a4X0 10131, 2024.
13

---

Appendix
A
Inter-segment Dynamies
Inter-segment dynamics quantifies the diversity of patterns between video segments by measuring the similarity of features across different segments. In this section, we investigate the influence of various factors on inter-segment dynamics, including the method for video segmentation, the proportion ratio and the overall length of the video.
How to segment videos? In the computation of global aperiodicity within inter-segment dynamics, video segmentation is essential. Table 7 compares the effects of proportional video segmentation and keyframe- based segmentation on the performance of inter-segment dynamics. We observe that both segmentation methods achieve comparable levels of correlation with human evaluations, with Pearson's correlation coefficients of 0.96 and 0.95, and Win Ratios of 0.85 and 0.87, respectively. Given that proportional video segmentation facilitates the simultaneous comparison of videos of varying lengths and the frequency of pattern changes, have opted to utilize proportional video segmentation in our implementation of global aperiodicity.
Table 7: Comparison of keyframe-based and proportional video segmentation methods
Video Segment Method
Pearson's 's Correlation Kendall's Correlation
Win Ratio
Key-frame-based Proportional
0.96
0.94
0.85
0,95
0.93
0.87
Influence of proportional factor r. Table illustrates the impact of the parameter r on the performance of inter-segment dynamics. When is set at 1/8, 1/4, and 1/2, the Pearson's 's correlation coefficients with human ratings are 0.92, 0.94, and 0.93 respectively. These results indicate that inter-segment dynamics is robust to variations in Ultimately, we selected r = 1/4 as it achieved the highest correlation with human evaluations.
Table 8: Influence of the proportion factor the performance of inter-segment dynamics Proportion Factor r'
Pearson's Correlation
Kendall's Correlation
1/8
0.92
0.90
1/4
0.94
0.91
1/2
0.93
0.90
Influence of Video Length. In Table 9 We group videos based on the video length (max is 8s in tested models) and study the relation between dynamics scores and human scores. Inter-segment dynamics robustly achieves over 90% correlation whatever the video length is.
Table 9: Influence of video length the performance of inter-segment dynamics.
Video Length
Pearson's 's Correlation Kendall's Correlation
0.96
0.94
4
0.93
0.91
8
0.94
0.90
B Model Weight of Human Alignment Module
Table 10 shows the weights of each dynamics score in the human alignment module.
14

---

Table 10: Weights of each dynamics score in the human alignment module.
Temporal Scale
Dynamics Score
Typical Value
Weight
Dofa
62.00
6. 6.70E-04
Inter-frame
Dad
1.00
0.17
Dd
33.00
0.03
D
0.20
2.20
Inter-segment
D
0.80
0.63
Die
7.00E+04
1.00E-05
Video
Disd
0.20
1.46
Table 11: Pearson correlation coefficient between the dynamics metrics and the existing metrics including aesthetic score [45) technical score [45] visual quality [45]. motion smoothness (24).
subject consistency [24] and background consistency [24] and our naturalness.
Aesthetic Score
Technical
Visual Quality
Motion Smoothness
Subject Consistency
Background Consistency
Naturalness
Score
GEN-2
-0.19
-0.09
-0.12
-0.54
-0.88
40.73
-0.50
Pika
-0.40
-0.20
-0.28
-0.65
-0.88
-0.78
-0.47
VideoCrafter
14
-0.25
-0.20
-0.24
-0.59
-0.87
-0.76
-0.36
OpenSora
23
-0.20
-0.27
-0.26
-0,70
-0.90
-0.83
-0.43
Streaming T2V
19
-0.15
-0.21
-0.23
-0.57
-0.89
-0.81
-0.36
FreeNoise-Laiie
31
-0.37
-0.31
-0.35
-0.75
-0.91
-0.86
-0.48
Average
-0.26
-0.21
-0,25
-0.63
-0.81
-0.79
-0.43
C Correlation Between Existing Metrics and Dynamics
In Section to identify the relevance between existing metrics with the dynamics metrics, we provide bi-variate analysis strategy. Based on bi-variate analysis, we provide detailed correlation results for the models. Table T1 the Pearson correlation coefficients between the dynamics scores and existing metrics, including aesthetic score, technical score, visual quality, motion smoothness, subject consistency, background consistency, and naturalness, are detailed.
The results indicate clear trade-off between video dynamics and various existing metrics in T2V models. As dynamic complexity increases, there tends to be decline in motion smoothness, subject consistency, background consistency, and naturalness. The aesthetic, technical, and visual quality metrics show relatively low correlation, which can attributed to the fact that these metrics evaluate video frames independently, ignoring temporal relationships between frames.
D Comprehensive Evaluation Metrics
Let (()) denote score of generated video i. Existing metrics simply average the scores of all videos obtain the metric score S of the T2V model:
T Σ
S
50)
(15)
where [7] is the total number of generated videos. Considering that some existing metrics show considerable negative correlation with the video's dynamics score, they fail prevent models from generating low-dynamic videos.
To address this issue, we enhance existing metrics by integrating human-aligned dynamics scores, preventing models from attaining high scores by producing low-dynamic videos. Specifically, we first equally divide the human- dynamics mean scores St at each interval I. The improved metric S* is defined as the average of S across all intervals:
into 13 intervals. We then calculate the
S
S
(16)
15

---

E Assigning Dynamies Grades to Text Prompts
As described in Section 3.1 we collect approximately 50,000 text prompts from existing benchmarks, including 19 object categories and 4 scene categories. Using GPT-4 coarse classification and human refinement, we construct the DEVIL prompt benchmark. The process of categorizing dynamics grades using GPT-4 is illustrated in Figure 7 In specific, we instruct GPT-4 to perform classification on the rate of content change. To enhance GPT-4's classification accuracy, we further provide detailed criteria and examples for each dynamics grade. In the post-processing step, we recruit six human annotators refine the dynamics grades over three months. Finally, we sample about 800 text prompts at different dynamics grades to ensure uniform distribution across the grades.
F Details of Naturalness
We employed the advanced multi-modal large model, Gemini-1.5 Pro [1] equipped with video understanding capabilities, assess and classify the naturalness of video content. As shown in Fig. [8] we demonstrate the process through which the model analyzes videos and assigns naturalness ratings. The figure details the five different levels used evaluate video naturalness, ranging from "Completely Fantastical" to "Almost Realistic". Each level is defined based on how closely the video content aligns with the real world. Additionally, the figure includes two examples of video evaluations: the first video is rated as "Almost Realistic" due to its high conformity with reality, while the second video, due minor distortions- such the unrealistic number of legs on dog -is rated as "Slightly Unrealistic". These examples validate the plausibility of the proposed naturalness metric.
G Human Annotation
To align human evaluations with automated metries, we annotated series of videos generated by SOTA T2V models. We initiated the process by generating videos using prompts from the DEVIL benchmark with six advanced T2V models including GEN-2, Pika, VideoCrafter2, OpenSora, Streaming and FreeNoise-Lavie. Subsequently, evaluating the dynamics and naturalness videos. As shown Figure Н the toolbox allows annotators assess the dynamics of the videos across five grades, from almost static to very high dynamics, and the naturalness from almost real to completely unreal. To guarantee high-quality and consistent evaluations, recruit six annotators who have undergraduate degrees and provided them with detailed training.
developed video annotation toolbox for
H Visual comparison
In Section 3 we use text prompts with different dynamies grades to generate videos with T2V models. Here, we provide visual results of the generated videos.
16

---

Given the provided text, classify each text segment according to the scene and background dynamics using the following criteria. For each text segment, inherit the serial number at the beginning of the text and classification label from the list below.
Classification Criteria:
Almost Static: Minimal changes scene or background, almost static.
Example: "A room where only the fading light changes." "
Low Dynamics: Slow and slight changes in scene, usually slow motion.
Example: "A balloon slowly rising, with focus its details."
Medium Dynamics: Noticeable activity and changes, but relatively smooth
overall.
Example: "A child and dog moving from grass to sand." "
High Dynamics: Fast actions and changes.
Example: "A chase scene with rapid transitions and complex maneuvers." " Very High Dynamies: Extremely rapid and frequent video content changes. Example: "A battle scene with quick cuts and intense action." Instructions:
For each section of text, assign a dynamics grade classification based on the provided criteria. List the serial number inherit from the beginning of the text followed by the classification.
User
A car drifts sharply around corner, almost hitting bystander.
2. little girl putting down and picking up her bear plush.
3. the dead bird is the ground.
User
1. High Dynamics
2.
Low Dynamics
3. Almost Static
AI
Figure 7: Illustration of prompt coarse categorization using GPT-4 30
17

---

Task: Analyze the video for anomalies and normal behaviors, then classify its realism based on the criteria below:
Displays complete detachment from reality throughout, with elements of fantasy or surrealism.
1. Completely Fantastical:
Clearly Unrealistic: Contains significant distortions over extended periods or large scale, making the overall scene unrealistic contrary physical laws, such unrealistic large objects scenes.
3. Moderately Unrealistic: Exhibits noticeable distortions temporarily or an intermediate scale, though the plot remains fairly coherent, c.g., medium-sized objects scenes appear unrealistic.
Slightly Unrealistie: Distortions are brief minute, hard notice, such as unnatural facial expressions or unnatural scene textures.
5. Almost Realistic: No noticeable distortions; aligns completely with reality.
Instructions:
List all the anomalies and normal aspects observed.
Based these observations, classify the video's realism using the above criteria.
Output Required:
User
Only return the classification the video's realism.
User
Almost Realistic
AI
User
AI
Clearly Unrealistic
Figure 8: Illustration of naturalness caleulation for generated videos using Gemini-1.5 Pro
18

---

Enter video index to start.
Load Video
Previous
Next
Question Which level of inter-frame change does this video belong
1. Almost Static: The scene almost completely still, there only minimal detail change, such as person standing still almost static sky.
2. Low Dynamics: There slow movements scene changes, such
walking slowly slowly shifting viewpoint,
moderate speed 4. High Dynamics: There are fast movements or scene changes, such as person running.
Dynamics:
scene changes,
person walking normal pace
bynamics: There very
transformations
vehicles significant
Very High Dynamics:
Almost Static: 1
Low Dynamics:
Medium Dynamics: 3
High Dynamics:
Question Which level of inter-segment dynamic variation repetition does this video belong to?
1. Almost Static: The video almost repetitive with only minor changes, such as nearly static objects.
Dynamics:
The video
almost repetitive with slight variations, such shaking. recurs,
Dynamics: Some
changes one major change background example, changes
pedestrian traffic flow.
contains repetition, with unpredictable esample, movements players in sports match.
High
objects changes background For
Dynamics: Content non-repatitive, oxtremely complex object changes. example, tightly edited sequences live broudsasts of high-speed speed races.
Very High Dynamics:
Almost Static:
Low Dynamics:
Medium Dynamics:
High Dynamics:
Question Which level of global dynamics does this video belong to?
changes, only minimal mation
changes.
2. Low Dynamics: Slight changes or slow movements, very little object variation, limited viewpoint changes.
3. Medium Dynamics: Noticeable changes objects or background, sigrificant movement changes quantity and appearance objects, the background
completely
still depicts
scene.
4 High Dynamics: Quick changes in objects and scenes, and rich actions.
Dynamics: Significant changes objects,
quantity frequent changes appearance, very repid background changes,
froquent scene and viewpoint switches, and changes the depicted scenes.
Very High Dynamics:5
Almost Static:1
Dynamics:2
Medium Dynamics:3
High Dynamics:4
Question What is the level of naturalness in this video's content?
fantasy Clearly Unreat: Long term macro scale distortions are significant, leading to an overall departure from reality, Entire scenes are unrealistic, defy physical laws, objects unreal.
Completely Completely
Moderately Unreal: There clear distortions short-term meso-scale, sccle, but overall plot remains relatively coherent. esample, medium-sized objects unrealistic.
Slightly Unreal: Distortions occur only instantaneoualy scale, not easily noticeable. unnatural expressions unnatural scene textures.
completely reality.
Completely Unreal:
Clearly Unreal:
Moderately Unreal:
Slightly Unreal:
Almost Real:
Figure 9: Toolbox for dynamies and naturalness annotation.
19

---

VidenCraiiter
OpenSom
GEN2
"Text Prompt": "Portrait of three business people looking the
"Dynamic grade":
"Static"
VidooCrafter2
Pika
FreeNoise
"Text Prompt": "Tringa glareola. two wood sandpipers in the summer. standing on land near the lake in the north of siberia."
"Dynamic grade": "Loow"
GEN2
Noive
Streaming
"Text Prompt": "Yachting wide city river. outdoor activities, summer vacation"
"Dynamic grade":
"Medium"
20

---

GEND
Video TidenCCafttr2
Pika
"Text Prompt": "Commercial uirplane dodging lightning during turbulent storm, rain-drenched windows."
"Dynamic grade": "High"
GEN2
FreeNoise
Streaming T2V
"Text Prompt": "High-speed shots of a volcanic eruption engulfing tropical island, with lava fountains spewing molten rock and the environment transforming from idyllic paradise hellish landscupe ash fire."
"Dynamic grade": "Very High"
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the claims made, including the proposal of the DEVIL evaluation protocol for evaluating T2V models focusing video dynamics, the creation of a benchmark with dynamics scores, and the improvement of existing evaluation metries from a dynamics perspective.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
21

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes section discussing limitations, such as the limited number of dynamics grades and the need for more comprehensive evaluation of T2V models. (Refer to the Limitations section).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results that require formal assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
22

---

Answer: [Yes]
Justification: The paper provides sufficient details on the experimental setup, including data construction, and evaluation metrics, allow reproduction of the main results. (Refer the method and experiments section).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis-
sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper does not provide open access to the data and code due to restrictions, but detailed instructions for reproducing the results are included. (Refer to the Supplemental Material for instructions).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
public/ wwblic/guides/ httii///uudiee//eeuuimieeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https //nips http:///c/ii///oo/////////aaaaaa
for more details.
23

---

The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specifies all relevant training and test details. (Refer to the Experi- ments section).
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
The paper reports error bars and confidence intervals for the main experimental results, providing information about the statistical significance of the findings. (Refer the Results and Analysis section).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
24

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The paper provides details on the type of compute workers (GPUs), memory requirements, and time of execution for each experiment, ensuring reproducibility. (Refer to the Experiments section).
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses potential positive impacts, such as promoting the develop- ment of T2V models through improved evaluation methods, and negative impacts, including the risk of misuse in creating realistic but misleading video content.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
25

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits the creators and original owners of the assets used, and the licenses and terms of use are explicitly mentioned and respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduces new assets, such as the DEVIL benchmark and associated metrics, and provides thorough documentation to ensure reproducibility and proper usage.
Guidelines:
26

---

The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
The paper includes detailed instructions provided to human subjects for the user study, along with information about compensation. (Refer the User Study section in the Methods and Appendix).
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper describes the potential risks to study participants, confirms that these risks were disclosed, and states that IRB approval was obtained for the user study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "f6Meo7Gwwk", "reviewer": "Reviewer_7oNg", "summary": "The paper presents a comprehensive study on the evaluation of Text-to-Video (T2V) generation models, with a particular focus on the dynamics of video content. The authors introduce a novel evaluation protocol named DEVIL, which aims to address the often overlooked aspect of dynamics in existing evaluation metrics. This protocol defines a set of dynamics scores that correspond to multiple temporal granularities and introduces a new benchmark of text prompts under various dynamics grades. \n\nIn addition, this paper verifies a significant issue with some existing evaluation metrics: they allow models to attain high scores by generating low-dynamic videos. The authors demonstrate that the metrics provided by DEVIL align closely with human in rating dynamics, indicating its potential to significantly advance T2V generation models. The paper also highlights the current limitations in T2V models and datasets, and provides valuable suggestions for future research in this field.", "questions": "Please refer to weakness section.", "limitations": "The authors addressed the limitations and societal impact in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces an innovative T2V model metric, DEVIL, which evaluates the content dynamics of generated videos—an aspect often overlooked by current evaluation methods but essential for realistic video generation. DEVIL assesses dynamics at multiple granularities, achieving high correlation with human evaluations. By revealing the negative correlation between existing metrics and dynamic scores, the paper challenges the current evaluation standards.\n2. The study employs robust experiments to validate the dynamics metric (DEVIL), achieving up to 90% consistency with human evaluations. Additionally, the paper identifies issues with existing T2V evaluation metrics, revealing their negative correlation with dynamics metrics and the tendency to favor low-dynamic content, which misrepresents model performance.\n3. The theoretical framework is detailed and thorough, covering the formulation of dynamics scores across multiple temporal granularities. The paper also highlights a reliable human rating collection process, amassing 50,000 prompt dynamic level classifications for training the linear regression model used in the metrics and evaluations of 4,800 generated videos from six T2V models for proving the consistency between the metric and human ratings. \n4. The paper is overall logically structured and easy to follow.", "weaknesses": "1. How to differentiate dynamic video from videos with low-quality motions, for example, flickering or temporal inconsistency? We desire videos with not only large dynamics but also high-quality dynamics, but some models which generate low-quality motion such as flickring frames and temporally-inconsistent videos may also achieve high dynamic scores under the proposed evaluation scheme.\n\n2. Some previous works also propose metrics to evaluate dynamic levels of generated videos. For example, VBench [23] proposes dynamic degree evaluated with RAFT, and EvalCrafter [31] proposes motion quality evaluated with RAFT. The proposed metric should be compared with existing metrics from those works.\n\n3. Frame rate of videos is an important factor that may affect the metrics. For example, a video with low fps may get higher score than a model with high fps if the evaluation is conducted on all frames without aligning their fps. Do the authors take it into consideration when designing the metrics? Are the models evaluated at their original fps or samped frames at the same fps?\n\n4. In Section 3.3, how many videos are used to fit the human alignment dynamic scores? I am concerned that the metric is overfitting the selected videos for human alignment, but may not generalize well to other videos. Are the human evaluation results in Table 3 computed from the same videos as the videos used to fit the dynamic scores in Section 3.3? There is a risk of cheating if using the same set of videos to fit the human alignment dynamic score (in Section 3.3) and to calculate the human correlation (in Table 3).\n5. The effectiveness of the improved metrics proposed in Section 4 is not validated with human correlation.\n6. The relationship between the naturalness score defined in Section 4 and the rest of the paper is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1: How to differentiate dynamic video from videos with low-quality motions?**\nWe aim to motivate models to produce videos with a wide range of dynamics while maintaining high quality. We recognize that some videos may exhibit high dynamics but poor quality.\nTo address, we have enhanced quality metrics by integrating dynamics scores, enabling a more comprehensive evaluation of models' generative capabilities rather than assessing quality and dynamics independently.\nWhen filtering individual videos based on both quality and dynamics, our dynamics scores are complementary to exisiting video quality scores.\n\n#### **Q2: Compare with dynamics degree in Vbench and motion quality in EvalCrafter**\n Thank you for this insightful comment. DEVIL is superior to existing T2V evaluation protocols in three key aspects.\n\n1. Dynamics Assessment\n- Vbench and EvalCrafter are bulit solely on optical flow, limiting analysis to inter-frame pixel motion dynamics. They fail to capture non-pixelmotion dynamics, such as variations in illumination, style, and color.\n- DEVIL employs seven dynamics scores across three temporal scales (inter-frame, inter-segment, and video level) to assess video dynamics comprehensively. It significantly improves correlation with human evaluation, achieving an 84% win ratio for inter-frame dynamics, substantially outperforming optical flow's 73%.\n2. Evaluation of Models' Capabilities on Dynamics\n- Vbench and EvalCrafter created Dynamic Degree and Flow Score metrics based on optical flow, both favoring videos with greater dynamics. However, they neglect the importance of matching video dynamics to prompt requirements, which can range from low to high.\n- EvalCrafter's Motion AC-Score assesses a model's ability to generate videos with low/high dynamics according to prompts. However, as a binary metric, it only roughly measures dynamic controllability and doesn't capture the full range of a model's dynamic capabilities.\n- We propose the Dynamics Range, $M_{range}$, to drive generating videos with various dynamics, and Dynamics Alignment, $M_{align}$ to evaluate how well models can control dynamics based on text prompts. Our approach enables a more reasonable and precise assessment of a model's dynamic capabilities.\n3. Quality Metrics\n- Vbench and EvalCrafter assess video dynamics and quality independently, neglecting their correlation. As models tend to generate low-dynamics videos, quality metrics are skewed towards these, failing to accurately represent model performance across the full range of dynamics.\n- EvalCrafter's Action Recognition assesses human actions using recognition scores, but it's limited to human activities and fails to evaluate model quality across various dynamics ranges.\n- We combine dynamics with existing quality metrics to quantitatively assess models' ability to generate high-quality videos across dynamic ranges, enabling more comprehensive evaluation.\n\n#### **Q3: Influence of Frame Rate**\nThanks for the constructive comment. In the paper, we DO standardize the frame rate of each video to 8 FPS, following Vbench. Experiments show our dynamics evaluation maintains high correlation (>0.9) with human ratings across various frame rates. The table below shows Pearson correlation between our dynamics scores and human ratings:\n\n| Dynamics    | 4FPS  | 8FPS  | 16FPS | Origin FPS |\n|-------------|-------|-------|-------|------------|\n| Inter-frame | 0.952 | 0.950 | 0.946 | 0.951      |\n| Inter-Segm  | 0.952 | 0.954 | 0.954 | 0.953      |\n| Video       | 0.967 | 0.967 | 0.967 | 0.967      |\n\n#### **Q4: Details of human alignment**\nWe used a 75/25 split of the annotated data, 75%(3600 videos) for fitting the human alignment dynamic scores, 25%(1200 videos) reserved as a test set. Results in Table 3 are computed exclusively from the 25% test set.\n\n#### **Q5: Effectivenss of the improved metrics**\nImproved Metrics aims to achieve a more comprehensive and accurate evaluation of model performance by employing grouping assessment based on video dynamics. The effectiveness of Improved Metrics can be demonstrated from two perspectives:\n1. Representativeness\n\nImproved Metrics can better represent models' capability to generate high-quality videos with varying levels of dynamics, as measured by the root mean squared bias. As the table below demonstrates, our Improved Metrics achieve a **lower root mean squared bias** than the Original Metrics.\n\n| Metric | Motion Smoothness | Subject Consistency | Background Consistency | Naturalness |\n|--------|-------------------|---------------------|------------------------|-------------|\n| Improved | 0.44 | 0.41 | 0.43 | 0.29 |\n| Original | 0.56 | 0.55 | 0.56 | 0.39 |\n\n2. Human Correlation\n\nOur metrics are defined for the entire model rather than individual videos. Annotators grouped videos by their dynamics and assessed their quality, focusing on naturalness due to time limitations. We then averaged the scores for each dynamic group to obtain a comprehensive human score for each model. The table below shows the human correlation evaluation of the original metrics and the improved metrics.\n\n| Metric | Pearson's Correlation | Kendall's Correlation |\n|--------|----------------------|----------------------|\n| Improved | 0.70 | 0.60 |\n| Original | -0.60 | -0.33 |\n\nOur improved metric achieved a Pearson correlation of 0.70 with human scores, confirming its effectiveness. The original metric, which averages video quality without considering dynamics, is dominated by low-dynamic videos. This oversight leads to a negative correlation with human scores, despite accurate individual video ratings.\n\n#### **Q6: Naturalness**\nNaturalness is inspired by the observation that dynamics also accompany unnatural scenarios, e.g., car wheels spinning rapidly while the vehicle remains stationary. We thus propose naturalness to reflect how much the generated videos are like camera-captured ones.  Note that it achieved a correlation of 79% with human ratings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dH4rfCt2yE", "reviewer": "Reviewer_HC61", "summary": "This paper proposes DEVIL, an evaluation suite of metrics for evaluating text-to-video generation, focusing on dynamics (the authors note that many previous works on video generation focus on other aspects but ignore dynamics). Proposed metrics are computed using a number of automatically extracted values based on e.g. optical flow or autocorrelations and then a linear combination of these values are learned such that they align with human perceptions of dynamics.\n\nThe authors also rate prompts (automatically using gpt4) to assign a dynamics “grade” which allows them to evaluate alignment with the level of dynamics asked for by a text prompt.  They also report a “dynamic range” of a given model across the set of prompts. Finally the authors use their proposed metrics to evaluate a number of recent video generation models.", "questions": "* I believe that the proposed measures are not necessarily able to separate camera motions from object motions.  Similarly, it is not able to distinguish between fast moving “video textures” (e.g. roaring flame) which tend to be easy to generate using these models vs more specific motions (e.g. ninja doing a backflip).  I would ask the authors to comment on whether these are real limitations or not.\n* Suggestion: Please add examples of prompts with high dynamics and low dynamics in the main body of the paper.\n* Suggestion: Report linear regression weights (given that there are just a few weights and would allow the reader to evaluate the relative importance of each factor)\n* Question: Will the code/prompts be made public?\n* Are the “improved metrics” better correlated with human judgements too?  I am not sure if I saw this somewhere in the paper\n* Suggestion (but lower priority): I believe Runway has a control that lets you control the dynamics (by changing CFG weight) — it would be interesting to see the effect of this control on measures proposed in this paper.", "limitations": "yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper attacks a useful problem in video generation that researchers talk about / are aware of, but don’t have good metrics to measure.  Overall the proposed metrics seem quite thorough and as shown in the results, the proposed measures do seem to be quite correlated with human judgements.  I think this would be a good contribution to the community particularly if made publicly available.", "weaknesses": "Conceptually the novelty of the paper is mostly incremental but since good measures of dynamism in video generation do not currently exist, the work should be of good value to the community.\n\nOne potential weakness is that I am not sure if I agree with the decision to segment videos based on length relative to total number of frames since this ignores the fact that different models can generate widely different video lengths or at quite different framerates(e.g. 2 seconds at 8 fps for VideoPoet videos and some examples from Sora that go to 1 min at 24 fps).\n\nThere is also sloppy writing in some parts of the paper.  Examples:\n* Some tables are never referenced in the body of the text\n* Authors mention that human “refine” the dynamics grades but no details are given other than to say that they did this for three months (which is mostly meaningless).\n* The word cloud in Figure 3 is not very interpretable — is this for just the high dynamics prompts or everything? What is the take-away?\nEqn 3 (which mentions 5%) does not agree with the paragraph above \n* The function SIM is never defined\n* Orphaned footnote in line 248\n* Plots in Fig 5 are attractive but very hard to actually compare across the different methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1: Concern about segmenting videos based on length relative to total number of frames.**\n1. **Why using relative length.** Segmenting videos by relative length enables standardized comparison no matter how long the video is. The following table shows consistently high correlation values with different ratios, demonstrating our method's robustness.\n\n| ratio | PC   | KC   |\n|-------|------|------|\n| 1/8   | 0.92 | 0.90 |\n| 1/4   | 0.94 | 0.91 |\n| 1/2   | 0.93 | 0.90 |\n\n2. **Influence of frame rate**: We standardized videos to 8 FPS, following Vbench protocol, to eliminate frame rate variability effects.\n\n3. **Influence of video length**: We group videos based on the video length (max is 8s in tested models) and study the relation between dynamics scores and human scores. DEVIL robustly achieves over 90% correlation whatever the video length is.\n\n| Video length | PC   | KC   |\n|----------------|------|------|\n| 2s             | 0.96 | 0.94 |\n| 4s             | 0.93 | 0.91 |\n| 8s             | 0.94 | 0.90 |\n\n#### **Q2: Necessity of Distinguishing Motion Types**\nThanks for highlighting this. Our paper focuses on creating a comprehensive dynamic scoring system for various motions, rather than differentiating specific motion types. In video generation, factors such as object categories, contexts, and actions affect efficacy. Therefore, we designed a benchmark with 19 object categories, 4 scenes, and 10 dynamic categories, covering over 40 subtypes. These subtypes include camera movements, actions, complex effects, and environmental dynamics, ensuring our evaluation system accurately measures video generation performance across diverse scenarios.\n\n#### **Q3: Examples of prompts**\nThe following are examples of prompts with different dynamics grades, we will add them to the main paper.\n\n**Static:**\na man is laying on the ground.\n\n**Low dynamics:**\nA male fencer adjusts his epee mask and prepares to duel with his sparring partner in slow motion.\n\n**Medium dynamics:**\nTilt up of shirtless sportsman doing pull-ups on bars during cross-training workout at gym.\n\n**High dynamics:**\nA runner explodes out of the starting blocks, racing down the track.\n\n**Very High dynamics:**\nA medieval siege with catapults launching, walls breaking, soldiers charging, and arrows raining down.\n\n#### **Q4: Report linear regression weight**\nThe linear regression weight of each dynamics score is as follows, we will add them to the main paper.\n\n| Temporal Scale | Dynamics Score | Typical Value | Weight |\n|----------------|----------------|---------------|--------|\n| Inter-frame | $D_{ofs}$ | 62.00 | 6.70E-04 |\n| | $D_{sd}$ | 1.00 | 0.17 |\n| | $D_{pd}$ | 33.00 | 0.03 |\n| Inter-segment| $D_{pa}$ | 0.80 | 0.63 |\n| | $D_{ga}$ | 0.20 | 2.20 |\n| Video | $D_{te}$ | 7.00E+04 | 1.00E-05 |\n| | $D_{tsd}$ | 0.20 | 1.46 |\n\n#### **Q5: Code release**\nYes, we will release the code, rompts, and weights.\n\n#### **Q6: Effetiveness of improved metrics.**\nImproved Metrics enhances model performance evaluation through grouping assessment based on video dynamics. Its effectiveness is demonstrated from two perspectives:\n1. **Representativeness**.\nImproved Metrics better represent a model's ability to generate quality videos with varying dynamics, as shown by lower root mean squared bias compared to Original Metrics. The table below demonstrates this improvement.\n\n| Metric | Motion Smoothness | Subject Consistency | Background Consistency | Naturalness |\n|--------|-------------------|---------------------|------------------------|-------------|\n| Improved | 0.44 | 0.41 | 0.43 | 0.29 |\n| Original | 0.56 | 0.55 | 0.56 | 0.39 |\n\n2. **Human Correlation**.\nImproved metrics evaluate the model as a whole, not individual videos. To evaluate consistency with human scores, annotators grouped videos by dynamics and assessed their quality(focusing on naturalness due to time limitations). We averaged scores for each group to derive a comprehensive human score per model. The table below compares the human correlation of original and improved metrics.\n\n| Metric | Pearson's Correlation | Kendall's Correlation |\n|--------|----------------------|----------------------|\n| Improved | 0.70 | 0.60 |\n| Origion | -0.60 | -0.33 |\n\nOur improved metric shows a 0.70 Pearson correlation with human scores, confirming its effectiveness. The original metric, dominated by low-dynamic videos, overlooks dynamics and shows a negative correlation with human scores, despite accurate individual video ratings.\n\n#### **Q7: Effetiveness of Runway CFG**\nDuring Runway GEN-2's evaluation, we considered CFG functionality by assigning different CFG weights based on the prompts' dynamics grade. With such strategy, the model achieves a high degree of dynamics control capability, as evidenced by the high Dynamics Alignment metric.\n\n#### **Q8: Some tables are never refered**\nWe will reference the previously uncited Table 1 and Table 2 in Section 3 and Section 4 respectively.\n\n#### **Q9: Details of Human Refinement**\nWe have designed detailed criteria and examples for each dynamics grade, used by both human annotators and GPT-4. See Fig. 7 in the Appendix for more information. This will be clarified in the paper.\n\n#### **Q10: Figure 3 is not very interpretable**\n\n1. Fig. 3 is a word cloud of all prompts in the DEVIL prompt benchmark. It shows that users often use similar terms like \"rapid\" and \"extremely fast\" for high-dynamics scenarios, making these words more prominent.\n2. The number in the preceding paragraph of Eqn 3 should be corrected to 5% to match Equation 3.\n\n#### **Q11: The function SIM is never defined**\nSIM represents the cosine similarity between two feature vectors.\n\n#### **Q12: Orphaned footnote in line 248**\nThank you for pointing out. We will make the necessary corrections.\n\n#### **Q13: Plots in Fig 5 are attractive but very hard to actually compare across the different methods**\nWe'll improve Figure 5 to make it more clear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MAb60mrdAJ", "reviewer": "Reviewer_VYfN", "summary": "Effective evaluation protocols are essential for developing advanced text-to-video (T2V) generation models. Current protocols primarily address temporal consistency and content continuity but often neglect the dynamics of video content, which are crucial for visual vividness and fidelity to text prompts. This study introduces DEVIL, a protocol that emphasizes dynamics by defining metrics across multiple temporal granularities and creating a benchmark of text prompts graded by dynamics levels. DEVIL evaluates T2V models using metrics of dynamics ranges and T2V alignment, enhancing existing metrics from a dynamics perspective. Experimental results show that DEVIL achieves up to 90% consistency with human evaluations, demonstrating its potential as a powerful tool for advancing T2V generation models.", "questions": "Please refer to the weaknesses.", "limitations": "This paper discusses the limitations in Section 6.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Introduction of New Dynamics Metrics**: The study presents a novel set of dynamics metrics that enhance the evaluation of text-to-video (T2V) generation models. These metrics focus on the dynamics dimension, which assesses the visual vividness and fidelity of video content to text prompts across multiple temporal granularities. \n\n2. **Comparative Analysis with Existing Methods**: The research includes a thorough comparative analysis of existing evaluation methods, highlighting their limitations in addressing video dynamics. By juxtaposing these methods with the newly proposed dynamics metrics, the study demonstrates how the latter offers a more detailed assessment of T2V models. \n\n3. **Code Submission for Reproducibility**: The authors have submitted the code used in their study. This openness is crucial for validating the proposed methods and fostering innovation in the community.", "weaknesses": "1. **Difficult to Assess Dynamics from Images Alone**: Evaluating the impact of dynamics solely through images is challenging. To better understand the effectiveness of the proposed dynamics metrics, it would be beneficial to provide several real videos along with their corresponding scores. This would allow for a more accurate assessment of how well the metrics reflect the dynamics and visual vividness of real video content. Without these examples, it is hard to gauge the practical utility and accuracy of the dynamics scores.\n\n2. **Comparison of Existing T2V Methods**: Similarly, assessing the dynamics of generated videos from existing T2V methods using only images is insufficient. To comprehensively evaluate the effectiveness of the proposed dynamics metrics, the authors are recommended to include several examples of fake videos generated by existing methods along with their corresponding dynamics scores. This comparison would help in understanding how the new metrics perform relative to current standards and whether they offer a significant improvement in evaluating video quality.\n\n3. **Computational Efficiency**: The computational efficiency of the proposed method is not well addressed. Understanding the resource requirements and processing time for the new dynamics metrics is crucial for practical applications. If the method is computationally intensive and very slow, it might limit its applicability in real-world scenarios where quick and efficient evaluations are necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1: Demonstrations of Dynamics Score of Real Videos (weakness1) and Fake Videos from Different T2V Methods (weakness2).**\nThank you for your valuable suggestion. We have collected some real videos and fake videos from different T2V models with dynamic scores for demonstration. However, due to NeurIPS guidelines, we **CAN'T** include links in the rebuttal. Therefore, we present them as images in the PDF instead. Please refer to the attached files. We will also create a project page containing the demo videos in the future for a more intuitive understanding, as you suggested.\n\nAs demonstrated in the paper, our dynamics scores correlate with human scores at a level exceeding 90%. \nThe improved metrics can better reflect the quality of generation across different levels of dynamics, as measured by root mean square bias.\nAs shown in the table below, the improved metrics achieve a **lower root mean square bias** in assessing video quality across various dynamic levels for Motion Smoothness, Subject Consistency, Background Consistency, and Naturalness.\n| Metric | Motion Smoothness | Subject Consistency | Background Consistency | Naturalness |\n|--------|-------------------|---------------------|------------------------|-------------|\n| Improved | 0.44 | 0.41 | 0.43 | 0.29 |\n| Original | 0.56 | 0.55 | 0.56 | 0.39 |\n\n\n#### **Q2: Computational Efficiency**\nThe evaluation of proposed dynamics metrics incur low computational cost, processing approximately 10 FPS on a single NVIDIA A100 GPU with support for multiple GPUs. Evaluating models like VideoCrafter 2 takes about 20 minutes, reducible to minutes with parallel GPU processing—a common practice when developing video generation system.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K5b06vaCFB", "reviewer": "Reviewer_Kt6U", "summary": "This paper proposes a new evaluation metric for the text-to-video model, and this metric in particular focus on the dynamics on the generated video. Their metric is based on three sub-scores: inter-frame dynamics score, inter-segment dynamics score, and video-level dynamics. They did", "questions": "- Why is this protocol called DEVIL?\n\n- What are the cross and circles in the Dynamics Evaluation plot in Figure 2? I'm confused what this plot wants to convey.\n\n- Figure 3 may need to be improved. (b) is only the word cloud for DEVIL instead of for all three benchmarks.\n\n- The reference in 136 line seems to be incorrect, it is not a paper for optical flow.\n\n- What's SIM in equation (2)? \n\n- What does I function mean in equation (14)?\n\nTypo:\n\nLine 223: overall h??", "limitations": ".", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "It explores a more fine-grained protocol to evaluate the dynamics of the T2V generation.\n\nThey have done extensive evaluation on the exiting T2V models, and obtained some insights.\n\nThey propose the \"improved metric\", which would give higher weight to the large-dynamic range compared to the \"existing metric\". It would be useful to detect whether the generated videos of a model cover all dynamic ranges.", "weaknesses": "I feel this paper over-emphasizes existing evaluation works ignore the dynamics, this is not quite what I saw. I think most of papers already paid attention to the dynamics evaluation, they are just not so detailed and fine-grained as in this paper. For example, EvalCrafter has Motion Quality section for considering the dynamics.\n\nI think there are already some existing metric to quantify the dynamics in the video, and they're simpler, such as Motion Quality in EvalCrafter. I think there should be a direct comparison between the proposed metric and the existing dynamics metric. Maybe the authors can consider to use the average of three motion quality metrics in EvalCrafter (after normalization) and show which one is aligned with human evaluation better?\n\nAnother problem is for the Inter-segment Dynamics Score, I think the way to take segments is too brutal-force and sketchy, for example, just take every 8 frames. This maybe totally misaligned with the actual semantic segments, in that case, I feel the Inter-segment Dynamics Score does not make much sense.\n\nFor the naturalness metric's definition, why can we trust Gemini-1.5pro? Authors may want to justify this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### **Q1: Compare to existing metrics, such as Motion Quality in EvalCrafter. Consider to use the average of three motion quality metrics in EvalCrafter (after normalization) and show which one is aligned with human evaluation better?**\nThe differences between our method and the existing metrics, e.g., Motion Quality in EvalCrafter, are as follows:\n1.  Dynamics Assessment\n- Vbench and EvalCrafte rely solely on optical flow, limiting analysis to inter-frame pixel motion dynamics. They fail to capture non-pixel-motion dynamics, such as variations in illumination, style, and color.\n- Our method employs seven dynamic indicators across three temporal scales (inter-frame, inter-segment, and full-video), comprehensively assessing video dynamics. This approach significantly improves correlation with human evaluation, when evaluating inter-frame dynamics it achieving an 84% win ratio, surpassing the 73% of optical flow.\n2. Evaluation of Models' Capabilities on Dynamics\n- Vbench and EvalCrafter created Dynamic Degree and Flow Score metrics based on optical flow, both favoring videos with greater dynamics. However, they overlook the need for videos to match varying dynamics in text prompts, such as low dynamics for low-dynamics descriptions and high dynamics for high-dynamics descriptions.\n- Although EvalCrafter's Motion AC-Score assesses a model's ability to generate videos with high or low dynamics, it is a binary metric, which is a coarse measurement and can not reflect the true dynamic controllability.\n- Instead, our Dynamics Range, $M_{range}$, measures a model's ability to generate videos with various dynamics, and Dynamics Alignment, $M_{align}$ measures how well models can align video dynamics to text prompts across five grades, enabling a more reasonable and precise assessment of a model's capabilities on dynamics.\n3. Quality Metrics\n- Vbench and EvalCrafter assess video dynamics and quality separately, neglecting their correlation. It results in that the models tend to generate low-dynamics videos to ensure high quality, limiting to reflect the model quality across different dynamics.\n- EvalCrafter's Action Recognition is limited to evaluating human actions using recognition scores and doesn't directly evaluate video quality across different dynamics.\n- By incorporating dynamics into existing quality metrics, we quantitatively assess a model's ability to generate high-quality videos across dynamic ranges, achieving a more comprehensive evaluation.\n\nFollowing your suggestion, we use the average of motion quality metrics in EvalCrafter (after normalization) and calculated the correlation of this average metrics with human evaluation. It obtains 83% Pearson's correlation andb 74% win ratio, which are significantly lower than those in our proposed dynamics metric ( 95% Pearson's correlation and an 84% win ratio).\n\n#### **Q2: Inter-segment dynamics Score**\nWe would like to clarify that the length of each segment is not 8 frames. Instead, segments are divided proportionally according to the video's duration when evaluating global aperiodicity. Experimental results indicate that the global aperiodicity in inter-segment dynamics is robust with respect to the proportion, maintaining a high correlation with human evaluations (>90%) when assessed at proportions of 1/8, 1/4, and 1/2 of the video's length.\n\n| Proportion |  PC  |  KC  |\n|----------|----|----|\n|    1/8     | 0.92 | 0.90 |\n|    1/4     | 0.94 | 0.91 |\n|    1/2     | 0.93 | 0.90 |\n\nBased  the on the observation that different patches within the video take different lengths of time to change, we assess the correlation of changes over different time interval (from 1 to the full video length) base on the temporal autocorrelation factor in this paper. This constructs the Inter-segment Dynamics feature combining with the global aperiodicity score, which achieves a 95% correlation with human judgments.\n\n#### **Q3: Why trust Gemini 1.5 Pro for Naturalness Evaluation**\n1. Gemini 1.5 Pro has been validated to have the best video comprehension capabilities among Multimodal Large Language Models (MLLM) currently available [a].\n2. Our validation indicates that Gemini 1.5 Pro achieves a 78% correlation with human ratings in terms of naturalness as shown in Table 3 in the paper.\n\n[a] Fu C, Dai Y, Luo Y, et al. Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis[J]. arXiv preprint arXiv:2405.21075, 2024.\n\n#### **Q4: Why is this protocol called DEVIL?**\n\"DEVIL\" stands for “**D**ynamics-based **E**valuation of **VI**deo generation mode**L**s”. The name also hints at the tough challenges that dynamics bring to video generation models.\n\n#### **Q5: What are Cross and circles in Figure 2**\nWe apologize for any confusion. The rightmost plot in Figure 2 illustrates the calculation process for Dynamics Alignment and Dynamics Range. A circle represents videos where the dynamics are correctly aligned with the prompt, while a cross indicates videos where the dynamics are misaligned.\n\n#### **Q6: Figure 3 may need to be improved. (b) is only the word cloud for DEVIL instead of for all three benchmarks.**\nThanks for pointing this out. It is true that Figure 3 (b) is the DEVIL's word cloud instead of all three benchmarks. We will rectify the caption.\n \n#### **Q7: Wrong reference in line 136**\nThank you for pointing out the error. We have updated the reference to:\nTeed, Zachary and Jia Deng. \"RAFT: Recurrent All-Pairs Field Transforms for Optical Flow.\" European Conference on Computer Vision (2020).\n\n#### **Q8: What's SIM in equation (2)?**\nSIM in equation (2) represents the cosine similarity between two feature vectors.\n\n#### **Q9: What does I function mean in equation (14)?**\nThe I in Equation (14) refers to the indicator function.\n\n#### **Q10: Line 223: overall h???**\nThank you for pointing out the typo. The correct sentence should read:\n\"The overall naturalness is then determined by averaging the scores of all videos.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thanks to all reviewers and ACs for the valuable comments and suggestions. In the original review, all the reviewers acknowledged the contributions of the proposed evaluation protocol(DEVIL). The strengths are summarized as follows:

1. Reviewer R-7oNg: "The paper introduces an **innovative** T2V model metric", "the theoretical framework is detailed and thorough", "employs **robust experiments**... achieving up to **90% consistency** with human evaluations", "The paper is overall logically structured and easy to follow."
2. Reviewer  R-HC61: "This paper attacks a **useful problem** in video generation that **researchers talk about / are aware of**, but don’t have good metrics to measure", "quite **correlated with human judgements**", "**good contribution** to the community".
3. Reviewer R-VYfN: "The study presents a **novel** set of dynamics metrics", "offers a **more detailed assessment** of T2V models".
4. Reviewer  R-Kt6U: "It explores a **more fine-grained protocol**", "propose the 'improved metric'... would be **useful** "

The major concerns and suggestions are mostly about novelty (R-7oNg, R-Kr6U), effectiveness of improved metrics (R-7oNg, R-HC61), computation efficiency (R-VYfN), technical details (R-7oNg, R-HC61) and paper writing (R-HC61, R-Kt6U). 

During the rebuttal, we carefully considered the reviewers' feedback and provided results of more experiments. We believe our responses can address reviewers' concerns and enhance our work.

Meta-review: This paper examines the evaluation protocols for text-to-video generation. Initially, the reviewers identified weaknesses in assessing motion dynamics, the fairness of comparing FPS, comparisons with existing metrics for evaluating dynamics, and the details about human evaluation. In the rebuttal, the authors clarified several points and provided additional results to support their claims, leading two reviewers to raise their ratings to borderline accept. Ultimately, no reviewers opposed the acceptance of this work. The remaining expectation is for the authors to incorporate their changes in the rebuttal into their camera-ready version. Therefore, the AC recommends accepting this submission.
Decision: Accept (poster)