Evaluation of Text-to-Video Generation Models: A Dynamics Perspective

Authors: mingxiang  liao
hannan  lu
xinyu  zhang
fang  wan
tianyu  wang
yuzhong  zhao
wangmeng  zuo
qixiang  ye
jingdong  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-03 04:34:40

Abstract: Comprehensive and constructive evaluation protocols play an important role when developing sophisticated text-to-video (T2V) generation models. Existing evaluation protocols primarily focus on temporal consistency and content continuity, yet largely ignore dynamics of video content. Such dynamics is an essential dimension measuring the visual vividness and the honesty of video content to text prompts. In this study, we propose an effective evaluation protocol, termed DEVIL, which centers on the dynamics dimension to evaluate T2V generation models, as well as improving existing evaluation metrics. In practice, we define a set of dynamics scores corresponding to multiple temporal granularities, and a new benchmark of text prompts under multiple dynamics grades. Upon the text prompt benchmark, we assess the generation capacity of T2V models, characterized by metrics of dynamics ranges and T2V alignment. Moreover, we analyze the relevance of existing metrics to dynamics metrics, improving them from the perspective of dynamics. Experiments show that DEVIL evaluation metrics enjoy up to about 90\% consistency with human ratings, demonstrating the potential to advance T2V generation models.

Main paper: 
 introduction: 
 With the rapid progress of video generation technology, the demand of comprehensively evaluating model performance continues to grow. Recent benchmarks [27, 24] have included various metrics, e.g., generation quality, video-text alignment degree, and video content continuity, to evaluate T2V generation models. Despite of the great efforts made, an essential characteristic of video, i.e., dynamics, remains ignored.Dynamics is a crucial dimension when evaluating video generation models for the following two reasons: (i) In practical applications, it is expected that generated video content is honest to text prompts, e.g., dramatic text prompts result in videos of high dynamics. (ii) In real-world scenarios, dynamics are negatively relevant to commonly used evaluation metrics, as observed by recent benchmark studies [24, 27] . This allows T2V models to 'cheat' by generating low-dynamic video content in many cases to achieve high scores upon these metrics.In this study, we introduce DEVIL, a comprehensive evaluation protocol, which assesses T2V generation models from a perspective of dynamics. DEVIL treats dynamics as a primary dimension for T2V model evaluation, as well as enhancing the completeness of existing metrics. To fulfill this purpose, we first establish a benchmark incorporating text prompts under multiple dynamics grades. The text prompts are collected from commonly used datasets [7, 6, 46, 41] and categorized using a Large Language Model (LLM), GPT-4 [30] , with further manual refinement. We then define a set of dynamics scores, which are aggregated into two dynamics metrics to reveal the temporal characteristics of generated videos. In addition, we conduct a user study to synthesize the proposed metrics into an overall dynamics score, facilitating a comprehensive evaluation of dynamics capacity, characterized by dynamics ranges and T2V alignment.To enhance the completeness of existing evaluation metrics, we introduce a bi-variate analysis strategy. Specifically, we use dynamics as an additional dimension to examine the distribution of existing metrics, such as aesthetics, continuity, and consistency. Through bi-variate analysis, we identify those metrics negatively related to dynamics, and update them by incorporating the dynamics factor. We also introduce a metric to evaluate the naturalness based on a multimodal large language model (MLLM), e.g., Gemini-1.5 Pro [1] .With the proposed DEVIL protocol, we evaluate state-of-the-art T2V models and commonly used benchmarks and find the following problems. (i) Existing generation models typically generate slow-motion videos, as most videos in existing benchmarks are of low dynamics. (ii)The text prompts in commonly used T2V benchmarks can not reflect the degrees of video dynamics. If such prompts are used to generate videos, they cause poor T2V alignment w.r.t. dynamics. (iii) By experiments, we observe that the naturalness of generated video decreases with video dynamics, which implies that the capability of simulating real-world scenarios remains to be elaborated.The contributions of this study are summarized as follows,• We propose a novel evaluation protocol, termed DEVIL, which benchmarks T2V generation models by integrating dynamics metrics. Together with existing evaluation metrics, DEVIL builds a more comprehensive evaluation protocol. • We establish a text prompt benchmark w.r.t. dynamics grades and propose a set of metrics to evaluate video dynamics across temporal granularities, facilitating the assessment of dynamics range and T2V alignment. • Extensive evaluation of existing T2V generation models allows us to thoroughly analyze the capabilities of T2V models through the proposed protocol and benchmarks. The results would inspire sophisticated T2V generation models.2 Related Work 
text-to-video generation model: 
 As a recent breakthrough in artificial intelligence, diffusion models have pushed video generation technology to a new height. Earlier studies [22, 21] explored the 3D U-Net and cascaded models for diffusion within pixel space. Recent solutions [13, 33] employed latent diffusion models to efficiently manage the diffusion process within a compressed latent space. Following these studies, a variety of approaches [40, 10, 26, 43, 16, 42, 47, 29, 25] updated and improved this paradigm. Building on these advancements, subsequent methods further explored generating videos of higher quality and extended duration. The Videocrafter approach [14] pursued high-quality video generation through disentangling spatial and temporal learning and tuning spatial modules using high-quality images.In a similar way, commercial models such as Pika [4] and GEN-2 [2] demonstrated substantial improvements, showcasing videos with exceptional visual clarity. For longer video generation, Gen-L-Video [39] aggregated short clips generated by base T2V models using temporal co-denoising to enhance continuity. Freenoise [31] extended pre-trained T2V models through rescheduling noise for longer-duration video inference. StreamingT2V [19] enhanced long-term content consistency by integrating short-term and long-term memory blocks.The rapid development of T2V models poses a growing demand for quality evaluation protocols. Unfortunately, existing protocols primarily focus on temporal consistency and content continuity, yet largely ignore temporal dynamics. This hinders the exploitation of video content vividness and the honesty of video content to text prompts.Early evaluation protocols [35] primarily relied on class labels to T2V models. For example, they commonly used video clips from the UCF-101 dataset and human-annotated video captions from the MSR-VTT [46] dataset as the evaluation data. For a more specific assessment, FETV [28] assigned fine-grained category labels to prompts and calculated the CLIP-SIM score for each category.However, conventional quality assessment metrics such as Inception Score (IS) [34] , Fréchet Inception Distance (FID) [20] , Frechet Video Distance (FVD) [37] , and CLIP-SIM typically operate on a single dimension while can not provide a comprehensive evaluation. When addressing the limitation, EvalCrafter [32] expanded both the prompt scale and the number of evaluation metrics so that the text-video alignment degree and the quality of generated videos can be better evaluated. Additionally, VBench [24] proposed a multi-dimensional, multi-category evaluation suite that not only considered the diversity of prompts but also encompassed a variety of assessment metrics.Despite of the evolution of evaluation metrics, we argue an essential characteristic of video, i.e., dynamics, remains ignored. In this study, we introduce the dynamics dimension to evaluate T2V generation models, as well as enhance the completeness of existing metrics. 
dynamics evaluation protocol: 
 As shown in Fig. 2 , we first establish a benchmark incorporating text prompts under multiple dynamics grades. The text prompts are collected from commonly used datasets [7, 6, 46, 41] and categorized to dynamics grades using GPT-4 [30] and human refinement. We then define a set of dynamics D of s = 1 N -1 N -1 i=1 FLOW(f i ) Structural Dynamics Score D sd = 1 -1 N -1 N -1 i=1 SSIM(f i , f i+1 ) Perceptual Dynamics Score D pd = 1 N -1 N -1 i=1 PHASHD(f i , f i+1 ) Inter-segment Patch-level Aperiodicity D pa = 1 -1 HW h,w ACF({F i,h,w } N i=1 ) Global Aperiodicity D ga = 1 -1 ⌊rN ⌋ ⌊rN ⌋ i=1 j̸ =i SIM(F r i , F r j ) Video Temporal Entropy D te = H(f 1 , f 2 , • • • , f N |f 1 ) Temporal Semantic Diversity D tsd = 1 N N i=1 ∥F i -F ∥ 2scores corresponding to multiple temporal granularities, to reveal the video characteristics at multiple temporal levels. We finally conduct a user study to aggregate the proposed dynamics into overall dynamics metrics about ranges of dynamics and T2V alignment.Building the benchmark includes a coarse categorization step and a post-processing step. In the coarse categorization step, the GPT-4 model is used to categorize approximately 50k text prompts collected from existing benchmarks into five grades. These benchmarks include VidProm [41] ,WebVid [8] , MSR-VTT [46] , Didemo [18] , etc. The five dynamics grades are summarized as follows: In the post-processing step, we recruit six human annotators to refine the dynamics grades following same criterion. Finally, we sample 800 text prompts at different dynamics grades for a uniform distribution along the grades.Fig. 3 shows the statistics of the DEVIL benchmark, which contains approximately 800 text prompts, and each dynamics grade includes 19 object categories and 4 scene categories. Unless otherwise specified, all experiments in this paper are conducted on the DEVIL benchmark.As illustrated in Fig. 4 , video dynamics can be classified into three categories based on their temporal granularities: (i) Inter-frame dynamics, which describes the variations between successive frames.The dynamics score at this level reflects rapid and prominent content variations. (ii) Inter-segment dynamics, which refers to the changes between video segments which contains K video frames.Define on a middle-level, this score captures middle-speed transitions and motion patterns. (iii) Video-level dynamics, which encompasses the overall content diversity and the frequency of changes throughout the video. (i) Inter-frame Dynamics Score. This is further categorized to optical flow strength, structural dynamics and perceptual dynamics.Optical flow strength. We first employ RAFT [49] to estimate the optical flow [9, 36] for each video frame. The mean optical flow magnitudes of each frame are averaged to calculate the optical flow strength of this frame. Averaging the optical flow strength values of all video frames, we have the optical flow strength D of s of the video, asEQUATIONwhere FLOW calculate the mean optical flow strength values of frame f i .Structural dynamics score. While optical flow excels in capturing motion, it is less effective when detecting structural dynamics such as lighting conditions. To capture such information, we calculate the average structural similarity index (SSIM) between consecutive frames from all frame pairs to quantify inter-frame structural variations of the video, asEQUATIONPerceptual dynamics. The human visual system is sensitive to changes in low-frequency regions of video frames. To reflect this characteristic, we introduce a perceptual dynamics metric that measures the difference between the perceptual hashes [38] of consecutive frames. The perceptual distance D pa is defined as the mean of all frame pairs, asEQUATIONwhere PHASHD denotes the Hamming distance [17] between the perceptual hash of f i and f i+1 .(ii) Inter-segment Dynamics Score. This is further categorized into patch-level aperiodicity and global aperiodicity, which measure the dynamics between video segments.Patch-level aperiodicity. We first calculate inter-segment dynamics at the patch level using the autocorrelation factor [11] , to evaluate the scene and temporal pattern dynamics. The auto-correlation factor measures the feature similarity of a time series, revealing periodicity and changing trends of features. Given features at position (h, w) across N frames, {F i,h,w } N i=1 , the auto-correlation factor of the features is defined asEQUATIONwhere SIM represents the cosine similarity between two feature vectors. The minimal segment length K 0 is empirically set to ⌊N/8⌋, as most generated videos contain more than 8 frames. H and W are the height and width of the feature map, respectively. With auto-correlation factors of all patches, we define the patch-level aperiodicity of the video, asEQUATIONGlobal aperiodicity. In addition to patch-level dynamics, we employ a global aperiodicity metric to measure the diversity of patterns between video segments. Specifically, we divide the video into segments. Each segment has a length rN , where r is a proportion factor, empirically set to 0.25. We use ViCLIP [44] to extract the spatial-temporal features for each segment. The features are denoted as {F r i } ⌊rN ⌋ i=1 . We then calculate the similarity of these features to assess the variation in spatial-temporal patterns across segments, asEQUATION(iii) Video-level Dynamics. The dynamics of a whole video sequence is defined upon the temporal entropy and temporal semantic dynamics.Temporal entropy. To evaluate the dynamics at the video level, we first measure the temporal information of each video. The temporal information is defined as the conditional entropy of the entire video sequence given the first frameEQUATIONTo estimate the conditional entropy D te , we employ the video encoding toolbox FFmpeg [15] .Temporal Semantic Dynamics. Beyond low-level dynamics, we further introduce a semantic diversity score to assess high-level dynamics across the whole video. The semantic diversity score D tsd is computed to reflect semantic-level dynamics and is defined as the variance of DINO [12] features {F i } N i=1 of each frame, asEQUATIONwhere F = 1 N N i=1 F i denotes the mean feature vector of all frames.To establish a reliable and robust assessment, we introduce a human alignment module, Fig. 2 , to refine the empirically defined dynamcis scores. It utilizes human ratings to provide ground-truth, based on which we fit a linear regression model at each temporal level, asEQUATIONEQUATIONEQUATIONwhere θ f , θ s , θ v respectively denote the model parameters of linear regression at each scale. 2 The overall dynamics score of the video is then defined as the average of aligned dynamics scores from all three levels, asEQUATIONThrough this learnable human alignment procedure, the empirically defined dynamics scores are more consistent with human perception, as validated in Sec. 5.1.After calculating the aligned dynamics scores of all generated videos at inter-frame, inter-segment, and video levels, we combine these scores together to obtain the following two evaluation metrics.(i) Dynamics Range. The metric evaluates model's capability to generate videos with vivid dynamics. A larger dynamics range implies higher dynamics capability. In detail, we determine the dynamics range M range by identifying the extremes of the dynamic scores over the benchmark, while excluding the top and bottom 1% scores to mitigate the influence of outliers. This is formulated asEQUATIONwhere Q 0.99 and Q 0.01 denote the 99th and 1st percentile values of the dynamics scores for videos generated with the DEVIL benchmark, respectively. This metric reflects a realistic spread of dynamics, excluding atypical extremes.(i) Dynamics Controllability. Let P (i) , P (j) ∈ [1, 5] respectively denote the ground-truth dynamics grades of prompt i and j, and D (i) , D (j) the predicted dynamics scores by prompt i and j. For P (i) > P (j) , we should have D (i) > D (j) so that the dynamics scores of the generated videos are consistent with the dynamics grades of text prompts. Accordingly, we can calculate the dynamics controllability metric byEQUATIONwhere |T | is the total prompt number and T i denotes the number of prompts at dynamics grade P (i) .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results that require formal assumptions or proofs. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
improving existing metrics with dynamics: 
 Table 2 : Correlation between the dynamics metric with the existing metrics including Naturalness (Nat), Visual Quality [45] (VQ), Motion Smoothness (MS) [24] , Subject Consistency(SC) [24] and Background Consistency(BC) [24] . 'PC' denotes Pearson's correlation, and 'KC' denotes Kendall's correlation.Evaluation Metrics PC KC Naturalness (Nat) -51.8 -44.2 Visual Quality (VQ) -24.8 -18.6 Motion Smoothness (MS) -64.0 -54.6 Subject Consistency (SC) -88.9 -74.9 Background Consistency (BC) -79. 4 -61.4 As observed by our experiments, existing metrics have negative relevance to video dynamics. To identify these metrics, we calculate the correlation, e.g., Pearson and Kendal correlation coefficients, between dynamics scores and existing metrics, Table 11 . These metrics include naturalness, motion smoothness, subject consistency, and background consistency. Under these metrics, models might 'cheat' for high-quality scores by generating low-dynamic videos.We improve the identified metrics by incorporating our proposed human-aligned dynamics score D. In specific, we propose to equally divide the human-aligned dynamics score into L = 13 intervals. Within each interval, we calculate the mean metric values. The mean values of the L intervals are further averaged as the improved metrics. Upon the improved metrics, to have a high score, the generated videos should spread all dynamics intervals, which implies a large dynamics range. 3Naturalness. In addition to the improved metrics, we introduce the Naturalness metric, which reflects how much the generated videos are like camera-captured ones. This is done by using the MLLM, i.e., Gemini-1.5 Pro [1] , to calculate a naturalness score for each video. The scores are categorized into five grades: "Almost Real" (100 points), "Slightly Unrealistic" (75 points), "Moderately Unrealistic" (50 points), "Noticeably Unrealistic" (25 points), and "Completely Fictitious" (0 points). The overall naturalness is then determined by averaging the scores of all videos. For evaluation, we invited five users to rate the naturalness of the generated videos and then perform a correlation analysis between human ratings and model scores. A high correlation (larger correlation coefficients) indicates the plausibility of the naturalness metric.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The paper provides sufficient details on the experimental setup, including data construction, and evaluation metrics, to allow reproduction of the main results. (Refer to the method and experiments section). Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
human alignment assessment: 
 To evaluate the plausibility of the proposed dynamics metrics and the naturalness metric, we conduct the following human alignment experiments.Ground-truth Annotation. We first generate videos using six state-of-the-art (SOTA) T2V models, including GEN-2 [2], Pika [4] , VideoCrafter2 [14] , Open-Sora [23] , StreamingT2V [19] and FreeNoise-Lavie [31] and DEVIL text prompts. For the generated videos, we collect human evaluated dynamics and naturalness as the ground-truth. Six persons are recruited to assess each video's grade of dynamics under three temporal levels (Frame, Segment, and Video). For each temporal level, evaluators are required to rate the grade of dynamics from "static" to "very high dynamics". To guide the annotation process, we provide specific prompts for each temporal level. 4 . We conduct between-group correlation analyses using Pearson's correlation, Kendall's correlation, and the win ratio to evaluate the consistency of dynamics scores with respect to human ratings. The evaluation of the naturalness metric follows the same process, where a higher human assigned grade indicates a greater degree of naturalness.Evalution. We calculate dynamics grades and naturalness for generated videos on the proposed DEVIL benchmark. For dynamics scores at multiple temporal levels, we integrate them using the linear regression model defined by Eq. 12. For each linear regression model, it takes the human evaluation results as ground-truths, trained upon 75% of the randomly selected videos and tests on the remaining 25% videos. During testing, the human alignment performance is reflected by the correlation e.g., Pearson and Kendall correlation coefficients and win ratio, between predicted and human-evaluated dynamics grades. The win ratio involves comparing each video against others with different grades of dynamics. For instance, a video rated as "high dynamics" by evaluators should score lower in dynamics than any video rated as "Very high dynamics" but higher than those rated as "static".Table 3 shows the assessment results of the six T2V models. It can be seen that the dynamics metrics and the naturalness metric exhibit a strong alignment with human evaluation. The improved metrics (D f , D s , D v defined in Sec. 3.3) further enhance the alignment with human evaluations.5.2 Influence of Frame Rate. 4 Please refer to Appendix G for details Table 5 demonstrates how frame rate influences the correlation between dynamics scores and human evaluations. Experiments indicate that our dynamics scores maintain a high correlation (>0.9) with human ratings across various frame rates. To mitigate the impact of frame rate variations on dynamics, we standardized the frame rate of each video to 8 FPS.Our dynamics metrics offer high computational efficiency, achieving around 10 frames per second on a single NVIDIA A100 GPU, and are scalable to multiple GPUs.Video Quantity Density We evaluate the dynamics range M range , and dynamics controllability M control of T2V models on the proposed DEVIL benchmark. We also evaluate dynamics ranges at different temporal scales: inter-segment dynamics rangeM f range , inter-segment dynamics range M s range , and video level dynamics range M v range . The results are shown in Table 4. The GEN-2 [2] and Pika [4] models score high in dynamics controllability but low in range due to their generation of low-dynamic videos. Conversely, the FreeNoise-Lavie method [31] attains high range but low controllability, suggesting it produces videos with dynamics that do not align well with text prompts.As shown in Table 6 , the existing metrics exhibit an obvious negative correlation when embedded with dynamics, indicating that these models can achieve high scores on these metrics by generating low-dynamic videos rather than high-quality content.Biased Dynamics Distribution of Existing Dataset. The distribution of dynamics of the video datasets (such as WebVid2M [8] ) is biased. The statistical result is shown in Fig. 5 . It can be seen that most of the videos have a small dynamics score (≤ 20). The limited number of videos with high dynamics scores restricts the model's ability to generate dynamics-rich videos which are common in , Pika [4] , VideoCrafter2(VC-2) [14] , Open-Sora(OS) [23] , StreamingT2V [19] and FreeNoise-Lavie(FN) [31] . Subplot (a) shows video quantity distribution. Subplots (b) display the distribution of quality score of generated videos in terms of Background Consistency, Motion Smoothness, and Naturalness, respectively. All videos are generated based on our text prompt benchmark. practical applications. Therefore, existing datasets should be expanded in terms of dynamics, and the proposed metrics can provide guidance for this expansion.Prompt-Video Bias of Existing Datasets. We used the dynamics controllability metric to evaluate two popular datasets, WebVid2M [8] and MSR-VTT [46] , by using the ground-truth text prompts and videos. Unfortunately, they respectively achieve alignment scores of 36.31 and 52.98. The poor performance indicates that the two datasets can not provide sufficient information/guidance while training the video generation models. To train better video generation models, the text prompt of these datasets requires to be elaborated on aspects of dynamics.Limited Real-World Simulation Ability of Existing Methods. As shown in Fig. 6 , we performed a statistical analysis of frequency, visual quality, motion smoothness and naturalness metric scores for SOTA methods based on the distribution of dynamics score. When the dynamics score is low, videos generated by these SOTA models perform well across the four metrics mentioned. As the dynamics score rises, these metrics, particularly naturalness, tend to decrease significantly. This decline may be due to the models' focus on optimizing the generation of simple, slow-motion content, with dynamics not considered in the evaluation metrics. 
conclusion: 
 We proposed DEVIL, a comprehensive and constructive evaluation protocol for T2V generation models. In the protocol, we defined a set of dynamics metrics corresponding to multiple temporal granularities, and a new benchmark of text prompts under multiple levels of dynamics. Based on the distribution of dynamics scores over the benchmark, we assessed the generation capacity of T2V models, characterized by dynamic ranges and degree of T2V alignment. Experiments show that DEVIL enjoys 90% consistency with human evaluation results, demonstrating the potential to be a powerful tool for advancing T2V generation models.Limitations. At present, the grades of dynamics remain limited, which should be improved to more fine-grained grades. Furthermore, only a limited number of T2V models are evaluated using the proposed protocol. A more comprehensive evaluation of T2V models should be done in future work. [45] , technical score [45] visual quality [45] , motion smoothness [24] , subject consistency [24] and background consistency [24] and our naturalness.  
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The paper reports error bars and confidence intervals for the main experimental results, providing information about the statistical significance of the findings. (Refer to the Results and Analysis section).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper discusses potential positive impacts, such as promoting the development of T2V models through improved evaluation methods, and negative impacts, including the risk of misuse in creating realistic but misleading video content. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [Yes] Justification: The paper includes detailed instructions provided to human subjects for the user study, along with information about compensation. (Refer to the User Study section in the Methods and Appendix). Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [Yes] Justification: The paper describes the potential risks to study participants, confirms that these risks were disclosed, and states that IRB approval was obtained for the user study. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Section 4, to identify the relevance between existing metrics with the dynamics metrics, we provide a bi-variate analysis strategy. Based on bi-variate analysis, we provide detailed correlation results for the models. In Table 11 , the Pearson correlation coefficients between the dynamics scores and existing metrics, including aesthetic score, technical score, visual quality, motion smoothness, subject consistency, background consistency, and naturalness, are detailed.The results indicate a clear trade-off between video dynamics and various existing metrics in T2V models. As dynamic complexity increases, there tends to be a decline in motion smoothness, subject consistency, background consistency, and naturalness. The aesthetic, technical, and visual quality metrics show relatively low correlation, which can be attributed to the fact that these metrics evaluate video frames independently, ignoring temporal relationships between frames.Let S (i) denote a score of generated video i. Existing metrics simply average the scores of all videos to obtain the metric score S of the T 2V model:EQUATION)where |T | is the total number of generated videos. Considering that some existing metrics show a considerable negative correlation with the video's dynamics score, they fail to prevent models from generating low-dynamic videos.To address this issue, we enhance existing metrics by integrating human-aligned dynamics scores, preventing models from attaining high scores by producing low-dynamic videos. Specifically, we first equally divide the human-aligned dynamics score into L = 13 intervals. We then calculate the mean scores S l at each interval l. The improved metric S * is defined as the average of S l across all intervals:EQUATIONAs described in Section 3.1, we collect approximately 50,000 text prompts from existing benchmarks, including 19 object categories and 4 scene categories. Using GPT-4 coarse classification and human refinement, we construct the DEVIL prompt benchmark. The process of categorizing dynamics grades using GPT-4 is illustrated in Figure 7 . In specific, we instruct GPT-4 to perform classification on the rate of content change. To enhance GPT-4's classification accuracy, we further provide detailed criteria and examples for each dynamics grade. In the post-processing step, we recruit six human annotators to refine the dynamics grades over three months. Finally, we sample about 800 text prompts at different dynamics grades to ensure a uniform distribution across the grades.We employed the advanced multi-modal large model, Gemini-1.5 Pro [1] , equipped with video understanding capabilities, to assess and classify the naturalness of video content. As shown in Fig. 8 , we demonstrate the process through which the model analyzes videos and assigns naturalness ratings. The figure details the five different levels used to evaluate video naturalness, ranging from "Completely Fantastical" to "Almost Realistic". Each level is defined based on how closely the video content aligns with the real world. Additionally, the figure includes two examples of video evaluations: the first video is rated as "Almost Realistic" due to its high conformity with reality, while the second video, due to minor distortions-such as the unrealistic number of legs on a dog-is rated as "Slightly Unrealistic". These examples validate the plausibility of the proposed naturalness metric.To align human evaluations with automated metrics, we annotated a series of videos generated by SOTA T2V models. We initiated the process by generating videos using prompts from the DEVIL benchmark with six advanced T2V models including GEN-2, Pika, VideoCrafter2, OpenSora, StreamingT2V, and FreeNoise-Lavie. Subsequently, we developed a video annotation toolbox for evaluating the dynamics and naturalness of videos. As shown in Figure 9 , the toolbox allows annotators to assess the dynamics of the videos across five grades, from almost static to very high dynamics, and the naturalness from almost real to completely unreal. To guarantee high-quality and consistent evaluations, we recruit six annotators who have undergraduate degrees and provided them with detailed training.In Section 3, we use text prompts with different dynamics grades to generate videos with T2V models.Here, we provide visual results of the generated videos.Given the provided text, classify each text segment according to the scene and background dynamics using the following criteria. For each text segment, inherit the serial number at the beginning of the text and a classification label from the list below. For each section of text, assign a dynamics grade classification based on the provided criteria. List the serial number inherit from the beginning of the text followed by the classification.High Dynamics 2.Low Dynamics 3.Almost Static … User AI User 1.A car drifts sharply around a corner, almost hitting a bystander. 2.little girl putting down and picking up her bear plush.the dead bird is on the ground. … Task: Analyze the video for anomalies and normal behaviors, then classify its realism based on the criteria below:1. Completely Fantastical: Displays complete detachment from reality throughout, with elements of fantasy or surrealism. 2. Clearly Unrealistic: Contains significant distortions over extended periods or on a large scale, making the overall scene unrealistic or contrary to physical laws, such as unrealistic large objects or scenes. 3. Moderately Unrealistic: Exhibits noticeable distortions temporarily or on an intermediate scale, though the plot remains fairly coherent, e.g., medium-sized objects or scenes appear unrealistic. 4. Slightly Unrealistic: Distortions are brief or minute, hard to notice, such as unnatural facial expressions or unnatural scene textures. 5. Almost Realistic: No noticeable distortions; aligns completely with reality.-List all the anomalies and normal aspects observed.-Based on these observations, classify the video's realism using the above criteria. "Text Prompt": "Yachting on wide city river. outdoor activities, summer vacation" "Dynamic grade": "Medium" Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The paper includes a section discussing limitations, such as the limited number of dynamics grades and the need for a more comprehensive evaluation of T2V models. (Refer to the Limitations section). Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Appendix B presents the model weights and typical values for various dynamics scores.Please refer to Appendix D for the details of the improved metrics. 


Decision: This paper examines the evaluation protocols for text-to-video generation. Initially, the reviewers identified weaknesses in assessing motion dynamics, the fairness of comparing FPS, comparisons with existing metrics for evaluating dynamics, and the details about human evaluation. In the rebuttal, the authors clarified several points and provided additional results to support their claims, leading two reviewers to raise their ratings to borderline accept. Ultimately, no reviewers opposed the acceptance of this work. The remaining expectation is for the authors to incorporate their changes in the rebuttal into their camera-ready version. Therefore, the AC recommends accepting this submission.

Reviews: review: paper_summary: paper_summary: The paper presents a comprehensive study on the evaluation of Text-to-Video (T2V) generation models, with a particular focus on the dynamics of video content. The authors introduce a novel evaluation protocol named DEVIL, which aims to address the often overlooked aspect of dynamics in existing evaluation metrics. This protocol defines a set of dynamics scores that correspond to multiple temporal granularities and introduces a new benchmark of text prompts under various dynamics grades. 

In addition, this paper verifies a significant issue with some existing evaluation metrics: they allow models to attain high scores by generating low-dynamic videos. The authors demonstrate that the metrics provided by DEVIL align closely with human in rating dynamics, indicating its potential to significantly advance T2V generation models. The paper also highlights the current limitations in T2V models and datasets, and provides valuable suggestions for future research in this field.
strength_weakness: strength_weakness: 1. How to differentiate dynamic video from videos with low-quality motions, for example, flickering or temporal inconsistency? We desire videos with not only large dynamics but also high-quality dynamics, but some models which generate low-quality motion such as flickring frames and temporally-inconsistent videos may also achieve high dynamic scores under the proposed evaluation scheme.

2. Some previous works also propose metrics to evaluate dynamic levels of generated videos. For example, VBench [23] proposes dynamic degree evaluated with RAFT, and EvalCrafter [31] proposes motion quality evaluated with RAFT. The proposed metric should be compared with existing metrics from those works.

3. Frame rate of videos is an important factor that may affect the metrics. For example, a video with low fps may get higher score than a model with high fps if the evaluation is conducted on all frames without aligning their fps. Do the authors take it into consideration when designing the metrics? Are the models evaluated at their original fps or samped frames at the same fps?

4. In Section 3.3, how many videos are used to fit the human alignment dynamic scores? I am concerned that the metric is overfitting the selected videos for human alignment, but may not generalize well to other videos. Are the human evaluation results in Table 3 computed from the same videos as the videos used to fit the dynamic scores in Section 3.3? There is a risk of cheating if using the same set of videos to fit the human alignment dynamic score (in Section 3.3) and to calculate the human correlation (in Table 3).
5. The effectiveness of the improved metrics proposed in Section 4 is not validated with human correlation.
6. The relationship between the naturalness score defined in Section 4 and the rest of the paper is not clear.
questions: questions: Please refer to weakness section.
limitations: limitations: The authors addressed the limitations and societal impact in the paper.
paper_summary: paper_summary: This paper proposes DEVIL, an evaluation suite of metrics for evaluating text-to-video generation, focusing on dynamics (the authors note that many previous works on video generation focus on other aspects but ignore dynamics). Proposed metrics are computed using a number of automatically extracted values based on e.g. optical flow or autocorrelations and then a linear combination of these values are learned such that they align with human perceptions of dynamics.

The authors also rate prompts (automatically using gpt4) to assign a dynamics “grade” which allows them to evaluate alignment with the level of dynamics asked for by a text prompt.  They also report a “dynamic range” of a given model across the set of prompts. Finally the authors use their proposed metrics to evaluate a number of recent video generation models.
strength_weakness: strength_weakness: Conceptually the novelty of the paper is mostly incremental but since good measures of dynamism in video generation do not currently exist, the work should be of good value to the community.

One potential weakness is that I am not sure if I agree with the decision to segment videos based on length relative to total number of frames since this ignores the fact that different models can generate widely different video lengths or at quite different framerates(e.g. 2 seconds at 8 fps for VideoPoet videos and some examples from Sora that go to 1 min at 24 fps).

There is also sloppy writing in some parts of the paper.  Examples:
* Some tables are never referenced in the body of the text
* Authors mention that human “refine” the dynamics grades but no details are given other than to say that they did this for three months (which is mostly meaningless).
* The word cloud in Figure 3 is not very interpretable — is this for just the high dynamics prompts or everything? What is the take-away?
Eqn 3 (which mentions 5%) does not agree with the paragraph above 
* The function SIM is never defined
* Orphaned footnote in line 248
* Plots in Fig 5 are attractive but very hard to actually compare across the different methods
questions: questions: * I believe that the proposed measures are not necessarily able to separate camera motions from object motions.  Similarly, it is not able to distinguish between fast moving “video textures” (e.g. roaring flame) which tend to be easy to generate using these models vs more specific motions (e.g. ninja doing a backflip).  I would ask the authors to comment on whether these are real limitations or not.
* Suggestion: Please add examples of prompts with high dynamics and low dynamics in the main body of the paper.
* Suggestion: Report linear regression weights (given that there are just a few weights and would allow the reader to evaluate the relative importance of each factor)
* Question: Will the code/prompts be made public?
* Are the “improved metrics” better correlated with human judgements too?  I am not sure if I saw this somewhere in the paper
* Suggestion (but lower priority): I believe Runway has a control that lets you control the dynamics (by changing CFG weight) — it would be interesting to see the effect of this control on measures proposed in this paper.
limitations: limitations: yes
paper_summary: paper_summary: Effective evaluation protocols are essential for developing advanced text-to-video (T2V) generation models. Current protocols primarily address temporal consistency and content continuity but often neglect the dynamics of video content, which are crucial for visual vividness and fidelity to text prompts. This study introduces DEVIL, a protocol that emphasizes dynamics by defining metrics across multiple temporal granularities and creating a benchmark of text prompts graded by dynamics levels. DEVIL evaluates T2V models using metrics of dynamics ranges and T2V alignment, enhancing existing metrics from a dynamics perspective. Experimental results show that DEVIL achieves up to 90% consistency with human evaluations, demonstrating its potential as a powerful tool for advancing T2V generation models.
strength_weakness: strength_weakness: 1. **Difficult to Assess Dynamics from Images Alone**: Evaluating the impact of dynamics solely through images is challenging. To better understand the effectiveness of the proposed dynamics metrics, it would be beneficial to provide several real videos along with their corresponding scores. This would allow for a more accurate assessment of how well the metrics reflect the dynamics and visual vividness of real video content. Without these examples, it is hard to gauge the practical utility and accuracy of the dynamics scores.

2. **Comparison of Existing T2V Methods**: Similarly, assessing the dynamics of generated videos from existing T2V methods using only images is insufficient. To comprehensively evaluate the effectiveness of the proposed dynamics metrics, the authors are recommended to include several examples of fake videos generated by existing methods along with their corresponding dynamics scores. This comparison would help in understanding how the new metrics perform relative to current standards and whether they offer a significant improvement in evaluating video quality.

3. **Computational Efficiency**: The computational efficiency of the proposed method is not well addressed. Understanding the resource requirements and processing time for the new dynamics metrics is crucial for practical applications. If the method is computationally intensive and very slow, it might limit its applicability in real-world scenarios where quick and efficient evaluations are necessary.
questions: questions: Please refer to the weaknesses.
limitations: limitations: This paper discusses the limitations in Section 6.
paper_summary: paper_summary: This paper proposes a new evaluation metric for the text-to-video model, and this metric in particular focus on the dynamics on the generated video. Their metric is based on three sub-scores: inter-frame dynamics score, inter-segment dynamics score, and video-level dynamics. They did
strength_weakness: strength_weakness: I feel this paper over-emphasizes existing evaluation works ignore the dynamics, this is not quite what I saw. I think most of papers already paid attention to the dynamics evaluation, they are just not so detailed and fine-grained as in this paper. For example, EvalCrafter has Motion Quality section for considering the dynamics.

I think there are already some existing metric to quantify the dynamics in the video, and they're simpler, such as Motion Quality in EvalCrafter. I think there should be a direct comparison between the proposed metric and the existing dynamics metric. Maybe the authors can consider to use the average of three motion quality metrics in EvalCrafter (after normalization) and show which one is aligned with human evaluation better?

Another problem is for the Inter-segment Dynamics Score, I think the way to take segments is too brutal-force and sketchy, for example, just take every 8 frames. This maybe totally misaligned with the actual semantic segments, in that case, I feel the Inter-segment Dynamics Score does not make much sense.

For the naturalness metric's definition, why can we trust Gemini-1.5pro? Authors may want to justify this.
questions: questions: - Why is this protocol called DEVIL?

- What are the cross and circles in the Dynamics Evaluation plot in Figure 2? I'm confused what this plot wants to convey.

- Figure 3 may need to be improved. (b) is only the word cloud for DEVIL instead of for all three benchmarks.

- The reference in 136 line seems to be incorrect, it is not a paper for optical flow.

- What's SIM in equation (2)? 

- What does I function mean in equation (14)?

Typo:

Line 223: overall h??
limitations: limitations: .

Comments: title: Official Comment by Reviewer 7oNg
Final thoughts
Final thoughts
Final thoughts
comment: We greatly appreciate you raising the score. As you recommended, we will include these discussions in the revised version.
Thanks! I have raised the score. Please include these discussions in the revised paper.
Thank you so much for your positive rating!
Given that humans can only group videos into groups based on their dynamics rather than precisely quantifying each video's dynamics, we conduct between-group analysis when calculating the win ratio and correlation.

- **Correlation metrics**  calculate the between-group correlations, focusing on overall trends.

- The **Win Ratio** focuses on the accuracy of pairwise comparisons, providing a more fine-grained level of analysis.
  
Overall, scores on all metrics are high. 
The slight differences observed between correlation and win ratio can be attributed to their distinct calculation methods.

We hope that all the rebuttals can solve your concerns towards positive rating.
Thank the authors for the rebuttal. The authors addressed most of my concerns so I raised my rating to borderline accept.
I'm a bit confused, why win rate and correlation are conflicting with each other?
Before the end of the rebuttal, we look forward to your valuable feedback and further suggestions. We will take full efforts to address your concerns. Looking forward to your reply. Thank you very much.
Before the end of the rebuttal, we welcome further discussion and hope for further raising of the rating. We would like to appreciate your further encouragement. Thank you very much.
Thank you for your positive feedback and score. We will ensure to include detailed efficiency information in camera-ready version. Your support is greatly important to us, and we look forward to carrying your good wishes forward. Thanks again.
Thank you for recognizing our work and rebuttal.

Inter-segment Dynamics measures the dynamics of temporal patterns by assessing the correlation/similarity between video segments. 
Proportional video segmentation allows for the simultaneous comparisons of videos with varying lengths while taking into account the frequency of pattern changes.
In our rebuttal, we have demonstrated that Inter-segment Dynamics is robust to segment ratios. 
Additionally, we compared it with keyframe-based segmentation methods, as shown below:

| Segment method| Pearson Correlation    | Kendall Correlation    |  Win Ratio    |
|---------------|:-------:|:-------:|:-------:|
| Keyframe-based| 0.963 | 0.940 | 0.848 |
| Ratio         | 0.954 | 0.934 | 0.865 |

Our method is comparable to keyframe-based segmentation. 
In the future, we will continue to explore better methods for calculating inter-segment dynamics.
We appreciate your consideration and hope for a favorable review.

Thank you so much!
Thank you for the rebuttal. I appreciate that the authors will also create a project page containing the demo videos in the future for a more intuitive understanding; this will help everyone better grasp the effectiveness of the new evaluation metrics. I would suggest that the authors include the information regarding efficiency in the camera-ready version, as it will aid in understanding the practical utility of the evaluation. I have no further questions. Best of luck to the authors.
I'll raise my score but I'm still a bit uncertain on the segment way of Inter-segment Dynamics.
Hi Reviewer VYfN

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you,
AC
Hi Reviewer Kt6U,

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you,
AC
Hi Reviewer 7oNg, 

The discussion period is ending soon. Please take a moment to review the author's rebuttal and share any remaining concerns or questions.

Thank you,
AC