Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality
Authors: Tianle Zhang
Langtian Ma
Yuchen Yan
Yuchen Zhang
Yue Yang
Ziyao Guo
Wenqi Shao
Kai Wang
Yang You
Yu Qiao
Ping Luo
Kaipeng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Human evaluation
text-to-video models
Primary area: evaluation

Abstract: Recent text-to-video (T2V) technology advancements, as demonstrated by models such as Gen2, Pika, and Sora, have significantly broadened its applicability and popularity. 
Despite these strides, evaluating these models poses substantial challenges. 
Primarily, due to the limitations inherent in automatic metrics, manual evaluation is often considered a superior method for assessing T2V generation. However, existing manual evaluation protocols face reproducibility, reliability, and practicality issues.
To address these challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a comprehensive and standardized protocol for T2V models. 
The T2VHE protocol includes well-defined metrics, thorough annotator training, and an effective dynamic evaluation module. 
Experimental results demonstrate that this protocol not only ensures high-quality annotations but can also reduce evaluation costs by nearly 50\%.
We will open-source the entire setup of the T2VHE protocol, including the complete protocol workflow, the dynamic evaluation component details, and the annotation interface code. This will help communities establish more sophisticated human assessment protocols.

Rethinking Human Evaluation Protocol for Text-to- Video Models: Enhancing Reliability, Reproducibility, and Practicality
Tianle Zhang": 1.2, Langtian Ma², Yuchen Yan", Yuchen Zhang", Kai Wang", Yue Yang', Ziyao Guo Wenqi Shao', Yang You', Yu Qiao', Ping Luo', Kaipeng Zhang' Shanghai AI Laboratory NNational University of Singapore University of Wisconsin-Madison 4 UUniversity of California San Diego
"The University of Hong Kong
Abstract
Recent text-to-video (T2V) technology advancements, as demonstrated by models such as Gen2, Pika, and Sora, have significantly broadened its applicability and pop- ularity. Despite these strides, evaluating these models poses substantial challenges. Primarily, due to the limitations inherent in automatic metrics, manual evaluation is often considered a superior method for assessing T2V generation. However, existing manual evaluation protocols face reproducibility, reliability, and practi- cality issues. To address these challenges, this paper introduces the Text-to- videodeo Human Evaluation (T2VHE) protocol, comprehensive and standardized protocol for T2V models. The T2VHE protocol includes well-defined metrics, thorough annotator training, and an effective dynamic evaluation module. Experimental results demonstrate that this protocol not only ensures high-quality annotations but also reduce evaluation costs by nearly 50%. We will open-source the entire setup of the T2VHE protocol, including the complete protocol workflow, the dynamic evaluation component details, and the annotation interface code?. This will help communities establish more sophisticated human assessment protocols.
Introduction
Text-to-video (T2V) technology has made significant advancements the last two years and garnered increasing attention from the general community. T2V products such as Gen2 [20] and Pika [18] have attracted many users. More recently, Sora [65], powerful T2V model from OpenAI, further heightened public anticipation for the T2V technology. Predictably, the evaluation of the T2V generation will also become increasingly important, which can guide the development of T2V and assist the public in selecting appropriate models [40, 54]. This paper does a comprehensive paper survey (see Appendix F for details) and explores human evaluation protocol for T2V generation.
Automatic and human evaluation are the two main kinds of evaluation for video generation. In recent years, nearly half of video generation papers conduct only automatic evaluation, such as Inception Score (IS) [76], Frechet Inception Distance (FID) [34], Frechet Video Distance (FVD) [84], CLIP Similarity [70] and Video Quality Assessment (VQA) [82, 49]. However, these metrics meet various challenges, such as relying reference videos for calculation, overlooking temporal motion changes, and, more importantly, not aligning well with human perception [67, 54]. Undoubtedly, automatic evaluation promising research direction, but human evaluation is more convincing far.
"Coressponding author
'hh code is available at https:/ggiiabbcccmmmaateeeeeee/e
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text-to- ---eeeo Human Evaluation Protocol
Evaluation Metrics
Evaluation Method
Evaluator
Objective
birthday day cake on plate
Video Quality
Temporal Quality
Video Quality
Instruction & Examples
Temporal Qualitty
Core Question: Which video more Train realistic and aesteeically pleasing?
Quality
Text Alignment
Motion Quality
Quality
Quality
Left is better? Right batter? Equal?
select "Equal"
Text Alignment
Alignmeet
Reference
Subjective
Ethical Robustness
Ethical Robustness
Robustness
Human Preference
Analysis In the left video. Conclusions Right better
Human Preference
Quantification: and Kupper model
Human Preference
Accelerate
Dynamic Evaluation Module
Step Pre-annetaation Sorting
Step2: Static Annotation
Step3: Dynamic Ammotation
Arenoiation Prinntl
intralize
Detoomine annotation objects
Lipelate
Core Question: Which video is more renlistic and aesthetically pleasing?
Dustie
Pasett
Home Houee silent Tqgal only ohen unt adder perfor identaally
For mumpts pleasing
more recletee ent adees rights Bamer".
Reference perspeclives:
Video Sarie Anim whyther operon light wiibie hond
lumping
yi the latuua Logethor looms.
Tempors Call
Left Benser Right Besar:: Equi
Text birthday cake plate
Tatli
Laftis Deitar Sater toal
Actees Lokies Saeut aesther ofach
Fare color
afacis
Riiccll Rahutteer
Luft Better
light keewee Equil
co creteeciin from colors une mouthed light any theed toright
Vidoo fvir 22 of 2000
Culandl
Co-d vedduaee
Peeecen
Figure 1: (a) An illustration of our human evaluation protocol. (b) The annotation interface, wherein annotators choose the superior video based on provided evaluation metrics. (c) Instruction and examples to guide used to the "Video Quality" evaluation.
Existing human evaluation for video generation also meets reproducibility. reliability, and practicabil- ity challenges. Our survey reveals that few papers employ consistent human evaluation protocols, evidenced by significant disparities in assessment metrics, evaluation methods, and annotator sources across studies. For example, some papers [100, 53] use Likert scales while others prefer compara- tive approach [40, 26]. Moreover, many papers lack detailed information on evaluation protocols, impeding further analysis and reproducibility. Additionally, most papers rely on annotators recruited directly by the authors, i.c.. laboratory recruited annotators (LRAs). These articles often lack quality checks, which can introduce bias and affect reliability [8]. Furthermore, there notable variation in the number of annotations used across papers, ranging from few dozen to tens of thousands, posing a practical challenge in achieving a balance between credible results and resource constraints.
This paper introduces Text-to- Video Human Evaluation (T2VHE), a standardized human evaluation protocol for T2V generation. T2VHE consists of well-designed evaluation metrics, annotator training encompassing detailed instructions and illustrative examples, and user-friendly interface, see Figure for illustrations. Additionly, introduces a dynamic evaluation module to reduce the annotation cost. We further introduce the details of our protocol as follows:

---

Evaluation metrics. Many previous research focuses on video quality and text alignment but neglects the critical aspects of temporal and motion quality inherent videos [103, 74], as well as ethical considerations [2]. T2VHE employs four objective evaluation metrics: video quality, temporal quality, motion quality, and text alignment, and two subjective metrics: ethical robustness and human preference. For different types of metrics, annotators were asked to rely more on the definitions of reference perspectives and their preferences to make judgments, respectively.
Evaluation method. Due to the complexities and potential noise inherent in absolute scoring [67], T2VHE employs the comparison-bas based method, which is relatively annotator-friendle [8]. Critiquing the reliance of traditional protocols on win rates, which may introduce biases and offer limited insights into model performance [33, 221, T2VHE adopts the Rao and Kupper model [72] to quantify annotations results. This approach enables more efficient management of pairwise comparison results, yielding improved model rankings and score estimations.
Evaluators. While crowdsourcing platforms are often considered to gather high-quality annota- tions [8. 15]. our survey reveals that researchers primarily rely on LRAs. However, concerns about LRAs' reliability due to inadequate training and quality checks have been raised, potentially biasing evaluation outcomes. To address this, T2VHE proposes comprehensive annotator training, including detailed guidelines and examples to improve understanding of evaluation metrics. Experimental validation shows that properly trained LRAs can achieve agreement with crowdsourced annotators, affirming the effectiveness of our training approach in ensuring high-quality annotations.
Dynamic evaluation module. T2VHE incorporates dynamic evaluation component enhance protocol efficiency. This component optimizes utility through two key functionalities: firstly, pre- annotation sorting of videos using automatic scoring results, prioritizing the annotation of video pairs considered more deserving of manual evaluation during the static annotation phase; secondly, the dynamic annotation phase, which determines whether to annotate pending video pairs based on differences in model scores. Experimental results indicate that this module reduces costs by approximately 50%, while concurrently ensuring the validity of the annotation results.
Our study reveals several findings: (1) Post-tral Post-training LRAs and annotators on crowdsoureing platforms achieve consensus. The low quality of the pre-training LRAs' annotations mainly stems from annotators' biased interpretations of evaluation metric definitions. Thus, furnishing detailed guidelines and example training can substantially improve annotation quality, yielding results comparable to those obtained by professional annotators. (2) Comparison-based evaluation exhibits significant potential for optimization. It manifests an O(N2) growth trend in the number of annotations the number of models compared increases, Our dynamic evaluation module reveals that judiciously selecting samples for annotation can produce model ranking outcomes consistent with those derived from fully annotated data, even when annotating only approximately half of them. (3) A substantial disparity persists between open-soure source and dlosed-ouuree models. While well in some evaluation metrics, videos generated by closed-source models generally exhibit superior quality and tend to garner greater popularity among annotators.
models perform
The main contributions of this paper are as follows:
1. We introduce standardized human evaluation protocol for T2V models, comprising meticulously crafted array of evaluation metrics alongside accompanying annotator training resources. Moreover, it lets acquire high-quality annotations utilizingLRAAs
2. Our dynamic evaluation component reduces annotation costs approximately half of the original expenditure while maintaining annotation quality.
3. We comprehensively evaluate the latest T2V models and commit to open-sourcing the entire evaluation process and code, empowering the community to assess new models with fresh data based on existing reviews. Our protocol is also easily extended.
2 Related work
Text-to-video generative models. Creating realistic and novel videos has been a compelling research area for many years [87, 71, 111]. Various generative models have been investigated in prior studies, including GANs [87, 75, 83, 81, 77], autoregressive models [80, 105, 61, 23, 37], and implicit neural representations [79, 1 113]. T2V generation focuses on producing videos from textual descriptions.
3

---

Recently, the success of diffusion models in image synthesis has spurred studies to adapt these models for conditional and unconditional video synthesis [86, 31, 123, 101, 5, 45, 107].
Evaluation metrics of video generative models. Evaluation metrics for video generative models can be broadly classified into automated evaluation metrics and benchmark methods. IS assesses image diversity and clarity through a pre-trained network [76], CLIP Similarity assesses alignment between textual descriptions and generated images [70], FID and FVD compare feature representations from real and generated images [34, 84]. Despite their usefulness, these metrics have limitations, including bias, sensitivity to surface similarity, and inconsistency with human perception [67, 54]. Additionally, VBench [40], EvalCrafter [53]. and FETV [54]. among other benchmarks, provide comprehensive scenarios, and these automated scorers typically require alignment training based on human evaluation results. Given the limitations of these
evaluations but may lack the diversity to cover all
automated metrics and benchmarks, high-quality manual assessments remain critical.
Human evaluation in video generation
We survey 89 papers on video generation models since 2016', and after reviewing how they use and report human assessments, we have the following findings:
Automatic vs. Human evaluation. Among the 89 papers reviewed, 44 rely exclusively on automated evaluation metrics. However, these metrics typically have limitations, such as only capturing certain characteristics of the generated video [76], dependence on pre-trained models [70], and the necessity for reference videos [84]. Moreover, most automated metrics have demonstrated inconsistency with human evaluations, thus rendering them unsuitable as the sole measure [67, 54].
Many benchmark studies have attempted to train automated scorers based on human- assessed results [54, 100]. However, the training data is often obtained by pre-training LRAs, and the protocols employed exhibit considerable variation across studies. Hence, though automated evaluation should be a promising research direction, human evaluation is more reliable so far, and also good human evaluation can assist the development of automated evaluation.
Human evaluation metrics. The setup of human evaluation metrics varies greatly across papers. Ten studies directly use overall human preference as an evaluation metric, while some develop 16 nuanced metrics to assess from various perspectives. Overall, assessing the video's 's quality and relevance to the text descriptions remains the main focus of human evaluation [74, 28]. Moreover, increasing number of studies focus on evaluating the video generation model's 's capabilities concerning motion, consistency, and continuity [53, 40J. In addition, we found no study introduces ethical evaluation in human evaluation, which is an aspect that warrants attention [48].
Human evaluation methods. Of the studies conducting human evaluations, nearly 70% employ a comparison- based approach in their evaluation protocols, wherein annotators select the superior video among two or more options make judgments. In contrast, seventeen papers employed 3-point or 5-point Likert scales (i.e. absolute scoring), necessitating annotators to directly rate individual video's performance across various dimensions. However, this approach increases annotation complexity and introduces higher levels of noise and disagreement [67].
Quantification of annotation results. In evaluation protocols employing absolute scoring, the final model score is usually an average of all annotation results from all samples. On the other hand, pair- wise comparison-bart based evaluation protocols often employ the win ratio to quantify annotations [40]. Thus, while comparison- based evaluations are more user-friendly, they often require the evaluation of all model combinations to generate stable and reliable outcomes [22].
Annotators. Many articles omit crucial annotator information, such the number of annotators, their recruitment sources, and remuneration details. Additionally, for studies utilizing crowdsourcing platforms, only a handful of articles mention eligibility screener settings. This information vital for gauging the reliability and ethicality of assessment results [67].
Annotator training and quality checking. Only eight articles provided instruction-based or example- based training while utilizing LRAs. Moreover, merely two articles employed inter-annotator
'e summarize the video generation models published in major conferences and journals as well as the popular ones since 2016. The full list of reviewed papers can be found in Appendix F.

---

agreement (IAA) for annotation quality checking. Most protocols not only neglected to train LRAs before annotation but also omitted quality checks, raising concerns about the reliability of results.
Prompts. For different evaluation needs, researchers usually choose different prompts for generating videos, such HuMMan [6], which is specifically designed for evaluating fine-grained spatiotem- poral motion generation. However, datasets sourced from real-world instances often lack diversity, motivating recent benchmarking studies to advocate for manually curated cue datasets encompassing comprehensive categories for model evaluation [40, 53, 54]. Due to different needs and different sources of prompts, the number of prompts used in human assessment often varies by more than a few dozen times from study to study, and many papers use fewer than 100 samples per model for human assessment, such small sample sizes are likely to produce biased results [67].
Annotation interface. Since videos generated by T2V models usually have different resolutions and sizes, how they are presented in the annotation interface also impacts the evaluation results. While some studies adopt methods such as adding watermarks, frame sampling, or cropping to standardize videos from different models [54], the majority only scale or leave generated videos unaltered during annotation. In addition, only nine articles provide the details of the annotation interface, with none sharing the interface code. This absence of information impedes the reproducibility of results for future studies adhering to similar protocols. Moreover, the scarcity of reusable resources hinders the ongoing enhancement of human evaluation protocols and practices [67].
Further, we provide more in-depth discussions of video processing, protocol settings, and samples for the human assessment process in Appendix C.
4 Our protocol for text-to-vedio models
Our T2VHE framework comprises four key components: evaluation metrics, evaluation method, evaluator, and dynamic evaluation module. To ensure a comprehensive assessment of the T2V model, we meticulously devise a set of evaluation metrics, accompanied by precise definitions and corresponding reference perspectives. For case of annotation, we employ a comparison- based scoring format evaluation method [7, 8] and develop annotator training ensure researchers can procure high-quality annotations using post-training LRAs. Furthermore, our protocol incorporates an optional dynamic evaluation component, enabling researchers to attain reliable evaluation results at reduced costs. More details can be found in Appendix D.
4.1 Evaluation metrics
Drawing from established protocols in image generation evaluation, prior studies have primarily used metrics like "Video Quality" and "Overall Alignment" for human assessment. However, these metries often suffer from vague definitions, leading annotators to base ratings on general impressions and fidelity the textual content. This lack of specificity can introduce subjectivity, potentially undermin- ing the quality of annotations. Recent research also underscores that motion and temporal quality are also vital metrics for assessing video generation models' capabilities [53, 40]. Additionally, as video generation technology gains popularity, its ethical and societal impacts are becoming increasingly critical factors in evaluation [48]. However, our survey reveals that none of the previous protocols con- sidered this indicator. Moreover, only ten studies provided specific training for annotators, suggesting that the majority of research has only offered basic definitions of each metric without comprehensive instructions or relevant examples, which are essential for ensuring high-quality annotations [15).
To this end, we establish comprehensive evaluation framework with explicit definitions and cor- responding reference perspectives for each metric. Additionally, to enable precise assessments, we also devise thorough annotator training, detailed in Section 4.3, Objective indicators require strict adherence to the reference perspectives to ensure consistency and repeatability in evaluations, while subjective indicators allow for personal interpretation, providing holistic assessment of the model's performance and potential. Recognizing the subjective nature of certain indicators, we categorize them into objective and subjective types. Objective indicators require strict adherence to the reference perspectives to ensure consistency and repeatability in evaluations, while subjective indicators allow for personal interpretation, providing a holistic assessment of the model's performance and potential. Detailed definitions and reference perspectives for each metric can be found in Table 1.
5

---

Table Comprehensive evaluation criteria for T2V models. The table presents T2VHE's evaluation metrics, their definitions, corresponding reference perspectives, and types. When considering different indicators, annotators rely differently on reference angles in making their judgments.
Metric
Definition
perspectives
Reference
Description
Type
Assess whether
uppears
Video Fidelity
highly realistic, making hard to
Which video more realistic and aesterically pleasing?
distinguish from actual footage. the artistic heauty aesthetic value of each video frame.
Video Quality
Objective
Aesthetic Appeal
including color coordination. composition, and lighting effects.
Evaluare whether subject's and hackground's is appearances remain unchanged throughout the vidco. Assess the consistency local and high-frequeacy details over time in the video.
Content Consistency
Which video has better consistency and less flickering over time!
Temporal Quality
Objective
Temporal Flickering
Evaluate the natural Quidity und adherence physical laws movemmnnts within the video. Assess whether the dynamic activities the and appropriate.
Movement Fluidity
Which video contains mations are more natural, smooth, and consistent with physical laws?
Motion Quality
Objective
Motion Intensity
sufficient
Assesa whether the video accurately reflects the types and quantities of objects deseribed the text. Evaluate whether the visual style of the video matches the text description.
Object Category
Text Alignment
Which video a higher degree of alignment with the prompt?
Objective
Style Consistency
Exaluate the video for any content that might be deemed toxic inappropriate.
Toxicity
Determine the faimesss in the portrayal and treatment of characters or subjects across different social dimensions.
Which video demonstrates higher ethical standards and fairness!
Ethical Robustness
Fairess
Subjectivity
Bias
Assess the presence and handling of biased content within the video.
Video Originality
Evaluate the originality of the vidco's contents.
Human Preference
As an annotator, which viden you prefer?
Overall Impact
intellectual value provided viden.
Assess the emotional and
Subjectivity
Assess the video based on the previous five metrics personal preferences.
Personal Preference
4.2
Evaluation method
There exist two primary scoring methods: comparative and absolute. The former requires annotators to compare a set of videos and select the one demonstrating superior performance, whereas the latter entails directly assigning scores the videos. Absolute scoring typically necessitates detailed instructions and precise question formulations due to its complexity [8]. However, even with these in place, absolute scoring could still result in noisy annotations and pose challenges in reaching consensus among annotators [67]. Hence, we use the less challenging comparative scoring method.
Quantification of annotations. Traditional comparative scoring protocols rely on the win ratio in pair- wise comparison, however, this method has several drawbacks. First, it can introduce bias if models are not uniformly compared [33]. For instance, model frequently pitted against stronger counter- parts might exhibit a lower win ratio compared to one facing weaker opponents more frequently. Consequently, significant number of comparisons are required to establish reliable rankings [22]. Moreover, the win ratio alone does not reliably indicate the likelihood of one model outperforming another [72]. To overcome these issues, we adopt the Rao and Kupper model [72], probabilistic approach that allows for more efficient handling of the results of pairwise comparisons using less data
than full comparisons. This model enables better estimation model rankings and scores, thereby
furnishing more precise and dependable evaluation compared to simply using the win ratio. The

---

Table Comparison of annotation consensus under different annotator qualifications. We compute Krippendorff's 'saiif'''' a [47] as an IAA measure. Higher values represent more consensus among annotators.
Metric
AMT & Pre-training LRAs
AMT & Post-training LRAs
AMT
Video Quality
0.185
0,411
0,451
Temporal Quality
0.131
0,340
0,369
Motion Quality
0,088
0.338
0.249
Text Alignment
0.069 Ethical Robustness
0.327
0.366
-0.057
0.100
0.177
Human Preference
0.167
0.281
0.297
estimation is conducted by maximizing the log-likelihood function:
Up, 0) = Σ
Taj log
pi + nij log
Pip; -1 Pj + They log (pr + (pp) + p35) (I)
1)
+ 0pj
0p: + Pj
5-++1
where the number of models, p = (pI.... pp) E R' is the vector representing the scores of each model, 0 is tolerance parameter, raj denote the number of times model is preferred to model j. and miy denotes the number of times the two models reached a tic. Further details about model's implementation and its parameter estimation process are provided in Appendix D.3.
4.3 Evaluators
For most video generation evaluation tasks, the evaluator does not need specific expertise. However, using annotators from crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), still results in higher annotation quality [67, 64, 66], as these workers have usually completed many tasks and carefully followed the publisher's requirements due to cost constraints, more studies tend to use non-professional, unpaid LRAs for the annotation tasks. Furthermore, our survey showed that most studies using LRAs lack annotator training and quality checking, raising concerns about the reliability of their annotations. To address this issue, we developed comprehensive training methodology for annotators and conducted experiments on pilot dataset to explore the impact of annotator qualifications on annotation quality.
ensure successful payment. Nevertheless,
Annotator training. We propose cost-effective training methods: instruction- and example- based training. Specifically, we furnish detailed guidance for each metric, complemented by two to three reference perspectives, each perspective paired with an example and an analytical process to aid annotators in understanding metric definitions and making accurate judgments. Detailed training interfaces are illustrated in Figure and Figures 4 8.
Comparison of annotator qualifications. We assess three annotator qualifications: AMT evalua- tors (who are required to hold an AMT Master designation), pre- pre-training LRAs, and post-training LRAs. Each AMT evaluator is compensated $0.05 per task and underwent both instruction based and example-based training maintain high standards of annotation quality [67]. In the case of pre-training LRAss, workers annotate tasks directly based on the problem definition for each indicator. Conversely, post-training LRAs familiarize themselves with guidelines and examples before annotat- ing. Five annotators are tasked with selecting the superior video from each pair in each qualification category, outlined in Section 1. Detailed annotation interfaces and the pilot dataset setup are presented in Figure and Section 5, respectively.
After collecting all annotations, notators compared to those from pre- pre-training LRAs across various metrics. To further evaluate these differences, calculate the internal I of AMT annotators and the external ones between them and the pre-and post-training annotators. For the latter two sets of experiments, we randomly select five annotators from the AMT group and the two LRAs groups, respectively, to calculate the corresponding IAA and average the results. As shown Table 2, pre-training annotators demonstrate lower agreement with professional annotators across multiple dimensions, evidenced by significantly
observe disparities in model rankings derived from AMT an-
A positive 1AA value indicates that the ratings are more consistent than random annotations. For example. the coherence rating of NLG in [43] achieves an IAA of 0.14
7

---

lower IAA than the other two groups. In contrast, post-training annotators exhibit improved agree- ment with the professional annotators, approaching levels of intra-AMT consensus. Thus, the model rankings obtained from the two sets of annotation results are identical. These findings suggest that effective annotator training within our protocol can yield high-quality annotation results using L.AAs comparable to those obtained using professional annotators.
4.4 Dynamic evaluation module
As the number of models increases, traditional evaluation protocols often become more costly, To minimize annotation costs and ensure stable model ranking with fewer comparisons, we develop dynamic evaluation module based on two key principles: the video quality proximity rule and the model strength rule. The first principle ensures that initially evaluated video pairs are of comparable quality, reducing unnecessary annotations, the second principle selects video pairs based on model strength, enhancing evaluation efficiency. The specific process is as follows:
Before annotation starts, each model receives an unbiased strength value. These scores are normalized and summed to generate feature score for each video. Groups of model pairs are then constructed for each prompt, with the difference between video scores input into an exponential decay model to determine pair scores and group total scores. These groups are then sorted based on their total scores prioritize those with close quality. In the follow-up phases, for each assessment indicator, all model scores are updated using the Rao and Kupper model after evaluating video pairs in the initial groups. Subsequent annotations occur in batches, with model strengths adjusted periodically. When evaluation results stabilize across all dimensions, i.e., the model rankings are unchanged for several consecutive batches, the evaluation terminated. We provide the implementation details of the module in Appendix D.2 and verify its effectiveness in Section 5.3.
5 Human evaluation of existing models
We evaluated five state-of-the-art T2V models, including Gen2 [20]. Pika [18], TF-T2V [94], Latte [56], and Videocrafter [11], see Appendix D.I for details. All videos were generated without any prompt engineering or filtering. Furthermore, to ensure uniformity and ease of comparison for evaluators, we standardized the height of all videos in the annotation interface.
5.1 Settings
Data preparation. We use the Prompt Suite per Category [40] as the source of prompts, which comprises prompts manually curated from eight distinct categories. We randomly select quarter of the prompts from each category to serve as our evaluation prompts. For each prompt, we construct 10 pairwise comparisons using videos generated by five models and ask annotators evaluate the superiority of one video over another across various metrics. This process results in 2,000 video pairs for annotation, from which we randomly sampled 200 video pairs to form the pilot dataset.
Annotators. To analyze the differences in results among different annotators and to affirm the protocol's generalizability and validity, following settings detailed in Section 4.3, we engage three distinct categories of annotators: AMT annotators, pre-training LRAs, and post-training LRAs. Each AMT annotator is limited to 250 tasks maintain quality, while both pre-training and post-training LRAs are tasked with annotating all video pairs. We also test the effectiveness of our dynamic evaluation component using post-training LRAs under the same conditions.
5.2 Evaluation results
For annotators who don't use the dynamic evaluation module, we collect the annotated data for all video pairs under the six metrics, resulting in total of x 5 x 2000 x 6 annotations (three categories of annotators, five in each category). Conversely, for annotators using the dynamic component, the average number of video pairs be annotated is only 1,068 per annotator. For AMT annotators, 76 participants contribute, with an average of 131 tasks per annotator.
Figure 2 summarizes the results of the quantified annotations, more detailed scores and rankings can be found Appendix D.4. As discussed in Section 4.3, the annotation results obtained by the pre-training LRAs markedly differ from those of the other three groups, evident in the discrepancy
8

---

Pre-training Vise Cuality
Post training LIAA
AMT Annotators
Post trainig LOAx
Quulity
Tripo Harron
Temporr Prufesnce
Temporal Humon
Ereal
Modion
Masse
Mation
theal
Misse
Set Algrewe
Tedt Alg sreet
Tout Migeneee
Tex Mlgeee
Gen2
Videscraltrr2
Figure 2: Scores and rankings of models across various dimensions for pre-training LRAs, AMT Annotators, and Post-training LRAs. Post-training LRAs (Dyn) refers the annotation results of Post-training LRAs using the dynamic evaluation component.
Table 3: Comparison of internal IAA of LRAs before and after training. The internal IAA of post- training LRAs rose all dimensions, implying a significant improvement in the annotation quality. We compute Krippendorff's a [47] as a measure of internal IAA. Higher values represent more con- sensus among annotators.
Table 4: Type and number of model pairs discarded in dynamic evaluation.
Model
Model
Count
Gen2
Latte
230
Videocrafter2
219
TF-T2V
215
Metric
Pre-training
Post-training
Pika
184
Pika
Videocrafter2
76
Video Quality
0,224
0.339
TF-T2V
65
Temporal Quality
0.178
0.288
Latte
76
Motion Quality
0.164
0.321
Latte
TF-T2V
19
Text Alignment
0.145
0,236
Videocrafter2
12
Ethical Robustness
0.055
0.107
TF-T2V
Videocrafter2
20
Human Preference
0.195
0.284
between the final model scores and rankings for each dimension. In addition, the annotation results of the trained LRAs closely mirror those of the AMT personnel, yielding consistent final model ranking outcomes. We also conduct a quality check of the annotation results for LRAs before and after training. As shown in Table 3, the annotations from the post-training LRAs exhibit higher quality.
However, regardless of the annotator sources, closed-source models typically perform better. In the annotated results from AMT personnel, Gen2 demonstrates significant superiority over other models across all metrics, while Pika also exhibits commendable performance across most metrics.
In contrast, the performances of open-source models show less disparity in terms of video quality, temporal quality, and motion quality metries. TF-T2''s generations typically excel in video quality and action timing, while Videocrafter2, an earlier open-source model, demonstrates notable profi- ciency in generating high-quality videos. However, distinctions among the three models become more apparent in the metrics of text alignment, ethical robustness, and human preference. Notably, Latte exhibits strong performance in text aligmment and ethical robustness, even surpassing Pika. This contributes to its higher ranking in human preference compared to other open- models despite marginal differences in other metrics.
5.3 Module validation
As detailed in Section 5.2, our protocol, augmented by the dynamic evaluation module, cuts annotation costs to about 53% the original expense while achieving comparable outcomes. We further explore the module's effectiveness and reliability in this section.
Effectiveness. Although protocols utilizing pairwise comparisons offer convenience to annotators, their evaluation costs can escalate rapidly as the number of models under examination increases. To assess the effectiveness of the dynamic evaluation component, we randomly select corresponding annotation results from 2-4 models (25 model combinations in total) out of all annotations. For each

---

bratersl ureganoee protice!
12.0
Dut potoca wit dyeams eviket an FO0080
3.0
3530
Number of annnttatoon
2.5
2530
Estimation
MODE!
1500
1030
1.0
Vadeo Quulity
kerporal
Nution Deanty Dimension
digrement
IShecal Rebesions Humen Prteerncee
Model count
Figure 3: The left figure shows how the number of annotations required for different protocols. The right figure represents model score estimations across different metries. Each boxplot illustrates the median, interquartile range, and 95% confidence intervals of the estimates.
combination, we simulate the dynamic annotation process, calculating average annotation demands and employing exponential model to forecast annotation needs with escalating model numbers. As depicted in Figure 3, our protocol with dynamic component demonstrates a nearly linear growth in annotation demands the number of models increases, greatly reducing the evaluation costs.
Reliability, We advocate for the reliability of the module both the design level of the algorithm and the result level. Before the start of the dynamic evaluation, annotators are required to annotate 200 video pairs where distinguishing differences between them based on automated metries challenging. This step ensures that the samples most deserving of human assessment will not be discarded in the dynamic assessment and that the initially estimated model scores are not biased by specific prompt types, as elaborated in detail in the Appendix C.4.
To enhance the demonstration of this module's reliability, we perform bootstrap confidence intervals for the score estimates of each model across various metrics. As shown in Figure 3, the confidence intervals for Latte, Pika, TF-T2V, and Videocrafter2 are consistently narrow, signifying precise estimations. In contrast, the confidence intervals for Gen2's scores are relatively wide, indicating less stable estimations. This variance primarily stems from our dynamic algorithm's 's frequent exclusion of comparisons involving Gen2 due to its significant superiority over the other models. Table 4 details the number of these omissions. Nevertheless, even at the lower bound of the confidence intervals, Gen2's ' score estimation remains superior to those of all other models. This highlights that the rank estimations remain robust despite some instability in score estimation. Thus, our dynamic evaluation provides reliable and consistent rank estimations while requiring fewer annotations.
6 Limitations
Our study conducts welllestablished human evaluation experiments on five state-of-the-art video generation models, yet some limitations persist. First, because the T2V models used are relatively new and contain two closed-source models, we do not offer technical improvement suggestions. Secondly, there is potential for refining our dynamic evaluation algorithm. As the number of models evaluated increases, improvements can be made to the initial settings of model strength.
Conclusion
To address the issues of reproducibility, reliability, and usability in previous human evaluation protocols for T2V generation, this paper introduces the T2VE protocol. By employing well-defined metrics, thorough annotator training, and dynamic evaluation module, T2VE enables researchers to obtain high-quality annotations by LRAs at low costs. We anticipate that future research could build upon this protocol to further expand the study of human evaluations of generative models.
10

---

Acknowledgement
This paper is partially supported by the National Key R&D Program of China (No.2022Z00610000 No. No.2022ZD010010,, No. No.2022ZD0160102) and the General Research Fund of Hong Kong No.17200622 and 17209324.
References
[1] Yogesh Balaji, Martin Renqiang Min, Bing Bai. Rama Chellappa. and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 1995-2000 International Joint Conferences on Artificial Intelligence Organization, 2019.
[2] Hritik Bansal, Da Yin. Masoud Monajatipoor, and Kai-Wei Chang. How well can text- to-image generative models understand ethical natural language interventions? In Yoav Goldberg, Zoriissa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1 3358-1370, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
[3] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets, 2023.
[4] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Bjöm Ommer. Understanding object dynamics for interactive image-to-video synthesis, 2021.
[5] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models, 2023.
[6] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang. and Ziwei Liu. Humman: Multi-modal 4d human dataset for versatile sensing and modeling, 2023.
[7] Chris Cellison-Burchh Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. (meta-) evaluation of machine translation. In Chris Callison-Burchh Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz, editors, Proceedings of the Second Workshop on Statistical Machine Translation, pages 136-158, Prague, Czech Republic, June 2007. Association for Computational Linguistics.
[8] Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey, 2021.
[9] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexci A. Efros. Everybody dance now. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), October 2019.
[10] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing. Yaofang Liu, Qifeng Chen, Xintao Wang. Chao Weng, and Ying Shan. Videocrafterl: Open diffusion models for high- qualityality video generation, 2023.
[1I] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models, 2024.
[12] Tsai-Shien Chen, Chich Hubert Lin, Hung- Yu Tseng, Tsung- Yi Lin, and Ming-Hsuan Hsuan Yang. Motion-conditioned diffusion model for controllable video synthesis, 2023.
[13] Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023.
11

---

[14] Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang. Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction, 2023.
[15] Elizabeth Clark, Tal August, Sofia Serrano, Nikita Haduong, Suchin Gururangan, and Noah A. Smith, All that's 'human' is not gold: Evaluating human evaluation of generated text, 2021.
[16] Kangle Deng, Tianyi Fei, Xin Huang, and Yuxin Peng. Ire-gan: Introspective recurrent convo- lutional gan for text-to-video generation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages Conferences on Artificial Intelligence Organization, 2019.
International Joint
[17] Michael Dorkenwald, Timo Milbich, Andreas Blattmann, Robin Rombach, Konstantinos G. Derpanis, and Bjöm Ommer. Stochastic image-to-video synthesis using cinns, 2021.
[18] Saleh Elmohamed. Pika lab discord server. Website, 2024. https://wwa pika. art/.
[19] Sefik Emre Eskimez, You Zhang, and Zhiyao Duan. Speech driven talking face generation from single image and an emotion condition, 2021.
[20] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models, 2023.
[21] Fanda Fan, Chunjie Luo, Wanling Gao, and Jianfeng Zhan. Aigebench: Comprehensive evaluation of image-to-video content generated by ai, 2024.
[22] Jianqing Fan, Zhipeng Lou, Weichen Wang, and Mengxin Yu. Ranking inferences based on the top choice of multiway comparisons. ArXiv, 0ts/2211.11957,, 2022.
[23] songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vqgan and time-sensitive trans- former, 2022.
[24] songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models, 2024.
[25] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samanch Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing tcxi-to-video generation by explicit image conditioning, 2023.
[26] Xianfan Gu, Chuan Wen, Weinui Ye, Jiaming Song, and Yang Gao. Seer: Language instructed video prediction with latent diffusion models, 2024.
[27] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning, 2024.
[28] Thilo Hagendorff. Mapping the ethics of generative ai: A comprehensive scoping review, 2024.
[29] Ligong Han, Jian Ren, Hsin- Bing Lee, Francesco Barbieri, Kyle Olszewski. Shervin Minaee, Dimitris Metaxas, and Sergey Tulyakov. Show me what and tell me how: Video synthesis via multimodal conditioning, 2022.
[30] Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajec- tories. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
[31] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long videos, 2022.
[32] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation, 2023.
12

---

[33] Reinhard Heckel, Nihar B. Shah, K. Ramchandran, and M. Wainwright. Active ranking from pairwise comparisons and when parametric assumptions do not help. The Annals of Statistics, 2016.
[34]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Neural Information Processing Systems, 2017.
[35] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models, 2022.
[36] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
[37] Wenyi Hong. Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers, 2022.
[38] Yaosi Hu, Chong Luo, and Zhenzhong Chen. A benchmark for controllable text -image-to- video generation. IEEE Transactions on Multimedia, 26: 66:1700 700-11799, 2024.
[39] Hanzhuo Huang. Yufan Feng. Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator, 2023.
[40] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang. Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Vhench: Comprehensive benchmark suite for video generative models, 2023.
[41] Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peckaboo: Interactive video generation via masked-diffusion, 2024.
[42] Yuming Jiang, Shuai Yang, Tong Liang Koh, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2performer: Text-driven human video generation, 2023.
[43] Marzena Karpinska, Nader Akoury, and Mohit lyyer. The perils of using Mechanical Turk to evaluate open-ended text generation. In Marie-Francin Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 1265-1285, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[44] Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemnllnaher--Shizzema.. Dreampose: Fashion image-to-vides to-ieeo synthesis via stable diffusion, 2023.
[45] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators, 2023.
[46] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, Krishna Somandepalli, Hassan Akbari, Yair Alon, Yong Cheng. Josh Dillon, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, Mikhail Sirotenko, Kihyuk Sohn, Xuan Yang, Hartwig Adam, Ming-Hsuan Yang, Irfan Essa, Huisheng Wang, David A. Ross, Bryan Seybold, and Lu Jiang. Videopoet: A large language model for zero-shot video generation, 2024.
[47] Klaus Krippendorff Content analysis: An introduction to its methodology. 1980.
[48] Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, Minguk Kang, Taesung Park, Jure Leskovee, Jun- Yan Zhu, Li Fei-Fei, Jiajun Wu, Stefano Ermon, and Percy Liang. Holistic evaluation of text-to-image models, 2023.
13

---

[49] Dingquan Li, Tingting Jiang, and Ming Jiang. Quality assessment of in-the-wild videos. In Proceedings of the 27th ACM International Conference on Multimedia, MM '19. ACM, October 2019.
[50] Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng. Errui Ding, and Jingdong Wang. Videogen: A reference- -giiddd latent diffusion approach for high definition text-to-video generation, 2023.
[51] Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin. Video generation from text. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018.
[52] Jiawei Liu, Weining Wang, Wei Liu, Qian He, and Jing Liu. Ed-12v: An efficient training framework for diffusion- text-to-vided generation. In 2023 International Joint Conference on Neural Networks (JCCN), pages 1-8, 2023.
[53] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan. Evalcrafter: Benchmarking and evaluating large video generation models, 2024.
[54] Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou. Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video genera- tion, 2023.
[55] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high- quality video generation, 2023.
[56] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024.
[57] Tanya Marwah, Gaurav Mittal, and Vineeth Balasubramanian. Attentive semantic video generation using captions, 2017.
[58] Kangfu Mei and Vishal M. Patel. Vidm: Video implicit diffusion models, 2022.
[59] Willi Menapace, Stéphane Lathuilière, Sergey Tulyakov, Aliaksandr Siarohin, and Elisa Ricci. Playable video generation, 2021.
[60] Gaurav Mittal, Tanya Marwah, and Vineeth N. Balasubramanian. Sync-draw: Automatic video generation using deep recurrent attentive architectures. In Proceedings of the 25th ACM international conference on Multimedia, MM 11.. ACM, October 2017.
[61] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis, 2021.
[62] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors, 2023.
[63] Haomiao Ni, Changhao Shi, Kai Li, Sharon X. Huang, and Martin Rengiang Min. Conditional image-to-video generation with latent flow diffusion models, 2023.
[64] Austin Lee Nichols and Jon K. Maner. The good-subject effect: Investigating participant demand characteristics. The Journal of General Psychology, 135:151 166, 2008
[65] OpenAL Sora. Website, 2024.
[66] Martin T. Orne. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist, 17:776-783, 1962.
[67] Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkilä, and Shin'ichi Satoh. Toward verifiable and reproducible human evaluation for text-to-image generation, 2023.
14

---

[68] Junting Pan, Chengyu Wang, Xu Jia, Jing Shao. Lu Sheng, Junjie Yan, and Xiaogang Wang. Video generation from single semantic label map. 2019.
[69] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei. To create what you tell: Generating videos from captions, 2018.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[71] MarcAurelio Ranzato, Arthur Szlam, Joan Bruna, Michael Mathieu, Ronan Collobert, and Sumit Chopra. Video (language) modeling: a baseline for generative models of natural videos, 2016.
[72] PV Rao and Lawrence I. Kupper. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62(317):194-200 1967.
[73] BYRD RH. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific and Statistical Computing, 16(5):1190 1ee111990-22,,, 1995.
[74] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning audio and video generation, 2023.
diffusion models for joint
[75] Masaki Saito, Eiichi Matsumoto, and Shunta Saito. Temporal generative adversarial nets with singular value clipping. 2017.
[76] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, 2016.
[77] Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Mostgan-v: Video generation with temporal motion styles, 2023.
[78] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An. Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman, Make- a-video: Text-to-video generation without text-video data, 2022.
[79] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2022.
[80] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms, 2016.
[81] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris Metaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis, 2021.
[82] Zhengzhong Tu, Yilin Wang. Neil Birkbeck, Balu Adsumilli, and Alan C. Bovik. Uge-vqa: Benchmarking blind video quality assessment for user generated content. IEEE Transactions on Image Processing, 00:4449-446/1 2021.
[83] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation, 2017.
[84] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michal- ski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges, 2019.
[85] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual description, 2022.
[86] Vikram Voleti, Alexia Molicoeur-Martinees and Christopher Pal. Mevd: Masked conditional video diffusion for prediction, generation, and interpolation, 2022.
15

---

[87] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics, 2016
[88] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to- rext-to-video technical report, 2023.
[89] Ting-Chun Wang, Ming-Yo Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot rideo-to-video synthesis, 2019.
[90] Ting-Chin Wang, Ming- Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. Video-to-video synthesis, 2018.
[91] Wei Wang, Xavier Alameda-Pinneda, Dan Xu, Elisa Ricci, and Nicu Sebe. Learning how smile: Expression video generation with conditional adversarial recurrent nets. IEEE Transactions on Multimedia, (2(1))2808 1):2082-289, 2020.
[92] Wenjing Wang, Huan Yang. Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu. Swap attention in spatiotemporal diffusions for text-to-video generation, 2024.
[93] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. 2023.
[94] Xiang Wang. Shiwei Zhang, Hangjie Yuan, Zhiwu Qing, Biao Gong, Yingya Zhang, Yujun Shen, Changxin Gao, and Nong Sang. A recipe for scaling up text-to-video generation with text-free videos. arXiv preprint arXiv:2312. 15770, 2023.
[95] Yaohui Wang, Piotr Bilinski,, Francois Bremond, and Antitza Dantcheva. G3an: Disentangling appearance and motion for video generation, 2020.
[96] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang. Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang. Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. Lavie: High-quality video generation with cascaded latent diffusion models, 2023.
[97] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions, 2021.
[98] Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang. Zicheng Liu, Yuejian Fang, and Nan Duan. Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis, 2022.
[99] Chenfei Wa, Jian Liang, Lei Ji, Fan Yang. Yuejian Fang. Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation, 2021.
[100] Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia- Wei Liu, Yuchao Gu, Rui Zhao, Weisi Lin, Wynne Hsu, Ying Shan, and Mike Zheng Shou. Towards a better metric for text-to-video generation, 2024.
[101] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for texi-to-video generation, 2023.
[102] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors, 2023.
[103] Zhen Xing, Qi Dai, Han Hu, Zuxuan Wu, and Yu-Gang Jiang. Simda: Simple diffusion adapter for efficient video generation, 2023.
[104] Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks, 2018.
[105] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. Videogpt: Video generation using vaevae and transformers, 2021.
16

---

[106] Ceyuan Yang, Zhe Wang, Xinge Zhu, Chen Huang, Jianping Shi, and Dahua Lin. Pose guided human video generation, 2018.
[107] Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation, 2022.
[108] Zipeng Ye, Mengfei Xia, Ran Yi, Juyong Zhang, Yu-Kun Lai, Xuwei Huang. Guoxin Zhang, and Yong-Jin Liu. Audio-driven talking face video generation with dynamic convolution kernels. IEEE Transactions on Multimedia, 25:2033-2046, 2023.
[109] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Dragmuwa: Fine-grained control in video generation by integrating text, image, and trajectory, 2023.
[110] Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang, Xiaodong Wang, Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, Jianlong Fu, Gong Ming, Lijuan Wang. Zicheng Liu, Houqiang Li, and Nan Duan. Nuwa-xl: Diffusion over diffusion for extremely long video generation, 2023.
[111] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G. Hauptmann, Ming- Hsuan Yang. Yuan Hao, Irfan Essa, and Lu Jiang. Magvit: Masked generative video transformer, 2023.
[112] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space, 2023.
[113] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics- aware implicit generative adversarial networks, 2022.
[114] David Junhao Zhang. Jay Zhangjie Wu, Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation, 2023.
[115] Jianfeng Zhang, Hanshu Yan, Zhongcong Xu, Jiashi Feng, and Jun Hao Liew. Magicavatar: Multimodal avatar generation and animation, 2023.
[116) Jiangning Zhang, Chao Xu, Liang Liu, Mengmeng Wang, Xia Wu, Yong Liu, and Yunliang Jiang. Dtvnet: Dynamic time-lapse video generation via single still image. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pages 300-315. Springer, 2020.
[117] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang. Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models, 2023.
[118] Wenxuan Zhang, Xiaodong Cun, Xuan Wang, Yong Zhang, Xi Shen, Yu Guo, Ying Shan, and Fei Wang. Sadtalker: Learning realistic 3d motion coefficients for stylized audio-driven single image talking face animation, 2023.
[119] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang. Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation, 2023.
[120] Yiming Zhang, Zhening Xing, Yanhong Zeng, Youqing Fang, and Kai Chen. Pia: Your personalized image animator via plug- and-pla modules text-to-image models, 2024.
[121] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris Metaxas. Learning to forecast and refine residual motion for image-to-video generation, 2018
[122] Rui Zhao, Yuchao Gu. Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models, 2023.
[123] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Mag- icvideo: Efficient video generation with latent diffusion models, 2023.
17

---

A Author contributions
Tianle Zhang: Led the project; Designed and implemented the overall evaluation protocol (metries, reference perspectives, dynamic evaluation component, etc.); Conducted experiments and analysis; Contributed to writing; Participation in data annotation.
Langtian Ma: Conduct validity analysis; Contributed to writing; Participation in data annotation.
Yuchen Yan: Participation in data annotation.
Yuchen Zhang: Participation in data annotation.
Kai Wang: Provided advice on the core framework; Contributed to writing.
Yue Yang: Participation in data annotation.
Ziyao Guo: Participation in data annotation.
Wenqi Shao: Provided advice on human evaluation.
Yang You: Provided advice on the core framework.
Yu Qiao: Provided advice on the core framework.
Ping Luo: Provided advice on the core framework.
Kaipeng Zhang: Provided overall supervision and guidance throughout the project. Provided advice on the core framework. Provided advice human evaluation. Contributed to writing.
B Statement of neutrality
The authors of this paper affirm their commitment to maintaining fair and independent evaluation of T2V models. We recognize that the authors' affiliations cover range of academic and industrial institutions, including those that developed some of the models we evaluated. However, the authors' involvement is based solely on their expertise and efforts in running and evaluating models, and the authors treat all models equally throughout the evaluation process, regardless of their origin. This study intended to provide an objective understanding and assessment of all aspects of the models, and it is not our intention to endorse specific models.
C Further Discussions
C.1 Preprocessing of the generated video
Our protocol is designed to ensure fair video evaluations by using simple scaling method that maintains clarity across all videos, regardless of aspect ratio or length. An extensive literature review confirms that preprocessing methods can bias comparisons, processing is applicd, allowing annotators to assess videos in their entirety [40, 21, 100J. This approach ensures that the unique strengths of each model are preserved and fairly evaluated. For instance, using a uniform frame rate can disadvantage high frame rate models, while extracting frames from long videos may not accurately reflect their advantages. Similarly, cropping adding watermarks can hinder text alignment evaluations.
emphasize that additional
In sum, while some preprocessing may be fair and essential for specific comparisons, simply scaling videos more equitable approach for human evaluation.
C.2 Trade-offs between crowdsourcing platforms and LRAs
Although our evaluation protocol has diligently established precise definitions, reference angles, and comprehensive annotator training for each metric and has eased task complexity through a comparison- based approach, it does not imply that LRAS can entirely supplant annotators sourced from crowdsoureing platforms. Firstly, evaluating subjective metrics, personnel from crowdsourcing platforms, given their diverse backgrounds, are better positioned to gauge the model's universality. Even post-training LRAs inevitably harbor certain biases, particularly evident assessing the metric of Ethical Robustness, where definitive consensus between LRAs and AMT personnel
18

---

remains elusive. Secondly, while employing LRAs facilitates easier training and quality assurance of annotations, they typically require more time to complete annotations as they often engage in voluntary work rather than salaried employment. Lastly, when conducting reviews of new models, the use of L.RAs introduces inherent biases, annotators naturally favor novel models, potentially leading to questionable annotation outcomes [64, 66].
Therefore, despite our study showcasing the capacity of trained LRAs to yield high-quality annota- tions, researchers must judiciously select annotators tailored to their specific research objectives.
C.3 Protocol settings for various needs
Our protocol designed with broad applicability, enabling researchers with diverse objectives to enhance and adapt it required. Researchers can draw upon the training and analysis methodologies outlined our protocol to establish corresponding reference points, guidelines, and illustrative examples for their own metrics. Furthermore, the guidelines and training proposed in this study can be easily extended to other fields, and similarly, by modifying the automated metrics in the dynamic module, one can also achieve efficient human evaluation of other types of protocols.
Regarding dynamic components, researchers retain the flexibility to adjust hyperparameters according to their specific requirements. Our simulation experiments indicate that the number of annotations necessary can be significantly reduced, potentially even below 53%, particularly when aiming to attain stable model rankings on specific metrics.
C.4 Samples more suitable for automated assessment
In designing the dynamic evaluation module, we first counted the positions of different prompts in the video pairs sorted according to the scoring results of the automated metrics, as shown in Figure 9. By instructing annotators to assess the initial 200 video pairs, we ensure the model's ' efficacy estimation before initiating the dynamic evaluation draws from assessments covering all prompt categories.
This discovery also facilitates a thorough analysis of prompt categories necessitating human eval- uation, where automated metrics might encounter challenges in identifying significant differences. Notably, video pairs categorized under vehicles, scenery, and animals tend to be more discernible by automated scorers, indicated by their relatively minor presence among the initial video pairs. Conversely, domains such as food and architecture often demand human assessment for nuanced differentiation. These insights are equally informative for future endeavors to construct datasets specifically tailored for human assessment.
C,5 Impact prompt types
In our investigation of model performance across various prompt types, we input human annotation results for eight distinct prompt categories into the Rao and Kupper model to rank the model scores, as shown in Tables 8 15. The findings indicate that while there are minor fluctuations in evaluation outcomes based on the prompt sources, models exhibiting superior performance, such as Gen2, consistently maintain their high rankings irrespective of prompt variations.
However, is worth noting that for cases where the overall strengths of the models are close, such as Latte, TF-22V and Videocrafter2, different kinds of prompts may directly affect the results of the rankings among models under each dimension. Consequently, our analysis underscores the significance of employing class-wide prompts in human evaluation protocols, they may provide more stable assessment of model capabilities.
D Protocol details
Due to the problematic nature of controlling the quality of annotations, final results may vary even when using the same assessment protocol and assessors [43]. However, can be effectively mitigated by a detailed description of the annotations and harmonized assessment process [67]. We, therefore, report full implementation details of protocol use and provide example templates for human assessment in the supplementary material.
19

---

D.1 Details of the model to be evaluated
Gen2 [20] a closed-source multi-moda Al system that generates novel videos with text, images, or video clips. We use the default parameter settings from the demo for T2V generation.
Pika [18] is also a popular closed-soure model that enables users to convert simple text descriptions into dynamic videos. We use the default settings on the Discord platform for video generation.
TE-T2V [94] is diffusion-based T2V model, and we use publicly available model parameters and commands to super-resolve the generated video after generating low-frame video.
Latte [56] is an open-source video generation model based on Latent Diffusion Transformer, trained on FaceForensics, SkyTimelapse, Taichi-HD, and UCFI01.
Videocrafter2 [11] is a high-quality open-source video generation model trained using low-quality video and synthesized high-quality images, its newly announced 51 512x320 checkpoint.
D.2 Dynamic evaluation component details
Evaluating multiple video models via pairwise comparisons becomes increasingly resource-intensivy as the number of models expands. To efficiently obtain stable model rankings, we propose pluggable dynamic evaluation module founded on two principles: the video quality proximity rule and the model strength rule. It is important to note that the component of this module responsible automatic metries calculations is executed on an A A100 GPU. However, the remainder of the protocol's caleulations be conducted CPU. The details of the design of the algorithm are as follows:
Principles
Video Quality Proximity Rule
Leverage the scoring results of automated metrics to allow human annotators to priori- tize the annotation samples that are difficult to distinguish with automated metries.
Ensures that initially evaluated video pairs have similar quality levels.
Model strength Rule
Determines the evaluation priority of subsequent video pairs based model strength
scores.
Reducing the number of comparisons between models with significant differences in strength to improve algorithmic efficiency.
Evaluation Process
1. Initial Model Strength Assignment
Each model assigned an initial neutral strength value, indicating no prior bias.
2. Automatic Metrics Computation
For each video, the following metrics were evaluated by pre-trained scorer [40]: subject consistency, temporal flickering, motion smoothness, dynamic fegree, aesthetic quality, imaging quality, and overall consistency.
Scores normalized and summed produce the feature for each video.
3. Group Construction
For each prompt, groups of model pairs are constructed.
The absolute value of the difference between the scores of two videos in pair is input
into an exponential decay model.
The output value is the score of each video pair, and the sum of these scores forms the total score for the group.
4. Sorting and Grouping
Groups were ranked according their total scores, with higher scores the top indicating less variation within the group.
This preprocessing step does not increase the cost of our evaluation protocol.
20

---

5. Human Evaluation Phase
initial of video pairs evaluated by humans, and model strengths are updated using the Rao and Kupper model.
Comparisons are then split into batches. For each video pair, the absolute difference in scores between the two models is entered into exponential decay model, whose output value is the probability that this pair will be discarded. After each batch, the model strength estimates were updated under the six evaluation dimensions.
The evaluation ends when the model rankings stabilize, meaning the rankings of models under each dimension remain unchanged for several consecutive batches.
have also provided pode and corresponding definitions D.2 for the dynamic evaluation component to make this easier to understand.
Definitions
V: : Set of videos
M Set of models
P Set of prompts
(pr,) = {(v:: v1) 1g and in E V, shares the same prompt pr,, and ek # m²
(vv) + IR. feature score for each video V
R Human evaluation results for model pairs
g( R) Estimate / using Rao and Kupper models based on R
I: An [ d matrix, representing scores under all metrics, where d is the number of metries,
and Ii represents the score of ith model in jth metric
mai__scor((u) uk) Score for video pair ut, U/ V
crrup_scoreefe Total pair score for group of model pairs V(p.)
sorted corted_groups Groups sorted by group_ score
F(Frk, v1) Difference in scores between the two models in the video pair
For the setting of hyperparameters, we set the number of No to 200 in our experiments, and each batch contains 8 groups, i.c., 80 video pairs, and update the model strengths under all dimensions every 5 batches, and stop annotating and outputting the final results after five consecutive times of equal model rankings.
D.3
Rao and Kupper model
naive method to evaluate the rank and performance of different models from the paired comparison data is ranking them based the win rate of each model. However, this method suffers from several disadvantages: (1) in scenarios where not every model is compared against others, or some models compared more frequently than others, win rates can be biased. If model only compared against strong models, its win rate might be lower than model compared mostly against weaker models [33]. (2) It requires large number of comparisons to accurately determine the rankings; otherwise, the estimation would suffer from high variance [22]. (3) Win rates could not provide reliable estimations of the probability of one model beating another model [72].
The Rao and Kupper model [72] is a probabilistic model for the outcome of pairwise comparisons between items, which effectively overcome the naive method's limitations. We adopt this model to characterize our evaluation results and obtain the estimations of ranking and scores of the text-to-video models.
Consider the paired comparison with models. Given a pair of models and j. let > j denote beating j i denote losing to j, and j denote a tie between i and j. The probabilities of
21

---

Algorithm Model Evaluation Algorithm
1: Input: Set of videos
2: Pre-processing:
3: for each video v E V do
4: compute and normalized automatic metric scores for
5: S(u) < sum of normalized scores
6: end for
7: for each prompt pr; P do
8: for each video pair {uk, 1/ E ((r,,) do
9: pair_score(Vi: vt) S(IS(v)) a())),
10: end for
11:
group_score(pt.) <
Σ (V
pair puir_score(ra,
12: end for
13: sorted geruidgrrouppo +- sort (V(pr, ep by group_ score in descending order
14: Hum-evaluation:
15: Evaluate the first No groups in sorted sorted_groups by human and update R.
16: + g(R)
17: for each batch in the remaining video pairs do
18: for each video pair batch do
19: Discard the video pair with probability f f(IF([ek, a.).
20: if the pair is not discarded then
21:
Evaluate the video pair by human and update R.
22:
end if end for
24: I + g
if model ranking stable over 5 consecutive batches then
26: break
end if
28:
end for
29:
Output: Final model rankings and updated intensities
each event specified
PiP;(02
P(i j)
P(i ~
(2)
P: + Op,
(pr + tp, (8p: p,)
where pt and Pj are positive real-valued scores of model i and model j. respectively, which can be interpreted as the strength the models. 0 positive real-valued parameter with larger implying two models are more likely to be tied.
Let May denote the number of times model i is preferred to model j. ū.g denote the number of times model is tied with model j. The likelihood function can be written as:
(02 1)
L(p,0)
II
II iilii
Pi (P
(3)
+ Op:
Op: + Ps
(ps + (0p; po5)
and the log- likelikel function follows:
PfP;(02 11 (p. + 0p;) + p,))
1(p, 0) = Σ Σ
May log
Pi
+ nys log
Pj + Ruj log
(4)
P: + 0p,
0p: + Pj
1=1 5=
where E R is a vector defined by (p..... ..).....)"" p.) The maximum likelihood estimation of p and I can be obtained by maximizing (4) numerically. However, obtaining human- numan-evaluation data could be costly. To reduce the cost, we use a dynamic evaluation algorithm to give reliable estimations with fewer data.
Parameter Estimation Details. To obtain a stable estimation of the Rao and Kupper model parame- ters, restrict the parameter reasonable range and use the L-BFGS-B [73] algorithm to optimize
22

---

the log- log-likelihood function. We restrict all p: to be greater 0.01 to avoid numerical instability caused by extremely small values. We also restrict [e0.01, e This is because / can be interpreted as the exponential of the tolerance within which the annotators cannot distinguish two models with different "true" merits, Formally, according [72], the probabilities in the model can be written as:
P(i 5) - - / -(Vi-V))+
sech² (22 ydy (V/-U))+
P(i 5) = / -(V+-v3)- sech? ( Jdy.
)dy
where Vi = In Pt. which is interpreted the "true" merit of each model and In 0 is the tolerance. We restrict r in [0.01, 10] and therefore 0 E "e0,01ee00
Confidence intervals by bootstrap. We employ bootstrap algorithm to derive 95% confidence intervals for score estimations. Our dynamic evaluation algorithm operates independently on data from each individual, with 2000 annotations per person. Rather than resampling from pooled data, generate 2000 bootstrap samples from the annotations of each person separately and then aggregate these samples. We resample 1000 times, resulting in 1000 bootstrap estimates using algorithm. The confidence intervals are then calculated based on the 2.5% and 97.5% percentiles of these bootstrap estimates.
D.4 Evaluation results
We further show the model scores obtained using different annotators and their rankings in Table 5, and the results of the scores and rankings under different prompt types are shown in Tables 8 to 15.
D.5 Annotation interface and annotator training
We show the annotation interface Figure and the presentation corresponding to each evaluation metric. For annotator training, inspired by [15], use lightweight annotator training methods, i.e., instruction-bart training and example-based training, which do not increase the cost of evaluation but can effectively improve the quality of annotation.
Instruction-based based training means providing detailed instructions for each metric based on the problem description. We provide two to three reference angles for each metric to help annotators better understand the definition of each metric.
Example-based training refers to providing examples for the evaluation process of each metric based instruction-based training. We provide example for each reference perspective and an analysis process based on the examples to help the annotator better perform the annotation task.
In addition, we informed all annotators involved in the study of the possible risks they might face. The detailed training interface is shown in Figure and Figures 8. We will releasing the full evaluation protocol code shortly.
E
More related work
More details about evaluation metries of video generative models Evaluation metrics for video generative models be broadly classified into automated evaluation metrics and benchmark methods. For automated evaluation metrics, the IS assesses image diversity and clarity through a pre- trained network [76], while the FID compares feature representations from real and generated images, enhancing sensitivity to image quality [34]. The FVD for videos examines temporal consistency and quality [84]. CLIPSIM uses the CLIP model assess alignment between textual descriptions and generated images, offering a measure for text-to sot-tto-mmae synthesis [70]. However, these metrics have limitations: IS may prioritize diversity over quality and carry biases from pre-trained networks. FID, despite improvements, can be influenced by superficial similarities and comparison dataset quality. FVD might emphasize temporal over spatial quality, and CLIPSIM's reliance on textual data can introduce language and cultural biases [67, 54]. For the benchmark frameworks, VBench evaluates video generation across multiple dimensions using tailored prompts and human preference annotations [40]. EvalCrafter integrates 17 objective metrics refined by human opinions with diverse
23

---

Table 5: Scores and rankings of models across various dimensions for pre-training LRAs, AMT Annotators, and Post-training LRAs. Post-training LRAs (Dyn) refer to the annotation results of Post-training LRAs using the dynamic evaluation component. higher score represents better performance of the model on that dimension.
Model
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical Robustness
Human Preference
Pre-training LRAs
Gen2
3.33 (1)
2.63 (1)
2.03 (1)
1.57 (1)
1.36 (1)
2.87 (1)
Pika
1.11 (2)
1.71 (2)
1.37 (2)
1.03 (3)
1.08 (3)
1.21 (2)
Latte
0.67 (5)
0.79 (5)
0.84 (5)
1.03 (4)
1.00
0.77 (5)
TF-T2V
0.76 (3)
1.09 (3)
1.01 (4)
0.90 (5)
1.06 (4)
0.87 (4)
Videocrafter2
0.72 (4)
0.92 (4)
1.06 (3)
1.24 (2)
1.12 (2)
0.91 (3)
Post-training LRAs
Gen2
2.71 (1)
2.37 (1)
2.16 (1)
2.71 (1)
2.57 (1)
2.96 (1)
Pika
1.16 (2)
1.34 (2)
1.24 (2)
(3)
1.18 (3)
1.24 (2)
Latte
0.82 (5)
0.89 (4)
0.89 (4)
1.43
1,42 (2)
0.89 (3)
TF-T2V
0.91 (3)
1.00 (3)
0.95 (3)
0.82 (4)
0.86 (4)
0.85 (4)
Videocrafter2
0.82 (4)
0.83 (5)
0.89 (5)
0.68 (5)
0.73 (5)
0.76 (5)
AMT Annotators
Gen2
2.25 (1)
2.29 (1)
2.11 (1)
2.76 (1)
3114
2.73 (1)
Pika
1.09 (2)
1.21 (2)
1.23 (2)
1.00 (3)
0.82 (3)
1.04 (2)
Latte
0,80 (5)
0.88 (4)
0.89 (4)
1,40 (2)
1.29 (2)
0.87 (3)
TF-T2V
0.90 (3)
0.88 (3)
0.91 (3)
0.71 (4)
0.49 (4)
0.71 (4)
Videocrafter2
0.86 (4)
0.76 (5)
0.87 (5)
0.51 (5)
0.29 (5)
0.56 (5)
Post-training LRAs (Dyn)
Gen2
2.75 (1)
2.42 (1)
2.30 (1)
2.90 (1)
2.66 (1)
2.98 (1)
Pika
1,22 (2)
1,46 (2)
1.35 (2)
1,21 (3)
1.23 (3)
1.31 (2)
Latte
0.86 (5)
0.97 (4)
0.92 (4)
1.62 (2)
1.53 (2)
0.98 (3)
TF-T2V
0,92 (3)
1.01 (3)
1300
0.86 (4)
0.91 (4)
0.89 (4)
Videocrafter2
0.87 (4)
0.86 (5)
0.88 (5)
0.69 (5)
0.76 (5)
0.81 (5)
prompts [53]. FETV categorizes text prompts into spatial and temporal attributes, combining manual and automatic evaluations [54]. Despite advancements, these benchmarks still have limitations such that they might lack the diversity needed to cover all real-world scenarios, potentially leading to biases [40, 53, 54, 38, 100].
Given the limitations of these automated metrics and benchmarks, high-quality human evaluations remain crucial. Human evaluations can capture nuanced visual and contextual quality aspects that automated metrics might miss, which helps bridge the gap between algorithmic evaluations and real-world applicability.
Human evaluation The natural language generation (NLG) community has long recognized the need for human evaluations to complement automated metrics. Similarly, text-to-image generation models have benefited from human evaluation to ensure the generated images are technically correct, contextually appropriate, and visually appealing. For example, Lee et al. [48] introduce compre- hensive evaluation framework for text-to-image models called Holistic Evaluation of Text-to-Image Models (HEIM), which evaluates models across 12 aspects and incorporates both human-rated and automated metrics to capture the full spectrum of model performance. Despite these advancements in the evaluation of text and image generative models, there remains a notable gap in the literature concerning the human evaluation of T2V generative models, which highlights the significance of our human evaluation protocol for T2V Models.

---

Details of reviewed papers
We counted the following characteristics of the surveyed articles: whether human evaluation was performed (Humeval), whether quality checking was performed (Validity), whether crowdsourcing platforms were used (Crowds), the number of annotators (Annotators), whether training was provided to the annotators (Training), the scoring format used for the evaluation protocol (Format), the conferences/journall published (Venue), the year of publication (Year), as shown Table 6 and
25

---

Table 6: Full list of surveyed papers, where indicates not mentioned in the article.
Title
Humeval
Validity
Crowds
Annotators
Training
Format
Venue
Year
GVSD [87]
x
150
comparative
NeurIPS
2016
TGAN [75]
x
X
ICCV
2017
Sync-DRAW [60]
V
x
37
absolute
MM
2017
ASVGC [57]
x
24
absolute
ICCV
2017
TGANs-C [69]
8
30
absolute
MM
2017
MoCoGANN [83]
240
comparative
CVPR
2018
CVGST [30]
comparative
CVPR
2018
V2VSynthesis [90]
comparative
NeurIPS
2018
FRGAN [121]
x
comparative
BCCV
2018
MD-GAN [104]
x
comparative
CVPR
2018
PSGAN+SCGAN [106]
x
50
absolute
BCCV
2018
Gist [51]
x
x
AAAI
2018
FBF+TS+FG [9]
x
100
comparative
ICCV
2019
Few-shotV2V [89]
6
x
60
comparative
NeurIPS
2019
Seg2Vid [68]
6
10
comparative
CVPR
2019
IRC-GAN [16]
20
comparative
UCAI
2019
TEGAN
ICAI
2019
G3AN [95]
comparative
CVPR
2020
DTVNet [6]
comparative
BCCV
2020
CAR-Nets [91]
x
comparative
TMM
2020
UOD [4]
x
CVPR
2021
SIVS [17]
x
x
x
CVPR
2021
PVG [59]
comparative
CVPR
2021
SDTFG [19]
5
x
60
comparative
TMM
2021
GODIVA [97]
/
x
200
comparative
arXiv
2021
MMVID [29]
x
CVPR
2022
Imagen Video [35]
arXiv
2022
VDM [36]
arXiv
2022
Make-A-Vicce Video [78]
comparative
arXiv
2022
StyleGAN-V [79]
CVPR
2022
DIGAN [113]
x
ICLR
2022
FDMLY [31]
X
x
x
NeurIPS
2022
DCK [108]
x
x
x
TMM
2U22
MMVID [29]
x
x
CVPR
2022
Phenaki [85]
x
x
ICL.R
2022
NÜWA [99]
x
BCCV
2022
NUWA-Infinig [98]
CVPR
2022
FETV [54]
/
absolute
NeurIPS
2023
MAGE [38]
absolute
TMM
2023
VideoL.DM [S]
comparative
CVPR
2023
PYoCo [24]
x
ICCV
2023
LVDM [32]
x
arviv
2023
Videogen [50]
x
17
comparative
arXiv
2023
ModelScope [88]
x
x
x
arXiv
2023
Linc-A-Vidoo [101]
V
x
x
5
comparative
ICCV
2023
26

---

Table 7: Full list of surveyed papers, where indicates not mentioned in the article.
Tidle
Humeval
Validity
Crowds
Annotators
Training
Format
Venue
Year
LAVIB [96]
x
30
x
comparative
arXiv
2023
NUWA-XL [110]
arXiv
2023
Show-1 [114]
/
comparative
arXiv
2023
MotionDirector [122]
comparative
arXiv
2023
MagicVider [123]
x
comparative
arXiv
2023
VideoCrafterl [10)
x
x
x
absolute
arXiv
2023
SadTalker [1]8]
x
X
20
x
comparative
CVPR
2023
Genl [20]
V
5
X
comparative
ICCV
2023
Text2Performer
20
absolute
ICCV
2023
Text2Video-Zero [45]
ICCV
2023
VideoFusion [55]
x
x
x
x
CVPR
2023
DynamiCrafter [102]
x
x
49
comparative
arXiv
2023
MCDifi [12]
x
x
X
x
arXiv
2023
DragNUWA [109]
X
arXiv
2023
Control-A- Vidco [13]
absolute
arXiv
2023
DreamPose [44]
x
absolute
ICCV
2023
VideoComposer [93]
x
x
arXiv
2023
MagicAvatar [115]
x
x
x
x
arXiv
2023
Emu Video [25]
x
5
<
comparative
arXiv
2023
MAGVIT [111]
x
CVPR
2023
12VGen-XL [117]
arXiv
2023
SVD [3]
x
comparative
arXiv
2023
LFDM [63]
x
CVPR
2023
MAGE [38]
x
X
16
x
absolute
TMM
2023
Video [37]
x
90
X
absolute
ICLR
2023
Dreamix [62]
10
X
absolute
arXiv
2023
ED-T2V [52]
UCNN
2023
Free-Bloom [39]
absolute
NeurIPS
2023
MM-Diffusion [74]
x
x
absolute
CVPR
2023
PVDM [112)
x
x
x
CVPR
2023
VIDM [58]
x
AAAI
2023
EvalCrafter [53]
3
V
absolute
CVPR
2024
AIGCBench [21]
comparative
TBench
2024
T2VScore [100)
absolute
arXiv
2024
VBench [40]
x
X
comparative
CVPR
2024
Seer [26]
x
X
54
/
comparative
ICLR
2024
Video Factory [92]
x
x
x
x
arXiv
2024
VideoCrafterl [1]]
comparative
arXiv
2024
AnimateDiff [27]
comparative
ICLR
2024
SEINE [14]
comparative
ICLR
2024
VidtrolVideo [1]
x
x
x
comparative
ICLR
2024
PIA [120]
x
x
comparative
CVPR
2024
SimDA [103]
x
x
comparative
CVPR
2024
PEEKABOO [41]
CVPR
2024
VidcoPoct [46]
comparative
ICML
2024
27

---

Contine
Roseet
Peewe ditk the compgending indiuato button to vame the quidkrre ras before selecing!
Core Question:
better consistency flickering over time?
Hote Pease "Toal" whon both viloon pertiree idveticalyy acress referree anglns and contlcting eress reformend perpeetives, plecee prieetice them better object beckgrennd
example,
tlowwing shouidsat
Reference perspecties:
wheler suljett
vides
Andlee heo ta reed ly fanend oert time and lignt cart, unvorerrble changes trai fox bacignos Howevr right ough changes
dwnn't dffirent
202/220
E00
Sompovi Selering
temorotooy food High hopenng
Eaumpls pumpt slling counn
Analas in uttvees the coaewwee change amoothy between difirmet hemss wheroo the rights iito echero not bcable ticheing
Condlatte::
JumI Vidio
Ritset
Figure 4: Instruction and examples guide used the "Temporal Quality" evaluation.
28

---

contsponding vew Mosines ecting
Core Question: Which video contains motions that natural, smooth, and consistent with physical laws?
Non: sdlct "Suall osly wfen vidos perform idovicall scroes refience sugles and thes are corficting viowe iofonce pesppeliie,, placto prioritoe thare ordec.
Reference perspectives
Fn Mevement Faildy Enfiare the mataral fually end vehoeros shevial lown rewereero willen rkles
gel cuoning sandroad
Right
Mosiee fnterity Acoeo whiche dynum adivkoo ador sullckett and appopriate
cousis dancing
innotins no rot partle cance novevers.
Conclallorc Left
SED/SCX
12
Figure 5: Instruction and examples to guide used to the "Motion Quality" evaluation.
29

---

conespordily
Core Question: Which video has higher degree of alignment with the prompt?
For eaaeple, the oilno the left centaios connect and mamber angeess furt muins piorpparry the widne style the small chea be "Lrtt Kertor"
perspactives
Objent Categery
whacher rideo ccouraily refecto typo que riti edjech rocebeed
bumple pormpt paper per
Analyas The vrtiidk lects siemert of hears and com not copley euch diemvnt the form peper outs Atrough the rightidke aloo lado dementt hands, does edhor so9cesble
light
ESG SE:
020/502
Mlye Comt Hemey Iaakatt wterrher veseligge on viteo metch the desceptinn
buanglic paxept frieg grogeee wilh puty do
Ki Kralysi Allhogg the encosatonn
atrad domnerss proceies wetteed unresst the sideo more dearly dea festa we desorttion
Figure 6: Instruction and examples to guide used to the "Text Alignment" evaluation.
Core Question: Which video demonstrate higher ethical standards and fairness?
onn rabjective atterios
perpacthoee for reforens, conticting
prieritte
Reforence perspectives
Earngle 2: Aus thes ane content dows not rduud Bny for pormerasily lagal achitte??
12 farsess Desennie the fimess part spal and vesameent chrocees sulped's scout difeeet sodaa mensions
Inample 1/s them meliaale sper cadiaa and ieaomen chenateo faom vena badigraundi in seme of fase gescer and Exmplee now rouue eeluce tertan groupe that right sugges
prosece humn dling of
witt
Cargga dee pepetaate pender roveecypove any democrea group cepacity reprseetaioon adde?
Figure 7: Instruction and examples guide used the "Ethical Robustness" evaluation.
30

---

Core Question: annotator, which video do prefer?
Nere: Dae adjectie
this only ofter possible perpectives contlcctig penpectier, plese priertte then
perspectives:
in Miees thiginal tiikute unr viginaally of the valess oxterts
buanslc Whoter vele hes ove moothe buskers lovel to untoppesd lig saditom?
banrle Goes the video atimuiin lina letuall surieityy or proude thought provaiing contert!
Ps: Nessonal Fefirance Astass the adeo based on the provicus flve netic and perional parferscca.
buansle Den the ataasa stelaal Uhe eeee
veouggo mirer your expeiouss
Right Better Expai
Figure 8: Instruction and examples guide used to the "Human Preference" evaluation.
Categories
architecture
200
food
human
lifestyle
plant
150
scenery
vehicles
Counts
100
50
Overall amount
Figure 9: The number of prompts corresponding to each category for different locations at the sorted video pairs.
31

---

Table Prompt Category Animal
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical
Human
Model
Robustness
Preference
Gen2
2.02 (1)
2.15 (1)
1.89 (1)
2,40 (1)
2.48 (1)
3.38 (1)
Pika
1.18 (2)
1.41 (2)
1.52 (2)
1.54 (2)
1.04 (2)
1.35 (2)
Latte
0.90 (4)
0.98 (3)
0.92 (4)
1.29 (3)
0.92 (3)
1.01 (3)
TF-T2V
0.98 (3)
0.93 (4)
0.96 (3)
0.99 (4)
0.73 (4)
0.52 (4)
Videocrafter2
0.84 (5)
0.72 (5)
0.86 (5)
0.53 (5)
0.69 (5)
0.35 (5)
Table 9: Prompt Category Architecture
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical Robustness
Human Preference
Model
Gen2
2.12 (1)
1.86 (1)
1.95 (1)
2.49 (1)
3.45 (1)
2.54 (1)
Pika
1.00 (3)
1.46 (2)
1.13 (2)
0.92 (3)
0.59 (3)
1.05 (2)
Latte
0.81 (5)
1.07 (4)
0.93 (4)
1,28 (2)
1.30 (2)
0.95 (3)
TF-T2V
0.82 (4)
1.10 (3)
0.94 (3)
0.85 (4)
0.50 (4)
0.72 (4)
Videocrafter2
1.09 (2)
0.73 (5)
0.89 (5)
0.62 (5)
0.25 (5)
0.62 (5)
Table 10: Prompt Category Food
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical Robustness
Human Preference
Model
Gen2
2.34 (1)
2.39 (1)
2.00 (1)
2.87 (1)
3.45 (1)
3,04 (1)
Pika
0.99 (2)
1.03 (2)
1.16 (2)
0.94 (3)
0.59 (3)
0.82 (3)
Latte
0.89 (3)
0.92 (3)
0.83 (5)
1.57 (2)
1.30 (2)
1.42 (2)
TF-T2V
0.79 (5)
0.74 (4)
1.04 (3)
0.51 (4)
0.50 (4)
0.47 (4)
Videocrafter2
0.80 (4)
0.74 (5)
0.87 (4)
0.47 (5)
0.25 (5)
0.35 (5)
Table 11: Prompt Category Human
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical Robustness
Human Preference
Model
Gen2
2.44 (1)
2.64 (1)
2.14 (1)
2.96 (1)
3.66 (1)
2.98 (1)
Pika
1.38 (2)
1.29 (2)
1.52 (2)
0.99 (3)
0.78 (3)
1.23 (2)
Latte
0.59 (5)
0.62 (5)
0.75 (5)
1.38 (2)
0.96 (2)
0.44 (5)
TF-T2V
0.94 (4)
0.79 (3)
0.91 (3)
0.61 (4)
0.37 (4)
0.62 (3)
Videocrafter2
0.97 (3)
0.64 (4)
0.87 (4)
0.45 (5)
0.27 (5)
0.55 (4)
32

---

Table 12: Prompt Category Lifestyle
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical
Human
Model
Robustness
Preference
Gen2
2.52 (1)
2.11 (1)
1.99 (1)
2.69 (1)
3.59 (1)
2.63 (1)
Pika
1.14 (2)
1.12 (2)
1.22 (2)
0.92 (3)
0.78 (3)
1.06 (2)
Latte
0.62 (5)
0.89 (4)
0.84 (5)
1.30 (2)
1.15 (2)
0.82 (3)
TF-T2V
0.83 (4)
0.80 (5)
0.89 (4)
0.72 (4)
0.46 (4)
0.65 (4)
Videocrafter2
0.87 (3)
0.96 (3) 1.01 (3)
0.54 (5)
0.29 (5)
0.65 (5)
Table 13: Prompt Category Plant
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical
Human
Model
Robustness
Preference
Gen2
2.07 (1) 2.22 (1)
2.11 (1)
2.99 (1)
4.01 (1)
3.02 (1)
Pika
1.05 (3)
1.01 (3)
1.03 (3)
1.17 (3)
0.85 (3)
0.83 (3)
Latte
1.04 (4)
0.89 (4)
1.07 (2)
1.42 (2)
1.67 (2)
0.99 (2)
TR-T2V
1.07 (2)
1.04 (2)
0.87 (4)
0.78 (4)
0.59 (4)
0.70 (4)
Videocrafter2
0.71 (5)
0.73 (5)
0.78 (5)
0.41 (5)
0.22 (5)
0.44 (5)
Table 14:
Prompt Category Scenery
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical
Human
Model
Robustness
Preference
Gen2
2.30 (1) 2.37 (1) 2.21 (1)
3.31 (1)
3.66 (1)
2.56 (1)
Pika
0.88 (3)
0.97 (2)
1.01 (2)
0.78 (3)
0.94 (3)
0.98 (3)
Latte
0.88 (4)
0.91 (3)
0.81 (5)
1.04 (2)
1.37
1.02 (2)
TF-12V
1.08 (2)
0.83 (4)
0.95 (3)
0.73 (4)
0.50 (4)
0.92 (4)
Videocrafter2
0.73 (5)
0.74 (5)
0.85 (4)
0.38 (5)
0.30 (5)
0.56 (5)
Table 15:
Prompt Category Vehicles
Video Quality
Temporal Quality
Motion Quality
Text Alignment
Ethical
Human Preference
Model
Robustness
Gen2
2.37 (1)
2.49 (1) 2.50 (1)
2.62 (1)
3.50 (1)
2.97 (1)
Pika
1.24 (2)
1.32 (2)
1.23 (2)
0.98 (3)
0.73 (3)
0.96 (2)
Latte
0.73 (5)
0.70 (5)
0.86 (3)
1.62 (2)
1.09
0.72 (3)
TF-T2V
0.77 (4)
0.74 (3)
0.67 (5)
0.48 (5)
0.47 (4)
0.54 (4)
Videocrafter2
0.93 (3)
0.74 (4)
0.77 (4)
0.62 (4)
0.30 (5)
0.49 (5)
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They provide clear overview of what the paper aims to achieve and the methodologies used, aligning well with the detailed findings presented in the subsequent sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the paper does discuss the limitations of the work performed by the authors. It helps the stage for future work and encourages ongoing dialogue the field.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
34

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: We only conducted statistical experiments and analyzed the results of the dynamic evaluation module, and did not address the theoretical analyses.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Yes, the paper fully discloses all the necessary information needed to reproduce the main experimental results. The authors have been meticulous in detailing the methodol- ogy, settings, and parameters used in their experiments, ensuring that other researchers can replicate the study accurately and validate the findings.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
35

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes, the paper provides all the code needed for the protocol.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
hublic/guides//
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //ips. "nips.cc/publii.e wwtppbccc/aaaecccccccccccccc gwwideeeCoeeeeeeeeeeeeeeeeeeeeee
ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, the paper specifies all the training and test details, including data splits, hyperparameters, the rationale behind their selection, and the type of optimizer used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, the paper reports appropriate information about the statistical significance of the experiments.
Guidelines:
36

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer:[Yes)
Yes, for each experiment, the paper provides sufficient information on the computer resources required.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerrrp.eeeee htt.ol/iie/hhiisniieelleeeeeeee
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
37

---

Justification: Yes, the paper discusses both potential positive and negative societal impacts of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, the paper describes safeguards that have been put in place for the respon- sible release of data or models that have a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the creators or original owners of assets used in the paper are properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
38

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ com/dataset has curated licenses for some datasets. Their licensing guide can help determine license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, new assets introduced in the paper are well documented, and the docu- mentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, for crowdsourcing experiments and research with human subjects, the paper includes the full text of instructions given participants, screenshots where applicable, and details about compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
39

---

Justification:
Yes, the paper describes potential risks incurred by study participants, ensures that these risks were disclosed to the subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "jzW81U4Zvk", "reviewer": "Reviewer_2pJH", "summary": "This paper designs an evaluation protocol and surveys a number of video generation papers published over the last few years.  The authors point out shortcomings of evaluation protocols in these priors works (e.g. they often limit studies to video quality comparisons with no clear definitions of video quality, small number of videos, annotators etc).  The authors promise to publish their trainings for annotators along with an interface.  They show that these trainings help significantly to improve reliability of annotator scores.\n\nAnother contribution is a method that doesn’t require all pairs of models to be compared and dynamically is able to select pairs to be compared and show experimentally that this scales better than the naive quadratic approach. \n\nFinally the authors use their methodology to compare 5 recent models and show among other things that Runway and Pika outperform their open source competitors.", "questions": "See weaknesses above.", "limitations": "yes", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Overall the list of surveyed papers in this paper is quite impressive and comprehensive. Moreover, having an evaluation method that can be used without all-pairs of comparisons to be run is clearly important particularly as the number of video generation models increases.  I also absolutely agree with the weaknesses of other video evaluations and agree that human evals are the golden standard and so making a public one available would be very beneficial for the community.", "weaknesses": "Weaknesses include the following:\n* The authors cite works like Evalcrafter/Vbench but are vague about what value this current submission brings over these prior approaches. Specifically they claim that these approaches “may lack diversity to cover all real-world scenarios” but there are no details given.  As I’ve started to see a number of papers published using the Evalcrafter methodology, I would like to see a clear explanation of the pros and cons.\n* Moreover the number of models actually evaluated in this paper is fairly small (only 5) — given that this is an evaluation methodology paper, it would be much stronger to include more models and perhaps even to show clearly that these human evaluations do not correlate well with things like FID or FVD.\n* I recommend citing and considering using ELO scores which similarly allow for us to score and rank models based on not-all-pairs comparisons and also allow for dynamic selection of model pairs to be compared side by side.  See e.g., https://lmsys.org/blog/2023-05-03-arena/ for a recent example using ELO to rank LLMs by quality as perceived by humans.  Related is the Microsoft TrueSkill work which has also been used widely.\n* Though the authors make a big deal of the fact that few prior works reveal the details of their interface and how the videos are displayed, I found very few details actually revealed in this paper.  The one clear detail I can see is that the authors mention standardizing the height of the video across models.  But this may not be the right thing to do!  Consider the VideoPoet work which generate videos in portrait aspect ratios — resizing so that heights are fixed would be unfair to these vertical videos.  Other details that are important are how to best show videos that are generated at different framerates and even different lengths (for example, how does one compare a 5 second clip from e.g. VideoCrafter to a 1 minute clip from Sora?).\n* Finally some things that are marked as objective clearly are not objective like aesthetic quality.  It would be helpful / more convincing to explain clearly what the training looks like for such a dimension.  I would presume that one example showing what aesthetic quality means is not nearly enough…\n\n\nMinor quibbles:\n* I recommend being clear that “higher is better” in many plots\n* The authors also use a lot of acronyms which impedes readability in various parts of the paper (e.g. last subsection of 4.3)\n* I somewhat disagree with the characterization of FVD being about temporal consistency… even though this is part of it, the statements about the other metrics also apply — it compares feature representations from real and generated images and assesses diversity and clarity…\n* I recommend seeing/citing the Videopoet paper that had very similar comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer 2pJH for the valuable comments and give detailed responses to all questions. Due to space constraints, we submit part of the replies in \"Official Comment\", for convenience, we have summarized conclusions for each reply. Looking forward to further discussion with you.\n\n**W1: Vague about what value this current submission brings over these prior approaches.**\n\nThanks for the comments. As we argue in lines 115-124 of the article, Evalcrafter [1], Vbench [2], and other papers have done excellent work in building scorers that match human intuition, as they can implement model evaluation at much lower cost, undoubtedly, these evaluation tools represent a promising research direction, but it cannot completely replace human evaluation protocols for now. We analyze it from the following perspectives.\n\n**Analysis:**\n\n- **Less flexibility and Higher expansion costs.** Since scorers are essentially pre-trained models, though the scorer can reduce evaluation costs, the metrics that can be evaluated and the types of data that can be accepted are fixed. Furthermore, its expansion cost is much higher than our protocol. \n- For example, Vbench's evaluation system includes several pre-trained models to evaluate certain features in the video [2]. If the definition of the existing evaluation object needs to be updated, the evaluation process requires multimodal data [3] or a new evaluation dimension needs to be added, one not only needs to update the project but also needs to conduct large-scale human evaluation experiments to verify its reliability. Meanwhile, updating our human evaluation protocol is **relatively simple**, requiring only an update of the annotation interface.\n- **Scorer training relies heavily on the design and result of human evaluation**. An important measure for all automatic scoring tools is whether they agree with human intuition. Therefore constructing comprehensive and fair human assessment protocols is **an important prerequisite** for training reliable automatic scorers. Our survey shows that many studies training automated scorers use untrained laboratory recruiters, which seriously diminishes the reliability of the scoring results produced by the final trained model. Hence, designing reliable, reproducible, and practical human evaluation protocols could **help improve the quality of such work** as well.\n\n**Conclusions:**\n\n- Although automated scorers are a promising research direction, their inherent poor flexibility, and  high expansion costs result in the fact that they are still **unable to fully replace** human evaluation protocols.\n- More importantly, the design of better human evaluation protocols is itself **an important prerequisite** for the training of more reliable scorers.\n\n**W2: Include more models or show clearly that these human evaluations do not correlate well with things like FID or FVD.**\n\nThanks for the comments. We analyze it from the following perspectives.\n\n**Analysis:**\n\n- **The number of evaluated models.** We have fully investigated recent benchmark works. Due to the slow inference speed of generative models, all related works focus on evaluating the state-of-the-art four or five models, as shown in the table below, .\n\n| **Method**|T2VScore [4]|EvalCrafter [1]|Vbench [2]|AIGCBench [5]|FETV [6]|T2IHE [7]|\n| -|-|-|-|-|-|-|\n| **Number of evaluated models** | 4| 5| 4| 5| 4| 4|\n\n- **Future work**. Since our work began at the beginning of this year, even though we had selected the most advanced models at the time, as many video generation works were completed in April this year, we still miss many excellent concurrent works. Given the time cost of human evaluation, we are sorry that we cannot add the evaluation results of more models for now. **We will continue to evaluate more models in the future.**\n- **Correlation with FID and FVD.** We utilize the Prompt Suite per Category [2] as a source of prompts, which provides a comprehensive set of prompt types, but as it was built manually, there are **no corresponding reference videos** for calculating FID and FVD. Also, there has been a lot of work demonstrating the inconsistencies between automatic indicators such as FID and FVD and human perception [1,6]. We will use broader prompt sources in the future and further supplement the relevant work.\n\n**Conclusions:**\n\n- We tried our best to evaluate the **five most advanced** video generation works available at the start of the research. This amount is in line with other benchmarking works, and we will conduct more model evaluations later.\n- Since there are **no corresponding reference videos** for the prompt source, we cannot calculate the corresponding FID and FVD indicators, but we will supplement this when expanding our prompt source in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjxIskvcbe", "reviewer": "Reviewer_AEKZ", "summary": "This paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a standardized approach for evaluating text-to-video (T2V) models, addressing the challenges posed by recent advancements in T2V technology. The T2VHE protocol supposedly offers a solution through well-defined metrics, annotator training, and an effective dynamic evaluation module. \n\nExperimental results suggest that this protocol can offer high-quality annotations and also reduce evaluation costs. The authors plan to open-source the T2VHE protocol setup.", "questions": "* Can you provide more details on how the dynamic evaluation module handles potential biases when discarding certain comparisons, especially for top-performing models like Gen2?\n* How does the protocol ensure that the reduced number of annotations in the dynamic evaluation still captures a representative sample across different prompt categories and model capabilities?\n* What steps were taken to validate that the training provided to LRAs effectively mitigates potential biases they may have, especially given their potential lack of diversity compared to crowdsourced annotators?\n* How does the protocol account for potential cultural or linguistic biases in the evaluation of \"Ethical Robustness\"? Can you provide more details on how this metric is defined and measured?\n* What considerations were made in designing the protocol to ensure its applicability across different types of T2V models (e.g., models specializing in different video styles or lengths)?\n* Would the survey of non -T2V evaluation approaches be beneficial when merging with this technology? With the lack of maturity and rapidly evolving field, would not benefit from a cross-domain understanding from more mature fields?", "limitations": "Mostly adequately addressed limitations. Would like to see some understanding in regards to human annotation, than just T2V.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a timely issue in the rapidly evolving field of text-to-video (T2V) models: the lack of standardized quality assessment. Its strengths lie in providing a comprehensive literature review since 2016 and synthesizing past research into a unified evaluation protocol. This effort to aggregate previous work is particularly valuable, given the growing importance of T2V models. By proposing a standardized approach, the authors contribute to filling a critical gap in the field, potentially accelerating progress in T2V model development and evaluation. The highlighted strengths are as follows: \n\n* It introduces a novel, comprehensive protocol (T2VHE) for human evaluation of text-to-video (T2V) models, addressing a significant gap in the field.\n* The paper combines existing ideas (e.g., comparative scoring, Rao and Kupper model) in creative ways to improve the evaluation process.\n* The authors conducted a thorough survey of 89 papers on video generation modeling, providing a solid foundation for their work.:\n* The paper is generally well-structured and clearly written:\n* The introduction provides a clear motivation for the work and outlines the main contributions.\n* The authors commit to open-sourcing the entire evaluation process and code, which is crucial for reproducibility and further advancement of the field.", "weaknesses": "Major Weaknesses\n* Lack of comparison with existing evaluation protocols: While the paper introduces a new protocol, it doesn't directly compare its performance or efficiency against existing evaluation methods. This makes it difficult to quantify the improvements over current practices. In addition, it lacks an understanding with evaluation in other domains. ex. T2I, NLP, or CV. With the lack of details or standards in T2V evaluation, it would have benefitted from an expansive understanding of human annotation as a whole. Thus, the ideas although well representative of the current state of evaluation in T2V, lacks the maturity, which affords the novelty or value, that hasn't been surveyed in other domains. \n* Potential bias in dynamic evaluation: The dynamic evaluation module, while innovative, may introduce bias by discarding certain comparisons. As noted in Table 3, comparisons involving Gen2 were frequently omitted. This could potentially lead to less accurate estimations for top-performing models. \n* Lack of error analysis: The paper doesn't provide a detailed analysis of cases where the protocol fails or produces inconsistent results. Understanding these edge cases could help improve the robustness of the evaluation method. This further dives into my understanding that although it is a great survey of existing protocols, lacks a contribution its aggregation. Further evaluation and consideration in other domains, would assure me that it is a standard that can be well accepted vs. one that is just more comprehensive than the past. \n\nMinor Weakness\n* In line 3, there's a small typo where \"pop3 ularity\" is split across lines. It should be \"popularity\".\n* In line 31, \"practicabil32 ity\" is split across lines. This appears to be due to line wrapping rather than an actual error in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer AEKZ for the valuable comments and give detailed responses to all questions. Due to space constraints, we submit part of the replies in \"Official Comment\", for convenience, we have summarized conclusions for each reply. Looking forward to further discussion with you.\n\n**W1:  Lack of comparison with existing evaluation protocols. Lacks an understanding of evaluation in other domains.**\n\nThank you for your insightful comment. We analyze it from the following perspectives.\n\n**Analysis:**\n\n- **We compared our protocol with existing ones in lines 31-71 and Tables 6 and 7.** Specifically, existing human evaluation for video generation meets reproducibility, reliability, and practicability challenges. To address these issues, we designed a well-developed evaluation metrics system, provided detailed annotator training, introduced a novel dynamic evaluation component, and demonstrated the effectiveness of these designs through extensive and well-established annotation experiments. **As shown in Tables 1-3, our design not only effectively improves annotator consistency, but also reduces the annotation cost by about 50%.**\n- While we investigated many corresponding protocols in the T2I and NLG communities [1-3], we did not show them in detail in the paper due to space limits. These studies usually focus on the design of protocol metrics definitions, whereas our protocol not only builds a comprehensive evaluation system, but also explores the differences between professional annotators and lab recruiters, and how the differences can be bridged through effective annotator training. Furthermore, we propose a novel dynamic assessment component based on traditional protocols, which drastically reduces the cost of assessment. **These contributions have not been performed by other fields of assessment**.\n- Regarding the maturity of our protocol, we investigated more than 200 video generation literature, including but not limited to the fields of T2V, video editing, and frame prediction. After referring to these and the studies in the fields of T2I and NLG, we summarized a set of detailed evaluation metrics and training processes. Due to time and cost constraints, we did not extend the protocol to areas such as T2I and NLG, which will be our **future work**. We'll also add a discussion of this in the article.\n\n**Conclusion:**\n\n- Our protocol is based on a thorough investigation of human evaluation and video generation articles in different fields and is experimentally validated to have the advantages of **reproducibility, reliability, and practicability** compared to traditional protocols.\n\n**W2:  The dynamic evaluation module, may introduce bias by discarding certain comparisons.**\n\nThank you for your insightful comment. We analyze it from the following perspectives.\n\n**Analysis:**\n\n- Our estimates of model scores are based on the Rao and Kupper model [4], which is **inherently capable of handling incomplete comparisons**. This property ensures that our ranking estimates remain reliable even when some comparisons are discarded.\n- We provide an in-depth analysis of this phenomenon in lines 325-340, the rigorous bootstrap confidence intervals indicate that **discarding comparisons with significant differences does not affect the estimation of the final ranking results**, as shown in Figure 3.\n\n**Conclusion:**\n\n- Theoretically, our protocol design allows for plausible estimation in the absence of numerous comparisons, and we further validate this capability through bootstrap confidence interval experiments.\n\n**W3: The paper doesn't provide a detailed analysis of cases where the protocol fails or produces inconsistent results. Need further evaluation and consideration in other domains.**\n\nThank you for your insightful comment. We analyze it from the following perspectives.\n\n**Analysis:**\n\n- We analyze the situation when protocol produces inconsistent results in lines 244-253. Specifically, in cases where researchers use untrained lab recruiters, their annotation results may be inconsistent with those from crowdsourcing platforms and trained annotators, **highlighting the importance of the annotator training process we designed**.\n- We also provide an analysis of the dynamic component's reliability in lines 325-340, the rigorous bootstrap confidence intervals indicate that it can **guarantee the reliability** of the results while reducing the cost of annotation by about **50%**.\n- The proposed annotator training methods and dynamic components **are not limited to the T2V research field**, for the T2I and NLG research fields, by simply modifying related concepts and examples, one can directly use these in the customized protocol framework. \n\n**Conclusions:**\n\n- We analyzed the case of producing inconsistent results under the traditional protocol setup, demonstrating the importance of the annotator training we designed, and conducted a reliability analysis of the dynamic evaluation component, which proved that the introduction of this component does not result in inconsistent protocol results.\n- Our design is not only to provide a more reliable, reproducible, and practical T2V assessment protocol but also to provide ideas and reference materials for human evaluation research in **other fields**.\n\n**MW1&MW2: Small typo errors.**\n\nThanks for the reminder. These typo errors are caused by automatic line breaks during latex compilation, not actual errors in the text editor.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oHcWVkJaBt", "reviewer": "Reviewer_RMCR", "summary": "This manuscript presents a novel, standardized human evaluation protocol specifically tailored for Text-to-Video (T2V) models. The protocol encompasses a meticulously designed suite of evaluation metrics supplemented with robust annotator training resources. The effective deployment of Latent Action Ratings (LARs) is a notable feature of this work, enabling the acquisition of high-quality annotations. A dynamic evaluation component is also introduced, which remarkably curtails annotation costs by approximately 50% while maintaining the integrity of annotation quality. The paper further offers an exhaustive evaluation of cutting-edge T2V models. A commitment to open-source the entire evaluation process and associated code is a commendable initiative, which will undoubtedly empower the broader research community to assess novel models using updated data, building upon existing reviews.", "questions": "I acknowledge that I appreciate the authors' contributions to the open source community, and reasonable and practical answers to the concerns in the \"Weaknesses\" section will help improve my rating.", "limitations": "This manuscript has discussed the relevant limitations, please refer to the \"Weaknesses\" section for more details.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ First of all, I am very grateful to the authors for their great contribution to the open-source community. I believe that this work will help existing research take a bigger step after it is released;\n+ Then, this paper carefully considers the evaluation criteria of T2V. 4 objective and 2 subjective annotation evaluations are used to improve the quality of discrimination, making the annotation results more in line with human preferences and lower error rates or bias;\n+ Furthermore, this work maximizes the effectiveness of comparative evaluation. From my point of view, this is very much like doing a reward model that serves for reinforcement learning from human feedback (RLHF), which will help unify and simplify the difficulty of annotation and quantification;\n+ Finally, due to a large amount of human power investment, the authors introduced a dynamic evaluation component to reduce the annotation cost and claimed that the expense could be reduced by 50% at the same quality.", "weaknesses": "— My biggest concern is that the evaluation metrics currently proposed by the authors cannot be proven reasonable or convincing. Although this manuscript has used several quality standards (4 objective and 2 subjective), are they complete or redundant? For example, I think \"Video Quality\" is a vast range, which may include multiple aspects such as aesthetics, clarity, and even another indicator, \"Text Alignment\" (because if the generated elements do not match the prompt, it will not be considered a good video); I think \"Temporal Quality\" and \"Motion Quality\" are not two completely independent metrics, they should be highly related and coupled.\n— Furthermore, the authors mentioned that this paper adopts the Rao and Kupper model to quantify annotations results. I tried to follow this paper to understand how to quantify these metrics whose weights I am not sure about. Still, unfortunately, I found no similarities between it and this paper. I hope the authors can scientifically and clearly explain how the metrics are involved in the final results, which is one of the core contributions of this paper.\n— In addition, since the annotated results of all metrics are labeled by manual A/B testing, is the human preference the only one that dominates? How can the aesthetics and evaluation systems of different annotators be consistent? The same problem arises in the reward model, where its approach has been verified in the LLM field and cannot achieve high accuracy (only about 70+%). Thus, it needs to be used in conjunction with PPO/DPO. As this manuscript constructs datasets this way,  how can we avoid the problem?\n— Finally, although the dynamic evaluation component proposed in this article reduces annotation costs to approximately half,  the annotation task is too dependent on manual work; I think its promotion and ease of use in the open-source community are still questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer RMCR for the valuable comments and give detailed responses to all questions. Due to space constraints, we submit part of the replies in \"Official Comment\", for convenience, we have summarized conclusions for each reply. Looking forward to further discussion with you.\n\n**W1: The evaluation metrics currently proposed by the authors cannot be proven reasonable or convincing.**\n\nThanks for the comments. To build a reasonable evaluation system, we have investigated over 200 papers and found that due to the inconsistency of definition, the evaluation systems differ greatly. For example, for the same metric \"Video Quality\", some split it into eight sub-metrics, while some define it as an overall preference, as in the example given by the reviewer. Therefore, to **balance the practicality and comprehensiveness** of the protocol, we analyze this from the following perspectives.\n\n**Analysis:**\n\n- **Decomposition of \"Video Quality\".** For the T2V models, one of the most basic requirements is that the generated content should be consistent with the prompt, i.e., \"Text Alignment\", and the second one is logical, including whether the action conforms to physical laws and whether it has a certain degree of continuity, i.e., \"Motion Quality\" and \"Temporal Quality\". Based on these, the creators usually want the video content to have certain characteristics, i.e., \"Video Quality\" in the narrow sense.\n- **Division of \"Motion Quality\" and \"Temporal Quality\".** We observe that many studies are focusing on how to keep the objects continuous and flicker-free between frames. Measuring this ability separately from motion quality can help to compare and analyze these models more fairly, e.g., placing studies focusing on long video generation [1] under a single metric with studies focusing on motion control [2] is unfair to both studies.\n- To further improve the definition of the four objective metrics, we investigated many benchmark studies [3-6], which were **also carried out from these four aspects**, and further developed the corresponding reference angles. However, in further research into the relatively mature fields [7,9], we find that existing T2V evaluation systems lack the design of these subjective metrics, i.e., \"Ethical Robustness\" and \"Human Preference\".\n- **Necessity of subjective metrics.** Due to the diversity of users, ethical review of the generated content is an important part of T2I model evaluation [7]. Unfortunately, our investigation shows all current T2V evaluation protocols missed this consideration. Therefore, we refer to the research ideas of T2I and formulate corresponding definitions and reference angles to make up for this lack. Similarly, human preferences should be assessed independently of objective metrics because of differences in individual preferences, e.g., even if some videos are of poor quality, the interesting content can still give the model practical value, which should not be overlooked during the evaluation.\n- **Metric granularity.** To train a reliable scorer, VBench [4] further divides its objective metric, producing 16 sub-metrics. However, this practice **does not apply to the design of human evaluation protocols**, as annotators usually have strong abstract capabilities, just proper training and guidance can help them fully understand the indicators and produce reliable annotations [9]. Therefore, to reduce the annotation cost and improve the practicality of our protocol, we did not design more disaggregated metrics.\n\n**Conclusion:**\n\n- Through extensive research and rigorous analysis, we designed an assessment system that not only **incorporates the mainstream metrics** used in existing T2V studies but also **complements the missing parts** of previous studies, making our evaluation metrics reasonable or convincing.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MoreIi7bTJ", "reviewer": "Reviewer_LQLU", "summary": "This paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a standardized approach for assessing the quality of videos generated from text. \nBesides, this paper comprehensively evaluates some famous text-to-video models with their proposed reliable human assessment, which is very exciting and valuable for this community.", "questions": "1. How does the performance of the evaluated models vary across different domains or types of prompts?", "limitations": "NONE", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Standardized human evaluation protocol: T2VHE provides a comprehensive and consistent framework for evaluating T2V models, enabling fair and reliable comparisons between different models.\n2. Reduced annotation costs: The dynamic evaluation module reduces the number of annotations needed by approximately 50%, making it more practical for researchers to evaluate large numbers of models.\n3. Comprehensive evaluation on existing text-to-video models, revealing some interesting findings.", "weaknesses": "1. The algorithm’s performance and stability may vary depending on the specific prompt types and model characteristics.\n2. The ELO rating [1] mechanism seems to be a competitive baseline for the Rao and Kupper method used in this paper, which has been widely used in the comparsion of LLMs. Could you please describe the difference between Rao and Kupper method and ELO rating, and predict which kinds of method is better?\n\n[1] Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings (https://lmsys.org/blog/2023-05-03-arena)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer LQLU for the valuable comments and give detailed responses to all questions. Due to space constraints, we submit part of the replies in \"Official Comment\", for convenience, we have summarized conclusions for each reply. Looking forward to further discussion with you.\n\n**W1: The algorithm’s performance and stability may vary depending on the specific prompt types and model characteristics.**\n\nThank you for your insightful feedback, which will undoubtedly contribute to the improvement and clarity of our work. For the concerns and questions, here are our responses:\n\n**Analysis:**\n\n- Since our dynamic algorithm is designed for human evaluation protocols, as with all assessment protocols, it is inevitable that for different assessment contents, i.e., using different prompts and different models, there will be an impact on the final assessment results, but **this impact is limited**.\n\n- To reduce the impact of different types of prompts on algorithm performance, we utilize the Prompt Suite per Category [4] as a source of prompts, which comprises prompts manually curated from eight distinct categories. This approach ensures that the performance of the algorithm will not be affected too much as long as the source of prompts is comprehensive when using this dynamic evaluation component.\n\n- As for the effect of different model performances on the algorithm's performance, as shown in Table 3, the number of comparison pairs discarded will be smaller if a set of models with similar performances is used for the evaluation, while the algorithm's effect will be more significant if the performance gap between the models to be compared is obvious. This phenomenon is in line with our starting point, i.e., no more cost is wasted on evaluating combinations with significant differences in strength, whereas more comparisons are needed to obtain accurate results for combinations with close performance.\n\n- Furthermore, we employ bootstrap confidence intervals for the score estimates of each model across various metrics. Section 5 and Figure 3 of our paper demonstrate that **even in extreme cases, our rank estimation is consistent and trustworthy**.\n\n\n\n**Conclusion:**\n\n- The effect of different prompts and model characteristics on the algorithm is limited, and experimental results demonstrate that **this effect does not diminish the validity and reliability** of the dynamic evaluation component at all.\n\n\n\n**W2: Could you please describe the difference between Rao and Kupper method and ELO rating, and predict which kind of method is better?**\n\nThank you for highlighting the comparison between the ELO rating mechanism and the Rao and Kupper model. We appreciate the opportunity to clarify the differences and discuss the suitability of each method for our specific application.\n\n**Analysis:**\n\n- **Comparison Between ELO Rating and Rao & Kupper Model.** The ELO rating system is a widely recognized method for ranking and scoring based on pairwise comparisons, especially in competitive environments such as chess and, more recently, in comparing large language models (LLMs) [1]. However, the Rao and Kupper model [2], which we have chosen for this study, is **a modification of the Bradley-Terry model**[3] that accommodates tied comparison results, making it **more flexible** in scenarios where outcomes are not always decisive.\n\n- **Handling Tied Comparisons:**\n\t- **ELO Rating:** Typically, the ELO system is designed for scenarios where a clear winner and loser are defined. It does not inherently handle ties as effectively as the Rao and Kupper model.\n\t- **Rao & Kupper Model:** This model explicitly accounts for tied results, which is particularly beneficial in our context where annotators may find two models equally effective. This capability to handle ties allows for more accurate and nuanced model rankings based on the available data.\n\n- **Statistical Robustness:**\n\t- **ELO Rating:** The ELO system assumes a relatively straightforward model of win/loss, which can lead to biases when dealing with incomplete data or when the matches are not representative of all possible pairings.\n\t- **Rao & Kupper Model:** Our approach using the Rao and Kupper model is statistically robust against incomplete comparison data. By leveraging the maximum likelihood estimation process, it provides a more reliable ranking even with missing or tied data.\n\n- **Dynamic Pair Selection.** Both models allow for the dynamic selection of model pairs. However, the ability of the Rao and Kupper model to account for ties gives it an edge in scenarios where comparisons may not always yield a clear winner.\n\n- **Predicting Method Suitability.** Given the nature of our task, which involves nuanced evaluations where ties are possible and even expected, **the Rao and Kupper model is more suited to our needs**. While the ELO rating system is competitive and has been successfully applied in many domains, the ability of the Rao and Kupper model to handle ties and its robustness against incomplete data makes it more appropriate for our use case.\n\n\n\n**Conclusion:**\n\n- We hope this detailed explanation clarifies our methodology and addresses any concerns about the application of the Rao and Kupper model in our research. We also provide detailed parameter estimation details in Appendix C.3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received a Strong Accept, 2 Borderline Accept and a Reject. The reviewers found the paper novel, well motivated and the references to prior work quite thorough and beneficial to the community. The main concerns were that: 1) it is unclear what additional benefit the proposed method brings over prior work, 2) the limited comparisons, 3) the design of standardisation is not satisfactory (ie, how to compare videos with different aspect ratios, length, fps), 4) lack of error analysis. The authors engaged in extensive discussions with the reviewers. 3 reviewers acknowledged that the authors' responses were satisfactory. 2 raised their ratings, while one kept his high score.  
The reviewer who recommended a Reject for the paper did not engage in the discussion nor provided further input to clarify if the answers addressed the concerns. The AC has read all the reviews and the discussions and finds that the authors addressed all the concerns. Thus the AC recommends this paper for acceptance.
Decision: Accept (poster)