Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing Reliability, Reproducibility, and Practicality

Authors: tianle  zhang
langtian  ma
yuchen  yan
yuchen  zhang
kai  wang
yue  yang
ziyao  guo
wenqi  shao
yang  you
yu  qiao
ping  luo
kaipeng  zhang
shanghai ai laboratory
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 15:31:09

Abstract: Recent text-to-video (T2V) technology advancements, as demonstrated by models such as Gen2, Pika, and Sora, have significantly broadened its applicability and popularity. 
Despite these strides, evaluating these models poses substantial challenges. 
Primarily, due to the limitations inherent in automatic metrics, manual evaluation is often considered a superior method for assessing T2V generation. However, existing manual evaluation protocols face reproducibility, reliability, and practicality issues.
To address these challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a comprehensive and standardized protocol for T2V models. 
The T2VHE protocol includes well-defined metrics, thorough annotator training, and an effective dynamic evaluation module. 
Experimental results demonstrate that this protocol not only ensures high-quality annotations but can also reduce evaluation costs by nearly 50\%.
We will open-source the entire setup of the T2VHE protocol, including the complete protocol workflow, the dynamic evaluation component details, and the annotation interface code. This will help communities establish more sophisticated human assessment protocols.

Main paper: 
 introduction: 
 Text-to-video (T2V) technology has made significant advancements in the last two years and garnered increasing attention from the general community. T2V products such as Gen2 [20] and Pika [18] have attracted many users. More recently, Sora [65] , a powerful T2V model from OpenAI, further heightened public anticipation for the T2V technology. Predictably, the evaluation of the T2V generation will also become increasingly important, which can guide the development of T2V and assist the public in selecting appropriate models [40, 54] . This paper does a comprehensive paper survey (see Appendix F for details) and explores a human evaluation protocol for T2V generation.Automatic and human evaluation are the two main kinds of evaluation for video generation. In recent years, nearly half of video generation papers conduct only automatic evaluation, such as Inception Score (IS) [76] , Frechet Inception Distance (FID) [34] , Frechet Video Distance (FVD) [84] , CLIP Similarity [70] and Video Quality Assessment (VQA) [82, 49] . However, these metrics meet various challenges, such as relying on reference videos for calculation, overlooking temporal motion changes, and, more importantly, not aligning well with human perception [67, 54] . Undoubtedly, automatic evaluation is a promising research direction, but human evaluation is more convincing so far. Existing human evaluation for video generation also meets reproducibility, reliability, and practicability challenges. Our survey reveals that few papers employ consistent human evaluation protocols, evidenced by significant disparities in assessment metrics, evaluation methods, and annotator sources across studies. For example, some papers [100, 53] use Likert scales while others prefer comparative approach [40, 26] . Moreover, many papers lack detailed information on evaluation protocols, impeding further analysis and reproducibility. Additionally, most papers rely on annotators recruited directly by the authors, i.e., laboratory-recruited annotators (LRAs). These articles often lack quality checks, which can introduce bias and affect reliability [8] . Furthermore, there is notable variation in the number of annotations used across papers, ranging from a few dozen to tens of thousands, posing a practical challenge in achieving a balance between credible results and resource constraints.This paper introduces Text-to-Video Human Evaluation (T2VHE), a standardized human evaluation protocol for T2V generation. T2VHE consists of well-designed evaluation metrics, annotator training encompassing detailed instructions and illustrative examples, and a user-friendly interface, see Figure 1 for illustrations. Additionly, it introduces a dynamic evaluation module to reduce the annotation cost. We further introduce the details of our protocol as follows:Evaluation metrics. Many previous research focuses on video quality and text alignment but neglects the critical aspects of temporal and motion quality inherent to videos [103, 74] , as well as ethical considerations [2] . T2VHE employs four objective evaluation metrics: video quality, temporal quality, motion quality, and text alignment, and two subjective metrics: ethical robustness and human preference. For different types of metrics, annotators were asked to rely more on the definitions of reference perspectives and their preferences to make judgments, respectively.Evaluation method. Due to the complexities and potential noise inherent in absolute scoring [67] , T2VHE employs the comparison-based method, which is relatively annotator-friendly [8] . Critiquing the reliance of traditional protocols on win rates, which may introduce biases and offer limited insights into model performance [33, 22] , T2VHE adopts the Rao and Kupper model [72] to quantify annotations results. This approach enables more efficient management of pairwise comparison results, yielding improved model rankings and score estimations.Evaluators. While crowdsourcing platforms are often considered to gather high-quality annotations [8, 15] , our survey reveals that researchers primarily rely on LRAs. However, concerns about LRAs' reliability due to inadequate training and quality checks have been raised, potentially biasing evaluation outcomes. To address this, T2VHE proposes comprehensive annotator training, including detailed guidelines and examples to improve understanding of evaluation metrics. Experimental validation shows that properly trained LRAs can achieve agreement with crowdsourced annotators, affirming the effectiveness of our training approach in ensuring high-quality annotations.Dynamic evaluation module. T2VHE incorporates a dynamic evaluation component to enhance protocol efficiency. This component optimizes utility through two key functionalities: firstly, preannotation sorting of videos using automatic scoring results, prioritizing the annotation of video pairs considered more deserving of manual evaluation during the static annotation phase; secondly, the dynamic annotation phase, which determines whether to annotate pending video pairs based on differences in model scores. Experimental results indicate that this module reduces costs by approximately 50%, while concurrently ensuring the validity of the annotation results.Our study reveals several findings: (1) Post-training LRAs and annotators on crowdsourcing platforms achieve consensus. The low quality of the pre-training LRAs' annotations mainly stems from annotators' biased interpretations of evaluation metric definitions. Thus, furnishing detailed guidelines and example training can substantially improve annotation quality, yielding results comparable to those obtained by professional annotators. (2) Comparison-based evaluation exhibits significant potential for optimization. It manifests an O(N 2 ) growth trend in the number of annotations as the number of models compared increases. Our dynamic evaluation module reveals that judiciously selecting samples for annotation can produce model ranking outcomes consistent with those derived from fully annotated data, even when annotating only approximately half of them. (3) A substantial disparity persists between open-source and closed-source models. While open-source models perform well in some evaluation metrics, videos generated by closed-source models generally exhibit superior quality and tend to garner greater popularity among annotators.The main contributions of this paper are as follows:1. We introduce a standardized human evaluation protocol for T2V models, comprising a meticulously crafted array of evaluation metrics alongside accompanying annotator training resources. Moreover, it lets us acquire high-quality annotations utilizingLRAs.2. Our dynamic evaluation component reduces annotation costs to approximately half of the original expenditure while maintaining annotation quality.Recently, the success of diffusion models in image synthesis has spurred studies to adapt these models for conditional and unconditional video synthesis [86, 31, 123, 101, 5, 45, 107] .Evaluation metrics of video generative models. Evaluation metrics for video generative models can be broadly classified into automated evaluation metrics and benchmark methods. IS assesses image diversity and clarity through a pre-trained network [76] , CLIP Similarity assesses alignment between textual descriptions and generated images [70] , FID and FVD compare feature representations from real and generated images [34, 84] . Despite their usefulness, these metrics have limitations, including bias, sensitivity to surface similarity, and inconsistency with human perception [67, 54] . Additionally, VBench [40] , EvalCrafter [53] , and FETV [54] , among other benchmarks, provide comprehensive evaluations but may lack the diversity to cover all real-world scenarios, and these automated scorers typically require alignment training based on human evaluation results. Given the limitations of these automated metrics and benchmarks, high-quality manual assessments remain critical.• Each model is assigned an initial neutral strength value, indicating no prior bias. 
automatic metrics computation: 
 • For each video, the following metrics were evaluated by a pre-trained scorer [40] : subject consistency, temporal flickering, motion smoothness, dynamic fegree, aesthetic quality, imaging quality, and overall consistency. • Scores are normalized and summed to produce the feature score for each video.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes, the paper does discuss the limitations of the work performed by the authors. It helps set the stage for future work and encourages ongoing dialogue in the field.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
human evaluation in video generation: 
 We survey 89 papers on video generation models since 2016 3 , and after reviewing how they use and report human assessments, we have the following findings:Automatic vs. Human evaluation. Among the 89 papers reviewed, 44 rely exclusively on automated evaluation metrics. However, these metrics typically have limitations, such as only capturing certain characteristics of the generated video [76] , dependence on pre-trained models [70] , and the necessity for reference videos [84] . Moreover, most automated metrics have demonstrated inconsistency with human evaluations, thus rendering them unsuitable as the sole measure [67, 54] .Many benchmark studies have attempted to train automated scorers based on human-assessed results [54, 100] . However, the training data is often obtained by pre-training LRAs, and the protocols employed exhibit considerable variation across studies. Hence, though automated evaluation should be a promising research direction, human evaluation is more reliable so far, and also good human evaluation can assist the development of automated evaluation.Human evaluation metrics. The setup of human evaluation metrics varies greatly across papers. Ten studies directly use overall human preference as an evaluation metric, while some develop 16 nuanced metrics to assess from various perspectives. Overall, assessing the video's quality and relevance to the text descriptions remains the main focus of human evaluation [74, 28] . Moreover, an increasing number of studies focus on evaluating the video generation model's capabilities concerning motion, consistency, and continuity [53, 40] . In addition, we found no study introduces ethical evaluation in human evaluation, which is an aspect that warrants attention [48] .Human evaluation methods. Of the studies conducting human evaluations, nearly 70% employ a comparison-based approach in their evaluation protocols, wherein annotators select the superior video among two or more options to make judgments. In contrast, seventeen papers employed 3-point or 5-point Likert scales (i.e. absolute scoring), necessitating annotators to directly rate individual video's performance across various dimensions. However, this approach increases annotation complexity and introduces higher levels of noise and disagreement [67] .Quantification of annotation results. In evaluation protocols employing absolute scoring, the final model score is usually an average of all annotation results from all samples. On the other hand, pairwise comparison-based evaluation protocols often employ the win ratio to quantify annotations [40] . Thus, while comparison-based evaluations are more user-friendly, they often require the evaluation of all model combinations to generate stable and reliable outcomes [22] .Annotators. Many articles omit crucial annotator information, such as the number of annotators, their recruitment sources, and remuneration details. Additionally, for studies utilizing crowdsourcing platforms, only a handful of articles mention eligibility screener settings. This information is vital for gauging the reliability and ethicality of assessment results [67] .Annotator training and quality checking. Only eight articles provided instruction-based or examplebased training while utilizing LRAs. Moreover, merely two articles employed inter-annotator agreement (IAA) for annotation quality checking. Most protocols not only neglected to train LRAs before annotation but also omitted quality checks, raising concerns about the reliability of results.Prompts. For different evaluation needs, researchers usually choose different prompts for generating videos, such as HuMMan [6] , which is specifically designed for evaluating fine-grained spatiotemporal motion generation. However, datasets sourced from real-world instances often lack diversity, motivating recent benchmarking studies to advocate for manually curated cue datasets encompassing comprehensive categories for model evaluation [40, 53, 54] . Due to different needs and different sources of prompts, the number of prompts used in human assessment often varies by more than a few dozen times from study to study, and many papers use fewer than 100 samples per model for human assessment, such small sample sizes are likely to produce biased results [67] .Annotation interface. Since videos generated by T2V models usually have different resolutions and sizes, how they are presented in the annotation interface also impacts the evaluation results. While some studies adopt methods such as adding watermarks, frame sampling, or cropping to standardize videos from different models [54] , the majority only scale or leave generated videos unaltered during annotation. In addition, only nine articles provide the details of the annotation interface, with none sharing the interface code. This absence of information impedes the reproducibility of results for future studies adhering to similar protocols. Moreover, the scarcity of reusable resources hinders the ongoing enhancement of human evaluation protocols and practices [67] .Further, we provide more in-depth discussions of video processing, protocol settings, and samples for the human assessment process in Appendix C.• For each prompt, groups of model pairs are constructed.• The absolute value of the difference between the scores of two videos in a pair is input into an exponential decay model. • The output value is the score of each video pair, and the sum of these scores forms the total score for the group.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: We only conducted statistical experiments and analyzed the results of the dynamic evaluation module, and did not address the theoretical analyses.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
our protocol for text-to-vedio models: 
 Our T2VHE framework comprises four key components: evaluation metrics, evaluation method, evaluator, and dynamic evaluation module. To ensure a comprehensive assessment of the T2V model, we meticulously devise a set of evaluation metrics, accompanied by precise definitions and corresponding reference perspectives. For ease of annotation, we employ a comparison-based scoring format as evaluation method [7, 8] and develop annotator training to ensure researchers can procure high-quality annotations using post-training LRAs. Furthermore, our protocol incorporates an optional dynamic evaluation component, enabling researchers to attain reliable evaluation results at reduced costs. More details can be found in Appendix D.Drawing from established protocols in image generation evaluation, prior studies have primarily used metrics like "Video Quality" and "Overall Alignment" for human assessment. However, these metrics often suffer from vague definitions, leading annotators to base ratings on general impressions and fidelity to the textual content. This lack of specificity can introduce subjectivity, potentially undermining the quality of annotations. Recent research also underscores that motion and temporal quality are also vital metrics for assessing video generation models' capabilities [53, 40] . Additionally, as video generation technology gains popularity, its ethical and societal impacts are becoming increasingly critical factors in evaluation [48] . However, our survey reveals that none of the previous protocols considered this indicator. Moreover, only ten studies provided specific training for annotators, suggesting that the majority of research has only offered basic definitions of each metric without comprehensive instructions or relevant examples, which are essential for ensuring high-quality annotations [15] .To this end, we establish a comprehensive evaluation framework with explicit definitions and corresponding reference perspectives for each metric. Additionally, to enable precise assessments, we also devise thorough annotator training, detailed in Section 4.3. Objective indicators require strict adherence to the reference perspectives to ensure consistency and repeatability in evaluations, while subjective indicators allow for personal interpretation, providing a holistic assessment of the model's performance and potential. Recognizing the subjective nature of certain indicators, we categorize them into objective and subjective types. Objective indicators require strict adherence to the reference perspectives to ensure consistency and repeatability in evaluations, while subjective indicators allow for personal interpretation, providing a holistic assessment of the model's performance and potential. Detailed definitions and reference perspectives for each metric can be found in Table 1 . Assess whether the dynamic activities in the video are sufficient and appropriate.Which video has a higher degree of alignment with the prompt?Assess whether the video accurately reflects the types and quantities of objects described in the text.Evaluate whether the visual style of the video matches the text description.Which video demonstrates higher ethical standards and fairness?Evaluate the video for any content that might be deemed toxic or inappropriate.Determine the fairness in the portrayal and treatment of characters or subjects across different social dimensions.Assess the presence and handling of biased content within the video.As an annotator, which video do you prefer?Evaluate the originality of the video's contents.Assess the emotional and intellectual value provided by the video.Assess the video based on the previous five metrics and personal preferences.There exist two primary scoring methods: comparative and absolute. The former requires annotators to compare a set of videos and select the one demonstrating superior performance, whereas the latter entails directly assigning scores to the videos. Absolute scoring typically necessitates detailed instructions and precise question formulations due to its complexity [8] . However, even with these in place, absolute scoring could still result in noisy annotations and pose challenges in reaching consensus among annotators [67] . Hence, we use the less challenging comparative scoring method.Quantification of annotations. Traditional comparative scoring protocols rely on the win ratio in pairwise comparison, however, this method has several drawbacks. First, it can introduce bias if models are not uniformly compared [33] . For instance, a model frequently pitted against stronger counterparts might exhibit a lower win ratio compared to one facing weaker opponents more frequently. Consequently, a significant number of comparisons are required to establish reliable rankings [22] . Moreover, the win ratio alone does not reliably indicate the likelihood of one model outperforming another [72] . To overcome these issues, we adopt the Rao and Kupper model [72] , a probabilistic approach that allows for more efficient handling of the results of pairwise comparisons using less data than full comparisons. This model enables better estimation of model rankings and scores, thereby furnishing a more precise and dependable evaluation compared to simply using the win ratio. The EQUATIONwhere t is the number of models, p = (p 1 , • • • , p t ) T ∈ R t is the vector representing the scores of each model, θ is a tolerance parameter, n ij denote the number of times model i is preferred to model j, and ñij denotes the number of times the two models reached a tie. Further details about model's implementation and its parameter estimation process are provided in Appendix D.3.For most video generation evaluation tasks, the evaluator does not need specific expertise. However, using annotators from crowdsourcing platforms, such as Amazon Mechanical Turk (AMT), still results in higher annotation quality [67, 64, 66] , as these workers have usually completed many tasks and carefully followed the publisher's requirements to ensure successful payment. Nevertheless, due to cost constraints, more studies tend to use non-professional, unpaid LRAs for the annotation tasks. Furthermore, our survey showed that most studies using LRAs lack annotator training and quality checking, raising concerns about the reliability of their annotations. To address this issue, we developed a comprehensive training methodology for annotators and conducted experiments on a pilot dataset to explore the impact of annotator qualifications on annotation quality.Annotator training. We propose two cost-effective training methods: instruction-based and examplebased training. Specifically, we furnish detailed guidance for each metric, complemented by two to three reference perspectives, each perspective is paired with an example and an analytical process to aid annotators in understanding metric definitions and making accurate judgments. Detailed training interfaces are illustrated in Figure 1 After collecting all annotations, we observe disparities in model rankings derived from AMT annotators compared to those from pre-training LRAs across various metrics. To further evaluate these differences, we calculate the internal IAA 4 of AMT annotators and the external ones between them and the pre-and post-training annotators. For the latter two sets of experiments, we randomly select five annotators from the AMT group and the two LRAs groups, respectively, to calculate the corresponding IAA and average the results. As shown in Table 2 , pre-training annotators demonstrate lower agreement with professional annotators across multiple dimensions, evidenced by significantly lower IAA than the other two groups. In contrast, post-training annotators exhibit improved agreement with the professional annotators, approaching levels of intra-AMT consensus. Thus, the model rankings obtained from the two sets of annotation results are identical. These findings suggest that effective annotator training within our protocol can yield high-quality annotation results using LRAs comparable to those obtained using professional annotators.As the number of models increases, traditional evaluation protocols often become more costly. To minimize annotation costs and ensure stable model ranking with fewer comparisons, we develop a dynamic evaluation module based on two key principles: the video quality proximity rule and the model strength rule. The first principle ensures that initially evaluated video pairs are of comparable quality, reducing unnecessary annotations, the second principle selects video pairs based on model strength, enhancing evaluation efficiency. The specific process is as follows:Before annotation starts, each model receives an unbiased strength value. These scores are normalized and summed to generate a feature score for each video. Groups of model pairs are then constructed for each prompt, with the difference between video scores input into an exponential decay model to determine pair scores and group total scores. These groups are then sorted based on their total scores to prioritize those with close quality. In the follow-up phases, for each assessment indicator, all model scores are updated using the Rao and Kupper model after evaluating video pairs in the initial groups. Subsequent annotations occur in batches, with model strengths adjusted periodically.When evaluation results stabilize across all dimensions, i.e., the model rankings are unchanged for several consecutive batches, the evaluation is terminated. We provide the implementation details of the module in Appendix D.2 and verify its effectiveness in Section 5.3.• Groups were ranked according to their total scores, with higher scores at the top indicating less variation within the group. • This preprocessing step does not increase the cost of our evaluation protocol.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Yes, the paper fully discloses all the necessary information needed to reproduce the main experimental results. The authors have been meticulous in detailing the methodology, settings, and parameters used in their experiments, ensuring that other researchers can replicate the study accurately and validate the findings. 
human evaluation of existing models: 
 We evaluated five state-of-the-art T2V models, including Gen2 [20] , Pika [18] , TF-T2V [94] , Latte [56] , and Videocrafter [11] , see Appendix D.1 for details. All videos were generated without any prompt engineering or filtering. Furthermore, to ensure uniformity and ease of comparison for evaluators, we standardized the height of all videos in the annotation interface.Data preparation. We use the Prompt Suite per Category [40] as the source of prompts, which comprises prompts manually curated from eight distinct categories. We randomly select a quarter of the prompts from each category to serve as our evaluation prompts. For each prompt, we construct 10 pairwise comparisons using videos generated by five models and ask annotators to evaluate the superiority of one video over another across various metrics. This process results in 2,000 video pairs for annotation, from which we randomly sampled 200 video pairs to form the pilot dataset.To analyze the differences in results among different annotators and to affirm the protocol's generalizability and validity, following settings detailed in Section 4.3, we engage three distinct categories of annotators: AMT annotators, pre-training LRAs, and post-training LRAs. Each AMT annotator is limited to 250 tasks to maintain quality, while both pre-training and post-training LRAs are tasked with annotating all video pairs. We also test the effectiveness of our dynamic evaluation component using post-training LRAs under the same conditions.For annotators who don't use the dynamic evaluation module, we collect the annotated data for all video pairs under the six metrics, resulting in a total of 3 × 5 × 2000 × 6 annotations (three categories of annotators, five in each category). Conversely, for annotators using the dynamic component, the average number of video pairs to be annotated is only 1,068 per annotator. For AMT annotators, 76 participants contribute, with an average of 131 tasks per annotator. between the final model scores and rankings for each dimension. In addition, the annotation results of the trained LRAs closely mirror those of the AMT personnel, yielding consistent final model ranking outcomes. We also conduct a quality check of the annotation results for the LRAs before and after training. As shown in Table 3 , the annotations from the post-training LRAs exhibit higher quality.However, regardless of the annotator sources, closed-source models typically perform better. In the annotated results from AMT personnel, Gen2 demonstrates significant superiority over other models across all metrics, while Pika also exhibits commendable performance across most metrics.In contrast, the performances of open-source models show less disparity in terms of video quality, temporal quality, and motion quality metrics. TF-T2V's generations typically excel in video quality and action timing, while Videocrafter2, an earlier open-source model, demonstrates notable proficiency in generating high-quality videos. However, distinctions among the three models become more apparent in the metrics of text alignment, ethical robustness, and human preference. Notably, Latte exhibits strong performance in text alignment and ethical robustness, even surpassing Pika. This contributes to its higher ranking in human preference compared to other open-source models despite marginal differences in other metrics.As detailed in Section 5.2, our protocol, augmented by the dynamic evaluation module, cuts annotation costs to about 53% of the original expense while achieving comparable outcomes. We further explore the module's effectiveness and reliability in this section.Effectiveness. Although protocols utilizing pairwise comparisons offer convenience to annotators, their evaluation costs can escalate rapidly as the number of models under examination increases. To assess the effectiveness of the dynamic evaluation component, we randomly select corresponding annotation results from 2-4 models (25 model combinations in total) out of all annotations. For each As depicted in Figure 3 , our protocol with dynamic component demonstrates a nearly linear growth in annotation demands as the number of models increases, greatly reducing the evaluation costs.Reliability. We advocate for the reliability of the module both at the design level of the algorithm and the result level. Before the start of the dynamic evaluation, annotators are required to annotate 200 video pairs where distinguishing differences between them based on automated metrics is challenging. This step ensures that the samples most deserving of human assessment will not be discarded in the dynamic assessment and that the initially estimated model scores are not biased by specific prompt types, as elaborated in detail in the Appendix C.4.To enhance the demonstration of this module's reliability, we perform bootstrap confidence intervals for the score estimates of each model across various metrics. As shown in Figure 3 , the confidence intervals for Latte, Pika, TF-T2V, and Videocrafter2 are consistently narrow, signifying precise estimations. In contrast, the confidence intervals for Gen2's scores are relatively wide, indicating less stable estimations. This variance primarily stems from our dynamic algorithm's frequent exclusion of comparisons involving Gen2 due to its significant superiority over the other models. Table 4 details the number of these omissions. Nevertheless, even at the lower bound of the confidence intervals, Gen2's score estimation remains superior to those of all other models. This highlights that the rank estimations remain robust despite some instability in score estimation. Thus, our dynamic evaluation provides reliable and consistent rank estimations while requiring fewer annotations.• An initial set of video pairs is evaluated by humans, and model strengths are updated using the Rao and Kupper model. • Comparisons are then split into batches. For each video pair, the absolute difference in scores between the two models is entered into an exponential decay model, whose output value is the probability that this pair will be discarded. After each batch, the model strength estimates were updated under the six evaluation dimensions. • The evaluation ends when the model rankings stabilize, meaning the rankings of models under each dimension remain unchanged for several consecutive batches.We have also provided pseudo-code 1 and corresponding definitions D.2 for the dynamic evaluation component to make this easier to understand.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes]Justification: Yes, the paper provides all the code needed for the protocol.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
limitations: 
 Our study conducts well-established human evaluation experiments on five state-of-the-art video generation models, yet some limitations persist. First, because the T2V models used are relatively new and contain two closed-source models, we do not offer technical improvement suggestions. Secondly, there is potential for refining our dynamic evaluation algorithm. As the number of models evaluated increases, improvements can be made to the initial settings of model strength.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Yes, the paper specifies all the training and test details, including data splits, hyperparameters, the rationale behind their selection, and the type of optimizer used.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 To address the issues of reproducibility, reliability, and usability in previous human evaluation protocols for T2V generation, this paper introduces the T2VE protocol. By employing well-defined metrics, thorough annotator training, and a dynamic evaluation module, T2VE enables researchers to obtain high-quality annotations by LRAs at low costs. We anticipate that future research could build upon this protocol to further expand the study of human evaluations of generative models.Our protocol is designed to ensure fair video evaluations by using a simple scaling method that maintains clarity across all videos, regardless of aspect ratio or length. An extensive literature review confirms that preprocessing methods can bias comparisons, so we emphasize that no additional processing is applied, allowing annotators to assess videos in their entirety [40, 21, 100] . This approach ensures that the unique strengths of each model are preserved and fairly evaluated. For instance, using a uniform frame rate can disadvantage high frame rate models, while extracting frames from long videos may not accurately reflect their advantages. Similarly, cropping or adding watermarks can hinder text alignment evaluations.In sum, while some preprocessing may be fair and essential for specific comparisons, simply scaling videos is a more equitable approach for human evaluation.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Yes, the paper reports appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, the paper discusses both potential positive and negative societal impacts of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Yes, the paper describes safeguards that have been put in place for the responsible release of data or models that have a high risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Yes, the creators or original owners of assets used in the paper are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: Yes, new assets introduced in the paper are well documented, and the documentation is provided alongside the assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: Yes, for crowdsourcing experiments and research with human subjects, the paper includes the full text of instructions given to participants, screenshots where applicable, and details about compensation.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [Yes]  
 
 Appendix: Although our evaluation protocol has diligently established precise definitions, reference angles, and comprehensive annotator training for each metric and has eased task complexity through a comparisonbased approach, it does not imply that LRAS can entirely supplant annotators sourced from crowdsourcing platforms. Firstly, in evaluating subjective metrics, personnel from crowdsourcing platforms, given their diverse backgrounds, are better positioned to gauge the model's universality. Even post-training LRAs inevitably harbor certain biases, particularly evident in assessing the metric of Ethical Robustness, where a definitive consensus between LRAs and AMT personnel remains elusive. Secondly, while employing LRAs facilitates easier training and quality assurance of annotations, they typically require more time to complete annotations as they often engage in voluntary work rather than salaried employment. Lastly, when conducting reviews of new models, the use of LRAs introduces inherent biases, annotators naturally favor novel models, potentially leading to questionable annotation outcomes [64, 66] .Therefore, despite our study showcasing the capacity of trained LRAs to yield high-quality annotations, researchers must judiciously select annotators tailored to their specific research objectives.Our protocol is designed with broad applicability, enabling researchers with diverse objectives to enhance and adapt it as required. Researchers can draw upon the training and analysis methodologies outlined in our protocol to establish corresponding reference points, guidelines, and illustrative examples for their own metrics. Furthermore, the guidelines and training proposed in this study can be easily extended to other fields, and similarly, by modifying the automated metrics in the dynamic module, one can also achieve efficient human evaluation of other types of protocols.Regarding dynamic components, researchers retain the flexibility to adjust hyperparameters according to their specific requirements. Our simulation experiments indicate that the number of annotations necessary can be significantly reduced, potentially even below 53%, particularly when aiming to attain stable model rankings on specific metrics.In designing the dynamic evaluation module, we first counted the positions of different prompts in the video pairs sorted according to the scoring results of the automated metrics, as shown in Figure 9 . By instructing annotators to assess the initial 200 video pairs, we ensure the model's efficacy estimation before initiating the dynamic evaluation draws from assessments covering all prompt categories.This discovery also facilitates a thorough analysis of prompt categories necessitating human evaluation, where automated metrics might encounter challenges in identifying significant differences. Notably, video pairs categorized under vehicles, scenery, and animals tend to be more discernible by automated scorers, as indicated by their relatively minor presence among the initial video pairs. Conversely, domains such as food and architecture often demand human assessment for nuanced differentiation. These insights are equally informative for future endeavors to construct datasets specifically tailored for human assessment.In our investigation of model performance across various prompt types, we input human annotation results for eight distinct prompt categories into the Rao and Kupper model to rank the model scores, as shown in Tables 8 to 15 . The findings indicate that while there are minor fluctuations in evaluation outcomes based on the prompt sources, models exhibiting superior performance, such as Gen2, consistently maintain their high rankings irrespective of prompt variations.However, it is worth noting that for cases where the overall strengths of the models are close, such as Latte, TF-T2V, and Videocrafter2, different kinds of prompts may directly affect the results of the rankings among models under each dimension. Consequently, our analysis underscores the significance of employing class-wide prompts in human evaluation protocols, as they may provide a more stable assessment of model capabilities.Due to the problematic nature of controlling the quality of annotations, final results may vary even when using the same assessment protocol and assessors [43] . However, it can be effectively mitigated by a detailed description of the annotations and a harmonized assessment process [67] . We, therefore, report full implementation details of protocol use and provide example templates for human assessment in the supplementary material.Gen2 [20] is a closed-source multi-modal AI system that generates novel videos with text, images, or video clips. We use the default parameter settings from the demo for T2V generation.Pika [18] is also a popular closed-source model that enables users to convert simple text descriptions into dynamic videos. We use the default settings on the Discord platform for video generation.[94] is a diffusion-based T2V model, and we use publicly available model parameters and commands to super-resolve the generated video after generating low-frame video.Latte [56] is an open-source video generation model based on Latent Diffusion Transformer, trained on FaceForensics, SkyTimelapse, Taichi-HD, and UCF101.Videocrafter2 [11] is a high-quality open-source video generation model trained using low-quality video and synthesized high-quality images, we use its newly announced 512x320 checkpoint.Evaluating multiple video models via pairwise comparisons becomes increasingly resource-intensive as the number of models expands. To efficiently obtain stable model rankings, we propose a pluggable dynamic evaluation module founded on two principles: the video quality proximity rule and the model strength rule. It is important to note that the component of this module responsible for automatic metrics calculations is executed on an A100 GPU. However, the remainder of the protocol's calculations can be conducted on a CPU. The details of the design of the algorithm are as follows:• Video Quality Proximity Rule -Leverage the scoring results of automated metrics to allow human annotators to prioritize the annotation of samples that are difficult to distinguish with automated metrics. -Ensures that initially evaluated video pairs have similar quality levels.• Model strength Rule -Determines the evaluation priority of subsequent video pairs based on model strength scores.-Reducing the number of comparisons between models with significant differences in strength to improve algorithmic efficiency.V : Set of videos M : Set of models P : Set of prompts V(pr i ) := {{v k , v l }|v k and v l ∈ V, shares the same prompt pr i , and For the setting of hyperparameters, we set the number of N 0 to 200 in our experiments, and each batch contains 8 groups, i.e., 80 video pairs, and update the model strengths under all dimensions every 5 batches, and stop annotating and outputting the final results after five consecutive times of equal model rankings.v k ̸ = v l } S(v) : V → RA naive method to evaluate the rank and performance of different models from the paired comparison data is ranking them based on the win rate of each model. However, this method suffers from several disadvantages: (1) in scenarios where not every model is compared against others, or some models are compared more frequently than others, win rates can be biased. If a model is only compared against strong models, its win rate might be lower than a model compared mostly against weaker models [33] . (2) It requires a large number of comparisons to accurately determine the rankings; otherwise, the estimation would suffer from high variance [22] . (3) Win rates could not provide reliable estimations of the probability of one model beating another model [72] .The Rao and Kupper model [72] is a probabilistic model for the outcome of pairwise comparisons between items, which can effectively overcome the naive method's limitations. We adopt this model to characterize our evaluation results and obtain the estimations of ranking and scores of the text-to-video models.Consider the paired comparison with t models. Given a pair of models i and j, let i ≻ j denote i beating j , i ≺ j denote i losing to j, and i ∼ j denote a tie between i and j. The probabilities of for each video pair {v k , v l } ∈ V(pr i ) do9: pair_score(v k , v l ) ← f (|S(v k ) -S(v l )|, α) 10:end for 11:group_score(pr i ) ← {v k ,v l }∈V(pr i ) pair_score(v k , v l )12: end for 13: sorted_groups ← sort {V(pr i )} pr i ∈P by group_score in descending order 14: Hum-evaluation: 15: Evaluate the first N 0 groups in sorted_groups by human and update R. Evaluate the video pair by human and update R. each event are specified as:EQUATIONwhere p i and p j are positive real-valued scores of model i and model j, respectively, which can be interpreted as the strength of the models. θ is a positive real-valued parameter with larger θ implying two models are more likely to be tied.Let n ij denote the number of times model i is preferred to model j, ñij denote the number of times model i is tied with model j. The likelihood function can be written as:EQUATIONand the log-likelihood function follows:EQUATIONwhere p ∈ R t is a vector defined by p = (p 1 , . . . p t ) T . The maximum likelihood estimation of p and θ can be obtained by maximizing (4) numerically. However, obtaining human-evaluation data could be costly. To reduce the cost, we use a dynamic evaluation algorithm to give reliable estimations with fewer data.Parameter Estimation Details. To obtain a stable estimation of the Rao and Kupper model parameters, we restrict the parameter to a reasonable range and use the L-BFGS-B [73] algorithm to optimize the log-likelihood function. We restrict all p i to be greater 0.01 to avoid numerical instability caused by extremely small values. We also restrict θ ∈ [e 0.01 , e 10 ]. This is because θ can be interpreted as the exponential of the tolerance within which the annotators cannot distinguish two models with different "true" merits. Formally, according to [72] , the probabilities in the model can be written as:P (i ≻ j) = 1 4 +∞ -(Vi-Vj )+τ sech 2 ( y 2)dyP (i ∼ j) = 1 4 -(Vi-Vj )+τ -(Vi-Vj )-τ sech 2 ( y 2 )dy,where V i = ln p i , which is interpreted as the "true" merit of each model and τ = ln θ is the tolerance. We restrict τ in [0.01, 10] and therefore θ ∈ [e 0.01 , e 10 ].Confidence intervals by bootstrap. We employ a bootstrap algorithm to derive 95% confidence intervals for score estimations. Our dynamic evaluation algorithm operates independently on data from each individual, with 2000 annotations per person. Rather than resampling from pooled data, we generate 2000 bootstrap samples from the annotations of each person separately and then aggregate these samples. We resample 1000 times, resulting in 1000 bootstrap estimates using our algorithm. The confidence intervals are then calculated based on the 2.5% and 97.5% percentiles of these bootstrap estimates.We further show the model scores obtained using different annotators and their rankings in Table 5 , and the results of the scores and rankings under different prompt types are shown in Tables 8 to 15 .We show the annotation interface in Figure 1 and the presentation corresponding to each evaluation metric. For annotator training, inspired by [15] , we use lightweight annotator training methods, i.e., instruction-based training and example-based training, which do not increase the cost of evaluation but can effectively improve the quality of annotation.Instruction-based training means providing detailed instructions for each metric based on the problem description. We provide two to three reference angles for each metric to help annotators better understand the definition of each metric.Example-based training refers to providing examples for the evaluation process of each metric based on instruction-based training. We provide an example for each reference perspective and an analysis process based on the examples to help the annotator better perform the annotation task.In addition, we informed all annotators involved in the study of the possible risks they might face. The detailed training interface is shown in Figure 1 and Figures 4 to 8. We will be releasing the full evaluation protocol code shortly.More details about evaluation metrics of video generative models Evaluation metrics for video generative models can be broadly classified into automated evaluation metrics and benchmark methods. For automated evaluation metrics, the IS assesses image diversity and clarity through a pretrained network [76] , while the FID compares feature representations from real and generated images, enhancing sensitivity to image quality [34] . The FVD for videos examines temporal consistency and quality [84] . CLIPSIM uses the CLIP model to assess alignment between textual descriptions and generated images, offering a measure for text-to-image synthesis [70] . However, these metrics have limitations: IS may prioritize diversity over quality and carry biases from pre-trained networks. FID, despite improvements, can be influenced by superficial similarities and comparison dataset quality. FVD might emphasize temporal over spatial quality, and CLIPSIM's reliance on textual data can introduce language and cultural biases [67, 54] . For the benchmark frameworks, VBench evaluates video generation across multiple dimensions using tailored prompts and human preference annotations [40] . EvalCrafter integrates 17 objective metrics refined by human opinions with diverse We counted the following characteristics of the surveyed articles: whether human evaluation was performed (Humeval), whether quality checking was performed (Validity), whether crowdsourcing platforms were used (Crowds), the number of annotators (Annotators), whether training was provided to the annotators (Training), the scoring format used for the evaluation protocol (Format), the conferences/journals published (Venue), the year of publication (Year), as shown in Table 6 and 7 . Justification: Yes, the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. They provide a clear overview of what the paper aims to achieve and the methodologies used, aligning well with the detailed findings presented in the subsequent sections.✓ × × 3 × comparative ECCV 2018 MD-GAN [104] ✓ × × - × comparative CVPR 2018 PSGAN+SCGAN [106] ✓ × × 50 × absolute ECCV 2018 Gist [51] × × - - × - AAAI 2018 FBF+TS+FG [9] ✓ × ✓ 100 × comparative ICCV 2019 Few-shotV2V [89] ✓ × ✓ 60 × comparative NeurIPS 2019 Seg2Vid [68] ✓ × × 10 × comparative CVPR 2019 IRC-GAN [16] ✓ × × 20 × comparative IJCAI 2019 TFGAN [1] × × - - × - IJCAI 2019 G3AN [95] ✓ × - 27 × comparative CVPR 2020 DTVNet [116] ✓ × × 30 × comparative ECCV 2020 CAR-Nets [91] ✓ × × 40 × comparative TMM 2020 UOD [4] × × - - × - CVPR 2021 SIVS [17] × × - - × - CVPR 2021 PVG [59] ✓ ✓ - - × comparative CVPR 2021 SDTFG [19] ✓ × ✓ 60 ✓ comparative TMM 2021 GODIVA [97] ✓ × × 200 × comparative arXiv 2021 MMVID [29] × × × - × - CVPR 2022 Imagen Video [35] × × × - × - arXiv 2022 VDM [36] × × × - × - arXiv 2022 Make-A-Video [78] ✓ × ✓ - × comparative arXiv 2022 StyleGAN-V [79] × × × - × - CVPR 2022 DIGAN [113] × × × - × - ICLR 2022 FDMLV [31] × × × - × - NeurIPS 2022 DCK [108] × × - - × - TMM 2022 MMVID [29] × × - - × - CVPR 2022 Phenaki [85] × × - - × - ICLR 2022 NÜWA [99] × × - - × - ECCV 2022 NUWA-Infinity [98] × × - - × - CVPR 2022 FETV [54] ✓ ✓ × - ✓ absolute NeurIPS 2023 MAGE [38] ✓ × × 16 × absolute TMM 2023 VideoLDM [5] ✓ × ✓ 4 × comparative CVPR 2023 PYoCo [24] × × × - × - ICCV 2023 LVDM [32] × × × - × - arXiv 2023 Videogen [50] ✓ × × 17 × comparative arXiv 2023 ModelScope [88] × × × - × - arXiv 2023 Tune-A-Video [101] ✓ × × 5 × comparative ICCV 2023EQUATIONEQUATIONEQUATIONEQUATIONEQUATIONVideoCrafter1 [11] ✓ × × - × comparative arXiv 2024 AnimateDiff [27] ✓ × × - ✓ comparative ICLR 2024 SEINE [14] ✓ × × 10 × comparative ICLR 2024 ControlVideo [119] ✓ × × 5 × comparative ICLR 2024 PIA [120] ✓ × - - × comparative CVPR 2024 SimDA [103] ✓ × - - × comparative CVPR 2024 PEEKABOO [41] × × - - × - CVPR 2024 VideoPoet [46] ✓ × × 7 ✓ comparative ICML 2024Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.The code is available at https://github.com/ztlmememe/T2VHE.38th Conference on Neural Information Processing Systems (NeurIPS 2024).We comprehensively evaluate the latest T2V models and commit to open-sourcing the entire evaluation process and code, empowering the community to assess new models with fresh data based on existing reviews. Our protocol is also easily extended.We summarize the video generation models published in major conferences and journals as well as the popular ones since 2016. The full list of reviewed papers can be found in Appendix F.A positive IAA value indicates that the ratings are more consistent than random annotations. For example, the coherence rating of NLG in[43] achieves an IAA of 0.14. 


Decision: This paper received a Strong Accept, 2 Borderline Accept and a Reject. The reviewers found the paper novel, well motivated and the references to prior work quite thorough and beneficial to the community. The main concerns were that: 1) it is unclear what additional benefit the proposed method brings over prior work, 2) the limited comparisons, 3) the design of standardisation is not satisfactory (ie, how to compare videos with different aspect ratios, length, fps), 4) lack of error analysis. The authors engaged in extensive discussions with the reviewers. 3 reviewers acknowledged that the authors' responses were satisfactory. 2 raised their ratings, while one kept his high score.  
The reviewer who recommended a Reject for the paper did not engage in the discussion nor provided further input to clarify if the answers addressed the concerns. The AC has read all the reviews and the discussions and finds that the authors addressed all the concerns. Thus the AC recommends this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper designs an evaluation protocol and surveys a number of video generation papers published over the last few years.  The authors point out shortcomings of evaluation protocols in these priors works (e.g. they often limit studies to video quality comparisons with no clear definitions of video quality, small number of videos, annotators etc).  The authors promise to publish their trainings for annotators along with an interface.  They show that these trainings help significantly to improve reliability of annotator scores.

Another contribution is a method that doesn’t require all pairs of models to be compared and dynamically is able to select pairs to be compared and show experimentally that this scales better than the naive quadratic approach. 

Finally the authors use their methodology to compare 5 recent models and show among other things that Runway and Pika outperform their open source competitors.
strength_weakness: strength_weakness: Weaknesses include the following:
* The authors cite works like Evalcrafter/Vbench but are vague about what value this current submission brings over these prior approaches. Specifically they claim that these approaches “may lack diversity to cover all real-world scenarios” but there are no details given.  As I’ve started to see a number of papers published using the Evalcrafter methodology, I would like to see a clear explanation of the pros and cons.
* Moreover the number of models actually evaluated in this paper is fairly small (only 5) — given that this is an evaluation methodology paper, it would be much stronger to include more models and perhaps even to show clearly that these human evaluations do not correlate well with things like FID or FVD.
* I recommend citing and considering using ELO scores which similarly allow for us to score and rank models based on not-all-pairs comparisons and also allow for dynamic selection of model pairs to be compared side by side.  See e.g., https://lmsys.org/blog/2023-05-03-arena/ for a recent example using ELO to rank LLMs by quality as perceived by humans.  Related is the Microsoft TrueSkill work which has also been used widely.
* Though the authors make a big deal of the fact that few prior works reveal the details of their interface and how the videos are displayed, I found very few details actually revealed in this paper.  The one clear detail I can see is that the authors mention standardizing the height of the video across models.  But this may not be the right thing to do!  Consider the VideoPoet work which generate videos in portrait aspect ratios — resizing so that heights are fixed would be unfair to these vertical videos.  Other details that are important are how to best show videos that are generated at different framerates and even different lengths (for example, how does one compare a 5 second clip from e.g. VideoCrafter to a 1 minute clip from Sora?).
* Finally some things that are marked as objective clearly are not objective like aesthetic quality.  It would be helpful / more convincing to explain clearly what the training looks like for such a dimension.  I would presume that one example showing what aesthetic quality means is not nearly enough…


Minor quibbles:
* I recommend being clear that “higher is better” in many plots
* The authors also use a lot of acronyms which impedes readability in various parts of the paper (e.g. last subsection of 4.3)
* I somewhat disagree with the characterization of FVD being about temporal consistency… even though this is part of it, the statements about the other metrics also apply — it compares feature representations from real and generated images and assesses diversity and clarity…
* I recommend seeing/citing the Videopoet paper that had very similar comparisons.
questions: questions: See weaknesses above.
limitations: limitations: yes
paper_summary: paper_summary: This paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a standardized approach for evaluating text-to-video (T2V) models, addressing the challenges posed by recent advancements in T2V technology. The T2VHE protocol supposedly offers a solution through well-defined metrics, annotator training, and an effective dynamic evaluation module. 

Experimental results suggest that this protocol can offer high-quality annotations and also reduce evaluation costs. The authors plan to open-source the T2VHE protocol setup.
strength_weakness: strength_weakness: Major Weaknesses
* Lack of comparison with existing evaluation protocols: While the paper introduces a new protocol, it doesn't directly compare its performance or efficiency against existing evaluation methods. This makes it difficult to quantify the improvements over current practices. In addition, it lacks an understanding with evaluation in other domains. ex. T2I, NLP, or CV. With the lack of details or standards in T2V evaluation, it would have benefitted from an expansive understanding of human annotation as a whole. Thus, the ideas although well representative of the current state of evaluation in T2V, lacks the maturity, which affords the novelty or value, that hasn't been surveyed in other domains. 
* Potential bias in dynamic evaluation: The dynamic evaluation module, while innovative, may introduce bias by discarding certain comparisons. As noted in Table 3, comparisons involving Gen2 were frequently omitted. This could potentially lead to less accurate estimations for top-performing models. 
* Lack of error analysis: The paper doesn't provide a detailed analysis of cases where the protocol fails or produces inconsistent results. Understanding these edge cases could help improve the robustness of the evaluation method. This further dives into my understanding that although it is a great survey of existing protocols, lacks a contribution its aggregation. Further evaluation and consideration in other domains, would assure me that it is a standard that can be well accepted vs. one that is just more comprehensive than the past. 

Minor Weakness
* In line 3, there's a small typo where "pop3 ularity" is split across lines. It should be "popularity".
* In line 31, "practicabil32 ity" is split across lines. This appears to be due to line wrapping rather than an actual error in the text.
questions: questions: * Can you provide more details on how the dynamic evaluation module handles potential biases when discarding certain comparisons, especially for top-performing models like Gen2?
* How does the protocol ensure that the reduced number of annotations in the dynamic evaluation still captures a representative sample across different prompt categories and model capabilities?
* What steps were taken to validate that the training provided to LRAs effectively mitigates potential biases they may have, especially given their potential lack of diversity compared to crowdsourced annotators?
* How does the protocol account for potential cultural or linguistic biases in the evaluation of "Ethical Robustness"? Can you provide more details on how this metric is defined and measured?
* What considerations were made in designing the protocol to ensure its applicability across different types of T2V models (e.g., models specializing in different video styles or lengths)?
* Would the survey of non -T2V evaluation approaches be beneficial when merging with this technology? With the lack of maturity and rapidly evolving field, would not benefit from a cross-domain understanding from more mature fields?
limitations: limitations: Mostly adequately addressed limitations. Would like to see some understanding in regards to human annotation, than just T2V.
paper_summary: paper_summary: This manuscript presents a novel, standardized human evaluation protocol specifically tailored for Text-to-Video (T2V) models. The protocol encompasses a meticulously designed suite of evaluation metrics supplemented with robust annotator training resources. The effective deployment of Latent Action Ratings (LARs) is a notable feature of this work, enabling the acquisition of high-quality annotations. A dynamic evaluation component is also introduced, which remarkably curtails annotation costs by approximately 50% while maintaining the integrity of annotation quality. The paper further offers an exhaustive evaluation of cutting-edge T2V models. A commitment to open-source the entire evaluation process and associated code is a commendable initiative, which will undoubtedly empower the broader research community to assess novel models using updated data, building upon existing reviews.
strength_weakness: strength_weakness: — My biggest concern is that the evaluation metrics currently proposed by the authors cannot be proven reasonable or convincing. Although this manuscript has used several quality standards (4 objective and 2 subjective), are they complete or redundant? For example, I think "Video Quality" is a vast range, which may include multiple aspects such as aesthetics, clarity, and even another indicator, "Text Alignment" (because if the generated elements do not match the prompt, it will not be considered a good video); I think "Temporal Quality" and "Motion Quality" are not two completely independent metrics, they should be highly related and coupled.
— Furthermore, the authors mentioned that this paper adopts the Rao and Kupper model to quantify annotations results. I tried to follow this paper to understand how to quantify these metrics whose weights I am not sure about. Still, unfortunately, I found no similarities between it and this paper. I hope the authors can scientifically and clearly explain how the metrics are involved in the final results, which is one of the core contributions of this paper.
— In addition, since the annotated results of all metrics are labeled by manual A/B testing, is the human preference the only one that dominates? How can the aesthetics and evaluation systems of different annotators be consistent? The same problem arises in the reward model, where its approach has been verified in the LLM field and cannot achieve high accuracy (only about 70+%). Thus, it needs to be used in conjunction with PPO/DPO. As this manuscript constructs datasets this way,  how can we avoid the problem?
— Finally, although the dynamic evaluation component proposed in this article reduces annotation costs to approximately half,  the annotation task is too dependent on manual work; I think its promotion and ease of use in the open-source community are still questionable.
questions: questions: I acknowledge that I appreciate the authors' contributions to the open source community, and reasonable and practical answers to the concerns in the "Weaknesses" section will help improve my rating.
limitations: limitations: This manuscript has discussed the relevant limitations, please refer to the "Weaknesses" section for more details.
paper_summary: paper_summary: This paper introduces the Text-to-Video Human Evaluation (T2VHE) protocol, a standardized approach for assessing the quality of videos generated from text. 
Besides, this paper comprehensively evaluates some famous text-to-video models with their proposed reliable human assessment, which is very exciting and valuable for this community.
strength_weakness: strength_weakness: 1. The algorithm’s performance and stability may vary depending on the specific prompt types and model characteristics.
2. The ELO rating [1] mechanism seems to be a competitive baseline for the Rao and Kupper method used in this paper, which has been widely used in the comparsion of LLMs. Could you please describe the difference between Rao and Kupper method and ELO rating, and predict which kinds of method is better?

[1] Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings (https://lmsys.org/blog/2023-05-03-arena)
questions: questions: 1. How does the performance of the evaluated models vary across different domains or types of prompts?
limitations: limitations: NONE

Comments: title: Looking forward to the reply
Thanks for your support!
We appreciate the support!
Looking forward to the reply
Looking forward to the reply
Looking forward to the reply
We appreciate the support!
Response to Authors
To save reviewer's time, we put a summary of rebuttal
To save reviewer's time, we put a summary of rebuttal
To save reviewer's time, we put a summary of rebuttal
To save reviewer's time, we put a summary of rebuttal
Looking forward to further discussion
Rebuttal (3/3)
Rebuttal (2/3)
Rebuttal (3/3)
Rebuttal (2/3)
Rebuttal (4/4)
Rebuttal (3/4)
Rebuttal (2/4)
Rebuttal (4/4)
Rebuttal (3/4)
Rebuttal (2/4)
comment: Dear Reviewer 2pJH,

We appreciate your constructive comments and have **responded in detail to the five weaknesses**, while we also **revised the article** to address the minor quibbles raised.

As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback.

Thanks again for the time and effort in our work!

Best wishes,

Authors
Thank you very much for your **support and for accepting** our work. We are pleased to hear that we have successfully addressed your concerns. Your constructive feedback has been invaluable in helping us improve our research.

If there are any further questions or if you require a more detailed explanation of any aspect of our work, we are more than willing to provide any additional information you may need.
Thanks. The authors have addressed most of my concerns, so I'll raise my rating.
Thanks very much for **supporting and accepting** our work. We are happy that we have addressed your concern. Your constructive feedback is very helpful for us in improving our work.

If there are any further questions, we are more than willing to provide a detailed explanation. Thank you!
I have updated the score, based on the rebuttal presented by the author.
Dear Reviewer 2pJH,

As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback.

Thanks again for the time and effort in our work!

Best wishes,

Authors
Dear Reviewer AEKZ,

As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback.

Thanks again for the time and effort in our work!

Best wishes,

Authors
Dear Reviewer RMCR,

As we draw closer to the rebuttal deadline, we would like to inquire if there are any additional questions or concerns about our work. We greatly value the feedback.

Thanks again for the time and effort in our work!

Best wishes,

Authors
Thanks very much for **supporting and accepting** our work. We are happy that we have addressed your concern. Your constructive feedback is very helpful for us in improving our work.

If there are any further questions, we are more than willing to provide a detailed explanation. Thank you!
Thank you for the detailed responses from authors. I have acknowledged the strengths of your wonderful paper and I will keep my score.
Dear Reviewer LQLU,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**.

1. \[**The algorithm’s performance may vary depending on the specific prompt types and model characteristics** \]: 

   **Response:** 
   - The effect of different prompts and model characteristics on the algorithm is limited, and experimental results demonstrate that this effect does not diminish the validity and reliability of the dynamic evaluation component at all.

2. \[**Compare Rao and Kupper method and ELO rating** \]: 

   **Response:** 
   - While the ELO rating system is competitive and has been successfully applied in many domains, given the nature of our task, which involves nuanced evaluations where ties are possible and even expected, the Rao and Kupper model is more suited to our needs. 

3. \[**How does the performance of the evaluated models vary across different domains or types of prompts** \]: 

   **Response:** 
   - We report the model scores and corresponding rankings obtained for the annotation results under the eight prompt categories.
   - The results show that different prompt categories do slightly affect the evaluation results of model performance, but this effect is not obvious for models with high performance, such as Gen2.


Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors
Dear Reviewer RMCR,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**.

1. \[**The evaluation metrics cannot be proven reasonable or convincing** \]: 

   **Response:** 
   - Through extensive research and rigorous analysis, we designed an assessment system that not only incorporates the mainstream metrics used in existing T2V studies but also complements the missing parts of previous studies, making our evaluation metrics reasonable and convincing.

2. \[**Explain how Rao and Kupper model quantifies annotations results** \]: 

   **Response:** 
   - We provide detailed explanation in Rebuttal (2/3) and detailed parameter estimation details in Appendix C.3.

3. \[**How to handle human preference** \]: 

   **Response:** 
   - Previous studies and our experimental results have shown that human preferences cannot and need not be eliminated during evaluation. 
   - By providing different guidelines for different types of metrics, our protocol achieves a balance between the pros and cons of human preferences.

4. \[**Dynamic evaluation component is too dependent on manual work** \]: 

   **Response:** 
   - Our dynamic components are proposed to improve the efficiency of the human assessment process that already requires manual work, without additional expense. At the same time, it can be used in other research fields with simple adjustments.


Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors
Dear Reviewer AEKZ,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**.

1. \[**Lacks an understanding of evaluation in other domains** \]: 

   **Response:** 
   - Our protocol is based on a thorough investigation of human evaluation and video generation articles in different fields and is experimentally validated to have the advantages of reproducibility, reliability, and practicability compared to traditional protocols.

2. \[**Dynamic evaluation module may introduce bias** \]: 

   **Response:** 
   - Theoretically, our protocol design allows for plausible estimation in the absence of numerous comparisons, and we further validate this capability through bootstrap confidence interval experiments..

3. \[**Lack detailed analysis of cases where the protocol fails. Need further evaluation and consideration in other domains** \]: 

   **Response:** 
   - We analyzed the case of producing inconsistent results under the traditional protocol setup, demonstrating the importance of the annotator training we designed, and conducted a reliability analysis of the dynamic evaluation component, which proved that the introduction of this component does not result in inconsistent protocol results.
   - Our design is not only to provide a more reliable, reproducible, and practical T2V assessment protocol but also to provide ideas and reference materials for human evaluation research in other fields..

4. \[**Small typo errors** \]: 

   **Response:** 
   - Thanks for the reminder. These typo errors are caused by automatic line breaks during latex compilation, not actual errors in the text editor.

5. \[**How the dynamic evaluation module handles potential biases** \]: 

   **Response:** 
   - From a methodological design perspective, the Rao and Kupper model we used can accurately estimate model scores using incomplete comparisons.
   - From an experimental validation perspective, our rigorous bootstrap confidence interval experiments confirm that even after discarding many comparisons on top models such as Gen2, the final estimate of the relative strength of the model is still accurate and credible.

6. \[**How to capture a representative sample** \]: 

   **Response:** 
   - The static evaluation in the protocol retains the samples that are most worthy of being annotated, ensuring that the evaluation results won't be biased by the lack of specific kinds of prompts and model combinations.


7. \[**How to handle potential biases and potential lack of diversity** \]: 

   **Response:** 
   - We minimize annotator bias in assessing objective metrics by providing detailed guidelines and training processes, and we retain annotator diversity by introducing subjective metrics..

8. \[**Provide more details on "Ethical Robustness"** \]: 

   **Response:** 
   - Our survey shows that while the measurement of the ethics of generated content is missing from existing T2V evaluation protocols. Therefore, after a thorough investigation of the relevant research, we use three reference angles, i.e., Toxicity, Fairness, and Bias, to define and measure the subjective metric of "Ethical Robustness".

9. \[**How to ensure its applicability across different types of T2V models** \]: 

   **Response:** 
   - In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that all videos can be displayed clearly. 
   - Large-scale literature research confirms the fairness of this approach, because any further processing may undermine the advantages of some models when the evaluation scope is very wide.

10. \[**Cross-domain understanding** \]: 

   **Response:** 
   - Our protocol builds on previous non-T2V evaluation studies and T2V model studies, and the proposed annotator training principles and dynamic evaluation components can also contribute to the development of non-T2V evaluation studies.


Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors
Dear Reviewer 2pJH,

Thanks so much again for the time and effort in our work. Considering the limited time available and to save the reviewer's time, we **summarized our responses here**.

1. \[**Value over prior approaches** \]: 

   **Response:** 
   - While automated scorers can implement evaluations at lower cost, their inherent poor flexibility, and high expansion costs result in the fact that they are still unable to fully replace human evaluation protocols.
   - The design of better human evaluation protocols is itself an important prerequisite for the training of more reliable automated scorers.

2. \[**Include more models or show human evaluations do not correlate FID or FVD** \]: 

   **Response:** 
   - We evaluate the five most advanced video generation works available at the start of the research. This amount is in line with other benchmarking works, and we will conduct more model evaluations later.
   - Since there are no corresponding reference videos for the prompt source, we cannot calculate the corresponding FID and FVD indicators, but we will supplement this when expanding our prompt source in the future.

3. \[**Citing and considering using ELO scores** \]: 

   **Response:** 
   - While the ELO rating system is powerful, our Rao and Kupper model better fits the requirements of our task due to its flexibility in handling ties.
   - We will also build and maintain a public, large-scale, long-term project in the future.

4. \[**Few details. How to best show videos with different characteristics** \]: 

   **Response:** 
   - In our response Rebuttal (3/4), we provided the locations of all the protocol details revealed and we've revised the article to include more detailed information in the body of the article.
   - We use the simplest scaling process to avoid biasing any model in comparison and guarantee that all videos can be displayed clearly. 
   - Large-scale literature research confirms the fairness of this approach, because any further processing may undermine the advantages of some models when the evaluation scope is very wide.

5. \[**Some things marked as objective are not objective** \]: 

   **Response:** 
   - The distinction between subjective and objective indicators stems from whether we can ask the annotator to evaluate according to predefined, accepted, and feasible criteria.
   - Experiments show that we can effectively reduce the influence of personal preferences on results through well-designed assessment protocols and detailed annotator training.

6. \[**Minor quibbles** \]: 

   **Response:** 
   - Thanks to your suggestion, we have revised the article to address the concerns raised.


Since the discussion stage is already halfway through, may I know if our rebuttal addresses the concerns? If there are further concerns or questions, we are more than happy to address them. Thanks again for taking the time to review our work and provide insightful comments.

Best Regards,

Authors
Thank you for confirming the reception of our rebuttals. We appreciate your attention to our submission.

Please let us know if you have any further queries or require additional information. Your guidance is invaluable to us, and we are eager to address any remaining concerns.

Best regards
To keep the authors updated: 

I confirm the reception of the rebuttals. I will follow up soon with any queries and/or updated scores to reflect my understanding.
Prompt Category - Human

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.44(1)       | 2.64(1)          | 2.14(1)        | 2.96(1)        | 3.66(1)             | 2.98(1)          |
| Pika          | 1.38(2)       | 1.29(2)          | 1.52(2)        | 0.99(3)        | 0.78(3)             | 1.23(2)          |
| Latte         | 0.59(5)       | 0.62(5)          | 0.75(5)        | 1.38(2)        | 0.96(2)             | 0.44(5)          |
| TF-T2V        | 0.94(4)       | 0.79(3)          | 0.91(3)        | 0.61(4)        | 0.37(4)             | 0.62(3)          |
| Videocrafter2 | 0.97(3)       | 0.64(4)          | 0.87(4)        | 0.45(5)        | 0.27(5)             | 0.55(4)          |


Prompt Category - Lifestyle

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.52(1)       | 2.11(1)          | 1.99(1)        | 2.69(1)        | 3.59(1)             | 2.63(1)          |
| Pika          | 1.14(2)       | 1.12(2)          | 1.22(2)        | 0.92(3)        | 0.78(3)             | 1.06(2)          |
| Latte         | 0.62(5)       | 0.89(4)          | 0.84(5)        | 1.30(2)        | 1.15(2)             | 0.82(3)          |
| TF-T2V        | 0.83(4)       | 0.80(5)          | 0.89(4)        | 0.72(4)        | 0.46(4)             | 0.65(4)          |
| Videocrafter2 | 0.87(3)       | 0.96(3)          | 1.01(3)        | 0.54(5)        | 0.29(5)             | 0.65(5)          |


Prompt Category - Plant

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.07(1)       | 2.22(1)          | 2.11(1)        | 2.99(1)        | 4.01(1)             | 3.02(1)          |
| Pika          | 1.05(3)       | 1.01(3)          | 1.03(3)        | 1.17(3)        | 0.85(3)             | 0.83(3)          |
| Latte         | 1.04(4)       | 0.89(4)          | 1.07(2)        | 1.42(2)        | 1.67(2)             | 0.99(2)          |
| TF-T2V        | 1.07(2)       | 1.04(2)          | 0.87(4)        | 0.78(4)        | 0.59(4)             | 0.70(4)          |
| Videocrafter2 | 0.71(5)       | 0.73(5)          | 0.78(5)        | 0.41(5)        | 0.22(5)             | 0.44(5)          |


Prompt Category - Scenery

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.30(1)       | 2.37(1)          | 2.21(1)        | 3.31(1)        | 3.66(1)             | 2.56(1)          |
| Pika          | 0.88(3)       | 0.97(2)          | 1.01(2)        | 0.78(3)        | 0.94(3)             | 0.98(3)          |
| Latte         | 0.88(4)       | 0.91(3)          | 0.81(5)        | 1.04(2)        | 1.37(2)             | 1.02(2)          |
| TF-T2V        | 1.08(2)       | 0.83(4)          | 0.95(3)        | 0.73(4)        | 0.50(4)             | 0.92(4)          |
| Videocrafter2 | 0.73(5)       | 0.74(5)          | 0.85(4)        | 0.38(5)        | 0.30(5)             | 0.56(5)          |


Prompt Category - Vehicles

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.37(1)       | 2.49(1)          | 2.50(1)        | 2.62(1)        | 3.50(1)             | 2.97(1)          |
| Pika          | 1.24(2)       | 1.32(2)          | 1.23(2)        | 0.98(3)        | 0.73(3)             | 0.96(2)          |
| Latte         | 0.73(5)       | 0.70(5)          | 0.86(3)        | 1.62(2)        | 1.09(2)             | 0.72(3)          |
| TF-T2V        | 0.77(4)       | 0.74(3)          | 0.67(5)        | 0.48(5)        | 0.47(4)             | 0.54(4)          |
| Videocrafter2 | 0.93(3)       | 0.74(4)          | 0.77(4)        | 0.62(4)        | 0.30(5)             | 0.49(5)          |
**Q1: How does the performance of the evaluated models vary across different domains or types of prompts?**

Thanks for the comments. 

We conducted experiments on the performance of the evaluated models that varied across different types of prompts. The results are shown in the tables below.

**Setting:**

- We input the human annotation results corresponding to the eight prompt categories into the Rao and Kupper model [2] to estimate and rank the model scores.

**Results:**

- We report the model scores and corresponding rankings obtained for the annotation results under the eight prompt categories.

- Some of the scores in the table are the same and the result of different rankings is due to the retention of two decimal places in the report.

**Analysis:**

- Our results show that using different prompt sources does lead to slight differences in evaluation results, whereas models with a clear advantage, such as Gen2, are unaffected and have the highest ranking regardless of which prompt is used.

- This result further highlights the importance of using class-wide prompts when using human evaluation protocols.



**Conclusion:**

- Our experimental results show that different prompt categories **do slightly affect the evaluation results** of model performance, but this effect is not obvious for models with high performance, such as Gen2.



**References:**

[1] Chatbot Arena.

[2] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204.

[3] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345. 

[4] Vbench: Comprehensive benchmark suite for video generative models.  CVPR 2024. 





Prompt Category - Animal

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.02(1)       | 2.15(1)          | 1.89(1)        | 2.40(1)        | 2.48(1)             | 3.38(1)          |
| Pika          | 1.18(2)       | 1.41(2)          | 1.52(2)        | 1.54(2)        | 1.04(2)             | 1.35(2)          |
| Latte         | 0.90(4)       | 0.98(3)          | 0.92(4)        | 1.29(3)        | 0.92(3)             | 1.01(3)          |
| TF-T2V        | 0.98(3)       | 0.93(4)          | 0.96(3)        | 0.99(4)        | 0.73(4)             | 0.52(4)          |
| Videocrafter2 | 0.84(5)       | 0.72(5)          | 0.86(5)        | 0.53(5)        | 0.69(5)             | 0.35(5)          |


Prompt Category - Architecture

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.12(1)       | 1.86(1)          | 1.95(1)        | 2.49(1)        | 3.45(1)             | 2.54(1)          |
| Pika          | 1.00(3)       | 1.46(2)          | 1.13(2)        | 0.92(3)        | 0.59(3)             | 1.05(2)          |
| Latte         | 0.81(5)       | 1.07(4)          | 0.93(4)        | 1.28(2)        | 1.30(2)             | 0.95(3)          |
| TF-T2V        | 0.82(4)       | 1.10(3)          | 0.94(3)        | 0.85(4)        | 0.50(4)             | 0.72(4)          |
| Videocrafter2 | 1.09(2)       | 0.73(5)          | 0.89(5)        | 0.62(5)        | 0.25(5)             | 0.62(5)          |


Prompt Category - Food

| |Video Quality|Temporal Quality|Motion Quality|Text Alignment|Ethical Roubustness|Human Preference|
|-|-|-|-|-|-|-|
| Gen2          | 2.34(1)       | 2.39(1)          | 2.00(1)        | 2.87(1)        | 3.45(1)             | 3.04(1)          |
| Pika          | 0.99(2)       | 1.03(2)          | 1.16(2)        | 0.94(3)        | 0.59(3)             | 0.82(3)          |
| Latte         | 0.89(3)       | 0.92(3)          | 0.83(5)        | 1.57(2)        | 1.30(2)             | 1.42(2)          |
| TF-T2V        | 0.79(5)       | 0.74(4)          | 1.04(3)        | 0.51(4)        | 0.50(4)             | 0.47(4)          |
| Videocrafter2 | 0.80(4)       | 0.74(5)          | 0.87(4)        | 0.47(5)        | 0.25(5)             | 0.35(5)          |
**W3:  Is the human preference the only one that dominates? How can the aesthetics and evaluation systems of different annotators be consistent? How can we avoid the problem?**

Thanks for the comments. We analyze this from the following perspectives.

**Analysis:**

- We observed similar problems in human evaluation studies in the T2I and NLG communities [8,9,12], as shown in the footnotes of the article, some NLG studies [11] achieve an IAA of only 0.14, i.e., even if constructing sufficiently objective protocols and using the most professional annotators, one **cannot avoid** the biases brought by human preferences.

- At the same time, however, it has also been shown that **the existence of human preferences is a vital part of human assessment** [9], i.e., a good protocol should both effectively reduce bias due to preferences and make reservations about this, as too much agreement (e.g., IAA greater than 0.7) may suggest that there is a lack of diversity among annotators.

- This is one of the core reasons our protocol distinguishes between subjective and objective metrics. For **objective** metrics, we minimize the influence of personal preferences through detailed metric definitions and annotator training, whereas for **subjective** metrics, we encourage annotators to evaluate based on their preferences. Unlike objective metrics, subjective metrics are difficult to be replaced by automated metrics and reflect the diversity of annotators.

**Conclusion:**

- Previous studies and our experimental results have shown that **human preferences cannot and need not be eliminated** during evaluation. By providing different guidelines for different types of metrics, our protocol achieves a balance between the pros and cons of human preferences.

**W4: The annotation task with the dynamic evaluation component is too dependent on manual work, its promotion and ease of use in the open-source community are still questionable.**

Thanks for the comments. We analyze this from the following perspectives.

**Analysis:**

- Our dynamic evaluation component was proposed to improve human evaluation protocols, i.e., this method should be compared to a traditional human evaluation protocol that **also relies on manual work**, and uses **all the video pairs**. Our component can reduce the annotation cost without additional intervention during the annotation phase, which is important for popularizing human assessment.

- For example, a researcher needs to recruit five annotators, each of whom can only evaluate 200 sets of video pairs; using the same budget, employing our dynamic component would allow the researcher to recruit 10 or more people to execute evaluation, as the workload for each individual is only about half of what it would otherwise be. Not only does this allow for a more varied and reliable set of annotated results, but also as the evaluation size increases, the effect of the component becomes more pronounced.

- In addition, **this component is not limited to the T2V research field**, for the T2I and NLG research fields, by simply replacing the automated indicators in the pre-preparation phase, one can directly use this component in the customized protocol framework. We provide the detailed flow of the component operation in Appendix C.2 and will open-source all the related code. We also hope and encourage researchers from different fields to conduct more comprehensive human assessment protocols at a lower cost.


**Conclusion:**

- Our dynamic components are proposed to improve the efficiency of the human assessment process that **already requires manual work**, without additional expense. At the same time, it can be used in other research fields with **simple adjustments**.



**References:**

[1] Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models. 

[2] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024.

[3] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models. CVPR 2024. 

[4] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024. 

[5] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI. TBench 2024. 

[6] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NIPS 2023. 

[7] Holistic Evaluation of Text-to-Image Models. NIPS 2023.

[8] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023. 
[9] Evaluation of Text Generation: A Survey.

[10] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204. 

[11] The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation. EMNLP 2021. 

[12] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021.
**W2: Explain how Rao and Kupper model quantifies annotations results.**

We appreciate your feedback on our description of the Rao and Kupper model [10]. We understand the importance of clarity and detail in explaining our methods. Here’s a more comprehensive overview of the model.

**Analysis:**

- **Probabilistic Model Overview:**

  - **Model Definition:** The Rao and Kupper model is a probabilistic framework used for pairwise comparisons, allowing us to estimate scores for each model and rank them accordingly.
  - **Maximum Likelihood Estimation:** We employ maximum likelihood estimation to derive the scores, which provides a statistical basis for the rankings.
- **Detailed Explanation:**

  - **Comparison Process:** Consider a scenario with $t$ models. For any pair of models $i$ and $j$, the notation $i \succ j$ indicates $i$ beating $j$, $i \prec j$ indicates $i$ losing to $j$, and $i \sim j$ indicates a tie. The probabilities of these events are defined as follows:
    $$
     \begin{align}
     P(i \succ j) &= \frac{p_i}{p_i + \theta p_j},
     P(i \sim j) &= \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)},
     \end{align}
    $$
     where $p_i$ and $p_j$ represent the positive real-valued scores of models $i$ and $j$. $\theta$ is a parameter indicating the likelihood of a tie between closely matched models.

   - **Estimation Method:** The likelihood function for our model is constructed as:
     $$
     \begin{equation}
     L(p, \theta) = \prod_{i = 1}^{t} \prod_{j = i + 1}^{t} \left( \frac{p_i}{p_i + \theta p_j} \right)^{n_{ij}} \left( \frac{p_j}{\theta p_i + p_j} \right)^{n_{ji}} \left( \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)} \right)^{\tilde{n}_{ij}},
     \end{equation}
     $$

     and the corresponding log-likelihood function is:

     $$
     \begin{equation}
     l(p, \theta) = \sum_{i = 1}^{t} \sum_{j = i + 1}^{t} \left( n_{ij} \log \frac{p_i}{p_i + \theta p_j} + n_{ji} \log \frac{p_j}{\theta p_i + p_j} + \tilde{n}_{ij} \log \frac{p_i p_j (\theta^2 -1)}{(p_i + \theta p_j)(\theta p_i + p_j)} \right),
     \end{equation}
     $$
     where $p \in \mathbb{R}^t$ is a vector of scores $(p_1, \dots, p_t)^T$. We estimate $p$ and $\theta$ by maximizing the log-likelihood function numerically.

- **Application and Results:**  
	- **Model Ranking:** After estimating the scores, models are ranked based on these scores, reflecting their relative strengths. Our detailed explanation in Appendix C.3 covers the model fitting process and the derivation of scores and rankings.


**Conclusion:**

- We hope this detailed explanation clarifies our methodology and addresses any concerns about the application of the Rao and Kupper model in our research. We also provide detailed parameter estimation details in Appendix C.3.
**Q6: Would the survey of non-T2V evaluation approaches be beneficial when merging with this technology? With the lack of maturity and rapidly evolving field, would not benefit from a cross-domain understanding from more mature fields?**

Thank you for your insightful comment. We analyze it from the following perspectives.

**Analysis:**

- The guidelines and training proposed in this study can **be easily extended** to the T2I and NLG community studies with only user interface modifications, and similarly, by modifying the automated metrics in the dynamic module, one can also achieve efficient human evaluation of other types of protocols.
- This study drew on some well-established T2I and NLG human assessment studies [2,16] to construct a more comprehensive protocol in terms of some important principles, such as ensuring clarity of definitions and reliability of annotated results, as well as the design of the subjective indicators, which are lacking in existing T2V assessment studies. However, the design of specific indicators, such as the four objective indicators, was derived from specific T2V research needs.
- We also plan to design prompt datasets specifically for evaluating T2V models based on the principles of dataset construction in T2I and NLG fields in the future.

**Conclusion:**

- Our protocol builds on previous non-T2V evaluation studies and T2V model studies, and the proposed annotator training principles and dynamic evaluation components can also contribute to the development of non-T2V evaluation studies.

**References:**

[1] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021.

[2] Evaluation of Text Generation: A Survey.

[3] Towards A Better Metric for Text-to-Video Generation.

[4] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204.

[5] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345.

[6] Holistic Evaluation of Text-to-Image Models. NIPS 2023.

[7] How well can Text-to-Image Generative Models understand Ethical Natural Language Interventions? EMNLP 2022.

[8] Adopting and expanding ethical principles for generative artificial intelligence from military to healthcare. NPJ Digital Medicine 6, Article number: 225 (2023).

[9] ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation.

[10] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024.

[11] Seer: Language Instructed Video Prediction with Latent Diffusion Models. ICLR 2024.

[12] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024.

[13] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI.

[14] VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. TBench 2024.

[15] OpenAI Sora.

[16] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023.
**Q4: How does the protocol account for potential biases in the evaluation of "Ethical Robustness"? Can you provide more details on how this metric is defined and measured?**

Thank you for your insightful comment. We analyze it from the following perspectives.

**Analysis:**

- Our survey shows that while the measurement of the ethics of generated content is an important evaluation metric for generative models [6], it is missing from existing T2V evaluation protocols. Therefore, after a thorough investigation of the relevant research [7,8], we defined the corresponding reference angles according to the following criteria.
- **Toxicity**. Evaluating toxicity in video generation models is crucial for ensuring user safety, compliance with community standards, and protecting brand reputation.
- **Fairness.** Assessing fairness in video content ensures a diverse and fair representation of characters and subjects across different social dimensions, preventing the perpetuation of stereotypes and marginalization.
- **Bias.** Evaluating bias in video generation models is essential for ensuring content accuracy, diversity of perspectives, and avoiding harm to specific groups. 

**Conclusion:**

- After studying numerous relevant literature, we use three reference angles, i.e., Toxicity, Fairness, and Bias, to define and measure the subjective metric of "Ethical Robustness".


**Q5: How to ensure its applicability across different types of T2V models?**

Thank you for your insightful comment. Regarding the preprocessing problem of different types of videos, we further analyze this from the following perspectives.

**Analysis:**

- **Discussion on the preprocessing of different types of video.** We reviewed over 200 video generation literature from a variety of fields. It is certain that the vast majority of articles, at least in the field of T2V, **do not further preprocess** the video to be evaluated [3,12,13], as further processing would have an impact on some of the characteristics of the original video, while only simple scaling would instead better preserve and evaluate the **model's original capabilities**. This point can be further illustrated with a few examples below:

  - For models that aim to produce high frame rate video [9], it is **not fair** to sample a uniform frame rate during the evaluation process, as this approach undermines the strength of the model.
  - For studies specializing in the generation of long videos [10], it is also **unfair** to extract frames or cut short pieces of the generated results to compare with short videos, because there is no guarantee that the displayed results can reflect the original advantages of the study.
  - For articles emphasizing text alignment [11], if the original video is cropped or watermarks are added, the original visible content may be cropped or blocked, which is also an **unfair** comparison.

- **Video comparison of different lengths.** We do not apply any extra processing to the length of the video, and annotators are asked to view the video in its entirety before annotating it, thus ensuring a fair and reliable comparison. We also experimented with comparing VideoCrafter [14] with Sora [15], which proved that even with video length editing, frame rate processing, etc., the annotators all agreed that Sora was significantly better than VideoCrafter on all dimensions.

**Conclusion:**

- In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that **all videos can be displayed clearly**. Large-scale literature research confirms the fairness of this approach, because any further processing may **undermine** the advantages of some models when the evaluation scope is very wide.
**Q1: More details on how the dynamic evaluation module handles potential biases.**

Thank you for your question regarding the dynamic evaluation module. We have designed our method with robustness and accuracy in mind. Here's how we address potential biases:

**Analysis:**

- **Robustness Against Incomplete Comparisons:**  It is well-documented that the Bradley-Terry model [5] is robust against incomplete comparison data. Our approach, based on the Rao and Kupper model [4], **inherits this robustness and is specifically designed to handle scenarios with missing or incomplete data**. This property ensures that our ranking estimates remain reliable even when some comparisons are discarded.
- **Handling Top-Performing Models (e.g., Gen2):**
  - **Bootstrap Confidence Intervals:** To mitigate potential biases in ranking models with incomplete data, we employ bootstrap confidence intervals for the score estimates of each model across various metrics. Section 5 and Figure 3 of our paper demonstrate that even in extreme cases, our rank estimation is consistent and trustworthy.
  - **Ensuring Fairness:** By using bootstrap methods, we can quantify the uncertainty around our model rankings and ensure that our evaluation is fair and unbiased, even when certain comparisons are omitted.
- **Transparency and Detail:**  We provide a detailed explanation of our dynamic evaluation module and its bias-handling mechanisms in the supplementary materials, ensuring that our methodology is transparent and replicable.

**Conclusions:**

- From a methodological design perspective, the Rao and Kupper model we used can **accurately estimate model scores using incomplete comparisons**.
- From an experimental validation perspective, we conducted rigorous bootstrap confidence interval experiments, the results of which confirm that even after discarding many comparisons on top models such as Gen2, the final estimate of the relative strength of the model is **still accurate and credible**.

**Q2: How to ensure that the reduced number of annotations still captures a representative sample?**

Thank you for your insightful comment. We analyze it from the following perspectives.

**Analysis:**

- Before the dynamic evaluation begins, the protocol undergoes a static evaluation, in which the annotator prioritizes a group of samples that are worthy of being annotated. These video pairs contain the **full range of prompt categories and model combinations**, as shown in Figure 4, and have little difference in the scores of the automated indicators, i.e., they are **the most representative samples** according to the reviewers.
- As these samples won't be discarded during the static evaluation, even if subsequent evaluations discard many comparisons, the final evaluation results can include all categories of prompts and the most representative comparisons.

**Conclusion:**

- The static evaluation in the protocol **retains the samples that are most worthy of being annotated**, ensuring that the evaluation results **won't be biased** by the lack of specific kinds of prompts and model combinations.

**Q3: How to validate that the training provided to LRAs effectively mitigates potential biases and their potential lack of diversity?**

Thank you for your insightful comment. We designed detailed guidelines and training processes to address this bias,  as following points:

**Analysis:**

- To help annotators understand the definition of each metric, we provide multiple **reference perspectives** for each indicator. For each reference perspective, we provide **detailed examples** including a set of reference videos, a description of the analysis process, and corresponding conclusions, so the annotators from different sources can evaluate with the same mode of thinking. The specific definition and training process are shown in Table 4, Figure 1 (c), and Figures 5 to 9, respectively.
- As shown in Tables 1 and 2, the consistency between the trained annotators and the crowdsourced annotators is significantly improved, i.e., the metric examples and the training process can help eliminate the inconsistency caused by the potential bias.
- Although the lack of LRA's diversity cannot be completely avoided [2], by distinguishing between subjective and objective indicators, we can **preserve the diversity of annotators** as much as possible during the evaluation process.

**Conclusion:**

- We minimize annotator bias in assessing objective metrics by providing **detailed guidelines and training processes**, and we retain annotator diversity by introducing **subjective metrics**.
**W5: Explain clearly what the training looks like for aesthetic quality.**

Thank you for your insightful comment. We analyze it from the following perspectives.

**Analysis:**

- We provide a **detailed display** of the training content in Figure 1 (c) and Figures 5 to 9. Specifically, for each reference angle, we provide a set of **reference videos** with the **corresponding analysis process**. For your reference, a detailed description of the "Aesthetic Appeal" and corresponding training examples can be found in Table 4 and Figure 1 (c).
- **Objective and subjective metrics are relative.** Our division between subjective and objective metrics stems mainly from the fact that for subjective metrics, such as human preferences, we cannot find a universally recognized and workable description for annotators to adhere to. Whereas for objective metrics, such as "Aesthetic Appeal", several studies have rigorously defined these [2,18], we can design effective training examples based on previous research and ask the annotators to try to be objective in their judgments.
- **Annotators are different from automatic scorers**. While they can understand the concept of more abstract indicators, **no amount of detailed guidance is enough** to get them rid of their subjective influence [7,16,17]. However, this effect can be reduced through annotator training. As shown in Table 1, there is better consistency in the annotators after the training.

**Conclusions:**

- The distinction between subjective and objective indicators stems from whether we can ask the annotator to evaluate according to predefined, accepted, and feasible criteria.
- Although it is impossible to achieve a completely objective human assessment, experiments show that we can **effectively reduce the influence** of personal preferences on results through well-designed assessment protocols and detailed annotator training.


**M1: Being clear that “higher is better” in many plots.**

Thank you for your comment, we have added that in the table captions.

**M2: Using a lot of acronyms may impede readability in various parts of the paper.**

Thanks to your suggestion, we have reduced using acronyms in the text, particularly in section 4.3.

**M3: Disagree with the characterization of FVD being about temporal consistency.**

Thank you for your comments, we have amended the description of the FVD indicator within the related work section.

**M4:Recommend seeing/citing the Videopoet paper.**

As the survey covers video generation work in many areas such as T2V, video editing, frame prediction, and motion control, as well as human evaluation work in related areas such as T2I, NLG, and so on, we apologize for neglecting such excellent work as Videopoet [19] in our research. Thanks for the reminder, we've read the work carefully and cited it in the revised article.

**References:**

[1] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models. CVPR 2024.

[2] VBench: Comprehensive Benchmark Suite for Video Generative Models. CVPR 2024.

[3] Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning. CVPR 2022.

[4] Towards A Better Metric for Text-to-Video Generation.

[5] AIGCBench: Comprehensive Evaluation of Image-to-Video Content Generated by AI. TBench 2024.

[6] FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation. NIPS 2023.

[7] Toward Verifiable and Reproducible Human Evaluation for Text-to-Image Generation. CVPR 2023.

[8] Rank analysis of incomplete block designs: I. The method of paired comparisons. _Biometrika_, _39_(3/4), 324-345.

[9] Ties in paired-comparison experiments: A generalization of the Bradley-Terry model. _Journal of the American Statistical Association_, _62_(317), 194-204.
[10] Chatbot Arena.

[11] ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video Generation.

[12] ControlVideo: Training-free Controllable Text-to-Video Generation. ICLR 2024.

[13] Seer: Language Instructed Video Prediction with Latent Diffusion Models. ICLR 2024.

[14] VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models. TBench 2024.

[15] OpenAI Sora.

[16] All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text. IJCNLP 2021.

[17] Evaluation of Text Generation: A Survey.

[18] Holistic Evaluation of Text-to-Image Models. NIPS 2023.

[19] VideoPoet: A Large Language Model for Zero-Shot Video Generation. ICML 2024.
**W4: Few details are revealed in this paper. Standardizing the height of the video across models may not be the right thing to do. How to best show videos that are generated at different framerates and even different lengths.**

Thank you for your insightful comment. We provide full details of the protocol in the article and **Appendix**. For your ease of reference, we provide **the location of all details** in the table below.

|| **Evaluation metrics**|**Scoring methods**|**Evaluator requirements**|**Dynamic component**|
|-|-|-|-|-|
| **Position** | Section 4.1, Line 177-197$ $     Appendix C.4, Line 829, Table 4 | Section 4.2, Line 199-219$ $     Appendix C.3, Line 787-820 | Section 4.3, Line 221-253 | Section 4.4, Line 254-271$ $      Appendix C.2, Line 737-786 |

|| **Annotator training** |**Annotation interface** | **Details of the models evaluated**| **Details of the videos evaluated** |
|-|-|-|-|-|
| **Position** | Section 4.3, Line 230-234 $ $     Figure 1 (c) and Figures 5 to 9 | Section 1, Line 41-45$ $     Figure 1 (b) | Section 5, Line 273-276$ $    Appendix C.1, Line 727-736 | Section 5.1, Line 278-283 |

We've revised the article to include more detailed information in the body of the article and will **open source** all the relevant interfaces and code. In addition, we are willing to provide **any information** related to the protocol to help the community.

Regarding the preprocessing problem of different types of videos, we further analyze this from the following perspectives.

**Analysis:**

- First of all, even if VideoPoet [19] produces videos in portrait aspect ratios, its characteristics such as clarity in the evaluation interface **won't be affected**.
- Specifically, T2VHE take the highest video among the videos to be evaluated as the video height displayed in the annotation interface, and zoom out of the videos smaller than this height, so as to guarantee that **every video can be displayed clearly**. We will add an explanation of this issue in Section 5.1 to avoid this misunderstanding.
- **Discussion on the preprocessing of different types of video.** We reviewed over 200 video generation literature from a variety of fields. The vast majority of articles, at least in the field of T2V, **do not further preprocess** the video to be evaluated [2,4,5], as further processing would have an impact on some of the characteristics of the original video, while only simple scaling would instead better preserve and evaluate the **model's original capabilities**. This point can be further illustrated with a few examples below:

  - For models that aim to produce high frame rate video [11], it is **not fair** to sample a uniform frame rate during the evaluation process, as this approach undermines the strength of the model.
  - For studies specializing in the generation of long videos [12], it is also **unfair** to extract frames or cut short pieces of the generated results to compare with short videos, because there is no guarantee that the displayed results can reflect the original advantages of the study.
  - For articles emphasizing text alignment [13], if the original video is cropped or watermarks are added, the original visible content may be cropped or blocked, which is also an **unfair** comparison.
    It must be admitted that the preprocessing methods used in the various articles may be fairer for  particular comparisons, but to fit the protocol into the extensive T2V studies, simply scaling the video is a viable approach for human evaluation protocols.
- **Video comparison of different lengths.** We do not apply any extra processing to the length of the video, and annotators are asked to view the video in its entirety before annotating it, thus ensuring a fair and reliable comparison. We also experimented with comparing VideoCrafter [14] with Sora [15], which proved that even with video length editing, frame rate processing, etc., the annotators all agreed that Sora was significantly better than VideoCrafter on all dimensions.

**Conclusion:**

- In order not to bias any model in comparison, we use the simplest scaling process, which only guarantees that **all videos can be displayed clearly**. Large-scale literature research confirms the fairness of this approach, because any further processing may **undermine** the advantages of some models when the evaluation scope is very wide.
**W3: Citing and considering using ELO scores.**

Thank you for your insightful comment. We recognize the significance of the ELO rating system and its broad applications in model comparisons. Below, we detail our response and the reasons for choosing the Rao and Kupper model.

**Analysis:**

- **ELO Rating as a Bradley-Terry Special Case:** The ELO rating system can be viewed as a special case of the Bradley-Terry model [8]. It estimates object scores based on the outcomes of pairwise comparisons, and the resulting calculated scores are fed into the sampling algorithm to determine subsequent competition model pairs [10].
- **Ties and Dynamic Selection:** While the ELO rating system provides a robust framework for scoring and ranking, it doesn't inherently account for ties, which can occur frequently in our dataset. In contrast, the Rao and Kupper model [9], which we adopted, is a modification of the Bradley-Terry model that accommodates tied comparison results. This capability is crucial for our specific use case, where annotators may not always have a clear preference.
- **Rao and Kupper Model Advantages.**  (1) **Handling Ties:** The Rao and Kupper model's ability to handle tied outcomes allows for more precise scoring and ranking when annotators express equal preference between models. (2) **Dynamic Pair Selection:** Although both models support dynamic pair selection, the Rao and Kupper model's handling of ties makes it more suitable for scenarios where comparisons may not always result in a decisive winner.
- **Consideration for Future Work:**  We acknowledge the benefits of the ELO system's dynamic selection algorithm and will consider incorporating these insights into our future research to enhance our model ranking process. We will also add discussions and references to similar content in our article revisions.

**Conclusions:**

- While the ELO rating system is powerful, the Rao and Kupper model **better fits the requirements of our task** due to its flexibility in handling ties.
- The main purpose of our work is to **design reliable, reproducible, and practical human evaluation protocol for T2V models**, but we will also build and maintain a public, large-scale, long-term project in the future.