Can LVLMs Describe Videos like Humans? A Five-in-One Video Annotations Benchmark for Better Human-Machine Comparison

Authors: Shiyu Hu, Xuchen Li, Xuzhao Li, Jing Zhang, Yipei Wang, Xin Zhao, Kang Hao Cheong
Primary area: datasets and benchmarks
Keywords: Video Caption, Video Understanding, LVLM Evaluation, Human-machine Comparison

Abstract:
Large vision-language models (LVLMs) have made significant strides in addressing complex video tasks, sparking researchers' interest in their human-like multimodal understanding capabilities. Video description serves as a fundamental task for evaluating video comprehension, necessitating a deep understanding of spatial and temporal dynamics, which presents challenges for both humans and machines. Thus, investigating whether LVLMs can describe videos as comprehensively as humans—through reasonable human-machine comparisons using video captioning as a proxy task—will enhance our understanding and application of these models. However, current benchmarks for video comprehension have notable limitations, including short video durations, brief annotations, and reliance on a single annotator's perspective. These factors hinder a comprehensive assessment of LVLMs' ability to understand complex, lengthy videos and prevent the establishment of a robust human baseline that accurately reflects human video comprehension capabilities. To address these issues, we propose a novel benchmark, FIOVA (Five In One Video Annotations), designed to evaluate the differences between LVLMs and human understanding more comprehensively. FIOVA includes 3,002 long video sequences (averaging 33.6 seconds) that cover diverse scenarios with complex spatiotemporal relationships. Each video is annotated by five distinct annotators, capturing a wide range of perspectives and resulting in captions that are 4 to 15 times longer than most existing benchmarks, thereby establishing a robust baseline that represents human understanding comprehensively for the first time in video description tasks. Using the FIOVA benchmark, we conducted an in-depth evaluation of six state-of-the-art (SOTA) LVLMs, comparing their performance with humans. To enhance this evaluation, we proposed FIOVA-DQ, a novel event-based metric that incorporates weighted event importance derived from human annotations. Results show that while current LVLMs demonstrate some perception and reasoning capabilities, they still struggle with information omission and descriptive depth. Moreover, we found significant discrepancies between LVLMs and humans in complex videos, particularly where human annotators exhibited substantial disagreement, whereas LVLMs tended to rely on uniform strategies for challenging content. These findings underscore the limitations of using a single human annotator as the groundtruth for evaluation and highlight the need for new evaluation perspectives. We believe this work offers valuable insights into the differences between LVLMs and humans, ultimately guiding future advancements toward human-level video comprehension.

Decision: reject
Reviewer scores: 6, 5, 3, 6

Reviews and discussion:
<thread>
creation date:
1730495478064

thread_type:
official_review

<thread_object>
creator: reviewer_pxyn

<content>
summary:
The authors of the paper investigate whether large vision-language models (LVLMs) can describe videos as comprehensively as humans. To facilitate the investigation, the authors propose a benchmark, FIOVA, designed to evaluate the differences between captions from LVLMs and humans, with 3,002 videos (averaging 33.6 seconds) that cover diverse scenarios.

Using the FIOVA benchmark, the authors conducted an in-depth evaluation of six state-of-the-art LVLMs (VideoLLaMA2, LLaVANEXT-Video, Video-LLaVA, VideoChat2, Tarsier, and ShareGPT4Video), comparing their performance with humans.

strengths:
1. FIOVA is unique and important for advancing video-language models further. Each video in FIOVA is annotated by five distinct annotators, establishing a robust baseline that, for the first time, comprehensively represents human understanding in video description tasks.

2. In-depth evaluations of six open-source SOTA models were performed, and details of the algorithms and implementations were provided.

3. Many useful insights are provided through fine-grained evaluation and analysis. For example:
- Current LVLMs still struggle with information omission and descriptive depth.
- Different LVLMs employ varying strategies for video captioning—some prioritize completeness, while others focus on accuracy.
- For videos that are relatively easy to describe, the models show significant variability in performance. In contrast, for more challenging videos, their performance becomes more consistent.
- Significant discrepancies exist between LVLMs and humans in complex videos, particularly where human annotators exhibit substantial disagreement, whereas LVLMs tend to rely on uniform strategies for challenging content.

weaknesses:
LLM hallucination and detail omission issues may be present in the ground truth description of each video, as the ground truth for each video is generated by an LLM, GPT-3.5-turbo, which synthesizes the five human-provided descriptions into a single, comprehensive video description.

Additionally, using an LLM instead of a VLM to summarize the five human captions is insufficient because an LLM cannot properly handle conflicting information in the five human captions. For example, in Figure 4, Human3 notes that the little boy cries at the end, while Human5 states that the boy smiles at the end. Since an LLM cannot 'see' the video, it may simply guess that the boy smiles at the end.

questions:
1. Were the same five distinct annotators used to annotate all 3,002 videos? I assume the answer is 'No.'

2. What are the sources of the videos? 

3. How do you collect and process the videos in order to obtain and control the resulting varied video complexity?

4. How does the GPT-summarized caption affect the distributions shown in Figure 3?

5. Line 306 states that each model was fine-tuned for video caption generation. Do you mean you further fine-tuned these models? On which dataset was the fine-tuning performed?

6. When the SOTA models were evaluated, 8 frames per video were used. Were 8 frames sufficient for FIOVA? Did the human annotators watch the actual video files, or were they presented with sampled frames?

7. AutoCQ only provides event-level evaluation. Why were the five dimensions (e.g., consistency, context, etc.) from Sec. 2.2 not considered in the model response evaluation metrics?

8. It appears that the comparison of an LVLM against humans is conducted by comparing the LVLM’s response with the GPT-synthesized human caption summary in the main paper. However, in the image-language domain, having multiple human annotators for each image is not new (e.g., COCO captions include 5 human captions per image), and researchers often compare a model’s caption with each human caption individually, then aggregate the metric values (e.g., averaging the 5 metric values obtained by comparing with each human caption). Is there a reason for not taking this approach?

9. In Table A.3 in the Appendix, comparing the model's response with the GPT-synthesized human caption summary tends to produce higher metric values than directly comparing the model's response with a single human caption. Do the authors have any comments or insights regarding this observation?

Minor comments:
In the qualitative results shown in Figures A12–A17, it would be helpful to highlight any hallucinated content in the model responses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730621809602

thread_type:
official_review

<thread_object>
creator: reviewer_ca5f

<content>
summary:
The paper proposes a new benchmark called FIOVA to evaluate LVLMs in their ability to describe videos with a depth and breadth comparable to human understanding.  FIOVA includes 3,002 long video sequences averaging 33.6 seconds, each annotated by five different human to provide multi-perspective detailed descriptions. The benchmark analyzes the quality and variability of human annotations and compares the performance of several LVLMs. on FIOVA, revealing their strengths and weaknesses. Overall, FIOVA provides a more rigorous and comprehensive framework for assessing the differences in video description capabilities between LVLMs and humans. The benchmark addresses the limitations of existing video captioning datasets, which often feature short video durations, brief annotations, and reliance on a single annotator's perspective. These limitations hinder the assessment of LVLMs' performance on complex, long-duration videos and the establishment of a robust human baseline.

strengths:
1. The paper collects a new benchmark with richer video descriptions, focusing on long-duration videos with complex spatiotemporal relationships, providing a more realistic and challenging test case for LVLMs. By consistency, context, correctness, detailed orientation, temporality, length, CV and other dimensions and grouping videos according to difficulty, observe the video caption capabilities of different models, showing the advantages and disadvantages of the model, which is more comprehensive than the previous benchmark.
2. The results are clearly presented and supported by quantitative metrics, providing a comprehensive comparison of model performance. A variety of evaluation methods are also introduced, and a baseline assessment of human annotations is conducted. This approach helps to understand the diversity and consistency of human annotations, which is essential for establishing a reliable human baseline. The discussion section provides an in-depth analysis of the findings and their implications. These conclusions are well supported by the results and provide direction for future research.

weaknesses:
1. The evaluation model has limitations. This article evaluates some selected LVLMs, but does not explore broader models, such as the business model Gemini-1.5-Pro, which has a strong understanding of long videos.
2. There are doubts about the collection of groundtruth in FIOVA. FIOVA carefully designed manual annotations composed of five human annotator annotations, and merged and rewrote human annotations with GPT-3.5-Turbo. However, since GPT-3.5-Turbo cannot directly see the video, induction based on human text order alone can easily bring errors such as illusions to groundtruth. As in Figure 4, the actions of the little boy riding a bicycle are described twice in the text, including the actions on the ground are repeated twice, which is inconsistent with the word order of normal human speech. Without the video frames in Figure 4, it is easy to lead to misunderstandings. At the same time, there is no verification for behaviors such as smiling and pointing at the camera, and it is uncertain whether there will be new errors. The results of relevant experiments performed on the benchmark of video descriptions with errors are not necessarily reliable.
3. This paper provides an overview of performance metrics, but lacks detailed error analysis to explain the types of errors made by LVLM and the reasons behind them. The authors should build on the proposed benchmark with a more fine grained error analysis and explore potential causes. This will provide valuable insights for improving the model and the benchmark itself. At present, both traditional metrics (e.g. METEOR, BLEU) and automated measurement methods that rely on GPT models (e.g. AutoDQ) have limitations. I hope the authors can conduct further research on metrics.

questions:
1. Ensure the accuracy of the ground truth of FIOVA, and ensure that the evaluation metrics can truly reflect the model's capabilities based on the ground truth.
2. Include more LVLMs for experiments and conduct in-depth analysis, preferably summarizing directions to improve the long video description capabilities of LVLMs.
3. If possible, improve existing evaluation metrics to enhance their objectivity.
4. There are still many typos in the text, one of the most obvious being that the evaluation metric proposed by Wang et al. (2024) is AutoDQ, not AutoCQ.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639690314

thread_type:
official_review

<thread_object>
creator: reviewer_sakt

<content>
summary:
The paper introduces FIOVA, a novel benchmark for evaluating the video description capabilities of LVLMs in comparison to human understanding. The work provides a dataset of 3,002 long, diverse video sequences, each annotated by five distinct annotators, resulting in more comprehensive and longer captions. The paper conducts an in-depth evaluation of six state-of-the-art LVLMs, revealing that while they show some perception and reasoning capabilities, they still struggle with information omission and descriptive depth, especially in complex videos. The findings underscore the need for new evaluation perspectives that capture semantic understanding, fluency, and content relevance, guiding future advancements toward human-level video comprehension.

strengths:
1. The work provides extensive materials (video theme definition, representative data, annotation rules, prompts) to make it less difficult to reproduce.
2. FIOVA will be a valuable resource for evaluating the video understanding capabilities of LVLMs.

weaknesses:
1. Some settings are not easy to follow, like the `Batch Ranking` in Sec 4.3. 
2. `Describe Videos like Humans` might be an interesting evaluation setting. However, it does not stand alone as a task. It would be meaningful to include further analysis to show the correlation between performance  of `Describe Videos like Humans` and other video understanding tasks (VideoQA, etc.). 
3. While this work has adopted multiple metrics to demonstrate the video caption performance, it lacks analysis of how those metrics align with human preference.

questions:
Sec 2.2 shows that GPT-3.5-turbo is adopted to assess the quality of video caption, while that does not sound make sense to me. How can GPT-3.5 (a legacy text-only model) evaluate dimensions like Context, Correctness for video captions without accessing the visual parts? Can you provide more evaluation samples?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719328092

thread_type:
official_review

<thread_object>
creator: reviewer_fqpc

<content>
summary:
In this paper, the authors propose FIOVA benchmark, designed to evaluate the video description capabilities of LVLMs in comparison with human understanding. The authors address the limitations of current benchmarks by introducing a long-video dataset with diverse scenarios, and annotations from multiple annotators. The paper reports an in-depth evaluation of six state-of-the-art LVLMs and compares their performance with human annotations across various metrics. The findings highlight the discrepancies between LVLMs and human annotators, particularly in complex videos.

strengths:
- The paper is clearly written and presents a well-structured methodology, results, and analysis.
- The collected dataset is of high quality, featuring perspectives from multiple annotators.
- The authors provide comprehensive evaluations of LVLMs using traditional metrics and AutoCQ-based metrics.

weaknesses:
The overall paper reads more like a collection of experimental analyses rather than a benchmark for the following reasons:
- There are no experiments that demonstrate why the authors' dataset is superior to other datasets. For instance, is the FIOVA dataset better than the DREAM-1K dataset used in Tarsier for evaluation? Do the models that perform better on FIOVA also perform better in human evaluations?
- No new metrics are proposed. Traditional metrics are not well-suited for current LVLMs due to the nature of long responses. Additionally, is AutoCQ sufficient for evaluating dense captioning?

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521292374

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We sincerely thank the reviewers and the Area Chair (AC) for their constructive feedback and their recognition of our work. The reviewers have highlighted the following key contributions of our paper:

1. **High-quality dataset**: The FIOVA dataset was praised for its richness and comprehensiveness, particularly the multi-perspective annotations provided by five annotators per video (**Reviewer fqpC, Reviewer CA5F**). The inclusion of long videos with complex spatiotemporal relationships was recognized as a significant advancement over existing benchmarks (**Reviewer pxyN**).

2. **Innovative evaluation framework**: The systematic and comprehensive nature of the multi-metric evaluation framework was highly appreciated (**Reviewer fqpC**).

3. **Performance analysis of LVLMs**: The fine-grained comparison of LVLM outputs and human annotations was commended for providing valuable insights into the strengths and limitations of LVLMs (**Reviewer CA5F, Reviewer pxyN**). The evaluation of six SOTA LVLMs was noted as a valuable benchmark contribution (**Reviewer pxyN**).

4. **Impact on future research**: FIOVA was considered a challenging and critical resource, especially for understanding complex real-world video scenarios (**Reviewer SaKT, Reviewer pxyN**).

5. **Paper quality and comprehensiveness**: The reviewers acknowledged the clarity of the paper's structure and its high-quality writing (**Reviewer fqpC, Reviewer CA5F**) as well as the richness of the supporting materials (**Reviewer SaKT**).

Based on the reviewers' invaluable suggestions, we have made extensive revisions to the paper, with all changes highlighted in blue. The key revisions are as follows:

1. **Introduction of the FIOVA-DQ metric:**
- Detailed explanations of the FIOVA-DQ metric were added in Sec 3.2 and Sec 4.1, with illustrative examples provided in the revised Fig. 4. This metric incorporates event importance weights based on human cognition, enhancing the fairness and accuracy of the evaluation.
- Additional experimental analyses based on FIOVA-DQ were included in Sec 4.2, with fine-grained results presented in Appendix E.2.
2. **Improved transparency and reliability of groundtruth generation:**
- Fig. 4 was revised, and its generation process was elaborated in Appendix B.6 to ensure transparency and accuracy.
- Explanations of video data sources and collection methods were added in Appendix B.1.
- Quality control measures for human annotations were detailed in Appendix B.2.
3. **Expanded experiments and evaluations:**
- Detailed justifications for selecting six evaluated models were added in Appendix D.2.
- Further clarification of batch ranking computations was provided in Sec 4.2, supplemented by a visual illustration (Fig. A8) in Appendix C.
- Fine-grained error annotations and analyses were added in Appendix E.4, exploring model limitations and proposing optimization directions.
4. **Additional refinements:**
- Annotation style-based grouping examples were included in Appendix B.5, showcasing case studies of Groups A and H.
- Further explanations for Table A.3 were added in Appendix E.1.

We sincerely thank the reviewers for their valuable suggestions and support. These insights have significantly enhanced the quality and impact of our paper. We believe the revised manuscript is more rigorous and compelling. In the following rebuttal, we first address the two common concerns raised by reviewers, followed by point-by-point responses to each reviewer's specific comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521525295

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern 1. Reasons and quality of using GPT-3.5-turbo to generate groundtruth

comment:
We sincerely thank the reviewer for their concerns regarding the Groundtruth (GT) generation process and quality. The FIOVA benchmark is designed to provide high-quality, semantically consistent, and diverse GT by employing a multi-annotator strategy combined with LLM-based integration, enabling a comprehensive evaluation framework for long video description tasks. **Multiple measures have been implemented to ensure the GT's quality.** Below is our detailed response:

**1. Design Philosophy of FIOVA: Ensuring GT Quality**
- **Multi-Annotator Strategy:** Each video was annotated by five rigorously selected and trained independent annotators. After watching the full video, the annotators provided detailed descriptions based on their observations, ensuring that all descriptions strictly aligned with the video content. This multi-perspective annotation method not only captures rich semantic details but also minimizes potential bias or omissions from a single annotator.
- **LLM-Based Text Integration:** GPT-3.5-turbo was employed to semantically merge and linguistically enhance the multi-perspective descriptions provided by annotators. Its role is strictly confined to integration and optimization of text rather than generating new content or subjectively interpreting the video. This approach avoids potential interference from multimodal information while preserving the diversity and accuracy of the annotators' descriptions. This methodology has been validated and widely applied in related research (e.g., Tarsier [1], MM-Ego [2]).

**2. Addressing Specific Issues in GT Generation**
- **Hallucination Issues:** We opted to use LLMs (e.g., GPT-3.5-turbo) instead of VLMs (Vision-Language Models) for the integration task to reduce hallucination issues that could arise from multimodal alignment, such as generating content not present in the video. With carefully designed prompts, GPT-3.5-turbo focuses on ensuring semantic consistency during integration, thereby avoiding redundant or irrelevant content.
- **Resolving Annotator Conflicts:** As analyzed in Appendix B.6 and illustrated in Figure A7 of the revised paper, when annotators provided conflicting descriptions for certain events (e.g., "smiling" vs. "crying"), GPT-3.5-turbo synthesized the descriptions and the contextual logic of the video to arrive at a reasonable semantic conclusion. For instance, by combining Human 1's mention of the "pretend" action and Human 5's description of the boy smiling, the final conclusion aligns better with the video's overall semantics.
- **Addressing Omission of Details:** The multi-annotator strategy ensures that GT covers all core events comprehensively. During the generation phase, we optimized prompt designs through multiple iterations and manually reviewed and corrected the GT to minimize omissions and redundancies.

**3. Measures to Enhance GT Quality**
- **Prompt Optimization:** We refined prompt designs through multiple rounds of experiments to guide GPT-3.5-turbo in producing high-quality, semantically consistent, and linguistically fluent text.
- **Manual Review and Correction:** All 3002 GTs underwent comprehensive manual checks, with problematic descriptions regenerated and corrected. Corresponding experimental results were also updated to ensure that the GT is comprehensive, accurate, and consistent.

**4. Significance and Future Directions**
Through the construction of high-quality GT, FIOVA provides a more comprehensive and impartial evaluation framework for long video description tasks. The integrated GT:
- **Comprehensive Coverage:** Reflects the core semantics of the video content, encompassing multi-perspective and diverse descriptive details.
- **Reliable Evaluation:** Significantly enhances the consistency and reliability of LVLM evaluation in long video tasks.
We firmly believe that the design philosophy and practical methodology of FIOVA not only address the reviewer's concerns but also provide strong support for future research.

[1] Wang J, Yuan L, Zhang Y, et al. Tarsier: Recipes for training and evaluating large video description models[J]. arXiv preprint arXiv:2407.00634, 2024.

[2] Ye H, Zhang H, Daxberger E, et al. MM-Ego: Towards Building Egocentric Multimodal LLMs[J]. arXiv preprint arXiv:2410.07177, 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732521898550

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Concern 2. Propose a New Evaluation Method FIOVA-DQ

comment:
We sincerely thank the reviewers for their insightful feedback on evaluation metrics. Based on the reviewers' suggestions, we extended the original AutoDQ method and **designed a novel weighted evaluation metric, FIOVA-DQ, tailored to the characteristics of the FIOVA benchmark.** 

Below, we provide a detailed explanation of this new metric：

AutoDQ, proposed by Wang et al., is an event-based evaluation method that extracts and matches events from reference and model-generated descriptions. While AutoDQ offers fine-grained assessment, it does not incorporate human annotators' cognitive understanding of event importance. FIOVA-DQ addresses this limitation by achieving the following goals:

- **Incorporating Human Cognition:** Assign weights to events based on human annotators' perspectives, ensuring more critical events have greater influence on the evaluation.
- **Improving Evaluation Precision:** Calculate weighted precision and recall based on event alignment, offering fairer and more representative evaluation results.

---

**FIOVA-DQ Calculation Process (refer to Fig.4 in the revised manuscript)**

**1. Event Extraction**
FIOVA-DQ starts by extracting events from human annotations, groundtruth descriptions, and model-generated outputs:
- Extract events from human annotations: Create event lists $H_1, H_2, H_3, H_4, H_5$ from the five annotators' descriptions.
- Extract events from Groundtruth (GT): Use GPT-3.5-turbo to extract events from the aggregated GT description, forming list $G$.
- Extract events from model outputs: Extract events from the model-generated descriptions, forming list $V$.

**2. Event Weighting**

To reflect human cognition, weights are assigned to each event in the GT. The process is as follows:
- Ranking for individual references:
    -  For each event $g_i$ in $G$, calculate its similarity with events in $H_1$.
    -  Rank $g_i$ based on similarity, repeating for all annotators ($H_1, H_2, H_3, H_4, H_5$), and calculate the average rank for each $g_i$.
-  Weight Generation:
    -  Normalize the average rank (average rank/total rank sum) to assign weights.
    -  For example, in Fig.4, the GT events $g_1, g_2, \dots, g_7$ have weights $w_{g1}=0.114, w_{g2}=0.182, \dots, w_{g7}=0.114$.
    -  The same process is applied to the model events $v_1, v_2, \dots, v_6$, resulting in weights $w_{v1}=0.053, w_{v2}=0.093, \dots, w_{v6}=0.173$.

**3. Precision and Recall Calculation**
Based on the weighted results, precision and recall are computed:
- Precision: The sum of weights for matched events in $V$ divided by the total weight of $V$. For Fig.4, matched events $v_2, v_4, v_6$ have weights $0.093, 0.227, 0.173$, yielding Precision = $0.493$.
- Recall: The sum of weights for matched events in $G$ divided by the total weight of $G$. For Fig.4, matched events $g_1, g_4$ have weights $0.114, 0.219$, yielding Recall = $0.333$.

---

**Comparison with AutoDQ**

FIOVA-DQ significantly improves upon AutoDQ:

1. **Incorporates Human Cognition:** Reflects human understanding of event importance.

2. **Enhances Evaluation Effectiveness:** Prioritizes key events over minor ones.

3. **Aligns with FIOVA's Multi-Perspective Philosophy:** Leverages multi-annotator insights for comprehensive and fair evaluation.

We believe FIOVA-DQ provides a robust framework for evaluating LVLM performance in long video description tasks and offers valuable insights for future research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a new benchmark to evaluate the video caption of LVLMs. Different from existing benchmarks, the proposed benchmark contains longer videos (33.6 seconds on average) with five human annotations for each video. It applies GPT-3.5 evaluate the quality of human annotations and also use it to aggregate all annotations to generate ground truth captions. A novel metric FIOVA-DQ is also proposed as a more human-aligned metric for LVLMs.

Strength:
1. The collected dataset is different from all existing dataset: (a) Average duration is longer than existing dataset; (b) Have multiple human annotations.
2. This paper provide in-depth evaluation of six open-sourced SOTA methods.
3. As suggested by reviewers, a new metric FIOVA-DQ which is more human-aligned is proposed.

Weakness:
The biggest concern raised by several reviewers is the quality of the ground truth. The authors spend a lot of effort to answer questions about this, the concerns are not fully addressed.