Neptune: The Long Orbit to Benchmarking Long Video Understanding

Authors: Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, Mikhail Sirotenko, Yukun Zhu, Tobias Weyand
Primary area: datasets and benchmarks
Keywords: video understanding, dataset, metric, long video understanding, benchmark

Abstract:
This paper describes a semi-automatic pipeline to generate challenging question-answer-decoy sets for understanding long videos.  Many existing video datasets and models are focused on short clips (10s-30s). While some long video datasets do exist, they can often be solved by powerful image models applied per frame (and often to very few frames) in a video, and are usually manually annotated at high cost. In order to mitigate both these problems, we propose a scalable dataset creation pipeline which leverages large models (VLMs and LLMs), to automatically generate dense, time-aligned video captions, as well as tough question answer decoy sets for video segments (up to 15 minutes in length). Our dataset Neptune covers a broad range of long video reasoning abilities and consists of a subset tha temphasizes multimodal reasoning. Since existing metrics for open-ended question answering are either rule-based or may rely on proprietary models, we provide a new open source model-based metric (GEM) to score open-ended responses on Neptune. Benchmark evaluations reveal that current open-source long video models perform poorly on Neptune, particularly on questions testing temporal ordering, counting and state changes. Through Neptune, we aim to spur the development of more advanced models capable of understanding long videos.

Decision: reject
Reviewer scores: 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730534088380

thread_type:
official_review

<thread_object>
creator: reviewer_j9hh

<content>
summary:
The paper introduces NEPTUNE, a new benchmark for evaluating multimodal understanding of long videos. It addresses the limitations of current video datasets and models, which often focus on short clips and can be solved by image models applied per frame. The authors propose a semi-automatic pipeline that leverages large Video Language Models (VLMs) and Large Language Models (LLMs) to automatically generate dense, time-aligned video captions and challenging question-answer-decoy sets for video segments up to 15 minutes long. The paper also discusses the related works in video question answering, the motivation behind the NEPTUNE dataset, and the detailed pipeline for dataset creation. It concludes with the limitations of the dataset and potential areas for future improvement, such as reducing biases introduced by the models used in the creation process and enhancing the dataset with additional annotations like temporal grounding or entity labels.

strengths:
1. The paper introduces an innovative semi-automatic pipeline designed to generate question-answer-decoy (QAD) sets. This method effectively reduces the annotation costs associated with manual video captioning and question generation. By leveraging large models such as VLMs and LLMs, the pipeline automates the creation of dense, time-aligned video captions, which are then used to derive challenging QAD sets for segments of video content. This approach not only scales well but also maintains a high standard of quality in the dataset.

2. A significant contribution of this paper is the training and proposal of a model for the automatic evaluation of open-ended QA responses. The authors address the limitations of existing metrics, which are either rule-based or rely on proprietary models, by finetuning an open-source model on a generic answer equivalence dataset. This results in the GEM (Gemma Equivalence Metric), a new metric that provides a more accurate assessment of open-ended responses on the NEPTUNE dataset, thus facilitating more reliable benchmarking and comparison of different models.

3. The NEPTUNE benchmark covers a wide range of video lengths and types, ensuring diversity in the dataset. This comprehensive approach guarantees that the models evaluated on this benchmark are tested on their ability to reason over a variety of video content, going beyond the capabilities required for understanding short video clips. The inclusion of different video domains and the emphasis on longer-form content make the NEPTUNE dataset a robust platform for assessing multimodal video understanding systems across a broad spectrum of real-world scenarios.

weaknesses:
1. The paper does not include an evaluation and comparison with the latest open source models such as InternVL, LLaVA-OneVision, and MiniCPM. These models are part of the current research landscape and offer a different perspective on video understanding capabilities 

2. The paper primarily focuses on the analysis of benchmarks like NextQA and EgoSchema but does not provide a thorough comparison with more recent benchmarks such as MLVU, Video-MME, and LongVideoBench, which are designed to evaluate long-form video understanding 

3. Although the semi-automatic pipeline proposed in the paper effectively reduces the workload of manual annotation, it lacks novelty in terms of a detailed analysis and comparison with other pipelines. The paper could benefit from a more in-depth exploration of the unique aspects of its pipeline and how it compares to existing methods 

4. While the proposed GEM metric may have lower evaluation costs compared to assessments using models like GPT or Gemini, the paper lacks a comparison with the consistency of human evaluations. Introducing human annotations to quantify and analyze the quality of assessments would strengthen the paper's findings

questions:
Why not compare with other benchmarks and test the latest open-source models?
How to ensure the reliability of GEM's evaluation results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730552813333

thread_type:
official_review

<thread_object>
creator: reviewer_mqoh

<content>
summary:
This paper introduces a scalable, semi-automatic pipeline for constructing the Neptune benchmark dataset, aimed at evaluating long video understanding capabilities. A new evaluation metric, GEM, is proposed, demonstrating its advantages over traditional rule-based evaluation metrics.

strengths:
1. Neptune addresses several essential question types related to temporal-aware video understanding, including the challenging temporal ordering and counting.
2.	Two subsets are introduced to comprehensively assess current multi-modal large language models.

weaknesses:
1. In Figure 4, the authors show that EgoSchema reaches saturation at approximately 16 frames, while performance continues to increase with Neptune. This conclusion is drawn based on the powerful Gemini model; it would be beneficial to additionally include results from some open-source models (e.g., short or long context MLLMs) to better promote the development of open-source MLLMs.
2. Model names should be consistently formatted (e.g., VideoLLaMA2 vs. VideoLlaMA2).

questions:
1. In Tables 3 and 4, it is evident that a language-only Gemini model (without ASR) even outperforms the baseline image model (BLIP2) and a short-context MLLM (Video-LLaVA). Could you provide possible reasons for this? Is it because the language models in BLIP2 and Video-LLaVA are not sufficiently robust to describe visual information?
2. In Table 3, the authors present the performance with and without ASR. The blind model (0 frame) achieves a comparable (Gemini) or even higher (VideoLLaMA2) GEM score than the model evaluated with full-frame input. Are there possible ways to mitigate the impact of ASR during dataset construction to encourage the design of vision-centric MLLMs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730603576663

thread_type:
official_review

<thread_object>
creator: reviewer_jd8f

<content>
summary:
The paper introduces a new benchmark dataset called Neptune, designed to assess multimodal, high-level understanding of long videos. The key contributions of the paper are as follows:

1. Semi-Automatic Pipeline: The authors propose a scalable, semi-automatic pipeline for generating challenging question-answer-decoy sets for long videos, leveraging large video language models (VLMs) and large language models (LLMs) to automatically generate dense, time-aligned video captions and tough QAD sets for video segments up to 15 minutes long.
2. Neptune Dataset: The dataset, Neptune, covers a broad range of long video reasoning abilities and includes a subset that emphasizes multimodal reasoning. It consists of 3,268 QAD annotations for 2,405 videos.
3. Evaluation Metrics: Given the limitations of existing metrics for open-ended question answering, the authors introduce a new model-based metric called GEM (Gemma Equivalence Metric) to score open-ended responses on Neptune, offering a static and reproducible evaluation method.
4. Benchmark Evaluations: The paper presents benchmark evaluations that reveal current open-source long video models perform poorly on Neptune, especially on questions of testing temporal ordering, counting, and state changes, indicating a significant gap between open-source and proprietary models like Gemini-1.5 and GPT-4.

In summary, the paper contributes a new benchmark dataset and metric aimed at advancing the field of long video understanding through multimodal reasoning, with the goal of spurring the development of more advanced models in this area.

strengths:
1. The semi-automated annotation method proposed in this paper overcomes some of thechallenges associated with manual annotation, such as the creation of complex temporal reasoning questions, which can be laborious for humans. The utilization of GPT to generate these questions reduces this burden.

2. The introduction of the GEM metric addresses the need for a static, open-source evaluation metric for open-ended VideoQA, which has been a limitation in prior work.

3. The paper clearly articulates the motivation behind the Neptune dataset and the GEM metric, making it easy for readers to understand the importance of the work.

weaknesses:
1. The key contributions that distinguish it from previous benchmarks are not clearly highlighted. Although the paper has made a simple comparison with EgoSchema on the impact of the number of input frames on performance, I feel that it is too simplistic. It would be constructive to see more comparisons with other existing datasets (e.g.,VideoMME) to understand how NEPTUNE's complexity and diversity align or differ from them, which could highlight the unique challenges it presents.

2. In addition to the Gemini bias discussed in the paper, I noticed that the frame captions are generated by PaLI-3. I am concerned that the hallucinations and preferences of the VLM itself (such as a preference for describing static images) may affect the quality of the
final generated QA.

3. As a benchmark for long videos, the discussion on the impact of the number of input frames on model performance is too simplistic. For example, Table 3 only discusses the performance of Gemini with 1 frame, 150 frames, and All frames. In my view, the performance with 150 frames has already reached saturation, but it is uncertain how the performance with 100 frames and 50 frames would be, that is, where the performance saturation point is.

questions:
1. As a benchmark for long videos, single-frame assessment fails to reflect the significance of temporal modeling. I believe that the sensitivity to temporal order should be evaluated by adopting methods such as reversing or shuffling the input frames.

2. I have observed that the Gemma Equivalence Metric shows similar or even inferior performance compared to direct assessment with Gemini 1.5 Pro, with an accuracy rate of only around 70%. In contrast, previous works such as Lingo-Judge[1] seem to demonstrate that fine-tuned model-based evaluation methods can achieve assessment accuracy close to human judgment. Is this due to differences in dataset complexity?

3. I have noticed that the distribution of generated QA types exhibits a clear long-tail distribution, with the largest proportion being Temporal Ordering tasks. Does this not suggest that the complexity and richness of the generated questions are insufficient? In my view, Ordering tasks belong to very simple temporal reasoning and do not meet the authors' claim of being complex questions that are difficult for humans to propose.

[1] LingoQA: Visual Question Answering for Autonomous Driving
</content>
</thread_object>

</thread>
<thread>
creation date:
1732388587438

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their insightful comments, highlighting the strengths of our dataset and suggesting areas for improvement. Here, we address points raised by multiple reviewers.

# Comparison to other Long Video dataset pipelines 

We highlight novelties of our dataset, and its creation pipeline compared to both previous and concurrent works \[1-11\].

1. **Reducing Manual Effort**: Our pipeline is semi-automatic, unlike manual pipelines \[4,5,3,11\]. In LVBench \[3\] and VideoMME \[11\], even the video selection is done manually, and for MoVQA \[1\], only the decoys are generated automatically. We asked raters to propose QAs (but not decoys) for videos from scratch, and found the time taken (19.03 minutes), longer than simply discarding or correcting QAs generated automatically (10.32 minutes), i.e. showing our pipeline almost halves rater effort. Manually proposing decoys is even more challenging, both in time as well as effort (as it requires more creativity to come up with incorrect answers). The closest to our pipeline is concurrent work VideoVista \[6\], which uses GPT4 to generate QADs automatically. However the performance of GPT4 and Gemini-1.5 on this dataset is close to saturated (98% on some categories).  
2. **Scalability to generic YouTube videos**: Our pipeline can be applied to any generic YouTube video. This is unlike EgoSchema \[2\], which relies on human generated captions, SFD \[9\], which requires movie titles, loglines and synopses (human-written), or MLVU \[4\], which re-uses annotations from existing datasets for many of their tasks. This makes the dataset scalable, as YouTube has a constantly growing set of videos. Potential further use cases could be applying this pipeline to generate large quantities of training data.   
3. **High Quality Video Captions using careful multi-stage prompting**. A key novelty is the ability to generate high quality video captions, using a multi-stage process (Sec 4.2) which involves generating shot-level captions, clustering shots into segments, and adding visual support captions. Examples of caption quality are provided in the supplementary (Fig. 13 and Sec. F), showcasing details such as visual support, numerous events, mood and atmosphere, details from the ASR, and even high level feelings and emotions. 

# Clarifications on the GEM metric 

***Training:*** We train GEM by fine-tuning the open source Gemma model on BEM, a dataset of general answer equivalence ratings. Therefore, GEM is not specific to Neptune and generally applicable to other answer equivalence problems as well.  
***Comparison to human assessments:*** To assess GEM (Sec. 5.1, appendix D.1, D.2), we **manually annotate** an answer equivalence dataset by sampling 97 question-answer pairs from Neptune. We then generate 3 candidate answers per question by prompting VideoLLAVA, Gemini-1.5-Pro and MA-LMM to write a free-form answer for each question without looking into the decoys or ground truth. We then used human raters to manually score these responses, resulting in a dev set with 291 QA-pairs that allows us to compare GEM to human performance. GEM metric reaches an F1-Score of 71.2, which very closely approximates the performance of the much more expensive, closed-source Gemini model (F1-Score 72.8). We will move some details from the appendix to the main paper.

# Additional Frame Experiments

We provide additional frame experiments for both Gemini-1.5-Pro and new open-sourced models below. We find that Gemini saturates at around 50 frames while open-sourced models have varying saturation points, VideoLLaMA2 (8 frames), MiniCPM (50 frames) and LLaVA-OneVision (100 frames). For both MiniCPM and LLaVA-OneVision we were unable to fit more frames into the context window.

This is an interesting experiment which we will add to the paper. Note that these are uniformly sampled frames, and hence cover the full span of the video.

| \# of Frames | 1 | 8 | 16 | 50 | 100 | 150 |
| :---- | ----: | ----: | ----: | ----: | ----: | ----: |
| **Gemini-1.5-Pro** | 55.57 | 63.80 | 66.62 | 70.08 | **70.44** | 69.31 |
| **LLaVA-OneVision**  | 56.56 | 62.51 | 63.98 | 65.70 | **66.22** | Out of context |
| **MiniCPM-V 2.6**  | 50.89 | 53.83 | 53.86 | **56.59** | 55.00 | Out of context |
| **VideoLLaMA2** | 40.88 | **44.74** | 44.74 | \- | \- | \- |

We also provide frame ablations comparing Neptune to other datasets in our response to reviewer jd8F.

\[1\]\* MoVQA, Zhang et al.  
\[2\] EgoSchema, Mangalam et al.  
\[3\]\* LVBench, Wang et al.  
\[4\]\* MLVU, Zhou et al.  
\[5\] MMBench-Video, Fang et al.  
\[6\]\* VideoVista, Li et al.  
\[7\]\* ReXTime, Chen et al.  
\[8\]\* LongVideoBench, Wu et al.  
\[9\]\* Short Film Dataset, Ghermi et al.  
\[10\] Towards long-form video understanding, Wu et al.  
\[11\]\* Video-MME, Fu et al.  
\[12\] CinePile, Rawal et al.  
\[13\] Perception Test, Pătrăucean et al.

\* (unpublished or concurrent work)
</content>
</thread_object>

</thread>
<thread>
creation date:
1733272757969

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We thank the reviewers for their helpful comments and insightful questions which have helped strengthen our paper! We summarize the improvements we have made as part of the rebuttal here:

1. **Comparisons to additional open source models**: We evaluated three more recent open source models: LLaVA-OneVision, MiniCPM and InternVL2, giving a more comprehensive and up-to-date view of the state of the art on our dataset.  
2. **More detailed ablation with different number of frames**: We added an ablation going from 1 to 150 frames in 6 steps. Besides Gemini, which we used in the original ablation in the paper, we also ablated on 3 open source models, providing insights into each model’s ability to reason with limited information and their saturation points.  
3. **Experimental comparison with three other video understanding benchmarks**: We compare Neptune against Video-MME, CinePile and Perception Test using Gemini-1.5-Flash with different numbers of frames, showing the model performance and saturation characteristics on each benchmark.
4. **An additional metric that gives equal weights to different task types**. This allows for a more balanced view of model performance and removes skew of the metric to the most prominent task types.  
5. **A breakdown of per-task accuracy**. We have added a table that breaks down performance of all benchmarked models per task and allows detailed insights into each model’s strengths and weaknesses.

Additionally, we will include clarifications that reviewers requested, e.g. the comparison of our pipeline to pipelines used for other benchmarks, and details on our evaluation of the GEM metric. Again, we sincerely thank the reviewers for taking the time to review our paper and providing us with their valuable feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces NEPTUNE, a benchmark designed to evaluate multimodal understanding of long videos, addressing the limitations of existing datasets that primarily focus on short clips. The paper contributes a semi-automatic pipeline leveraging large video language models (VLMs) and large language models (LLMs) to generate dense, time-aligned video captions and challenging question-answer-decoy (QAD) sets for video segments up to 15 minutes long. Additionally, it proposes a new evaluation metric, GEM (Gemma Equivalence Metric), to assess open-ended responses in a static and reproducible manner. Reviewers commend the dataset's focus on long video reasoning, its innovative annotation pipeline, and the introduction of GEM, which tackles the limitations of prior evaluation metrics. However, concerns about the lack of comparisons with recent benchmarks (e.g., MLVU, Video-MME) and state-of-the-art open-source models (e.g., InternVL, LLaVA-OneVision) are raised. Reviewers also note potential biases in dataset creation due to reliance on VLMs and LLMs, limited novelty in the annotation pipeline, and insufficient evaluation against human judgment for GEM. While the contributions are valuable, the absence of more substantial comparisons and deeper analysis leaves room for improvement.