Frame-Voyager: Learning to Query Frames for Video Large Language Models

Authors: Sicheng Yu, CHENGKAI JIN, Huanyu Wang, Zhenghao Chen, Sheng Jin, ZHONGRONG ZUO, XU XIAOLEI, Zhenbang Sun, Bingni Zhang, Jiawei Wu, Hao Zhang, Qianru Sun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video-LLM, Adaptive Frame Sampling

Abstract:
Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks. However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos. Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos or the complex instructions in the tasks, leading to sub-optimal performance. In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task. To train Frame-Voyager, we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM. Given a video of M frames, we traverse its T-frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM's prediction losses. Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses. In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs. The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings, highlighting its potential as a plug-and-play solution for Video-LLMs.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729844669767

thread_type:
official_review

<thread_object>
creator: reviewer_ults

<content>
summary:
The paper presents FRAME-VOYAGER, a novel approach for improving Video Large Language Models by selecting informative frame combinations based on textual queries. Traditional methods like uniform frame sampling and text-frame retrieval do not account for variations in information density or complex instructions, leading to sub-optimal performance. FRAME-VOYAGER addresses this by learning to query frame combinations with lower prediction losses, using a pre-trained Video-LLM for supervision. The method is demonstrated to enhance performance in video question answering tasks across various benchmarks. The approach also includes a new data collection and labeling pipeline, which ranks frame combinations to train the model efficiently. FRAME-VOYAGER is proposed as a plug-and-play solution, improving Video-LLMs without significant computational overhead. Experiment results show that FRAME-VOYAGER achieves significant performance in several VQA datasets.

strengths:
1.The authors designed an efficient keyframe data construction method, exploring keyframe selection using combinations of all frames. This approach achieved significant improvements across multiple datasets with 12K high-quality clips.

2.The authors state that they will open-source the data and related code, which is valuable for the open-source community.

3.The proposed method demonstrate strong performance across multiple datasets, proving the robustness of the approach, especially in Anet dataset. As a plug-and-play method, Frame-Voyager can serve as a valuable tool for video understanding.

weaknesses:
1.Data scaling limitations. As the authors mentioned, selecting 4 frames from 32 results in 35960 candidates, leading to prohibitively high costs for data scaling. Furthermore, keyframe selection is even more critical for longer videos, and the costs become unmanageable when dealing with selections from 128 or 256 frames.

2.When testing on short video benchmarks, did VILA and VILA+FRAME-VOYAGER evaluate performance by selecting 2 frames out of 16? If so, please provide the results of VILA using 16 frames on the Next-QA and ANQA datasets. Additionally, I believe that comparing results based on only 2 frames is not meaningful, as most models typically evaluate performance using 8 or 16 frames.

3.Please provide additional results of the model on more benchmarks, such as MVbench and egoSchema.

questions:
See the above weakness. I am interested in how the authors consider data scaling. Constructing data with only 16 or 32 frames is not very helpful for understanding long videos.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730119371458

thread_type:
official_review

<thread_object>
creator: reviewer_qbz8

<content>
summary:
This paper introduces Frame-Voyager, a novel frame selection method designed to improve the performance of Video-LLMs on video question-answering (QA) tasks. The key contributions are (1) an automated data collection strategy that ranks frame combinations according to the QA loss from a reference Video-LLM, and (2) a frame selection module integrated with a Video-LLM, trained using this ranked data. When applied to VILA, Frame-Voyager outperforms the uniform sampling baseline across four video QA benchmarks.

strengths:
**1. Originality:** The novel formulation of frame selection as a ranking problem is compelling. This approach minimizes the need for extensive manual labeling to obtain ground truth, offering a fresh and efficient perspective on the task.

**2. Significance:** Frame selection is a critical challenge in video understanding due to the high dimensionality of video data and the computational demands involved. Traditional uniform sampling often misses relevant content, making this problem a key focus for advancing the field.

**3. Clarity:** The paper is well-written, with a clear explanation of the motivation and methodology. The authors effectively communicate the concepts, making the approach easy to understand.

weaknesses:
**1. Data Collection Cost:**
The number of frame combinations increases exponentially with M and T (L166), leading the authors to limit M to 16 or 32 and T to 2 or 4. Despite these restrictions, evaluating $C(32, 4) \approx 36K$ combinations for a single QA sample is still costly, raising concerns about data collection efficiency. Thus, this paper needs to provide more details on the computational resources and time required for data collection.

**2. Generalization to More Frames:** 
The method is trained to select up to T=4 frames from M=32 candidates. Although the authors claim the model generalizes well to larger M and T values (L170), Fig. 3 shows diminishing gains over uniform sampling while increasing the number of frames. Despite the authors attributing this to benchmark constraints (L431), prior research (e.g., LongVILA [1], LongVA [2]) has shown that additional input frames can enhance QA performance, suggesting potential limits in generalization. Given the impracticality of increasing M and T for data collection, robust generalization is crucial. Thus, the authors should extend their experiments to more input frames, possibly integrating Frame-Voyager with LongVILA, to help clarify the generalization capabilities and limitations of Frame-Voyager.

**3. Fairness of Comparisons:** 
The comparison between Frame-Voyager and CLIP is not fully equitable due to three factors: (i) Frame-Voyager uses superior backbones (SigLIP or InternViT-6B); (ii) it is trained on specialized ranking data, unlike CLIP; and (iii) it has additional parameters like self-attention modules. Since (ii) and (iii) are core contributions of Frame-Voyager, a fairer comparison would involve using SigLIP or InternViT-6B models with and without training on the collected data. Additionally, the authors should specify the versions of CLIP and SigLIP used.

**4. Latency Analysis:** 
The authors claim that Frame-Voyager outperforms LongVILA while processing fewer frames, but this only holds for the "Long" subset of the Video-MME benchmark. In other scenarios, LongVILA performs better. Given the extra parameters introduced by Frame-Voyager, a detailed efficiency comparison between VILA+Frame-Voyager and LongVILA is necessary, including latency, memory usage, and computational complexity.

---
[1] Xue F, Chen Y, Li D, et al. Longvila: Scaling long-context visual language models for long videos[J]. arXiv preprint arXiv:2408.10188, 2024.

[2] Zhang P, Zhang K, Li B, et al. Long context transfer from language to vision[J]. arXiv preprint arXiv:2406.16852, 2024.

questions:
My questions focus primarily on the weaknesses outlined above:

1. What are the data collection costs, particularly for the VideoChatGPT data?

2. How well does the method generalize when increasing the number of frame candidates and selected frames?

3. Can the proposed model outperform SigLIP and InternViT-6B, especially when they are fine-tuned on the same collected data?

4. What is the latency of the method?

5. A suggestion: Could CLIP-like models enhance data collection efficiency? Specifically, using them to estimate query-frame similarities might identify frames with extremely high or low similarities, indicating positive or negative frame combinations. This could reduce the search space and improve data collection efficiency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730436505450

thread_type:
official_review

<thread_object>
creator: reviewer_6u8s

<content>
summary:
In this paper, the authors present Frame-Voyager which learns to query informative frame combinations, based on the given textual queries for video qa tasks. The authors create a data collection pipeline for the proposed model/training objection and evaluate it on four video QA benchmarks.

strengths:
1. The motivation is clear and easy-to-follow. 

2. The presentation of the paper is of high quality. 

3. The analysis is comprehensive.

weaknesses:
1. The inflexibility of the proposed frame selection method. Different videos usually have different “information density”, which means that some video could be represented by even a single frame and some need densely sampled frames. This also would be affected by the query type. The number of keyframes in the proposed framework seems to be a fixed hyper-parameter, which would be hard to generalize to all different videos in the wild (for different length / query types, need different hyper-parameter selection). Is it possible for the model to do adaptive/dynamic frame selection? 

2. The authors mention in line 170-172 that “the smaller combinations exhibit generalization capabilities when larger values of M and T are used during inference for longer video”. However, the process of this generalization is unclear (the introduction of the inference process, top-K selection is also unclear, in line 258-261). Does the inference directly apply the model to higher frame number or is the concatenation of several combinations with the highest score, what is the K setting for inference? Also, based on this, could the authors elaborate more on how Frame-Voyager makes good use of global video context? 

3. In Table 1 and Figure 3, the authors only compare Frame-Voyager with the uniform sampling baseline using the same frame number of frames for Question-Answering. However, Frame-Voyager actually sees more frames for keyframe selection (128 frames and select 8 frames), which creates unfair comparisons. Also there seems to be very little improvement in performance while using 8 keyframes from 128 frames compared to just uniform sample 16 frames. Could the authors show the comparison of Frame-Voyager against the best uniform sampling configurations for a more fair comparison?

questions:
All question in weakness and: 

1. Since the authors claim that the proposed method is a plug-and-play module, does the reference Video-LLM for data collection have to be the same model family with the MLLM for Frame-Voyager? Since different models have different features, whether the generated data only works for the ViLA-1.5 model family? It would be interesting to see the performance of these annotated data’s effectiveness on different model families like LLaVA-onevision and so on. 

2. The authors are encouraged to discuss the comparison with a related line of research which leverages LLM agents to select meaningful keyframes from the video input, including VideoAgent [1], VideoTree[2] … 

3. Also, could the authors validate the effectiveness of the frame selection method on temporal grounding as well? Which seems to be very similar in high-level (localizing key information given the text query). 

[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent

[2] VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688436478

thread_type:
official_review

<thread_object>
creator: reviewer_pmfu

<content>
summary:
This paper proposed Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task.
Authors introduced a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM.
Extensive experiments are conducted to support the effectiveness of the proposed Frame-Voyager.

strengths:
This is a reasonable extension from previous keyframe selection work [1] where it relies more on single keyframe selection, and does not consider temporal relations/modeling among frames. The proposed pseudo-label scheme from a VLM + list of combinations of frames makes sense to me. 
Authors also conduct extensive experiments across diverse popular benchmarks to show their effectiveness. 
Overall, I think the proposed Frame-Voyager makes a good contribution to the keyframe selection in video-language studies. 

[1] Self-chained image-language model for video localization and question answering. NeurIPS23

weaknesses:
1. As the paper mentioned, this pseudo-label strategy is not scalable, and I think the frame combination part is a bit tricky. This is like creating artificial rewards according to video (like a sandbox in RL) to train the reward model. However,  the reward is not always reliable from a VLM even though we compute it according to GT answers. For example, as shown in some previous studies [2],  the model will generate correct answers when provided with wrong localized clips. It is true that we might give performance improvement when using such reward data created by a human-free pipeline, but it is more like an adapter / DPO style adaptation to the model rather than letting the model truly be grounded on query-related / informative frames. 

2. I think the weakness in 1 somehow affects the proposed method to require a relatively complex pre-training data construction/filtering/design, since the positive/negative signal is too sensitive according to video conditions in this framework from my view (e.g. frame blurring / redundancy).

3. Also, the keyframe selection for video-language understanding is not a brand new topic, many related works in track try to propose different ways, from continuous space learning to discrete pipeline, to address question-aware moment detection. I would suggest to include those works [1,2,3,4] for a more comprehensive study. 


[1] ViLA: Efficient Video-Language Alignment for Video Question Answering. ECCV24.  
[2] Can i trust your answer? visually grounded video question answering.  CVPR24.  
[3] TimeCraft: Navigate Weakly-Supervised Temporal Grounded Video Question Answering via Bi-directional Reasoning. ECCV24.  
[4] Self-Adaptive Sampling for Accurate Video Question Answering on Image Text Models. ACL24.

questions:
Please see the weaknesses. And extra questions:

(1) What is the trade-off between efficiency and effectiveness? Can you provide some metrics like running time/memory usage/flops to the proposed methods? As the model contains an extra keyframe localization stage, it is worthy showing those results to see the trade-off.

(2) I notice that the proposed method uses VILA as a reference model to label rewards, and uses the same VILA series models for downstream tasks. Is this also a kind of self-rewarding strategy shown in the sevilla work? Or the proposed method/modules can zero transfer to other VLM like llava-ov/qwen-vl-2?

(3) to answer my 1st weakness, I would suggest authors conduct extra experiments on Next-GQA to see the grounded QA results with grounded metrics.

(4) Can you show some off-shelf llm-based localization tools like sevila localizer in Table 2? It would be interesting to see the comparison between a llm-based reasoning method with the proposed reward imitation learning method.

(5)  Regarding the claim in Figure 3 and RQ3 (Lines 431-433), is this true? Related work (e.g., [1]) suggests that some observed issues may stem from limitations in the model side (VILA). Could the authors clarify this?

[1] LONGVIDEOBENCH: A Benchmark for Long-context Interleaved Video-Language Understanding
</content>
</thread_object>

</thread>
<thread>
creation date:
1732122707103

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response of First Round Rebuttal (1/4)

comment:
We thank all the reviewers for their valuable feedback! In this post, **we would like to first address the common questions raised by the reviewers with additional experimental results. In the sooner individual replies, we will address other specific comments.**

**1.** ```[pMFu, 6u8S]``` **Generalization on Different Backbones**: Could our method and generated data be generalized to other Video-LLMs such as LLaVA-One-Vision?
To answer this question, we directly apply the same data obtained by VILA-8B to train Frame-Voyager on one of the SOTA Video-LLMs, LLaVA-One-Vision-7B. We evaluate the models on the same benchmarks used in our paper. The results are shown in the table below.

|      | **Video-MME (w/o subtitles, %)** | **MLVU (%)** | **ANQA (%)** | **NextQA (%)** |
|---------------------|---------------|----------|----------|------------|
| LLaVA-One-Vision-7B | 53.3          | 58.5     | 41.7     | 72.5       |
| **+Frame-Voyager**  | 57.5          | 65.6     | 48.4     | 73.9       |

From the table, we can observe that the Frame-Voyager module can consistently improve the performance of LLaVA-One-Vision-7B across different benchmarks. It shows that Frame-Voyager trained with data generated by one Video-LLM (in our case, VILA-8B) can be zero-transferred to another Video-LLM (in our case, LLaVA-One-Vision-7B). This demonstrates the generalizability and plug-and-play ability of Frame-Voyager.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732122926594

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response of First Round Rebuttal (2/4)

comment:
**2.** ```[pMFu, 6u8S, qbz8]``` **Generalization on Higher Numbers of Frames**: How does Frame-Voyager generalize to higher numbers of selected frames? 

The generalization on the number of frames in our paper is: when training, we sample 4 frames from 32 frames (or 2 frames from 16 frames), and when inference, we directly choose 8 frames from 128 frames, so it is generalized from smaller numbers to larger numbers. It is aligned with most of the existing Video-LLMs, i.e., trained primarily on short videos, yet they can also generalize to long videos during testing, e.g., VILA and LLaVA-One-Vision [1][2]. In the case of Frame-Voyager, for instance, considering temporal reasoning tasks where the question requires locating the beginning of a video (case study shown in Fig. 6), the Frame-Voyager consistently focuses on the video openings regardless of total duration. Similarly, for content identification tasks, e.g., finding ingredients as shown in Fig. 9, it selects relevant frames containing ingredients independent of video length. 

As mentioned in our paper (Lines 430-431), “as the number of extracted frames increases, the performance gap between our method and uniform sampling narrows”. There are mainly two factors, i.e., dataset characteristics and model capacity, that contribute to this pattern observed. We conduct further analysis below for clarification.

1. **Dataset Characteristics**: Our case studies (Fig. 6, Lines 445-485; Fig. 9, Fig. 10, Lines 922-1022) show that 8 or fewer frames selected by Frame-Voyager are sufficient to answer some of the questions. Additional frames may introduce redundant or noisy information for some question types, reducing the model performance (Lines 037–040). This finding is consistent with LongVA's results [3].
2. **Model Capability** [suggested by the Reviewer ```pMFu```]: Given the selected frame combination, the Video Question Answer performance is determined by the inference model, e.g., the frozen VILA used in our paper. For example, in Figure 5 (Lines 402-420), it is shown that VILA is generally not good at addressing the OCR and counting problems using any frame selection methods (uniform sampling, CLIP, or ours).

Due to the model capability limitation of VILA, we augment additional experiments based on another SOTA Video-LLM, i.e., LLaVA-One-Vision-7B. We vary the number of frames (while keeping the candidate frame number M = 128). The results on the Video-MME (without subtitles) benchmark are presented as follows.

| **_LLaVA-One-Vision-7B_** | **2 Frames** | **4 Frames** | **8 Frames** | **16 Frames** | **32 Frames** | **64 Frames** | **128 Frames** |
|---------------------------|--------------|--------------|--------------|---------------|---------------|---------------|----------------|
| **Uniform Sampling**      | 44.3         | 49.6         | 53.3         | 57.2          | 58.2          | 58.0          | 58.2           |
| **Frame-Voyager**         | 51.4         | 54.8         | 57.5         | 59.2          | 60.4          | 60.5          | 58.2           |
| **Gains**                 | +7.1         | +5.2         | +4.2         | +2.0          | +2.2          | +2.5          | Same Input     |

It is observed that Frame-Voyager shows **consistently better performance when varying the number of frames from 2 to 64**. While the uniform sampling method plateaus at 32 frames (58.2), our Frame-Voyager continues to improve up to 64 frames. Notably, Frame-Voyager achieves with 16 frames (59.2) what uniform sampling requires 32 or more frames to approach (58.2).

[1] VILA: On Pre-training for Visual Language Models. CVPR 2024

[2] ​​LLaVA-OneVision: Easy Visual Task Transfer. ArXiv 2024.

[3] Long Context Transfer from Language to Vision. ArXiv 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123067322

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response of First Round Rebuttal (3/4)

comment:
**3.** ```[qbz8, ULTS]``` **Efficiency & Scalability on Data Collection**: How can we make our data collection pipeline more cost-efficient at scale?

First, we would like to highlight that our data collection process is automated and **one-off**, allowing the resulting data (generated by VILA-8B) to be **directly used** by other Video LLMs, i.e., results of VILA-40B in Table 1 (Lines 292-293) and results of LLaVA-One-Vision-7B in the “1. Generalization on Different Backbones” of this general response. We will open-source our algorithm and also the generated dataset after the paper's acceptance.

Next, for rebuttal, we implement two straightforward approaches to address efficiency concerns and reduce computational resource consumption.

* **Method (1)**: We attempt some optimization strategy on our data collection process, revealing the potential for efficiency improvements. Specifically, we conduct dynamic pruning by filtering combinations based on **frame-to-frame similarity** and **temporal proximities among frames**. This strategy helps us discard the frame combinations containing multiple similar and redundant frames. It reduces the average number of combinations per sample to **14% on the VideoChatGPT dataset with processing time reduced to ~5 hours using 32 A100 GPUs** and to **27% on the NextQA dataset with ~15 minutes using 8 A100 GPUs**. 

* **Method (2)**: Following the Reviewer ```qbz8```'s suggestion, we conduct experiments by using CLIP to identify relevant frames. Building upon Method (1), we use CLIP to compute the similarity between all frames and the question, and rank them accordingly. We mark lower-ranked frames as irrelevant.  We believe that only a small portion of video frames directly relate to the question, while the majority of frames are irrelevant. As a result, most frame combinations only containing irrelevant frames can be discarded during the training process. Based on this pruning strategy, we filter out most combinations consisting of only irrelevant frames while retaining a few as low-ranking training samples. **This approach can further reduce data collection time by 60-70%**.

The comparison among the original version data collection, +Method (1), and +Method (1) & (2) is shown below. The backbone model is kept as VILA-8B. **The results demonstrate that the data construction time of Frame-Voyager could be largely reduced to just 4.4% of the original version, while maintaining comparable performance across all benchmarks**. We will include these results in the revised version, and we leave the exploration of more sophisticated efficiency improvements to future work.

| **_Training Data Construction_** | **GPU Hours (VideoChatGPT + NextQA)** | **Time Costs (%)** | **Video-MME (%)** | **MLVU (%)** | **NextQA (%)** | **ActivityNetQA (%)** |
|-------------------------|---------------|---------------------|---------------|----------|------------|-------------------|
| **Original Version**    | 1280+8        | 100%                | 50.5          | 49.8     | 51.1       | 51.6              |
| **+Method (1)**          | 160+2         | 12.6%               | 50.6          | 50.0     | 50.8       | 52.0              |
| **+Method (1) & (2)**        | 56+0.8        | **4.4%**                | 50.5          | 49.7     | 51.0       | 51.9              |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732123153222

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response of First Round Rebuttal (4/4)

comment:
**4.** ```[pMFu, qbz8]``` **Overall Efficiency**: How much additional computational overhead is required for training and inference?

We conduct a comprehensive analysis of running time, model parameters, and latency comparisons between the baseline (uniform sampling) and our method, as presented in Appendix C (Lines 219-226).
* Parameter size: our method adds only several MLPs, increasing the total parameter count by just a negligible 0.2% compared to the original model. This minimal parameter increase translates to a modest impact on training and inference resources. 
* Training efficiency: our Frame-Voyager module requires 64 GPU hours for separate training, representing only 1.25% additional training time compared to the original model's 5.1k GPU hours as in [1]. 
* Inference speed: we evaluate latency using 100 randomly sampled examples from Video-MME. Our method averages 1.696 seconds per example, and the baseline is 1.329 seconds per sample. It shows a 27.6% increase in inference time.
* Additionally, we add the memory usage comparison based on the Reviewer ```pMFu```’s suggestion. The memory footprint remains similar: our Frame-Voyager uses 23,069 MiB and the baseline uses 22,425 MiB, making a difference of 2.9%.

|                        | **Parameters Overhead** | **Training Hours** | **Latency** | **Memory** |
|------------------------|-------------------------|--------------------|-------------|------------|
| **VILA**       | -                       | 5.1k GPU Hours     | 1.329s      | 22,425 MiB |
| **VILA+Frame-Voyager** | +0.2%                   | 64 GPU hours       | 1.696s      | 23,069 MiB |

These metrics demonstrate that Frame-Voyager only introduces a tiny computational overhead. The minimal increases in parameter count and memory usage suggest that our method scales efficiently while achieving superior video understanding capabilities.

[1] VILA: On Pre-Training for Visual Language Models. CVPR 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733049538833

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response and Revision of Manuscript

comment:
We sincerely thank the reviewers for their valuable feedback!

In this post:

*  We summarize and highlight the key strengths of our work as noted by the reviewers.
*  We summarize the changes to the updated PDF document (marked in blue). Additional experimental results after the PDF revision deadline will be added in the final version.


****(1) Key Strengths****
* **Motivation**
  - ```pMFu```: "*reasonable extension from previous keyframe selection work*"
  - ```6u8S```: "*motivation is clear and easy-to-follow*"
  - ```qbz8```: "*a critical challenge in video understanding…making this problem a key focus for advancing the field*"

* **Method**
  - ```pMFu```: "*good contribution to the keyframe selection in video-language studies*"
  - ```qbz8```: "*novel formulation of frame selection as a ranking problem is compelling…offering a fresh and efficient perspective on the task*"
  - ```ULTS```: "*an efficient keyframe data construction method*", "*a novel approach for improving Video Large Language Models*", "*a valuable tool for video understanding*"

* **Experiments & Results**
  - ```pMFu```: "*extensive experiments across diverse popular benchmarks to show their effectiveness*"
  - ```6u8S```: "*analysis is comprehensive*"
  - ```ULTS```: "*FRAME-VOYAGER achieves significant performance in several VQA datasets*, *strong performance across multiple datasets proving the robustness of the approach*"

* **Presentation**
  - ```6u8S```: "*presentation of the paper is of high quality*"
  - ```qbz8```: "*well-written, with a clear explanation of the motivation and methodology*"



****(2) Changes to the PDF****

* **Section 2 Related Work**
  - ```pMFu```: Adding related works in frame selection and video grounding.
  - ```6u8S```: Adding related works leveraging LLM agents. 

* **Section 3 Frame-Voyager**
  - ```pMFu```: Adding the explanation why our loss-based desigen is able to avoid language prior shortcuts in Section 3.1.

* **Section 4 Experiments**
  - ```pMFu,6u8S```: Augmenting results of Frame-Voyager on LLaVA-One-Vision in Table 1.
  - ```ULTS```: Modifying the results on short video benchmarks with higher number of frames in Table 1.
  - ```pMFu,6u8S,qbz8```: Adding more frame selection baselines in Table 2.
  - ```pMFu,6u8S,qbz8,ULTS```: Frame-Voyager generalized to more frames with LLaVA-One-Vision in Figure 3b and related explanation in Research Question 3.
  - ```qbz8,ULTS```: A new Research Question 7 elaborating the efficiency and scalability of Frame-Voyager when processing more frames.

* **Appendix**
  - ```pMFu,6u8S,qbz8```: Updating implementations for the new added frame selection baselines in Appendix A.
  - ```pMFu,qbz8```: Adding extra memory usage of Frame-Voyager in Appendix C.
  - ```ULTS```: Adding results on three additional benchmarks in Appendix D.
  - ```6u8S,qbz8```: Dynamic frame selection experiment in Appendix F.
  - ```qbz8,ULTS```: Optimization of data collection pipeline in Appendix G.
  - ```pMFu```: Case study on the ranking data for training in Appendix H.

* **References**
  - ```[1]``` "On Calibration of Modern Neural Networks". ICML 2017. Guo et al.
  - ```[2]``` "Self-Adaptive Sampling for Accurate Video Question Answering on Image Text Models". Findings of NAACL 2024. Han et al.
  - ```[3]``` "Language Models (Mostly) Know What They Know". arXiv 2022. Kadavath et al.
  - ```[4]``` "MVBench: A Comprehensive Multi-modal Video Understanding Benchmark". CVPR 2024. Li et al.
  - ```[5]``` "TimeCraft: Navigate Weakly-Supervised Temporal Grounded Video Question Answering via Bi-directional Reasoning". ECCV 2024. Liu et al.
  - ```[6]``` "EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding". NeurIPS 2023. Mangalam et al.
  - ```[7]``` "VideoAgent: Long-form Video Understanding with Large Language Model as Agent". ECCV 2024. Wang et al.
  - ```[8]``` "ViLA: Efficient Video-Language Alignment for Video Question Answering". ECCV 2024. Wang et al.
  - ```[9]``` "VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos". arXiv 2024. Wang et al.
  - ```[10]``` "STAR: A Benchmark for Situated Reasoning in Real-World Videos". NeurIPS 2021. Wu et al.
  - ```[11]``` "Can I Trust Your Answer? Visually Grounded Video Question Answering". CVPR 2024. Xiao et al.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Frame selection is a popular paradigm to improve the computational cost of video-language models. This paper improves upon prior works like Sevila by not choosing frames independently, but by considering temporal relations among frames (the authors address the combinatorial search space of this task by phrasing it as ranking instead).

The rebuttal and revised version of the paper addressed most of the reviewers' initial concerns, and each reviewer is now positive about the paper. Therefore, the decision is to accept the paper.