Frame-Voyager: Learning to Query Frames for Video Large Language Models

## Abstract

Abstract Video Large Language Models (Video-LLMs) have made remarkable progress in video understanding tasks.
However, they are constrained by the maximum length of input tokens, making it impractical to input entire videos.
Existing frame selection approaches, such as uniform frame sampling and text-frame retrieval, fail to account for the information density variations in the videos
or the complex instructions in the tasks,
leading to sub-optimal performance.
In this paper, we propose Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task.
To train Frame-Voyager , we introduce a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM.
Given a video of M 𝑀 M italic_M frames, we traverse its T 𝑇 T italic_T -frame combinations, feed them into a Video-LLM, and rank them based on Video-LLM’s prediction losses.
Using this ranking as supervision, we train Frame-Voyager to query the frame combinations with lower losses.
In experiments, we evaluate Frame-Voyager on four Video Question Answering benchmarks by plugging it into two different Video-LLMs.
The experimental results demonstrate that Frame-Voyager achieves impressive results in all settings,
highlighting its potential as a plug-and-play solution for Video-LLMs.

## 1 Introduction

Recent studies (Liu et al., ; ; Li et al., ; Lin et al., ) explore integrating Large Language Models (LLMs, Stiennon et al. (); Gao et al. (); OpenAI (); Touvron et al. (); Jiang et al. (); Yang et al. ()) with visual foundation models (*e.g.,* Vision Transformer (ViT, Dosovitskiy et al. ()) and cross-modal projectors (Li et al., ; Lin et al., ; Liu et al., ).
In this paper, we focus on Video-LLMs.
Existing Video-LLMs (Zhang et al., ; Cheng et al., ; Li et al., ) usually treat the video as a sequence of image frames.
The key challenge is that the entire video can not be fed into the model due to LLMs’ token length limitation (Xue et al., ).
Meanwhile, arbitrarily increasing the token length of the model (Miao et al., ; Wan et al., ; Xiong et al., ; Zhang et al., )
may lead to the “lost-in-the-middle” issue (Liu et al., ) and introduce significant computational complexity.

To mitigate this issue, some efforts propose to select only a subset of frames as input, *e.g.,* through uniform sampling (Liu et al., ; Lin et al., ; Cheng et al., ) or text-frame matching (Liang et al., ; Wang et al., ; Yu et al., ).
The uniform sampling strategy evenly samples frames in videos, while text-frame matching typically retrieves a set of relevant frames by calculating semantic similarities, *e.g.,* using CLIP (Radford et al., ), between the query and each frame.
However, the uniform sampling fails to account for the information density variations in the videos. For instance, in the video question answering task, answering different questions may rely on distinct video segments or frames (Fu et al., ).
Meanwhile, text-frame matching is inadequate for complex video understanding tasks that require multi-frame or temporal reasoning, such as tracking the progression of an action or understanding cause-and-effect relationships over time.
For instance, in the video summarization task, simply matching frames might overlook the subtle transitions that connect scenes, while in temporal reasoning tasks—such as answering why does the woman need to drink water at the beginning of the video?—it is crucial to concentrate on the beginning part of the video.
These matching-based methods fail to account for these frame-to-frame interactions and the relative positional information essential for a comprehensive understanding of the video.
To solve these problems, we introduce an innovative approach named Frame-Voyager that
learns to query the subset of frames in a combinational manner, rather than retrieving individual frames separately.
This capability is essential for understanding dynamic scenes and the global context of events.

To train Frame-Voyager, we encounter two main challenges:
1) High Learning Complexity: Learning the optimal combination of frames poses a combinatorial optimization problem. For instance, selecting $8$ frames from a $128$-frame video yields around $1.4\times 10^{12}$ possible frame combinations.
2) Lack of Labeled Data: There are currently no available datasets to facilitate the learning of such combinatorial problems in videos. We must address the question of how to construct a training dataset with minimal human effort.
To address the first challenge, we formulate the combinatorial problem as a ranking task. Specifically, we train the model to rank the given frame combinations (*i.e.,* subsets) based on supervised ranking scores, which proves to be more efficient than forcing the model to search the optimal frame combination in a huge search space (Cao et al., ).
Assume that for each frame combination, we have an annotation of ranking based on its usefulness in eventually generating the correct answer (by addressing the second challenge). Given a batch of frame combinations, the model learns to assign a higher reward to those with higher rankings. In other words, the model’s objective is to maximize the reward for higher-ranked frame combinations.
To tackle the second challenge, we propose leveraging a pre-trained Video-LLM to generate a ranking score for each frame combination, based on the prediction loss when the combination is input together with the query into this Video-LLM. The intuition is that a more effective frame combination will result in a lower prediction loss, indicating a higher likelihood of generating correct answers. Specifically, assume the original video has $M$ frames and the Video-LLM accepts only $T$ frames for input. We evaluate all frame combinations (*i.e.,* the total number is $\mathcal{C}(M,T)$) and rank them based on the prediction losses provided by the Video-LLM. The frame combinations with lower losses rank higher.
These sorted frame combinations are then used for training Frame-Voyager. Finally, the trained Frame-Voyager is used for choosing a frame combination to input into Video-LLMs for downstream tasks.

To evaluate Frame-Voyager, we plug it into two versions of the state-of-the-art Video-LLM named VILA (VILA-8B and VILA-40B, Lin et al. ()) and conduct experiments on four widely-used Video Question Answering benchmarks, Video-MME (Fu et al., ), MLVU (Zhou et al., ), NextQA (Xiao et al., ) and ActivityNet-QA (Yu et al., ). Experiment results show that using the frame combination “chosen” by Frame-Voyager achieves significant performance improvements, compared to the conventional uniform sampling and text-frame retrieval (*i.e.,* individual text-frame matching) methods, especially for the cases requiring reasoning and information synopsis in long videos.
Our contributions
are thus two-fold: 1) We unveil the importance of combinational frame selection for video understanding tasks and propose an efficient method Frame-Voyager that learns to do this frame selection automatically. The Frame-Voyager itself is a plug-and-play module that can be applied to different Video-LLM architectures; 2) We formulate and learn Frame-Voyager in a task of ranking frame combinations and introduce an automatic labeling pipeline to generate training datasets.

## 2 Related Work

Transformer-based LLMs have revolutionized the field of natural language processing, achieving remarkable advancements by scaling up model sizes and expanding pre-training datasets (Brown et al., ; OpenAI, ; Chowdhery et al., ; Touvron et al., ).
Researchers further extend LLMs into a multi-modality manner by fusing multi-modality information into the inputs of LLMs (Zhang et al., ; Liu et al., ). In this work, we focus on Video-LLMs, and we consider the most widely-used structure (Lin et al., ), where frames are adapted as visual tokens and then temporally fed into LLMs alongside text tokens in an auto-regressive way.

However, in existing Video-LLM models, a single frame is typically represented by $64$-$256$ visual tokens (Liu et al., ; Lin et al., ; Cheng et al., ). Due to the input limitations of LLMs, the number of frames that can be processed by Video-LLM models is often constrained.
Some studies apply techniques for handling long LLM inputs to support more frames (Miao et al., ; Wan et al., ; Xiong et al., ; Xue et al., ; Song et al., ), but this approach significantly increases computational complexity and can lead to issues such as the “lost-in-the-middle” effect and hallucinations (Liu et al., ).
Other studies, while keeping the frame input limit unchanged, use alternative sampling strategies instead of the default uniform sampling to obtain higher-quality frames as inputs.
Some initial attempts focus on identifying transition frames (Lu & Grauman, ; Rochan et al., ; Rochan & Wang, ) or using frame clustering (Liang et al., ) to find central frames, but these methods often overlook the information from the query.
Subsequent research view this problem as an individual text-frame matching task (Wang et al., ; Yu et al., ), attempting to select frames that are semantically closest to the query.
However, this method is still sub-optimal as it ignores frame-to-frame relationships, failing in complex video understanding tasks that require multi-frame or temporal information.
Therefore, to the best of our knowledge, our method is the first to consider the combination of frames as a whole, aiming to find the optimal combination that can best answer the query under the constraint of frame length limitations.

## 3 Frame-Voyager

In the research context of Video-LLMs, typical video-language tasks such as video understanding, summarization, and reasoning can be formulated as video question answering tasks.
The input and output of Video-LLMs are thus in the format of (*video*, *query*) and *answer*, respectively.
The *video* here is typically not the entire video but rather a subset of frames, *i.e.,* frame combination, due to the token length limitations.
Our research question is thus how to get the “optimal” subset of frames in this *video* to answer the *text query* correctly.
Our method is called Frame-Voyager. To train it with manageable complexity, we downsample the entire video to a fixed number of $M$ frames using uniform sampling. Then, we use Frame-Voyager to evaluate $T$-frame combinations sampled from these $M$ frames, where $M\gg T$.
The training is supervised and the labeled data are generated by a pre-trained reference Video-LLM, as elaborated in Section 3.1.

Given that the optimal combination of frames must be identified in a huge search space, one may wonder: *How is Frame-Voyager’s training supervised?*
We answer this question by formulating the problem as a ranking task (for which it is easy to get labeled data, *i.e.,* the second challenge), rather than looking for “optimal” (as it is non-trivial to learn, *i.e.,* the first challenge).
In the following subsections,
we elaborate on this task by introducing the pipeline of ranking data collection (Section (3.1)) as well as the training and inference processes of Frame-Voyager (Section (3.2)).

### 3.1 Data Collection

We propose a human-free data collection and annotation pipeline for frame combinations.
The overall process is demonstrated in Figure 1.

Figure: Figure 1: The data collection pipeline of Frame-Voyager. Given a video of $M$ frames, we traverse its $T$-frame combinations, feed them into a Video-LLM, and rank them based on the reference Video-LLM’s prediction losses. At last, we train Frame-Voyager to query the frame combinations with lower losses. Please note that we omit filtering steps in this figure for clarity. $\mathcal{C}(M,T)$ is the binomial coefficient representing the number of ways to choose $T$ items from $M$.
Refer to caption: x1.png

Our pipeline is based on a simple intuition: if one frame combination is better than another, it will produce a lower language modeling loss when used as input to any trained Video-LLM. For each (*video*, *query*) pair,
we evaluate all possible combinations of $T$ frames selected from a total of $M$ video frames, resulting in $\mathcal{C}(M,T)$ combinations, where $\mathcal{C}(M,T)$ is the binomial coefficient representing the number of ways to choose $T$ items from $M$.
Each combination, along with the *query*, is then input into a trained reference Video-LLM to calculate the combination loss, *i.e.,* language modeling loss against the ground-truth *answer*.
We collect the loss values for each combination and rank them from best to worst by sorting the losses in ascending order.
It is worth noting that as $M$ increases, the potential number of combinations $\mathcal{C}(M,T)$ grows exponentially, making exhaustive traversal computationally infeasible. For example, when $M=64$ and $T=8$, the total number of combinations is approximately $\mathcal{C}(64,8)\approx 4\times 10^{9}$.
Considering that the majority of the training data are with short videos, we use smaller combinations during training,
such as $\mathcal{C}(16,2)$ or $\mathcal{C}(32,4)$.
We observe from experiments that models trained with smaller combinations
exhibit generalization capabilities when larger values of $M$ and $T$ are used during inference for longer video or complex reasoning.
The specific choices of $M$ and $T$ employed in our experiments are detailed in Section 4.1.

To improve the ranking efficiency, we apply two filters for (*video*, *query*) pairs: we filter out 1) the pairs with an excessively high averaged loss, as these pairs may represent outlier cases or weak video-query correlations; and
2) the pairs with low variance in the losses across their combinations, as these pairs are not sensitive to the quality of combinations, *e.g.,* the correct answer may be generated solely based on the Video-LLM’s inherent language prior without referring to any input content.

As a result, for each (*video*, *query*), we can obtain rankings for all $\mathcal{C}(M,T)$ frame combinations, *i.e.,* the combination ranking data in Figure 1.
Each item in combination ranking data contains the indices of frames within the combination and its corresponding rank.
For instance, given $M=8$ and $T=2$, the combination $\textsl{Comb}=\langle\{\textsl{Frame}^{2},\textsl{Frame}^{5}\},\#6\rangle$ means that it contains the $2$-th and $5$-th frames from $M$ candidate frames, and it ranks at the $\#6$ position given all traversed $\mathcal{C}(8,2)$ combinations.

### 3.2 Model Training and Inference

In this section, we elaborate on the training and inference details for Frame-Voyager.
Give a pre-trained Video-LLM, *e.g.,* VILA-8B (Lin et al., ), we implement Frame-Voyager as a lightweight plug-and-play module on it. For training this module, we use the $M$ candidate frames and query as input to the model, and our Frame-Voyager is thus learned to capture both query-frame and frame-to-frame relationships. The overall process is demonstrated in Figure 2.

Figure: Figure 2: Training and inference processes of Frame-Voyager. In the training, Frame-Voyager is fed with all $M$ candidate frames and learns to rank $K$ sampled combinations from pre-generated combination ranking data in Section 3.1. Each combination contains $T$ frames. As for the inference, Frame-Voyager selects top-$T$ frames with highest rewards to form the predicted frame combination. Note that there is no parameter to update during the inference.
Refer to caption: x2.png

Frame and Query Features.
Given uniformly sampled $M$ frames as candidate frames for each input (video, query) pair, we utilize the visual encoder followed by the Video-LLM projector to convert each frame into a sequence of visual tokens. Each visual token has the same size as the word token embedding of the LLM backbone.
Thus, for $M$ frames, we have an initial feature map $\bm{X}^{\prime}\in\mathbb{R}^{M\times N\times d}$, where $M$ is the number of frames, $N$ denotes the number of visual tokens per frame, and $d$ represents the feature dimension. We further perform token-wise average pooling before feeding them into LLMs for computational efficiency, *i.e.,* averaging $N$ visual tokens, on the initial frame feature map $\bm{X}^{\prime}\in\mathbb{R}^{M\times N\times d}$ to obtain refined frame feature $\bm{X}\in\mathbb{R}^{M\times d}$. Concurrently, the query can be tokenized as $Q$ tokens, and filled with word embeddings of the backbone LLM. The operation will produce $\bm{Y}\in\mathbb{R}^{Q\times d}$ for textual information.

Cross-Modality Interaction Modeling.
To model both query-frame and frame-to-frame interactions, we leverage the bottom layers of LLMs. These transformer layers, which utilize the self-attention, are well pre-trained for vision-language tasks (Vaswani et al., ; Stan et al., ).
Thus we concatenate the frame feature $\bm{X}$ and query feature $\bm{Y}$, and feed them together into LLMs’ bottom layers. Importantly, all $M$ candidate frames are processed simultaneously, rather than individually feeding $T$ frames per combination, as Frame-Voyager needs to model the entire set of $M$ candidate frames to capture the relationships within frames. The generated cross-attentive multimodal features are denoted as $\bm{X}_{\texttt{BL}}\in\mathbb{R}^{M\times d}$ for frames and $\bm{Y}_{\texttt{BL}}\in\mathbb{R}^{Q\times d}$ for the query. The BL refers for “Bottom Layer”.

Frame Combination Reward. As mentioned, we formulate the combinatorial problem as a ranking task. Thus, for each frame combination, we need to compute its combination reward for further ranking-based training.
First, we apply the token-wise average pooling on the cross-attentive multimodal features $\bm{Y}_{\texttt{BL}}$ of query, and feed features into a feed-forward network (FFN).
The generated final query feature is $\bm{Y}_{\texttt{FFN}}\in\mathbb{R}^{h}$, where $h$ is the output dimension of the feed-forward network.
The frame feature map $\bm{X}_{\texttt{BL}}$ is also converted by another feed-forward network to get the final features $\bm{X}_{\texttt{FFN}}\in\mathbb{R}^{M\times h}$.
Then we simply measure the reward for a given frame combination as the averaged reward of the frames within the combination as:

$$ $r(\textsl{Comb})=\mathbb{E}_{i\in\textsl{Comb}}[{r(\textsl{Frame}^{i})}].$ (1) $$

The reward for $i$-th frame (*i.e.,* $i$-th row in $\bm{X}_{\texttt{FFN}}$) with respect to the query and other frames, is computed as:

$$ $r(\textsl{Frame}^{i})=\mathrm{cosine}(\bm{Y}_{\texttt{FFN}},\bm{X}_{\texttt{ FFN}}^{i}).$ (2) $$

Training. Inspired by the reward ranking loss function in (Ouyang et al., ), we train Frame-Voyager via reward modeling to align its outputs with the optimal combination.
To be specific, given the combination ranking data generated by the pipeline in Section 3.1, we uniformly sample $K$ ranked combinations, with each combination consisting of $T$ frames.
Any two combinations sampled from the $K$ frame combinations are selected to form a training pair ($\mathcal{C}(K,2)$ training pairs in total), with the frame combination having a lower loss in each pair designated as the chosen sample and the one with a higher loss as the rejected sample.
Overall, given $K$ ranked combinations, the training objective, *i.e.,* reward ranking loss, is calculated as:

$$ $\mathcal{L}=-\frac{1}{\mathcal{C}(K,2)}\mathbb{E}\bigg{[}\log\Big{(}\delta\big {(}r(\textsl{Comb}_{w})-r(\textsl{Comb}_{l})\big{)}\Big{)}\bigg{]},$ (3) $$

where $\textsl{Comb}_{w}$ denotes the preferred frame combination, and $\textsl{Comb}_{l}$ represents the rejected one.

Inference.
During inference, we plug the conventional Video-LLM models with our Frame-Voyager module.
Suppose that we uniformly sample $M$ candidate frames and expect $T$ frames as the visual input for Video-LLMs.
The process remains consistent as the training procedure until the computation of the reward for frames.
After that, we adopt the most efficient way to sample $T$ frames, *i.e.,* selecting the top $T$ frames with the highest rewards while maintaining their original temporal order in the video.
The reason is that each reward of the frame here is optimized under combinational ranking supervision and incorporates the interaction information of the query and other frames.

## 4 Experiments

### 4.1 Experiment Settings

Backbone Models.
We use two variants of the state-of-the-art Video-LLM called VILA (Lin et al., ): VILA-8B and VILA-40B. VILA-8B employs SigLIP (Zhai et al., ) as its visual encoder and Llama3-8B (Dubey et al., ) as its LLM backbone, while VILA-40B utilizes InternViT-6B (Chen et al., ) for visual encoding and Yi-34B (Young et al., ) as its LLM backbone.

Training Data. To ensure that Frame-Voyager is trained on a diverse range of questions, we examine the video datasets used in VILA (Lin et al., ) and LLaVA-OneVision (Li et al., ). We select the training set of NextQA (Xiao et al., ) and VideoChatGPT (Maaz et al., ), on which we apply our proposed pipeline to create a training dataset for Frame-Voyager.
We empirically set the values of $M$ and $T$ based on the video length and question difficulty in these two datasets.
Specifically, for NextQA, which features shorter videos and simpler questions, we select $16$ candidate frames per video and explore all $120$ possible $2$-frame combinations. For VideoChatGPT, we select $32$ candidate frames from each video and evaluate all $35,960$ possible $4$-frame combinations.
During the filtering process, we exclude the (video, query) pairs with an average loss larger than $7$ and select pairs within the top $30\%$ and $10\%$ ranked by the variance of losses for the two datasets, respectively.
After filtering, we obtain about $5,500$ and $7,000$ samples for these two datasets.

**Table 1: Comparing Video-LLMs with and without Frame-Voyager as an additional module. Except for ours (+Frame-Voyager) and the models with ^∗, all results are copied from the related papers of benchmarks or models. The two VILA baselines utilize uniform sampling. For the Video-MME benchmark, we report results under two standard settings: without subtitles (no sub.) and with subtitles (sub.). ANQA refers to ActivityNetQA. Accuracy sign $\%$ is omitted for clarity.**
| Model | LLM<br>Size | Video-MME (no sub. / sub.) | MLVU | ANQA | NextQA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Overall | Short | Medium | Long |  |  |  |  |  |
| Video Length | 17min | 1.3min | 9min | 41min | 12min | 2min | 0.8min |  |
| Video-LLaVA | 7B | 39.9 / 41.6 | 45.3 / 46.1 | 38.0 / 40.7 | 36.2 / 38.1 | 47.3 | 45.3 | - |
| Qwen-VL-Chat | 7B | 41.1 / 41.9 | 46.9 / 47.3 | 38.7 / 40.4 | 37.8 / 37.9 | - | - | - |
| ST-LLM | 7B | 37.9 / 42.3 | 45.7 / 48.4 | 36.8 / 41.4 | 31.3 / 36.9 | - | 50.9 | - |
| VideoChat2 | 7B | 39.5 / 43.8 | 48.3 / 52.8 | 37.0 / 39.4 | 33.2 / 39.2 | 44.5 | 49.1 | - |
| Chat-UniVi-V1.5 | 7B | 40.6 / 45.9 | 45.7 / 51.2 | 40.3 / 44.6 | 35.8 / 41.8 | - | 46.1 | - |
| VideoLLaMA2 | 7B | 47.9 /    - | 56.0 /    - | 45.4 /    - | 42.1 /    - | - | 49.9 | - |
| LLaVA-NeXT-QW2 | 7B | 49.5 /    - | 58.0 /    - | 47.0 /    - | 43.4 /    - | - | - | - |
| LongVILA${}^{\text{128frm}}$ | 8B | 49.2 /    - | 60.2 /    - | 48.2 /    - | 38.8 /    - | - | - | - |
| LongVILA${}^{\text{256frm}}$ | 8B | 50.5 /    - | 61.8 /    - | 49.7 /    - | 39.7 /    - | - | - | - |
| \hdashline VILA^∗ | 8B | 47.5 / 50.0 | 57.8 / 61.6 | 44.3 / 46.2 | 40.3 / 42.1 | 46.3 | 44.3 | 48.5 |
| +Frame-Voyager | 8B | 50.5 / 53.6 | 60.3 / 65.0 | 47.3 / 50.3 | 43.9 / 45.3 | 49.8 | 51.1 | 51.6 |
| VideoLLaMA2 | $\text{8}\!\!\times\!\!\text{7B}$ | 47.9 / 49.7 | -    /    - | -    /    - | -    /    - | - | 50.3 | - |
| VITA | $\text{8}\!\!\times\!\!\text{7B}$ | 55.8 / 59.2 | 65.9 / 70.4 | 52.9 / 56.2 | 48.6 / 50.9 | - | - | - |
| LLaVA-NeXT-Video | 34B | 52.0 / 54.9 | 61.7 / 65.1 | 50.1 / 52.2 | 44.3 / 47.2 | - | 58.8 | - |
| \hdashline VILA^∗ | 34B | 58.3 / 61.6 | 67.9 / 70.7 | 56.4 / 59.8 | 50.4 / 52.1 | 57.8 | 47.7 | 67.4 |
| +Frame-Voyager | 34B | 60.0 / 63.8 | 70.3 / 73.1 | 58.3 / 62.7 | 51.2 / 55.7 | 61.1 | 55.2 | 69.4 |

Benchmarks. We evaluate Frame-Voyager on four widely-adopted video benchmarks: Video-MME (Fu et al., ), MLVU (Zhou et al., ), NextQA (Xiao et al., ) and ActivityNet-QA (Yu et al., ).
The former two evaluation datasets are tailored for long video assessments, while the latter two focus on short videos.
For the former (the latter), we uniformly downsample the video to $128$ ($32$) candidate frames and each time select $8$ ($2$) frames to compose a frame combination.
The LMMs-Eval Library (Li et al., ) is used for evaluation, and accuracy is reported across all benchmarks. Note that we report the accuracy scores of Video-MME under both without (no sub.) and with (sub.) subtitles settings.

Implementation Details.
We reuses one bottom layer from the LLM for Frame-Voyager in Section 3.2.
During the Frame-Voyager dataset construction, VILA-8B is consistently adopted as the reference Video-LLM for generating loss due to resource limitations.
During training, we set $K\!=\!4$ for sampling combination ranking data.
VILA-8B is trained using DeepSpeed (Aminabadi et al., ) ZeRO2 with $8$ H100 GPUs, while VILA-40B is trained using ZeRO3 setting with $32$ H100 GPUs. The batch size (with accumulation) is set to $64$ and the learning rate is $1e^{-3}$. The training of Frame-Voyager is conducted over $40$ epochs requiring approximately $8$ hours for VILA-8B whereas over $20$ epochs for VILA-40B, taking around $20$ hours. All model inferences are performed on $8$ H100 GPUs.

Table 2: RQ1. Accuracies ($\%$) for using different frame extraction methods on Video-MME (without subtitles). Q: whether query information is used. Comb: whether considering frame combination.
Q
Comb
Video-MME
VILA-8B (Uniform)
✘
✘
47.5
+ RGB Histogram
✘
✘
45.9
+ Edges Change Ratio
✘
✘
47.3
+ Optical Flow
✘
✘
46.7
+ Katna
✘
✘
45.7
+ CLIP
✔
✘
48.5
+ VILA-Embedding
✔
✘
48.8
+ Frame-Voyager
✔
✔
50.5
Table 3: RQ2. The ablation study ($\%$) on different dataset collection methods. All results are evaluated on Video-MME (without subtitles). “Comb.”: frame combination.
Video-MME
(1) NextQA
48.7
(2) VideoChatGPT w/ Filtering
49.1
(3) VideoChatGPT w/o Filtering
48.3
(4) All Data + Top-$1$ Rank Comb.
48.9
(5) All Data + $K\!=\!2$
49.4
(6) All Data + $K\!=\!8$
49.7
Frame-Voyager ($K\!=\!4$)
50.5

### 4.2 Results and Analysis

Comparison with state-of-the-art methods.
Table 1 presents the comparison of Frame-Voyager with leading Video-LLMs, organized by the size of their backbone LLMs. For models with LLMs of 8B parameters or fewer, we evaluate Video-LLaVA (Lin et al., ), Qwen-VL-Chat (Bai et al., ), ST-LLM (Liu et al., ), VideoChat2 (Li et al., ), Chat-UniVi-V1.5 (Jin et al., ), VideoLLaMA2 (Cheng et al., ), LLaVA-NeXT-QW2 (Liu et al., ) and LongVILA (Xue et al., ). For larger models, we compare Frame-Voyager with VideoLLaMA2 (Cheng et al., ), VITA (Fu et al., ) and LLaVA-NeXT-Video (Zhang et al., ).

Among the models with LLMs of 8B parameters or fewer, Frame-Voyager achieves the best overall performance. On the Video-MME benchmark (without subtitles), it outperforms the vanilla VILA-8B by $3.0\%$, with a notable $3.6\%$ gain on long videos.
Remarkably, Frame-Voyager, using only $8$ frames as input into VILA, surpasses the VILA variant LongVILA, which utilizes $128$ and $256$ frames and requires additional training and system design.
LongVILA improves its performance by inputting more frames, while Frame-Voyager improves through querying more informative frame combinations, without changing the frame length limits of VILA.
Frame-Voyager outperforms LongVILA even on extremely long videos (average $41$ minutes).
This suggests that simply increasing the input number of frames may not always lead to better performance, since incorporating more frames might introduce noises and irrelevant information. More frames also reduce computing efficiency.
Besides, among the larger Video-LLMs (with 8$\!\times$​7B and 34B LLM backbones), Frame-Voyager consistently brings notable improvements over the vanilla VILA and other state-of-the-art models. These results highlight the importance of selecting and utilizing the optimal information from video for efficient video understanding. Last but not least, VILA-8B and VILA-40B employ distinct vision encoders and LLM backbones, as outlined in Section 4.1. Thus, the consistent performance improvements indicate that Frame-Voyager functions as a plug-and-play module compatible with different Video-LLM architectures.

Research Question (RQ) 1: How does Frame-Voyager compare to other frame extraction methods?

To evaluate the effectiveness of Frame-Voyager on Video-LLMs, we conduct experiments with several baseline methods, all utilizing the same VILA-8B backbone, and the same number of frames for a fair comparison. We test rule-based shot boundary detection (SBD) methods, including Histogram (Sheena & Narayanan, ), Edges Change Ratio (Nasreen & Dr Shobha, ), and Motion (Wolf, ), which select frames based on significant transitions in texture, structure, and motion. Frames with the most substantial changes are chosen as extracted frames. We also include Katna(^1^11[https://github.com/keplerlab/katna](https://github.com/keplerlab/katna)), a cluster-based method that extracts histograms from all frames and uses K-means clustering to select most representative frames near cluster centers. In addition, two frame-text matching methods (Liang et al., ; Wang et al., ), VILA-Embedding and CLIP (Radford et al., ), are employed to retrieve frames by calculating cosine similarity between query inputs and individual frames, leveraging embedding layers from VILA’s LLM and CLIP. Implementation details are presented in Appendix A.

Table 4.1 presents the comparison results on Video-MME (without subtitles). Rule-based methods perform worse than uniform frame sampling, likely due to inherent biases in these techniques. For instance, optical flow methods prioritize motion-heavy frames, while RGB histogram methods emphasize texture changes. These approaches overlook the input *query* and thus often fail to answer the *query*.
Furthermore, although VILA Embedding and CLIP outperform uniform sampling, they still fall short compared to Frame-Voyager. Their frame-by-frame extraction approach lacks a holistic understanding of the video, making them struggle with complex tasks requiring temporal reasoning and comprehensive video comprehension. Overall, the proposed Frame-Voyager outperforms all the baselines, demonstrating superiority for frame subset selection and efficient video understanding.

Figure: Figure 3: RQ3. Accuracies ($\%$) of uniform sampling and Frame-Voyager on Video-MME (without subtitles) regarding number of frames.
Refer to caption: x3.png

Figure: Figure 5: RQ5. Accuracies ($\%$) of uniform sampling, CLIP, and Frame-Voyager on six question types of the Video-MME. For each type, the maximum and minimum results are marked.
Refer to caption: x5.png

RQ2: What is the impact of each component on data collection?

To evaluate the effectiveness of each component in our dataset construction phase, we conduct a series of ablation studies summarized in Table 4.1. Specifically, we investigate the impact of individual datasets (1-2), analyze the role of data filtering (2-3), and explore the effects of different data usage strategies during training (4-6).
In (4), we directly optimize the Frame-Voyager by the combination ranked highest.
In (5-6), we modify $K$, number of sampled combinations, during training.
The results of (1-2) demonstrate additive performance gains from each dataset and the significant distribution differences between the two datasets enhance the query diversity. The results of (3) underscore the critical role of data filtering in eliminating instances unsuitable for training Frame-Voyager. Experiments (4-6) provide some insights on data usage during training, revealing that varying the number of combinations can adversely affect reward computation. Moreover, results from (4) highlight the necessity of teaching Frame-Voyager to discern better from worse combination via reward modeling, rather than merely training it to identify the combination.

RQ3: How does the number of frames impact the performance of Frame-Voyager?

In Figure 3, we demonstrate the performance of Frame-Voyager on the Video-MME (without subtitles) by varying the number of chosen frames and comparing it to uniform sampling. Across different numbers of frames, Frame-Voyager consistently outperforms uniform sampling. Notably, Frame-Voyager achieves better results using only half the frames, *e.g.,* the $8$-frame Frame-Voyager surpasses the $16$-frame uniform sampling. However, as the number of extracted frames increases, the performance gap between Frame-Voyager and uniform sampling narrows. We think this is reasonable on the current public benchmarks which usually do not require a very large number of frames to answer the query. Using more frames may introduce unnecessary information, which could even limit performance gains.

Figure: Figure 6: Case study from Video-MME: The horizontal lines represent the timeline, with points marking the time positions of frames extracted by different methods. Uniform sampling captures only a limited number of frames relevant to the query. While CLIP extracts more relevant frames, it struggles to capture the temporal dynamics of gradual zoom-in transitions. In contrast, Frame-Voyager effectively selects a combination of frames that are both highly relevant to the query and accurately reflect the correct temporal sequence.
Refer to caption: x6.png

RQ4: How does the design of Frame-Voyager contribute to the performance?

Frame-Voyager reuses the embedding layer and the first transformer layer of the backbone LLM. To validate this design choice, we assess the impact of reusing different components of the LLM, as presented in Figure 4. Reusing only the embedding layer’s bag-of-words features yields noticeable improvements. Incorporating one to two layers of the LLM enhances question understanding, leading to better performance. However, reusing additional layers results in a gradual performance decline. This decline occurs because the LLM remains frozen during Frame-Voyager’s training, while lower-layer features are general-purpose, higher layers—with increased attention and fusion operations—shift the features towards language modeling, leading to diminished performance (Jin et al., ).

RQ5: How does Frame-Voyager perform on different types of questions?

Leveraging the question types defined in the Video-MME benchmark, we conduct a comparative analysis among three methods, *i.e.,* our Frame-Voyager, the uniform sampling approach, the individual frame-query matching method based on CLIP, across six distinct categories of questions.
The results are presented in Figure 5, where the maximum and minimum values are attached to each type in the figure.
The results indicate that Frame-Voyager achieves significant improvements in four of these categories, including an accuracy enhancement of $4.8\%$ on the recognition task compared to uniform sampling.
However, slight performance fluctuations are observed in the counting and OCR tasks.
For instance, Frame-Voyager results in one additional error in the counting problem compared to uniform sampling.
We attribute these minor inconsistencies to VILA’s inherent limitations in effectively handling these specific types of tasks.

We can also observe that, although CLIP-based method shows improvement over uniform sampling in the perception, recognition, and reasoning question types, they still fall short compared to Frame-Voyager.
In the information synopsis type, CLIP-based method performs even worse than uniform sampling.
The reason is that CLIP-based methods ignore the global video information, whereas Frame-Voyager explicitly models both the query-frame and frame-to-frame interactions.

RQ6: What does the combination extracted by Frame-Voyager look like?

In Figure 6, we present one sampled case from Video-MME (Fu et al., ). The (video, query) pair, with the query “What is the small flying black dot at the start of the video”, is evaluated across different methods. The frames obtained using uniform sampling provide limited information, with irrelevant background frames included.
When using CLIP for frame-query matching, the retrieved frames show a small black dot in the first frame, followed by frames related to the terms “virus” and “protein”. However, these frames do not clarify what the small black dot represents.

In contrast, our model, Frame-Voyager, effectively queries frame combinations by analyzing relationships between frames and modeling temporal information. This enables Frame-Voyager to capture the critical details at the start of the video. The selected frame combinations reveal the trajectory of the “small black dot in flight”, ultimately identifying the “dot” as a “virus”. Additional case studies are provided in Appendix 10.

## 5 Conclusions

In this work, we introduce Frame-Voyager, a plug-and-play frame combination selection method that enhances Video-LLMs in video understanding tasks.
We address the challenges of combination optimization by formulating it as a ranking task and implementing a ranking-based reward learning framework with a human-free data collection pipeline. Extensive experiments show that Frame-Voyager not only significantly boosts the performance of baseline Video-LLMs like VILA, achieving state-of-the-art results, but also outperforms other frame selection methods. Comprehensive ablation studies further confirm its effectiveness. Overall, our work sets a strong baseline for Video-LLMs and guides future research on frame selection and optimization. As Video-LLMs evolve to tackle diverse tasks, our method offers an efficient solution to enhance broader video understanding.

## Limitations

For the data construction, due to resource constraints, 1) we only generate the ranking data using VILA-8B, precluding experiments with more powerful Video-LLMs that might yield superior results. 2) The combinations for data construction are limited in size as the training set primarily focuses on short videos. Despite the promising improvements achieved, we highlight that applying Frame-Voyager to longer videos with a greater number of frame combinations could potentially lead to further performance enhancements in processing extended video content.

For the model framework, 1) we integrate our approach into existing Video-LLMs as a plug-in to ensure parameter efficiency, without additional fine-tuning of the backbone model.
However, directly reusing the backbone’s parameters may not yield optimal results. Moreover, our simplified plug-in module could benefit from a more sophisticated design to further enhance overall performance. 2) Integrating our approach into the pre-training process of Video-LLMs remains unexplored, we hypothesize that learning frame combinations during pre-training could produce a more robust and effective model.

## Appendix A Implementation Details of Frame Extraction Baselines in RQ1

In executing the SBD methods, we utilize the OpenCV(^2^22[https://github.com/opencv/opencv](https://github.com/opencv/opencv)) library.
We generate the disparity between each frame and its adjacent frame, utilizing both the difference of the RGB Histogram and Canny, as well as the optical flows. This process aids us in picking out $T$ frames with the top-$T$ greatest disparity values.

For the cluster-based approach, Katna, we directly use
Katna API. It initially segments the videos according to content specifics and extracts a list of keyframes from each individual segment, employing the Histogram with K-means technique. Afterward, all segmented keyframe lists are merged to pick out the final $T$ frames.

For VILA-Embedding, we utilize the visual feature after the projector and the text feature from word embedding to measure the cosine similarity after the average pooling. The cosine similarity is then used for ranking the frames.
For CLIP, we use the hosted model on HuggingFace(^3^33[https://huggingface.co/docs/transformers/model_doc/clip](https://huggingface.co/docs/transformers/model_doc/clip)) and directly rank and select top-$T$ candidate frames according to their output logits.

## Appendix B Question type analysis of the generated datasets

Figure: (a) Question type of origin dataset.
Refer to caption: x7.png

Figure: (a) Question type of origin dataset.
Refer to caption: x9.png

We observe similar patterns in Figure 7 and Figure 8.
Taking the NextQA dataset as an example, the proportion of “what” type questions in the original dataset is $27.4$%, whereas in our generated dataset, this proportion increases to $38.7$%. Conversely, the proportion of “why is/does” type questions significantly decreases from $39.4$% in the original dataset to $11.4$% in our generated dataset.

The reason is that answers to questions starting with “what” tend to be shorter compared to those starting with “why is/does”. When we calculate the loss for each subset combination using Video-LLM, the auto-regressive loss for each token in the answer is computed based on all preceding tokens.
In the case of a long answer, the LLM’s prior knowledge may diminish the impact of the subset combination (as the LLM can predict subsequent tokens based on the earlier ones in the answer). This leads to more “why” cases being removed during our filtering process.

## Appendix C Computation and Parameter Cost

In this section, we analyze the additional overhead introduced by Frame-Voyager based on VILA-8B. First, in terms of parameter size, our method introduces two MLPs, and the additional parameters account for $0.2$% of the original model’s parameter.
Next, we analyze the time overhead. During training, the original model required $5.1$k GPU hours(Lin et al., ), while Frame-Voyager requires $64$ GPU hours, representing an additional training time of $1.25$% of the original.

For inference, we randomly sample $100$ examples from Video-MME to measure the model’s inference latency. Using the experimental setting from our main paper, the average inference latency for the uniform sampling baseline is $1.329$ seconds per example, while for Frame-Voyager it is $1.696$ seconds, indicating that our method introduces an additional latency overhead of approximately $27.6$%.

## Appendix D A Study on the Number of Candidate Frames

**Table 4: The ablation results on different number of candidate frames. For all experiments, we expand the candidate frames from $8$ to $256$, while freeze the number of chosen frames as $8$. Results are reported on Video-MME dataset (without subtitles).**
| #candidate frames | 8 | 16 | 32 | 64 | 128 | 256 |
| --- | --- | --- | --- | --- | --- | --- |
| Video-MME (%) | 47.5 | 48.2 | 48.6 | 49.7 | 50.5 | 50.8 |

As the number of candidate frames increases, more video information is captured within the selection pool. While selecting $8$ frames from a larger candidate set, our method consistently improves results on the Video-MME dataset.
As shown in Table D, selecting $8$ frames from a candidate pool of $128$ frames yields a $3$% improvement compared to selecting from just $8$ frames (*i.e.,* uniform sampling). However, as the candidate set size increases further (from $128$ to $256$), the additional information brought by the expanded set becomes limited.

## Appendix E More Case Studies

Figure: Figure 9: Case study from Video-MME: The horizontal lines represent the timeline, with points marking the time positions of frames extracted by different methods. We can see that the uniform sampling fails to capture key objects relevant to the query, while the CLIP-based method, due to its limited OCR capabilities on special fonts, incorrectly matches terms like “blue food dye”. In contrast, Frame-Voyager effectively identifies the used ingredients, providing the necessary context to accurately answer the query about which ingredient is not used.
Refer to caption: x11.png

Figure: Figure 10: Case study from Video-MME: The horizontal lines represent the timeline, with points marking the time positions of frames extracted by different methods. It shows that uniform sampling produces a sequence containing many irrelevant background frames. The CLIP-based method, though focused on keywords like “basketball” and “boy”, fails to capture the temporal relationships between frames. Our approach can select high-quality frames that align with the query, illustrating key events in temporal order—from “boy’s training” to the “basketball match” and finally the “podium”.
Refer to caption: x12.png

In Figure 9 and 10, we present additional case studies to highlight the effectiveness of our model.
In Figure 9, the uniform sampling fails to capture key objects mentioned in the query. Similarly, the CLIP-based method struggles due to its limited OCR capabilities on special fonts, often match objects like “blue food dye”. In contrast, our Frame-Voyager, accurately identifies the used ingredients in this video, providing sufficient video context to answer the query about which ingredients are not used.

Figure 10 further demonstrates the limitations of uniform sampling, which produces a lot of irrelevant background frames. While the CLIP-based method focuses on isolated keywords like “basketball” and“boy”, it lacks the ability to connect frames meaningfully. Our method, however, selects representative frames based on the query, illustrating key events in temporal order—from the boy’s training to the basketball match, and finally, the podium.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: Please see the weaknesses. And extra questions:

(1) What is the trade-off between efficiency and effectiveness? Can you provide some metrics like running time/memory usage/flops to the proposed methods? As the model contains an extra keyframe localization stage, it is worthy showing those results to see the trade-off.

(2) I notice that the proposed method uses VILA as a reference model to label rewards, and uses the same VILA series models for downstream tasks. Is this also a kind of self-rewarding strategy shown in the sevilla work? Or the proposed method/modules can zero transfer to other VLM like llava-ov/qwen-vl-2?

(3) to answer my 1st weakness, I would suggest authors conduct extra experiments on Next-GQA to see the grounded QA results with grounded metrics.

(4) Can you show some off-shelf llm-based localization tools like sevila localizer in Table 2? It would be interesting to see the comparison between a llm-based reasoning method with the proposed reward imitation learning method.

(5)  Regarding the claim in Figure 3 and RQ3 (Lines 431-433), is this true? Related work (e.g., [1]) suggests that some observed issues may stem from limitations in the model side (VILA). Could the authors clarify this?

[1] LONGVIDEOBENCH: A Benchmark for Long-context Interleaved Video-Language Understanding
soundness: 3
strengths: This is a reasonable extension from previous keyframe selection work [1] where it relies more on single keyframe selection, and does not consider temporal relations/modeling among frames. The proposed pseudo-label scheme from a VLM + list of combinations of frames makes sense to me. 
Authors also conduct extensive experiments across diverse popular benchmarks to show their effectiveness. 
Overall, I think the proposed Frame-Voyager makes a good contribution to the keyframe selection in video-language studies. 

[1] Self-chained image-language model for video localization and question answering. NeurIPS23
summary: This paper proposed Frame-Voyager that learns to query informative frame combinations, based on the given textual queries in the task.
Authors introduced a new data collection and labeling pipeline, by ranking frame combinations using a pre-trained Video-LLM.
Extensive experiments are conducted to support the effectiveness of the proposed Frame-Voyager.
weaknesses: 1. As the paper mentioned, this pseudo-label strategy is not scalable, and I think the frame combination part is a bit tricky. This is like creating artificial rewards according to video (like a sandbox in RL) to train the reward model. However,  the reward is not always reliable from a VLM even though we compute it according to GT answers. For example, as shown in some previous studies [2],  the model will generate correct answers when provided with wrong localized clips. It is true that we might give performance improvement when using such reward data created by a human-free pipeline, but it is more like an adapter / DPO style adaptation to the model rather than letting the model truly be grounded on query-related / informative frames. 

2. I think the weakness in 1 somehow affects the proposed method to require a relatively complex pre-training data construction/filtering/design, since the positive/negative signal is too sensitive according to video conditions in this framework from my view (e.g. frame blurring / redundancy).

3. Also, the keyframe selection for video-language understanding is not a brand new topic, many related works in track try to propose different ways, from continuous space learning to discrete pipeline, to address question-aware moment detection. I would suggest to include those works [1,2,3,4] for a more comprehensive study. 


[1] ViLA: Efficient Video-Language Alignment for Video Question Answering. ECCV24.  
[2] Can i trust your answer? visually grounded video question answering.  CVPR24.  
[3] TimeCraft: Navigate Weakly-Supervised Temporal Grounded Video Question Answering via Bi-directional Reasoning. ECCV24.  
[4] Self-Adaptive Sampling for Accurate Video Question Answering on Image Text Models. ACL24.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: All question in weakness and: 

1. Since the authors claim that the proposed method is a plug-and-play module, does the reference Video-LLM for data collection have to be the same model family with the MLLM for Frame-Voyager? Since different models have different features, whether the generated data only works for the ViLA-1.5 model family? It would be interesting to see the performance of these annotated data’s effectiveness on different model families like LLaVA-onevision and so on. 

2. The authors are encouraged to discuss the comparison with a related line of research which leverages LLM agents to select meaningful keyframes from the video input, including VideoAgent [1], VideoTree[2] … 

3. Also, could the authors validate the effectiveness of the frame selection method on temporal grounding as well? Which seems to be very similar in high-level (localizing key information given the text query). 

[1] VideoAgent: Long-form Video Understanding with Large Language Model as Agent

[2] VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos
soundness: 2
strengths: 1. The motivation is clear and easy-to-follow. 

2. The presentation of the paper is of high quality. 

3. The analysis is comprehensive.
summary: In this paper, the authors present Frame-Voyager which learns to query informative frame combinations, based on the given textual queries for video qa tasks. The authors create a data collection pipeline for the proposed model/training objection and evaluate it on four video QA benchmarks.
weaknesses: 1. The inflexibility of the proposed frame selection method. Different videos usually have different “information density”, which means that some video could be represented by even a single frame and some need densely sampled frames. This also would be affected by the query type. The number of keyframes in the proposed framework seems to be a fixed hyper-parameter, which would be hard to generalize to all different videos in the wild (for different length / query types, need different hyper-parameter selection). Is it possible for the model to do adaptive/dynamic frame selection? 

2. The authors mention in line 170-172 that “the smaller combinations exhibit generalization capabilities when larger values of M and T are used during inference for longer video”. However, the process of this generalization is unclear (the introduction of the inference process, top-K selection is also unclear, in line 258-261). Does the inference directly apply the model to higher frame number or is the concatenation of several combinations with the highest score, what is the K setting for inference? Also, based on this, could the authors elaborate more on how Frame-Voyager makes good use of global video context? 

3. In Table 1 and Figure 3, the authors only compare Frame-Voyager with the uniform sampling baseline using the same frame number of frames for Question-Answering. However, Frame-Voyager actually sees more frames for keyframe selection (128 frames and select 8 frames), which creates unfair comparisons. Also there seems to be very little improvement in performance while using 8 keyframes from 128 frames compared to just uniform sample 16 frames. Could the authors show the comparison of Frame-Voyager against the best uniform sampling configurations for a more fair comparison?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My questions focus primarily on the weaknesses outlined above:

1. What are the data collection costs, particularly for the VideoChatGPT data?

2. How well does the method generalize when increasing the number of frame candidates and selected frames?

3. Can the proposed model outperform SigLIP and InternViT-6B, especially when they are fine-tuned on the same collected data?

4. What is the latency of the method?

5. A suggestion: Could CLIP-like models enhance data collection efficiency? Specifically, using them to estimate query-frame similarities might identify frames with extremely high or low similarities, indicating positive or negative frame combinations. This could reduce the search space and improve data collection efficiency.
soundness: 1
strengths: **1. Originality:** The novel formulation of frame selection as a ranking problem is compelling. This approach minimizes the need for extensive manual labeling to obtain ground truth, offering a fresh and efficient perspective on the task.

**2. Significance:** Frame selection is a critical challenge in video understanding due to the high dimensionality of video data and the computational demands involved. Traditional uniform sampling often misses relevant content, making this problem a key focus for advancing the field.

**3. Clarity:** The paper is well-written, with a clear explanation of the motivation and methodology. The authors effectively communicate the concepts, making the approach easy to understand.
summary: This paper introduces Frame-Voyager, a novel frame selection method designed to improve the performance of Video-LLMs on video question-answering (QA) tasks. The key contributions are (1) an automated data collection strategy that ranks frame combinations according to the QA loss from a reference Video-LLM, and (2) a frame selection module integrated with a Video-LLM, trained using this ranked data. When applied to VILA, Frame-Voyager outperforms the uniform sampling baseline across four video QA benchmarks.
weaknesses: **1. Data Collection Cost:**
The number of frame combinations increases exponentially with M and T (L166), leading the authors to limit M to 16 or 32 and T to 2 or 4. Despite these restrictions, evaluating $C(32, 4) \approx 36K$ combinations for a single QA sample is still costly, raising concerns about data collection efficiency. Thus, this paper needs to provide more details on the computational resources and time required for data collection.

**2. Generalization to More Frames:** 
The method is trained to select up to T=4 frames from M=32 candidates. Although the authors claim the model generalizes well to larger M and T values (L170), Fig. 3 shows diminishing gains over uniform sampling while increasing the number of frames. Despite the authors attributing this to benchmark constraints (L431), prior research (e.g., LongVILA [1], LongVA [2]) has shown that additional input frames can enhance QA performance, suggesting potential limits in generalization. Given the impracticality of increasing M and T for data collection, robust generalization is crucial. Thus, the authors should extend their experiments to more input frames, possibly integrating Frame-Voyager with LongVILA, to help clarify the generalization capabilities and limitations of Frame-Voyager.

**3. Fairness of Comparisons:** 
The comparison between Frame-Voyager and CLIP is not fully equitable due to three factors: (i) Frame-Voyager uses superior backbones (SigLIP or InternViT-6B); (ii) it is trained on specialized ranking data, unlike CLIP; and (iii) it has additional parameters like self-attention modules. Since (ii) and (iii) are core contributions of Frame-Voyager, a fairer comparison would involve using SigLIP or InternViT-6B models with and without training on the collected data. Additionally, the authors should specify the versions of CLIP and SigLIP used.

**4. Latency Analysis:** 
The authors claim that Frame-Voyager outperforms LongVILA while processing fewer frames, but this only holds for the "Long" subset of the Video-MME benchmark. In other scenarios, LongVILA performs better. Given the extra parameters introduced by Frame-Voyager, a detailed efficiency comparison between VILA+Frame-Voyager and LongVILA is necessary, including latency, memory usage, and computational complexity.

---
[1] Xue F, Chen Y, Li D, et al. Longvila: Scaling long-context visual language models for long videos[J]. arXiv preprint arXiv:2408.10188, 2024.

[2] Zhang P, Zhang K, Li B, et al. Long context transfer from language to vision[J]. arXiv preprint arXiv:2406.16852, 2024.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See the above weakness. I am interested in how the authors consider data scaling. Constructing data with only 16 or 32 frames is not very helpful for understanding long videos.
soundness: 3
strengths: 1.The authors designed an efficient keyframe data construction method, exploring keyframe selection using combinations of all frames. This approach achieved significant improvements across multiple datasets with 12K high-quality clips.

2.The authors state that they will open-source the data and related code, which is valuable for the open-source community.

3.The proposed method demonstrate strong performance across multiple datasets, proving the robustness of the approach, especially in Anet dataset. As a plug-and-play method, Frame-Voyager can serve as a valuable tool for video understanding.
summary: The paper presents FRAME-VOYAGER, a novel approach for improving Video Large Language Models by selecting informative frame combinations based on textual queries. Traditional methods like uniform frame sampling and text-frame retrieval do not account for variations in information density or complex instructions, leading to sub-optimal performance. FRAME-VOYAGER addresses this by learning to query frame combinations with lower prediction losses, using a pre-trained Video-LLM for supervision. The method is demonstrated to enhance performance in video question answering tasks across various benchmarks. The approach also includes a new data collection and labeling pipeline, which ranks frame combinations to train the model efficiently. FRAME-VOYAGER is proposed as a plug-and-play solution, improving Video-LLMs without significant computational overhead. Experiment results show that FRAME-VOYAGER achieves significant performance in several VQA datasets.
weaknesses: 1.Data scaling limitations. As the authors mentioned, selecting 4 frames from 32 results in 35960 candidates, leading to prohibitively high costs for data scaling. Furthermore, keyframe selection is even more critical for longer videos, and the costs become unmanageable when dealing with selections from 128 or 256 frames.

2.When testing on short video benchmarks, did VILA and VILA+FRAME-VOYAGER evaluate performance by selecting 2 frames out of 16? If so, please provide the results of VILA using 16 frames on the Next-QA and ANQA datasets. Additionally, I believe that comparing results based on only 2 frames is not meaningful, as most models typically evaluate performance using 8 or 16 frames.

3.Please provide additional results of the model on more benchmarks, such as MVbench and egoSchema.

Meta-review: Frame selection is a popular paradigm to improve the computational cost of video-language models. This paper improves upon prior works like Sevila by not choosing frames independently, but by considering temporal relations among frames (the authors address the combinatorial search space of this task by phrasing it as ranking instead).

The rebuttal and revised version of the paper addressed most of the reviewers' initial concerns, and each reviewer is now positive about the paper. Therefore, the decision is to accept the paper.

Decision: Accept (Poster)