Prompt-guided Visual Perception for Efficient Training-free Video LLM

## Abstract

Abstract Vision-language large models have achieved remarkable success in various multi-modal tasks, yet applying them to video understanding remains challenging due to the inherent complexity and computational demands of video data. While training-based video-LLMs deliver high performance, they often require substantial resources for training and inference. Conversely, training-free approaches offer a more efficient alternative by adapting pre-trained image-LLMs models for video tasks without additional training, but they face inference efficiency bottlenecks due to the large number of visual tokens generated from video frames. In this work, we present a novel prompt-guided visual perception framework (abbreviated as Free Video-LLM ) for efficient inference of training-free video LLMs. The proposed framework decouples spatial-temporal dimension and performs temporal frame sampling and spatial RoI cropping respectively based on task-specific prompts. Our method effectively reduces the number of visual tokens while maintaining high performance across multiple video question-answering benchmarks. Extensive experiments demonstrate that our approach achieves competitive results with significantly fewer tokens, offering an optimal trade-off between accuracy and computational efficiency compared to state-of-the-art video LLMs. The code will be available at https://github.com/contrastive/FreeVideoLLM .

## 1 Introduction

Recently, vision-language models (VLMs) have rapidly revolutionized multi-modal understanding and generation, enabling models to comprehend and produce responses from both visual and textual inputs (visual inputs include images, videos, etc). Due to the vast amount of image-text data and foundational models like CLIP Radford et al. () and large language models (LLMs) such as GPT Brown () and LLaMA Touvron et al. (), image-LLMs Zhang et al. (); Liu et al. (); Achiam et al. (); Chen et al. (); Jie et al. () have already made significant progress. InstructBLIP Dai et al. () extends the capabilities on new visual tasks by incorporating instruction-aware features. LLaVA Liu et al. () generate multimodal language-image instruction-following data using GPT-4 Achiam et al. (), demonstrating impressive multimodel chat abilities. The evolution of VLMs has also given rise to video-based LLMs that are specifically tailored for video understanding. These methods, as discussed by Tang et al. Tang et al. (), can be categorized into two primary approaches based on their training strategies: training-based video LLMs and training-free video LLMs.

In the training-based video LLMs, AV-LLM Shu et al. () and Vid2Seq Yang et al. () trained all the parameters in the LLM, which can be resource-intensive. Other main stream methods in this fine-tuning category either externally(e.g., Q-former) or internally fine-tune the bridge between the video encoder and the LLM, or adopt a phased fine-tuning approach for connective adapters and insertive adapters. Video-LLMs, such as Video-ChatGPT Maaz et al. (), Chat-UniVi Jin et al. (), and MovieChat Song et al. (), primarily focus on general question-answering (QA) or captioning tasks. They utilize video encoders to produce video embeddings, which they then decode into text outputs based on given prompts or instructions. Video-LLaMA Zhang et al. () integrates a video Q-former to learn video-language correspondence and an audio Q-Former to learn audio embeddings for the LLM. Additionally, Chat-UniVi Jin et al. () employs a set of dynamic visual tokens to uniformly represent both images and videos.

Despite the remarkable success of training-based video LLMs, training-free video understanding models have emerged as an attractive alternative due to the high cost of training large-scale video models. Training-free methods leverage pre-trained image-based models and minimally adapt them for video tasks without additional training on video data. Models such as IG-VLM Kim et al. () and FreeVA Wu () have demonstrated
the potential of transforming video frames into a format compatible with high-performance VLMs. These approaches utilize parameter-free temporal aggregation techniques and composite image grids, enabling robust zero-shot performance on video question-answering tasks. These methods offer promising results without requiring time-consuming and resource-intensive video-specific
training.

However, existing training-free video LLMs, struggle with efficiency issues due to the large number of visual tokens generated by video frames. Video data consists of multiple frames, significantly increasing the number of visual tokens that need to be processed by the model. As the input sequence length increases, the computational cost also scales, especially in transformer architectures where self-attention and feed-forward networks dominate resource usage. This results in slow inference speeds and high memory consumption, limiting the practical application of video LLMs in real-time or resource-constrained environments. Reducing token count without compromising performance is a key challenge in scaling video LLMs efficiently.

To address these challenges, we propose a prompt-guided visual perception framework (abbreviated as *Free Video-LLM*) that significantly improves the efficiency of training-free video LLMs. Our method introduces prompt-guided temporal and spatial sampling, which reduces the number of visual tokens based on the specific requirements of the input prompt. By discarding redundant frames and focusing only on the regions and time segments relevant to the prompt, our model achieves superior performance across multiple video QA benchmarks with a significantly reduced computational burden. Table [1](#S1.T1) compares several prominent open-source models, our proposed method uniquely possesses training-free capabilities, inference efficiency, and effective video understanding, demonstrating a comprehensive advantage in the field. The proposed Free Video-LLM approach not only enhances the model’s efficiency but also maintains competitive accuracy compared to existing state-of-the-art video LLMs, offering a balanced solution for scalable video understanding tasks.

**Table 1: Comparison of the existing representative open-sourced vision-language models.**
| Method | Training-free | Inference-efficient | Video understanding |
| --- | --- | --- | --- |
| Image-LLM (*e.g.,* LLaVA) | ✓ | ✗ | ✗ |
| Training-based Video LLM (*e.g.,* Video-LLaVA) | ✗ | ✗ | ✓ |
| Training-free Video LLM (*e.g.,* IG-VLM) | ✓ | ✗ | ✓ |
| Ours | ✓ | ✓ | ✓ |

## 2 Related Works

In this section, we briefly revisit the related works including image-LLMs, video-LLMs (especially training-free video LLMs).

### 2.1 Image-LLMs

The rapid advancement of image-language models (image-LLMs) can be attributed to two key factors: the foundational work of CLIP Radford et al. (), which introduced a shared representation space for vision and language, demonstrating strong zero-shot capabilities and robust performance across various computer vision benchmarks; and the emergence of powerful Large Language Models (LLMs) like GPT Brown () and LLaMA Touvron et al. (), which can be further enhanced through instruction tuning Peng et al. ().
Flamingo Alayrac et al. () excels in few-shot learning by seamlessly integrating pre-trained vision and language models and high-quality interleaved multimodal data. BLIP-2 Li et al. () utilizes a lightweight Querying Transformer to connect modalities.

Further progress on image-LLMs has been achieved through multimodal instruction tuning. LLaVA Liu et al. () uses GPT-4 Achiam et al. () to generate robust multi-modal instruction data, pre-training on image-text pairs and fine-tuning for end-to-end multimodal understanding. InstructBLIP Dai et al. () extends the capabilities of BLIP-2 with instruction-aware features, while mPLUG-Owl Ye et al. () employs a two-stage modular approach to enhance task performance across different modalities. MiniGPT-4 Zhu et al. () aligns a frozen visual encoder with a frozen LLM (Vicunna Peng et al. ()) using one projection layer and showcasing enhanced capabilities such as detailed image descriptions and creative storytelling. MiniGPT-5 Zheng et al. () extends MiniGPT-4 to output text interleaved with images.

### 2.2 Video LLMs

Building on the foundation of LLM and image-LLMs, video-language models (video-LLMs) ia also rapidly developed. FrozenBiLM Yang et al. () leverages frozen bidirectional language models for zero-shot video question answering, achieving leading performance on zero-shot VideoQA without the need for manual annotations.
VideoChat Li et al. () and Video-LLaMA Zhang et al. ()
both utilize dual streams to handle audio and visual signals. Specifically, Video-LLaMA integrates Q-Former for these two streams, whereas VideoChat incorporates a video embedder alongside a perception tools for captions, and introduce a video-centric multimodal instruction fine-tuning dataset. Video-ChatGPT Maaz et al. () utilizes a pretrained visual encoder to extract both spatial and temporal features from videos by averaging frame-level features. These features are then projected into the input space of large language models (LLMs). Additionally, it also contributes a high-quality dataset of 100,000 video-instruction question-answer pairs. Chat-UniVi Jin et al. () use a set of dynamic visual tokens to uniformly represent images and videos tokens. PLLaVA Xu et al. () introduce a post-training weight fusion methods to alleviate forgetting phenomenon during multi-modality fine-tuning.

### 2.3 Training-free Video LLMs

In contrast to these tuning methodologies for Video LLMs, there is a growing interest in exploring training-free video LLMs, specifically how existing image model architectures can be minimally adapted to accommodate video inputs. IG-VLM Kim et al. (), as a pioneer in this exploration, transforms videos into single composite image grids, enabling the direct application of a high-performance VLM without the need for video-data training. FreeVA Wu () also investigates the potential of leveraging offline image-LLMs as training-free video assistants by simply adding a parameter-free temporal aggregation, achieving strong performance in zero-shot video question-answering tasks Chen & Dolan (); Caba Heilbron et al. (); Xu et al. (), even surpassing video instruct-tuning models. Meanwhile, SLOWFAST-LLAVA Xu et al. () utilizes a dual-stream approach to efficiently capture spatial and temporal video features, demonstrating competitive performance on various video benchmarks without requiring additional training.

## 3 Approach

### 3.1 Preliminaries

The image-LLM has a remarkable progress in the past two years due to the large amount of image-text pairs data and image instruction data. The representative open-sourced image-LLMs like LLaVA Liu et al. () and InternVL Chen et al. () obtain high performance on various image conversation, description and reasoning tasks. The image-LLMs take one image $I$ as input, and the visual encoder $g_{V}$ (including the projector) extracts the image features and converts into language embedding tokens:

$$ $H_{I}=g(I),$ (1) $$

where $H_{V}\in\mathbb{R}^{N\times D}$, $N$ is the number of visual tokens per frame, and $D$ is the token embedding dimension.
Then the image tokens $H_{I}$ and the text tokens $H_{T}$ are fed into the LLM $f$ for generating responses:

$$ $Y=f([H_{I},H_{T}]).$ (2) $$

### 3.2 Computational Burden of Video LLM

The training-free video LLM can leverage the well-trained image-LLMs for video without training on any data. Given a video, a number of frames are usually uniformly extracted and forms a sequence of images $\{I_{1},I_{2},\cdots,I_{T}\}$ where $T$ is the number of frames. These images are fed into the visual encoder to obtain visual tokens:

$$ $X_{V}=g_{V}(\{I_{1},I_{2},\cdots,I_{T}\}),$ (3) $$

$X_{V}\in\mathbb{R}^{T\times N\times D}$. Then the training-free video LLMs directly use LLMs to receive visual tokens $X_{V}$ and text prompt to generate responses for video understanding:

$$ $Y=f([H_{V},H_{T}]).$ (4) $$

This simple pipeline is intuitive yet effective Wu (); Kim et al. ().

However, the video usually consists of many frames, that is, the number of visual tokens will be significantly large, proportional to the product of the frame number and the image size $L\propto TN$. The input sequence length directly influences the computational cost and inference efficiency of LLM. In transformer architecture, the computational cost is mostly occupied by the self-attention and feed-forward network. The self-attention module performs token-to-token relation computation and requires a $O(L^{2})$ computational cost. The feed-froward network makes nonlinear transformation of each token with the computation budget proportional to the sequence length. Compared to image-LLMs, the computational complexity of video-LLMs will increase by an order of magnitude, which will significantly reduce the inference speed and efficiency of video-LLMs, affecting the actual usage experience.

Figure: Figure 1: Illustration of the proposed *Free Video-LLM* with prompt-guided visual perception.
Refer to caption: /html/2410.10441/assets/fig/diagram1.png

### 3.3 Prompt-guided Visual Perception

We introduce the prompt-guided visual perception for efficient video LLMs, by temporally and spatially sampling visual information respectively, as demonstrated in Figure [1](#S3.F1).

#### Prompt-guided Temporal Sampling

The video is usually captured in a high frame rate, resulting in large redundancy in the neighbor frames. In order to reduce the number of frames and maintain the discriminative information, we propose to temporally sample the prompt-related frames.
The previous video-LLMs take all the visual tokens generated by the visual encoder as inputs for LLMs. However, this approach does not consider the text context for different questions or tasks, which may introduce useless tokens during inference. For instance, given a video depicting two women shopping in a supermarket, one might ask how many apples they bought, while another person could ask whether they bought eggs. Different questions necessitate the model to focus on different periods of the video. On the other hand, different regions of the video are also required for different questions. For example, one may inquire about the number of persons, while another may ask about the color of the left woman’s T-shirt. Here, we introduce prompt-guided spatial and temporal sampling to reduce the number of visual tokens for different input prompts.

In the temporal dimension, we use prompt features to guide the selection of temporal frames most relevant to the prompt. In particular, We utilize the text encoder $g_{T}$ that is matched with the visual encoder (such as the two in CLIP) to extract prompt features:

$$ $F_{P}=g_{T}(P),$ (5) $$

where one prompt sequence $P$ will be represented as a vector $F_{P}\in\mathbb{R}^{D}$. The visual tokens of video frames after the visual encoder contains the information inside each frame, and the tokens are in the same subspace of the prompt features. The visual features after global average pooling $F_{V}\in\mathbb{R}^{T\times D}$ are used to represent each frame. We calculate the relation score (*i.e.,* cosine similarity) between the frame features and the prompt feature:

$$ $s_{i}=\frac{{F_{V}}_{i}}{\|{F_{V}}_{i}\|_{2}}\cdot\frac{F_{P}}{\|F_{P}\|_{2}},$ (6) $$

where $i=1,2,\cdots,T$ is the frame temporal index. We sample the frames based on the similarities so as to maintain the most related frames to the prompt and discard the useless ones.

#### Prompt-guided Spatial Sampling

As mentioned above, different questions require the model to focus on a specific part of the spatial regions for answering. We propose to utilize the prompt information to guide the selection of regions of interest (RoI) in the video. For the visual tokens of a frame, we reshape them back to the spatial size as ${X_{S}}_{i}\in\mathbb{R}^{H\times W\times D}$ where $H\times W$ is the feature map size.
For a feature vector in spatial position $(h,w)$, its relation score with the prompt feature is calculated:

$$ $s_{h,w}=\frac{{X_{S}}_{i,h,w}}{\|{X_{S}}_{i,h,w}\|_{2}}\cdot\frac{F_{P}}{\|F_{P}\|_{2}},$ (7) $$

where $1\leq h\leq H$ and $1\leq w\leq W$. Suppose we need to box out a RoI with an area that is $\alpha\times$ the original area where $0<\alpha<1$. The top-$K$ tokens with the largest similarity scores are selected, and their positions are $\{(h_{1},w_{1}),(h_{2},w_{2}),\cdots,(h_{K},w_{K})\}$ where $K=\alpha HW$. The center coordinates of the RoI can be obtained by the mean of these positions:

$$ $\displaystyle h_{c}$ $\displaystyle=\frac{1}{K}\sum_{k=1}^{K}h_{k},$ (8) $\displaystyle w_{c}$ $\displaystyle=\frac{1}{K}\sum_{k=1}^{K}w_{k}.$ (9) $$

The height and width of the RoI is calculated by

$$ $\displaystyle H^{\prime}$ $\displaystyle=\sqrt{\alpha}H,$ (10) $\displaystyle W^{\prime}$ $\displaystyle=\sqrt{\alpha}W.$ (11) $$

With the center coordinates and box size, we can easily crop the RoI from the frame feature maps, and reshape them as a sequence of tokens as the video representation for the specific prompt. The obtained compact visual tokens are concatenated with the text prompt embeddings as the inputs of LLMs. In this way, the training-free video LLMs can process a video with fewer tokens and enjoy more efficient inference.

## 4 Experiments

### 4.1 Benchmarks and Implementation Details

#### Benchmarks

As our method is training-free, we directly evaluate the proposed methods on open-ended video understanding and question-answering benchmarks, including MSVD-QA Chen & Dolan (), MSRVTT-QA Xu et al. (), ActivityNet-QA Caba Heilbron et al. () and TGIF-QA Jang et al. (). The GPT APIs are utilized to assess the model accuracy and response quality. Following the previous works Wu (); Xu et al. (; ), GPT-3.5-Turbo-0125 version is used for fair comparison.

#### Base Models

The image MLLM we utilized as base models is LLaVA-v1.6 Liu et al. (; ). The two versions with different model sizes are used, *e.g.,* 7B and 34B. Both the visual encoder and text encoder are from OpenAI’s CLIP-L-14. The pretrained weights of LLaVA-v1.6(^1^11[https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2](https://huggingface.co/collections/liuhaotian/llava-16-65b9e40155f60fd046a5ccf2)) and CLIP-L(^2^22[https://huggingface.co/openai/clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336)) can be downloaded on HuggingFace. We fix all the pretrained weights of the base models and only slightly modify the hyperparameters.

#### Implementation Details

The input video is resized to 336$\times$336 for matching the CLIP-L visual encoder. Each frame will be transformed into 24$\times$24 tokens. The proposed method will squeeze the frame number and crop an RoI for efficient video understanding. The scaling factor of RoPE (rotary position embedding) in LLaVA-v1.6 is set as 2 to enable processing context with 8192 tokens. All the models are implemented using PyTorch.

### 4.2 Main Results

In Table [2](#S4.T2), we evaluate several models with a focus on efficiency, as reflected by the number of visual tokens (second column), which directly impacts the computational cost and memory requirements. Our method demonstrates notable efficiency, using only 1026 and 2600 visual tokens across two configurations, which is significantly fewer than competing models such as IG-VLM (3456 tokens) and SF-LLaVA (3680 tokens). Despite this reduced token count, our model consistently achieves competitive or superior performance across various QA benchmarks. For instance, in the MSVD-QA task, our method scores 76.8/4.0 with 1026 tokens, outperforming FreeVA (73.8/4.1 with 2304 tokens). Overall, the average accuracy and score of our method are comparable to other models with fewer inference tokens.

**Table 2: Main results of the proposed method and comparison with other training-free video LLMs. All models are 7B-level and with CLIP-L visual encoder. The two numbers in each cell are ‘Accuracy/Score’ respectively. The bold numbers indicate that our method requires significantly fewer tokens during inference.**
| Model | #visual | Size | MSVD | MSRVTT | ANet | TGIF | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| tokens | -QA | -QA | -QA | -QA |  |  |  |
| LLaVA-NeXT-Image Zhang et al. () | 2304 | 7B | - | - | 53.8/3.2 | - | - |
| FreeVA Wu () | 2304 | 7B | 73.8/4.1 | 60.0/3.5 | 51.2/3.5 | - | - |
| Free Video-LLM (ours) | 513 | 7B | 74.9/3.9 | 60.8/3.4 | 51.2/3.4 | 73.8/3.9 | 65.2/3.7 |
| Free Video-LLM (ours) | 1026 | 7B | 76.8/4.0 | 62.9/3.5 | 53.9/3.4 | 75.6/4.0 | 67.3/3.7 |
| IG-VLM Kim et al. () | 3456 | 7B | 78.8/4.1 | 63.7/3.5 | 54.3/3.4 | 73.0/4.0 | 67.5/3.8 |
| SF-LLaVA Xu et al. () | 3680 | 7B | 78.1/4.0 | 64.1/3.4 | 55.3/3.4 | 78.4/4.1 | 69.0/3.7 |
| Free Video-LLM (ours) | 2648 | 7B | 78.2/4.0 | 65.6/3.6 | 54.8/3.4 | 77.8/4.1 | 69.1/3.8 |

This efficiency does not come at the expense of performance; rather, it highlights the ability of our approach to achieve optimal trade-offs between token usage and task accuracy. On the TGIF-QA benchmark, our model with 2600 tokens achieves a competitive score of 78.5/4.1, slightly surpassing IG-VLM (73.0/4.0 with 3456 tokens) and SF-LLaVA (77.3/4.0). Table [3](#S4.T3) shows the results on larger LLM, *i.e.,* 34B. The similar efficiency gain to that of 7B models can be seen. Overall, our model provides a balanced approach to both computational efficiency and task performance, offering substantial improvements in resource allocation without sacrificing accuracy.

**Table 3: Main results of the proposed method and comparison with other training-free video LLMs. All models are 34B-level and with CLIP-L visual encoder. The bold numbers indicate that our method requires significantly fewer tokens during inference.**
| Model | #visual | Size | MSVD | MSRVTT | ANet | TGIF | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- |
| tokens | -QA | -QA | -QA | -QA |  |  |  |
| IG-VLM Kim et al. () | 3456 | 34B | 79.6/4.1 | 62.4/3.5 | 58.4/3.5 | 79.1/4.2 | 69.9/3.8 |
| SF-LLaVA Xu et al. () | 3680 | 34B | 79.3/4.1 | 67.0/3.6 | 58.8/3.5 | 80.2/4.2 | 71.3/3.9 |
| Free Video-LLM (ours) | 2648 | 34B | 79.2/4.1 | 67.2/3.7 | 59.0/3.5 | 79.8/4.2 | 71.3/3.9 |

### 4.3 Comparison with SOTA Methods

In Table [4](#S4.T4), we present a comparison between our model and state-of-the-art (SOTA) methods, with a particular emphasis on the trade-off between accuracy and efficiency, as reflected by the number of visual tokens and performance on four video question-answering (QA) benchmarks. Our method utilizes 2600 visual tokens, which is more efficient than IG-VLM (3456 tokens) and SF-LLaVA (3680 tokens), while achieving highly competitive results across all evaluated tasks.

On the MSVD-QA benchmark, our model scores 78.2/4.0, being comparable with SF-LLaVA (78.1/4.0), but with much fewer tokens, demonstrating a substantial gain in efficiency. A similar trend is observed on the MSRVTT-QA benchmark, where our model achieves 65.6/3.6, outperforming SF-LLaVA’s 64.1/3.4 while using fewer visual tokens. Additionally, our model shows strong performance on the ANet-QA and TGIF-QA benchmarks, with faster inference and lower computational cost. This balance of high accuracy and reduced computational load highlights the strength of our approach in delivering comparable or superior performance with significantly fewer resources.

In comparison to other training-free models, such as FreeVA (2304 tokens) and IG-VLM (3456 tokens), our method offers a superior balance, outperforming FreeVA on all tasks and matching IG-VLM’s performance with fewer tokens. These results demonstrate that our approach achieves an optimal trade-off between efficiency and performance, making it an excellent choice for scalable video understanding tasks.

**Table 4: Comparison with SOTA video LLMs on the four benchmarks. The bold numbers indicate that our method requires significantly fewer tokens during inference.**
| Model | #visual | Size | Visual | MSVD | MSRVTT | ANet | TGIF |
| --- | --- | --- | --- | --- | --- | --- | --- |
| tokens | Encoder | -QA | -QA | -QA | -QA |  |  |
| Trained models |  |  |  |  |  |  |  |
| Video-LLaMA Zhang et al. () | - | 7B | CLIP-G | 51.6/2.5 | 29.6/1.8 | 12.4/1.1 | - |
| Video-LLaVA Lin et al. () | - | 7B | ViT-L | 70.7/3.9 | 59.2/3.5 | 45.3/3.3 | 70.0/4.0 |
| Vista-LLaMA Ma et al. () | - | 7B | CLIP-G | 65.3/3.6 | 60.5/3.3 | 48.3/3.3 | - |
| VideoChat Li et al. () | - | 7B | CLIP-G | 56.3/2.8 | 45.0/2.5 | 26.5/2.2 | 34.4/2.3 |
| VideoChat2 Li et al. () | - | 7B | UMT-L | 70.0/3.9 | 54.1/3.3 | 49.1/3.3 | - |
| MovieChat Song et al. () | - | 7B | CLIP-G | 75.2/3.8 | 52.7/2.6 | 45.7/3.4 | - |
| Video-ChatGPT Maaz et al. () | - | 7B | CLIP-L | 64.9/3.3 | 49.3/2.8 | 35.2/2.7 | 51.4/3.0 |
| Video-LLaMA2 Cheng et al. () | - | 7B | CLIP-L | 70.9/3.8 | - | 50.2/3.3 | - |
| PLLaVA Xu et al. () | - | 7B | CLIP-L | 76.6/4.1 | 62.0/3.5 | 56.3/3.5 | 77.5/4.1 |
| Training-free models |  |  |  |  |  |  |  |
| FreeVA Wu () | 2304 | 7B | CLIP-L | 73.8/4.1 | 60.0/3.5 | 51.2/3.5 | - |
| IG-VLM Kim et al. () | 3456 | 7B | CLIP-L | 78.8/4.1 | 63.7/3.5 | 54.3/3.4 | 73.0/4.0 |
| SF-LLaVA Xu et al. () | 3680 | 7B | CLIP-L | 78.1/4.0 | 64.1/3.4 | 55.3/3.4 | 78.4/4.1 |
| Free Video-LLM (ours) | 2648 | 7B | CLIP-L | 78.2/4.0 | 65.6/3.6 | 54.8/3.4 | 77.8/4.1 |

### 4.4 Ablation Studies

#### Inference Speed

We compare the inference speed of three representative training-free video LLMs in Table [5](#S4.T5). Two important metrics for measuring the inference speed of large models are pre-filling latency and output speed. Pre-filling latency represents the time taken by the model to generate the first output token after receiving the input. A lower pre-filling latency means the model can begin producing results faster. Output speed refers to the rate at which the model generates subsequent tokens after the first one is produced. A higher TPS indicates that the model can output tokens more quickly once the generation process has started. The inference speed is evaluated on an NVIDIA V100 GPU with standard transformers framework. Our Free Video-LLM outperforms the others with fewer visual tokens (2,648), the fastest pre-filling latency (0.578 seconds), and the highest output speed (20.4 TPS). In comparison, IG-VLM and SF-LLaVA have higher visual token counts, slower pre-filling latency, and lower output speeds.

**Table 5: The inference speed of the compared training-free video LLMs. Pre-filling latency means time to the first output token. TPS denotes tokens per second.**
| Method | #visual tokens | Avg Accuracy | Pre-filling latency | Output speed |
| --- | --- | --- | --- | --- |
| IG-VLM Kim et al. () | 3456 | 73.0/4.0 | 0.894 s | 18.2 TPS |
| SF-LLaVA Xu et al. () | 3680 | 77.3/4.0 | 0.961 s | 17.9 TPS |
| Free Video-LLM (ours) | 2648 | 77.8/4.1 | 0.578 s | 20.4 TPS |

#### Prompt-guided Temporal Sampling

We evaluate the effectiveness of the proposed prompt-guided temporal sampling on MSVD task. The base image-LLM is LLaVA-v1.6-7B. The methods evaluated include uniform temporal sampling baseline, prompt-guided temporal sampling, and a combination of uniform and prompt-guided temporal sampling. Uniform temporal sampling with 3 frames and 864 visual tokens achieves an accuracy of 71.7 on MSVD. This method provides a baseline for comparison. Prompt-guided temporal sampling also uses 3 frames and 864 visual tokens but yields a higher score of 75.0. This indicates that incorporating prompt information to guide the temporal sampling process can lead to improved performance. Finally, the combination of uniform and prompt-guided temporal sampling, with 6 frames and 1728 visual tokens, achieves a score of 77.2. This suggests that a hybrid approach may further enhance results.

**Table 6: The effectiveness of prompt-guided temporal sampling.**
| Method | #frames | #visual tokens | MSVD-QA |
| --- | --- | --- | --- |
| Uniform temporal sampling | 3 | 864 | 71.7 |
| Prompt-guided temporal sampling | 3 | 864 | 75.0 |
| Uniform + Prompt-guided temporal sampling | 6 | 1728 | 77.2 |

#### Prompt-guided Spatial RoI Cropping

We conduct experiments to evaluate the proposed prompt-guided spatial sampling for RoI cropping. The experiments involve uniform temporal sampling and prompt-guided temporal sampling, with and without the addition of RoI.
Uniform temporal sampling and prompt-guided temporal sampling serves as baselines without RoI cropping, with 3 frames and 864 visual tokens resulting in MSVD accuracies of 71.7 and 75.0 respectively.
When adding RoI to prompt-guided temporal sampling, with an RoI ratio of 0.6 and 3 frames, the number of visual tokens is reduced to 513 and the MSVD accuracy is 74.9. Replacing the proposed RoI method with adaptive average pooling will decrease the accuracy to 73.8 with the same number of visual tokens.
These experiments demonstrate combining prompt-guided sampling with RoI can enhance performance, showing the effectiveness of prompt-guided spatial RoI cropping.

**Table 7: The effectiveness of prompt-guided spatial RoI cropping.**
| Method | RoI ratio | #frames | #visual tokens | MSVD-QA |
| --- | --- | --- | --- | --- |
| Uniform temporal sampling | - | 3 | 864 | 71.7 |
| Prompt-guided temporal sampling | - | 3 | 864 | 75.0 |
| Prompt-guided temporal sampling + AvgPool | 0.6 | 3 | 513 | 73.8 |
| Prompt-guided temporal sampling + RoI | 0.6 | 3 | 513 | 74.9 |

**Table 8: The effect of RoI ratio. The setting is prompt-guided temporal sampling + RoI.**
| RoI ratio | #frames | #visual tokens | MSVD-QA |
| --- | --- | --- | --- |
| 0.4 | 3 | 360 | 74.1 |
| 0.5 | 3 | 408 | 74.2 |
| 0.6 | 3 | 513 | 74.9 |
| 0.7 | 3 | 600 | 74.8 |
| 1.0 | 3 | 864 | 75.0 |

Table [8](#S4.T8) presents the results of our experiments on the effect of RoI ratio during prompt-guided spatial cropping on MSVD-QA performance. The findings reveal a general trend of increasing accuracy with higher RoI ratios. Specifically, an RoI ratio of 0.4 yields 360 visual tokens and an MSVD-QA score of 74.1, while increasing the RoI ratio to 0.5 results in a modest improvement to 74.2 with 408 tokens. As the RoI ratio increases beyond 0.6, performance stabilizes, with the scores hovering around 74.8 to 74.9, despite an increase in the number of visual tokens. This suggests that while a larger RoI allows for more comprehensive spatial information, there may be diminishing returns in accuracy beyond a certain threshold (*i.e.,* 0.6). Overall, these results indicate that optimizing the RoI ratio is crucial for enhancing model performance in video question-answering tasks, highlighting the effectiveness of our prompt-guided approach in balancing token utilization and accuracy.

### 4.5 Video QA Examples

We also show the video QA examples to analyze the advantages and disadvantages of the method more intuitively. Figure [2](#S4.F2) summarizes example object detection results from the MSVD-QA benchmark, comparing the outputs of SF-LLaVA and our proposed method. For the first video, both models correctly identify that someone is folding a square green paper, with SF-LLaVA scoring slightly higher (5 vs. 4). In the second video, both models describe multiple men diving into a swimming pool, with neither able to accurately determine the number of divers, resulting in the same lower score of 2. Overall, while both models perform similarly on these tasks, our method is more efficient with much fewer tokens.

Figure: Figure 2: Example video question answering results on MSVD-QA benchmark.
Refer to caption: /html/2410.10441/assets/fig/bXsKw3TOQXs_30_55.png

## 5 Conclusion

In conclusion, this paper introduces a novel framework for efficient inference of training-free video LLMs, named Free Video-LLM. The proposed method addresses the computational challenges associated with video understanding by introducing a prompt-guided visual perception approach that significantly reduces the number of visual tokens processed by the model. Through temporal and spatial token sampling techniques tailored to the input prompt, Free Video-LLM achieves a remarkable reduction in computational burden without compromising accuracy. Extensive experiments demonstrate that our method not only matches but often exceeds the performance of current state-of-the-art video LLMs, while using a fraction of the visual tokens. This represents a substantial step forward in the practical application of video LLMs, offering a competitive balance between accuracy and computational efficiency. The prompt-guided framework paves the way for more scalable and efficient video understanding systems, potentially transforming real-time and resource-constrained environments.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - What would the story/pitch look like if multi-image VLMs were taken into consideration?
- What is the complete breakdown of runtimes and comparisons to prior work?
- Can you produce a graph-based version of Tables 6 and 7, `#tokens` vs performance?
- How do the performance characteristics (in concrete numbers) change if multiple questions are asked about the same video (allowing for kv-caching)?
- How much correlation is there in the sampled frames? How do the authors prevent near duplicate frames from being sampled?
soundness: 3
strengths: - The paper is mostly well written and easy to follow.
- The technical contribution is well explained, concise, and clearly separated from prior approaches.
- The method is explained in enough detail, and likely easy to reproduce
summary: The submission presents a way to convert a pre-trained image-language model into a "video"-language model. The submission presents a method to pack image-tokens of a video into a stream of visual tokens that can be fed into a pre-trained VLM. At it's core the method filters input frames using a CLIP embedding, and uses spatial nearest neighbors to define a prompt-specific region of interest. The results show some improvement over baselines.
weaknesses: - **Framing:** The comparison, and motivation through prior work is incomplete. Throughout the submission, image-language models are presented as single-image models. For example L.150 "The image-LLMs take *one* image I as input". This is clearly no longer the case. Yes the original LLaVA architecture supported just one image, but LLaVA-next [Zhang et al. (2024b)], InternVL (cited as supporting evidence for L.150), EMU2, Mantis, etc all support multiple images at least at inference time. This significantly changes the framing of the paper and will require a major rewrite. The current framing is: "Convert a video into a image-size token stream, and feed this through the VLM". A more accurate framing would be "Use an existing multi-image VLM and subsample the visual tokens, right? This puts a much larger burden on the runtime evaluation.
- **Evaluation:** The evaluation seems incomplete.
  - One major claimed advantage of the proposed method is faster runtime. However, runtime is never explicitly compared. Instead, the submission presents `# tokens` as a proxy metric. This is not an ideal proxy, as it completely ignores the additional cost of running CLIP, and potential batching or pipeline inefficiencies. I would like to see a complete breakdown of runtimes and comparisons to prior work on the rebuttal. 
  - Another issue are baselines. I really appreciate the ablations in Tables 6, 7 and 8, however they are not quite apples-to-apples, as the number of visual tokens varies drastically (for Table 8 this is understandable as it is part of the experiment, but Table 6 and 7 should control for it). Both Tables 6 and 7 would be much more informative as a graph with varying number of tokens for each baseline. This would also provide the reader with a calibrated baseline if the llava-1.6 performance on the input videos (uniform sampling of *all* input tokens).
  - Finally, it is quite standard for VLMs to encode the image once, and answer multiple questions at inference time. How would the proposed framework perform under this more standard setting? Presumably, caching the image tokens will no longer work, as they depend on the prompt. Ideally, the rebuttal shows some evidence that caching vision tokens does not fully take away the advantage of the proposed method.
- **Technical:** The proposed CLIP-based sampling technique might lead to highly correlated inputs. Since CLIP is an embedding-based model, it might always find the k-closest images to the text-prompt. It seems likely that these k-closest frames are all the same (if the video is sampled at a high enough frequency). In fact, this may be happening, the results in Table 6 show that combining uniform and CLIP-based sampling works significantly better.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: I left questions in the weakness section
soundness: 2
strengths: The paper introduces Free Video-LLM, a training-free framework that enhances efficiency in video LLMs through prompt-guided temporal and spatial sampling, reducing visual tokens without compromising performance, and achieving superior results across video QA benchmarks with lower computational costs.
summary: Free Video-LLM introduces a training-free, prompt-guided visual perception framework that improves inference efficiency in video LLMs by decoupling spatial-temporal dimensions through temporal frame sampling and spatial RoI cropping based on task-specific prompts, reducing visual tokens while maintaining high performance across video question-answering benchmarks.
weaknesses: Inconsistency in Reported Results: The paper mentions that "On the TGIF-QA benchmark, our model with 2600 tokens achieves a competitive score of 78.5/4.1, slightly surpassing IG-VLM (73.0/4.0 with 3456 tokens) and SF-LLaVA (77.3/4.0)." However, 2600 tokens and the corresponding accuracy values are not found or do not match those presented in the tables. This inconsistency makes it difficult to verify the claimed performance improvements. Clarification is needed. Could you clarify the origin of the reported score of 78.5/4.1 for the 2600-token model on the TGIF-QA benchmark? This score is mentioned in the text but does not seem to match any data in the tables. Could you ensure consistency between the text and the tables?

Lack of Direct Comparisons and Generalizability: In Tables 2 and 3, each model utilizes a different baseline Vision-Language Model (VLM), which complicates the assessment of the efficiency gains from temporal sampling and spatial ROI cropping. To accurately evaluate the effectiveness of the proposed token reduction methods, it is essential to compare the performance of the original LLAVA 1.6 model to that of LLAVA 1.6 enhanced with the proposed techniques. Additionally, applying the proposed token reduction method to various baseline VLMs such as LLaVA-NeXT would demonstrate its generalizability and effectiveness across different models. Without these direct comparisons, it is challenging to understand the true impact and applicability of the proposed approach. Could you include an experiment or table comparing the original LLAVA 1.6 model with your enhanced LLAVA 1.6 model, detailing both performance metrics and token counts? This would help in assessing the efficiency gains of your proposed methods. Could you also demonstrate the effectiveness of your token reduction method on another Vision-Language Model baseline, such as LLaVA-NeXT, to support the claims of generalizability?

Insufficient Discussion of Related Work: The concept of question/text-guided token reduction has been previously introduced in works like LVNet [1] and VideoAgent [2]. The paper does not adequately compare the similarities and differences between the proposed method and these existing approaches. A thorough discussion of how the proposed method builds upon or differs from prior work is necessary to establish its novelty and contribution to the field. Without this comparison, the paper's unique contributions remain unclear. Could you add a subsection in the Related Work or Discussion section that explicitly compares your method with LVNet and VideoAgent? This would be helpful to highlight key similarities and differences, as well as clarify how your approach advances beyond these existing methods.

References:
[1] Park, Jongwoo, et al. "Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA." arXiv preprint arXiv:2406.09396 (2024).

[2] Wang, Xiaohan, et al. "Videoagent: Long-form video understanding with large language model as agent." arXiv preprint arXiv:2403.10517 (2024).

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: In addition to the questions in the weakness section, can the authors also provide a density map for the center coordinates obtained with Eq. (8-9)?
soundness: 2
strengths: 1. The paper has a clear statement and is well organized.
2. Figure 1 is well illustrated, and the proposed method is intuitive to understand.
3. Experiments show that Free Video-LLM performs on par with baseline models with fewer visual tokens, and this improves the output speed of the LLM.
summary: This paper proposed a training-free inference method for video-LLM, namely Free Video-LLM, aiming to enhance the model efficiency without sacrificing the model performance. Specifically, it reduced the number of visual tokens to the LLM by selecting important frames in time and important tokens (with a ROI pooling) in space per frame with top-K prompt-guided selection, where the prompt was from the CLIP text encoder. Experiments were conducted with LLaVA-v1.6 and OpenAI CLIP-L-14 on 4 video QA benchmarks, showing performance on par with other baseline video-LLM models.
weaknesses: **Major**:  

1. The novelty of this paper is somewhat limited since using sparse video tokens is not new in the literature, e.g. [a] selects top-K visual tokens, [b] adopts token merging, [c] compresses visual tokens into context and content tokens using text decoder.  It is unclear how the performance and computation of the proposed token reduction method compared to these baselines under the same number of visual tokens.  

2. (Sec 3.3) The proposed Prompt-guided Spatial Sampling crops an ROI box (with alpha ratio of the image size) with a center coordinate obtained by the mean of top-K visual patch tokens. However, why not using these top-K visual tokens directly but using the ROI box?  It is also unclear how the proposed method is compared to using the image center as the center coordinate (with the same alpha). Note that all the ablations should be conducted with the same Video-LLM for fair comparison.  

3. The paper addresses the model efficiency as the number of input tokens into the LLM. It is true that longer sequence requires more computations in the LLM part. However, there is no computation analysis on the token selection part (before the LLM).  

4. (Table 6) The comparisons between prompt-guided temporal sampling and uniform temporal sampling only considers a low-frame-rate scenario, i.e. 3 frames. It is unclear whether the prompt-guided method is still important when more frames are taken (e.g. Table 2).

**Minor**:

5. The proposed method only keeps visual tokens that are relevant to the query question, which is helpful for the speed in this run, but if the user wants to ask follow-up questions, the visual tokens need to be processed again.  

6. Table 6 & 7 are not referenced in paragraphs.  

7. When citing prior works, it would be more readable to put authors’ names in bracket, e.g. Chat-UniVi Jin et al. (2024) -> Chat-UniVi (Jin et al., 2024).  

8. Typo: H_V in L154 -> H_I  

**Ref**:   
- [a] Li et al., SViTT: Temporal Learning of Sparse Video-Text Transformers, CVPR 2023.  

- [b] Bolya et al., Token Merging: Your ViT but Faster, ICLR 2023.  

- [c] Li et al., LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models, ECCV 2024.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Is it possible to use recent egocentric datasets and benchmarks for assessing QA performance of this Free Video LLM method?
soundness: 3
strengths: The core idea of the paper is to avoid unnecessary temporal frames and spatial areas as per the prompt, which leads to compute efficiency and a training-free video-LLM framework. We see that the framework produces competitive or sometimes better results on QA tasks, that too without requiring any training.
summary: The paper proposes prompt-guided temporal and spatial sampling for visual tokens. By virtue of being able to focus only on query-relevant regions, the method achieves QA performances efficiently.
weaknesses: For the problem statement of a training-free video LLM, the proposed framework achieves what it aims for. 
A bigger question for a video-llm field, in general, would be how to assess a model if it does temporal reasoning. The datasets shown in this paper aren't the best uses cases. I get that this is an a focus of this particular paper.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: Could you also discuss the limitations of the proposed framework/techniques?
soundness: 2
strengths: - The proposed prompt-guided visual perception framework is straightforward and intuitive. The two prompt-guided sampling techniques are all based on cosine similarity, which is easy to understand.
- The paper reads well and is easy to follow. It looks well-organized, with the required tables and equations placed throughout.
- The proposed method shows a good trade-off between accuracy and efficiency regarding the number of visual tokens.
summary: This paper introduces an efficient inference framework for training-free video LLMs. The presented prompt-guided visual perception framework reduces the number of visual tokens in two ways: First, it samples the prompt-related frames to temporarily drop unnecessary frames. Second, it selects a RoI to make the video LLMs focus on a smaller spatial area. This method, called Free Video-LLM, achieves competitive results while using fewer visual tokens. The authors show the effectiveness of Free Video-LLM on multiple QA benchmarks using the LLaVa-v1.6 model.
weaknesses: - [W1] Although the authors argue that the proposed method can focus only on the regions and time segments relevant to the prompt (l.74), they have not provided any empirical results supporting this argument. Does the prompt-guided temporal sampling technique really sample only the prompt-related frames? Does the prompt-guided spatial sampling select the most informative area? The presented results seem only anecdotal, and I believe the authors should provide direct evidence and deeper analysis.
- [W2] The prompt-guided spatial sampling technique is too naive. It finds a center position by averaging the top-K tokens and builds a RoI around the computed center position with a fixed ratio of 0.6. This has three problems: First, it might have a high degree of center bias, meaning that the center position of the RoI is highly biased to the center of the frame. This is because the proposed technique simply takes the mean value of the top-K tokens. Second, the fixed RoI ratio may not be optimal for other tasks or datasets. More advanced techniques can have an adaptive RoI ratio depending on the context of each video. Third, a single RoI might not be enough for some cases. For example, some tasks might require multiple regions that lie scattered. The proposed technique assumes that the prompt-related region is a single, connected area, which may not be valid for many other cases. I believe that more advanced techniques should be developed for better spatial sampling.
- [W3] The authors have used only one architecture, LLaVa-v1.6. The proposed sampling techniques may not work for other MLLMs. For example, it seems like they work only with CLIP models, which have aligned visual and text encoders.