BLIP-3-Video: You Only Need 32 Tokens to Represent a Video Even in VLMs

## Abstract

Abstract We present xGen-MM-Vid (BLIP-3-Video): a multimodal language model for videos, particularly designed to efficiently capture temporal information over multiple frames. BLIP-3-Video takes advantage of the ‘temporal encoder’ in addition to the conventional visual tokenizer, which maps a sequence of tokens over multiple frames into a compact set of visual tokens. This enables BLIP-3-Video to use much fewer visual tokens than its competing models (e.g., 32 vs. 4608 tokens). We explore different types of temporal encoders, including learnable spatio-temporal pooling as well as sequential models like Token Turing Machines. We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models (e.g., 34B), while being much smaller (i.e., 4B) and more efficient by using fewer visual tokens. The project website is at https://www.salesforceairesearch.com/opensource/xGen-MM-Vid/index.html

## 1 Introduction

Large Vision-Language Models (VLMs), benefiting from large-scale image-text training, have been dominating the field of computer vision.
Recently, open-source VLMs are obtaining strong results, despite having much smaller size than the commercial models (e.g., 4B vs. Trillions).

Further, in addition to such VLMs trained with images, VLMs for videos are becoming increasingly popular. The key component in a VLM for videos is the temporal abstraction of tokens over multiple frames. Models like Video-ChatGPT (Maaz et al., ) and PLLaVA (Xu et al., ) rely on a simple spatial/temporal pooling on top of image frame-level tokens to represent the entire video. Some models rely on a separate video encoder to capture temporal information in videos (Lin et al., ). Similarly, some models use additional convolutional layers (or Transformer layers) over frames to reduce their representation size (e.g., Video-LLaMA (Zhang et al., ), Kangaroo (Liu et al., )). Approaches that simply collect all the visual tokens from all the frames (e.g., MiniGPT4-video (Ataallah et al., ), LLaVA-NeXT (Li et al., ), Tarsier (Wang et al., ) and LLaVA-OneVision (Wang et al., )) also have been very popular recently, as they allow capturing all the details from the frame-level tokens.

Figure: Figure 1: SOTA video VLM model comparison: (Left) Number of visual tokens vs. video-QA accuracy. (Right) Model size vs. video-QA accuracy.
Refer to caption: x1.png

However, this often makes the number of tokens for video to be very huge (e.g., thousands even for 8 frames). Such large number of video tokens could be critical for longer videos as the LLM computation is quadratic to the number of total tokens.

In this paper, we introduce xGen-MM-Vid (BLIP-3-Video), which is an efficient compact vision-language model with an explicit *temporal encoder*, designed particularly for videos. BLIP-3-Video particularly focuses on incorporating a learnable ‘temporal encoder’ within it. We explore different types of temporal encoder, and demonstrate that the model can abstract each video into much fewer visual tokens (e.g., 16) while being successful in open-ended question-answering and captioning tasks. We include a space-time attentional pooling as well as a sequential model as our temporal encoder, relying on token operations to iteratively abstract a series of frame-level tokens into a learnable memory.

There has been prior work investigating the role of pooling (Jin et al., ), convolutions, and cross attention layers (Zhang et al., ; Liu et al., ; Li et al., ), but study on full space-time attentional pooling or sequential model to this extent has been limited in the past. Our objective in this paper is to provide a fundamental alternative to more brute-force way of collecting all the visual tokens which have been increasing popular recently. We experimentally confirm that $16\sim 32$ video tokens abstracted by the temporal encoder is often sufficient to represent the entire video for question-answering (Figure 1).

## 2 BLIP-3-Video

### 2.1 Model architecture

Figure: Figure 2: An illustration of the BLIP-3-Video model architecture. It has the explicit temporal encoder inserted to BLIP-3.
Refer to caption: x3.png

We build the BLIP-3-Video model based on the image-based vision-language model (VLM), BLIP-3 (Xue et al., ).

The model architecture is composed of the following four components: (1) the vision encoder (ViT) taking each frame input, (2) the frame-level tokenizer to reduce the number of tokens, (3) the temporal encoder to build video-level token representations, and (4) the autoregressive LLM generating output text captions based on such video tokens and text prompt tokens. Figure 2 shows an overview.

First, we apply a pretrained SigLIP as the vision encoder, designed to take one single image frame at a time. Perceiver-Resampler is then applied to map such visual tokens into $N=128$ visual tokens per frame, independently.
Once the model has such visual tokens over time (i.e., over multiple frames in the video), they are provided to an explicit ‘temporal encoder’. The role of the temporal encoder is to build a video-level token representation from such sequence of image-level tokens, serving as a mapping function between a set of $N\cdot T$ image tokens to $M$ video tokens where $T$ is the number of frames and $M$ is a constant number of tokens. We explore various forms of the temporal encoder, including temporal pooling as well as sequential models, which we discuss further in the following subsection. The resulting tokens are given to the LLM together with the encoded text tokens in a prefix manner, as in many standard VLMs.

For computational efficiency, the model takes uniformly sampled 8 frames per video. As a result, in our model, ViT first maps a video into $8*729$ visual tokens, which is then mapped to $8*128$ visual tokens using Perceiver-Resampler, and then to $16\sim 128$ video tokens using the temporal encoder.

We use Phi-3 (Abdin et al., ) as our LLM backbone taking such video tokens in addition to the text prompt tokens. This enables the model to take text+video as an input and generate text sentences as an output.

### 2.2 Temporal encoders

A temporal encoder is a function of tokens, taking $N\cdot T$ tokens as an input and returning $M$ tokens as an output: $x_{1,\ldots,M}=f(v_{(1,1),\ldots,(N,T)})$.

We explore different types of encoders as part of our model. The simplest form of the temporal encoder will be temporal pooling, e.g., summating per-frame tokens over time: $x_{1,\ldots,M}=\left\{\sum_{t}(v_{(i,t)})\right\}_{i=1}^{M}$ where $M$ is always restricted to be identical to $N$, which was also used in (Maaz et al., ). Another possible implementation would be the use of a temporal Transformer, modeling the entire token sequence and selecting the last $m$ tokens similar to Mirasol3B (Piergiovanni et al., ):

$$ $\displaystyle x_{1,\ldots,M}=\left\{\textrm{Transformer}(v)\right\}_{N\cdot T- M+1}^{N\cdot T}$ (1) $$

In addition to the straightforward temporal encoders mentioned above, we explore two important temporal encoders considering space-time nature of tokens: spatio-temporal attentional pooling and sequential models (Figure 3).

Figure: Figure 3: Visually comparing different types of temporal encoders we explored in our model architecture. (c) and (d) are particularly effective, as we discuss further in the experiments.
Refer to caption: x4.png

#### Spatio-temporal attentional pooling:

Attentional pooling allows learnable ‘soft selection’ of multiple tokens given a larger set of tokens. Such attentional pooling have been previously developed for Transformers (e.g., Perceiver (Jaegle et al., ) and TokenLearner (Ryoo et al., )), and also used in earlier foundation models (e.g., CoCa (Yu et al., )) for images.

In our model, we use TokenLearner (Ryoo et al., ), making it explicitly serve as our space-time aware temporal encoder. Unlike previous per-image-frame usage of poolings where spatial pooling and temporal pooling are applied separately (e.g., Video-ChatGPT), our temporal encoder directly takes all $N\cdot T$ tokens and ‘learns’ to soft-select $M$ informative tokens spatio-temporally. Here, $N$ tokens could be viewed as spatial representations of a frame and we have $T$ of them, suggesting it is a spatio-temporal representation selection.

Our attentional pooling in its simplest form is expressed as:

$$ $\displaystyle x_{i}=A(V)\cdot V=\textrm{softmax}\left(\alpha(V^{T})\right)\cdot V$ (2) $$

where $V$ is a matrix formed by concatenating input tokens $v_{(1,1),\ldots,(N,T)}$. The function $A(\cdot)$ computes the summation weights for $V$, performing soft selection of tokens. This is further decomposed to the softmax and the function $\alpha(\cdot)$. In Perceiver, a matrix multiplication with a latent query tokens (i.e., cross attention where $|Q|=m$) have been used to implement this: $\alpha(V)=Q\cdot V^{T}/c$. TokenLearner uses a convolution/MLP on top of $V$: $\alpha(V)=\textrm{MLP}_{m}(V^{T})$, which we use in our model. This allows selecting a smaller number of tokens (e.g., $M=32$ tokens).

We experimentally confirm that such learnable spatio-temporal attentional pooling has advantages over the conventional approach of non-learnable spatial pooling and temporal pooling, in Section 3.3.

#### Sequential Model:

We also deploy Token Turing Machines (TTM) (Ryoo et al., ) as a temporal encoder, which is a sequential model capable of taking any number of frames to generate a video-level token representation (e.g., $M=32$ regardless the number of frames). Our use of TTM is similar to its usage in Mirasol3B (Piergiovanni et al., ), except that our model uses TTM directly to encode a sequence of image tokens while Mirasol3B uses TTM to encode a sequence of sets of video tokens.

Here, we also further extend TTM by adding time-stamped positional encodings to embded the frame index of each token in the latent space. This enables the tokens in the ‘memory’ of TTM to preserve the temporal ordering information, which is crucial when representing complicated or long video scenes. In addition, we use TTM temporal encoder in a ‘grouped’ fashion, maintaining a separate memory of size $G=4$ for each of $N=128$ tokens over time. The memory is maintained to have the size is $N\cdot G$, and the final output from the sequence model is attentionally pooled from the final memory to give $M$ tokens.

### 2.3 Training recipe

BLIP-3-Video follows a three-stage curriculum learning: (1) image caption pretraining, (2) video caption pretraining, and (3) video instruction tuning. In all its training we freeze the vision encoder, only training the model parameters after the vision encoder. First, we directly use the pretrained weights from BLIP-3 (Xue et al., ). BLIP-3 is for images and it does not contain weights for the temporal encoder, so we randomly initialize those weights.

As its 2nd stage, the model is then trained on LLaVA-Hound-DPO’s video caption data (Zhang et al., ), featuring over 900k video captions. Instead of directly using the text captions provided in LLaVA-Hound-DPO, we used GPT-4 to rephrase such text captions so that they become more GPT-style captions.

Finally, we tuned the model using a mix of video question-answering datasets, including VideoChatGPT’s 99k-sample video instruction tuning data (Maaz et al., ), along with the training splits of the MSVD-QA (Xu et al., ), MSRVTT-QA (Xu et al., ), ActivityNet-QA (Yu et al., ), TGIF-QA (Jang et al., ), and NExT-QA (Xiao et al., ) datasets, which contain 30k, 149k, 32k, 71k, and 34k samples, respectively. For TGIF-QA, we only used the training data associated with the Repeating Action and State Transition tasks. In our video instruction tuning recipe, we employ both open-ended and multiple-choice video QA formats for TGIF-QA and NExT-QA. For the open-ended video QA training data sourced from the MSVD-QA, MSRVTT-QA, TGIF-QA, and NExT-QA training sets, we used GPT-3.5 to rephrase the original single-word or single-phrase answer into a natural language sentence, providing the question in the LLM prompt context. For open-ended TGIF-QA and NExT-QA, we also double the sample size by using both the original short-phrase answers and the rephrased sentence-based answers.
In addition, we added a filtered version of the Mira caption dataset (Ju et al., ) for our video instruction tuning. That is, we are using both video question-answering and video captioning for our final training. We excluded captions for Mira videos longer than one minute, totaling 935k video caption samples.

We trained our model with 8 $\times$ H100 GPUs. For the video caption pretraining, we use the batch size of 16 per GPU, 500 warmup steps, and the learning rate of 2e-5 with the cosine decay. We trained the model for 1 epoch. The video QA sft (i.e., instruction tuning) was done with the batch size of 4 per gpu, 500 warmup steps, and the learning rate of 1e-5 with the cosine decay. We trained the model for 1 epoch in this case as well. The entire training (combining both video pretraining and the sft) takes around 12 hours, confirming the efficiency of our model.

## 3 Experiments and Results

### 3.1 Model implementation details

We share the model details with BLIP-3 (4B), except that BLIP-3-Video has the new temporal encoder component in its architecture. This model takes the video with the input resolution of 384$\times$384, using SigLIP encoder to map it to 729 tokens per frame with the channel size 1152. Perceiver-Resampler is implemented with multiple cross-attention layers with the same channel dim, which is then given to the temporal encoder.

TokenLearner serving as the spatio-temporal attentional pooling was implemented using a MLP as the attention function. The size of its inner dim was the number of target tokens * 2. The grouped TTM serving as the sequential model temporal encoder was implemented using 4 Transformer layers (with the channel dim of 1152) as the processor module while using TokenLearners for read/write modules. Memory size was set to $N*4=512$ tokens total.

The resulting $16\sim 128$ tokens are mapped to the text embedding with the channel dimension of 3072, before given to the LLM (Phi-3).

### 3.2 Public benchmarks

We conducted experiments measuring video question-answering accuracies on multiple public datasets. This includes open-ended answer generation tasks like MSVD-QA, as well as multiple choice questions like NExT-QA. We follow their standard settings in all cases.

Table 1 compare open-ended question answering accuracies of BLIP-3-Video against reported numbers of other models. We use four commonly used public datasets, MSVD-QA, MSRVTT-QA, ActivityNet-QA, and TGIF-QA, following standard VideoLLM evaluation settings. Note that our MSVD-QA and MSRVTT-QA accuracy was measured by training our model with a subset (i.e., Video-ChatGPT dataset-only) of our training data, as this allows more direct comparison to some of the prior work and enables more stable results due to its data distribution. We are including the model size as well as the number of visual tokens in the table. We are able to observe that, despite its smaller size (i.e., 4B vs. 7B or 34B), our model is obtaining superior or comparable performance.

With the temporal encoder, BLIP-3-Video was able to retain the performance with much fewer tokens, which we discuss more in the following subsection. Our results suggest that not too many visual tokens are really necessary to be successful on these open-ended question answering benchmarks, as long as we have a carefully designed temporal encoder.

In addition, we evaluated BLIP-3-Video’s ability to solve multiple choice questions (MCQ). Table 2 shows the results on NExT-QA. Due to the nature of its questions requiring understanding of multiple frames, many prior models use quie a bit of tokens. For instance, GPT-4 uses a minimum of 255 tokens per frame. It is interesting that BLIP-3-Video achieves comparable accuracy while representing the entire video with only 32 (or 128) tokens.

**Table 1: Comparison against reported numbers of other models on open-ended question answering evaluation. The number of visual tokens are also reported. The numbers after ‘/’ are answer quality scores. ^∗ indicates our evaluation using the checkpoint and inference code provided by the author, with the identical videos used in our model (8 frames of 384$\times$384 resolution).**
| Method | Size | #tokens | MSVD-QA | MSRVTT-QA | ActivityNet-QA | TGIF-QA |
| --- | --- | --- | --- | --- | --- | --- |
| VideoChat (Li et al., ) | 7B | 32 | 56.3 / 2.8 | 45.0 / 2.5 | - / 2.2 | 34.4 / 2.3 |
| Video-LLaMA (Zhang et al., ) | 7B | 32 | 51.6 / 2.5 | 29.6 / 1.8 | 12.4 / 1.1 | - / - |
| Video-ChatGPT (Maaz et al., ) | 7B | 264+ | 64.9 / 3.3 | 49.3 / 2.8 | 34.2 / 2.8 | 51.4 / 3.0 |
| Chat-UniVi (Jin et al., ) | 7B | 112 | 69.3 / 3.7 | 55.0 / 3.1 | 46.1 / 3.3 | 69.0 / 3.8 |
| LLaMA-VID (Li et al., ) | 7B | 32 | 69.7 / 3.7 | 57.7 / 3.2 | 47.4 / 3.3 | - |
| LLaMA-VID (Li et al., ) | 13B | 32 | 70.0 / 3.7 | 58.9 / 3.3 | 47.5 / 3.3 | - |
| Video-LLaVA (Lin et al., ) | 7B | 2048 | 71.8 / 3.9 | 59.2 / 3.5 | 45.3 / 3.3 | 70.0 / 4.0 |
| MiniGPT4-Video (Ataallah et al., ) | 7B | 2880+ | 73.9 / 4.1 | 59.7 / 3.3 | 46.3 / 3.4 | 72.2 / 4.1 |
| PLLaVA (Xu et al., ) | 7B | 576+ | 76.6 / 4.1 | 62.0 / 3.5 | 56.3 / 3.5 | 77.5 / 4.1 |
| SlowFast-LLaVA Xu et al. () | 7B | 3680 | 79.1 / 4.1 | 65.8 / 3.6 | 56.3 / 3.4 | 78.7 / 4.2 |
| LLaVA-Hound-DPO Zhang et al. () | 7B | 2048 | 80.7 / 4.1 | 70.2 / 3.7 | - / - | 61.4 / 3.5 |
| LLaVA-OneVision^∗ (Wang et al., ) | 7B | 1568 | 72.9 / 3.9 | 57.8 / 3.4 | 55.3 / 3.6 | 41.1 / 3.1 |
| Tarsier (Wang et al., ) | 7B | 4608+ | 77.0 / 4.1 | 62.0 / 3.5 | 59.5 / 3.6 | 79.2 / 4.2 |
| Tarsier ^∗ (Wang et al., ) | 7B | 4608 | 74.4 / 4.0 | 59.1 / 3.4 | 54.3 / 3.5 | - / - |
| PLLaVA (Xu et al., ) | 34B | 576+ | 79.9 / 4.2 | 68.7 / 3.8 | 60.9 / 3.7 | 80.6 / 4.3 |
| LLaVA-NeXT-Video^∗ (Li et al., ) | 32B | 1152 | 73.6 / 4.0 | 56.8 / 3.4 | 58.4 / 3.6 | 73.5 / 4.1 |
| Tarsier (Wang et al., ) | 34B | 4608+ | 80.3 / 4.2 | 66.4 / 3.7 | 61.6 / 3.7 | 82.5 / 4.4 |
| Tarsier ^∗ (Wang et al., ) | 34B | 4608 | 79.3 / 4.1 | 62.2 / 3.5 | 61.5 / 3.7 | - / - |
| BLIP-3-Video | 4B | 32 | 77.7 / 4.2 | 60.0<br>/ 3.6 | 55.7 / 3.5 | 76.5 / 4.3 |
| BLIP-3-Video | 4B | 128 | 77.9 / 4.3 | 59.7 / 3.6 | 56.9 / 3.6 | 77.1 / 4.3 |

**Table 2: Comparison against reported numbers of other models on multiple choice question-answering (MCQ) benchmark.**
| Method | Size | #tokens | NExT-QA |
| --- | --- | --- | --- |
| LangRepo (Kahatapitiya et al., ) | 7B | 3136+ | 54.6 |
| LangRepo (Kahatapitiya et al., ) | 12B | 3136+ | 60.9 |
| Tarsier (Wang et al., ) | 7B | 4608+ | 71.6 |
| LLoVi (Zhang et al., ) | 157B | 1000s | 67.7 |
| IG-VLM (Kim et al., ) | 34B | 1536+ | 70.9 |
| VideoAgent (Wang et al., ) | GPT-4 | 2091+ | 71.3 |
| VideoTree (Wang et al., ) | GPT-4 | 3978+ | 73.5 |
| Tarsier (Wang et al., ) | 34B | 4608+ | 79.2 |
| BLIP-3-Video | 4B | 32 | 76.4 |
| BLIP-3-Video | 4B | 128 | 77.1 |

### 3.3 Ablations

We conducted an ablation comparing different temporal encoders. These include: (1) the base single frame model (i.e., BLIP-3 trained with videos), (2) mean pooling similar to Video-ChatGPT, and (3) transformer temporal encoder similar to Mirasol3B. We also tried the (4) vanilla Token Turing Machines, which is not the grouped version we use as our temporal encoder.

Table 3 shows the result, comparing the question-answering accuracies of different types of temporal encoders when abstracting a video into 128 tokens. We are able to observe that they all do a reasonable job, while some temporal encoders are more effective.

In addition, we compared different pooling approaches similar to the ones tried in prior works, when they are required to select a much smaller number of tokens (e.g., 32) from a large set of visual tokens. We compare our spatio-temporal attentional pooling as well as the sequential model against its alternatives, including (1) fixed-window (non-learnable) space-time pooling and (2) learnable ‘per-frame’ pooling. In particular, (2) is similar to the approach taken in LLaMA-VID (Li et al., ), which independently selected a fixed number of tokens (e.g., 2) per frame. Table 5 shows the results. Table 5 explicitly compares the impact of having smaller visual tokens. We are able to observe that 32 visual tokens or more gives a reasonable video QA accuracy.

**Table 3: Ablations comparing different temporal encoders: 128 tokens. ^∗A slightly different training recipe using a subset of the entire dataset (without Mira data) was used for the ablations.**
| Encoder | MSVD-QA | TGIF-QA | ActivityNet-QA | NExT-QA |
| --- | --- | --- | --- | --- |
| 1 frame | 71.49 / 4.01 | 72.74 / 4.16 | 51.83 / 3.39 | 72.79 |
| Mean pooling | 76.75 / 4.17 | 77.01 /4.30 | 55.89 / 3.53 | 76.24 |
| Transformer | 76.24 / 4.15 | 76.33 / 4.28 | 55.59 / 3.50 | 76.34 |
| Vanilla Token Turing Machine | 76.42 / 4.15 | 75.80 / 4.26 | 54.45 / 3.48 | 75.42 |
| Ours (Space-time) | 77.49 / 4.18 | 76.90 / 4.29 | 56.94 / 3.56 | 76.27 |
| Ours (Sequential) | 77.86 / 4.20 | 77.10 / 4.31 | 56.66 / 3.56 | 77.07 |

**Table 4: Ablations comparing different pooling strategies for 32 tokens.**
| Encoder | MSVD-QA |
| --- | --- |
| Space-time pooling (4*8) | 76.04 |
| Per-frame (4*8) | 76.78 |
| Ours (Space-time) | 77.71 |
| Ours (Sequential) | 77.11 |

#### Speed:

Reducing the number of visual tokens increases the computational efficiency of the models, as the total computation is quadratic to the number of tokens fed to the LLM. We measure the runtime of our models in the training setting for the fair comparison. Here, we report ‘samples per second per GPU’. Without the temporal encoder (i.e., directly using 1024 visual tokens), the model processed 3.3 samples per second. With 16/32/128 tokens using the temporal encoder, the model was able to process 8.5 / 8.2 / 7.5 samples per second.

Figure: Figure 4: Example video captioning results on Mira dataset, formed in question-answering style.
Refer to caption: x5.png

Figure: Figure 5: Example video captioning results on MSVD and MSRVTT caption dataset.
Refer to caption: x6.png

### 3.4 Video captioning evaluation

**Table 6: Video caption evaluation results using 8 frames. We employ VideoChatGPT’s LLM evaluation and report Average Accuracy / Average Score in this table. The ‘captioning-only model’ was trained only using Mira video caption data (without QA data), making it specialized for the captioning.**
| Method | Size | # tokens | MSVD-Cap | MSRVTT-Cap | Mira-Cap |
| --- | --- | --- | --- | --- | --- |
| LLaVA-OneVision | 7B | 1152 | 61.62 / 3.31 | 38.60 / 2.71 | 48.83 / 3.10 |
| Tarsier | 7B | 4608 | 62.26 / 3.37 | 40.27 / 2.77 | 40.55 / 2.87 |
| BLIP-3-Video | 4B | 32 | 63.59 / 3.38 | 42.06 / 2.82 | 80.67 / 3.96 |
| BLIP-3-Video | 4B | 128 | 64.17 / 3.41 | 43.05 / 2.85 | 81.13 / 3.97 |
| BLIP-3-Video (captioning-only model) | 4B | 128 | 69.50 / 3.52 | 50.45 / 2.98 | 81.76 / 4.00 |

We evaluate our model on the video captioning task by comparing it against state-of-the-art models on the test splits of MSVD-Caption and MSRVTT-Caption, as well as a custom evaluation split from the Mira dataset. For the Mira dataset, we randomly selected 6,000 samples from our full, filtered data to create the evaluation split, with the remainder used for training. We employed Video-ChatGPT’s LLM evaluation, specifically using GPT-3.5 to compare model-predicted captions with ground truth captions. The LLM assesses accuracy by checking if the predicted caption matches the ground truth, and assigns a score on a scale of 0 to 5 for each sample.

Table 6 demonstrates the results. All three models were provided with 8 frames per video, and consistent visual input and prompts were ensured across the models. Our BLIP-3-Video consistently outperforms LLaVA-OneVision-7B and Tarsier-7B across all three video captioning benchmarks, with particularly notable improvements on the Mira video captioning task.

We present qualitative video captioning results for the Mira dataset in Figure 4 (more details in Appendix) and for the MSVD and MSRVTT datasets in Figure 5. BLIP-3-Video generates high-quality, detailed captions.

## 4 Related Works

### 4.1 Image-Text LLMs

Among recent advances in image-text multimodal models (Li et al., ; Alayrac et al., ; Liu et al., ; Dai et al., ; Xue et al., ; Laurençon et al., ; Deitke et al., ), one common strategy enable image understanding in LLM is to start with a pre-trained image encoder (e.g., ViT (Radford et al., ; Zhai et al., )) and a pre-trained language-only LLM (Abdin et al., ; Bai et al., ; Dubey et al., ). The two components are connected via a vision-language connector, which is trained to project vision embeddings output from the vision encoder into “vision tokens" that can be ingested by the LLM. The vision tokens are of the same shape as language embeddings, so the image-text LLM can be trained in the same way as regular language models using the next token prediction loss. There are many design choices for the VL connector, for example, BLIP-2 (Li et al., ) chooses to use a Q-Former to extract vision tokens from the vision embeddings, Flamingo (Alayrac et al., ) uses “perceiver resampler" as the connector plus cross-attention layers throughout the language model, while a simpler choice is to use MLP layers to transform the vision embeddings.
Image-text LLMs are usually trained with a multi-stage training strategy, including pre-training, instruction tuning, and sometimes, post-training (e.g., DPO (Rafailov et al., )). In addition to simple structured image-text data such as image captioning and single-image VQA, recent works also explore free-from image-text data for model training such as interleaved image-text understanding (Laurençon et al., ; Awadalla et al., ) and multi-image VQA (Jiang et al., ; Li et al., ).

### 4.2 Video LLMs

Video LLMs extend the architecture of image-based LLMs to handle video input. Zhang et al. () integrates pre-trained encoders and frozen LLMs to process multimodal input through a Video Q-Former and Audio Q-Former, generating video and audio embeddings compatible with LLM without retraining encoders. Maaz et al. () adapts the CLIP visual encoder for video by incorporating temporal features and fine-tunes the model using video-instruction pairs collected by tools like BLIP-2 (Li et al., ) and GRiT (Wu et al., ). Li et al. () generates frame-level embeddings using a visual encoder but condenses visual information into two tokens per frame. However, it does not account for temporal recency across frames. Similarly, models like Video-LLaVA (Lin et al., ) and LLaVa-OneVision (Li et al., ) treat videos as long multi-image contexts but lack token efficiency optimization, making them computationally costly. SlowFast-LLaVA (Xu et al., ) adopts a two-stream architecture—Slow and Fast pathways—to capture both spatial and temporal video semantics without extra fine-tuning. Finally, LLaVa-hound-DPO (Zhang et al., ) uses Direct Preference Optimization (DPO) (Rafailov et al., ) with GPT-4V to annotate preference data, enhancing video question-answering performance by detecting inconsistencies or hallucinations in model responses.

### 4.3 Token Pruning

Token pruning is a widely used technique to reduce redundant and overlapping information in Vision Transformers (ViTs) and large language models (LLMs). Bolya et al. () merges similar tokens within ViTs, combining redundant content while retaining task-relevant information across tasks like image, video, and audio processing. Similarly, Ren et al. () employs the Temporal Aggregation Module to combine redundant consecutive video frames and the Spatial Aggregation Module to merge similar patches within each frame, reducing the number of processed tokens by up to 75%. Shen et al. () focus on temporal redundancy and progressively merges tokens across neighboring clips, which reduces the number of tokens by preserving important video-level features. All these methods focus on visual token merging in ViTs, where token processing is challenging in video-based LLMs. In addition, Chen et al. () improves attention efficiency in deeper layers by dynamically pruning or merging redundant image tokens based on attention scores without extra training. Shang et al. () introduces adaptive token reduction through the Adaptive Important Token Selection and Token Supplement, which can be integrated into VLM models without fine-tuning. In LLMs, KV cache pruning is popular for efficient model serving, as seen in (Fu et al., ), which uses attention maps to progressively prune tokens and reduce the time-to-first-token (TTFT). Wan et al. () extends KV cache pruning to VLMs, employing different token merging strategies to cut computational costs and support longer multimodal contexts.

## 5 Conclusion

We introduce BLIP-3-Video, which is an efficient, compact vision-language model for videos with 4B parameters. BLIP-3-Video incorporates a temporal encoder in its architecture, which allows the model to abstract the entire video with as few as 16 or 32 tokens. In contrast to many state-of-the-art video VLMs taking advantage of thousands of visual tokens to represent a video (e.g., 4608), BLIP-3-Video shows a competitive performance while utilizing much fewer visual tokens (e.g., 32).

## Appendix

Figures 6 and 7 compares the video captioning results of BLIP-3-Video against Tarsier and LLaVA-OneVision. The models take the exact same video input with the same number of frames, for a fair comparison. We observe that BLIP-3-Video generate captioning much more similar to the ground truth. It also is able to provide more holistic understanding of the video (rather than focusing on unnecessary details such as texts in the background) while suffering from less hallucinations.

Figure: Figure 6: Example video captioning results on Mira dataset, formed in question-answering style. We compare the outputs of BLIP-3-Video, Tarsier, and LLaVA-OneVision. GT stands for the ground truth. Different colored texts are different parts of ground truth captions and their corresponding sentences in the model outputs. Underlined texts are hallucinations.
Refer to caption: x7.png

Figure: Figure 7: Example video captioning results on Mira dataset, formed in question-answering style. We compare the outputs of BLIP-3-Video, Tarsier, and LLaVA-OneVision. GT stands for the ground truth. Different colored texts are different parts of ground truth captions and their corresponding sentences in the model outputs. Underlined texts are hallucinations.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What novel designs does this method introduce compared to TTM? Are there ablation studies for these designs?

2. The model utilizes the VideoChatGPT instruction set. Why hasn't it been evaluated on that benchmark?
soundness: 2
strengths: 1. An impressive tradeoff between efficiency and accuracy on the MSVD-QA benchmark.
2. Extensive explorations on temporal encoders to reduce visual tokens.
3. The paper is well written and easy to follow.
summary: This paper presents BLIP-3-Video, which introduces a "temporal encoder" alongside a conventional visual tokenizer, allowing it to significantly reduce visual tokens (32 tokens compared to thousands in other models). The study explored various temporal encoders, including learnable spatio-temporal pooling and sequential models like token turning machines (TTM). Detailed experiments showed that different encoder types had a noticeable impact on performance, particularly in handling complex video scenarios. Experimental results show that BLIP-3-Video achieved video question-answering accuracies comparable to much larger state-of-the-art models while being smaller and more efficient.
weaknesses: 1. Novelty: The primary weakness is the insufficient novelty. As detailed in Section 2.2, the only improvements to TTM include (1) time-stamped positional encodings and (2) a 'grouped' TTM temporal encoder. These minor changes do not substantiate a significant contribution.

2. Evaluation Benchmarks: The evaluated benchmarks are unconvincing for assessing Video LMMs. The model was only evaluated on MSVD-QA, MSRVTT-QA, ActivityNet-QA, TGIF-QA, and NExT-QA, which are not so ideal for testing LMMs. The authors may consider newer benchmarks like VideoMME and MVBench, which are proposed for assessing Video LMMs.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - The (video part of the) training of this model is on video captioning data and video question-answering datasets. If the downstream task were to change to a more complex task, like temporal reasoning, would the model require more tokens or would 16~32 still be sufficient? i.e. is there enough visual information encoded in the 16~32 tokens?
- In addition, if the downstream task requires remembering multiple details and nuanced events over a long diverse scenario, how would this approach perform? Is there a built-in mechanism that prevents information loss during token pooling?
soundness: 3
strengths: - The temporal encoder which aggregates visual tokens across frames in a highly efficient manner results in computational efficiency of training the model.
- Extensive ablation study in the temporal encoder design validates the design choice while also demonstrating the flexibility in the design of temporal encoder. 
- Competitive performance on various video question-answering benchmarks, despite its smaller size.
summary: The paper introduces BLIP-3-Video, a multimodal vision language model which demonstrates strong performance on video understanding with high token accuracy. BLIP-3-Video uses as few as 16~32 tokens to encode an entire video sequence, which is highly efficient to other video VLMs, enabled by the incorporation of a temporal encoder. BLIP-3-Video achieves competitive accuracy on various video question-answering benchmarks while having smaller model parameters.
weaknesses: - The model proposed in the paper utilizes 8 frames per video which are uniformly sampled. This approach might not work for tasks that inherently require more than 8 frames to understand the video. If this method could scale up, an explanation of why that might be would be helpful.
- The experiments of the paper focuses on video question-answering benchmarks only, and this limited experimentation may not capture the model's ability in other video-based tasks. Further evaluation on other video tasks, such as temporal understanding would demonstrate the applicability of this approach to more general and diverse video-related tasks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Can one BLIP-3-Video model produce 32 and 128 tokens on demand as a hyper parameter? Or they are different models trained individually?

2. What does text encoder in Fig 2 mean? The text tokenizer?
soundness: 3
strengths: This paper investigates an important topic, how to efficiently & effectively understand videos by LLMs, which is underexplored so far. This paper proposes the BLIP-3-Video to understand videos with just 32 tokens (in LLMs) based on Phi-3, and it outperforms both parameter-heavy or visual-token-heavy models (as shown in Fig 1) on QA and captioning benchmarks. Additionally, this paper presents a compelling finding (somewhat): a video can be effectively represented by just 32 tokens in LLMs for QA and captioning tasks. I think this research line is promising and can benefit several downstreaming tasks, e.g., captioning for text-to-video generation.
summary: This paper presents an efficient Video LLM, coined as BLIP-3-Video, by incorporating extra modules to transform dense video tokens into sparse tokens (e.g., # can be 32). Specifically, the authors use a sequential transformer (so called token turing machine) on top of frame-wise image tokens and perceiver-resampler to produce limited 32 tokens. Diverse video QA benchmarks show the competitive performance of BLIP-3-Video on video question answering tasks. The authors also test the captioning ability of BLIP-3-Video.
weaknesses: Although this is overall a good paper, several concerns are here:
1. The presentation of the main method (Sec 2.2) somwhat presents confusion: Does BLIP-3-Video both use spatio-temporal attentional pooling and TTM? Is there a perceive resampler before temporal encoder in BLIP-3-Video (cannot be infered from Figure 2)?

2. Compressing a video into 32 tokens is a compelling and exciting idea. However, I am worried that spatial-temporal details will be missing through compression, which is crucial for some detailed reasoning in LLMs. More evaluation of BLIP-3-Video on diverse tasks beyond captioning and MCQ are encouraged.
(also, as the compression is not text query guided, the compression is solely dominated by the visual information itself. That is to say, 32 tokens per a video are fixed under different text query, which might not be appropriate in general)

3. Relating with the weakness 1, extra modules introduced besides the visual encoder (SigLIP) and LLM sound too complicated. If I understand correctly, there are a perceiver-resampler and a temporal encoder (attention pooling or TTM). My idea is naive and simple, can we just finetune a perceiver-resampler in BLIP-3 into a temporal encoder, rather than just compressing tokens per frame? Given the strong performance of cross attention layers in the perceiver resampler, this seems to be a missing but promising ablation study in this paper.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: 1. **Temporal Encoder Generalization**: A smaller number of visual tokens is particularly important for understanding longer videos. However, this paper only tested on a few simple short video benchmarks. Please provide test results on video benchmarks of different lengths and scenarios, such as VideoMME, MVBench, etc.

2. **Scalability Concerns**: How does the model's performance and efficiency scale with longer videos, and have you observed any limitations in terms of the number of frames the model can effectively process?

3. **Model Interpretability**: The paper mentions the use of different types of temporal encoders. Are there any plans to provide insights into how these encoders make decisions?

4. **Comparison with Other Efficient Models**: How does BLIP-3-Video compare with other recent models that also focus on efficiency, such as those employing knowledge distillation or sparse attention mechanisms? Could the authors provide some insights into the trade-offs involved? Please provide persuasive evidence from experimental studies.

5. **Novelty**: Compared to LLaMA-VID, where is the core novelty of this paper? Although the experimental results show that 32 tokens achieved better performance on four short video benchmarks, this standard will change with different video lengths, video scenarios, and the complexity of question answering. The scalability and generalizability of this method are questionable. Perhaps a more effective mechanism for accommodating more frames and selecting key information for video question answering from a large number of visual tokens is worth exploring, rather than determining the specific numerical value of a visually overfitted token on a few benchmarks. Similar architectures have been explored enough in a series of works such as Video-LLaVA, LLaVA-VID, LLaVA-NEXT, and so on.
soundness: 2
strengths: - **Originality**: It introduces an innovative temporal encoder that significantly reduces the number of visual tokens needed to represent videos, offering a new approach to efficiency in video understanding models.

- **Quality**: The model is thoroughly evaluated against state-of-the-art benchmarks, demonstrating competitive performance. The ablation studies are providing insightful analyses into the model's components.

- **Clarity**: The paper is well-organized, with clear explanations and visual aids that effectively convey complex information, making the technical content accessible to readers.

- **Significance**: BLIP-3-Video's efficiency in handling video data with fewer resources.
summary: The paper introduces BLIP-3-Video, a novel multimodal language model designed for video understanding that efficiently captures temporal information across frames. A key innovation is the integration of a 'temporal encoder' that maps a sequence of tokens from multiple frames into a compact set of visual tokens, allowing BLIP-3-Video to operate with significantly fewer visual tokens compared to its competitors. The model explores various temporal encoders, including learnable spatio-temporal pooling and sequential models like Token Turing Machines. Experiments demonstrate that BLIP-3-Video achieves comparable video question-answering accuracies to much larger models, while being more efficient and smaller in size due to its reduced token usage. The paper also details the model's architecture, training recipe, and experimental results, highlighting the effectiveness of the temporal encoder in representing videos for question-answering tasks with a small number of tokens.
weaknesses: - **Diversity of Datasets**: The experiments primarily rely on a limited set of public benchmarks for evaluation. Expanding the evaluation to include a more diverse range of benchmarks, particularly those with varying lengths and complexities of videos, could provide a more comprehensive assessment of the model's generalizability and robustness.

- **Scalability Analysis**: While the paper demonstrates the model's efficiency, there is a lack of analysis on how the model scales with increasing video length and complexity. Future work could benefit from exploring the model's performance as it processes longer videos, which is crucial for real-world applications.

- **Comparison with State-of-the-Art**: Although comparisons are made with other models, the paper could benefit from a more detailed analysis comparing the trade-offs between BLIP-3-Video and the state-of-the-art models in terms of accuracy, computational resources, and inference time.

- **Implementation Details**: Some aspects of the model's implementation, such as the specific choices made in the architecture of the temporal encoder, could be elaborated upon with more technical depth. This additional detail would aid other researchers in understanding the design decisions and potentially replicating or improving upon them.

Meta-review: This paper provides BLIP-3-Video, a Video LLM that build on the previous BLIP-3 architecture but focuses on incorporating temporality into the architecture by learning spatio-temporal pooling to obtain video representations in only 32 tokens. The paper is clearly written, provides a thorough analysis of various pooling strategies and tackles a topic of high interest to the community.
The paper's key weakness is the limited novelty, it is rather an exploration of architectures. Especially compared to TTM, the novelty, as also stated by the authors is only the addition of grouping and time-stamp. For this reason, the AC recommends the rejection of the paper as its key contributions don't warrant a full paper at ICLR.

Decision: Reject