Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers

## Abstract

Abstract Recent advancements in vision-language models (VLMs) have expanded their potential for real-world applications, enabling these models to perform complex reasoning on images.
In the widely used fully autoregressive transformer-based models like LLaVA, projected visual tokens are prepended to textual tokens. Oftentimes, visual tokens are significantly more than prompt tokens, resulting in increased computational overhead during both training and inference. In this paper, we propose Vi sual C ompact To ken R egisters ( Victor ), a method that reduces the number of visual tokens by summarizing them into a smaller set of register tokens. Victor adds a few learnable register tokens after the visual tokens and summarizes the visual information into these registers using the first few layers in the language tower of VLMs. After these few layers, all visual tokens are discarded, significantly improving computational efficiency for both training and inference. Notably, our method is easy to implement and requires a small number of new trainable parameters with minimal impact on model performance.
In our experiment, with merely 8 8 8 8 visual registers—about 1 % percent 1 1\% 1 % of the original tokens— Victor shows less than a 4 % percent 4 4\% 4 % accuracy drop while reducing the total training time by 43 % percent 43 43\% 43 % and boosting the inference throughput by 3.3 × 3.3\times 3.3 × .

## 1 Introduction

Vision-language models (VLMs) have attracted considerable attention for their capability to process visual and textual information, enabling various real-world applications, such as image captioning, visual question answering, and multimodal reasoning (OpenAI, ; Liu et al., ). For example, GPT-4V (OpenAI, ) demonstrates the potential of these models in helping visually impaired individuals to “see” the world through cell phone cameras.

Recent transformer-based vision-language models, such as LLaVA (Liu et al., ), employ a pre-trained vision encoder as the model’s “eye” to extract visual features and use a pre-trained language model as the “brain” to perform reasoning and text generation. This simple architecture is highly effective and requires only a small instruction-based fine-tuning dataset for achieving state-of-the-art results on standard benchmarks. The visual tower in VLMs decomposes high-resolution images into a large number of visual tokens, which are then concatenated with prompt tokens as an input to the language tower. This process significantly increases the computational cost due to the quadratic attention cost with respect to tokens. As an example, LLaVA-NeXT (Liu et al., ) uses $2,880$ tokens to represent a single image, which can be overly redundant in many scenarios. In contrast, the average text instruction length across all benchmarks used in LLaVA-NeXT has fewer than $70$ tokens, as shown in Section A.1. Therefore, to improve the efficiency of VLMs, reducing the number of visual tokens is essential.

The recent state-of-the-art method, FastV (Chen et al., ), achieves this by dropping unimportant visual tokens. This approach is highly effective when reducing the number of tokens by up to half. However, the model’s performance drops significantly when more than half of the tokens are removed. Moreover, FastV requires the retrieval of attention scores from the self-attention block. Efficient attention implementations, such as FlashAttention (Dao et al., ; Dao, ), do not support this feature because they avoid storing the entire attention score matrix to optimize memory usage and computational efficiency. Consequently, FastV relies on standard attention implementations to retrieve the attention scores, which limits its deployment scenarios. We also find that this constraint slows down the fine-tuning stage (Section 5.3).

Alternatively, another popular approach for token reduction involves using a transformer-based projection model to condense visual tokens into a smaller set of queries. Notable examples include Perceiver Resampler (Jaegle et al., ; Alayrac et al., ; Bai et al., ) and Q-Former (Li et al., ). These methods outperform FastV when the reduced set of visual tokens is significantly smaller than the original. However, they require many more trainable parameters.

Figure: Figure 1: Efficiency-Performance Trade-Off Curve. We compare our proposed method, Victor, with the state-of-the-art method FastV. We report the normalized average score across $12$ benchmarks and the corresponding throughput increase relative to the original baseline model (details in Section 4.2). The size of the circles represents the number of visual tokens for each method, with larger circles representing more tokens. Victor establishes a more favorable Pareto frontier than FastV, demonstrating a significantly smaller performance drop as throughput increases.
Refer to caption: x1.png

In this paper, we introduce Visual Compact Token Registers (Victor), a simple yet effective early visual token summarization method that provides a better efficiency-quality trade-off compared to the state-of-the-art methods. For visual tokens, we observe that they often contain redundant information, with many tokens being highly similar, as discussed in Section 3.1. To address this, our approach leverages the LLM to summarize these visual tokens into compact registers. Victor begins by appending a small set of register tokens to the visual tokens and uses the initial $k$ layers of LLM to summarize visual information into these registers. Notably, during training, no additional loss function or operation is required to explicitly force summarizing the visual information into the registers. However, empirical results show that the language model naturally uses these tokens to store visual information. As a result, starting at layer $k$, all visual tokens are discarded, leaving only the summarized registers and textual tokens for efficient inference in the subsequent layers.

Our method does not rely on attention scores and can be seamlessly integrated with efficient attention implementations. Moreover, unlike approaches like Perceiver Resampler which deploy a separate transformer, Victor leverages the powerful language model itself to perform this task, introducing only $1.78$M additional parameters, accounting for only $0.03\%$ of the total model size, and achieves a significantly better performance.

As illustrated in Figure 1, Victor establishes a more favorable Pareto frontier than the state-of-the-art method, particularly exhibiting a noticeably smaller performance drop as throughput increases. For instance, when the number of visual registers is set to $8$—approximately only $1\%$ of the original visual tokens—our model experiences a performance drop of less than $4\%$ while reducing total training time by $43\%$ and achieving a $3.36\times$ increase in inference throughput. Our extensive experiments demonstrate the effectiveness of Victor in balancing both efficiency and performance, making it a promising solution for reducing visual tokens in vision-language models.

## 2 Related Work

### 2.1 Vision-Language Models

Modern vision-language models typically combine a pre-trained image encoder with a large language model to handle multimodal data (Li et al., ). One popular approach, often referred to as the autoregressive or LLaVA-style model (Li et al., ; Liu et al., ), directly projects visual features into the input embedding space of the language model, treating these features as part of the input tokens. However, in this design, the number of visual tokens is large and often exceeds the number of textual tokens, leading to inefficiencies. Another common approach is cross-attention-based fusion (Alayrac et al., ), where added cross-attention blocks inside of the language transformer layers allow textual tokens to attend to visual tokens. More recently, early-fusion models like Fuyu (Bavishi et al., ), MoMA (Lin et al., ), and Chameleon (Team, ) use a unified transformer that processes raw textual tokens and visual patches simultaneously. Additionally, a key component of modern vision-language models recipe is instruction fine-tuning (Dai et al., ; Zhu et al., ; Liu et al., ; ; Singla et al., ), which enables the model to function as a typical chatbot while also processing images, even with a small synthetic fine-tuning dataset. In this paper, we focus on LLaVA-style models.

### 2.2 Visual Token Reduction

To improve the efficiency of vision or vision-language models, pruning or distilling visual tokens has been widely studied. Rao et al. () introduce DynamicViT, which uses a small module to predict the importance of each visual token, dropping less important ones to enhance efficiency. Similarly, EViT (Liang et al., ) retains important tokens and fuses the less important ones within the model, using attention scores from the class token to the visual tokens. Further, PuMer (Cao et al., ) reduces the number of both textual and visual tokens by progressively pruning and merging them throughout the cross-modal encoder. Another interesting approach by Saifullah et al. () involves discretizing visual features into textual tokens to reduce dimensionality. For more recent vision-language models, Perceiver Resampler (Jaegle et al., ; Alayrac et al., ; Bai et al., ) and Q-Former (Li et al., ) are commonly used to pool visual tokens into a smaller set of queries using a transformer-based model. Additionally, in the FastV paper (Chen et al., ), the authors observe that in LLaVA-style models, the attention from textual tokens to visual tokens significantly diminishes after the first few layers, with the attentiveness declining close to zero after $10\%$ of the transformer layers. Intuitively, their proposed state-of-the-art method drops the unimportant visual tokens accordingly after the first few layers.

### 2.3 Visual Registers

Burtsev et al. () first introduce memory tokens, which function similarly to registers. These tokens are used to store global information, enabling the model to effectively handle long-context tasks. Darcet et al. () apply the idea of registers to ViTs. In their work, the authors observe that vision transformer models implicitly use low-information tokens to store global information for internal computations. However, this leads to abnormally high norms for these tokens, making it difficult to interpret the attention maps. Therefore, to address this, they introduce additional register tokens at the end of the sequence to handle this task. This approach not only improves the interpretability of attention maps but also boosts model performance. In our work, we show that these register tokens also enhance the performance of vision-language models, as demonstrated in Section 5.6. However, our primary focus in this paper is on using these registers for information distillation, enabling the model to condense visual information into the registers to improve efficiency.

## 3 Method

Figure: Figure 2: Method Overview. Victor is a simple yet effective method for enhancing the efficiency of vision-language models. The process involves four key steps based on the LLaVA-style model: (I) appending learned visual register tokens after the visual tokens, where the number of visual registers is much smaller than the number of the visual tokens, (II) using the first $k$ layers of the language tower to summarize visual information into the visual registers, (III) discarding all visual tokens before layer $k$, and (IV) starting from layer $k$, the model performs efficient inference using only the visual registers and textual tokens with significantly reduced sequence length.
Refer to caption: x2.png

### 3.1 Motivation

We start with a key observation: many visual tokens exhibit significant redundancy. To demonstrate this, we calculated the cosine similarities between visual tokens generated by vision towers in VLMs. As shown in Figure 3, these similarities tend to cluster around $1$, indicating a high degree of similarity among the visual tokens. This suggests that compressing the visual tokens into a smaller set would result in a minimal information loss. To achieve this, we append a set of learnable register tokens to the visual tokens and leverage the language model to summarize the visual information into these registers. Rather than using a separate model, such as the Perceiver Resampler, we utilize the more powerful language model for this task, as it inherently understands which visual tokens are important and how to organize the information. Consequently, as demonstrated in Figure 3, our compact visual registers show reduced redundancy compared to the baseline visual tokens.

Figure: Figure 3: Token Similarities.
Refer to caption: x3.png

Furthermore, based on observations from FastV (Chen et al., ), textual tokens in the language model primarily attend to visual tokens in the early transformer layers, with attention scores to visual tokens dropping to nearly zero after these layers. This suggests the language model requires only a few layers to process the visual information. Therefore, instead of using the entire model, we employ only the first few transformer layers to summarize the visual tokens into the register tokens. After summarization, the visual tokens are discarded, improving model efficiency. An overview of our method is provided in Figure 2(^1^11Image credit: [https://unsplash.com/photos/green-grass-field-near-lake-under-blue-sky-during-daytime-d3Jf3avtXSg](https://unsplash.com/photos/green-grass-field-near-lake-under-blue-sky-during-daytime-d3Jf3avtXSg)).

### 3.2 Victor

We now formally introduce our method: Victor (Visual Compact Token Registers). A LLaVA-style vision-language model consists of three main components: (1) the image tower $\mathcal{I}$, which is a pre-trained vision model, such as the CLIP image encoder (Radford et al., ); (2) the language tower $\mathcal{T}$, a pre-trained LLM, such as LLaMA (Touvron et al., ); and (3) a projector $\mathcal{P}$ that bridges the two, mapping the image features into the input embedding space of the language model. Given an image $x_{\text{img}}$, we first extract its features using the image tower $\mathcal{I}$ and produce a set of projected visual tokens $x_{V}=\{x_{V}^{0},x_{V}^{1},\ldots,x_{V}^{N-1}\}$ from the projector $\mathcal{P}$.

Figure: Algorithm 1 Victor - Visual Compact Token Registers

As described in Algorithm 1, for Victor, we additionally introduce a set of learnable visual registers $x_{R}=\{x_{R}^{0},x_{R}^{1},\ldots,x_{R}^{M-1}\}$, where $M$ is a hyperparameter controlling the number of visual registers. A smaller $M$ results in a more efficient model, and usually $M\ll N$. We then concatenate the projected visual tokens, visual registers, and textual tokens to form the input: $x=[x_{V};x_{R};x_{T}]$. This input is processed through the language tower for the first $k$ layers. At the start of layer $k$, all visual tokens are discarded, and the model continues with the truncated hidden states for the remaining layers.

During training, we do not explicitly force the language model to summarize the visual information into the visual registers, but we empirically observe that it does so implicitly. In Section 5.7, we provide an empirical analysis showing that the visual registers effectively summarize important information from the visual tokens. Moreover, because Victor leverages the language model itself for this summarization, rather than relying on an external model, and the language model is both powerful and knowing at identifying the most useful image features, our method experiences minimal performance drop compared to approaches like Perceiver Resampler while requiring much fewer additional model parameters.

In practice, we typically set $k$ to $3$, which is approximately $10\%$ of the language tower. This means the language tower processes the full-length hidden states for only the first $10\%$ of its layers. For the remaining $90\%$ layers, it operates on a significantly shorter context, thereby improving model efficiency. FastV (Chen et al., ), a state-of-the-art method, follows a similar idea and drops unimportant tokens in the early layers and achieves a comparable theoretical FLOPs reduction. However, we find that since FastV relies on attention scores to determine which tokens to drop, it cannot utilize efficient attention implementations like FlashAttention (Dao et al., ; Dao, ) or PyTorch High-Performance Scaled Dot Product Attention (SDPA). Consequently, FastV delivers less improvement in throughput than Victor when applying the same token-drop ratio in practice, and also empirically experiences a greater performance degradation in high token-drop ratio regimes.

## 4 Experiments

### 4.1 Experimental Setup

In this paper, we primarily follow the setting of the open-sourced LLaVA-v1.5 (Liu et al., ). The training consists of two main stages: pre-training and instruction fine-tuning.

Pre-trained Models. For the image tower, we use the pre-trained OpenAI CLIP ViT-Large model (Radford et al., ), and for the text tower, we use the Vicuna-7B-v1.5 model (Zheng et al., ), an instruction fine-tuned version of LLaMA-2-7B (Touvron et al., ). Additionally, we present results using different language towers in Section 5.4, including Vicuna-13B-v1.5 (Zheng et al., ), Meta-Llama-3-8B-Instruct (Dubey et al., ), Mistral-7B-Instruct-v0.2 (Jiang et al., ), and Qwen2-7B-Instruct (Yang et al., ).

Datasets. For pre-training, we use the LLaVA CC3M pre-training dataset (Liu et al., )(^2^22[https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K](https://huggingface.co/datasets/liuhaotian/LLaVA-CC3M-Pretrain-595K)), a subset of $595$K images from the CC3M dataset (Sharma et al., ). For instruction fine-tuning, we use LLaVA-v1.5-mix$665$K(^3^33[https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json)), a mixed dataset comprising COCO 2017 (Lin et al., ), GQA (Hudson & Manning, ), OCR-VQA (Mishra et al., ), TextVQA (Singh et al., ), and VisualGenome (Krishna et al., ).

Implementation.
We follow the hyperparameters used by Liu et al. (). During pre-training, we freeze both the image and text towers, training only the projector and registers. We use the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $0.0001$ and no weight decay for one epoch. In the fine-tuning stage, we unfreeze the text tower while keeping the image tower frozen. As in pre-training, we use the AdamW optimizer with a learning rate of $0.00002$ and no weight decay for one epoch.

### 4.2 Evaluation

We use LMMs-Eval (Li et al., ) and run evaluations over the $11$ tasks reported in the LLaVA-v1.5 (Liu et al., ) (i.e. VQAv2 (Goyal et al., ), GQA (Hudson & Manning, ), ScienceQA (Lu et al., ), TextVQA (Singh et al., ), VizWiz-VQA (Gurari et al., ), POPE (Li et al., ), MME (Yin et al., ), MMBench (Liu et al., ), SEED-Bench (Li et al., ), LLaVA-Bench-in-the-Wild (Liu et al., ), MM-Vet (Yu et al., )), supplemented by the MMMU task (Yue et al., ). These benchmarks provide a comprehensive assessment of models’ multi-modal reasoning capabilities, encompassing academic-task-oriented and instruction-following tasks. For simplicity, we primarily report the average of normalized benchmark scores. Specifically, for MME, the score is divided by $2,000$, which represents the full score, as its metric is calculated by summing the accuracies of individual subtasks.

We evaluate efficiency by measuring the increase in throughput with the KV-cache enabled (Pope et al., ). After gathering statistics from all $12$ benchmarks, presented in Section A.1, we evaluate two settings: 1) $2$-token generation and 2) $128$-token generation. The $2$-token generation simulates scenarios where questions expect a single word, as in GQA (Hudson & Manning, ) and TextVQA (Singh et al., ). In contrast, the $128$-token generation represents open-ended question scenarios, such as in LLaVA-Bench-in-the-Wild (Liu et al., ) and MM-Vet (Yu et al., ). In both settings, we use a text prompt length of $64$ and a batch size of $16$. We choose a batch size of $16$ because it is the largest batch size that fits in memory. All training is conducted on $8$ NVIDIA A100 GPUs, with efficiency profiling performed on a single NVIDIA A100. By default, we set $k$ to $3$ and vary the number of final visual tokens to $256$, $128$, $64$, $32$, $16$, and $8$ for all methods to thoroughly assess the trade-off between efficiency and performance, where the number of visual tokens for the original LLaVA-v1.5 model is $576$.

We compare our method against two baselines: FastV (Chen et al., ) and Perceiver Resampler (Jaegle et al., ). FastV is the state-of-the-art token reduction method that filters out less important vision tokens based on attention scores, while the Perceiver Resampler is a compact, transformer-based model designed to condense input tokens into a smaller query set.

## 5 Results

### 5.1 Throughput Increase

Figure: (a) $2$-Token Generation
Refer to caption: x4.png

We present the efficiency and performance trade-offs for both generation settings in Figure 4, and the performance on individual benchmarks is provided in Section A.2. As shown, our method has a better Pareto frontier than FastV and the Perceiver Resampler in both scenarios.

Both Victor and FastV maintain minimal performance degradation when the throughput increases by approximately $1.5\times$ to $2\times$ and the number of tokens decreases from $576$ to $256$ or $128$. However, FastV’s performance declines rapidly beyond this point. In contrast, our method exhibits only a $4\%$ performance drop even when the number of visual tokens is reduced to $8$, which is roughly $1\%$ of the original visual token count. Additionally, with the same number of final visual tokens, our method has slightly higher theoretical FLOPs than FastV due to the inclusion of extra register tokens in the initial layers. However, in practice, our method achieves a greater increase in throughput compared to FastV. This is due to the fact that, in the layer where FastV performs filtering, the model is constrained to using the original attention mechanism to compute attention scores, as it cannot leverage more efficient attention implementations. In contrast, our method is compatible with a wide range of efficient attention implementations including those that do not support returning attention scores. As a result, Victor not only achieves better throughput and more effectively retains the accuracy but is also more adaptable across different devices than FastV.

**Table 1: Number of Extra Parameters for Different Methods. The final number of visual tokens is $256$.**
| Method | # of Extra Parameters |
| --- | --- |
| FastV | 0 0 ($0.00\%$) |
| Perceiver | $252.86$M ($3.61\%$) |
| Ours | $1.78$M ($0.03\%$) |

In contrast, the Perceiver Resampler experiences a substantial performance drop of approximately $10\%$ relative to the original model, and performs significantly worse than Victor. Interestingly, its performance remains stable across different reduction ratios, consistent with the findings of Laurençon et al. (). Despite this performance decline, the Perceiver Resampler achieves a much higher throughput increase than FastV and Victor. However, as shown in Table 1, the Perceiver Resampler requires a substantially larger number of additional parameters—$252.86$M, representing $3.61\%$ of the total model—while our method adds only $1.78$M, approximately $0.03\%$ of the entire model.

Figure: Figure 5: Performance vs. FLOPs Reduction.
Refer to caption: x6.png

### 5.2 FLOPs Reduction

We also report the theoretical FLOPs reduction of the methods, calculated using the FLOPs formula from Chen et al. (). As demonstrated in Figure 5, while our method shows a slightly smaller FLOPs reduction due to the presence of additional register tokens at the start of the language tower, the overall reduction is comparable under the significantly higher token reduction rate. Although the Perceiver Resampler achieves a notable increase in throughput, its FLOPs reduction is substantially lower than that of FastV and Victor, primarily due to the additional transformer layers it employs.

### 5.3 Training-Time Reduction

Figure: (a) Pre-Training
Refer to caption: x7.png

Victor not only reduces inference costs but also lowers training costs. As indicated in Figure 6, both Perceiver Resampler and Victor significantly reduce training time in both pre-training and fine-tuning stages, with the reduction being especially notable during pre-training due to the shorter text tokens. Victor achieves a greater overall time reduction. In contrast, training with FastV only reduces pre-training time and does not improve fine-tuning efficiency. This is because fine-tuning typically involves a large number of text tokens (often exceeding a thousand), and the use of a naive attention implementation in this phase introduces significant overhead, reducing training efficiency. Additionally, we observe that training with FastV does not match the performance of inference-time FastV. However, it exhibits slower benchmark performance decay as the number of visual tokens decreases and outperforms inference-time FastV when the number of visual tokens drops below $32$.

Figure: (a) Vicuna-13B-v1.5
Refer to caption: x9.png

### 5.4 Different Language Towers

We extensively evaluate the effectiveness of our method with different language towers. As shown in Figure 7, replacing the original Vicuna-7B-v1.5 language model with Vicuna-13B-v1.5 (Zheng et al., ), Meta-Llama-3-8B-Instruct (Dubey et al., ), and Mistral-7B-Instruct-v0.2 (Jiang et al., ), Victor remains highly effective and significantly outperforms the two baseline methods. For both Meta-Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2, Victor demonstrates minimal performance drop and a slow decay in performance as the number of visual tokens decreases. Notably, for these two models, when the number of visual tokens is reduced by half, the method shows no performance degradation at all.

We further demonstrate the performance of our method on a different vision-language model design: LLaVA-NeXT (LLaVA-v1.6) (Liu et al., ). LLaVA-NeXT follows a similar architecture to LLaVA-v1.5 but increases the number of visual tokens from 576 to 2,880 by incorporating different aspect ratios, enhancing the model’s capabilities. Additionally, LLaVA-NeXT utilizes Qwen2-7B-Instruct (Yang et al., ) as its language tower, benefiting from its extended context length. In our experiments, we reduce the number of visual tokens to $512$, $256$, $128$, $64$, $32$, and $16$. As indicated in Figure 7(d), our method remains highly effective in the LLaVA-NeXT setting, consistently outperforming both FastV and the Perceiver Resampler.

### 5.5 Different Layers to Drop Visual Tokens

Figure: Figure 8: Ablation on Token Dropping Layers.
Refer to caption: x13.png

We show the results of the ablation study on which layer to drop the visual tokens (hyperparameter $k$) in Figure 8. In terms of throughput improvement, it is clear that the earlier we drop the visual tokens, the more efficient the model becomes. For lower-layer numbers, such as $k=1$ or $k=2$, the model’s efficiency significantly increases, with throughput reaching nearly a $4\times$ improvement. However, this comes with a substantial performance drop, suggesting that one or two layers are likely insufficient for the summarization process. In contrast, when $k\geq 3$, the performance degradation is minimal, staying within a $5\%$ performance score loss. Notably, when $k=5$, with half of the visual tokens dropped, the model experiences no performance loss.

### 5.6 Effect of Visual Registers on Regular VLMs

Figure 9 presents the results of not dropping the visual tokens and instead using visual registers as a means for the model to store useful information, similar to those proposed by Darcet et al. ().

Figure: Figure 9: Results without Dropping Visual Tokens. From left to right, we add $8$, $16$, $32$, $64$, $128$, and $256$ visual tokens resp.
Refer to caption: x14.png

As reflected in Figure 9, there is a slight performance improvement over the baseline, but it is limited to around a $2\%$ increase. However, once the number of visual registers exceeds $64$, a further increase does not result in additional performance gains. On the other hand, adding more visual tokens leads to a decrease in throughput. Interestingly, adding just $8$ visual tokens offers a minimal throughput reduction while still providing a $1\%$ performance boost, making it almost a “free lunch” for visual-language models.

### 5.7 Analysis

In Section 3.1, we empirically demonstrate that the visual registers are more compact than the original visual tokens. In this subsection, we perform a simple analysis to examine whether and how visual registers summarize visual information. The attention map from visual registers to visual tokens is shown in Figure 10. Although the model is not explicitly trained to summarize visual information into the visual registers, they implicitly encode the visual tokens, as indicated by the significant attention scores between visual registers and visual tokens. Interestingly, the visual registers exhibit low attention to visual tokens in the first two layers, and the summarization primarily occurs in the third layer, just before the visual tokens are removed. This may be because the first two layers focus on processing the visual tokens or aligning the visual tokens and registers into a shared space to facilitate communication in later layers. This observation aligns with the ablation results discussed in Section 5.5, where dropping visual tokens in the first or second layer causes a significant performance drop. This suggests that it is more effective for the summarization process to occur in the later layers.

Figure: Figure 10: Attention Map from Visual Registers to Visual Tokens. We prompt the model with a test image from the COCO dataset and the instruction, “Describe the image.”
Refer to caption: x15.png

As shown on the right side of Figure 10, when examining the attention mapped back to the original image, the visual registers primarily focus on key elements like the rock in the water and the boat mast, while also capturing broader regions of the image. Overall, even without supervision, Victor implicitly learns to summarize the image information both effectively and efficiently.

## 6 Limitation and Future Work

While Victor is simple and effective, we identified some limitations and directions for future improvements. Currently, Victor is not a training-free method, and it must be incorporated at the training stage of the vision-language modeling. Developing a version of Victor that could be applied post-training would be a valuable advancement. However, this might be challenging, as the language tower may need to be specifically trained to learn to effectively utilize the visual registers. Another limitation is the inflexibility of the number of visual registers. As discussed in Section A.5, the performance degrades when the number of visual tokens is changed on the fly without retraining. In future work, we believe incorporating certain auxiliary loss functions could help make Victor more adaptable and flexible. Additionally, while this paper focuses on applying Victor to vision-language models, we believe this technique could also benefit language models, particularly in long-context tasks. We leave these for future exploration.

## 7 Conclusion

In this paper, we introduce Victor, a novel visual token summarization method that significantly enhances the efficiency-performance trade-off in vision-language models. Without explicit enforcement, the language tower utilizes register tokens to summarize visual information within the first $10\%$ of the layers. After summarization, Victor eliminates the need for visual tokens in the following layers. Our approach offers a superior balance in efficiency compared to state-of-the-art methods. Moreover, with just up to $0.03\%$ additional parameters, Victor is compatible with various attention mechanisms, providing a user-friendly and efficient solution across different hardware environments for future applications.

## Acknowledgements

We would like to thank Rasoul Shafipour, Seyed-mohsen Moosavi-dezfooli, and Max Horton for their valuable feedbacks and discussions.

## Appendix A Appendix

### A.1 Length Statistics for Individual Benchmarks

We show the length statistics of benchmarks in Table 2. Based on the representative lengths of these benchmarks, there are two main categories: 1) short-generation, represented by the $2$-token generation scenario in our experiments, and 2) long-generation, represented by the $128$-token generation scenario.

Table: Table 2: Prompt and Generation Length Stats of Individual Benchmarks.

### A.2 Performance on Individual Benchmarks

We show the performance on individual benchmarks of Section 5.1 in Figure 11.

Figure: (a) VQAv2
Refer to caption: x16.png

### A.3 Total Training-Time Reduction

The total training-time reduction is shown in Figure 12.

Figure: Figure 12: Performance vs. Total Training-Time Reduction..
Refer to caption: x28.png

### A.4 Extra Results with Different Language Towers

The extra result of Section 5.4 with $128$-token generation scenario is presented in Figure 13.

Figure: (a) Vicuna-13B-v1.5
Refer to caption: x29.png

### A.5 Ablation on Adjusting the Number of Visual Registers at Inference

Figure: Figure 14: Ablation on Adjusting the Number of Visual Registers at Inference.
Refer to caption: x33.png

In our main experiments, we retrain the model whenever a different number of visual registers is required. In this subsection, we explore two strategies for dynamically adjusting the number of visual registers at inference time. Given a Victor model with $M$ visual registers, if we want to use $M^{\prime}<M$ registers, we either select the first $M^{\prime}$ registers (referred to as “head”) or the last $M^{\prime}$ registers (referred to as “tail”). As shown in Figure 14, the performance of these adjustments is not as effective as retraining the model from scratch. However, we believe adding certain auxiliary losses during training can make our method more flexible, and we leave this for future work.

### A.6 Importance of Visual Registers for Summarization

Figure: Figure 15: Importance of Visual Registers for Summarization.
Refer to caption: x34.png

In this subsection, we conduct an ablation study to demonstrate the necessity of using visual registers for summarization. Specifically, we compare our approach to an alternative method where instead of prepending additional tokens to the visual tokens, we retain the last $M$ visual tokens at layer $3$. This requires the model to summarize all visual information into these last existing $M$ visual tokens. As shown in Figure 15, while the ablated method results in a slight improvement in throughput, overall the performance drops significantly. This highlights the importance of incorporating visual registers for effective summarization.

### A.7 Different Visual Registers

Figure: Figure 16: Ablation with Different Visual Registers.
Refer to caption: x35.png

We also experiment with various types of visual registers. In addition to using learnable tokens, we test three alternative methods for visual registers: 1) Pooled Image Feature: utilizing average-pooled visual tokens as the register tokens, 2) Zeros: initializing with all zeros, and 3) “Image” Token: using the embedding of the word “Image.” The results are presented in Figure 16. The “Image” Token method is effective for the visual registers, especially when the number of visual tokens is reduced to $256$ and $128$, as there is no performance drop. However, all alternative methods showed relatively worse performance compared to learnable queries in the low visual token regime. Therefore, we adopt learnable queries for Victor as they offer better overall performance.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See the weaknesses.
soundness: 2
strengths: 1. The paper is clearly written and easy to read.
2. The proposed Victor is compatible with high-efficiency attention implementations, such as FlashAttention.
summary: This paper introduces Victor (Visual Compact Token Registers), a method for reducing the number of visual tokens in vision-language models to improve efficiency without significant performance loss. By summarizing visual tokens into compact registers during the initial layers of a language model, Victor reduces computational overhead, achieving substantial reductions in training time and improvements in inference throughput. Victor’s approach is compatible with efficient attention mechanisms and minimizes parameter increase.
weaknesses: 1. The approach to visual token summarization feels overly simplistic. Prior studies suggest that fewer visual tokens are often sufficient, and Victor’s method lacks a more sophisticated or in-depth design for how register tokens are used to retain critical information. The current version seems to only provide a simple and straightforward baseline, while an exploration of more refined register token strategies could potentially minimize performance drops further or increase acceleration gains, maybe reducing >43% training time. For example, an advanced token-drop strategies, the relationship of register token drop ratios with increasing layer depth should be explored further.
2. Some comparisons with visual token compression models are missed. Victor’s efficiency claims would be more compelling if compared against models that perform early token compression before inputting to the LLM, such as LLaMA-VID. LLaMA-VID uses only two visual tokens to achieve core functionality, this paradigm seems to be also or even more efficient in theory.
3. While Victor is evaluated using different LLM architectures, it relies on only the LLaVA architecture for vision-language fusion. Comparisons with alternative VLM architectures, such as Mini-Gemini, Cambrian-1, or InternVL, would have offered a more robust evaluation.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: **Extend to Other VLM Architectures:**  
Can Victor be extended to other types of vision-language models, such as those using cross-attention mechanisms or early-fusion architectures like MoMA or Chameleon? Clarification on how adaptable the method is across different architectures would provide more insight into it.
soundness: 4
strengths: 1. **Efficiency Gains**: The method significantly reduces both training time and inference computational requirements without heavily impacting performance. The reported 43% training time reduction and 3.36× throughput boost are compelling.
2. **Implementation Simplicity**: Victor introduces only 1.78M additional parameters, representing 0.03% of the total model size, which is minimal compared to alternatives.
3. **Compatibility with Attention Mechanisms**: Unlike FastV, Victor is compatible with efficient attention implementations such as FlashAttention. This further improve its compatibility and throughput during implementation.
4. **Empirical Validation**: The paper provides extensive experimental evidence across 12 benchmarks, comparing Victor's performance to that of FastV and the Perceiver Resampler. It also provide insightful analysis of how visual registers summarize visual information, the results are reasonable and also align with the experiment discovery.
5. **Flexibility Across Models**: Demonstrated effectiveness with different language towers and vision-language model configurations, showcasing the method’s adaptability.

Overall, This paper presents a solid, well-validated approach to reducing computational overhead in vision-language models with register tokens. The idea is novel and solid, experiment results well support the design choices and also provide valuable insights.
summary: This paper introduces a novel method named Visual Compact Token Registers (Victor) aimed at improving the computational efficiency of vision-language models (VLMs). Based on the finding of [1], Victor minimizes computational overhead by summarizing extensive visual tokens into a smaller set of register tokens that distill the global information from input visual tokens. This process occurs within the initial k layers of the language tower, after which the original visual tokens are discarded to increase the overall throughput of the VLM. Victor is shown to reduce training time by 43% and enhance inference throughput by 3.36×. It only introduce extra 1.78M parameters and even when there is only 8 visual register tokens, it can still maintaining less than a 4% drop in performance.



[1] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2024). Vision Transformers Need Registers. ICLR.
weaknesses: This paper’s weaknesses are mainly discussed in Sec. 6

1. **Dependence on Training Integration**: The method is not training-free, which may limit its use for pre-trained models that cannot be modified or retrained.
2. **Limited Adaptability of Visual Registers**: The performance drops when the number of visual registers is adjusted post-training. The paper acknowledges this and suggests potential auxiliary loss functions as future work but does not address it in the current method.

Despite these weaknesses, the paper's contributions remain impactful, offering a compelling method that balances efficiency and performance effectively.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. Could you provide a more detailed explanation for the poor training performance observed with FastV in Figure 6? 
2. Additionally, according to Figure 16, the learnable registers appear to show significant differences from the image features only when the number of visual tokens is very low (≤ 16). The necessity and advantages of the additional register design need to be demonstrated through more ablation experiments.
3. More broadly, just out of curiosity, can this method be generalized to multiple images?
soundness: 3
strengths: 1. The proposed method is simple and effective.
2. The authors conduct experiments on various LLMs, aiming to demonstrate the generalizability of their method across different models.
summary: Motivated by observations from FastV, the authors utilize only the initial transformer layers to summarize the visual tokens into register tokens. Following this summarization, the original visual tokens are discarded, enabling more efficient processing while minimizing information loss. This method can be applied during both the training and inference stages and demonstrates a relatively small performance drop when utilizing fewer visual tokens.
weaknesses: 1. Since the advantages of additional register are not adequately presented. It's hard to attribute the superiority of the experimental results to the proposed method.
2. It is advisable to add a baseline experiment that directly uses fewer visual tokens. For instance, a pooling layer could be applied directly after the image tokenizer to achieve a similar computation FLOPs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: As stated in the weakness section
soundness: 2
strengths: The paper is well-written. The proposed method Victor is simple yet effective. The authors conducted numerous experiments to clearly show the superior performance of Victor, showcasing its ability to reduce computational costs without significantly compromising model accuracy. The experimental results are presented in a clear and compelling manner.
summary: The paper introduces Visual Compact Token Registers, a novel method aimed at improving the efficiency of vision-language models by reducing the number of visual tokens processed. Victor’s approach involves adding a few learnable register tokens after the initial visual tokens, then using only the first few layers of the language model to summarize the visual information into these registers. After these layers, the model discards all original visual tokens, substantially decreasing computational demands for both training and inference. This method requires minimal new parameters and speed up the training and inference.

Overall, the paper is well-written and the contributions are clear and sound.
weaknesses: 1) The paper aims at improving the efficiency of the VLMs. I think more baselines should be incorporated to demonstrate the effectiveness of the method. 
Some examples are:
[1] PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
[2] LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inference
[3] Efficient Inference of Vision Instruction-Following Models with Elastic Cache

2) Also, I think more methods should be discussed in related work section, about the development of efficient VLMs, currently still only fastV is mentioned.

3) Some questions in the experiments. 
    [1] why using 'zero' or 'image' can still maintain the performance to some extend as shown in Figure 16. I do not really get why zero can do something.
    [2] I notice that some benchmarks, such as scienceqa, even perform much better with only 8 visual tokens. Can you provide some explanations?

Due to the questions above, I think some revisions are needed, including more baseline comparisons, revision of the related work section and more explanations on experiment results. Still, I will consider raising my score if my questions are well-solved.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. In Section 5.6, the paper claims that retaining visual tokens can boost performance by 2%, please specify the benchmark and score in which this phenomenon is observed. It is quite unusual, because even in the original register paper[1], retaining visual tokens did not yield notable gains in visual comprehension tasks such as ImageNet.
2. It is worth noting that in Appendix A.2, the baseline score in the MME-P line graph appears to be only around 1,400, which does not align with the 1,510 score reported in the original LLaVA paper, could you explain it?

[1]Vision Transformers Need Registers, https://arxiv.org/abs/2309.16588
soundness: 3
strengths: 1. The paper introduces Visual Compact Token Registers (Victor), a method to enhance computational efficiency in vision-language models like LLaVA by reducing the large number of visual tokens.
2. This work continues to explore the performance of registers in VLM, which is a valuable perspective.
3. The paper is well-written and clearly presents the proposed framework.
summary: The paper introduces Visual Compact Token Registers (Victor), a method to enhance computational efficiency in vision-language models like LLaVA by reducing the large number of visual tokens. Victor replaces these tokens with a small set of “register tokens,” which summarize the visual information within the first few layers. Afterward, the original visual tokens are discarded, minimizing computational load. This approach maintains performance with minimal parameter addition, achieving only a 4% performance drop while cutting training time by 43% and increasing inference speed by 3.36×.
weaknesses: 1. This paper repeatedly emphasizes that the main limitation of FastV lies in its reliance on attention maps, which restricts it to using only the simplest attention implementation. However, this is fundamentally an engineering issue related to code implementation and does not merit extensive discussion in the paper. There are two straightforward solutions to address this: (1) set only the second layer’s attention to “eager” mode while employing flash_attention_2 for the remaining layers, or (2) store the key and value states of the second layer and recompute the attention map with (Lt, Lv) shape. Both methods can significantly reduce latency and flops. Therefore, all data related to FastV in the experiments should be reconsidered, particularly the comparison of training times in Section 5.3, where setting all FastV layers to eager mode is highly unreasonable, making the training time reported for FastV invalid.
2. When conducting research on efficiency, the primary focus should be on performance. Ideally, the efficiency strategies we aim to achieve should provide the greatest possible acceleration while maintaining comparable performance levels. However, this paper lacks any tables presenting the specific benchmark scores, relying solely on line graphs, which makes it difficult to assess the exact performance degradation. Moreover, the latter half of the line graphs has limited practical significance; for instance, when the benchmark performance drops by around 4%, the actual performance decreases far more, rendering such acceleration strategies unusable. It would be beneficial to select a few settings and display the key benchmark scores, such as for TextVQA, GQA, and SEED-Bench. 
3. When comparing inference acceleration, this paper uses FastV as a major baseline. However, FastV is a training-free method, whereas the approach  proposed by this paper requires complete training process before inference acceleration is possible. Therefore, this comparison is unreasonable from a cost perspective.
4. The reported benchmark performance of fastv after training was even lower than that of inference-only. Based on experimental experience, this result appears highly unreasonable, suggesting a review of the code for potential issues, and there is reason to suspect the baseline may have been artificially lowered.
5. Please provide a simple baseline, just pooling after the first layers to retain visual tokens. According to the paper’s methodology, retained visual tokens are intended to aggregate information from other image tokens. If using learnable tokens is feasible, a simple and straight approach would be to directly leverage part of the original image tokens to gather information.
6. In Section 4.1, it states, "We use the AdamW optimizer (Loshchilov & Hutter, 2017) with a learning rate of 0.0001 and no weight decay for one epoch." I noticed that this learning rate differs from the original LLaVA's rate of 0.001, indicating that you have adjusted the training hyperparameters. If this method were to be applied to another model, it would also require extensive hyperparameter tuning, which could incur significant computational costs.