DeCo: Decoupling Token Compression from Semantic Abstraction in Multimodal Large Language Models

## Abstract

Abstract The visual projector, which bridges the vision and language modalities and facilitates cross-modal alignment, serves as a crucial component in Multimodal Large Language Models (MLLMs).
However, measuring the effectiveness of projectors in vision-language alignment remains under-explored, which currently can only be inferred from the performance of MLLMs on downstream tasks.
Motivated by the problem, this study examines the projector module by interpreting the vision-language semantic flow within MLLMs.
Specifically, we trace back the semantic relevance flow from generated language tokens to raw visual encoder patches and the intermediate outputs produced by projectors.
Our findings reveal that compressive projectors (e.g., QFormer), abstract visual patches into a limited set of semantic concepts, such as objects or attributes, resulting in a “double abstraction” phenomenon. Therefore, in MLLMs, this involves a first visual semantic abstraction by the projector referring to pre-defined query tokens (in the vision modality), and a second extraction by the LLM based on text instructions (in the language modality).
The double abstraction is inefficient in training and will result in cumulative vision semantics deficiency.
To mitigate this issue, we propose the key insight of “ De couple Co mpression from Abstraction (DeCo) ”, that is compressing the visual token number at the patch level by projectors and allowing the LLM to handle visual semantic abstraction entirely.
Consequently, we adopt a simple compressor, i.e., 2D Adaptive Pooling, to downsample visual patches in a parameter-free manner.
Empirical evaluation demonstrates that DeCo surpasses traditional compressive projectors regarding both performance and efficiency. It achieves performance gains of 0.9%, 7.1%, and 2.9% across the MLLM Benchmarks, Visual Localization, and Open-ended VQA tasks with fewer trainable parameters and faster convergence speed.
Furthermore, DeCo preserves vision spatial locality and exhibits robustness across various MLLM configurations, including different vision backbones, image resolutions, and LLMs.
Our code will be available at https://github.com/yaolinli/DeCo .

## 1 Introduction

Multimodal Large Language Models (MLLMs) [, , ] endow Large Language Models (LLMs) with vision perception capability, which have shown their versatility and expertise in diverse vision-language tasks [, , , , , , , ].
For MLLMs, learning good vision-language alignment is at the core of their intelligence [, , , ].
To achieve cross-modal alignment, recent studies utilize an intermediate module, i.e., the projector [, , , ], to map representations of image patches [] into the LLM embedding space as visual tokens.
Widely used projectors can be roughly summarized into two branches: non-compressive and compressive. The non-compressed projector [] directly uses linear layers that translate the visual token dimension to the LLM’s while keeping the visual token number unchanged.
Despite its simplicity and effectiveness,
the linear projector struggles with high training resources and costs due to the length of the visual token sequence.
The sequence would be long in two common scenarios: (i) the length increases quadratically with the input resolution [, ]; (ii) the length increases linearly with the image number for handling video frames [, , ],
potentially resulting in sequences up to a million tokens long [].

On the other branch, prevalent compressive projectors, e.g., QFormer [, ], Resampler [], and D-Abstractor [], condense the original visual tokens into fewer query tokens to reduce visual redundancy, which have a better balance between performance and efficiency.

However, how existing projectors affect the vision-to-language semantic alignment in an explainable perspective is still under-explored. Understanding this question is crucial for facilitating better architectural improvement and providing broader practicability in demanding scenarios such as high image resolutions and video applications.
In this study, we investigate this problem by analyzing the relevance between generated textual tokens, raw visual patches and intermediate projector outputs. We start by tracing the language-to-vision semantic flow using a novel R-GAE explainability tool. Specifically, we decouple the overall Text-to-Patch semantic relevance to Text-to-Query and Query-to-Patch sub-flows during the image-to-text generation. Among the sub-flows, the Text-to-Patch relevance reveals the effective visual context from ViT [] (i.e., Patch) leveraged by the LLM (i.e., Text). Meanwhile, the Query-to-Patch relevance interprets the visual patterns learned from original image patches (i.e., Patch) by query tokens (i.e., Query).

Based on the R-GAE analysis, we derive two important findings: Firstly, the query tokens compress the number of visual tokens by abstracting semantic-level visual concepts, leading to visual semantics deficiency such as loss of fine-grained attributes and spatial locality. As Figure 1 illustrates, different query tokens are activated with varied visual concepts such as objects, attributes or backgrounds from the original images. For the top image with zebras in the grassland, query tokens attend to visual patterns such as three zebras, their body parts, surface textures, and distant backgrounds respectively.
However, the fixed number of queries can only express limited visual semantics. Specifically, different query tokens show repetitive patterns across images (highlighted by color frames in Figure 1). Moreover, they tend to lose fine-grained visual attributes (e.g., “purple and red” in Figure 3). Furthermore, the vanilla QFormer has been demonstrated to lose visual spatial locality [] during semantic abstraction.

Secondly, the LLM acts as an excellent visual-semantic abstractor directly from patch features. As Figure 3 first row shows, utilizing a non-compressive linear projector allows the LLM to perceive patch-level visual representations and attend to accurate vision regions without prior semantic deficiency.
Consequently, the QFormer-based MLLM system redundantly extracts the visual semantics twice using QFormer and LLM, which we refer to as the *Redundant Double Abstraction* phenomenon. This double abstraction introduces two major drawbacks: (i) an accumulation of vision deficiencies at the semantic level, meaning that the loss of fine-grained semantics and spatial locality in the QFormer abstraction will propagate to the LLM and (ii) increased training complexity required to optimize an efficient visual semantic abstractor. Therefore, a more efficient compressive projector is needed to simplify the training complexity and preserve more visual context.

In this study, we propose to Decouple token number Compression (DeCo) from vision semantic abstraction. The core of DeCo is using a simpler projector, which operates and outcomes visual tokens directly at the patch level to reduce the visual token number. Subsequently, the LLM independently abstracts vision semantic concepts from the reduced tokens.
In the DeCo framework, we adopt a simple Adaptive Average Pooling as a natural down-sampler at the patch level and then use the linear layers to map the vision dimension. This projector has three-fold advantages. Firstly, it can flexibly compress the visual tokens into an arbitrary indicated number by automatically calculating the pooling kernel size and stride. It is also parameter-free and thus converges faster. Besides, it preserves the vision spatial locality via the kernel-based operation and neighbor patch integration.
Experiments comparing prevailing compressed projectors
under the same settings verify the effectiveness and efficiency of our DeCo framework.
Meanwhile, DeCo shows stronger spatial understanding ability, as well as demonstrates robustness across various MLLM configurations, including different vision backbones, image resolutions, and LLMs.

In short, we make three key contributions in this work: (i) We design a novel analysis tool R-GAE to dissect the learned visual semantics in projectors of MLLMs.
(ii) Using this tool, we reveal a double abstraction phenomenon at semantic level, which leads to deficiencies in MLLM performance. To address this, we propose DeCo, a simple architecture with an adaptive pooling mechanism to decouple token compression and visual semantic abstraction. (iii) Experimental results demonstrate that our DeCo framework is simple yet effective, significantly boosting the spatial understanding capabilities of MLLMs across various benchmarks.

## 2 Related Work

#### Multimodal Large Language Models.

The development of large vision-language models has accelerated recently [, , , ]. Flamingo [, ] and IDEFICS [] have showcased the effectiveness of consolidating LLMs with vision encoders. The Q-Former from BLIP-2 [] has helped bridge the gap between the visual and text modalities. InstructBLIP [], Ying-VLM [] and MM-ICL [] further integrate instructions into the vision-text alignment process for improved in-context learning ability [].
Various approaches have been proposed to align visual encoders and LLMs effectively. MiniGPT-4 [] and LLaVA [, ] use a single projection layer, while mPLUG-Owl [] adopts LoRA tuning [, ], showing promising results. Qwen-VL-Chat [] has scaled up multi-modal pre-training with more datasets. Fuyu-8 [] proposes a new architecture by segmenting images into pixel patches, treating them as visual tokens to train a conditional multi-modal language model directly.
However, these works employ projector modules empirically or simply refer to the final performance of the MLLMs on downstream tasks without conducting an in-depth analysis of the projectors’ effectiveness.
In this paper, we examine this significant component by tracking the vision-and-language semantic flow within MLLMs. We visualize the internal patterns learned by projectors and highlight their drawbacks, offering valuable insights for future development.

#### Transformer Explainability.

Explainability tools have been widely explored for Transformers to better visualize their inner decision-making processes. Raw attention maps in Transformers usually provide interpretations for a single layer. Abnar et al. [] combine the attention scores across multiple layers and propose the rollout method. Chefer et al. [] introduce the relevance map through information propagation from all layers and components in Transformers. LRP [] captures the relative importance between different attention heads using gradients. Casual Interpretation [] can identify the most important input tokens corresponding to the model output. However, these methods are only applicable to Transformers with self-attention layers. As an alternative, the GAE [] method extends the propagated relevance maps to bi-modal scenarios with cross-attention layers. Moreover, several studies [, , , , ] focus on the multimodal system interpretation. Recently, LVLM-Interpret [] has developed an interactive application for interpreting MLLMs. Despite these efforts, in-depth explainability of existing MLLMs is rarely explored. In this study, we propose the R-GAE method derived from GAE for MLLMs to investigate how projector modules affect the vision-and-language semantic alignment of MLLMs.

## 3 Visual Projector Analysis

Figure: Figure 2: The overall analysis framework of a typical MLLM. During image-to-text generation, we trace back the language-to-vision semantic flow utilizing R-GAE relevance maps.
Refer to caption: x4.png

In this section, we analyze the impact of projector modules in Multimodal Large Language Models (MLLMs) from a semantic flow perspective using a novel R-GAE explainability tool.
During image-to-text generation, visual context plays an indispensable role in the perception of Large Language Models (LLMs). The related relevance maps between image and text, such as attention maps [], can serve as an interpretation of the vision-language semantic alignment [, , , ].
As Figure 2 shows, given an oracle description in the MLLM architecture, the backtracking relevance map from text words to visual patches (referred to as Text-to-Patch) exhibits the visual semantics aligned with the LLM and further indicates the effective visual context leveraged by the LLM.
To examine the impact of projectors as the intermediate module, we dissect the Text-to-Patch relevance map into Text-to-Query and Query-to-Patch sub-maps, as illustrated in Figure 3. The Query-to-Patch map can explain the visual patterns learned by the query (or compressed) tokens, while the difference between Text-to-Patch and Text-to-Query, exerted by the projector, reveals its impact on the vision-language semantic alignment.

A typical MLLM architecture comprises a Vision Transformer (ViT) to acquire visual representations $\mathcal{I}\in\mathbb{R}^{N\times d_{I}}$ containing $N$ patches, a projector to transform visual representations into the textual embedding space, and an LLM that handles both vision and instruction tokens to output hidden states $\mathcal{T}\in\mathbb{R}^{L\times d_{T}}$ and generate responses $Y=\{y_{1},y_{2},\dots,y_{L}\}$. We summarize widely adopted projectors into two branches:

Non-compressive Projectors maintain the number of patch tokens $N$ and only transform the visual embedding dimension to match the dimension of the LLM, as exemplified by the linear projector []. The projected visual tokens can be denoted as $\mathcal{Q}\in\mathbb{R}^{N\times d_{T}}$.

Compressive Projectors reduce the number of patch tokens $N$ to a specified lesser number $M$ ($M<N$), conserving training resources. For instance, QFormer [] learns pre-defined query tokens to compress original visual tokens. These compressed query tokens $\mathcal{Q}\in\mathbb{R}^{M\times d_{T}}$
are then fed into the LLM providing vision information.

For clear clarification, we distinguish the compression and abstraction concepts in this study. The compression refers to the reduction of vision token number in particular, whereas abstraction denotes the extraction of vision semantic concepts (e.g., objects and attributes, etc.).

Figure: Figure 3: Visualization of the R-GAE relevance maps across the same MLLM architecture except for projector modules. The linear projector is non-compressive while the QFormer and Adaptive Average Pooling (ours) compress the original 576 vision tokens to 64 tokens. Text-to-Patch relevance reveals the effective vision semantics aligned with the LLM during image-to-text generation. For QFormer in the second row, its Query-to-Patch map discards the fine-grained visual semantics about “purple and red”. This semantic deficiency is transmitted to the final Text-to-Patch map and leads to a misalignment of vision patches and textual words.
Refer to caption: x5.png

### 3.1 R-GAE: Relevance Maps in MLLMs Derived from GAE

We aim to employ the dissected Text-to-Query and Query-to-Patch relevance maps to examine the projector module. A straightforward attempt is utilizing the raw attention maps in MLLM layers as the relevance map []. However, the attention map exhibits the interaction between tokens in a single layer [].
Instead, we require a relevance map that traces back inter-token alignment in arbitrary two layers in the MLLM, for instance, the alignment from intermediate-layer query tokens to initial-layer input patch tokens. To achieve this goal, we propose a novel R-GAE relevance map derived from the Generic Attention Explainability (GAE) []. R-GAE extends the GAE method originally designed for classification tasks, to generative MLLMs, and adapts it to the typical MLLM architecture consisting of a ViT, a projector, and an LLM. The R-GAE can acquire relevance maps from any two arbitrary layers within the MLLM through propagation.

We initialize three R-GAE relevance maps including a Text-to-Patch map as $\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{I}}$, a Text-to-Query map as $\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{Q}}$, and a Query-to-Patch map as $\mathbf{R}_{\mathcal{Q}\rightarrow{}\mathcal{I}}$. Each map is an identity matrix based on the intuition that each input token’s relevance score is equal in the beginning.
Given an image and an instruction (e.g., “Please describe the image with a concise sentence”), an MLLM will generate a textual description $Y=\{y_{1},y_{2},\dots,y_{L}\}$ referring to the visual information. During the generation step $t$, we can cache the attention maps across the ViT, the projector and the LLM during a forward pass. Then, specifying a word class $\widehat{y}_{t}$ as the target prediction, we can obtain the related gradients through a backward pass. For each layer, a single R-GAE relevance map is obtained by utilizing gradients to average across the attention heads. For step $t$, we can propagate the Text-to-Query map $\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{Q}}\in\mathbb{R}^{1\times M}$ from the LLM’s first layer to its last layer to get the final map. Similarly, the Query-to-Patch map $\mathbf{R}^{t}_{\mathcal{Q}\rightarrow{}\mathcal{I}}\in\mathbb{R}^{M\times N}$ can be propagated from the first layer to the last layer of the projector. Subsequently, the overall Text-to-Patch relevance map can be obtained by matrix multiplication of Text-to-Query and Query-to-Patch maps:

$$ $\displaystyle\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{I}}$ $\displaystyle=\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{Q}}\times \mathbf{R}^{t}_{\mathcal{Q}\rightarrow{}\mathcal{I}}$ (1) $$

For a complete sentence $Y$, we integrate the R-GAE relevance maps from each time step $t$ by averaging to obtain the overall visual relevance related to a factual sentence.
We set the ground-truth description from an image-text pair as the target response to perform the backward process. This limits MLLMs with different projectors having the same Oracle Text-to-Patch visualization.
We provide the background of GAE and the specific propagation formula of R-GAE in Appendix A. Moreover, we compare the visualization between R-GAE and original attention maps in Appendix B.

### 3.2 A Redundant Double-Abstraction Phenomenon Resulting from Compressive Projectors

Based on the R-GAE maps, we analyze the different types of projectors and investigate how they affect the vision-to-language semantic alignment. For fair comparison and analysis, we train MLLMs under the same architecture, except for the projector module, and keep all other variables the same (experimental details are provided in § 5.1). We visualize the R-GAE maps of a non-compressive projector (i.e., linear layers) and a compressive projector (i.e., QFormer) in Figure 3 and draw the following findings.

Observation 1. LLMs are good visual semantic abstractors directly from patch representation.

The non-compressive projector directly inputs the patch representation to the LLM. As shown in the first row of Figure 3, given a description containing visual objects (i.e., the remote and buttons) and attributes (i.e., purple and red), the LLM can highlight the most relevant visual regions in a fine-grained manner, as it discriminates the accurate remote with purple and red buttons among other similar remotes. This indicates that the LLM has built a strong alignment between textual and visual semantics based on the patch representation. The recent success of MLLMs [, , ] with non-compressive projection further demonstrates that the LLM itself is an efficient visual semantic abstractor. For instance, LLaVA-Next [], which employs a simple Multi-layer Perceptron (MLP), achieves state-of-the-art performance across diverse multimodal benchmarks.

Observation 2. Compressive projectors extract limited visual semantic concepts from patches.
Compressive projectors like QFormer pre-extract visual semantic concepts from patches and provide reduced visual tokens at the semantic level to the LLM. As the Query-to-Patch map in Figure 3 shows, the compressed 8x8 query tokens are activated with visual semantic patterns such as different remotes, buttons, control panels, and the black background board. However, the fixed number of query tokens can only cover limited visual semantic concepts from the image. Comparing the visual patterns among 64 tokens, we find that they are visually repetitive and semantically sparse.
For instance, query tokens indexing $(0,1)$ and $(2,0)$ are nearly identical and all attend to the bottom-right panel of the right remote. These sparse query tokens lead to a deficiency in visual semantics, losing the fine-grained attribute of “purple and red buttons”. Consequently, the LLM suffers from this irreversible visual semantic deficiency when re-extracting visual context in the query semantic space. As the Text-to-Query map shows, the LLM primarily attends to the query tokens indexing $(0,2)$, $(0,4)$, and $(4,5)$ (framed in red), resulting in a misalignment of text words and patches verified in the Text-to-Patch map. More visualization cases are presented in Appendix D.

Insight. An inefficient MLLM system due to the double abstraction of visual semantics.

Based on these observations, we conclude that existing compressive projectors, which learn a fixed number of query tokens, are inefficient compressors for reducing the number of vision tokens. They result in a “Double Abstraction” MLLM system, where visual semantics are first abstracted by projectors and then re-extracted by the LLM. This dual-abstraction procedure has two main shortcomings:
(i) Accumulative visual semantics loss. The projector serves as an intermediate module bridging the ViT and LLM, therefore, the visual semantics lost during the initial abstraction by the projector become a bottleneck for the MLLM system.
(ii) Increased training complexity. Optimizing a projector to be an effective semantic abstractor is essential for alleviating semantic loss; however, this increases the training cost and complexity. For instance, Qwen-VL-7B [], which uses a resampler projector, requires 1.4B pretraining and 50M fine-tuning data across three training stages.

## 4 DeCo: Decoupling Vision Token Compression

Figure: Figure 4: Visualization of DeCo method.
Refer to caption: x6.png

Inspired by the analysis in §3, we propose a DeCo method to Decouple vision token Compression from semantic abstraction in MLLMs. In this approach, the compressive projectors focus on reducing the number of visual tokens with patch-level outcomes, while the LLM serves as the expert semantic abstractor. Consequently, the DeCo system only requires a simple projector that compresses visual tokens at the patch level. This design removes the intermediate semantic bottleneck and simplifies the training process.

We employ the 2D Adaptive Average Pooling (abbreviated as AdaptiveAvgPool) as a natural downsampler of the visual tokens at the patch level. As Figure 4 illustrates, given $N$ patch tokens from the ViT, the adaptive pooling can reduce the token number to a lesser square number $M$. Specifically, we reshape the N visual tokens to 2D tensors with size $(N^{\frac{1}{2}},N^{\frac{1}{2}})$ and utilize a 2D adaptive average pooling to get compressed tokens with size $(M^{\frac{1}{2}},M^{\frac{1}{2}})$. Subsequently, the compressed 2D tensor is flattened into $M$ tokens. These tokens are finally projected by the linear layer to match the textual embedding dimension, serving as visual inputs to the LLM. During compression, the adaptive pooling (^1^11 Apply the torch.nn.AdaptiveAvgPool2d function in the PyTorch framework.) automatically calculates the stride $S$ and kernel size $K$ in a parameter-free mode. It averages patches in a spatial $K\times K$ window into a mixed token. In essence, the 2D AdaptiveAvgPool merges the spatial neighbor patch tokens which tend to have high visual redundancy.

As illustrated in the third row of Figure 3, the Query-to-Patch mapping of the AdaptiveAvgPool projector forms a 2D grating pattern. It uniformly down-samples the grouped patches over the 2D spatial space of the original image. This uniform patch-level sampling preserves dense visual context compared to the QFormer abstractor. For instance, the compressed token indexed at $(3,3)$, highlighted in the red frame, retains the fine-grained representation of the “purple and red buttons”. Subsequently, the LLM can attend to the accurate visual region by leveraging the visual context from the AdaptiveAvgPool, as shown in the Text-to-Patch map. Furthermore, the Text-to-Patch maps of the linear projector and AdaptiveAvgPool are nearly identical. This similarity reveals that the AdaptiveAvgPool projector achieves a superior combination of (i) effectiveness, approximating the linear projector in preserving visual context, and (ii) efficiency, reducing the number of vision tokens, similar to the QFormer abstractor.

Table: Table 1: Overall performance compared to existing compressive projectors. All results are conducted under the same architecture and settings: CLIP ViT-L/14 [] as the vision backbone with $336^{2}$ image resolution and Vicuna-v1.5-7B [] as the LLM with LoRA [] tuning strategy. The training data contains 558K image-text pairs for pre-training and 665K chat instances for instruction tuning, the same as the LLaVA-1.5 []. All compressive projectors reduce the vision token number (#V) from 576 to 144. * indicates reproduced results using LoRA while ${\dagger}$ denotes the full-training results reported in LLaVA v1.5. The best and second-best results are bolded and underlined, respectively.

Figure: Table 2: Vision spatial understanding capability across different projectors. Task names are abbreviated as follows: Position (POS) for MME [], Spatial Relationship (SR), Object Localization (OL), and Physical Relation (PR) for MMBench [], and Spatial Relation (SR) and Instance Location (IL) for SEED-Bench [].
Refer to caption: extracted/5635172/images/train_loss.png

## 5 Experiments

### 5.1 Experiment Setting

#### Training data and Evaluation.

We utilize the open-sourced 558K pre-training data (sourced from LAION [], Conceptual Captions [] and SBU Captions []) and 665K instruction-following data (containing LLaVa Synthetic Data [], VQA${}^{\text{v2}}$ [], GQA [], OK-VQA [], OCR-VQA [], A-OKVQA [], TextCaps [],
RefCOCO [], Visual Genome [] and ShareGPT []) following LLaVA v1.5 [].
For evaluation, we measure model performance spanning three aspects. Multimodal LLM Benchmarks including SEED-Bench [] (report image-only set as SEED${}^{\text{I}}$), MME [] (report perception set as MME${}^{\text{P}}$) and POPE [] are specially designed for instruction-following MLLMs. Visual Localization task encompassing RefCOCO, RefCOCO+, and RefCOCOg [, ] is to measure the bounding box prediction accuracy. Open-Ended Visual Question Answering task consisting of VizWiz [], VQA${}^{\text{v2}}$ [], GQA [] and TextVQA [] aims to evaluate visual reasoning capability.

#### Implementation Details.

DeCo is primarily built on the LLaVA v1.5 framework, encompassing model architectures, training data, and training strategies. We replace the original two-layer MLP projector with QFormer [], C-Abstractor [], D-Abstractor [] and AdaptiveAvgPool respectively for fair comparison. The default configuration includes a CLIP ViT-L/14 336px [] and Vicuna v1.5 7B [] with a two-stage training strategy. The first pre-training stage updates only
the projector while the second instruction-tuning stage optimizes both the projector and the LLM using LoRA []. The main results are derived from this default configuration. Additionally, we conduct generalization experiments using a more lightweight setup that involves only the instruction tuning stage as outlined in PRISM []. Specific training hyper-parameters are detailed in Appendix C.

### 5.2 Compared with Existing Projectors

To showcase the efficiency and effectiveness of the DeCo method, we compare it with common projectors including the Linear projector [], QFormer [], C-Abstractor [], and D-Abstractor[].

Table: Table 3: Comprehensive comparison between the C-Abstractor (C-Abstr) and Adaptive Averaging Pooling (AvgPool) across various settings including different vision backbones, image resolutions and LLMs. All experiments are conducted on the one-stage instruction tuning (665K data) referring to PRISM [] to speed up training. Res. denotes image resolution. Compress. means the compression ratio of each projector from the raw visual token number to the projected vision token number.

#### Performance Effectiveness.

Table 1 presents the overall performance of different projectors. The non-compressive linear projector preserves all vision information and achieves the best overall performance. In the compressive projector category, DeCo outperforms existing solutions across most benchmarks. Specifically, DeCo achieves gain margins of SEED${}^{\text{I}}$ +2.3 and POPE +1.3 in the instruction-following MLLM benchmarks, RefCOCO/RefCOCO+/RefCOCOg +2.8/4.2/0.9 for visual localization, and VQA${}^{\text{v2}}$ +3.9, GQA +1.0, VQA${}^{\text{Text}}$ +0.3 for open-ended visual question answering. The superior results of DeCo under the same compression ratio (576->144) demonstrate that naive compression at the patch level effectively transmits visual context while reducing the token number. Among the existing projectors, the locality-enhanced C-Abstractor produces results comparable to DeCo. Additionally, we observe that QFormer performs poorly on the visual localization task, particularly in predicting visual coordinates. This poor performance is due to the loss of spatial locality during projector compression, resulting in cumulative spatial context deficiency.

#### Training Efficiency.

Besides the remarkable performance, DeCo also has efficiency advantages because it conducts parameter-free compression clarified in § 4.
Among existing compressive projectors, the sub-optimal C-Abstractor comprises 3-layer ResNet blocks [], the adaptive average pooling and another 3-layer ResNet blocks. Meanwhile, we adopt a two-layer QFormer consisting of a self-attention and a cross-attention layer initialized from the BLIP-2 [] pretraining weights.
Compared with them, the AdaptiveAvgPool in DeCo method is more lightweight and efficient. Figure 5 depicts that DeCo has faster training convergence during pre-training.

#### Spatial Locality Reservation.

Spatial understanding capability in vision modality is essential to achieve accurate visual location, fine-grained vision reasoning, object relation perception and etc. We verify the spatial understanding capability of DeCo in Table 2 across six spatial understanding tasks from MLLM benchmarks. As Honeybee [] points out, the vanilla resampler architecture like QFormer will lose the visual spatial locality, therefore, it obtains a low average score of 40.9. The locality-enhanced projector, i.e., C-Abstractor, has remarkable improvements and achieves 54.4. Overall, the DeCo with AdaptiveAvgPool well reserves the significant spatial context and achieves the closest score (56.0) to the linear projector (56.4). This benefits from the kernel and stride operation of 2D AdaptiveAvgPool similar to the convolutional network [].

Figure: Figure 6: Compression ratio reducing 576 tokens to 400/256/144/64 tokens.
Refer to caption: x7.png

### 5.3 Generalization Results

To explore the performance of DeCo under different configurations, we select varied vision backbones, image resolutions and LLMs, and report results in Table 3. To speed up training, all results are obtained through the one-stage training (i.e., instruction tuning) according to PRISM []. We select the most comparative baseline C-Abstractor (refer to Table 1) as a reference.

For vision backbones (B2, B3, and B4), we adopt the CLIP ViT-L, SigLIP ViT-SO [], and the DINOv2 []+SigLIP ensemble in embedding dimension. For scaling image resolution (B1 and B3), we compare 224px and 384px image inputs using the SigLIP ViT-SO backbone. For LLMs (B4, B5, and B6), we employ three levels of model scope, including Qwen-Chat-0.5B [], Phi-2-2.7B [], and Vicuna-v1.5 [].

The overall results in Table 3 under six different settings demonstrate the robustness of DeCo as a compressive projector across diverse MLLM architectures. It surpasses the C-Abstractor notably in almost all metrics and all settings.

### 5.4 Ablation Study

Compression Ratio Analysis.
There is a trade-off between visual information deficiency and training cost based on the compression ratio. In Figure 8, we compress the visual tokens from $24\times 24$ to $20\times 20$, $16\times 16$, $12\times 12$, and $8\times 8$ respectively, and report the average Accuracy@IoU=0.5 on the visual localization task. Results reveal that a quarter compression from $24\times 24$ to $12\times 12$ provides the best balance for AdaptiveAvgPool.

Average Pooling vs. Max Pooling.
Average pooling and max pooling are two widely-used downsampling operations. We compare these two operations in the DeCo method in Figure 8. Results show that adaptive average pooling performs better across almost all metrics, especially visual localization. The reason is that the averaging operation integrates each patch within the kernel-size window and can serve more visual context.

One-Stage vs. Multi-Stage Training.
PRISM [] indicates simple linear projectors only require one-stage instruction tuning. Inspired by this, we compare the one-stage and two-stage training results of DeCo and find that two-stage training is recommended, as shown in Figure 8.

## 6 Limitations

We present limitations in this work to facilitate future research.
Firstly, the AdaptiveAvgPool adopted in the DeCo method may cause severe visual information deficiency in an increasingly high compression ratio compared to semantic-level compression projectors. In a high-compression scenario, the averaging pooling will erase the fine-grained visual context in a kernel scope.
Secondly, the superiority of DeCo lies in a limited training resource application including limited GPUs to train a long visual token sequence and limited training data to optimize a desirable semantic QFormer-type projector. Otherwise, when have abundant training resources, the architecture of projectors tend to be insignificant in an MLLM system as pointed out in the MM1 [].

## 7 Conclusion

We introduce DeCo to decouple visual token compression from semantic abstraction. It is motivated by the “Double Abstraction” problem of existing projectors disentangling the Text-to-Patch, Text-to-Query and Query-to-Patch R-GAE maps in the vision-and-language semantic alignment. The DeCo method simplifies existing compressive projectors with a naive AdaptiveAvgPool, which downsamples spatial vision tokens directly at the spatial level. Experiments across diverse configurations demonstrate the efficiency, effectiveness, and robustness of DeCo. Eventually, the intuition of “DeCo” is not limited to the specific AdaptiveAvgPool projector design, there is great potential to improve it to perform more effectively under more demanding scenarios like high compression ratio.

## Appendix A Details of R-GAE Explainability Tool

### A.1 Background: GAE Explainability for Transformer Layers

The Generic Attention Explainability (GAE) [] is a powerful method to interpret predictions for bi-modality Transformer-based architectures. It has the advantage of acquiring the relevance map from two arbitrary layers in the Transformer through propagation.
Essentially, the GAE method generates a relevance map $\mathbf{\bar{A}}$ for each self-attention layer or cross-attention layer by integrating raw attention maps and gradients. Then it aggregating the relevance maps of all layers into a overall single map $\mathbf{R}$. Formally, denote a Transformer architecture as $\phi$, its attention map of each layer as $\mathbf{A}$, the input modality tokens as $I\in\mathbb{R}^{N\times d}$ and the output predict class as $y$. We aim to visualize the relevance map $\mathbf{R}_{y\rightarrow{}I}\in\mathbb{R}^{N}$ from class $y$ to input tokens $I$. Take the self-attention layer as an example, the relevance map $\mathbf{\bar{A}}$ for each layer and the propagation of final map $\mathbf{R}_{y\rightarrow{}I}$ are termed as:

$$ $\displaystyle\mathbf{\bar{A}}=\mathbb{E}_{h}((\nabla\mathbf{A}\odot\mathbf{A}) ^{+}),$ (2) $$

$$ $\displaystyle\mathbf{R}=\mathbf{R}+\mathbf{\bar{A}}\cdot\mathbf{R},$ (3) $$

where each layer’s attention map $\mathbf{A}$ can be obtained through a forward pass and the related gradient $\nabla\mathbf{A}:=\frac{\partial\mathcal{\phi}(y)}{\partial\mathbf{A}}$ can be cached during a backward pass.
$\odot$ is the Hadamard product, $(\cdot)^{+}$ represents the operation of setting negative values to 0, and $\mathbb{E}_{h}$ is the mean across the attention heads dimension.
The overall map $\mathbf{R}$ is initialized as the identity matrix with the intuition that each input token’s relevance score is identical in the beginning. The propagation Formula 3 updates the $\mathbf{R}$ from a start layer $L_{s}$ to an end layer $L_{e}$ ($L_{e}>L_{s}$ ) in the Transformer. The cross-attention propagation is similar, which maintains two relevance matrices for two modalities and updates them through the layer interaction. Please refer to the details of the propagation formula across cross-attention layer from the original paper.

### A.2 R-GAE Propagation for MLLMs

The traditional GAE map is designed for a classification task with the special $CLS$ token. We adapt it to MLLM architectures and propose the R-GAE explainability tool.
As Figure 2 shows, a typical MLLM architecture comprises a Vision Transformer (ViT) $\phi_{v}$ to acquire patch-level visual representations $\mathcal{I}\in\mathbb{R}^{N\times d_{I}}$
(containing $N$ patches), a projector $\phi_{p}$ to transform visual representations into the textual embedding space as $\mathcal{Q}$, and an LLM $\phi_{t}$ that handles both vision and instruction tokens to output hidden states $\mathcal{T}\in\mathbb{R}^{L\times d_{T}}$ and generate responses $Y=\{y_{1},y_{2},\dots,y_{L}\}$. We summarize widely adopted projectors into two branches:

Non-compressive Projectors maintain the number of patch tokens $N$ and only transform the visual embedding dimension to match the dimension of the LLM, as exemplified by the linear projector []. The projected visual tokens can be denoted as $\mathcal{Q}\in\mathbb{R}^{N\times d_{T}}$.

Compressive Projectors reduce the number of patch tokens $N$ to a specified lesser number $M$ ($M<N$), conserving training resources. For instance, QFormer [] learns pre-defined query tokens to compress original visual tokens. These compressed query tokens $\mathcal{Q}\in\mathbb{R}^{M\times d_{T}}$
are then fed into the LLM providing vision information.

We initialize three GAE relevance maps including a Text-to-Patch map as $\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{I}}$, a Text-to-Query map as $\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{Q}}$, and a Query-to-Patch map as $\mathbf{R}_{\mathcal{Q}\rightarrow{}\mathcal{I}}$. As Figure LABEL:supp_overall depicts, given an image and an instruction (e.g., “Please describe the image with a concise sentense”), an MLLM will generate a textual description $Y=\{y_{1},y_{2},\dots,y_{L}\}$ referring to the visual information. During the generation step $t$, we can cache the attention map $\mathbf{A}_{v}$, $\mathbf{A}_{p}$, $\mathbf{A}_{t}$ across the ViT, the projector and the LLM during a forward pass. Then specifying a word class $\widehat{y}_{t}$ as the target predict, we can get the gradients $\nabla\mathbf{A}_{t}$, $\nabla\mathbf{A}_{p}$, $\nabla\mathbf{A}_{v}$ in each module through a backward pass. The LLM module in MLLMs substantially contains self-attention layers, therefore, we can propagate the $\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{Q}}\in\mathbb{R}^{1\times M}$ according to Formula 3 from LLM’s first layer to its last layer. The QFormer-type projector consisting of self-attention and cross-attention layers can also be propagated similarly to get $\mathbf{R}^{t}_{\mathcal{Q}\rightarrow{}\mathcal{I}}\in\mathbb{R}^{M\times N}$. Subsequently, the overall text-to-patch relevance map can be obtained by matrix multiplication of text-to-query and query-to-patch maps:

$$ $\displaystyle\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{I}}$ $\displaystyle=\mathbf{R}^{t}_{\mathcal{T}\rightarrow{}\mathcal{Q}}\times \mathbf{R}^{t}_{\mathcal{Q}\rightarrow{}\mathcal{I}}$ (4) $$

For a complete sentence $Y$, we integrate the GAE relevance maps from each time step $t$ by averaging them to obtain the overall visual relevance related to a factual sentence. The final three maps are formulated as followings, in which
$\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{I}}\in\mathbb{R}^{1\times N}$,
$\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{Q}}\in\mathbb{R}^{1\times M}$, and
$\mathbf{R}_{\mathcal{Q}\rightarrow{}\mathcal{I}}\in\mathbb{R}^{M\times N}$.

$$ $\displaystyle\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{Q}}$ $\displaystyle=\frac{1}{L}\sum_{t=1}^{L}\mathbf{R}^{t}_{\mathcal{T}\rightarrow{ }\mathcal{Q}},$ $\displaystyle\mathbf{R}_{\mathcal{Q}\rightarrow{}\mathcal{I}}$ $\displaystyle=\frac{1}{L}\sum_{t=1}^{L}\mathbf{R}^{t}_{\mathcal{Q}\rightarrow{ }\mathcal{T}},$ $\displaystyle\mathbf{R}_{\mathcal{T}\rightarrow{}\mathcal{I}}$ $\displaystyle=\frac{1}{L}\sum_{t=1}^{L}\mathbf{R}^{t}_{\mathcal{T}\rightarrow{ }\mathcal{I}}$ (5) $$

For non-compression projectors maintaining the number of original patches, such as linear layers, the Query-to-Patch map is an identity mapping based on the one-to-one correspondence between queries and patches. Consequently, the Query-to-Image map visualizes the original image consisting of 576 patches. The Text-to-Query map is obtained in the same manner as in the QFormer, which propagates from the R-GAE maps in the Language Model (LLM).

For the AdaptiveAvgPool projector in the DeCo method, a 2D spatial down-sampling mapping is constructed from the original tokens to the compressed tokens. For an operation window with kernel size $K$, the merged token is assigned a relevance score equal to $1/K^{2}$ of the sum of the relevance scores of each raw token within the window. The corresponding Query-to-Patch map can be calculated using this simple mapping rule. Similar to the QFormer, the Text-to-Query map is obtained from the LLM layers.

## Appendix B Comparison between R-GAE and Raw Attention Maps

The R-GAE map offers two advantages over raw attention maps: (i) it demonstrates better explainability [] by integrating both attention maps and gradients, and (ii) it can track token relevance from a target layer (e.g., output textual tokens) to the first layer (e.g., original patch tokens). In contrast, the attention map commonly used from the last layer of the Large Language Model (LLM) can only show the relevance of mixed tokens in that layer, where the tokens related to the vision input position or the output word position have incorporated the semantics of other tokens through attention operation in previous layers.

Figure 9 visualizes the R-GAE relevance maps and the raw attention maps for a comparative analysis. The Query-to-Patch map of raw attention is obtained from the last cross-attention layer in the QFormer, while the Text-to-Query map of raw attention is derived from the last layer in the LLM. By visualizing the same model and image-text pair, it becomes evident that R-GAE provides a more interpretable representation of the inner vision-language alignment of an MLLM.
In contrast, the raw attention map highlights an unrelated visual patch, such as the sky, which introduces an additional error in the explainability procedure when analyzing semantic alignment. An in-depth analysis reveals that the error in the raw attention map primarily originates from the Text-to-Query map of the last LLM layer. This can be attributed to the fact that the LLM consists of 32 self-attention layers, and the relevance among query tokens and text tokens in the last layer has deviated due to the fusion of semantics from other tokens in previous layers.
On the other hand, the Query-to-Patch map exhibits relatively similar characteristics to the R-GAE map. This similarity can be explained by the architecture of the QFormer, which only employs a single cross-attention layer, thus minimizing the influence of token fusion across layers for raw attention.

Figure: Figure 9: Comparison between the R-GAE and raw attention map explainability on the same case from the QFormer projector, which compresses 576 vision tokens to 64 query tokens.
Refer to caption: x10.png

## Appendix C Training Hyper-parameters

#### Architecture of Used Projectors.

- 1.
C-Abstractor comprises 3-layer ResNet blocks [], the adaptive average pooling and another 3-layer ResNet blocks.
- 2.
D-Abstractor leverages Deformable Attention [] to replace the vanilla attention and conduct well-designed initialization of query tokens. We adopt a two-layer D-Abstractor.
- 3.
QFormer is a two-layer BERT [] architecture same as the the BLIP-2 [] and we load the BLIP-2 pre-training weights as an initialization.
- 4.
Linear projector is a two-layer MLP with the GELU activation same as the LLaVA v1.5 [].
- 5.
AdaptiveAvgPool is parameter-free, we utilize a two-layer MLP as the linear projector to map the vision feature dimension to the LLM’s.

#### Training Parameters.

Our experiments are conducted under two primary training settings. The main experiments are built on the LLaVA v1.5 framework, as shown in Table 4. The generalization experiments are constructed using a more lightweight setup that involves only the instruction tuning stage, referring to the PRISM [] approach. Specific training hyperparameters are detailed in Table 5.

**Table 4: Hyper-parameters of main experiments.**
| Hyperparameter | Pretrain | Finetune |
| --- | --- | --- |
| batch size | 256 | 128 |
| lr | 1e-3 | 2e-5 |
| lr schedule | cosine decay |  |
| lr warmup ratio | 0.03 |  |
| weight decay | 0 |  |
| epoch | 1 |  |
| optimizer | AdamW |  |
| DeepSpeed stage | 2 | 3 |

**Table 5: Hyper-parameters of generalization experiments.**
| Hyperparameter | Value |
| --- | --- |
| Batch Size | 128 |
| Max Gradient Norm | 1.0 |
| Weight Decay | 0.1 |
| Learning Rate | 2e-5 |
| Optimizer | AdamW |
| Scheduler | Warmup & Cosine Decay |
| Warmup Ratio | 0.03 |

## Appendix D More R-GAE Relevance Maps

Figure 10 presents additional visualized cases of the R-GAE relevance map across different projectors.

Figure: (a) R-GAE maps related to the generated text “small orange Nissan pick up truck in traffic”. In this case, all projectors reserve the effective visual representation and translate it to the LLM. Specifically, the Qformer-based MLLM attends to the query indexing $(0,2)$ which highlights the “Nissan” semantics on the image. This indicates extracting effective visual semantic concepts in the first abstraction by the QFormer is important for the traditional compressive projectors.
Refer to caption: x11.png

## Appendix E Broader Impacts

Our work utilizes off-the-shelf frozen LLMs, which means it shares some of their intrinsic drawbacks, such as generating hallucinated, ungrounded text or biased outputs. We mitigate these issues by enhancing the model’s grounding in both visual and instruction inputs. Additionally, our training dataset includes 40K examples of safety data sourced from ShareGPT, instructing the models to refuse responses to toxic, inappropriate, or otherwise unsafe inputs. However, we do not recommend applying our models to any downstream applications without a prior assessment of safety and fairness specific to that application.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. There is still no solid conclusion/reasoning w.r.t. why convolution based approach like C-abstractor fails.
soundness: 2
strengths: 1. The writing is clear. Reader know the contributionL use avg pooling to reduce the number of vision tokens.
summary: This paper use avg pooling to reduce the number of vision tokens.
weaknesses: 1. Contribution not enough as a ICLR paper. Average pooling has been using in a lot of other papers, including Phi-3.5-Vision, Qwen2-VL, etc. Not a new technique. 
2. For all content up to page 5, all observations with Q-former are visualization. Picked qualitative study should not and can not lead to the solid conclusion of why Q-former fails.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. What's the language instruction used in Figure 3?
soundness: 3
strengths: 1. The authors present an analysis of the relevancy between compressed tokens and input images.

2. The proposed method (adaptive average pooling comprehension) is easy to follow.
summary: The authors first reveal the drawbacks of the previous comprehension-based projector such as Q-Former. The authors use the R-GAE explainability tool to reveal limited visual semantics and repetitive patterns. Then the authors propose a new token comprehension technique based on adaptive average pooling. The authors conduct a series of experimental results to demonstrate the effectiveness of the proposed module.
weaknesses: 1. About the selected benchmarks.

    The authors claim that 'The non-compressed projector' struggles with high-resolution or video benchmarks. However, the selected benchmarks in this paper (e.g., SEED, MME) do not belong to these two categories. Thus, the non-compressed projector performs best in Table 1, and can not demonstrate the necessity of using the compressed projector. I suggest the author provides more experimental results on (1) high-resolution benchmarks, such as InfographicVQA; and (2) video benchmarks, such as Video-MME. And the authors may consider adding the time-efficiency comparison.

2. Limited MLLM baselines

    This paper is based on LLaVA-1.5 merely, which is far behind current SOTA models. I suggest the authors provide the experimental results on more recent MLLM models.

3. More related works such as token reduction

    Instead of designing a compressive projector, another direction is using the non-compressive projector plus the token reduction techniques, such as [1,2]. However, this paper does not discuss these related works.
    - [1] An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models, ECCV 2024
    - [2] Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations, ICLR 2022

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: please refer to weaknesses.
soundness: 4
strengths: 1. The analysis of existing visual projectors is convincing.
2. A simple 2D Adaptive Pooling outperforms traditional compressive projectors in both performance and efficiency.
summary: The manuscript conducts a thorough analysis of existing visual projectors and based on the proposed R-GAE, reveals that compressive projectors (e.g., QFormer) experience a "double abstraction" problem. To address this issue, the authors propose "Decouple Token Compression from Semantic Abstraction (DeCo)."
weaknesses: 1. The novelty of this paper is limited; the proposed method simply uses 2D adaptive average pooling, which may result in loss of detailed information. Also, the “double abstraction” problem is not solved; the pooling operation is also an abstraction in the vision modality.
2. In the manuscript, Q-Former, C(D)-Abstractor, and DeCo all use 144 visual tokens. If the number of visual tokens increases (reduce the compression rate), will DeCo still outperform other methods?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: 1. Although 2D Adaptive Average pooling can reduce the computational burden, the MLLM computation is mainly in the LLM. The increase in the computation of the projector does not have a significant impact on the overall computation. Therefore, from this point of view, does the module with learnable parameters such as convolution achieve better performance than 2D Adaptive Average pooling?

2. Table 3 shows that DeCo's performance is very close to MLP's performance, but the number of tokens is much lower. The reasons why DeCo can outperform MLP require further explanation.

3. Writing typo. L215 "an MLLM" -> "a MLLM".

**Post-rebuttal**

Unfortunately, the authors don't provide detailed responses to my concerns. Therefore, I have decided to give the final recommendation: **Strong reject**. After reading the comments from other reviewers, I conclude the reasons for the rejection decision.

1. We achieve the agreement that the novelty of this paper is limited (Reviewer NfRX, nzvd, and YoWL). Average pooling is a common practice in modern MLLM. This paper is just an analysis of its effectiveness, which does not mean so much to the ICLR community.

2. Limited experimental analysis. This paper lacks a detailed experimental analysis of why other visual projectors perform worse than average pooling (Reviewer NfRX and YoWL). Also, the MLLM benchmarks are limited (Reviewer prwT). The existing problems make the paper unable to validate the proposed method.

3. The proposed R-GAE is close-source, underweighting the significance of R-GAE.

Besides, **it seems that the authors have given up this paper, which is not a good example for open discussion in ICLR**. Overall, I agree with other reviewers that **this paper should be rejected**.
soundness: 1
strengths: 1. DeCo deeply analyzes the problem of double abstraction in MLLM, providing good insights for the MLLM community. Moreover, DeCo reveals why QFormer fails to achieve high performance.

2. R-GAE is able to facilitate the analysis of MLLM, which helps the researchers to analyze the problems that existed in MLLM.
summary: This paper proposes a novel visual projector, named DeCo. It adopts a 2D Adaptive Pooling to reduce the number of visual tokens. Besides, This article designs a novel R-GAE explainability tool to deeply analyze double abstraction in MLLM. The experimental results showed that DeCo achieves superior results across diverse benchmarks.
weaknesses: 1. The framework and methodology lack innovation. 2D Adaptive Average pooling is not a novel approach to reduce the number of visual tokens. For instance, PLLaVA [1] adopts the 2D Adaptive Average pooling to reduce the visual tokens.

2. The experimental analysis is not comprehensive enough, leading to the results being hard to understand. For example, in Table 1,  C-Abstractor and DeCo have the same number of visual tokens fed into LLM, and C-Abstractor performs local enhancement on the compressed tokens. It seems that C-Abstractor should outperform DeCo, but the experimental results show that DeCo performs better. The reasons need to be further explained.

3. Lack of necessary experiments. All experiments in DeCo are performed at normal resolution, but the visual token compression is useful for high-resolution scenes. Unfortunately, this part of the experiment is not reflected in the paper. Authors are suggested to add this experiment.

[1] Xu L, Zhao Y, Zhou D, et al. Pllava: Parameter-free llava extension from images to videos for video dense captioning[J]. arXiv preprint arXiv:2404.16994, 2024.