Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

## Abstract

Abstract To bridge the gap between vision and language modalities, Multimodal Large Language Models (MLLMs) usually learn an adapter that converts visual inputs to understandable tokens for Large Language Models (LLMs).
However, most adapters generate consistent visual tokens, regardless of the specific objects of interest mentioned in the prompt.
Since these adapters distribute equal attention to every detail in the image and focus on the entire scene, they may increase the cognitive load for LLMs, particularly when processing complex scenes.
To alleviate this problem, we propose prompt-aware adapters.
These adapters are designed with the capability to dynamically embed visual inputs based on the specific focus of the prompt.
Specifically, prompt-aware adapters utilize both global and local textual features to capture the most relevant visual clues from the prompt at both coarse and fine granularity levels.
This approach significantly enhances the ability of LLMs to understand and interpret visual content.
Experiments on various visual question answering tasks, such as counting and position reasoning, demonstrate the effectiveness of prompt-aware adapters.

## 1 Introduction

Recent advances in Large Language Models (LLMs) ; ; ; ; have significantly enhanced their performance across various natural language processing tasks.
These models are able to perform language comprehension and logical reasoning, enabling them to handle complex linguistic functions such as summarizing texts, answering questions, processing dialogues, and composing new essays or articles.
However, LLMs are inherently limited by their inability to process visual information.
This has led to the development of Multimodal Large Language Models (MLLMs) ; ; ; ; , which expand upon LLMs by incorporating visual processing abilities.
MLLMs offer a more holistic understanding and interaction by synthesizing both textual and visual data, thereby broadening their utility in diverse real-world applications.

Figure: Figure 1: Illustration comparing prompt-unaware and prompt-aware adapters. Left: Prompt-unaware adapter treats visual patches as a kind of words and directly converts these patches into “readable” tokens for LLMs, without considering the specific objects of interest. In this case, whether the question involves “pool” or “drinks,” they consistently generate the same tokens and allocate equal attention to every detail in the scene, which may increase the cognitive load for LLMs. Right: Prompt-aware adapter leverages prompts to collect the most relevant visual clues and generate adaptive tokes, thus enhancing the ability of LLMs to understand and interpret visual content.
Refer to caption: x1.png

To equip LLMs with visual perception ability, MLLMs typically employ a trainable adapter that connects a frozen visual encoder and a frozen LLM.
Adapters play a crucial role in bridging the gap between vision and language, enabling profound visual understanding while leveraging the powerful reasoning capabilities of LLMs.
However, most existing adapters ; ; ; ; ; ; ; approach visual patches as if they were words, and directly translate these patches into tokens comprehensible to LLMs (e.g., through linear projection), regardless of the specific objects of interest in the prompt.
These prompt-unaware adapters may enable LLMs to correctly analyze simple images but could struggle to understand complex scenes.
As shown in Fig. 1, no matter the query is “is there a pool in this image” or “how many drinks are on the table”, these adapters consistently convert the image into the same tokens.
Consequently, the subsequent LLMs have to independently parse the scene to deduce the spatial context and
shift attention to “pool” or “drinks” entirely on their own.

A few studies have explored using prompts to guide the behavior of adapters.
For instance, VisionLLM  and Flamingo  employs the cross-attention mechanism to learn adaptive visual tokens, where prompt words serve as the query and image patches are treated as the key and value.
InstructBLIP  first injects the prompt information into learned queries  (via self-attention) and then uses cross-attention to gather visual clues.
These cross-attention-based adapters encounter two main challenges.
First, they search for visual clues at the word level, thereby neglecting global information for capturing an overview of the prompt-related region.
Second, the use of the softmax function in cross-attention normalizes attention distribution from word to patch such that the total attention allocated to each word equals 1.
This implies that every word in the prompt, including function words like “a”, “the” and “is”, is forced to correspond to a specific region in the image.
Because these irrelevant words may correspond to different regions each time, this unrealistic assumption may lead adapters to produce unstable visual tokens, inevitably causing significant confusion for LLMs.

In this paper, we produce a prompt-aware adapter to adaptively embed visual inputs based on the prompt’s global and local representations.
Our adapter comprises two key components: prompt-aware global attention and prompt-aware local attention.
The global attention is designed to capture coarse-grained, prompt-related visual perceptions, while the local attention focuses on refining responses to specific, fine-grained regions of interest.
This dual approach allows adapters to effectively uncover visual contexts and shift attention to relevant areas as needed.
We conducted extensive experiments on the COCO-QA  and MME  datasets.
Compared to the prompt-unaware baseline, our method shows significant improvements on COCO-QA, enhancing object classification, counting, color recognition, and positional reasoning by 7.71%, 18.42%, 12.84%, and 9.51%, respectively. On the MME dataset, it boosts the total scores for perception tasks and cognition tasks by 59.43% and 46.91%, respectively.
Our method demonstrates superior performance in handling complex scenarios and parsing intricate problems, effectively sensing prompts and capturing informative details necessary for answering questions.
The contributions are as follows:

- •
Among the early efforts, we conduct a comprehensive study on the impact of prompts on adapters.
Our research reveals that prompt-independent adapters may be not sufficient to capture the most informative visual clues for visual understanding.
Furthermore, existing prompt-relevant adapters still suffer from issues such as failing to focus on crucial words in prompts or adequately aligning visual and semantic elements.
- •
We propose prompt-aware adapters that dynamically embed visual inputs based on prompts at both coarse and fine granularities.
Our method provides effective and convenient attention mechanisms for existing adapter-empowered MLLMs.
- •
Experiments on complex scene understanding applications demonstrate that the proposed method effectively improves the visual perception and reasoning capabilities of MLLMs.

## 2 Related Work

Multimodal Large Language Models and Adapters. The integration of the perceptual abilities of vision models ; ; ; with the reasoning capabilities of LLMs ; ; ; ; ; has given rise to Multimodal Large Language Models (MLLMs) ; ; ; ; ; .
In MLLMs, visual signals are converted into tokens that LLMs can understand, typically using adapters.
For example, MLLMs such as LLaVA , Shikra , MiniGPT , Matters , PandaGPT , Kosmos , and InfMLLM  utilize linear projection adapters.
QWEN-VL  and mPlUG-Owl  employ Perceiver-like architectures  as adapters.
Since the introduction of the Q-Former in BLIP-2 , many MLLMs ; have adopted it as an adapter for modal alignment.
The visual descriptions generated by the aforementioned prompt-independent adapters typically encompass the entire image, without emphasizing the specific visual details relevant to the prompt.
VisionLLM , Flamingo , and subsequent studies ; utilize cross-attention adapters to improve prompt-aware capabilities.
InstructBLIP  and Cheetor  incorporate prompt information into learned queries using self-attention and then employ cross-attention to collect visual clues.
These adapters often struggle with issues such as failing to focus on crucial words in prompts or adequately aligning visual and semantic elements.
In this paper, we introduce a new type of adapter designed to align visual signals with textual semantics for MLLMs.

Attention Mechanisms. The attention mechanism  enables vision-language models to focus on relevant information within the input while minimizing focus on irrelevant details. Its effectiveness and interpretability have inspired numerous subsequent studies ; ; . For example, introduces both hard and soft attention mechanisms that automatically learn to describe the contents of images.
Additionally, in the Visual Question Answering (VQA) task, it is crucial to focus on the relevant parts of both the image and the question.
Consequently, Co-Attention  was developed to facilitate simultaneous attention learning across multiple inputs (i.e., image and text).
Self-Attention  enables interactions between different positions within a single input sequence, thereby capturing content that is relatively more important. Drawing from sources such as ; ; , the Transformer , recognized as a foundational component of large models, incorporates Scaled Dot-Product Attention and Multi-Head Attention. Cross-Attention  has been developed within the Transformer  architecture to facilitate interaction between two different sequences. To enhance the interaction between images and text, we propose global and local attention mechanisms that automatically focus on the visual content of interest mentioned in the text.

## 3 Proposed Method

In this section, We first provide a brief overview of existing adapters for the visual perception of Multimodal Large Language Models (MLLMs).
Then, we describe how the proposed prompt-aware adapter enables MLLMs to effectively uncover visual context and adaptively shift attention for enhancing visual reasoning.

### 3.1 Preliminary: MLLM and Adapter

MLLM is an advanced type of artificial intelligence model that processes and understands information from multiple types of data, such as text, images, and sometimes audio or video.
This allows the model to perform tasks that involve more than one mode of communication, providing a richer understanding of the content than the text-only LLM.
MLLMs integrate capabilities from different domains.
For example, they can extract meaning from both the text in an image caption and the visual content of the image itself.
This is particularly useful in applications such as image captioning,
Visual Question Answering (VQA) and video understanding.

In MLLMs, a specialized module called adapter is commonly employed to enhance the model’s capability to process and integrate information from different modalities.
Adapters are particularly valuable because they allow pre-trained models to be adapted to new tasks or modalities without the need for extensive retraining of the entire model.
Formally, for a given visual input, a frozen visual encoder (such as ViT and Q-Former) is typically employed to extract the visual features ${\bm{X}}\in\mathbb{R}^{N\times C}$, where $N$ represents the number of patches and $C$ denotes the number of feature channels.
Because these visual features are not directly comprehensible to the subsequent LLM, the adapter is trained to translate them into tokens that the LLM can understand as follows,

$$ ${\bm{X}}^{\prime}=\mathrm{Adapter}({\bm{X}};\theta),$ (1) $$

where $\theta$ is the learnable parameters of the adapter, ${\bm{X}}^{\prime}\in\mathbb{R}^{N^{\prime}\times C^{\prime}}$ and $N^{\prime}$ represents the number of converted tokens and $C^{\prime}$ denotes the dimension of the LLM’s input.
In most MLLMs, the adapter is implemented as a linear projection layer that projects visual features into the textual feature space ; ; ; ; .
Additionally, the adapter in Honeybee  first applies a convolution to convert visual features and then downsamples them, resulting in $N^{\prime}<N$.
However, no matter linear projection or convolution, these adapters distribute equal attention to every detail in the visual input and focus on the entire scene, regardless of the specific objects of
interest mentioned in the prompt, converting visual input to invariant tokens.
In this case, these prompt-unaware adapters may struggle to effectively convert complex scenes for its subsequent LLM.

Figure: Figure 2: Illustration comparing the cross-attention (left) and proposed (right) adapters. (a) Methods like VisionLLM and Flamingo utilize text features as queries and visual features as keys and values in cross-attention. It assumes that each word in the prompt corresponds to specific regions. The number of converted visual tokens is equal to that of text features. (b) InstructBLIP first injects prompt information into learnable queries via self-attention, and then employs cross-attention. It assumes that each query in the learnable queries corresponds to specific regions. The number of converted visual tokens is equal to that of learnable queries. (c) Our adapter comprises global and local attention components. Due to the new attention calculation mechanism used in local attention, the number of converted visual tokens remains unchanged.
Refer to caption: x2.png

### 3.2 Prompt-Aware Adapter

To translate informative visual tokens for LLMs, we can use prompts to guide the behavior of adapters.
Suppose ${\bm{Y}}\in\mathbb{R}^{M\times D}$ is the prompt, where $M$ represents the number of words in the prompt and $D$ denotes the dimension of the word embedding.
This mechanism can be formulated as follows,

$$ ${\bm{X}}^{\prime}=\mathrm{Adapter}({\bm{X}},{\bm{Y}};\theta).$ (2) $$

There are many ways to implement Eq. (2).
As shown in Fig 2, VisionLLM  and Flamingo  utilize text features as queries and visual features as keys and values in cross-attention processes to facilitate modal interaction.
InstructBLIP  first incorporates prompt information into learned queries through self-attention, and then employs cross-attention to collect visual clues.
In summary, these methods are based on cross-attention, which can be formulated as follows,

$$ ${\bm{X}}^{\prime}=\mathrm{softmax}(\frac{{\bm{Q}}\cdot{\bm{K}}^{T}}{\sqrt{E}}) \cdot{\bm{V}},~{}~{}~{}~{}~{}~{}~{}{\bm{Q}}={\bm{Y}}\cdot{\bm{W}}_{q},~{}~{}{ \bm{K}}={\bm{X}}\cdot{\bm{W}}_{k},~{}~{}{\bm{V}}={\bm{X}}\cdot{\bm{W}}_{v},$ (3) $$

where ${\bm{W}}_{q}\in\mathbb{R}^{D\times E}$, ${\bm{W}}_{k}\in\mathbb{R}^{C\times E}$, ${\bm{W}}_{v}\in\mathbb{R}^{C\times E}$ and $\cdot$ denotes matrix multiplication.
In this paper, we use the lowercase $\mathrm{softmax}$ to indicate that the function is applied at each row.
This mechanism results in two outcomes:

- •
Attention. Because the use of the softmax function in cross-attention normalizes the attention distribution from each word to all patches, the total attention allocated to each word equals 1. Specifically, if ${\bm{a}}_{ij}$ represents the attention of the $i$-th word to the $j$-th patch, then $\sum_{j=1}^{N}{\bm{a}}_{ij}=1$.
- •
Output. Because the prompt ${\bm{Y}}$ serves as the query in cross-attention, it leads to ${\bm{X}}^{\prime}\in\mathbb{R}^{M\times E}$. This indicates that the number of converted visual tokens is $M$ (the number of words in the prompt), rather than $N$ (the number of patches in the visual input).

This suggests that every word in the prompt, including function words like “a”, “an”, “the”, “is” and “are”, is forced to correspond to a specific region in the image.
Given that these irrelevant words might correspond to different regions each time, this unrealistic assumption can cause adapters to generate unstable visual tokens, leading to significant confusion for LLMs.

Figure: Figure 3: Illustration of the proposed prompt-aware adapter. The adapter consists of a global attention component and a local attention component. The global attention, integrated into the visual encoder, is designed to capture coarse-grained, prompt-related visual perceptions. Meanwhile, the local attention focuses on refining responses to specific, fine-grained regions of interest.
Refer to caption: x3.png

Global Attention. To extract the most informative clues, we propose the prompt-aware adapter (shown in Fig. 3).
The adapter consists of a prompt-aware global attention component and a prompt-aware local attention component.
The global attention component searches for visual clues at the scene level, aiming to capture an overview of the region related to the prompt.
To this end, we first utilize CLIP’s text encoder to extract the global feature of the prompt.
Second, we employ a learned projection layer to map the global feature into the same space as the visual patch, i.e., ${\bm{y}}\in\mathbb{R}^{1\times C}$.
Third, the global feature of the prompt is appended to the visual patches, resulting in $[{\bm{X}};y]\in\mathbb{R}^{(N+1)\times C}$.
Fourth, self-attention is employed to allow the adapter to incorporate the global prompt information, which is as follows,

$$ ${\bm{I}}=\mathrm{SelfAttention}([{\bm{y}};{\bm{X}}]).$ (4) $$

In this way, a coarse extraction of visual clues mentioned in the prompt is conducted.
Finally, the global attention discards the prompt representation at the last position, yielding $N$ visual features, i.e., ${\bm{I}}\in\mathbb{R}^{N\times C_{i}}$, where $C_{i}$ is the feature dimension.

Local Attention. Global attention reflects the overall correlation between the visual input and prompt.
To capture local details, we design a local attention component.
First, local attention calculates a similarity matrix ${\bm{S}}\in\mathbb{R}^{N\times M}$ between text and visual features, as follows,

$$ ${\bm{S}}=\mathrm{SOFTMAX}\Big{(}\frac{\big{(}{\bm{I}}\cdot{\bm{W}}_{i})\cdot({ \bm{Y}}\cdot{\bm{W}}_{y})^{T}}{\sqrt{E}}\Big{)}$ (5) $$

where ${\bm{W}}_{i}\in\mathbb{R}^{C_{i}\times E}$ and ${\bm{W}}_{y}\in\mathbb{R}^{D\times E}$.
Here, we use the capital $\mathrm{SOFTMAX}$ to denote that the function is applied to the entire matrix.
Compared to the $\mathrm{softmax}$ function used in Eq. (3), $\mathrm{SOFTMAX}$ suggests that each word may or may not correspond to a visual patch in the scene, and similarly, each patch may or may not correspond to a word in the prompt.
This assumption is more flexible and realistic than that of cross-attention.

Second, the sum of each row ($\mathbb{R}^{1\times M}$) in ${\bm{S}}$ can be interpreted as the correlation between a specific patch and the entire prompt description.
Therefore, we calculate the weight of each visual patch by summing all attention focused on it, as follows,

$$ ${\bm{a}}_{i}=\sum_{j=1}^{M}{\bm{S}}_{ij},~{}~{}~{}{\bm{X}}^{\prime}=\mathrm{ MLP}({\bm{a}}^{T}\cdot{\bm{I}}),$ (6) $$

where ${\bm{a}}\in\mathbb{R}^{1\times N}$.
Then, local attention is applied to visual features. This process produces prompt-aware visual tokens. Finally, a multilayer perceptron (MLP) is employed to transform them.

Our method intrinsically differs from the cross-attention mechanism described in Eq. (3).
In cross-attention, prompt features ${\bm{Y}}$ function as ${\bm{Q}}$ (queries) and visual features ${\bm{X}}$ as ${\bm{K}}$ (keys) and ${\bm{V}}$ (values).
Here, each text token distributes its attention across visual patches such that the total sums to 1.
This means that even function words (like conjunctions and prepositions) must have their similarity scores with all visual tokens collectively equal 1.
This requirement is impractical, as it assumes that every text token is equally significant in directing attention to visual cues.
The proposed local attention mechanism ensures that $\sum_{i=1}^{N}\sum_{j=1}^{M}{\bm{S}}_{i,j}=1$. As a result, text tokens that are more relevant to the visual context (and have higher similarity scores) exert greater influence on visual encoding. This approach, akin to emphasizing certain words in a sentence when reading aloud, allows for the extraction of prompt-related visual features in a more detailed and nuanced manner.
In summary, there are two differences between cross-attention and the proposed local attention.

- •
Attention. The cross-attention implies that each word corresponds to a specific region, whereas local attention does not enforce such a correspondence.
- •
Output. The cross-attention produces $M$ (the number of words in the prompt) visual tokens, whereas local attention generates $N$ (the number of patches in the visual input) tokens.

## 4 Experiments

### 4.1 Implementation Details

Network Details.
In this paper, we utilize the open-sourced LLaMA2 (7B) model  as our Large Language Model (LLM). We employ the text encoder from CLIP  to ensure that the extracted text features closely align with the corresponding visual features in the embedding space.
ViT-g/14 from EVA-CLIP  is used as the vision encoder.
The global text token is injected into the vision encoder to conduct prompt-aware global attention.
The use AdamW   as the optimizer, with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay rate of $0.05$.
The learning rate is linearly warmed up from $10^{-6}$ to $8\times 10^{-5}$ over the first $1,000$ steps to accelerate model convergence. This is followed by a cosine decay of the learning rate to a minimum of $10^{-5}$. The training is set to a maximum of $50$ epochs, with $1,000$ iterations per epoch.
The model is trained with a batch size of 4, across a period of 3 days on a single NVIDIA RTX A6000 GPU.

Training Details and Datasets.
Following ; ,
the conversation template is as follows,

| 
 <Img> <Image Feature>  </Img> [Task Identifier] <Prompt Feature><br>
 |
| --- |

where 
 and 
 represent the user role and chat assistant, respectively.
During training, the <Image Feature> is replaced with the visual embeddings, and the <Prompt Feature> is substituted with textual prompt embeddings.
The [Task Identifier] is replaced based on specific circumstances (e.g., [vqa] and [caption]), making our model more adept at understanding multiple tasks.

We initialize our model with the pre-trained parameters of MiniGPT-V2 . MiniGPT-V2 has demonstrated impressive performance across multiple tasks, having undergone three stages of training on various fine-grained datasets ; ; ; ; ; ; ; ; ; ; ; ; ; .
We fine-tune the model on downstream tasks at a low cost to validate the effectiveness of our method. Our model is trained on the COCO-QA dataset , which includes a mix of question-answer pairs covering object classification, color recognition, counting, and positional reasoning.
The training image-text pairs account for around $67\%$, and the remaining pairs are used for the zero-shot image-to-text generation task.
Throughout the entire training process, the text encoder, global prompt-aware vision encoder, and LLM remain frozen.
Only the local prompt-aware vision adapter and additional projection layers are fine-tuned.
Evaluation is primarily conducted on the COCO-QA  test dataset and the MME  benchmark.
The latter includes $10$ perceptual (i.e., existence, count, position, color, poster, celebrity, scene, landmark, artwork, and OCR) and $4$ cognitive (i.e., commonsense reasoning, numerical calculation, text translation, and code reasoning) tasks.

Figure: Figure 4: Visualization of prompt-aware global and local attention. Global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question.
Refer to caption: x4.png

### 4.2 Evaluation

Attention Visualization.
To demonstrate the effectiveness of the proposed method, we visualize global and local attention in Fig. 4.
The global attention map effectively targets entire regions of the image that semantically align with the entire prompt sentence. The local attention map, associated with the question, highlights its ability to explore fine-grained semantics when extracting visual clues.

**Table 1: Quantitative results of $10$ perception tasks on the MME benchmark. Evaluation metric is scores for correct answers, with higher scores indicating better performance. $\ast$: frozen, $\checkmark$: fine-tuned. The best and second-best scores are highlighted in bold and underlined, respectively.**
| Method | Existence | Count | Position | Color | Poster | Celebrity | Scene | Landmark | Artwork | OCR | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 68.33 | 55.00 | 43.33 | 75.00 | 41.84 | 54.41 | 71.75 | 54.00 | 60.50 | 57.50 | 581.66 |
| PandaGPT | 70.00 | 50.00 | 50.00 | 50.00 | 76.53 | 57.06 | 118.00 | 69.75 | 51.25 | 50.00 | 642.59 |
| ImageBind-LLM | 128.33 | 60.00 | 46.67 | 73.33 | 64.97 | 76.47 | 113.25 | 62.00 | 70.75 | 80.00 | 775.77 |
| VPGTrans | 70.00 | 85.00 | 63.33 | 73.33 | 84.01 | 53.53 | 141.75 | 64.75 | 77.25 | 77.50 | 790.45 |
| LaVIN | 185.00 | 88.33 | 63.33 | 75.00 | 79.59 | 47.35 | 136.75 | 93.50 | 87.25 | 107.50 | 963.60 |
| mPLUG-Owl | 120.00 | 50.00 | 50.00 | 55.00 | 136.05 | 100.29 | 135.50 | 159.25 | 96.25 | 65.00 | 967.34 |
| LRV-Instruction | 165.00 | 111.67 | 86.67 | 165.00 | 139.04 | 112.65 | 147.98 | 160.53 | 101.25 | 110.00 | 1299.79 |
| BLIP-2 | 160.00 | 135.00 | 77.33 | 148.33 | 141.84 | 105.59 | 145.25 | 138.00 | 136.50 | 110.00 | 1293.84 |
| Flamingo | 70.00 | 60.00 | 46.67 | 73.33 | 41.84 | 57.06 | 68.00 | 62.00 | 60.50 | 65.00 | 604.40 |
| Multimodal-GPT | 61.67 | 55.00 | 58.33 | 68.33 | 57.82 | 73.82 | 68.00 | 69.75 | 59.50 | 82.50 | 654.72 |
| InstructBLIP | 185.00 | 143.33 | 66.67 | 153.33 | 123.81 | 101.18 | 153.00 | 79.75 | 134.25 | 72.50 | 1212.82 |
| Linear$\checkmark$ | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82 |
| Q-Former$\ast$ + Linear$\checkmark$ | 83.18 | 49.45 | 50.66 | 68.95 | 55.05 | 55.68 | 49.93 | 53.02 | 30.29 | 49.54 | 545.78 |
| Q-Former$\checkmark$ + Linear$\checkmark$ | 65.00 | 33.22 | 51.66 | 54.21 | 42.70 | 52.60 | 73.75 | 73.04 | 52.13 | 72.50 | 570.84 |
| Cross-Attention$\checkmark$ | 91.66 | 63.33 | 39.99 | 21.66 | 57.14 | 38.23 | 110.25 | 99.25 | 76.75 | 70.00 | 668.29 |
| Ours | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02 |

Quantitative Results.
For quantitative evaluation, we exclusively consider questions with precise and concise answers, due to the statistical challenges posed by open-ended responses from MLLMs ; ; .
First, we compare our model with MLLMs equipped with prompt-unware adapters, such as MiniGPT-4 , mPLUG-Owl , BLIP-2 , and MLLMs that consider prompts during visual encoding, like Multimodal-GPT , InstructBLIP .
The above comparison models are zero-shot evaluation results.
Second, we compare the proposed prompt-aware adapters with other popular adapters.
To ensure fairness, we strive to exclude the influence of training data and the number of parameters on MLLM performance.
Hence, we use the same dataset  to train MLLMs with MiniGPT-4/v2 ; as the uniform backbone, changing only the adapters, including linear projection, Q-Former family, cross-attention, and our prompt-aware adapter. Results are from the MME leaderboards(^1^11https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models/tree/Evaluation)

**Table 2: Quantitative results of object classification, counting, color recognition, and position reasoning on the COCO-QA test dataset. Evaluation metric is accuracy (%). $\ast$: frozen parameters, $\checkmark$: fine-tuned parameters. The best results are highlighted in bold.**
| Method | Object | Count | Color | Position | Total |
| --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 73.69 | 36.01 | 63.32 | 57.23 | 68.42 |
| Linear$\checkmark$ | 74.86 | 62.15 | 69.87 | 63.97 | 72.44 |
| Q-Former$\ast$ + Linear${\checkmark}$ | 56.50 | 43.04 | 29.75 | 35.23 | 49.73 |
| Q-Former${\checkmark}$ + Linear${\checkmark}$ | 28.46 | 38.83 | 32.09 | 26.43 | 29.67 |
| Cross-Attention${\checkmark}$ | 76.47 | 71.79 | 69.95 | 65.98 | 74.39 |
| Ours | 81.12 | 76.19 | 80.17 | 70.70 | 79.95 |

Tab. 2 shows the quantitative results on the COCO-QA  text dataset.
The proposed method is capable of selectively focusing on objects mentioned in the question during visual feature extraction, leading to a noticeable improvement (approximately 5.78%) in quantity perception compared to cross-attention adapters.
In the object classification, color recognition and positional reasoning tasks, our prompt-aware adapters surpass cross-attention adapters by more than 5.73%, 12.75%, and 6.95%, respectively.
As depicted in Tab.s 1 and 3, compared to prompt-unaware adapters, our method excels in both perception tasks ($1375.02$ vs $1299.79$) and cognition tasks ($289.28$ vs $210.31$) on the MME  benchmark.

Qualitative Results.
We qualitatively compare our model with several popular MLLMs on perception and cognition tasks with more diverse visual inputs and prompts.
Among the compared methods, LLaVA , MiniGPT-4 , BLIP-2  adopt prompt-unaware adapters.
While Flamingo  and InstructBLIP  inherently extract prompt-related visual signals through cross-attention.
For visual results, refer to Fig.s 6$\sim$5 and Appendix.
Thanks to the prompt-aware global and local attention, our model shows a clear improvement in following prompts to focus on specific visual clues.

Figure: Figure 5: Qualitative results of the proposed method on diverse perception and cognition tasks.
Refer to caption: x5.png

### 4.3 Ablation Study

**Table 3: Quantitative results of four cognition tasks on the MME benchmark. Evaluation metric is scores for correct answers, with higher scores indicating better performance. $\ast$: frozen, $\checkmark$: fine-tuned.**
| Method | Common | Calculate | Translate | Code | Total |
| --- | --- | --- | --- | --- | --- |
| MiniGPT-4 | 59.29 | 45.00 | 0.00 | 40.00 | 144.29 |
| Linear$\checkmark$ | 58.57 | 42.50 | 5.00 | 47.50 | 153.57 |
| Q-Former$\ast$ + Linear$\checkmark$ | 37.24 | 78.07 | 50.00 | 45.00 | 210.31 |
| Q-Former$\checkmark$ + Linear$\checkmark$ | 38.45 | 0.00 | 47.50 | 5.00 | 90.95 |
| Cross-Attention$\checkmark$ | 40.00 | 52.50 | 15.00 | 57.50 | 165.00 |
| Ours | 99.28 | 50.00 | 87.50 | 52.50 | 289.28 |

Ablation Details. In this section, we analyze the necessity of global and local attention. During validation for the former, the global text token is not added to the visual encoder for guidance, while for the latter, a simple linear projection layer is employed to replace the designed local attention component.
We also test the baseline that does not incorporate any prompt-related attention mechanism during visual feature extraction.
To ensure fairness, the other experimental settings are identical.

Prompt-Aware Global Attention.
Results (see Tab.s 4$\sim$6) suggest that the prompt-unaware method struggles to perceive visual inputs and make reasonable inferences, as it does not focus on visual signals emphasized by prompts.
The global attention component directs attention allocation with complete prompt embeddings in the initial phase, thus extracting semantically aligned visual embeddings.
Without prompt-aware global attention, the answer accuracy of MLLM drops obviously.

Prompt-Aware Local Attention.
We evaluate the local prompt-aware adapter against linear projection without local attention.
According to the results in Tab.s 4$\sim$6, the local prompt-aware attention component effectively contributes to the perceptual and cognitive capabilities of our MLLM.
Especially in counting and position reasoning tasks, the correct scores are enhanced by $16.05\%$ and $31.66\%$ (shown in Tab. 4), respectively.
Responding to fine-grained prompt guidance during the visual translation process relieves the comprehension load of LLMs.

Figure: Figure 6: Comparison of qualitative results between our method and other popular MLLMs.
Refer to caption: x6.png

**Table 4: Ablation study of prompt-aware global and local attention on MME perception tasks. The best and second-best scores are highlighted in bold and underlined, respectively.**
| Method | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| prompt-unware | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82 |
| w/o global-atten | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49 |
| w/o local-atten | 175.00 | 113.33 | 68.33 | 99.99 | 152.38 | 131.17 | 155.25 | 119.75 | 122.75 | 80.00 | 1217.97 |
| w/ global + local | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02 |

**Table 5: Ablations on COCO-QA test set.**
| Method | Object | Count | Color | Position | Total |
| --- | --- | --- | --- | --- | --- |
| prompt-unware | 74.86 | 62.15 | 69.87 | 63.97 | 72.44 |
| w/o global-atten | 78.67 | 73.94 | 79.12 | 68.40 | 77.75 |
| w/o local-atten | 77.08 | 74.59 | 76.06 | 67.83 | 76.15 |
| w/ global + local | 81.12 | 76.19 | 80.17 | 70.70 | 79.95 |

## 5 Conclusions

In this paper, we introduce novel prompt-aware adapters designed to adaptively embed visual inputs based on the given prompt. Our method aims to extract the most informative visual clues relevant to the prompt, thereby enhancing the visual understanding capabilities of LLMs. We first present a global attention mechanism that utilizes a global text token for coarse-grained visual guidance.
Then, we introduce a local attention mechanism that leverages local textual features to extract highly relevant visual clues with fine granularity. The resulting prompt-aware global and local visual tokens significantly reduce the visual perception burden on LLMs. Experimental results show that our method achieves competitive accuracy on multiple visual perception and cognitive tasks. We discuss the limitations and broader impacts of our method in Appendix.

## Appendix A Appendix Overview

In summary, the appendix is organized as follows:

Appendix B: Additional experiments and analysis.

Appendix C: More visualization results of prompt-aware global and local attention.

Appendix D: More qualitative results and analysis.

Appendix E: More details of COCO-QA dataset and MME benchmark.

Appendix F: Prompts used for fine-tuning.

## Appendix B Additional Experiments and Analysis

### B.1 Comparison of Text Encoders

During the implementation of the global and local attention mechanisms, we compare the quantitative results of text feature extraction by different text encoders.
Tab.s 7$\sim$9 show comparison results of adopting CLIP [] and Llama 2 [] to extract prompt features for global and local attention.
G-Num refers to the number of text tokens used in global attention.
0 0 means no global attention is used in the model architecture, $1$ means only one global text token is used, and $L$ means the total number of text tokens.

The experimental results show that using global text tokens performs better than not using them, confirming the effectiveness of our prompt-aware global attention. Besides, CLIP [] generally performs better than Llama 2 [] on perceptual tasks, while Llama 2 [] excels in cognitive tasks.
This may be because, in perceptual tasks, the text features extracted by CLIP can better align with visual features. In cognitive tasks, Llama 2 [], having been trained on vast amounts of data, possesses superior reasoning abilities.

**Table 7: Comparison results of using different text encoders (CLIP [] and Llama 2 []) to extract prompt features for global and local attention. G-Num refers to the number of text tokens used in global attention, and $L$ is the total amount of text tokens. The evaluation is built on the MME [] perception tasks, the evaluation metric is correct answer scores, with higher scores being better. The best and second-best scores are highlighted in bold and underlined, respectively.**
| Method | Perception Tasks |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| G-Num | Global | Local | Existence | Count | Position | Color | Poster | Celebrity | Scene | Landmark | Artwork | OCR | Total |
| 0 0 | - | - | 88.33 | 35.00 | 20.00 | 43.33 | 24.82 | 93.82 | 74.75 | 108.75 | 41.50 | 27.5 | 557.82 |
| 0 0 | - | CLIP | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49 |
| $1$ | CLIP | - | 175.00 | 113.33 | 68.33 | 99.99 | 152.38 | 131.17 | 155.25 | 119.75 | 122.75 | 80.00 | 1217.97 |
| $1$ | CLIP | Llama 2 | 170.00 | 121.66 | 48.33 | 99.99 | 130.27 | 127.94 | 149.50 | 134.25 | 119.50 | 125.00 | 1226.46 |
| $L$ | CLIP | Llama 2 | 170.00 | 106.66 | 48.33 | 70.00 | 128.23 | 118.23 | 154.25 | 139.50 | 119.25 | 102.50 | 1156.96 |
| $L$ | CLIP | CLIP | 160.0 | 111.66 | 73.33 | 90.00 | 133.33 | 131.76 | 158.00 | 146.50 | 119.25 | 95.00 | 1218.84 |
| $1$ | CLIP | CLIP | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02 |

**Table 8: Comparison results of using different text encoders (CLIP [] and Llama 2 []) to extract prompt features for global and local attention. G-Num refers to the number of text tokens used in global attention, and $L$ is the total amount of text tokens. The evaluation is built on the MME [] cognition tasks, the evaluation metric is correct answer scores, with higher scores being better. The best and second-best scores are highlighted in bold and underlined, respectively.**
| Method | Cognition Tasks |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| G-Num | Global | Local | Common Sense | Calculation | Translation | Code | Total |
| 0 0 | - | - | 58.57 | 42.50 | 5.00 | 47.50 | 153.57 |
| 0 0 | - | CLIP | 96.42 | 50.00 | 55.00 | 55.00 | 256.42 |
| $1$ | CLIP | - | 99.28 | 47.50 | 50.00 | 52.50 | 249.28 |
| $1$ | CLIP | Llama 2 | 83.57 | 57.50 | 57.50 | 47.50 | 246.07 |
| $L$ | CLIP | Llama 2 | 92.14 | 50.00 | 65.00 | 47.50 | 254.64 |
| $L$ | CLIP | CLIP | 96.42 | 45.00 | 57.50 | 47.50 | 246.42 |
| $1$ | CLIP | CLIP | 99.28 | 50.00 | 87.50 | 52.50 | 289.28 |

**Table 9: Comparison results of using different text encoders (CLIP [] and Llama 2 []) to extract prompt features for global and local attention. G-Num refers to the number of text tokens used in global attention, and $L$ is the total amount of text tokens. The evaluation is built on the COCO-QA [] cognition tasks, the evaluation metric is correct answer scores, with higher scores being better. The best and second-best scores are highlighted in bold and underlined, respectively.**
| Method | Perception Tasks |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| G-Num | Global | Local | Object | Count | Color | Position | Total |
| 0 0 | - | - | 74.86 | 62.15 | 69.87 | 63.97 | 72.44 |
| 0 0 | - | CLIP | 78.67 | 73.94 | 79.12 | 68.40 | 77.75 |
| $1$ | CLIP | - | 77.08 | 74.59 | 76.06 | 67.83 | 76.15 |
| $1$ | CLIP | Llama 2 | 80.57 | 76.13 | 78.64 | 68.76 | 79.25 |
| $L$ | CLIP | Llama 2 | 80.41 | 76.62 | 77.41 | 70.94 | 79.07 |
| $L$ | CLIP | CLIP | 80.51 | 76.40 | 79.07 | 69.00 | 79.25 |
| $1$ | CLIP | CLIP | 81.12 | 76.19 | 80.17 | 70.70 | 79.95 |

### B.2 Residual Visual Details

We also try to fuse the visual features obtained through global attention and the visual features obtained through local attention.
$Ratio$ represents the retention ratio of local visual features, and the retention ratio of global visual features is (1 - $Ratio$).
For example, when $Ratio$=0.2, it means that the local features are multiplied by 0.2 and the global features are multiplied by 0.8 and then added.

Tab.s  10$\sim$11 show the impact of using different $Ratio$ values on the model’s perceptual and cognitive tasks.
Experimental results show that the weighted sum of global and local features helps to improve model performance to a certain extent but not very much.
In addition, when local features account for greater weight, the model seems to perform better.

### B.3 Exploration of Prompt-Aware Q-Former

Inspired by InstructBLIP [], we further explore novel architectures for prompt-aware Q-Former.
In this section, we implement a Q-Former-based prompt-aware adapter.
Specifically, we try to add prompt cross-attention to different blocks in Q-Former.
Table 12 shows the quantitative evaluation results of various model-specific implementations.
Through extensive experiments, we found that the Q-former is difficult to train, and it tends to lose previously learned knowledge when fine-tuning on downstream tasks. Additionally, the effects of injecting text features into different layers of the Q-Former vary significantly.

**Table 10: Comparison results of using different $Ratio$ to combine global and local visual features. The value range of $Ratio$ is 0-1. The evaluation is built on the MME [] perception tasks, the evaluation metric is correct answer scores, with higher scores being better. The best and second-best scores are highlighted in bold and underlined, respectively.**
| $Ratio$ | Existence | Count | Position | Color | Posters | Celebrity | Scene | Landmark | Artwork | OCR | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 0.2 | 85.00 | 65.00 | 58.33 | 51.66 | 71.42 | 40.29 | 52.25 | 99.75 | 56.25 | 70.00 | 649.97 |
| 0.4 | 175.00 | 108.33 | 68.33 | 73.33 | 130.27 | 136.17 | 156.75 | 148.25 | 136.00 | 95.00 | 1227.44 |
| 0.6 | 175.00 | 126.66 | 91.66 | 90.00 | 152.04 | 129.11 | 147.25 | 117.50 | 122.75 | 132.50 | 1284.49 |
| 0.8 | 185.00 | 135.00 | 99.99 | 99.99 | 144.89 | 141.47 | 159.00 | 145.15 | 132.00 | 132.50 | 1375.02 |
| 1.0 | 190.0 | 128.33 | 91.66 | 150.00 | 152.72 | 144.41 | 155.50 | 141.00 | 139.75 | 80.00 | 1373.38 |

**Table 11: Comparison results of using different $Ratio$ to combine global and local visual features. The value range of $Ratio$ is 0-1. The evaluation is built on the MME [] cognition tasks. the evaluation metric is correct answer scores, with higher scores being better. The best and second-best scores are highlighted in bold and underlined, respectively.**
| $Ratio$ | Commonsense | Calculation | Translation | Code | Total |
| --- | --- | --- | --- | --- | --- |
| 0.2 | 67.14 | 65.00 | 5.00 | 47.50 | 184.64 |
| 0.4 | 97.14 | 47.50 | 50.00 | 47.50 | 242.14 |
| 0.6 | 96.42 | 50.00 | 55.00 | 55.00 | 256.42 |
| 0.8 | 92.85 | 57.50 | 57.50 | 45.00 | 252.85 |
| 1.0 | 103.57 | 55.00 | 72.50 | 47.50 | 278.57 |

**Table 12: Quantitative evaluation results of various model-specific implementations. The “No." is the number of the experimental setup. The “BlockNum" denotes the number of attention blocks in which the text cross-attention layer is inserted. The “OC", “CON", “CR", and “PR" represent the four tasks of object classification, counting, color recognition, and position reasoning, respectively. The “Acc (%)“ denotes accuracy and “Total Acc (%)" refers to the overall accuracy on four tasks.**
| No. | BlockNum | OC | CON | CR | PR | Total |
| --- | --- | --- | --- | --- | --- | --- |
| Acc (%) | Acc (%) | Acc (%) | Acc (%) | Acc (%) |  |  |
| 1 | 0 | 78.09 | 73.22 | 74.23 | 65.54 | 76.33 |
| 2 | 0+1 | 65.46 | 70.84 | 51.55 | 52.31 | 62.73 |
| 3 | 0+1+2 | 77.67 | 65.48 | 73.49 | 63.08 | 75.23 |
| 4 | 0+1+2+3 | 78.20 | 69.95 | 69.89 | 62.47 | 75.29 |
| 5 | 0+1+2+3+4 | 77.72 | 69.35 | 79.06 | 62.47 | 76.38 |
| 6 | 0+1+2+3+4+5 | 78.52 | 71.43 | 74.85 | 67.39 | 76.73 |
| 7 | 1 | 29.31 | 36.31 | 8.68 | 10.47 | 25.23 |
| 8 | 1+3+5+7+11 | 33.21 | 29.47 | 9.79 | 22.77 | 28.50 |
| 9 | 0+2+4+6+8+10 | 77.98 | 69.05 | 72.37 | 64.31 | 75.59 |
| 10 | 11 | 77.02 | 68.46 | 60.97 | 62.77 | 72.93 |
| 11 | 10+11 | 75.15 | 71.73 | 72.37 | 64.62 | 73.79 |
| 12 | 9+10+11 | 74.89 | 69.95 | 72.87 | 66.47 | 73.69 |
| 13 | 8+9+10+11 | 75.09 | 69.95 | 73.49 | 65.85 | 73.89 |
| 14 | 7+8+9+10+11 | 78.65 | 72.03 | 78.31 | 67.69 | 77.26 |
| 15 | 6+7+8+9+10+11 | 77.72 | 68.76 | 75.97 | 64.92 | 76.1 |
| 16 | 0+1+2+3+4+5+6+7+8+9+10+11 | 76.70 | 69.05 | 74.73 | 64.62 | 75.09 |

## Appendix C More Visualization Results of Prompt-Aware Global and Local Attention

In this section, we present more results of global and local attention visualization. Fig. 7 aims to verify the role of the proposed attention mechanism in object recognition tasks. Fig.s 8 to 10 re-validate the superiority of the proposed prompt-aware adapter in counting. Fig. 11 and Fig. 12 illustrate the focal points of the attention mechanism in color recognition and position reasoning tasks, respectively. A large number of visualization results illustrate the effectiveness of our method and provide a degree of interpretability.

Figure: Figure 7: Visualization of prompt-aware global and local attention on the object perception task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question.
Refer to caption: x7.png

Figure: Figure 8: Visualization of prompt-aware global and local attention on counting task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question. Our method performs well in accurately identifying and counting objects.
Refer to caption: x8.png

Figure: Figure 9: Visualization of prompt-aware global and local attention on counting task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question. Our method performs well in accurately identifying and counting objects.
Refer to caption: x9.png

Figure: Figure 10: Visualization of prompt-aware global and local attention on counting task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question. Our method performs well in accurately identifying and counting objects.
Refer to caption: x10.png

Figure: Figure 11: Visualization of prompt-aware global and local attention on the color recognition task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question. Our method achieves impressive performance in accurately identifying and distinguishing colors.
Refer to caption: x11.png

Figure: Figure 12: Visualization of prompt-aware global and local attention on the position reasoning task. Visualization results show that global attention spans the entire prompt content, while local attention concentrates predominantly on the specific object in question. Our method has the capability of location recognition.
Refer to caption: x12.png

## Appendix D More Qualitative Results and Analysis

We also provide more qualitative results with a wider range of image and prompt inputs (as shown in Fig.s13$\sim$26). Specifically, Fig. 13 focuses on the color recognition task, Fig. 14 on the counting task, Fig. 15 on the position reasoning task, Fig.
17 on the OCR task, Fig. 18 on the landmark recognition task, Fig. 19 on the celebrity recognition task, and Fig. 20 on the poster recognition task. Additionally, Fig. 21, Fig. 22and Fig. 23 respectively showcase the model’s capabilities in computation, coding, and common sense reasoning. Finally, Fig.s 24$\sim$26 provide visualization results for mixed perception tasks.

Figure: Figure 13: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the color recognition task.
Refer to caption: x13.png

Figure: Figure 14: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the counting task.
Refer to caption: x14.png

Figure: Figure 15: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the position reasoning task.
Refer to caption: x15.png

Figure: Figure 16: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the object detection task.
Refer to caption: x16.png

Figure: Figure 17: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the OCR task.
Refer to caption: x17.png

Figure: Figure 18: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the landmark recognition task.
Refer to caption: x18.png

Figure: Figure 19: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability on the celebrity recognition task.
Refer to caption: x19.png

Figure: Figure 20: Zero-shot image-to-text generation using an MLLM based on the poster recognition task.
Refer to caption: x20.png

Figure: Figure 21: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual cognition ability on the calculation task.
Refer to caption: x21.png

Figure: Figure 22: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual cognition ability on the code reasoning task.
Refer to caption: x22.png

Figure: Figure 23: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual cognition ability on the commonsense reasoning task.
Refer to caption: x23.png

Figure: Figure 24: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability.
Refer to caption: x24.png

Figure: Figure 25: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability.
Refer to caption: x25.png

Figure: Figure 26: Zero-shot image-to-text generation using an MLLM based on the prompt-aware adapter, where it shows favorable visual perception ability.
Refer to caption: x26.png

## Appendix E More Details of COCO-QA Dataset and MME Benchmark

### E.1 COCO-QA Dataset

Object Classification.
The primary objective of the object classification task is to accurately recognize objects in the given visual input and determine the category to which they belong.
The metric used to evaluate this task is accuracy.
For this task, we utilize $82,198$ question-answer pairs related to objects from the COCO-QA dataset [], with $78,666$ used for fine-tuning and $3,532$ for testing.
The questions typically involve the use of “what" to inquire about the object type. In addition, a succinct single-word answer is employed to precisely indicate the response.

Counting.
The main goal of counting is to recognize and count the number of questioned objects from the given visual input.
The metric employed to assess this task is accuracy.
In the counting task, we use $19,568$ question-answer pairs related to the number of objects from the COCO-QA dataset [], with $1,8761$ used for fine-tuning and $807$ for testing.
When posing questions, the inquiry typically involves the use of “how many" to inquire about the object amount. Likewise, we only use a single word describing the quantity in the response.

Color Recognition.
The color recognition task is designed to detect objects in questions and perceive color information from the input visual signal.
The metric employed to assess this task is also accuracy.
In this task, we utilize $8,640$ question-answer pairs related to objects from the COCO-QA dataset [], with $8,304$ used for fine-tuning and $336$ for testing.
Questions related to color are relatively straightforward, usually beginning with “What is the color of".
To facilitate quantitative evaluation, responses responses are kept as brief as possible for color-related words.

Position Reasoning.
In the position reasoning task, our primary goal is to infer the location information of the queried object based on the input visual content.
Similarly, we adopt the accuracy as the evaluation metric.
We adopt $7,278$ question-answer pairs related to objects from the COCO-QA dataset [], with $6,953$ used for fine-tuning and $325$ for testing.
In this task, questions are formatted to start with “where" to ask for the position of an object.
Answers mostly be places, scenes, or large objects that contain smaller objects.

Examples of questions and answers from four perceptual tasks are shown in Table 13.

**Table 13: Examples of questions and answers from four visual perception tasks.**
| Visual Perception Task | Question | Answer |
| --- | --- | --- |
| Object Classification | what are sitting down on the ground | bears |
| what is parked on the side of the grass | motorcycle |  |
| what are two mhttps://www.overleaf.com/project/64eb0aa69f2b1a658d35a15ben playing with some elephants | ball |  |
| what is the color of the shirt | blue |  |
| what is laying on the bed next to some pillows | cat |  |
| Counting | how many men is sitting on the street in front of a building | two |
| how many red velvet cup cakes with no frosting on a flowered plate | three |  |
| how many pairs of shoes on a mat with a cat is sitting in the middle | eight |  |
| how many dessert treats in the white cardboard box | six |  |
| how many trays of itallian food are in large pans | four |  |
| Color Recognition | what is the color of the airplane | black |
| what is the color of the motorcycle | orange |  |
| what is the color of the brush | green |  |
| what is the color of the bird | white |  |
| what is the color of the flowers | red |  |
| Position Reasoning | where is the cat lounging | chair |
| where do the mother and son make sundaes | kitchen |  |
| where is the cheese pizza | box |  |
| where is the person sitting | bed |  |
| where do the large and over-sized stuffed teddy bear sitting | chair |  |

### E.2 MME Benckmark

MME [] benchmark evaluates both the perception and cognition abilities of MLLMs. In addition to OCR, perception encompasses the recognition of both coarse-grained and fine-grained objects. Coarse-grained recognition includes the identification of objects’ existence, count, position, and color. Fine-grained recognition involves identifying movie posters, celebrities, scenes, landmarks, and artworks. Cognition tasks include commonsense reasoning, numerical calculation, text translation, and code reasoning. There are a total of 14 sub-tasks, detailed as follows.

Coarse-Grained Recognition.
The contents of coarse-grained recognition include identifying the existence of common objects and determining their count, color, and position. Images are sourced from the COCO dataset [], but all instruction-answer pairs are manually constructed rather than directly using available annotations. Even if MLLMs have previously encountered these COCO images, the manually prepared pairs are not included in their training sets. This setup requires MLLMs to comprehend the instructions and infer the correct answers. For each perception subtask of existence, count, color, and position, 30 images are used, with a total of 60 instruction-answer pairs prepared.

Fine-Grained Recognition.
Fine-grained recognition evalutates the knowledge resources of MLLMs, encompassing subtasks such as recognizing movie posters, celebrities, scenes, landmarks, and artworks. These subtasks include 147, 170, 200, 200, and 200 images respectively. For the celebrity subtask, a red box is placed around a person with a clearly visible face in the image, with the instruction being, “Is the actor inside the red box named [celebrity name]? Please answer yes or no.” Similar to the coarse-grained recognition, images for these subtasks are sourced from publicly available datasets [, , , , ], and all instructions are manually crafted.

Optical Character Recognition.
Optical Character Recognition (OCR) is a foundational capability of MLLMs, supporting subsequent text-based tasks such as text translation and text understanding. Images are sampled from [], with all instruction-answer pairs manually designed. Given that MLLMs are still in their early stages, only relatively simple samples are chosen for this version of MME. This section includes 20 images and 40 instruction-answer pairs.

All image-prompt-answer pairs are manually created. The few public datasets used in this benchmark utilize only the images without relying on their original annotations.
Additionally, images are collected through real photographs and image generation.
The prompts in MME are designed to be concise to minimize the influence of prompt engineering on the model’s output.
Due to the instruction design of “please answer yes or no” quantitative statistics can be easily performed based on the “yes” or “no” responses from MLLMs, ensuring accuracy and objectivity.

## Appendix F Prompts used for Fine-tuning

The prompt templates employed in the end-to-end prompt fine-tuning process across various tasks are illustrated in Table 14.

**Table 14: Prompt templates used for converting datasets into prompt-tuning data. In VQA tasks, two formats of prompts are available. Note that the “{Question}" format is mandatory for the prompt-aware adapter, and needs to be replaced with the specific question in Table 13.**
| Task | Prompt Template |
| --- | --- |
| Image Captioning | 
 <Img> <Image Feature>  </Img> [Caption] <Describe this image in detail.> 
 |
| 
 <Img> <Image Feature>  </Img> [Caption] <Take a look at this image and describe what you notice.> 
 |  |
| 
 <Img> <Image Feature>  </Img> [Caption] <Please provide a detailed description of the picture.> 
 |  |
| 
 <Img> <Image Feature>  </Img> [Caption] <Could you describe the contents of this image for me?> 
 |  |
| 
 <Img> <Image Feature>  </Img> [Caption] <Is this image common in real world?> 
 |  |
| Visual Question Answering | 
 <Img> <Image Feature>  </Img> [vqa] <Describe this image in detail. > 
 |
| 
 <Img> <Image Feature>  </Img> [vqa] <Take a look at this image and describe what you notice. > 
 |  |
| 
 <Img> <Image Feature>  </Img> [vqa] <Could you describe the contents of this image for me? > 
 |  |
| 
 <Img> <Image Feature>  </Img> [vqa] <Is this image common in real world? > 
 |  |
| 
 <Img> <Image Feature>  </Img> [vqa] <Question > 
 |  |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: - The experiments are sufficient and demonstrate that the proposed method performs well across multiple visual perception and cognitive tasks, indicating good generalizability suitable for a wide range of applications. 
- The paper is well-written with a clear structure, making it easy to follow.
- The proposed new adapter and dual attention mechanism appears to be an innovative approach to processing visual inputs.
summary: This paper studies the impact of prompts on adapters, revealing that prompt-independent adapters still have deficiencies in visual understanding. To this end, this paper proposes a new adapter that dynamically injects visual input into LLM through prompts. The proposed global attention mechanism and local attention mechanism can utilize the overall content or local content of textual information for visual guidance at different granularities. The experiments are conducted on a large number of complex scene understanding datasets.
weaknesses: - The motivation of dual attention is not clear. The visual clues can be captured by cross-attention at token level from prompt. Why do we need to split it into global attention and local attention? And the visual clues are typically given at a word/token level. Why do we need a global attention?
- The differences in the metrics in the "Calculate" column of Table 3 are quite substantial and require further explanation. The Q-Former∗ + Linear model achieves a score of 78.07, while the proposed method only scores 50.
- There are two bolded numbers in the "Position" column of Table 3. Please correct this issue.
- The results in Table 1 indicate that the proposed method performs poorly on the color recognition task (99.99 vs. 165.00). What factors might contribute to this discrepancy? Could you provide an appropriate analysis or evidence to support this observation?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to the Weaknesses section.
soundness: 2
strengths: 1.	The paper is well-written, logically structured, and easy to follow.
2.	The motivation and the proposed prompt-aware global and local attention are generally reasonable.
3.	The ablation studies are extensive, and the selection of comparison methods is appropriate.
summary: This paper proposes a text-guided feature transformation for the vision encoder. Similar to InstructBLIP, it enhances the correlation between visual representations and text instructions by introducing a global text token and a local attention mechanism.
weaknesses: 1.	The major weakness is the lack of novelty. Multimodal fusion for vision encoders has already been extensively explored and validated. Compared to VisionLLM and InstructBLIP, the proposed prompt-aware global and local attention is merely a combination of existing techniques, offering minimal contribution.
2.	The experimental comparison in Table 1 is not fair. This paper uses LLaMA2-7B as the large language model, while its main counterparts, such as InstructBLIP, use the weaker Vicuna-7B model. Since the choice of vision encoder and LLM directly impacts multimodal performance, the performance improvements in the experiments can not reflect the advantages of the proposed method.
3.	As shown in Eq. 5, compared to cross-attention, the proposed prompt-aware local attention uses global softmax to create a single association between text tokens and vision tokens. This implicitly assumes that each text token will have a high correlation with only one visual token. However, this assumption does not hold in many cases, especially when visual patches are densely divided.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Please discuss given the recent progress of Open o1/4o and Gemini which can describe the image contents in details, why it is important to use different way to transform the visual tokens in the adapter. 

Please discuss how to extend the proposed method to more complicated multi-modal recognition tasks.
soundness: 3
strengths: The paper proposes a simple and effective way to improve the adapter between a visual encoder and an LLM. The mapping of CLIP features of a prompt to visual features and the adaptive weights of visual patches w.r.t. individual words in the prompt are easy to reproduce.

The experiments show that the proposed method is effective to improve the VQA task on COCO-QA and MME datasets.
summary: The paper proposed a prompt-aware adapter method that matches both global and local texture features in the prompt to the relevant image contents. Therefore, the adapter can adaptively inject the visual tokens to the semantic space of an LLM according to individual textual prompts. Specifically, the proposed method extracts the CLIP features of the entire prompt which are mapped and appended to the visual features extracted by ViT. Next, the similarity matrix between individual words in the prompt and the features of all visual patches is calculated to derive the weights of the visual patches. The proposed method has been evaluated on COCO-QA and MME datasets and outperforms some recent methods noticeably.
weaknesses: The contributions of this work may not be as significant as the paper claimed.

First of all, given the recent efforts on the native multi-modal foundation model, such as EMU3, the adapter approach may be a temporary solution for MLLM. 

The motivation to come up different ways to inject visual tokens to an LLM according to different prompts may not be that strong. The fundamental way to improve VQA shall be on how to generate detailed and accurate description and representation of image. Using the case in Fig.1 as an example, if MLLM can describe the image as “Along the pool in the backyard of a house, there is a table with 5 drinks on it and 4 chairs”, surely an LLM can conduct VQA and answer “Is there a pool in this image?” and “How may drinks are on the table?” without an adaptive adapter. Please send this photo to OpenAI o1 and check if it can generate a detailed description of the image. So it is arguable that do we really need to change the way to encode visual patches according to each textual prompt.  

The technical contributions are a bit thin which are from line 253 to line 284 and Fig.3, while, most of the paper discusses the background and difference to previous works. The length of the paper could be trimmed, e.g., like Fig.1, the specific image is not informative, perhaps only keeping the same color tokens and different color tokens is enough. 

The experiments verified the proposed schemes are useful to the VQA tasks on two datasets. What about its performance on more complicated tasks such as multi-round conversation w.r.t. images or video understanding.

Regarding to the writing, the paper tends to use some generic and vague expressions, like “understands information”, “to extract the most informative clues”. Btw: if your goal “aims to extract the most informative visual clues”, it does not mean you did “extract the most informative visual clues”, you still need to justify why you achieve this, in what sense it is “informative” (which is a very vague term), in what sense it is the “most” informative, etc.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See Weaknesses.
soundness: 3
strengths: 1. The paper introduces a prompt-aware adapter that adapts visual inputs based on the textual prompt, which is a novel approach compared to traditional, prompt-unaware adapters.
2.  By focusing on the specific visual elements mentioned in the prompt, the proposed adapter reduces the cognitive load on the LLM, leading to more efficient and accurate processing of visual information.
summary: The paper introduces a novel approach to enhance the visual understanding capabilities of Multimodal Large Language Models (MLLMs). The authors propose a prompt-aware adapter that dynamically embeds visual inputs based on the specific focus of the textual prompt. This adapter utilizes both global and local textual features to capture relevant visual cues at different granularity levels, reducing the cognitive load on the Large Language Model (LLM) and improving its ability to interpret visual content. The method is evaluated on various visual question answering tasks, including counting and positional reasoning, demonstrating significant improvements over prompt-unaware baselines.
weaknesses: 1.  The paper presents a method for enhancing visual understanding in MLLMs by focusing on visual details relevant to the textual prompt. While the concept of improving model performance through prompt-related visual cues is not entirely new, with prior works like API[1], AGLA[2], MAVEN[3], and ConrtolMLLM[4] exploring similar ideas. Despite some innovation in the design, the contributions do not seem to offer substantial new value or advancements in the field.
2. There is an inconsistency in the choice of baseline models. Line 353 mentions initializing parameters with MiniGPT-V2, yet Tables 2 and 3 use MiniGPT-4 as the baseline. 
3. Ablation studies, particularly those involving parameter tuning (as shown in Table 7), should ideally be conducted on a validation set to ensure the generalizability of the findings. Although most researchers have currently overlooked this point.
4. Since Table 1 compares earlier models that utilize different visual encoders and language decoders, it would be beneficial to include comparisons with the latest methods that are not contemporaneous, such as LLaVA-Next. Additionally, it would be insightful to see if the authors' proposed method could be applied to other state-of-the-art models like LLaVA-Next or others.
5. It would be valuable to see comparisons with models that possess referring and grounding capabilities, such as Qwen-VL[5], Ferret[6], and ViP-LLaVA[7].
6. Figure 1 could be further refined to make it more concise and intuitive. Improving the figure to be more straightforward would enhance the reader's understanding of the concepts presented in the paper.

[1] Attention Prompting on Image for Large Vision-Language Models.

[2] AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention.

[3] MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model.

[4] ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models.

[5] Qwen-vl: A frontier large vision-language model with versatile abilities.

[6] Ferret: Refer and Ground Anything Anywhere at Any Granularity.

[7] ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts.