Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model
Authors: Haogeng Liu
Quanzeng You
Xiaotian Han
Yongfei Liu
Huaibo Huang
Ran He
Hongxia Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Anchors; Vision Language Learning; Multimodal Large Language Models
Primary area: machine_vision

Abstract: In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays a crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connector has been relatively less explored. In this study, we aim to propose a strong vision-language connector that enables MLLM to simultaneously achieve high accuracy and low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to progressively extract them. Building on these findings, we introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. 
Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds, while simultaneously outperforming baseline methods. This highlights the effectiveness and efficiency of AcFormer.

Visual Anchors Are Strong Information A ggrreatoor For Multimodal Large Language Model
Haogeng Liu Quanzeng You?, Xiaotian Han Yongfei Liu
Huaibo Huang!.2, Ran He Hongxia Yang MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences Schooo of Artifcial Intelligence, University of Chinese Academy of Sciences BByteDance, Inc iuuhogengg20maills ucas. ac. cn, huaibo. huaibo. haabbb.hanngcrip.c.. ia La.ac ac. cn
Abstract
In the realm of Multimodal Large Language Models (MLLMs), vision-language connector plays crucial role to link the pre-trained vision encoders with Large Language Models (LLMs). Despite its importance, the vision-language connec- tor has been relatively less explored. In this study, aim propose strong vision-language connector that enables MILMs to achieve high accuracy while maintain low computation cost. We first reveal the existence of the visual anchors in Vision Transformer and propose a cost-effective search algorithm to extract them. Building these findings, we introduce the Anchor Former (AcFormer), a novel vision-language connector designed to leverage the rich prior knowledge obtained from these visual anchors during pretraining, guiding the aggregation of information. Through extensive experimentation, we demonstrate that the proposed method significantly reduces computational costs by nearly two-thirds compared with baseline, while simultancously outperforming bascline methods. This highlights the effectiveness and efficiency of AcFormer. Codes are available at httpp///gghhbbbbcomm/uuuggggg/g/ Former
Introduction
Multimodal Large Language Models (MLLMs) have emerged as a focal point within contemporary research discourse 47 Prominently showcased by seminal works such LLaVA B3] 34), BLIP-2
(7], Qwen-VLL 4 and Flamingo (1, these models exhibit exceptional efficacy across a broad spec- trum of tasks, spanning from nuanced image description [29] 48 to complex visual reasoning. Their versatility transcends conventional boundaries, finding practical application in quotidian scenarios such smartphone interface design [2I] and consequential real-world decision-making processes
[3] This advancement is attributed to the availability of pre-trained Large Language Models (LLMs)
44 and vision encoders [1I) By utilizing these pre-trained components and introducing a connecting layer between them, it is possible to construct robust MLLMs with only training the lightweight vision- languae connector. This enables the development of MLLMs with the capacity to process visual inputs while retaining the linguistic prowess characteristic of LLMs. These enhanced MLLMs exhibit proficiency across various tasks, including narrative generation, code composition, and addressing complex queries 45
In the construction of above MLLMs, the vision-language connector plays a pivotal role. A funda- mental approach, as demonstrated in LLaVA (34), employs a linear projection layer as the connector. In the enhanced version, VLa 33 the linear projection layer is expanded into multilayer
Conresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

AcFormer (Ours)
visual tokens
MLP
65
257
C-Abstractor
Pooling generate
Pooling
145 tokens at most
145
S77
PR
63
62
Average Accuracy (%)
61
Vision Transformer
60
Entormatoo
aggregasion
59
58
57
56
1.5
2.0
2.5
3.0
3.5
4.5
5.5
Step time (s/step)
Visual Anchors
Figure 1: Comparison of the average normalized accuracy (MMB, TextVQA, GQA). PR means Perceiver Resampler, which utilize the learnable query information aggregator. Our method achieves highest accuracy comparing with the others while maintaining high training speed.
perceptron, enhancing the model's efficacy. Despite achieving notable performance, the large number of visual tokens, which extends the time required for computing attention and other processes (8] indicating potential for optimization to decrease the computation cost. Prior efforts have sought to
address this concern. For instance, BLIP-2 [27] introduces Q-Former and Flamingo []] proposes Perceiver Resampler, both leveraging learnable queries visual information aggregators. This mechanism utilizes the cross-attention between learnable queries and the outputs of visual encoders to effectively reduce the length of the visual sequence, thereby lowering the computation cost. Similarly, Qwen- VL 4 adopts a comparable structure but eliminates self-attentic among the learnable queries. While these rision-languue connectors substantially improve efficiency compared to naive linear projectors, they also exhibit notable decrease accuracy, as detailed in (5]
The primary computational bottleneck in MILMs is the LLM component, where computational costs escalate significantly with the length the input sequence, including both visual and textual tokens. To mitigate the computational cost, a straightforward approach is to reduce the number of input visual tokens with vision-language connector. commonly adopted method is attention pooling, which offers greater flexibility than traditional pooling techniques. This method focuses aggregating information within the visual tokens, necessitating aggregators with high information gathering capabilities. Current attention pooling methods typically use randomly initialized learnable queries as information aggregators. We identify two main drawbacks with this approach: (1) The queries are randomly initialized without prior knowledge for aggregating visual information. They necessitates training on massive datasets (hundreds of millions) to be effective, as showed Table (2) The queries are fixed and invariant to different input images, potentially leading significant information loss and low specificity for uncommon inputs, as illustrated in 122 39].
To address the aforementioned issue, we propose Anchor Former (AcFormer), novel vision- connector that enhances both the accuracy and efficiency of MLLMs compared with baselines. In order to build AcFormer, we identify more effective information aggregators by analyzing the visual features obtained from pre-trained vision encoder from two perspectives: the feature map and the attention map. Our analysis reveals the presence of "visual anchors" within the visual tokens. Fundamentally, the transformers in the neural network aggregate information related to these visual anchors, central to the transformation process. Moreover, the positions of these anchors vary across different images. Despite this variability, we can effectively identify them using the attention matrix to carry out the cost-effective progressively search algorithm. With these observations, propose

---

Anchor Selector, an important part of our AcFormer. The Anchor Selector utilize the progressive search algorithm with respect to the attention map. By this way, effectively extracts visual anchors from the visual tokens generated by the Vision Transformer. And with the visual anchors, AcFormer utilize the naive cross attention to aggregate visual information for generating dense and complete visual representation.
In summary, our contributions can be summarized as follows:
We reveal the existence of visual anchors within the visual tokens generated by pre-trained Vision Transformer, and subsequently propose cost-effective Anchor Selector to effectively extract these visual anchors.
We propose the Anchor Former (AcFormer), a novel vision-language connector designed to improve the accuracy and efficiency of Multimodal Large Language Models (MLLMs) by leveraging the rich prior of information aggregation within visual anchors.
We conduct comprehensive experiments across various vision-language tasks to empirically validate the efficacy of AcFormer.
2 Related Work
2.1
Multimodal Large Language Models
The development of MLLMs has become financially viable due to the utilization of pre-trained
and Large Language Models (LLMs) 44 43] [52]. Spearheaded by initiatives like Flamingo and BLIP-2 [28), the field has witnessed significant advancements Studies like LLaVA and MiniGPT-4 have introduced methodologies such as visual instruction tuning, enabling robust MILLMs capable of understanding human instructions. Additionally, efforts such as Emu and LaVIT 24 have proposed unified frameworks for generation and comprehension, integrating visual decoders and consolidating the training loss of visual and textual inputs. The progression of MLLMs has been supported by the availability of extensive visual- language training datasets 77 33 Innovations like Sphinx [32] and Monkey [3]] have facilitated high-resolution image processing through techniques like sub-image cropping. thus advancing open- source MLLMs. Moreover, Mobile VLM [10] has introduced a compact vision language model suitable for deployment on mobile devices.
vision encoders 41
38
12
34
42
45
21
2.2 Vision-Language Connectors
Vision-language connectors typically employ either direct linear projection (LLaVA) or an informa- tion aggregation module, such as Flamingo, BLIP-2, and C-Abstractor, followed by linear projection. LLaVA, utilizing a simple multi-layer perceptron, effectively aligns visual features with the embed- ding space of Large Language Models (LLMss, but it suffers from high computational costs due to redundant input visual tokens, highlighted by ( BI.IP-2 uses the Q-Former to aggregate visual information and establish robust baselines, while Flamingo employs the Perceiver Resampler. Both architectures leverage the cross cross-attention mechanism to aggregate visual information into Iearnable queries. However, these approaches require extensive data for training and may have limitations in tasks requiring fine-grained visual perception due to the constrained nature of the learned query's ability to capture all visual patterns. In contrast, Honeybee [5] proposes the C-Abstractor and D- Abstractor to address these challenges by introducing spatial priors into the feature representation. Our proposed method, AcFormer, consists of three parts: Anchor Selector, Information Aggregation Module and Linear Projection. While Flamingo and BLIP-2 use learnable queries for information aggregation, C-Abstractor applies a convolution network directly. Our method utilizes visual anchors as information aggregators, generating dense and complete visual representations for input images.
3 Methods
3.1 Preliminaries
For MILMs, their visual encoders are typically off-the-shelf pre-trained Vision Transformers (CLIP). Given input images I 18 xBxxxxXX and Vision Transformer denoted as F(.), the vision feature
3

---

Parch Emb
Layer
Layer
Layer
Layer 12
Layer 15
Layer 18
Layer 21
Layer 25
Layer 24
Patch Emb
Layer
Layer 15
Layer
Layer
Layer 24
Head
Head 2 Head 3
Head
Head 5
Head 6
Hrad 7
Head 8
Head
Head 10
Figure 2: Visualizations of the visual feature map and attention map pertaining to the [CLS] token. Here we select 10 layers in Vision Transformer to show their output. We present the attention maps corresponding to the [CLS] token in the final layer. Notably, special tokens within both the feature map and attention map are identified using red circles. These marked points are referred to as "visual anchors". Details can be found in Section 3.2
R. is obtained as follows:
R. = F(I) R BXNxD
(I)
Here, C, H, and W represent channel, height, and width the input image, respectively, while N and D denote the number and dimension of the image tokens. There is one additional token, the [CLS] token added for the global representation the image in contrastive learning. After obtaining the feature map. two common methods used to combine it with LLM.
The first method utilizes gated cross-attention for modality fusion. Assuming the corresponding input instruction with the image is T N the computation can be expressed as:
T₁ = G(query = T, key = R,, value = RR,),
where Tn represents the hidden states of the LLM and G(-) denotes the gated cross-attention layer.
The other approach involves converting the visual tokens into soft embeddings nd concatenating them with the text embeddings the input of the LLM. Suppose the vision-language connector V-L-Connector,
LM,, Conjat(Proj (o-LCooooooooooooetorrr (R.))) T),
(3)
Where Proj means the Linear Projection and T means the text embedding. LM,,, represents the input embeddings of the L.M.
3.2 Visual Anchors
Within the ViT, the input pixel-level features undergo series of transformations. Understanding how the visual semantic is learned will bring us better insight of building the vision-language connector. To analyse in a more intuitive way, visualize the feature map and attention map.
Given set of Vision Transformer's feature maps, denoted V N where N represents the number of tokens and D signifies the dimension of these tokens, we leverage dimension reduction for visual feature visualization. Initially, we extract all hidden states from the Vision Transformer. Subsequently, employing Principal Component Analysis (PCA) on each individual feature map, we derive low-dimensional features. We select the first three principal components to yield V E RRNxs followed by normalizing the pixel values to the range of 00, 255). To visually represent these features, we construct an image of the input size. Fach patch within the image then encoded using obtained fire-ddimensional representation, effectively encapsulating the corresponding region's

---

value within the image. By this way, we obtain the visualization of the features, as depicted in F (Row 1 and Row 3). Regarding the attention map, we derive by extracting the attention weights associated with the last layer's [CISS] token. These attention scores reflect the significance of respective tokens. We exclude the attention directed towards the [CLS] token itself, yielding the attention map A E HR where H and N denote the number of attention heads and visual tokens. This process is depicted in Figure Row 2 and Row 4).
Upon visual inspection, several noteworthy observations come to light. Specifically, the transfor- mation of visual information exhibits a gradual obscuration of the feature map, accompanied by an increasing activation of specific tokens (depicted as pink and light yellow patches in the first and third lines, respectively). Initially, these activated tokens are just parts of the background, appearing indistinguishable from surrounding elements. However, over time, they progressively differentiate themselves from the background. Nevertheless, this evolutionary process lacks a discernible pattern. Examining the attention map, the [CLS] token, commonly used for aggregate global information of the input image, is anticipated to attend to the most salient regions. Paradoxically, the attention map of the [CLS] token predominantly focuses on limited subset of tokens. We calculate the overlap among the activated tokens in feature map and the salient regions in attention map. The ratio reaches up to 69.38% (500 images are sampled for calculation), conjecturing that this alignment is not coincidental. For further validation, we visualize the pre- pre-trained MLLMs text generation attention matrix (text visual tokens), result can be found in Figure 5
Based on above observation, we name these tokens "visual anchors" and assume them serving as pivotal points for information aggregation during the transformation of visual features. While the [CLS] token indeed integrates visual information, its reliance alone proves insufficient, necessitating the involvement of other visual anchors in conveying information to the [CLS] token. Consequently, this process facilitates the extraction of meaningful representations from the image.
Anchor Former
Large I. Language Model
Flexibality Usilized Visual Archors prive informtios
Embedding
AcFormer
MLP
Anchor Selector
Information Aguregator
Tokenize
Food Forward
Tokke
A27 201337,,255 134 30030055
Deseribe the image
Cross Atteation
Vision Feature
Parcriver Recample
Flesibility Uillived x sandoos quary begining
Tokes rndt
Anchor Selector
300 304 30655)
Information Aggregation Module
Tokan
Vision Encoder
Lincar Projector
Fleubility x Prioe Utilized Y
Vision-l Language Connectors
Figure 3: Visualization of Anchor Former (AcFormer). We propose our token selection algorithm code in detail at Section
3.3 Anchor Former
As illustrated above, the vision information is aggregated through visual anchors. Similarly, within Multimodal Large Language Models (MLLMs), structures such as Q-Former and Perceiver Resampler also leverage information aggregation modules. However, these models use learnable queries as Information Aggregator to extract information from the visual feature map produced by a pre-trained Vision Transformer. In this method, the same queries are used for all images, which can lead to two issues. First, their effectiveness requires extensive datasets for training as demonstrated Table 5
Second, they may lead to significant information loss, as illustrated in [5] 39 22
5

---

To address the aforementioned issues, we propose Anchor Forner (AcFormer), which consists of Anchor Selector, Information Aggregation Module and Linear Projector. We show a rough view of Anchor Former in Figure 3 Our approach integrates insights from visual anchors with the established framework of the Perceiver Resampler. A key innovation in our method the use of visual anchors associated with each image Information Aggregator for aggregating visual information. The effectiivnness of our approach relies on the selection of these salient points. This visual anchors allows for more precise and specific information extraction, improving both the accuracy and efficiency of the model.
Anchor Selector: To avoid additional computation, we leverage the attention map of the [CLS] token for visual anchor selection. We introduce a progressive search algorithm to construct the Anchor Selector. Assuming the attention map is A E R where H represents the number of attention heads and N - denotes the number of visual tokens ([CLS) excluded). Let the token index list be TL, initially containing only the index O, corresponding to the [CLS] token, an essential anchor. Suppose we still need Ty tokens in addition to the [CLS] token. We select these tokens head by head, assuming each head will provide by tokens. For each head,, we first sort the indices of the visual tokens based on their attention scores. Then, select the top и the chosen token is already in TL. we choose the next token in the sorted order until we have the required number of unique tokens. We provide detailed algorithm in Figure 7
Tx tokens from this sorted list.
Information Aggregation Module. In our approach, we employ selected visual anchors as In- formation Aggregator, combined with a cross-attention module to aggregate information. Let the Information Aggregator be denoted as IA E R(Xx+1)xD and the origin visual tokens as R. E R xD Our proposed model, Information Aggregation Module, is a bidirectional transformer encoder. We denote the cross attention module as Attn and the feedforward module FF. Let LN represent layer normalization. For a single layer in the model, the operations are as follows:
Aout IA + Aun(query=LN(IA), key=LN(R.), value=LN(R.)),,
H A out + (A.t)((A
Where H means the hidden states. We use H. to represent the last hidden states. Let T represents text embedding. We use the Proj to represent the Multilayer Perception. We obtain the final multimodal input embeddings for LLM as bellow,
LM = (Poo(()))) T)-
Experiments
4.1 Settings
Benchmarks. We employ nine distinct benchmarks to comprehensively assess the overall efficacy of our proposed method. The specifics of these benchmarks delineated in Table Notably, in experimental setup, as we reduce the number of image tokens, our focus is primarily directed towards enhancing visual perception capabilities. As result, we pay particular attention benchmarks such as Tex which challenged the model's fine-grained visual perception ability [11
Implementation details. In our experimental setup, we utilize 7B and 13B Vicuna-v1.5 as Large Language Models (LLMMs) (.. The CLIP ViT-L/14 model, pre-trained with a resolution of 336, serves as our vision encoder. We select the last but one layer's 's output from the vision encoder as our vision feature. For Anchor Former, we configure it with 6 layers and hidden dimension of 512, employing 8 attention heads, each with a dimension of 64. The feedforward module utilize 2048 as the hidden dimension. Regarding the training dataset, we leverage the dataset utilized in LLaVA-1.5 [33, with 558k samples for pre- pre-training and 665k samples for instruction tuning. Our experimentation encompasses the evaluation of various vision- lision-language connectors, including the Perceiver Resampler, Anchor Former, pooling, C-Abstractor and utilizing pooled tokens as queries for the Perceiver Resampler. To maintain consistency, we construct our model using the official code provided by V... Specifically, we adjust the pre-training initial learning rate from le -3 5e Other configuration is the same with origin VA-15.

---

Table 1: Results on benchmark designed for MLLMs. V-T Num means the visual tokens number. V-T Num influences the computation cost that the bigger the V-T Num the heavier the computation cost is. Speed here means the relative pre-training speed with respect to LLaVA-1.5.
Model
I.IM
Connector
V-T Num
Res
POPE:
MMB
MM
Speed (1)
Approaches using 7B Large Language Models
MiniGPT-4 [53]
Vicuna-7B LLaMA2-7B
Resampler
32
224
72.2
726.0
24.3
22.1
mPLUG-OW12 46
Resampler
32
224
1243.4
49.4
InstreetLLP
LLaMA2-7B
Q-Former
32
224
78.9
36.0
26.2
ILaVA (v]) M4
11aMA-7B
Linear
257
224
67.7
717.5
38.7
LLaMA-AdapterV2 16
LLaMA2-7B
LLaMA-Adapeer
257
224
1221.6
41.0
31,4
Shikra [6]
Vicuna-7B
Lincar
257
224
58.8
Qwen-VLJJ
Qwen-7B
Resampler
256
448
38.2
Qwen Vha
Qwen-7B
Resampler
256
448
1845.3
60.6
LLaVA-1.5 3
Vicuna-7B
Linear
577
336
85.9
1794.6
64.3
30,5
1.00x
Ours
Vicuna-7B
AcFormer
145
336
86.4
1846.1
68.4
30.3
2.23x
Approaches using 13B Large Language Models
MiniGPT-4 [53
Vicuna-13B
Resampler
32
224
1158.7
24.4
InstructBLIP
Vicuna- 13B
Q- Q-Former
32
224
78.9
1504.6
25.6
BLIP-2
Vicuna- 13B
Q-Former
32
224
85.3
22.4
LLaVA-1.5
3
Vicuna-13B
Linear
577
336
85.9
1826.7
67.7
35.4
1.00x
Ours
Vicuna-13B
AcFormer
145
336
86.1
1870.0
69.2
34.1
2.30
4.2 Main Results
We present the main results in Tables I and 2 organized by benchmark type. Upon observation of these tables. it becomes apparent that our model achieves robust performance despite being trained with limited data and visual tokens. Notably, even with only 145 or 257 tokens, our model achieves performance comparable to that of the original LLaVA-1.5 model, which utilizes 577 visual tokens as input. This performance holds across various benchmarks, including those that require high-level visual perception (e.g., VisWiz, VeX and those that assess overall capability (e.g.. MME, GQA),
However. should be noted that although our model overall effective, it performs slightly worse than 11.15 on certain benchmarks such as GQA and VQAv2. Given that our method only applies significantly fewer visual tokens (less than half), the slightly performance drop meets expectations. Nevertheless, the performance gap is relatively small and can be considered marginal in light of the substantial increase speed.
Table 2: Results on General VQA tasks. V-T Num means the visual tokens number. V-T Num influences the computation cost that the bigger the V-T Num the heavier the computation cost is. Speed here means the relative pre-training speed with respect to LL..
Model
LLM
Connector
V- Num
Res
TeaVQA
GQA
VQA2
VisWiz
SQA,.g
Speed (t)
Approaches using
Large Language Models
Instrect11.IPDD
MM2-7BB
Q-Former
32
224
49.2
34.5
60.5
Shikra 6
Vicuna 7B
Linear
257
224
77.4
IDEFICS.9B
LLaMA-7B
Cross Attn
257
38.4
50.9
35.5
Qwen V1
Qwen 7B
Resampler
256
448
59.3
78.8
35.2
67.1
Qwen-VL hat
Qwen-7B
Resampler
256
57.5
78.2
38.9
68.2
I.I.VA-1.5
33
Vicuna-7B
Linear
577
336
58.2
62.0
78.5
50.0
66.8
1.00x
Ours
Vicuna-7B
AcFormer
257
336
58.2
61.2
78.4
52.8
69.4
1.65x
Approaches using 13B Large Language Models
InstructBLIP
Vicuna-138B
Q-Former
49.5
33.4
63.1
BLIP-2
Vicuna-13B
Q-Former
32
224
41.0
41.0
19.5
61.0
LLaVA-1.5
34
Vicuna- 13B
Lincar
577
336
61.2
63.3
80.0
53.6
71.6
1.00x
Ours
Vicuna-133
AcFormer
257
336
61.3
63.0
79.8
53.7
71.8
1.60x
4.3 Ablation Results
We mainly compare different visual Connectors. To facilitate understanding, we provide definitions for some terms. Pooling denotes direct pooling of visual token. Pooling-PP PR employs the pooled
7

---

Table Ablation studies. "Pooling" denotes direct pooling of visual token. "Pooling-PR" employs the pooled tokens as queries for the Perceiver Resampler. "Random-PR" means the Perceiver Resampler using randomly selected tokens from the vision feature map as query. "PR" refers the Perceiver Resampler using learnable queries. "AcFormer" represents our proposed Anchor Former. The configuration of the C-Abstractor follows Honeybee
15]
V-T Num means the visual tokens
number.
Model
LLM
Connector
V-T Num.
TextVQA
GQA
MMB
MME
Vicuna-7B Vicuna-7B BB
Pooling
65
53.4
59.8
66.8
1734.0
Pooling-PR
65
53.9
60.0
66.8
1728.9
LLVVA-1.5
Vicuna-71 7B
Random-PR
65
53.9
59.1
66.9
1728.7
Vicuna-17 7B
PR
65
51.0
56.1
63.2
1702.8
Vicuna-7B
C-Abstractor
65
52.8
59.0
67.0
1743.3
Vicuna-7B
AcFormer
65
56.1
59.2
67.3
1744.2
Vicuna-7B
Pooling
145
55.1
60.9
68.0
1791.4
Vieuna-78 7B
Pooling-PR
145
54.7
60.9
68.0
1759.1
Vicuna-78
Random-PR
145
54.6
59.7
67.0
1772.7
LLaVA-1.5
Vicuna-17 7B
PR
145
52.1
56.4
65.4
1720.8
Vicuna-T7
C-Abstractor
145
53.4
60.2
67.8
1775.4
Vicuna-17 7B
AcFormer
145
58.0
61.3
68.4
1846.1
Vicuna- 7B
PR
257
52.3
56.8
65.7
1735.9
LLaVA-1.5
Vicuna-7B
C-Abstractor
53.7
60.8
68.3
1790.0
Vicuna-7B
AcFormer
257
58.2
61.2
68.3
1848.8
Vicuna- 13B
PR
145
53.4
56.9
64.7
1749.3
LLaVA-1.5
Vicuna- 13B
C-Abstractor
145
58.5
62.1
68.8
1823.6
Vicuna-138
AcFormer
145
60.7
62.8
69.2
1869.3
Table 4: Ablation studies whether to directly use the selected tokens input.
Model
L.M
Connector
V-T Num.
TextVQa
GQA
MMB
MME
Vicuna-7B
Top-P
145
56.3
60.8
68.2
1798.8
LLaVA-155
Vicuna-7B
E-VIT
146
57.1
61.0
68.3
1808.4
Vicuna-7B
AcFormer
58.0
61.3
68.4
1846.1
tokens as queries for the Perceiver Resampler. Random-PR means the Perceiver Resampler using randomly selected tokens from the vision feature map as query. PR refers the commonly used Perceiver Resampler. Our experimental findings validate the efficacy of our model from multiple perspectives. A comparison between PR and AcFormer indicates that the observed enhancement does not stem solely from an increase in trainable parameters. Additionally, comparison between AcFormer and Random-PP underscores the critical role of Anehor Selector in model performance. Furthermore, our evaluation of AcFormer against C-Abstractor reveals that the significance of inductive bias diminishes, as spatial coherence can be effectively maintained through cross-attentie mechanisms within the Perceiver Resampler.
C-Abstractor. We trained the model with training data from LLaVA-1.5. Our findings, as presented in Table indicate that the C-Abstractor method achieves comparable performance to AcFormer across most benchmarks. However, in tasks such as TextVQA, which necessitates high-level visual perception (often demanding fine-graine ena-gnaieed visual analysis), C-Abstractor exhibits inferior performance. This empirical evidence underscores the efficacy of our proposed AcFormer.
Pooling. One immediate consideration is aggregate visual tokens based their spatial positions by pooling and combine them with the [CLS] token to form the input visual features. Our empirical investigation reveals that direct pooling emerges as a potent technique for compressing visual information. However, it fails in TexiVQA, leading to around 3 points drop.
Pooling-PR. Given our direct utilization pooled tokens within Large Language Models, a concern arises regarding whether the observed degradation stems from the reduction in trainable parameters within the Anchor Former. To address this concern, we conducted additional experiments wherein the pooled tokens served as queries for the Perceiver Resampler, termed Pooling-PR. Examination

---

of the results in the table reveals that this approach yielded even poorer performance compared to directly inputting the pooled tokens.
Random-PR. While our method demonstrates superior performance compared with other ap- proaches, lingering question pertains whether this improvement indeed stems from the Anchor Former. To address the concern, we randomly select tokens from the vision feature map as the Information Aggregator. The empirical results presented in Table corroboratt the efficacy of our proposed anchor selection method, substantiating its discernible impact on model performance.
PR. While substantiated the effectiveness of AcFormer, an unresolved question pertains to its superiority over the commonly used Perceiver Resampler, which employs randomly initiallized learnable queries for visual information aggregation. Examination of the table reveals that the Perceiver Resampler exhibits the poorest performance among the aforementioned methods. This observation underscores the diminished performance of the structure under conditions of limited training data.
Anehor Direct-in. A related study [20 employ Top-K selection methodology for token com- pression. This approach shares similar token selection method with ours and involves the direct utilization of selected tokens as representations (without further cross attention) for visual infor- mation. However, it primarily caters classification tasks, which prioritize global understanding, potentially rendering it less suitable for tasks requiring nuanced comprehension. Noteworthy the proposition by the authors of Haurum [20] regarding EVIT. They employ the selected tokens and incorporate pooling mechanisms for the remaining tokens as input. The findings, as detailed in Table 4 empirically underscore the significance of retaining unselected tokens, they still encapsulate valuable information.
Table 5: Ablation studies on the visual connector when scaling up the training data.
Model
LLM
Connector
V-T Num.
TextVQA
GQA
OKVQA
VQAr,
VizWiz
MME
Pretrain Dataset: 60M image-text pairs from LAION-I1SM, COYO, LAION COCO
Instruction Finetuning Lataset:
Op OpenLLaMA 3B
PR AcFormer
145
35.03
54.35
54.14 55.15
70.04 72.76
31.16
1592.7
LLaVA-1.5
145
35.89
55.45
33.88
1622.3
Pretrain Dataset: 60M image-lexi pairs from LAION-IIMM, COYO, LATON COCO
Instruction Finetuning Cauldet: (roughly 1.8M)
LLAVA-1.5
OpcnLLaMA-33 Ope 3B
PR
145
38.76
44.98
49.66
70.83
32.87
1502.5
AcFormer
145
40.49
45.67
50.57
73.01
32.97
1523.1
4.4 Scaling Up the Training Dataset
We conducted data scaling experiment to further evaluating our proposed method's performance on large-scale data scene. Considering the expensive training cost, we replaced the Large Language Model with OpenLLaMA-3 pre-training, sampled from Laion 15M, coy0-238M, and laion-coco 100M datasets. For instruction fine-tuning, we utilized two datasets, covering LLaVA-660 VA-665k and Cauldron [26]. Notably, Cauldron (1.8) is a larger instruction finetuning dataset than LLaVA-6655. (0.66M) for MLLMs.
[17]. Approximately 60 million image-text pairs were employed for
Our focus summarized Table 5 consistently demonstrate the superiority of AcFormer over the Perceiver Resampler. This illustrate that our method only works well for the scene of limited data but also for larger data (both pre-train dataset and instruction tuning dataset).
comparing Perceiver Resampler with proposed AcFormer. The results,
Conclusions
In this study, we reveal the existence of the visual anchors and present the hypothesis that visual anchors are strong visual information aggregators. With the observation, propose the Anchor Former, an effective vision-language connector, Notably, Anchor Former distinguishes itself from the conventional information aggregation methods (c.g., Q-Former and Perceiver Resampler) by adopting

---

visual anchors as Information Aggregator. To build Anchor Former, we present the progressive search algorithm to effectively extract the visual anchors. Extensive experiments on different benchmarks consistently underscores the efficacy of Anchor Former in improving the models' accuracy while simultaneously removing the redundant visual tokens in MI.LMs.
Acknowledgements
This research is partially funded by National Natural Science Foundation of China (Grant No U21B2045, U20A20223), Beijing Nova Program (20230484276), and Youth Innovation Promotion Association CAS (Grant No. 2022132).
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick. Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binikowsi.. Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 23716-23736. Curran Associates, Inc., 2022.
[2] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick. and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision
(ICCV), 2015.
[3] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openftamingo: An open- framework for training
large autoregressive vision-language models. arXiv preprint arXiv:2308.01
2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[5] Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enhanced
projector for multimodal Ilm. arXiv preprint arXiv:2312.06744 2023.
[6] Keqin Chen, Zhao Zhang. Weili Zeng. Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal I1m's 's referential dialogue magic, 2023.
[7] Lin Chen, Jinsong Li, Xiaoyi Dong. Pan Zhang. Conghui He, Jiaqi Wang. Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi- modal models with better captions, 2023.
[8] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models, 2024.
[9] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[10] Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang. Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilcylm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023.
[11] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpon vision- Asionnlanguage models with instruction tuning, 2023.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhom, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1:Xiv:2010.11940
2020.
[13] Danny Driess, Fei Xia, Mehdi SM Sajjadi. Corey Lynch, Aakanksha Chowdhery. Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, ct al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03377 2023.
[14] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulci Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li. Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: comprehensive evaluation benchmark for multimodal large language models, 2023.
[15] Chaoyou Fu, Poixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
[16] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He. Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapier v2: Parameter-efficient visual instruction model, 2023.
[17] Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023.
[18] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018
[19] Xiaotian Han, Quanzeng You, Yongfei Liu, Wentao Chen, Huangjie Zheng, Khalil Mrini, Xudong Lin, Yigi Wang, Bohan Zhai, Jianbo Yuan, et al. Intimm-eval: Complex open endeeddedd reasoning evaluation for multi- modal large language models. arXiv e-prints, pages arXiv-233 2311, 2023.
[20] Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 773-783, 2023.
[21] Wenyi Hong. Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.088 08914. 2023.
11

---

[22] Yi-Xin Huang. Hou-I Liu, Hong-Han Shuai, and Wen-Huang Cheng. Dq-detr: Detr with dynamic query for tiny object detection. arXiv preprint arXiv:2404.03507, 2024.
[23] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709. 2019.
[24] Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song. Xiaoqiang Lei, et al. Unified language-vision pretraining with dynamic discrete visual tokenization. arXiv preprint 2309 230,0069 2023.
[25] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karameheti, Alexander Rush, Douwe Kicla, et al. Obelies: An open web-scale filtered dataset of interlcaved image-text documents. Advances in Neural Information Processing Systems, 36, 2024.
[26] Hugo Laurençon, Léo Tronchon, Matthicu Cord, and Victor Sanh. What matters when building vision- language models?, 2024.
[27] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-iman -mage pre- training with frozen image encoders and large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19730-19742. PMLR, 23-29 Jul 2023.
[28] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-iman pre-training with frozen image encoders and large language models. In International conference on machine learning.
pages 19730-19742. PMLR, 2023.
[29] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888 12900. PMLR, 2022.
[30] Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-R Ji-Rong Wen. Evaluating object ballucination in large vision-languae models. arXiv preprint arXiv:2305. 10355, 2023.
[31] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi modtll models. arXiv preprint 2rXhv:2311.066/ 2023.
[32] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wengi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi modal large language models. arXiv preprint MxXi:2331.005722 2023.
[33] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. 2023.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[35] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all around player? arXiv preprint N3Ki:2337.0028// 2023.
[36] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurlPS), 2022.
[37] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattemn recognition, pages 3195-3204, 2019.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
[39] Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang. Yihao Chen, Yuda Xiong, Hao Zhang, Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024.
[40] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317 8326, 2019.
[41] Quan Sun. Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.1 203.:23333, 2023.
[42] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint MrXiv:2307.05222 2023.
[43] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint vrXiv:2403008299 2024.
12

---

[44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288, 2023.
[45] Weihan Wang, Qingsong Lv. Wenmeng Yu. Wenyi Hong. Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, al. Cogvlm: Visual expert for pretrained language models. arXiv preprint
arXiv:2311.0303
2023.
[46] Qinghao Ye. Haiyang Xu, Guohai Xu. Jiabo Ye. Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengeheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Ji Zhang. Fei Huang, and Jingren Zhou. mplug-owl: Modularization empowers large language models with multimodality, 2024.
[47] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models, 2024.
[48] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Scyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205 wwx.i:2200001917, 2022.
[49] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and
Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023.
[50] Weihao Yu, Zhengyuan Yang. Linjie Li. Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint
OmXiv:2338.002455 2023.
[51] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Qiao. Llama- -aana-rdppter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint 1rXiv:2303.16111
2023.
[52] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging Ilm-as-a-judge with mt-bench and chatbot arena, 2023.
[53] Deyao Zhu, Jun Chen, Xiaogian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
13

---

A Visualization of The Feature From OpenAI CLIP
We have shown the visualization of the feature map and attention map of EVA- in the main text. To validate whether other models (OpenAI-CLIP) also has the same characteristic, we offer analogous visualizations in Figure 4 revealing comparable phenomena. While diverging from EVA- CLIP-L in certain aspects, OpenAT-CLIP-L nevertheless manifests certain anchor points. These shows that our observation is not isolated case.
Patch
Layer
Layer
Layer
Layer
Layer
Layer
Layer 21
Layur
Layur
Head
Head
Iead
Head
Patch Fash
Layer
Layer
Laver
Layer 12
Tayer 15
Layer 18
Lay er 21
Layor 21
Tayer 24
Head
Head 2
Head
Head 4
Head
Head 6
Head
Head
Head 9
Head
Figure 4: Visualization of the Openai CLIP. Figure shows the visualization of EVA CLIP. We show that the phenomenon happens in different CLIP.
B Visualization of The Attention Map From The Pre-trained MLLMs
To further substantiate hypothesis, conducted visualizations of the attention matrix of the pre- trained Multimodal Language Models (MILMss). We believe directly obtaining the generated text's attention will give better understanding of the visual anchors. Recognizing the potential generation of multiple tokens simultaneously, we adopted a methodology akin to L.LVA''s approach, constraining the model to provide single-word phrase responses for visualization of the attention mechanism on the answer (key word), Given the multilayered architecture of MLLMs, we specifically chose middle layers for visualization, prior work [8] has indicated their pivotal role in comprehending visual signals.
Two types of MLLMs considered, exemplified by Flamingo and LLaVA. As Flamingo segregates attention between text-only and image-text modalities, we opted for this model for visualization purposes. is noteworthy that excluded the Perceiver Resampler and directly leveraged all visual tokens to discern their relative importance. The visualization accessible Figure 5
C More Samples For Visual Anchor Visualization
We provide more visualization samples in Figure 6 to demonstrate that the phenomenon of vision anchors exists both complex real-world scenes and animated scenes.
D Detailed Algorithm
We have provided rough description in Figure 3 token selection in Figure The complete code is available in the supplementary material. We use a top-k search to select visual tokens. Specifically, assume the attention matrix A HxINN where H is the number of heads and N is the number of image patches. We traverse each head of the attention matrix, and for each head, we select T-1 tokens based on the sorted indices and already selected sequence avoid duplication. Here, T represents the total number of desired tokens.
Here, we present the detailed Python code for
H
14

---

Figure 5: Visualization of the attention map from the pre-trained Flamingo model (Removing the Perceiver Resampler). The attention map is the generated text's attending to the corresponding image patches.
Layer
Layer
Layer
Layer
Head
Head 3
Head 6
Head
Head
Patch Emih
Layer 5
6
Layer
Layer 12
Layer 15
Layer 18
Layer 21
Layer 23
Layer 24
Head
Ilead
Ilead
Tlead
Head 5
Head
[lead
Head
Head
Head 10
Putch Emnh
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Iliedd
Head
Ileed 3
Ieedd
lllcdd 5
Ilaad
Iledd
Ieedd
Hlead
Hlcd 10
Patch Emb
Laver
Layer
Layer
Layor 12
Layer
Layer 1s
Layer 21
Layor 23
Layer 24
Head
Head
Head
Heed
Head
Head
Hled
Head
Head 9
Head 10
Figure More samples for visualization of the visual anchors.
15

---

Algorithm: Anchor Selection
Language Model
Input:
Visual Feature Map VBB,N,
Visual Attention Map (B, H,N,
Finhalding
MLP
Token Number T
Caculation:
Tokenize
selected anehor None
Feod Forward
Per head pum in(((E-))/H) H[CLS] choosen default
image
for in range(B]):
Attesion
max indice - [0]
for in range(HH:
ump_atte 0,
Selecter
ind sorted Argsott-g alin)
tmp tmp_res- rmp_res-setinnd set(ind count
someed|-Pe Per head num:] indice)
Vision
Encoder
while len(mmp ((++)) head num + I):
tmp_res res set(ind sore head nurn-coutt]] max indice)
count count
max indices sorted(list(impp rre))
if selected anchor None:
selected anchor borch.cat((selecter anchor, VÍji], max indices, :D), dim-D)
cbs:
selected anchor V[[i), max indices, :]
selected anchor
Figure Detailed code for the anchor selector within Anchor Former (AcFormer).
Table 6: Details the training time. Pt Bz means the pre-train batch size, And IFT Bz means the instruction finetune batch size.
Methods
LLM
Training resource
Visual Token Num
Pt Bz
IFT Bz
Pt time
IFT time
Pooling
Vicuna-77 7B
8 A100(80G)
256
128
9h25m
Pooling-PR
Vicuna-77 7B
A100(80G)
128
42m
9h51m
Random-PR
Vicuna-7B
A100(80)
128
42m
9h52m
Origin-PR
Vicuna-78 7B
8 A100SSOG)
128
42m
9h52m
C-Abstractor
Vicuna-T7 7B
8 A100(80G)
65
256
128
41m
9h51m
AcFormer
Vicuna-77 7B
8 A100(80G)
65
256
128
42m
9h52m
Pooling
Vicuna-77 7B
8 A100(80G)
145
256
128
1h03m
10h13m
Pooling-PR
Vicuna-77 7B
8 A100(80G)
145
256
128
1h20m
10h5 mm
Random-PR
Vicuna-7B
8 A100880)
145
256
128
1h19m
10h50m
Origin-PR
Vicuna-T7 7B
A110(800)
128
1h20m
10h52m
C-Abstractor
Vicuna-T7 7B
A100(800)
128
1h18m
10h50m
AcFormer
Vicuna-
A100(80G)
128
1h20m
10h12m
Origin-PR
Vicuna-71 7B
8 A100(80G)
257
256
128
1h50m
10h29m
C-Abstractor
Vicuna-7B
8 A100(80G)
257
256
128
1h49m
10h25m
AcHormer
Vicuna-7B
8 A100880G)
257
256
128
1h50m
10h30m
Origin-PR
Vicuna- 138
8 A1OIK80G)
145
256
128
2h04m
17h31m
C-Abstractor
Vicuna- 13B
8 A100K80G)
145
256
128
2h02m
17h29m
AcFormer
Vicuna-133
8 A100S80G)
145
256
128
2h04m
17h32m
E
Training Resources
We provide details for training resource for main experiment in Table Notably, for the data scaling experiment, we utilize 16 Nvidia A100 (80G). It takes roughly 28 hours for pre-train and hours for IFT.
F
Detailed Description of The Benchmarks
There exist numerous benchmarks for assessing the proficiency of Multi-Modal Large Language Models (MLLMs),, each imbued with its own inherent biases. For instance, benchmarks such as OK- VQA (37 primarily concentrate appraising the model's pre-existing knowledge base. Conversely,
16

---

Table 7: Details on the chosen benchmark.
Benchmark
Description of the task
Metric
TexIVQA
40
QAs about text image (Visual Perception)
VQA score (t)
VizWiz VQA
[8]
QAs about image from blinds (Visual Perception)
VQA score (1)
GQA 23
QAs of real world comprehension and complex reasoning
EM (t)
VQAv2
2
QAs require vision, language and prior world knowledge
VQA score (1)
POPE
QAs for Object Hallucination evaluation
F1 Score (t)
Sci-QA(Img)
36
QAs about Science
Accuracy (1)
MME 14
Comprehensive Evaluation Benchmark for MILMs
Accuracy (f)
MMbench
33
Comprehensive Evaluation Benchmark for MLLMS
Accuracy (1)
MM-Vet
49
Integrated Capabilities Benchmark for MLLMs
GPT-4 score(
benchmarks like Te 40 and VizWiz-VQA [8] scrutinize the models' prowess in visual perception. Notably, a plethora newly introduced benchmarks tailored specifically for MLLMs have surfaced. MME [1]] and MMBench [35] stand out comprehensive benchmarks aimed at gauging the overall performance of MLLMs. However, they mandate MLLMs to furnish responses in binary multiple-choice formats. POPE (30] has been devised to assess MLLMs' propensity for hallucination. MM-VET (50) and InfIMM [9 scrutinize the open-ended question-answering capability of MILMs, aided by the assistance of GPT-4.
G Limitations
Although we have conducted extensive experiments, there are still aspects requiring further investiga- tion. For instance, the utilization of larger training datasets with corresponding larger models remains unexplored due to resource constraints in our experiments. Additionally, more theoretical analysis are needed for better elucidating the underlying reasons for the emergence of these visual anchors.
H Broader Impacts
Our model, despite its capabilities, may encounter certain risks. As is built upon LLaMA, Vicuna, and CLIP, it inherits some issues associated with large language models (LLMs) and vision encoders. One significant risk is hallucination, where the model might generate content that contradicts the facts. This poses a concern, especially when applied in critical fields such as medicine. Additionally, biases present in the LLM and vision encoder (CLIP) could be transferred to model, potentially resulting in biased outputs. Lastly, while energy consumption is not a primary concern due to the smaller pretraining dataset used, it may become issue when scaling up the pretraining dataset increasing the model size.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We give main claims accurately in abstract and instruction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations at section G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
18

---

This is an observational and experimentally validated study, with no theoretical argumentation involved.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This work can be fully reproducted with the information from the paper. We provide the main code in the appendix in detail.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our model built based on the open-sourced LLaVA, so our provided code in Figure iss enough for reproduction of the method.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle pwwppppguuuiieeeeep http:///uuiiees/deeuuuubiiis
(CodeSubiisss wwtauubmmsssooni ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the implementation details in paragraph 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We experiment on many different benchmarks to eliminate statistical error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the training resources in section Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: There is no bias or other special about the research.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide Broader Impacts in section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
Our proposed method is easy for reproduct that we will not release the model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow the license of the data and code used in our experiment.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There not any asset in the paper.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "SqRv4eYFsk", "reviewer": "Reviewer_U61n", "summary": "This paper proposes a way to select visual tokens, e.g. token pruning, by using attention map scores. This reduces the number of tokens needed in the network which saves compute costs. The method is evaluated on multiple datasets and shows reduced compute while maintaining performance.", "questions": "The compute cost overall seems to go down, but I'm curious about the implementation of the anchor selection. How long does that part take? How optimized is that algorithm?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is fairly well written and the experiments are good and show the benefit from the approach.", "weaknesses": "The approach isn't especially novel, many works have explored token pruning, token learning, token reduction before. While this approach is different from the previous ones, the differences are small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your hard work in reviewing! We will address your concern below.\n\n**Q1:** About the novelty.\n\n**R1:** \nThank you for highlighting this issue. Though there are indeed methods for token reduction or token pruning, such as CAbstractor and Perceiver Resampler, ours are different with them in many aspects.\n1. Motivation. For C-Abstractor, it mainly focus on maintaining the locality information. For Perceiver Resampler, it use the learnable queries to aggregate visual information. However, in our work, we attempt to propose image-specific information aggregation.\n2. Method. In C-Abstractor, the output tokens of vision transformer are aggregated with the conv pooling. In perceiver Resampler, the output tokens of vision transformer are aggregated with learnable queries (all of the image shares the same queries). While in Acformer, we use the anchors as queries and use all of the visual tokens as keys and values to carry out cross attention. By this way, it better maintain important info.\n3. Performance. Thanks to the visual anchors, it enables us to aggregate visual infromation with nearly no performance loss, which is evident Table below (parts of Table 3 from our paper).\n\n| Connector      | TextVQA($\\uparrow$)      | GQA($\\uparrow$)      | MMB($\\uparrow$) |MME($\\uparrow$) |\n| :--------- |:--------- | :----------: | :----------: | :----------: |\n| C-Abstractor  |53.4   |60.2  |67.8 |1775.4|\n| Perceiver Resampler  |52.1   |56.4  |65.4 |1720.8|\n| Anchor Former |58.0    |61.3|68.4|1846.1|\n\nThrough the analyse above, we believe this work provides **valuable insights into aligning visual and language modalities, and it offers a foundation for further exploration into the interpretability of the vision-language integration process**.\n\n**Q2:** About the implementation of anchor selection.\n\n**R2:** Thanks for pointing out the problem for understanding. We employ the top-p method for anchor selection. Specifically, we consider the attention map of the [CLS] token in the corresponding layer of the Vision Transformer (the penultimate layer in our implementation), denoted as $A \\in \\mathbb{R}^{H \\times 1 \\times N}$, where $H$ represents the number of heads and $N$ the number of visual tokens (excluding the [CLS] token itself). Our goal is to select $M$ tokens. To achieve this, we calculate the number of tokens each head should contribute using $p = (M-1)/H$ (we include the [CLS] token by default, so we subtract one). For each head, we sort the token indices based on $A$ and select the top-p indices for our results. In cases of duplication, we iteratively select the top-(p+1) indices until the desired number of tokens is achieved. By this way, we finally obtain the chosen visual anchors. \n\nFor example, if we aim to select 145 tokens and we are using the OpenAI-CLIP-L model, which has 16 attention heads, we first calculate each head's token contribution as (145-1)/16=9. We then initialize our selected tokens index list with the [CLS] token, res=[0]. For head 0, assuming the sorted token index with the attention map is [1,2,4,6,21,64,33,78,24,98,23,99...], then we append the top-8 index into the result list -> res = [0,1,2,4,6,21,64,33,78,24]. We then process head 1, assuming the sorted index with the attention map is [1,24,2,4,6,98,23,99,45,32,75,34,38,70...], as \"1,2,4,6\" are already in res, to avoid duplication, we append [24,98,23,99,45,32,75,34,38] into the result list. After processing each head, we have a list with 145 unique tokens' index. Finally, we sort this list and fetch the anchors according to the indext.\n\n**Q3:** About the cost of anchor selection.\n\n**R3:** \nThanks for your concern on the cost of anchor selection. In our experiment, we employ the OpenAI CLIP-L-336 model, which features 16 attention heads in its Vision Transformer and processes 576 tokens (excluding the [CLS] token). The anchor selection primarily involves a single list sort of length 576. After evaluating this anchor selection process over 1000 iterations, we observe an average execution time of 7.5 ms (800 ms for one token generation). The baseline method takes 1300 ms to generate one token. Although our method adds an extra 7.5 ms for token selection, it reduces attention computation time by nearly 500 ms, highlighting its efficiency and lightweight nature. Despite the additional 7.5 ms from anchor selection, the overall attention cost is reduced by approximately 500 ms, leading to a net decrease in total computation time.\n| Connector      |One token generation time|Anchor selection time|\n| :--------- | :----------: | :----------: | \n| MLP              |1300ms     |NA|\n| Anchor Former    |800ms (-500ms)     |7.5ms |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7G9Ip5l0xv", "reviewer": "Reviewer_i5LS", "summary": "This paper introduces a novel vision-language connector, Anchor Former (AcFormer), designed to enhance the efficiency and accuracy of multimodal models. By identifying visual anchors within Vision Transformers and utilizing a cost-effective progressive search algorithm, AcFormer leverages these anchors to aggregate information more effectively. \n\nExtensive experiments demonstrate that AcFormer significantly reduces computational costs while maintaining or improving performance across various vision-language tasks compared to existing methods like Q-Former and Perceiver Resampler.", "questions": "1. Besides slight accuracy gain and training time speedup, are there other benefits of the proposed method over the baseline connectors? \n3. Only pretraining time results are provided. Is the proposed method faster at inference time as well?\n2. It seems that the proposed method is more efficient in terms of number of visual tokens. Will this make a big accuracy difference on tasks with high resolution images (documents for example)? \n\nI will raise my rating if the authors can convince me the significance of the key technical contribution of this paper.", "limitations": "The authors discussed them in the limitation section.", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is strong, attempting to improve a fundamental building block in recent multimodal models. \n2. The paper is well written and easy to follow. Tables and figures are helpful for readers to quickly understand. \n3. Experiments are extensive and solid. They cover various types of connectors (linear projection (LLaVA), Q-Former (BLIP-2), and Perceiver Resampler) and datasets (POPE, MME, MMB, MM-Vet, TextVQA, GQA, VQAv2, VisWiz and SQAimg).\n4. This paper is insightful and can be useful to the community.", "weaknesses": "1. The proposed method is not very effective. The key result comparison for this paper is AcFormer vs LLaVA-1.5 (linear), but in Table 1 and 2 the accuracy results are mixed (slightly in favor of AcFormer). The proposed method is not significantly better than the linear baseline. \n2. Linear connector results are missing in Table 3. \n3. Minor: The term Multimodal Large Language Model (MLLM) is self-conflicting. I suggest authors use Large Multimodal Model (LMM).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your review. And we will address your concern below.\n\n**Q1:** About the effectiveness.\n\n**R1:** Thank you for highlighting this concern. \nOur primary motivation in this work is to enhance the efficiency of Large Multimodal Models (LMMs). While various token reduction methods exist, they commonly suffer from performance degradation.Our ablation study consistently indicates that our method outperforms other token reduction techniques, such as C-Abstractor and Perceiver Resampler.\n\nMoreover, in Tables 1 and 2, while the results between LLaVA (Linear projection) and our method are mixed, our method shows an overall improvement when averaged across each benchmark. Although there are slight performance drops on some benchmarks (e.g., MM-Vet ~-0.3, VQAv2 ~-0.1), the speed is greatly improved. Overall, our method reduces computation cost while maintaining comparable performance.\n| Connector      | TextVQA($\\uparrow$)      | GQA($\\uparrow$)      | MMB($\\uparrow$) |MME($\\uparrow$) |\n| :--------- |:--------- | :----------: | :----------: | :----------: |\n| C-Abstractor  |53.4   |60.2  |67.8 |1775.4|\n| Perceiver Resampler  |52.1   |56.4  |65.4 |1720.8|\n| Anchor Former |58.0    |61.3|68.4|1846.1|\n\n**Q2:** About missing the linear connector result in Table 3.\n\n**R2:** Thanks for pointing out this issue. We have listed the result of linear connector (i.e. the result of LLaVA that uses linear connector) in Table 1 and Table 2. We will add them to Table 3 to make it more completed.\n\n**Q3:** About the name.\n\n**R3:** Thanks for your suggestion. We will use Large Multimodal Model to replace MLLM.\n\n**Q4:** About other benefits. (Inferencing time, Training cost)\n\n**R4:** Thanks for your concern. Our proposed method not only improves accuracy and reduces training time but also significantly decreases training expenses (GPU memory) and accelerates inference. This is particularly important for building high-resolution large multimodal models.\n\nWe show the **inference time** and **training memory** needed below. We report the inference time for each benchmark using the prompt \"Answer the question using a single word or phrase.\" For inference, we use 8 A100 GPUs, each with a batch size of 1. To test the training memory, we use a batch size of 4.\n| Connector      | Resolution      | TextVQA($\\downarrow$)           | DocVQA($\\downarrow$)          | ChartQA($\\downarrow$) | GPU Mem($\\downarrow$) |\n| :--------- | :----------: | :----------: | :----------: | :----------: |:----------: |\n| MLP              |336     |125s|198s|64s|31.24g|\n| Anchor Former    |336     |97s |115s|36s|22.04g|\n| MLP              |672     |571s|803s|276s|71.58g|\n| Anchor Former    |672     |384s|470s|141s|32.95g|\n| MLP              |1008     |-|-|-|OOM|\n| Anchor Former    |1008     |505s|653s|223s|50.76g|\n\nFrom the table, it is evident that our proposed method significantly reduces both inference and training costs without significant performance loss (refer to Q6 for more detailed analyse). In addition, this reduction allows us to develop higher resolution models with limited resource. For example, with 8 A-100 GPUs, our method can develop a model for input resolution of 1008x1008 while LLaVA-Next may suffer from out-of-memory issue.\n\n**Q5:** About the inference time.\n\n**R5:** We are grateful for your concern. We list the inference time in the table in Q4. The results demonstrate our method's effectiveness on accelerating the inference.\n\n**Q6:** About performance on high resolution image.\n\n**R6:** We present the performance of the MLP and our method on DocVQA and ChartQA below. For pretraining, we use LLaVA-558k. As LLaVA-Next's dataset is not publicly available, we augment LLaVA-665k with the DocVQA and ChartQA document datasets for Instruction-Finetuning (IFT).\n\n| Connector      | Resolution      | DocVQA($\\uparrow$)      | ChartQA($\\uparrow$) |Inference Speed($\\uparrow$) |\n| :--------- |:--------- | :----------: | :----------: | :----------: |\n| MLP           |672    |65.4|57.2|1 $\\times$|\n| Anchor Former |672    |64.9|56.7|1.66 $\\times$|\n| MLP           |1008   |OOM  |OOM |-|\n| Anchor Former |1008   |68.8|58.1|1.21 $\\times$|\n\n\nFrom the table above, it is evident that our method incurs only a slight performance drop compared to the baseline method. Additionally, our method can handle input resolutions up to 1008 × 1008 with less inference time than an MLP processing 672 × 672 resolutions. Notably, at higher input resolutions, any slight performance drops are compensated for, and even show significant improvements. For example, compared to MLP, our method may have a slight performance drop at 672 × 672, but it operates faster. By using higher resolutions like 1008 × 1008, we achieve better accuracy and still faster speeds, compensating for any initial performance loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7nmrg3j76e", "reviewer": "Reviewer_r5Ke", "summary": "This paper propose AcFormer, a novel vision-language connector in MLLMs.\nAcFormer is driven by visual anchors observed in vision transformer's PCA of the feature map and attention map of [CLS] token, where high value is observed in both of the maps.\nThe author use the attention value of [CLS] token to select the visual anchor in a progressive way, the selected visual anchors are then used as information aggregation tokens through cross attention.\nThrough extensive experiments the author prove the effectiveness and efficiency of AcFormer comparing with other multimodal connectors.", "questions": "* In line 186-195, what is the difference between AcFormer on $i-1$ th layer and selecting some features from $i$ th layer?\n* In line 203-214, how are the 6 layers selected and how are the features fed before the projection layer? This part is not clear enough.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* As is shown in the main experiment result, AcFormer achieves about 2.3/1.7 time faster pretraining speed while retaining the overall performance.\n* Extensive ablation result is provided to further prove the effectiveness of AcFormer.\n* The writing and experimental setup is clear.", "weaknesses": "* It seems to the reviewer that the main advantage of AcFormer comparing with original LLaVA-1.5 is its efficiency. However, only the training time is reported in the main table. Since all of them is less than 20 hours as is shown in table 6, apparently training time is not the main bottleneck of developing LLaVA-1.5 model. To further justify the efficiency of AcFormer, it's enouraged to report the inference time per token of the resulting models.\n* The motivation is not clear enough. The visual anchors discovered by the authors is a common phenomenon recently in vision tranformers, for example, [1] finds the outliers in ViT and provs that the outliers emerge because the ViT need some additional tokens to preserve golbal information. In this paper, more experiments should be done to unveil the reason behind the emergence of visual anchors, otherwise it not convincing enough to directly use the anchors to aggregate information, especially given that the performance improvement is limited comparing with the origial model.\n* AcFormer selects the visual tokens without considering the actual question, which means the token reduction could be harmful when the question is not about the major subject of the image.\n\n\n[1] Darcet, T., Oquab, M., Mairal, J., & Bojanowski, P. (2023). Vision transformers need registers. arXiv preprint arXiv:2309.16588.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thanks for your review. We will address your concern below.\n\n**Q1:** About the inference time.\n\n**R1:** Thanks for your concern of the effectiveness. We show the **inference time** and **training memory** needed below. We report the inference time for each benchmark using the prompt \"Answer the question using a single word or phrase.\" (Max generation length is set to 3) For inference, we use 8 A100 GPUs, each with a batch size of 1. To test the training memory, we use a batch size of 4 to avoid out-of-memory issues.\n| Connector      | Resolution      | TextVQA($\\downarrow$)           | DocVQA($\\downarrow$)          | ChartQA($\\downarrow$) | GPU Mem($\\downarrow$) |\n| :--------- | :----------: | :----------: | :----------: | :----------: |:----------: |\n| MLP              |336     |125s|198s|64s|31.24g|\n| Anchor Former    |336     |97s |115s|36s|22.04g|\n| MLP              |672     |571s|803s|276s|71.58g|\n| Anchor Former    |672     |384s|470s|141s|32.95g|\n| MLP              |1008     |-|-|-|OOM|\n| Anchor Former    |1008     |505s|653s|223s|50.76g|\n\nFrom above results, it is evident that our model not only accelerate the training stage but also accelerate the inference. Also, it can greatly reduce the training memory, which enables us to handle higher resolution input. (For LLaVA-Next it supports maximum resolution of 672 $\\times$ 672)\n\n**Q2:** About the motivation.\n\n**R2:** Thank you for your suggestion. We appreciate you bringing the paper \"Vision Transformers Need Registers\" to our attention. However, it was not formally published before the NeurIPS 2024 submission deadline, so we had not reviewed it earlier. After examining the paper, we acknowledge that while both our work and this paper identify similar phenomena, our motivation and methodology are distinct. \n\n**Motivation**. The paper \"Vision Transformers Need Registers\" believe that special tokens can be harmful. They propose using registers (extra tokens) to eliminate these tokens and thus enhance vision transformer backbone. In contrast, our work, delves deeper into the utility of these tokens and utilizing them to build Multimodal Large Language Model (MLLM).\n\n\n**Analyse on the effectiveness of AcFormer**. We analyze the feature transformation process in Vision Transformers through both the attention map and feature map. Our experiments reveal that information tends to aggregate around these anchors. In Acformer, we leverage these anchors to be queries and all of the visual tokens to be keys and values. By this way, the anchors are trained to extract infromation from the whole image. This is different from Q-Former or Perceiver Resampler, which utilizes learnable queries to aggregate visual information.\n\nThrough extensive experiments, we evaluate our method. Results demonstrates comparable performance while significantly reducing costs, as shown above in Q1.\n\n**Q3:** About the token reduction's harm on different question.\n\n**R3:** Thanks for your concern. In our work, the Anchor Former consists of a stack of attention and feedforward layers. Instead of using anchors directly as image features, we use them as queries for the Anchor Former, with all image tokens serving as keys and values. During training, the model optimizes the Anchor Former parameters to implicitly extract visual features that is enough to answer different questions. (Covering both fine-grained and coarse-grained)\n\nThis is also evident by our experiment on DocVQA and ChartQA. These two benchmarks are about the document quesiton answering, which is more related to fine-grained visual information understanding. From below results, it can be found that our method can also handle the fine-grained visual question answering. Though sacrificing little performance, it greately accelerate the inference process.\n\n| Connector      | Resolution      | DocVQA($\\uparrow$)      | ChartQA($\\uparrow$) |Inference Speed($\\uparrow$) |\n| :--------- |:--------- | :----------: | :----------: | :----------: |\n| MLP           |672    |65.4|57.2|1 $\\times$|\n| Anchor Former |672    |64.9|56.7|1.66 $\\times$|\n| MLP           |1008   |OOM |OOM |-|\n| Anchor Former |1008   |68.8|58.1|1.21 $\\times$|\n\n \n\n**Q4:** About the difference between AcFormer on i-1 th layer and selecting some features from i th layer?\n\n**R4:** Thank you for your question. AcFormer is a stack of multi cross attention layers. So in the Anchor Former, we do not directly select specific features. The anchor selection occurs only after the Vision Transformer stage. We utilize the selected anchors as queries and all visual tokens as keys and values to perform cross-attention. Assuming the Anchors are $IA\\in R^{M\\times D}$ and the vision tokens are $V \\in R^{N\\times D}$, the computation of the i-th layer in AcFormer can be illustrated as below.\n\n$IA_i = IA_i + Attn_i(query=IA_i, key=value=V)$\n\n$IA_{i+1} = IA_i + FFN_i(IA_i)$\n\n$IA_0$ is the selected anchors after the vision transformer. V is all of the visual tokens.\nThe final vision representation is $IA_{6}$ from the last layer (We have 6 layers for anchor former). We will revise the relevant sections of the article to achieve a clearer description.\"\n\n**Q5:** how are the 6 layers selected and how are the features fed before the projection layer?\n\n**R5:** Thanks for your question. \nThere are six layers in Acformer with each contains an attention module and feed forward module. So there is not layer selection within the Acformer. There are only one anchor selection process between the Vision Transformer and Acformer. The selected anchors are applied as queries while all the visual tokens are applied as keys and values. They are intergrated with cross attention mechanism in Acformer as illustrated in Q4 above. Acformer are trained during all the stage (Pretrain and IFT).\n\nFollowing the Anchor Former, the output features have a shape of number_of_anchors $\\times$ dimensions. We then use an MLP to project these dimensions to match the LLM's hidden size.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYB471AGkq", "reviewer": "Reviewer_Ld45", "summary": "This paper introduces a new method to make a strong connector between a language model and a vision encoder, in order to build a vision-language model. The method is derived from the Perceiver Resampler, but uses a clever initialization for the queries. The authors validate their approach with a series of ablations, comparing the other connectors to theirs.", "questions": "- What is the effect of N, the depth of the AcFormer, on the performance on the benchmarks? Which depth do you recommend taking in practice?\n- In the paper, you mention that the strategy of the Anchor Selector is to take the attention map already computed to avoid having to re-compute something, and use it after modification for the initialization to the queries in your AcFormer. Have you tried other approaches where you are not focusing on efficiency and you are allowed to do additional operations, to see if it performs better?\n- In Figure 2, we see the effect of the layer on the attention map. Which layer are you considering for the final attention map that you are using for your algorithm? Have you tried different layers to see if it performs better?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This work meets a demand in the field of vision-language models, by creating a stronger connector between a language model and a vision encoder.\n- The authors obtained better scores with their connector than with the most used and best ones researchers currently use, while being efficient too.", "weaknesses": "- The Anchor Selector is to me the meat of the paper. The AcFormer is a Perceiver Resampler with queries benefiting from a better initialization that depends on the input image, instead of the same queries for all images. However, I find that the description of the Anchor Selector algorithm and the justification for why it could work could be better explained in the main paper (the full algorithm is in the appendix). Providing a pseudocode instead of the code could also help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thanks your comments. We will address your concern below.\n\n**Q1:** About the anchor selection algorithm.\n\n**R1:** Thanks for your suggestion. We will provide you with the pseudocode below.\nAssuming the Visual Feature Map is $V \\in \\mathbb{R}^{B \\times N \\times D}$, the Visual Attention Map $A \\in \\mathbb{R}^{B \\times H \\times N \\times N}$, and the desired Anchor Token Number $T$.\n\n1. **Initialize Variables**:\n   - `result` is set to empty list `[]`.\n   - Calculate `Per_head_num` as $(T - 1) / H$, where `[CLS]` is chosen by default.\n\n2. **Iterate Over Batches**:\n   - For each batch $i$ in the range $B$:\n     - Initialize `max_indices` with `[0]` (the index of the `[CLS]` token).\n\n     - **Iterate Over Heads**:\n       - For each head $j$ in the range $H$:\n         - Extract and sort attention scores for the `[CLS]` token, excluding the `[CLS]` token itself.\n         - Select top `Per_head_num` indices and add to `max_indices`.\n         - If duplicates occur, select additional indices to ensure the required number of unique tokens.\n\n     - **Update Selected Anchors**:\n       - Fetch `selected_anchor` according to `max_indices` and append it to `result` \n\n3. **Return**:\n   - Return `result`.\n\n\n**Q2:** The effect of the depth of the Anchor Former.\n\n**R2:** Thanks for your concern. We present an ablation experiment below, conducted using the Vicuna-7b model, in which we select a total of 145 tokens.\n| Depth      | TextVQA($\\uparrow$)      | GQA($\\uparrow$)           | MMB($\\uparrow$)          | MME($\\uparrow$) | Para Num (M) |\n| :--------- | :----------: | :----------: | :----------: | :----------: | :----------: |\n| 3    |57.7     |60.9    |68.1     |1816.1   |32.1    |\n| 6    |58.0     |61.3    |68.4     |1846.1   |63.1    |\n| 9    |58.2     |61.2    |68.3     |1856.1   |125.9   |\n\nFrom the table above, it is evident that using a depth of 6 with the training data for LLaVA-1.5 is sufficient. Further increasing the depth does not yield significant gains.\n\n**Q3:** About other approaches for anchor selection.\n\n**R3:** Thanks for your question. As we have mentioned in our paper, the **PCA** operation on the hidden states can also be employed for anchor selection. Assuming the output of the Vision Transformer is $H \\in R^{N \\times D}$, we apply PCA and select the first 3 components to obtain $H_d \\in R^{N \\times 3}$. We then sum these three components and sort the results to identify visual anchors without relying on the attention map. Experimental results show similar outcomes. Additionally, we calculate the Jaccard distance for the indexs of the selected anchors between the attention method and PCA-based method, which reaches 70%, indicating a high degree of consistency between the two approaches. However, due to the computational expense of PCA, we are not able to fully training the model with this method. In our future work, We will further explore other methods to extract these tokens.\n\n**Q4:** About the layer choice.\n\n**R4:** Thanks for your concern. In our experiment, we choose the last but one layer. We observe that before layer 12, the [CLS] token's attention is primarily focused on the object, gradually integrating with other visual anchors. We test the attention maps of the last 10 layers for anchor selection and compute the Jaccard distance of the selected tokens' index. Our findings indicate that the selected tokens' index exhibit at least 90% overlap across different attention maps. Therefore, to ensure consistency with feature selection, we choose the penultimate layer for our configuration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel approach, AcFormer, which leverages visual anchors in vision transformers to improve the efficiency and accuracy of multimodal models. The paper addresses the challenge of integrating vision and language models by introducing a strong vision-language connector that reduces computational costs while maintaining or even improving performance. The authors provide extensive experiments, demonstrating significant efficiency gains, especially in inference speed and memory usage, without compromising performance across various benchmarks. While some reviewers raised concerns about the novelty and motivation behind using visual anchors, the authors effectively addressed these concerns by providing detailed explanations and additional experimental results in their rebuttal. The responses clarified the method's advantages and demonstrated its broader applicability, particularly in handling high-resolution images. Given the strong empirical results, the thoroughness of the experiments, and the authors' satisfactory responses to reviewer concerns, I recommend accepting this paper.
Decision: Accept (poster)