Efficient Large Multi-modal Models via Visual Context Compression
Authors: Jieneng Chen
Luoxin Ye
Ju He
Zhao-Yang Wang
Daniel Khashabi
Alan Yuille
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-modal LLMs; Efficiency
Primary area: machine_vision
TL;DR: This study introduces a Visual Context Compressor for multi-modal LLMs, demonstrating that significant compression of visual tokens during training can enhance efficiency and maintain performance, accelerating training by a large margin.

Abstract: While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments
show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

Efficient Large Multi-modal Models via Visual Context Compression
Jieneng Chen", Luoxin Ye, Ju He", Zhao-Yang Yang Wang, Daniel Khashabi", Alan Yuille? Johns Hopkins University
Abstract
While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments show that eliminating up 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accu- racy on the GQA benchmark, indicating significant redundancy visual context. Addressing this, we introduce Visual Context Compressor, which reduces the num- ber of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression tokens while maintaining training efficiency, we develop LLa Volta as a light and staged training scheme that incorporates stage-wiss wiigewiie visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding loss of information when testing. Extensive experi- ments demonstrate that our approach enhances the performance of MLLMs in both image-language and lideo-language understanding, while also significantly cutting training costs and improving inference efficiency.
visual
Website
Code
httss///gthhaa.ccm//poooccccer/e
Introduction
The advent of LLMs 33
has marked natural language processing. LLMs can play a role as a universal interface for general-purpose assistant, where various task instructions can be explicitly represented language and guide the end-to-end trained neural assistant to solve a task of interest. For example, the recent success of ChatGPT [33] and GPT-4 K have demonstrated the power of aligned LLMs in following human instructions, and have stimulated tremendous interest in developing open-source LLMs 41 43). As the horizon of LLM applications broadens and the availability of open-source LLMs increases, the integration of muult--modalln into these models presents new frontier in expanding their capabilities. Multi-modal L.LMs
34
44
era in the field of artificial intelligence and
28 40 54 (MLLMs). which can process and understand not just text but also visual information, stand at the cutting edge of this evolution.
While MLLMs have made significant strides, crucial aspect that remains relatively unexplored is the efficient representation and processing of visual information within these models. Substantial have been dedicated to optimizing the efficient representation of text tokens aimed at enhancing inference efficiency by
efforts [8 35 3
through various compression techniques [8 35 53
"Contributed equally. t Advised equally.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Performance vs. Visual Tokan Compression Rate
(b) Attention to Visual Tokens vs. System Prompt Takens
12:00
13:05
115
10"2
Average Attenton
000 000-000mry..0
Visual Tikers
10-4
System Prompt
20:47
Tandfomer Lagen 2
10-
Tanofrrmr Layer 0
Tondformr Lagen 18
1.
20:7
30.0 43.1
conntaage Visual Sakrr
23.5 60.3
1000
Transtormer Layer
25 20
25
30
Figure 1: Visual tokens are redundant MLLMs. Left:
The accuracy of the LLaVA-1.5-7B 28
without re-train) on the GQA 20] benchmarks varies with different percentages of retained visual tokens. The z-axis represents the percentage of original visual tokens preserved after applying 1D average pooling with varying stride sizes S applied in i-th Transformer layer. Right: Visual tokens receive less attention from the IANS] ] token as we go deeper into its layers LLaVA-1.55 1.B-7B model. These findings collectively suggest significant redundancy within the visual tokens of the MLLMs.
attentively selecting important tokens. However, the efficient learning of visual tokens in MLLM has not garnered comparable attention. Naturally, this raises questions about the potential redundancy present in visual tokens and its implications for the overall computational efficiency of MILMs.
We start our work by addressing the question: Are visual tokens redundant in multi-modal LLMs? To explore this, we first experiment with simply reducing the number of visual tokens in pre-trained LLaVA-1.5-7B (8 at the inference stage via average pooling ($3.).. As shown in Fig (left), our initial results demonstrate that eliminating up to 70% of visual tokens by pooling them with stride of 4 starting from Transformer layer incurs only a minimal performance loss on the GQA benchmark, specifically a 3% accuracy reduction. Additionally, values from the [ANS] token to visual tokens and system prompt tokens across different Transformer layers in the pre-trained LLLVA--155-B 28 As revealed in Fig. (right; blue trends), the visual tokens are generally less attended to, measured based on average attention from the [ANS] token, as the layers get deeper. These two early explorations indicate significant redundancy in visual tokens.
compute and present the average attention
Addressing this, this work we develop an effective Visual Context Compressor that can be integrated into the training of MILMs. Surprisingly, a simple average pooler nested in LLMs stands out as the most effective compressor, outperforming the attention-based [88] 53 and parametric (23 counterparts. We attribute this to two reasons: (1) The simple pooling operation makes training stable, whereas prior attention- based approaches [8 (33 are specifically designed for accelerating inference rather than training. (2) Visual tokens in the deeper Transformer layers are less attended to (see Fig. [ (right)) and particularly redundant, making a simple compressor placed in a deeper Transformer layer effective enough. At lower training cost, the LLa VA- 28 trained with the proposed Visual Context Compressor is competitive with the con-compeessed baseline across various multi-modal benchmarks (e.g., GQA [20] and MM- Met [50)). This dual achievement highlights Visual Context Compressor's 's role as a pivotal advancement in enhancing the efficiency and performance of MLLMS across various multi-modal guestion-answering benchmarks.
To further mitigate the information loss caused by compressing visual tokens, especially under a large compression ratio (CR), we have devised LLaVA--pwered lite training scheme, dubbed LLa Volta, which progressively employs Visual Context Compressor at multiple training stages with different compression ratios ($ (3.3). Specifically, LLaVolta progresses through several stages, beginning with a high level of visual token compression and gradually reducing the compression ratio until the final stages, where full visual tokens utilized. This multi-stage approach allows for adaptive compression levels that ensure training efficiency without losing information at testing, thus maintaining the overall effectiveness of the model.
Extensive experimental evaluations of L.aaVolta have been conducted on thirteen widely-adopted MLLM benchmarks for both image-language understanding and video-language understanding,

---

showing promising results, We observe that LLaVolta not only enhances the performance of MLLMs, but also achieves a substantial reduction in training costs. These experiments validate the effectiveness of our method, demonstrating its capability to optimize resource utilization while maintaining even improving model performance.
In summary, our paper makes the following contributions:
We present two initial studies to verify the redundancy of visual tokens in MLLMs.
We propose the Visual Context Compressor, a simple yet effective compression technique that utilizes an average pooler, enhancing the efficiency of multi-modal models.
We propose the VLaVolta as an efficient training scheme by leveraging Visual Context Compressor at multiple training stages with progressively decreasing compression ratio. To the best of our knowledge, we are among the first to explore efficient training of MLLMs.
Extensive experiments show that our approach not only improves the performance of MLLMs in image-language and video-language understanding across various benchmarks but also showcases efficiency gains by reducing training costs by 16% and inference latency by 24%.
2 Related Works
Multi-modal LLMs. The evolution of large language models [10
33 34
into their multi-moda
counterparts [28 40) represents a significant leap in their ability to follow instructions and generalize across tasks. This transition has been marked by seminal works such Flamingo M, BLIP-2 (23 and LLaVA (28). which have extended LLM capabilities to encompass visual tasks, demonstrating impressive zero-shot generalization and in-context learning abilities. Progress in multi-modal LLMs has primarily been driven by advancements in visual instruction tuning [28] [54). leveraging vision- language datasets and refining visual instruction-followe
data. Additionally, efforts have been made to enhance the grounding capabilities of multi-moddl LLMs through the use of specialized datasets aimed at improving task-specific performance. Despite these advancements, the exploration of visual compression within multi-modal LLMs remains relatively underdeveloped. The design and optimization of compression strategies are crucial for maximizing the effectiveness and efficiency of multi-modal LLMs, suggesting a potential area for future research and development.
Visual Redundancy. In computer vision, reducing redundancy is crucial for creating efficient yet effective models without losing accuracy [4) Redundancy in images often arises from the inherent characteristics of natural scenes, including repetitive patterns, textures, and areas of uniform color. These features, while contributing to the richness and detail of visual perception, can lead to inefficiencies in both storage and processing when not adequately addressed. Image compression algorithms [46] can reduce file size by climinating or efficiently encoding redundant data. These methods take advantage of human visual perception's 's tolerances to subtly reduce data without significantly impacting image quality. Advanced machine learning models, particularly CNNs and autoencoders B offer sophisticated approaches minimizing redundancy. Transformers 45 as fundamental architecture for LLMs [10] 34 apply self-attention mechanisms to dynamically bind the most informative parts of tokents. Vision Transformers 6
16) trained with CLIP objective 7 36 encode image to sequence of visual features for multi-modal LLMs 22
Nevertheless, visual tokens receive less attention in LLMS due to attention shrinkage [47, resulting a waste of computation. In this work, we focus reducing the redundancy of visual tokens in MI..Ms.
Efficient LLMs. Efficient inference and training for LLMs are important. Compressing input sequences for efficiency reasons in Transformers is not a new idea for NLP. Much work is being done to accelerate the inference of LMs. For example, Pyramid Transformer variants [T] and [I9 are proposed in Encoder- -ecooer LMs that progressively compress the sequence as the layers grow deeper via pooling or core-set selection. Nawrot et al. [32) propose adaptively compressing the sequence based on the predicted semantic boundaries within the sequence. Rae et al. [37 propose compressing the fine-grained past activations to coarser memories. VCC (53) compress the sequence into a much smaller representation at each layer by prioritizing important tokens. Besides efficient inference, accelerating training for L.MMs attracts attention as well. A staged training setup (38) is proposed which begins with a small model and incrementally increases the amount of compute used
3

---

for training by applying growth operator to increase the model depth and width. However, efficient training for LLMs in multi-modal scenarios rarely explored.
3 Method
In this section, we first introduce an overview of multi-modal LLMs in $ 3.1 problem of visual redundancy and introduce Visual Context Compressor in 3.2 Finally, we present our proposed LLaVolta in $ 3.3
Then, we define the
3.1 Preliminaries: Multi-modal LLM
We start by reviewing the design of the LLaVA family 27 28
For processing an input image X... we utilize the pre-trained CLIP visual encoder ViT-L/14, as detailed by [36], to extract the visual feature Z. (X.), where g(.) indicates the visual encoder. To bridge the gap between visual and linguistic modalities, the LLaVA [27] (28] framework as an MLLM implements a straightforward linear/MLP transformation. This involves trainable projection matrix W. which maps the visual features Zu into the linguistic embedding space, producing language embedding tokens H, = WZ. These tokens are designed match the dimensionality of the word embeddings within the L.M.
For each image X., one can generate multi-tur turn conversation data (X), X.. ..... X x with T as the number of turns. One can organize them as a sequence, by treating all answers as the assistant's response and the instruction X instsuct at the t-th turn as:
Random Choose X.) or X. X 1>
t=1
X Instruct
(1)
X ??
This approach establishes a standardized format for the multi-modal instruction-follovin sequence. It allows for the instruction-bart based tuning of the LLM to be applied the prediction tokens, utilizing the model's native auto-regressive training objective. Specifically, for a sequence with length L, the likelihood of the target responses X, is calculated as:
PX., X X Xinstruot) =
I1 pa( (x, X Iisttrctt <is Xa,ei))
(2)
3.2 Visual Context Compressor
The redundancy observed in images often arises from inherent traits of natural scenes, including repetitive patterns, textures, and regions with uniform color. While these traits enrich visual perception by offering detail and depth, they also present challenges in terms of storage and processing efficiency. Considering the inherent limitations Transformers in handling long sequences (2) 29 49], it is critical to minimize any length redundancies to obtain a more effective accuracylefficiency trade-off.
Problem Formulation:
The objective of this study is to decrease the length of visual tokens X, (i.e., its hidden states H. if inside LLMs), while simultaneously maximizing the probability of the target response (X,XX, Xinstruct) described in Equation
Visual Context Compressor: key design change that we introduce is a compressor layer that compresses the dimensions of the visual inputs by reducing the effective number of visual tokens. As depicted in Fig. 2 the compressor is simply an average pooler in our setting. It is applied to the visual tokens in k-th Transformer layer of an LLM. Formally, given the hidden visual tokens at k-th Transformer layer H3 E IR L the compressor is expected to fulfill the following projection: f: BXCxL + R BxCLL., which results in compressed visual tokens H, E BxCx Las, where Lowr = - with as the compression stride. In $4 we explore multiple variants of compressor to reduce the token length, including random token dropping [17 with dropping ratio 1 5. K- Means 21 with number of centroids set to No 5. attention-based token-centric compression [53) attention- based token dropping 9 18] and average pooling with stride To our surprise, we find that the simple average pooler the most effective compressor for vision tokens within MLLMs, due to its stability during training detailed in $ 4.4 Thus, choose average pooler the compressor.

---

Note that the proposed Visual Context Compressor can be directly applied to any off-the-shelf MLLMs assess the visual redundancy, as conducted in $4.2 One can also train an MLLM with Visual Context Compressor to reduce the number of visual tokens while maintaining competitive multi-modal performance.
Compression Ratio (CRl For an LLM with N Trans- former decoder layers, the compression ratio for visual tokens can be calculated as:
"What dog up to?"
visual token text token
L CR (N- - K) Low + K
(3)
layer 1
where K is the K-th Transformer layer of multi-modaa LLM; T. is the the length of visual tokens input into Vi- sual Context Compressor; Lour is the compressed length of visual tokens generated by Visual Context Compres- sor, as illustrated in Fig. 2
LLM
Layer 2
layer K
Compression Ratio
N-L
Visual Compressor
(V - kmm R
Loue
Our architecture modifications thus far mostly impacts the inference efficiency of MLLM, however, its impact trade-off remains unclear. We will study this question in the context of training MLLMs with goal of enhancing efficiency without compromising performance. We then move on to further utilize Visual Context Compressor to design efficient training scheme to incorporates Visual Context Com- pressor at various stages of the training process.
layer K+1
on ietforrannnc-oompresseee
layer
ANS
Figure 2: Example of Visual Context Com- pressor in multi-modal LLM.
3.3 LLaVolta as a Light, Staged Training Scheme
Training with Visual Context Compressor not only facilitates efficient inference but also enhances training efficiency. However, devising an effective training scheme poses challenges when ensuring fair comparisons with the original LLaVA (27), primarily due to differences in the number of tokens involved in inference. This discrepancy may lead to information loss, particularly when operating under a scenario with a high compression ratio. To tackle this issue, we have developed a lite training scheme for VLa dubbed as LLaVolta, which employs stage- visual context compression. Generally, assuming there are N₂ total stages, stage i involves N. of the total training epochs with a compression ratio of ri and the final stage proceeds without any compression. Essentially, as training progresses, i increases while r; decreases.
stage 1
In this work, as depicted in Fig. 3 we primarily explore a three-stage training pipeline that progres- sively reduces the compression ratio, as detailed below:
Training Stage I: Heavy Compression. The MILM training at the first one-third of the total training iterations commences with a heavy compression ratio (> 500%). where Visual Context Compressor is applied in an early layer of the LLM with large pooling stride. This setup enables very fast training speed.
Training Stage II: Light Compression. The MLLM continues training with another one-third of the total training epochs. At this stage, Visual Context Compressor is applied at only the deeper layers of the LLM with a smaller pooling stride compared to Training Stage I.
Training Stage III: No/subtle Compression. The MLLM continues training during the final one- third of the total epochs, with either no compression or subtle compression applied. This stage is designed to align with the inference process, where visual tokens may also undergo compression. By maintaining consistency between training and inference, this approach ensures that critical information is preserved while still allowing for compression, minimizing any potential discrepancies between training and real-world use.
Given the above meta framework, we can instantiate a family of training schemes, as demonstrated in Tab. I The single-stage (non-compression) scheme is equivalent to the MLLM baseline. For
Definition of compression ratio from Wikipedia
5

---

stati training, the compression stage can either go deeper or wider. "deeper" implies an increase in K (Transformer layer), while "wider" means decrease in the stride of the pooler.
ANS
Training Contes Comoression
many
"How
yellow
ANS
text token
visual
aypar
Voual sivere per appt
wider more tokens)
layir
Transformer
Transformer
full-toleens training & Inference
training stage! heavy compression
training stage second 1/3 of iterations
iIl & Inference last Itorations subtile compression
compression
Traditional Paradigm
Light Paradigm
Figure 3: Training & inference paradigm comparison for conventional setting (A) and LLaVolta (B). Meta framework of LLaVolta consists three training stages: Stage I with heavy visual compression: Stage II with light visual compression in deeper layer; Stage III with subtle compression with wider token window without loss of performance. This accelerate the training and inference 18+% while maintaining performance.
#Stages
Scheme
Stage
Stride
CR
HEpoch
0.25
#Singes
Schemne
Siuge
Layer Stride
CR
HEpoch
deeper
52
188%
0.25
Single no compression
S1
100%
133%
0.25
8
557%
Two
compression
51
557%
0.25
51
8
557%
0.33
Four
deeper then wider
1789
0.25
Three
compe. deeper
52
16
8
178%
0.33
0.25
100%
0.33
54
100%
0.25
S1
8255
Three
compr. wider
52
2
188%
stage compression
0.33
53
7
100%
0.33
5a
16
160%
0.33
Table 1: Instantiations of VLatolta schemes. deeper indicates that the compressor's position in the LLM
shifts from the shallow layer (e.g., 2) to deeper layer (e.g., 16). wider indicates that the compressor's stride decreases while the number of visual tokens increases. Last stage compression refers to using compressor stage for efficient inference.
Note that all training schemes will be standardized to complete just one epoch. Thus, in the three-stage training, each stage will receive one third an epoch, while in the four-stage training, each stage will receive one fourth of an epoch. Effects of non-uniform stage splitting presented in the Appendix.
4 Experiments
In this section, we begin by detailing the experimental setup in $ 4.1 Next, we elaborate on the proof-of-concept in Section $ 4.2 Following this, we validate the proposed I.LaVolaa in 44.3 with an ablation study in 4.4 Finally, assess the extensibility to video-language in 4.5

---

4.1 Experimental Setup
We adopt the Vicuna-v1.5--77 .5-7B [T] the language model, leveraging the LLaMA2 codebase 433 We
leverage the pre-trained CLIP ViT-L/14 [2 36 with an input resolution of 336 x 336, resulting in 576 visual tokens. We employ the LLaVA framework [27 connect the frozen CLIP vision encoder and the Vicuna LLMs. Along with the projector. we train the entire LLM instead of parameter- efficient finetuning. We follow VL- ( to perform data preparation and training schedule for pretraining and instruction tuning. We conduct all the experiments with the machine of 8x Nvidia RTX 6000 Ada. Due to multiple invalid image links in the dataset of instruction tuning stage, the scores of LL- reported in our analysis are reproduced by ourselves to ensure a fair comparison under the same experimental environment.
It is worth mentioning that assessing visual token redundancy only necessitates the inference of existing off-the-shelf models, whereas the other experiments involve the training of multi-modal LLMs. specifically projectors and LLMs.
Benchmarks and Metrics: We adopt thirteen benchmarks specifically designed for MLLM eval- uation, including GQA [20]. MM-Vet [50], ScienceQA (II) MME[]]] TextVQA [39],
POPE 24], MMBench 130], MMBench-CN 30), VQA-v2
14) LLaVA-Bench-in-tie Will
(LLaVA 28). Wis [15). SEED-Image [22 and MMMU [52). GQA and VQA-v2 evalu-
ate the model's visual perception capabilitiess open-ended short answers. MME-Perception evaluates model's visual perception with yes/no questions. ScienceQA with multiple choice are used evaluate the zero-shot generalization on scientific question answering. TextVQA contains text-rich visual question answering. MMBench and the CN version evaluate a model's answer robustness with all-round shuffling on multiple choice answers. MM-Vet evaluates a model's capabilities in engaging in visual conversations. Additionally, we extend VLaVolta to vidco-language understanding, and follow Video-LLaVA (26) evaluate the models on MSVD-QA (5). MSRVTT-QA (48) and ActivityNet-QA [ST, where the accuracy and score are assessed using GPT-Assistant.
We report the official metrics calculated using the standard implementations provided for each bench- mark for a fair comparison. Latency is reported as the time taken during inference until the first answer token is produced. When reporting average performance in Table _ the score of MME is divided by 2000, as its range is from 800 to 2000. TFLOPs are profiled via DeepSpeed, For total number of tokens, #Tokens = ΣN # #Token'. The training time is reported for one epoch of training during the LLaVA instruction-tuning stage. The Compression Ratio (CR) is defined as in Equation 3
4.2 Proof of Concept: Visual Context Redundaney
To assess the redundancy of visual tokens, we perform average pooling within an off-the-shelf LLaVA-1.5-7B checkpoint at the testing stage, using different pooling stride sizes S across various Transformer layers K. As shown in Fig-[] the model still exhibits strong performance even when retaining only 62.5% of the visual tokens (S = 4, K 16) in the MM-Vet benchmark, without the need for additional training. When adopting the same setting (S = 4, K = 16), a similar trend can be observed in the GQA benchmark as well, where the compressed model only has 1 performance drop than the uncompressed counterpart. Surprisingly, in the GQA benchmark, eliminating up to 70% of visual tokens (S = K 16) results in a mere 3% decrease in performance. This proof-of-concep. shows a certain level of redundancy in the visual tokens within MLLMs.
4.3 Main Results: Volta
In this section, we present the main results of LLa schemes instantiated in $ 3.3 We conduct a thorough evaluation of the multi-modal capability across 13 benchmarks. Tab. 2 demonstrates that our proposed LLa Volta not only consistently lowers training costs by 19% (15.3 hours vs. 12.4 hours) but also surpasses the non-compression baseline. The last-stage-oompression training schemes achieves the best performance across thirteen benchmarks and obtains 62. 1% average performance, improving I.IVVA-vI.5-7B (27] with much less inference TFLOPs and training time, This indicates the necessity of designing an optimally lite training scheme.
7

---

Lad Siie
Lusacy
Aöuggs
Scleee
Vickees"
CK!
TELOPs
Tieee
dig
SQA MNE Vige" PoeE MMB
vgvi
LLaws"
Viswii sBED'
SIMM Avp.
Segle
to compopvace
19432
1.26
15.3a 62 21.5 T0 1966s Skaas Malay todas MM4
cosppionies
1934
1:35
91.7a 709 1400, 583,, B65as
185.a
649tr
Thom
eeegge Sequr
10587
1745
8.76
68.5
12:09 47
50.5 70.5 1777 19 Mid 08f
195
18.50
67.51 48 a5 659
350 0.
Thrse
coerpe vider
10407
177%
1.26
12.33 6i.ka St.ae 71.8x 1830 3655 Mabas bikas MMaas
187
543m 613
1466, 587, 168a
658
Foar
Aaper the wide:
18863
1709
8.36
68.5
12,90 42. 51.5 70.5 1472 R87 56.3,s 68.6 PAR
78.8
68.00 4831, (il
Thaor led Mage cueppionire 1666
2359
5.77
522
1244 023.g 51.5,, 71,0 1819, 580, 56.5 68.3, 20. 75 692.., 5821. 654, 38,4 624
Table 2:
Performance of Voltta See the definition of each training scheme in Tab. t: average across stages. First five derived schemes for training acceleration achieve competitive results while reducing 16% training time. The last scheme, last stage compression, achieved the shortest training time (12.4 hours) and the lowest inference cost (5.47 TFLOPs), but also the highest average performance (62.1%) We report average results across three runs, with the standard deviation written the bottom right of the average result. The last stage compression training achieves the best average performance across thirteen benchmarks, outprroorming the baseline (LLaVA-v1.5-7) while requiring significantly less training time.
4.4 Ablation Study
In this section, we perform an ablation study on the choice of visual compressors by comparing different compression methods. Additionally, we examine the effects of varying the stride and LLM layer in training Visual Context Compressor.
Compeesur
Uikers
UK
GUA
MM--ee
SQA
MMI!
VIDA"
PUPE
MMB
MM
VUAA
ELaVA"
VieWar
SHEEY
MMMU
withost compossiin:; Texing comepraation
Randan Deopping
3312
S565
$0.5
21.4
1142
46.5
55.6
53.3
50.3
47.6
47.2
S2.2
47.3
K-Mcar
8312
5569
$4.6
25.9
1155
7K6
69.3
57.6
56.1
8312
556%
$2.1
30.6
1296
65.6
53.0
65.6
$4.8
$6.1
VCC
3582
514%
54.7
25.9
69.2
1346
49.2
72.3
60.8
52.0
68.1
55.6
47.8
57.0
34.8
54.7
Averige Pooling
3312
556%
53.7
25.6
1150
47.7
70.1
56.4
46.5
67.0
55.6
55.7
Triis with compreziion; Tonin wit compression
Randon Despping
3312
556%
53.4
25.0
69.4
1186
49.4
64.9
52.0
41.3
59.7
51.5
47.9
52.6
34.6
50.8
MeMas
3312
556%6
57.5
25.9
1379
79.4
62.6
54.6
25.7
59.2
Facy
3312
556%
55.9
21.9
70.4
1327
79.8
55.9
09.5
51.2
56.8
3582
534%
57.7
29.3
1396
83.6
55.8
00.1
Averup
3312
5566
60.0
30.7
1450
85.5
59.5
75.9
62.6
Table 3:
Comparison among different visual compressors. Higher values preferred. All methods except VCC are set to the compression ratio of 556% to approximate VCC's 514% [53] for fair comparison. The best scores are marked as gray and the second best are underlined. Attention-based compressors (i.e., FastV and VCC) excel during the inference phase, yet their application to the training phase proves challenging. Average pooling shows more stable performance during the training phase.
Choice of Visual Compressors. The design choices include (1) random token dropping. (2) K-Means clustering. (3) average pooling, (4) FastV [9. (5) VCC [18), (6) parametric pre-trained Q-Former (23). We have the following three observations. Firstly, Tab. 3 shows that the attention- based methods, including FastVV and VCC win 9/13 best and second best scores, showcasing the high performance when compressing visual tokens in inference. However, they are ineffective when applied to training because the in- traininnng attention scores are unstable. Secondly, and surprisingly, the average pooling obtains the highest scores on eleven out of thirteen benchmarks when it is used to train MLLMs with a high CR. Thirdly, Tab. 4 shows that both Q-Former and average pooling can obtain reasonably good performance when trained with extremely high CRs, and the average pooling performs better with less training cost. The reason could be that the Q-Former resamples tokens outside the LLM, potentially causing the LLM to overlook crucial information relevant to the response. In contrast, approach employs average pooling subsequent to Transformer layer K, allowing the initial K layers of the LLM to effectively retain important information from uncompressed tokens. Given these three insights, we select average pooling as our favored approach for visual compression.
Performance Across Compression Ratios. Herein, we train the multi-modal LLM with our Visual Context Compressor in various settings. As demonstrated in Tab. 5 the proposed method offers certain improvements and trade-offs compared to the state-of-the-art method, LLaVA-1.5-7B. We have the following two observations. Firstly, in the heavy compression level, the performance of MLLM is inversely proportional to the compression ratio (linearly scaling to the number of visual
8

---

Train
Method
4Puram Vlokens
CR
Time
GQA
MMVei SQA
MME VQA"
POPE MMB MMB
VQA
LLaVA"
VisWiz SEED"
MMMU Avg.
Q Q-Former
[X]
105M
1024
1800%
10.
55.7
26.4
69.3
1217
49.2
83.0
57.7
50.7
71.4
64.6
52.6
55.1
34.0
56.2
Ours
855
2156%
9.2h
55.9
26.3
71.0
1321
51.6
82.5
63.3
55.9
74.5
63.1
47.8
57.3
35.7
57.8
Table 4: Parametric vs. nonparametric visual compressor. We follow miniGPT-4 [54] that uses Q- Former pre-trained from BLIP-2 [23] as the parametric compressor (All other aspects are maintained as in LLaVA to ensure fair comparison). Ours: pooling with stride 64 on LLM layer 1 to ensure comparable CRs. Our nonparametric compressor outshines the parametric Q- Q-Former counterpart in terms of both performance and training efficiency.
tokens). Secondly, the performance of MLLMs at the light compression level does not correlate directly with the number of visual tokens, making this observation somewhat unexpected. We attribute this to the MLLMs at this level of compression being relatively insensitive to changes in the compression ratio. This indicates that MILMMs trained at a light compression level will not hurt the model performance at all. For instance, the setting of stride 16 in light compression level attains a 188% CR and also outperforms the baseline LLaVA-v1.55 observations pave the way for developing more systematic training scheme.
across all four metrics. The above
Skise
Tidces
CR
Lakser
TELOP.
Tan
CQA
MMW
SQA
XNE
VQA
POPE
MHB
MMBB
VQA
LLWW
ViWic
SEE
MNMU
Avg
Heasy conprestion in 2
39M
30.3p0
Par
1443
35.13
m
05.12
40.62
620
$472
3596
3/02
122
613
323
714.>
1486,
366
856
6688
19-5.1
17.uz
47.0g
304
65.
34.9
61.5
VnQ
18955
4.77
12A
610
310.
71.bg
1400
S7A mm
16.3g
672
760ur
60.5
46.2
659
56.1 31
61A.
capeesse I2M
(6
12355
1769
$1.320
510
118
33
71.3r
1462
38:2
16l
60.3
788.2
63%
5L4.)
668
618
11530
52.20
624
305
1458
58.3
162.1
65.
ml
78.7
65.0gz
405
618g
13824
158%
Sk.ro
5.40
610
315.
1460g
58.5
$64.33
66.4
596
65.5
40.5
662
55.1
688
Base
(8432
1006
60.50
3.56
1530
626
31.
20
1407
58.3
56.1
66.3
59.4
76.
655
458 06.7
618
Table 5: Training MLLMS with Visual Context Compressor in various compression levels. We report the average results across three runs, with the standard deviation written at the bottom right the average result. In the heavy compression range, the performance is inversely proportional to the compression ratio. In the light compression range. the performance is not sensitive to compression. Performance remains high fir models at the light compression level.
Scalability to Larger Models. As modern multimodal LLMS (MLLMs) continue to grow in size and complexity, it is crucial to determine whether the performance gains observed in smaller models can be extended to larger architectures. This ablation allows us to verify if our compression strategies maintain or even enhance their effectiveness as the model scales, ensuring their applicability to more complex real-world scenarios. As demonstrated in Tab, 6. our four-stage scheme achieved comparable performance with standard training while saving 16%(21. vs 17.6) training time.
Train
Model
#Tokens
CR
Time
GQA MMVei SQA MME VQA POPE
MMB
MMB
VQA
LLAVA"
VisWiz SEED!
MMMU Avg.
LLaVA-13b
18432
100%
21.1h 63.0
35.0
74.1 1503
57.0
86.6
68.2
63.5
79.6
71.0
53.6
66.4
37,9
63.9
Ours
10863
170%
17.6h 63.0 35.4
74.2
1502
56.7
86.8
68.0
63.3
79.7
71.3
53.8
66.4
37.8
64.0
Table 6: Training larger MILMS with LLa Volta. Our method achieves comparable performance across various benchmarks while reducing the training time by 16% (21.1 hours vs. 17.6 hours) and increasing the compression ratio to 170%. These results demonstrate the scalability of our approach to larger models
Compairson with Layer-wise progressive Compression. Given the success of stage-wise com- pression in accelerating training, we hypothesize that it's also beneficial for layer-wise progressive compression. To explore this, we applied nested compressors with varying strides across layers, with smaller strides in the shallower layers, where visual tokens receive more attention. As shown in Tab. 7 we experimented with a multi-stage configuration: layers 0-3 with stride=1, layers 4-11 with stride=2, layers 12-23 with stride=4, and layers 24-31 with SCride= This was compared to single-stage compression setup: layer=8, Stride=((R=2666)) While the progressive layer-wise compression showed superior performance in direct inference, it underperformed when retrained. We

---

attribute this to the compounded pooling of visual tokens across layers, which imposes additional challenges on the model's learning, ultimately leading to suboptimal retraining outcomes.
Comsgrsssor
GQA
SQA
VQJ"
MMB
VQA
LLaVA
SHED
Avg.
Direct Inference
Sinlge Stage
267%
57.8
70.2
52.1
86.0
48.1
58.3
57.0
Malti Stage
260%
61.7
28.9
70.3
1403
35.4
55.1
65.2
57.1
77.7
60.6
49.1
64.8
35.2
00.0
Inference Re-tnoin
Single Stage
267%%
60.7
30.7
71.3
1456
36.9
86.4
64.6
58.0
77.9
67.0
48.8
66.0
35.3
61.3
Stage
7665
70.5
1408
65.4
60.2
Table Comparison between single stage compressor and multi stage compressor. -tati-stage compres- sion outperforming single-stage direct inference across most tasks. However, in retrained models, multi-stage compression only shows marginal improvements, with a slight increase in the average performance.
Furthermore, we conduct an ablation study on the number of iterations different stages (uniform vs. non-uniform stage splitting), which is detailed in the Appendix.
4.5 Extensibility to Video MILMs
We extend our training scheme to VidcoLLaVA [26] and the results in Tab. rreveal similar findings before: the proposed training scheme achieve competitive results while reducing 9% training time. is worth mentioning VideoL.aVV does not support DeepSpeed ZeRO-3, unlike LLaVA, which results in different relative efficiency gains.
MSVD-QA MSRVTT-QA ActivityNet-QA
Average
#Stages
Scheme
#Tokens
CR
TFLOPs
Train-time
Score
Ace
Score
Acc
Score
Ace
Score
Acc
Single
no compression
147456
29.68
40.7h
3.69
69.1
3.48
56.8
3.28
47.5
3.48
57.8
Two
compression
80496
183%
17.73
37.1h
3.71
69.0
3.50
56,9
3.29
47.9
3,50
57.9
Three
compr. deeper
84776
174%
17.29
37.1h
3.73
69.3
3.51
57.2
3.28
47.4
3,51
58.0
Three
compr. wider
83256
177%
16.86
37.0h
3.72
69.0
3.51
57.2
3.29
47.7
3,51
58.0
Four
wider then deeper
88704
166%
18.32
37.2h
3.72
69.1
3.51
57.2
3.27
48.0
3.50
58.1
deeper then wider
86904
170%
37.1h
3.74
56.9
47.8
Table 8: Performance of Volto VideoLLaVA (266) See the definition of each training scheme in Tab, average across stages. To implement multi- stage training. we apply the same compression processing the frames representing the video respectively. The derived six training schemes achieve competitive results while reducing 9% training time.
5 Conclusion
In this work, we conduct two initial studies to investigate and verify the redundancy of visual tokens in multi-modal LLMs. To address this, we propose Visual Context Compressor, straightforward yet effective compression technique that employs simple average pooler, seamlessly integrating into the training MLLMM. This approach enhances training efficiency without compromising performance. To further mitigate the information loss brought by the token compression, we introduce IL multi-stage training scheme that utilizes Visual Context Compressor with a progressively decreasing compression rate. Experimental results various visual question answering benchmarks verify the effectiveness of LLaVolta in boosting performance while demonstrating efficiency gains by reducing training costs by 16% and inference latency by 24%. To the best of our knowledge, we are the first to accelerate the training of multi-modal LLM from the compression perspective. We hope that the proposed Visual Context Compressor and LLa Volta will inspire more in-depth analysis of visual redundancy existing in current MLLMs and call for future designs of efficient training for MLLMs.
Acknowledgement: We thank Zhanpeng Zeng for the discussions regarding the comparison with VCC. We are also grateful for the insightful advice from our anonymous reviewers. This work was supported by Siebel Scholarship and ONR with -00014-23-1-2441.
10

---

References
[1] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican. Reynolds, et al. Flamingo: visual language model few-shot learning. Advances neural information processing systems, 13-23716-23736 2022.
[2] C. Anil, Y. Wu, A. Andreassen, A. Lewkowyez, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur. Exploring length generalization in large language models, arXiv preprint arXiv:2207.04000 2022.
[3] P. Baldi. Autoencoders, unsupervised learning. and deep architectures. In Proceedings of ICML workshop unsupervixed and transfer learning, pages 37-49. JMLR Workshop and Conference Proceedings, 2012.
[4] H. Barlow. Redundancy reduction revisited. Network: computation in neural systems, 12(3):241, 2001.
[5] D. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguisties: human language technologies, pages 190-200, 2011.
[6] Chen, Mei, X. Li, Lu, Q. Yu, Q. Wei, Luo, Xie, E. Adeli, Y. Wang, et al. Transunet: Rethinking the u-net architecture design for medical image segmentation through the lens of transformers. Medical Image Analysis. 97:103280, 2024.
[7] J. Chen, Q Yu, X. Shen, A. Yuille, and L.-C. Chen. Vitamin: Designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[8] J.-N. Chen, Sun, J. He, P. Torr, A. Yuille, and S. Bai. Transmix: Attend mix for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12135- 12144, 2022.
[9] L. Chen, H. Zhao, T. Liu, S. Bai, Lin, C. Zhou, and B. Chang. An image is worth 1/2 tokens after layer 2: huu-and--lay inference acceleration for large vision- language models. arXiv preprint arXiv:2403.06764 2024.
[10] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. E. Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See hutps://vicuaa Imsys. org (accessed 14 April 2023), 2(3)66, 2023.
[11] Z. Dai, G. Lai, Y. Yang, and Le. Funnel-transfore transformer: Filtering out sequential redundancy efficient language processing. Advances in neural information processing systems, 33:4271 14722, 2020.
[12] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Hcigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19999, 2020.
[13] C. Fu, P. Chen, Y. Shen, Y. Qin. M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, et al. Mme: A com- prehensive evaluation benchmark for multimodal large language models, arXiv preprint arXiv:2306.133: 133944 2023.
[14] Y. Goyal, Khot, D. Summers- Stay, D. Batra, and Parikh. Making the in vqa matter: Elevating the role image understanding in visual question answering. In CVPR, 2017.
[15] D. Gurari, Q. Li, A. J. Stangl, Guo, C. Lin, K. Grauman, Luo, and P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617 2018.
[16] J. He, J.NN Chen, S. Liu, A. Kortylewski, C. Yang, Y. Bai, and C. Wang. Transfg: A transformer architecture for fine-grained recognition. In Proceedings of the AAAI conference artificial intelligence, volume 36, pages 852 860, 2022.
[17] K. He, X. Chen, S. Xic, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[18] L. Hou, R. Y. Pang. Т. Zhou, Y. Wu, X. Song. X. Song, and D. Zhou. Token dropping for efficient bert pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. 2022.
11

---

[19] X. Huang, A. Khetan, R. Bidart, and Z. Karnin. Pyramid-bert: Reducing complexity via successive core-set based token selection. arXiv preprint WxXiv:2203.14380. 2022.
[20] D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and composi- tional question answering. In Proceedings of the IEEEJCVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[21] T. Kanungo, D. M. Mount, N. S. Netanyahu, C. D. Piatko, R. Silverman, and A. Y. Wu. An efficient k-means clustering algorithm: Analysis and implementation. IEEE transactions pattern analysis and machine intelligence, 24(7):881-88)) 1-892, 2002.
[22] B. Li, R. Wang. G. Wang. Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[23] J. Li, D. Li, S. Savarese, and S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image enenders and large language models. In International conference on machine learning, pages 197.0-19742. PMIR, 2023.
[24] Y. Li, Y. Du, K. Zhou, J. Wang. W. X. Zhao, and J.-R. Wen. Evaluating object hallucination large vision- lssion-anguuge models. arXiv preprint arXiv:2305 2023.
[25] Y. Li, Y. Zhang, C. Wang, 7. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia, Mini-gemini: Mining the potential of multi-modality vision language models, 2024.
[26] B. Lin, B. Zhu, Y. Ye, M. Ning. P. Jin, and L. Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arX1v:2311.0112, 2023.
[27] H. Liu, C. Li, Y. Li, and Y. Lee. Improved baselines with visual instruction tuning. 2023.
[28] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[29] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua. F. Petroni, and P. Liang, Lost in the middle: How language models use long contexts. arXiv preprint urXiv:2307.03172, 2023.
[30] Y. Liu, H. Duan, Y. Zhang, B. S. Zhang, W. Zhao, Y. Yuan, J. Wang. C. He, 7. Liu, al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0626 2023.
[31] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neurul Information Processing Systems, 2022.
[32] P. Nawrot, J. Chorowski, A. Laácucki, and E. M. Ponti. Efficient transformers with dynamic token pooling. arXiv preprint 2rX1v:211 2022.
[33] OpenAL. ChatGPT. http::/://penaa..commboo////aaaa
2023.
[34] OpenAL. Gpt-4 technical report, 2023.
[35] G. Qin and B. Van Durme. Nugget: Neural agglomerative embeddings of text. In International Conference Machine Learning, pages 28337 28350. PMLR, 2023.
[36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[37] J. W. Rae, A. Potapenko, S. M. M. Jayakumar, and T. P. Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint /91iv:9911.05500 1.05077, 2019.
[38] S. Shen, P. Walsh, K. Keutzer, J. Dodge, M. Peters, and Beltagy, Staged training for transformer language models. In International Conference on Machine Learning, pages 19893-19908. PMLR, 2022.
[39] A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, Parikh, and M. Rohrbach. Towards vqa models that can read. In CVPR, 2019.
[40] G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
12

---

[41] G. Team. T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju. S. Pathak. L. Sifre, M. Rivière, M. S. Kale. J. Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829) 2024.
[42] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, Love, P. Tafti, L. Hussenot, G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson. B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Chooe C. Crepy. D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishehenko, J. Austin, Keeling, Labanowski, J.-B. Lespiau, Stanway, J. Brennan, Chen, Ferret, J. Chiu, Mao- Jones, K. Lee, K. Yu, K. Millican, L. Sjoesund. L. Lee, L. Dixon, Reid, M. Mikula, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotow, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. Mcllroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowice, Y. hui Chen, Z. Ahmed, Z. Gong. T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals. J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, Barral, F. Percira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy. Gemma: Open models based on gemini research and technology, 2024.
[43] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 137711,00 2023.
[44] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykow, S. Batra, P. Bhargava,
S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, Femnandes, J. Fu, W. Fu, B. Fuller, C. Gao. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov. P. Mishra, I. Molybog, Y. Nie. A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[45] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, and I. Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[46] G. K. Wallace. The jpeg still picture compression standard. IEEE transactions on consumer electronics, 38(1 xxxii,iiii,,,,,,, 1992.
[47] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 17453, 2023.
[48] J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2016.
[49] X. Ye, A. Wang. J. Choi, Y. Lu, S. Sharma, L. Shen, V. Tiyyala, N. Andrews, and D. Khashabi. AnaloBench: benchmarking the identification of abstract and long-conter analogies. arXiv preprint a2X7v22402.13370. 2024.
[50] W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 02490, 2023.
[51] Z. Yu, D. Xu, J. Yu, T. Yu. Z. Zhao, Y. Zhuang, and D. Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 9914,, 2019.
[52] X. Yue, Ni, Zhang, Zheng. R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Sun, et al. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[53] Z. Zeng, C. Hawkins, M. Hong, A. Zhang, N. Pappas, V. Singh, and S. Zheng. Vec: Scaling transformers to 128k tokens or more by prioritizing important tokens. Advances in Neural Information Processing Systems, 36, 2024.
[54] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhosciny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv: 10592, 2023.
13

---

Appendix
In the appendix, we provide additional information as listed below:
Alpovviess the additional experimental results.
provides the dataset information and licenses.
A Additional Experimental Results
A.1 Non-uniform Stage Splitting
By default, the training time is evenly divided across each stage. To explore how the compression stage affects total training time, we modify the relative proportion of different stages. This variation tested in the two-stage setup referenced in Tab. [ adjusting from the standard 50% in Stage 1 and 50% in Stage 2 to different distributions. Tab. 9 below displays the results of these experiments.
Stage
Stage 2
#Tokens
CR
GQA
MMVer
SQA
MME
VQA
POPE
MMB
MMBCN
0%
100%
18432
62.0
31.1
70.1
1453.0
58.2
85.9
64.3
58.3
25%
75%
11088
166%
62.1
31.7
70.6
1474.5
58.8
86.4
65.1
59.6
50%
50%
10863
170%
62.2
30.0
70.3
1443.5
57.5
85.8
64.8
59.7
75%
25%
10597
174%
61.6
32.2
70.8
1471.5
57.5
86.6
65.2
58.9
90%
10%
10407
177%
61.2
31.0
70.5
1447.5
56.3
86.4
64.4
56.9
100%
0%
10062
183%
55.9
29.5
64.1
1257.8
49.1
86.6
47.4
29.2
Table Effects of non-uniform stage splitting the two-stage set-up. Performance decreases as the proportion of Stage 2 decreases, albeit the expense of lower compression ratios.
We observe that as the Stage 2 increases from 0% to 100%, there is a gradual decrease in the model's performance across various metrics (such GQA, MMVet, SQA, MME, VQA, POPE, MMB, and MMBCN ). Although there is decline in performance, it is relatively minor when the compression stage makes up 50% of the training duration. However, when the proportion of the compression stage is reduced below 50%, the decline in performance becomes more significant. In conclusion, keeping the compression stage between 0-50% of the training time minimizes performance loss while still achieving significant compression ratios.
A.2 Adaptability to Different Structures.
In addition scaling across model sizes, it is essential evaluate the adaptability of our approach different model structures. As shown in Tab. [00 we conduct an experiment on Mini- Mini-Gemini [25), structurally distinct baseline. Since Mini-Gemini employs multi-resolution visual encoding strategy and Gemma 42] as language model. This ablation experiment assesses LLaVolta's 's compatibility with different sophisticated visual encoding strategies.
Train Time
Model
#Tokens
CR
GQA MMVei SQA MMF VQA" POPE MMR MMBCN
VQA"
LLAVA"
VieWir SEED
MMMU Avg.
MGM-2B
18432 100%
18.1h 60.7
30.1
62.7
1327
57.1
86.0
61.9
50.6
76,3
65.9
48.3
63.8
28.1 1
58.3
Ours
10863 170% 14.8h 58.8
30.2
62.2
1325
54.3
87.0
62.5
52.5
76.3
65.7
48.9
63.1
27.3
58.1
Table 10: Training struturally distinet MILMs with Volta. Comparison of our method with the Mini-Gemini (MGM-20 2B) baseline, which uses a multi-resolution visual encoding strategy. Our approach demonstrates competitive performance while reducing training time by 18% (18.1 hours vs. 14.8 hours) and achieving higher scores. This ablation highlights LLaVolta's 's ability to adapt to different model structures and sophisticated visual encoding strategies.
14

---

B
Datasets Information and Licenses
GQA:
The GQA: 20
dataset, consists of 22M questions about various day-to-day images.
License: N/A
Dataset
website:
html
MM-Vet:
The MM-Vet [50]
dataset, defining 6 core VL capabilities and examines the 16 integra-
tions of interest derived from the capability combination.
License:
Apache License.
LICENSE
Dataset website: ht..ps::/githab..com
SQA:
The SQA: [3] dataset, consisting of 21k multimodal multiple choice questions with a diverse
set of science topics and annotations of their answers with corresponding lectures and explanations.
License:
CC BY-NC-SA
https://
creativecormons.com
Dataset website:
POPE:
The POPE [4] dataset can evaluate the object hallucination in a more stable and flexible
way.
License:
MIT
License.
MIT 1ET----00 ov- fi filetreadme
Dataset website:
MMBench: The MMBench 30 dataset is a collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models.
License: Apache License. httpss///gttthb..cc///e/o..ooooe
Apache-2.0
Dataset website: https: http://///ettaaaacooooooooooo/aa
MMBench-CNN The MMBench-CN [30] dataset is a collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models.
License: Apache License.
Apache 2.0-1 11.0011ov--i #
Dataset website:
MME: The MME [13 dataset containing 30 images with 60 instruction answer pairs for coarse-
grained recognition task; 917 images for fine-grained recognition task; 20 images with 40 instruction- answer pairs for OCR task.
Dataset website:
http:://sh.aaccceeee///////////a mm/
mme/ /EnLLVVAL MME.N. mc
Qwen Vt.ttee htt.:rreeemaatterttabb..ii 1- http::///ithttb.cmmm///ee
License: Tongyi Qianwen LICENSE AGREEMENT.
TextVQA: The TexiVQa 39 dataset containing images with 60 instruction-answer pairs for coarse coarse-grained recognition task; 917 images for fine- fine-grained recognition task; 20 images with 40
instruction-answer pairs for OCR task.
Dataset website: http://///aattaaaaa//eeeeeeeeeee
License: BSD LICENSE. http:///ttttubccomcchobbeeeeeerr main/ LICENSE
VQA-v2: The VQA-v2 14 dataset, containing 265,016 images, dataset containing open-ended questions about images. These questions require an understanding vision, language and common- sense knowledge to answer.
15

---

License: Commons Attribution 4.0 International License.
html
Dataset website: ht..ps://viis.t warpaaalaa.roo.......o...o org/
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec. [ for the main claims and Sec. and Sec. 4 for the detailed contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Sec. for the discussion on the limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: See Sec. results.
for the information needed to reproduce the main experimental
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We open-sourced the full code through github repo: https: http::://qithnn
com/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
public/ http://udddeeee/eeeeeeeee
wwwonsssiissoooooeooocioiiiisiis for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https)
unips.cc/publiin
SerodeSubbe. ss assonPoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec. for all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes))
we report our experiment results with statistics by running times and calculating mean and standard deviation.
Justification: See Sec. 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Sec. for the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httnggtttpp/.//.rrr.......pppppa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Sec. for the discussion on the broader impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: See Sec. 5 for the discussion on the safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Sec. B for dataset licenses.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thcode com dataaettettt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "kP0PuaJZlJ", "reviewer": "Reviewer_dJy4", "summary": "The paper presents a compelling study on redundancy of visual tokens in MLLMs and practical approaches to reduce them. The paper first verifies that one could eliminate up to 70% of visual tokens at testing time via simple average pooling with minial performance degradation. Then they experiments with several approaches for compressing the visual tokens, finding that a simple average pooling works the best. They also propose a staged training recipe, where we could save computation during early training stages and gradually remove compression.", "questions": "1. When doing average pooling, why choose the 1D average pooling instead of 2D average pooling over the grid features?\n2. What is K in table 2? I cannot seem to be able to find discussion on tuning k v.s. stride in Section 4.2.\n3. For ablation, why report only the performance on the 4 benchmarks? I suspect that the compression performance will also be quite dataset-dependent.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is clean and simple. It is nice to empirically verify the redundancy in visual tokens.\n2. The proposed staged training and the discussion around wider then deeper v.s. deeper then wider is interesting.", "weaknesses": "- There seems to be an easy baseline missing from the discussion, such as 2D conv (as in Honeybee). \nHoneybee: Locality-enhanced Projector for Multimodal LLM\n- There are some ambiguous experimental details. Please see questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.\n\nFor weakness 1, we have included 2D-Conv and C-Abstractor (from Honeybee) as a baseline in Table below. The stride of 2D-Conv and C-Abstractor are the same with other compressor to ensure fair comparison under same compression ratio (CR). However, we found that (1) The performance of convolution-based compressors is significantly lower, with 2D-Conv achieving 55.1\\%, C-Abstractor achieving 50.6\\%, and 1D Pool reaching 60.4\\% in average accuracy across 13 benchmarks. (2) there is overhead parameters of conv. We analyze that it is hard to optimize the extra convolution kernel within LLM at the same time (this is an underexplore area). \n\n| Compressor    | #Tokens | CR   | GQA  | MM-Vet | SQA  | MME  | VQA^T | POPE | MMBench | MMB-CN | VQAv2 | LLaVA^W | VisWiz | SEED^I | MMMU | Avg. |\n|---------------|---------|------|------|--------|------|------|-----------|------|------|-----------|------------|-------------|--------|---------|------|------|\n| 2D-Conv       | 2232    | 826% | 58.6 | 28.6   | 71.6 | 1366 | 51.8      | 84.4 | 63.8 | 55.6      | 74.0       | 63.8        | 48.1   | 60.1    | 25.6 | 55.1 |\n| C-Abstractor  | 2232    | 826% | 53.7 | 23.7   | 70.9 | 1209 | 48.7      | 82.8 | 58.0 | 50.3      | 68.2       | 48.6        | 48.0   | 53.4    | 23.9 | 50.6 |\n| 2D Pool       | 2232    | 826% | 57.5 | 28.7   | 71.5 | 1426 | 53.1      | 84.0 | 64.2 | 58.7      | 74.3       | 64.2        | 50.0   | 66.5    | 34.3 | 56.1 |\n| 1D Pool       | 2232    | 826% | 58.3 | 29.2   | 71.4 | 1434 | 53.6      | 83.8 | 64.8 | 58.6      | 74.5       | 65.0        | 49.1   | 66.8    | 35.0 | 56.3 |\n\nIt is worthy mentioning that Honeybee focus on the design of projector outside LLM while our focus is the visual context compression within LLM. Therefore, our method has the potential to accelerate the inference and training of Honeybee (i.e., keep Honeybee's 2D Conv on projector before LLM, and add our 1D pool compressor on visual tokens within LLM). We will include these new results in our revision. \n\nFor question 1, we thank the reviewer for the comments. We add experiment with 2D pooling in the same Table above. As results, 1D pooling perform slightly better than 2D pooling (+0.2\\%) under the same compression ratio. \nThis could be due to the visual tokens being processed in a 1D manner within several LLM layers. Besides, we opted for 1D pooling because it allows for more adaptable compression ratio. For instance, using a 1D pool with a stride can reduce the number of tokens by 2x, whereas a 2D pool with a stride of 2 reduces the token count by 4x. We will include these new results in our revision.\n\nFor question 2, we appreciate the reviewer's feedback. K is set to 2 with stride set to 8 in Table 2, resulting in a compression ratio of 556\\% to approximate [41]’s 514\\% (fixed CR) for a fair comparison of compressor.\n\nFor question 3,  we thank the reviewer for the comments. The previous validated four benchmarks are considered representative because they assess diverse capabilities: GQA evaluates visual understanding, SQA focuses on scientific knowledge across 26 topics, MME tests perception and cognition through 14 subtasks, and MM-Vet measures ensembling skills in recognition, OCR, math, and spatial awareness (encompassing other benchmarks like VQAv2, COCO, and TextVQA). \n\nIn Table 1 of GLOBAL RESPONSE PDF, we have included all 13 benchmarks in our ablation now and evaluated based on the average performance across these benchmarks.  The evaluated 13 benchmarks include not only VQA but also multimodal benchmarks. We achieve consistent conclusions with our original paper for both the ablation study and final results. We will include these new results in our revision.\n\nWe sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.\n\nKind regards,\n\nAuthors of Paper 2615", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qdfukhanOM", "reviewer": "Reviewer_ENde", "summary": "The paper presents a novel approach to reducing redundancy in visual tokens within MLLMs by introducing the Visual Context Compressor (VCC) and Stage-wise MLLM Training. The VCC uses simple average pooling to compress visual tokens during training, enhancing efficiency without sacrificing performance. The Stage-wise MLLM Training progressively reduces compression as training proceeds, ensuring no loss of information during testing. The proposed methods demonstrate significant improvements in training efficiency and performance across multiple benchmarks.", "questions": "Please see the weakness.\n\nMinor issues:\n\n1. What does the #Tokens mean in the tables?  Is it inversely proportional to CR?\n2. How do you compute the average performance in Table 5?\n3. Some suggestions for future improvements: (1) multi-stage compression: use different compressors’ strides in the different positions of the LLM. For example, layer 1-3: stride=1; layer 4-12: stride=2; layer 13-24: stride: 4. (2) extend this technique to other modalities, such as video understanding.", "limitations": "The authors have adequately discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper addresses an underexplored area in MLLMs by focusing on the redundancy of visual tokens and introducing effective compression techniques.\n2. The proposed methods achieve significant reductions in training costs while maintaining or improving performance on various benchmarks.\n3. The paper provides thorough experimental validation across multiple benchmarks, demonstrating the effectiveness of the proposed methods.", "weaknesses": "1. While the paper shows improvements across several benchmarks, the evaluation is primarily focused on visual question answering tasks. Additional evaluation on other multi-modal tasks could strengthen the claims.\n2. The paper mentions potential information loss due to compression but does not provide a detailed analysis/visualization of how this might affect tasks requiring dense visual information.\n3. This paper lacks some essential experiments. (1) The paper does not explore the effect of using larger input resolutions. For example, evaluating the method with 448×448 input images, which contain 2.25 times more visual tokens than the 336×336 input, could provide insights into the method's scalability. Designing a compression setting with a ratio of 225% for this input size and comparing it with the original LLaVA setting (compression ratio of 100%) would be valuable. (2) Testing the method on larger models such as LLaVA-13B or models with more input visual tokens (e.g., LLaVA-Next) could solidify the experimental section and demonstrate the robustness of the proposed approach.\n4. The best stage-wise MLLM training scheme looks difficult to transfer to other model & data settings. The training scheme has too many options and variables. If the model size and dataset size increase significantly, the time cost and computation cost for finding the best scheme can become prohibitively large.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.\n\nTo address weakness 1, we follow the reviewer's suggestion, and add the evaluation of multi-modal tasks in Table 1 of Global Response, especially MMBench and MMMU (A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI)[49].\n\nFor weakness 2, we thanks for the reviewer's comment on information loss. We have discussed this in the limitation section of original paper. The naive compression may not be ideal for tasks requiring dense comprehension, grounded reasoning, and OCR/text capabilities. Nonetheless, our staged training approach, which does not involve compression in the final stage, is capable of managing tasks that require dense visual information. We provide analysis and visualization in Global Response.\n\nTo address weakness 3.1, we try to implement the experiment of larger 448×448 input resolution. However, we note that the default openai-CLIP-336 encoder can only input 336x336 and interpolating positional embedding to 448x448 degrades the performance, resulting in -9% drop in GQA. We are experimenting the dynamic high resolution of LLaVA-NeXT (its training code has not yet been made public as of Aug/8) by employing openai-CLIP-224 to encode four 224x224 images derived from a 448x448 image. We plan to include the large input experiment in the revision once our new results are available.\n\nFor weakness 3.2, we thanks for the reviewer's suggestion.  As LLaVA-NeXT does not make the training code public upon the submission of rebuttal, we are in the progress of reproducing the code. We supplement the experiment of testing the method on larger model of LLaVA-13B in Table 3 of Global Response, and we can achieve 177% compression rate and accelerate training efficiency by 15% while keeping the performance. We will include the new results in our revision.\n\nFor weakness 4, we thanks for the reviewer's suggestion. We aimed at providing comprehensive analysis of design space. We will provide a simple version for the practical use in the near future. For instance, we might implement a universal training schedule that gradually reduces the compression ratio over time. \n\nWe thanks for the reviewer's comments for the minor issues. We address them as follows.\n\nQ1. We thanks for the reviewer's comment. #Tokens means the summation of number of visual tokens in each LLM transformer layer. It is inversely proportional to CR.\n\nQ2.  We mentioned this in line 242: \"When reporting average performance in Tab. 5, the score of MME is normalized by 2000, as its range is from 800 to 2000\". The average performance is simply the mean of the (normalized when it is MME) individual performances.\n\nQ3 (1) multi-stage compression. We thanks for the reviewer's suggestion. The multi-stage compression is a great idea. We follow the suggestion and experiment with the setting of multi-stage (layer 0-3: stride=1; layer 4-11: stride=2; layer 12-23: stride: 4; layer 24-31: stride: 8), resulting in a CR of 267%. We compare this to single-stage compression (on layer 8 stride 8) with a CR of 266%.\n\n| Compressor    | Phase | CR   | GQA  | MM-Vet | SQA  | MME  | VQA^T | POPE | MMBench | MMB-CN | VQAv2 | LLaVA^W | VisWiz | SEED^I | MMMU | Avg. |\n|---------------|---------|------|------|--------|------|------|-----------|------|------|-----------|------------|-------------|--------|---------|------|------|\n| layer=8 stride=8 | inference | 266%| 57.8|25.3|70.2|1337|52.1|86.0|60.4|52.2|74.6|56.0|48.1|58.3|33.3|57.0|\n| layer=8 stride=8| training| 266%| 60.7 | 30.7 | 71.3 | 1456 | 56.9 | 86.4 | 64.6 | 58.0 | 77.9 | 67.0 | 48.8 | 66.0 | 35.3 | 61.3 |\n| multi-stage | inference| 267% | 60.7 | 28.9 |70.3 | 1403 | 55.4 | 85.1 | 65.2 | 57.1 | 77.7 | 60.6 | 49.1 | 64.8 | 35.2 | 60.0 |\n| multi-stage| training| 267% | 60.9 | 29.5 |70.5 | 1408 | 55.9 | 84.8 | 65.4 | 57.4 | 76.6 | 61.1 | 48.9 | 64.7 | 34.9 | 60.2 |\n\nThe multi-stage compressor exhibits strong performance when applied directly during inference, outperforming layer 8 stride 8 by 3% under same CR. However, it's surprising that training the multi-stage compressor yields only a marginal average performance improvement of 0.2%. We analyze that the complexity of multi-stage operations makes the LLM more challenging to train. We will have more future analysis.\n\nQ3 (2) video extension. We thanks for the reviewer's suggestion.  For extension of modalities, we have experimented on video-language understanding (based on video-LLaVA). We observe consistent enhancements (an average improvement of +0.4\\% across three video datasets and a 9% reduction in training time) over the baseline using our new training setting.  We will include the new results in our revision.\n\n\n| Scheme           | #Tokens | CR | TFLOPs | Train-time | MSVD-QA Score | MSVD-QA Acc | MSRVTT-QA Score | MSRVTT-QA Acc | ActivityNet-QA Score | ActivityNet-QA Acc | Average Score | Average Acc |\n|------------------|-------------------|--------------|------------------|------------|---------------|-------------|-----------------|---------------|----------------------|--------------------|---------------|-------------|\n| Video-LLaVA-7B   | 147456      | 100%  | 29.68   | 40.7h      | 3.69     | 69.1        | 3.48            | 56.8          | 3.28                 | 47.5               | 3.48          | 57.8        |\n| Ours             | **86904**  |  **170%** | **18.64**            | **37.1h**      | 3.74      | 69.8    | 3.49           | 56.9          | 3.27                 | 47.8               | **3.50**          | **58.2**   \n\n\nWe sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.\n\nKind regards,\n\nAuthors of Paper 2615", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ruuFJRMZpA", "reviewer": "Reviewer_PsS7", "summary": "This paper shows that visual tokens are redundant in MLLM and can be compreseed by a large ratio without significantly hurting the model performance. Based on this observation, this paper studied several different approaches to compress visual tokens and identified that the simple average pooling method is the most effective one. Based on this, the paper further studied several different stage-wise MLLM training strategies, where the training starts with heavy compression and ends with no compression. The proposed training strategy could save the training cost by 16% while achieving even better performance than the baseline.", "questions": "Please see the weakness section", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The observation that visual tokens are redundant and can be largely compressed without significantly hurting the model performance is good. It could serve as a direction for future works.\n\n2. The paper conducted thorough empirical studies about different compression methods, and proposed several training methods based on visual token compression.\n\n3. The proposed method shows better performance than the baseline while reducing the training cost by 16%.", "weaknesses": "1. It would be better to provide more discussions and investigations on why other visual compressors are significantly worse than average pooling. This may provide more insights on how to design good compressor.\n\n2. The paper only studied LLaVA-1.5-7B. It would be better to show that if the method could scale up to larger models like 13B, 34B, or other structures such as Mini-Gemini [1]. Showing the efficiency on those large variants can better demonstrates the effectiveness of the method.\n\n3. The 16% training efficiency improvement is marginal in practice, especially given that the four-stage training would be more cumbersome compared to the baseline.\n\n[1] Li, Yanwei, et al. \"Mini-gemini: Mining the potential of multi-modality vision language models.\" arXiv preprint arXiv:2403.18814 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive suggestions. We address the concerns raised on a point-by-point basis, including additional benchmarks on the global response PDF. We will include the new results in our revised manuscript.\n\nFor weakness 1: \"more discussions and investigations on why other visual compressors are significantly worse than average pooling\", we thanks for the reviewer's suggestion.  In line 262-263, we analyzed that \"they are ineffective when applied to training because the in-training attention scores are unstable.\". Our insight is that while advanced compressors (e.g., attention-based token pruning) excel in inference-only scenarios (See bottom rows in Table 2), the simple pooling method performs better during training (See top rows in Table 2). We hypothesize that this is because training advanced compressors, such as attention-based pruning, necessitates (1) differentiable token selection and (2) stable attention mechanisms of LM Transformers.\n\n\nFor weakness 2:  \"scale up to larger models like 13B, 34B, or other structures such as Mini-Gemini [1]\", we thanks for the reviewer's suggestion.  To show the scalability, we scale the method up to 13B model, and observe consistent performance and efficiency improvements. \n| Scheme    | #Token | CR | train-time  | GQA  | MM-Vet | SQA  | MME  | VQA^T | POPE | MMBench | MMB-CN | VQAv2 | LLaVA^W | VisWiz | SEED^I | MMMU | Avg. |\n|---------------|---------|------|---|---|--------|------|------|-----------|------|------|-----------|------------|-------------|--------|---------|------|------|\n| LLaVA-13B | 18432             | 100%            | 21.1h      | 63.0 | 35.0  | 74.1 | 1503 | 57.0      | 86.6 | 68.2 | 63.5       | 79.6       | 71.0        | 53.6   | 66.4       | 37.9 | 63.9  |\n| Ours-13B  | 10863             | 170%         | 17.6h      | 63.0 | 35.4  | 74.2 | 1502  | 56.7      | 86.8 | 68.0 | 63.3       | 79.7       | 71.3        | 53.8   | 66.4       | 37.8 | 64.0  |\n\n\nTo show the generalizability to different structures, we supplement the experiment on Mini-Gemini (MGM-2B). Our four-stage compression training delivers comparable results with a 17% increase in training efficiency and significantly fewer vision tokens as input. We plan to incorporate additional structures in future versions.\n\n| Scheme    | #Token | CR   | GQA  | MM-Vet | SQA  | MME  | VQA^T | POPE | MMBench | MMB-CN | VQAv2 | LLaVA^W | VisWiz | SEED^I | MMMU | Avg. |\n|---------------|---------|------|------|--------|------|------|-----------|------|------|-----------|------------|-------------|--------|---------|------|------|\n| MGM-2B      | 18432    | 100% | 60.7 | 30.1 | 62.7 | 1327  | 57.1 | 86.0 | 61.9 | 50.6 | 76.3 | 65.9 | 48.3 | 63.8 | 28.1 | 58.3 |\n| Ours      | 10863    | 170% | 58.8 | 30.2 | 62.2 | 1325 | 54.3 | 87.0 | 62.5 | 52.5 | 76.3 | 65.7 | 48.9 | 63.1 | 27.3 | 58.1 |\n\n\nWe thanks for the reviewer's comments on Weakness 3. It's worth noting that our method not only improves training efficiency by 16% but also enhances average performance by 0.5\\% across 13 benchmarks. To address the cumbersomeness, we are exploring the development of a simple training schedule (linearly decreasing the compression ratio over time; like learning rate schedule), aimed at balancing training efficiency and model simplicity.\n\nWe sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.\n\nKind regards,\n\nAuthors of Paper 2615", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their feedback. Reviewer dJy4 commends the paper for presenting "The paper presents a compelling study. The idea is clean and simple". Reviewer ENde notes "the paper addresses an underexplored area in MLLMs", and Reviewer PsS7 appreciates the thorough empirical studies conducted. Additionally, Reviewer PsS7 suggests that the work could provide a direction for future research, highlighting its potential to influence further advancements.

We thanks the reviewers for the constructive comments. To address them, we provide **a global response PDF (attached to this post)** with more benchmark results (dJy4, ENde), scalability to large 13B model (ENde, PsS7), and additional visual analysis (ENde). We also address each reviewer's questions with new results on a point-by-point basis. 

We sincerely hope our rebuttal addresses your concerns, and we look forward to your feedback. Your response will motivate us to refine our paper into a more solid version.

Kind regards,

Authors of Paper 2615

Meta-review: This paper presents a study on the analysis of redundancy concerning visual tokens and efficient training within large multimodal models. The paper received 2 borderline accepts and 1 weak accept recommendations from reviewers. Positive points included a clean and simple idea, thorough empirical studies on an underexplored area, and significant reductions in training costs while maintaining or improving performance. Negative points included lacking discussions and investigations on comparisons to other visual compressors, missing experiments for more benchmarks, larger input resolutions, and larger models, and marginal efficiency improvements.  There were also questions including the use of 1D average pooling instead of 2D average pooling, and other questions regarding some approach and experiment details.  Although the reviewers did not follow up post-rebuttal, the ACs feel that most of the major concerns and questions were adequately addressed by the rebuttal. Overall, after carefully considering the paper and rebuttal, the ACs feel that the paper's strengths outweigh the negatives, and thus recommend accept. Please incorporate the rebuttal points into the camera ready version.
Decision: Accept (poster)