Efficient Large Multi-modal Models via Visual Context Compression

Authors: jieneng  chen
luoxin  ye
ju  he
zhao-yang  wang
daniel  khashabi
alan  yuille
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 01:23:36

Abstract: While significant advancements have been made in compressed representations for text embeddings in large language models (LLMs), the compression of visual tokens in multi-modal LLMs (MLLMs) has remained a largely overlooked area. In this work, we present the study on the analysis of redundancy concerning visual tokens and efficient training within these models. Our initial experiments
show that eliminating up to 70% of visual tokens at the testing stage by simply average pooling only leads to a minimal 3% reduction in visual question answering accuracy on the GQA benchmark, indicating significant redundancy in visual context. Addressing this, we introduce Visual Context Compressor, which reduces the number of visual tokens to enhance training and inference efficiency without sacrificing performance. To minimize information loss caused by the compression on visual tokens while maintaining training efficiency, we develop LLaVolta as a light and staged training scheme that incorporates stage-wise visual context compression to progressively compress the visual tokens from heavily to lightly compression during training, yielding no loss of information when testing. Extensive experiments demonstrate that our approach enhances the performance of MLLMs in both image-language and video-language understanding, while also significantly cutting training costs and improving inference efficiency.

Main paper: 
 introduction: 
 The advent of LLMs [33, 34, 44] has marked a new era in the field of artificial intelligence and natural language processing. LLMs can play a role as a universal interface for a general-purpose assistant, where various task instructions can be explicitly represented in language and guide the end-to-end trained neural assistant to solve a task of interest. For example, the recent success of ChatGPT [33] and GPT-4 [34] have demonstrated the power of aligned LLMs in following human instructions, and have stimulated tremendous interest in developing open-source LLMs [41, 43] . As the horizon of LLM applications broadens and the availability of open-source LLMs increases, the integration of multi-modality into these models presents a new frontier in expanding their capabilities. Multi-modal LLMs [1, 28, 40, 54] (MLLMs), which can process and understand not just text but also visual information, stand at the cutting edge of this evolution.While MLLMs have made significant strides, a crucial aspect that remains relatively unexplored is the efficient representation and processing of visual information within these models. Substantial efforts [18, 35, 53] have been dedicated to optimizing the efficient representation of text tokens through various compression techniques [18, 35, 53] , aimed at enhancing inference efficiency by The accuracy of the LLaVA-1.5-7B [28] model(without re-train) on the GQA [20] benchmarks varies with different percentages of retained visual tokens. The x-axis represents the percentage of original visual tokens preserved after applying 1D average pooling with varying stride sizes S applied in i-th Transformer layer. Right: Visual tokens receive less attention from the [ANS] token as we go deeper into its layers of LLaVA-1.5-7B model. These findings collectively suggest a significant redundancy within the visual tokens of the MLLMs.attentively selecting important tokens. However, the efficient learning of visual tokens in MLLM has not garnered comparable attention. Naturally, this raises questions about the potential redundancy present in visual tokens and its implications for the overall computational efficiency of MLLMs.We start our work by addressing the question: Are visual tokens redundant in multi-modal LLMs? To explore this, we first experiment with simply reducing the number of visual tokens in a pre-trained LLaVA-1.5-7B [28] at the inference stage via average pooling ( §3.2). As shown in Fig. 1 (left), our initial results demonstrate that eliminating up to 70% of visual tokens by pooling them with a stride of 4 starting from Transformer layer 2 incurs only a minimal performance loss on the GQA benchmark, specifically a 3% accuracy reduction. Additionally, we compute and present the average attention values from the [ANS] token to visual tokens and system prompt tokens across different Transformer layers in the pre-trained LLaVA-1.5-7B [28] . As revealed in Fig. 1 (right; blue trends), the visual tokens are generally less attended to, measured based on average attention from the [ANS] token, as the layers get deeper. These two early explorations indicate significant redundancy in visual tokens.Addressing this, in this work we develop an effective Visual Context Compressor that can be integrated into the training of MLLMs. Surprisingly, a simple average pooler nested in LLMs stands out as the most effective compressor, outperforming the attention-based [18, 53] and parametric [23] counterparts. We attribute this to two reasons: (1) The simple pooling operation makes training stable, whereas prior attention-based approaches [18, 53] are specifically designed for accelerating inference rather than training. (2) Visual tokens in the deeper Transformer layers are less attended to (see Fig. 1 (right)) and particularly redundant, making a simple compressor placed in a deeper Transformer layer effective enough. At a lower training cost, the LLaVA-1.5-7B [28] trained with the proposed Visual Context Compressor is competitive with the non-compressed baseline across various multi-modal benchmarks (e.g., GQA [20] and MM-Vet [50] ). This dual achievement highlights Visual Context Compressor's role as a pivotal advancement in enhancing the efficiency and performance of MLLMs across various multi-modal question-answering benchmarks.To further mitigate the information loss caused by compressing visual tokens, especially under a large compression ratio (CR), we have devised a LLaVA-powered lite training scheme, dubbed LLaVolta, which progressively employs Visual Context Compressor at multiple training stages with different compression ratios ( §3.3). Specifically, LLaVolta progresses through several stages, beginning with a high level of visual token compression and gradually reducing the compression ratio until the final stages, where full visual tokens are utilized. This multi-stage approach allows for adaptive compression levels that ensure training efficiency without losing information at testing, thus maintaining the overall effectiveness of the model.Extensive experimental evaluations of LLaVolta have been conducted on thirteen widely-adopted MLLM benchmarks for both image-language understanding and video-language understanding , showing promising results. We observe that LLaVolta not only enhances the performance of MLLMs, but also achieves a substantial reduction in training costs. These experiments validate the effectiveness of our method, demonstrating its capability to optimize resource utilization while maintaining or even improving model performance.In summary, our paper makes the following contributions:• We present two initial studies to verify the redundancy of visual tokens in MLLMs.• We propose the Visual Context Compressor, a simple yet effective compression technique that utilizes an average pooler, enhancing the efficiency of multi-modal models.• We propose the LLaVolta as an efficient training scheme by leveraging Visual Context Compressor at multiple training stages with a progressively decreasing compression ratio.To the best of our knowledge, we are among the first to explore efficient training of MLLMs.• Extensive experiments show that our approach not only improves the performance of MLLMs in image-language and video-language understanding across various benchmarks but also showcases efficiency gains by reducing training costs by 16% and inference latency by 24%. 
related works: 
 Multi-modal LLMs. The evolution of large language models [10, 33, 34] into their multi-modal counterparts [28, 40] represents a significant leap in their ability to follow instructions and generalize across tasks. This transition has been marked by seminal works such as Flamingo [1] , BLIP-2 [23] and LLaVA [28] , which have extended LLM capabilities to encompass visual tasks, demonstrating impressive zero-shot generalization and in-context learning abilities. Progress in multi-modal LLMs has primarily been driven by advancements in visual instruction tuning [28, 54] , leveraging visionlanguage datasets and refining visual instruction-following data. Additionally, efforts have been made to enhance the grounding capabilities of multi-modal LLMs through the use of specialized datasets aimed at improving task-specific performance. Despite these advancements, the exploration of visual compression within multi-modal LLMs remains relatively underdeveloped. The design and optimization of compression strategies are crucial for maximizing the effectiveness and efficiency of multi-modal LLMs, suggesting a potential area for future research and development.Visual Redundancy. In computer vision, reducing redundancy is crucial for creating efficient yet effective models without losing accuracy [4] . Redundancy in images often arises from the inherent characteristics of natural scenes, including repetitive patterns, textures, and areas of uniform color. These features, while contributing to the richness and detail of visual perception, can lead to inefficiencies in both storage and processing when not adequately addressed. Image compression algorithms [46] can reduce file size by eliminating or efficiently encoding redundant data. These methods take advantage of human visual perception's tolerances to subtly reduce data without significantly impacting image quality. Advanced machine learning models, particularly CNNs and autoencoders [3] , offer sophisticated approaches to minimizing redundancy. Transformers [45] , as a fundamental architecture for LLMs [10, 34] , apply self-attention mechanisms to dynamically bind the most informative parts of tokents. Vision Transformers [6, 7, 8, 12, 16] trained with CLIP objective [7, 36] encode an image to a sequence of visual features for multi-modal LLMs [28] . Nevertheless, visual tokens receive less attention in LLMs due to attention shrinkage [47] , resulting a waste of computation. In this work, we focus on reducing the redundancy of visual tokens in MLLMs.Efficient LLMs. Efficient inference and training for LLMs are important. Compressing input sequences for efficiency reasons in Transformers is not a new idea for NLP. Much work is being done to accelerate the inference of LMs. For example, Pyramid Transformer variants [11] and [19] are proposed in Encoder-Decoder LMs that progressively compress the sequence as the layers grow deeper via pooling or core-set selection. Nawrot et al. [32] propose adaptively compressing the sequence based on the predicted semantic boundaries within the sequence. Rae et al. [37] propose compressing the fine-grained past activations to coarser memories. VCC [53] compress the sequence into a much smaller representation at each layer by prioritizing important tokens. Besides efficient inference, accelerating training for LLMs attracts attention as well. A staged training setup [38] is proposed which begins with a small model and incrementally increases the amount of compute used for training by applying a growth operator to increase the model depth and width. However, efficient training for LLMs in multi-modal scenarios is rarely explored. 
method: 
 In this section, we first introduce an overview of multi-modal LLMs in § 3.1. Then, we define the problem of visual redundancy and introduce Visual Context Compressor in § 3.2. Finally, we present our proposed LLaVolta in § 3.3.We start by reviewing the design of the LLaVA family [27, 28] . For processing an input image X v , we utilize the pre-trained CLIP visual encoder ViT-L/14, as detailed by [36] , to extract the visual feature Z v = g(X v ), where g(.) indicates the visual encoder. To bridge the gap between visual and linguistic modalities, the LLaVA [27, 28] framework as an MLLM implements a straightforward linear/MLP transformation. This involves a trainable projection matrix W, which maps the visual features Z v into the linguistic embedding space, producing language embedding tokens H v = WZ v . These tokens are designed to match the dimensionality of the word embeddings within the LLM.For each image X v , one can generate multi-turn conversation data (X 1 q , X 1 a , • • • , X T q , X T a ) with T as the number of turns. One can organize them as a sequence, by treating all answers as the assistant's response and the instruction X t instruct at the t-th turn as:EQUATIONThis approach establishes a standardized format for the multi-modal instruction-following sequence.It allows for the instruction-based tuning of the LLM to be applied to the prediction tokens, utilizing the model's native auto-regressive training objective. Specifically, for a sequence with length L, the likelihood of the target responses X a is calculated as:EQUATIONProblem Formulation: The redundancy observed in images often arises from inherent traits of natural scenes, including repetitive patterns, textures, and regions with uniform color. While these traits enrich visual perception by offering detail and depth, they can also present challenges in terms of storage and processing efficiency. Considering the inherent limitations of Transformers in handling long sequences [2, 29, 49] , it is critical to minimize any length redundancies to obtain a more effective accuracy/efficiency trade-off.The objective of this study is to decrease the length of visual tokens X v (i.e., its hidden states H v if inside LLMs), while simultaneously maximizing the probability of the target response p(X a |X v , X instruct ) as described in Equation ( 2).A key design change that we introduce is a compressor layer that compresses the dimensions of the visual inputs by reducing the effective number of visual tokens.As depicted in Fig. 2 , the compressor is simply an average pooler in our setting. It is applied to the visual tokens in k-th Transformer layer of an LLM. Formally, given the hidden visual tokens at k-th Transformer layer H k ∈ R B×C×L , the compressor is expected to fulfill the following projection: f : R B×C×L → R B×C×Lout , which results in compressed visual tokens Hk ∈ R B×C×Lout , where L out = L S with s as the compression stride. In §4, we explore multiple variants of compressor f to reduce the token length, including random token dropping [17] with dropping ratio 1 -1 S , K-Means [21] with number of centroids set to N C = L S , attention-based token-centric compression [53] , attention-based token dropping [9, 18] , and average pooling with stride s. To our surprise, we find that the simple average pooler is the most effective compressor for vision tokens within MLLMs, due to its stability during training detailed in § 4.4. Thus, we choose average pooler as the compressor.Note that the proposed Visual Context Compressor can be directly applied to any off-the-shelf MLLMs to assess the visual redundancy, as conducted in §4.2. One can also train an MLLM with Visual Context Compressor to reduce the number of visual tokens while maintaining competitive multi-modal performance. Compression Ratio (CR) 3 . For an LLM with N Transformer decoder layers, the compression ratio for visual tokens can be calculated as:EQUATION)where K is the K-th Transformer layer of a multi-modal LLM; L is the the length of visual tokens input into Visual Context Compressor; L out is the compressed length of visual tokens generated by Visual Context Compressor, as illustrated in Fig. 2 .Our architecture modifications thus far mostly impacts the inference efficiency of MLLM, however, its impact on performance-compression trade-off remains unclear.We will study this question in the context of training MLLMs with a goal of enhancing efficiency without compromising performance. We then move on to further utilize Visual Context Compressor to design an efficient training scheme to incorporates Visual Context Compressor at various stages of the training process.Training with Visual Context Compressor not only facilitates efficient inference but also enhances training efficiency. However, devising an effective training scheme poses challenges when ensuring fair comparisons with the original LLaVA [27] , primarily due to differences in the number of tokens involved in inference. This discrepancy may lead to information loss, particularly when operating under a scenario with a high compression ratio. To tackle this issue, we have developed a lite training scheme for LLaVA, dubbed as LLaVolta, which employs stage-wise visual context compression. Generally, assuming there are N s total stages, stage i involves 1 Ns of the total training epochs with a compression ratio of r i , and the final stage proceeds without any compression. Essentially, as training progresses, i increases while r i decreases.In this work, as depicted in Fig. 3 Training Stage III: No/subtle Compression. The MLLM continues training during the final onethird of the total epochs, with either no compression or subtle compression applied. This stage is designed to align with the inference process, where visual tokens may also undergo compression. By maintaining consistency between training and inference, this approach ensures that critical information is preserved while still allowing for compression, minimizing any potential discrepancies between training and real-world use.Given the above meta framework, we can instantiate a family of training schemes, as demonstrated in Tab. 1. The single-stage (non-compression) scheme is equivalent to the MLLM baseline. For multi-stage training, the compression stage can either go deeper or wider. "deeper" implies an increase in K (Transformer layer), while "wider" means a decrease in the stride of the pooler.  
experiments: 
 In this section, we begin by detailing the experimental setup in § 4.1. Next, we elaborate on the proof-of-concept in Section § 4.2. Following this, we validate the proposed LLaVolta in § 4.3 with an ablation study in § 4.4. Finally, we assess the extensibility to video-language in § 4.5.We adopt the Vicuna-v1.5-7B [10] as the language model, leveraging the LLaMA2 codebase [43] . We leverage the pre-trained CLIP ViT-L/14 [12, 36] with an input resolution of 336 × 336, resulting in 576 visual tokens. We employ the LLaVA framework [27] to connect the frozen CLIP vision encoder and the Vicuna LLMs. Along with the projector, we train the entire LLM instead of parameterefficient finetuning. We follow LLaVA-1.5 [27] to perform data preparation and training schedule for pretraining and instruction tuning. We conduct all the experiments with the machine of 8× Nvidia RTX 6000 Ada. Due to multiple invalid image links in the dataset of instruction tuning stage, the scores of LLaVA-1.5 reported in our analysis are reproduced by ourselves to ensure a fair comparison under the same experimental environment.It is worth mentioning that assessing visual token redundancy only necessitates the inference of existing off-the-shelf models, whereas the other experiments involve the training of multi-modal LLMs, specifically projectors and LLMs.Benchmarks and Metrics: We adopt thirteen benchmarks specifically designed for MLLM evaluation, including GQA [20] , MM-Vet [50] , ScienceQA (SQA) [31] , MME [13] , TextVQA [39] , POPE [24] , MMBench [30] , MMBench-CN [30] , VQA-v2 [14] , LLaVA-Bench-in-the-Wild (LLaVA W ) [28] , VisWiz [15] , SEED-Image [22] and MMMU [52] . [26] to evaluate the models on MSVD-QA [5] , MSRVTT-QA [48] and ActivityNet-QA [51] , where the accuracy and score are assessed using GPT-Assistant. We report the official metrics calculated using the standard implementations provided for each benchmark for a fair comparison. Latency is reported as the time taken during inference until the first answer token is produced. When reporting average performance in To assess the redundancy of visual tokens, we perform average pooling within an off-the-shelf LLaVA-1.5-7B checkpoint at the testing stage, using different pooling stride sizes S across various Transformer layers K. As shown in Fig. 1 , the model still exhibits strong performance even when retaining only 62.5% of the visual tokens (S = 4, K = 16) in the MM-Vet benchmark, without the need for additional training. When adopting the same setting (S = 4, K = 16), a similar trend can be observed in the GQA benchmark as well, where the compressed model only has 1% performance drop than the uncompressed counterpart. Surprisingly, in the GQA benchmark, eliminating up to 70% of visual tokens (S = 4, K = 16) results in a mere 3% decrease in performance. This proof-of-concept shows a certain level of redundancy in the visual tokens within MLLMs.In this section, we present the main results of LLaVolta schemes instantiated in § 3.3. We conduct a thorough evaluation of the multi-modal capability across 13 benchmarks. Tab. 2 demonstrates that our proposed LLaVolta not only consistently lowers training costs by 19% (15.3 hours vs. 12.4 hours) but also surpasses the non-compression baseline. The last-stage-compression training schemes achieves the best performance across thirteen benchmarks and obtains 62.1% average performance, improving LLaVA-v1.5-7B [27] with much less inference TFLOPs and training time. This indicates the necessity of designing an optimally lite training scheme. Choice of Visual Compressors. The design choices include (1) random token dropping, (2) K-Means clustering, (3) average pooling, (4) FastV [9] , (5) VCC [18] , (6) parametric pre-trained Q-Former [23] .We have the following three observations. Firstly, Tab. 3 shows that the attention-based methods, including FastV and VCC win 9/13 best and second best scores, showcasing the high performance when compressing visual tokens in inference. However, they are ineffective when applied to training because the in-training attention scores are unstable. Secondly, and surprisingly, the average pooling obtains the highest scores on eleven out of thirteen benchmarks when it is used to train MLLMs with a high CR. Thirdly, Tab. 4 shows that both Q-Former and average pooling can obtain reasonably good performance when trained with extremely high CRs, and the average pooling performs better with less training cost. The reason could be that the Q-Former resamples tokens outside the LLM, potentially causing the LLM to overlook crucial information relevant to the response. In contrast, our approach employs average pooling subsequent to Transformer layer K, allowing the initial K layers of the LLM to effectively retain important information from uncompressed tokens. Given these three insights, we select average pooling as our favored approach for visual compression.Performance Across Compression Ratios. Herein, we train the multi-modal LLM with our Visual Context Compressor in various settings. As demonstrated in Tab. 5, the proposed method offers certain improvements and trade-offs compared to the state-of-the-art method, LLaVA-1.5-7B. We have the following two observations. Firstly, in the heavy compression level, the performance of MLLM is inversely proportional to the compression ratio (linearly scaling to the number of visual Table 5 : Training MLLMs with Visual Context Compressor in various compression levels. We report the average results across three runs, with the standard deviation written at the bottom right of the average result. In the heavy compression range, the performance is inversely proportional to the compression ratio. In the light compression range, the performance is not sensitive to compression. Performance remains high for models at the light compression level.Scalability to Larger Models. As modern multimodal LLMs (MLLMs) continue to grow in size and complexity, it is crucial to determine whether the performance gains observed in smaller models can be extended to larger architectures. This ablation allows us to verify if our compression strategies maintain or even enhance their effectiveness as the model scales, ensuring their applicability to more complex real-world scenarios. As demonstrated in Tab. Compairson with Layer-wise progressive Compression. Given the success of stage-wise compression in accelerating training, we hypothesize that it's also beneficial for layer-wise progressive compression. To explore this, we applied nested compressors with varying strides across layers, with smaller strides in the shallower layers, where visual tokens receive more attention. As shown in Tab. 7, we experimented with a multi-stage configuration: layers 0-3 with stride=1, layers 4-11 with stride=2, layers 12-23 with stride=4, and layers 24-31 with stride=8(CR=267%). This was compared to a single-stage compression setup: layer=8, stride=8(CR=266%). While the progressive layer-wise compression showed superior performance in direct inference, it underperformed when retrained. We attribute this to the compounded pooling of visual tokens across layers, which imposes additional challenges on the model's learning, ultimately leading to suboptimal retraining outcomes. Furthermore, we conduct an ablation study on the number of iterations in different stages (uniform vs. non-uniform stage splitting), which is detailed in the Appendix.We extend our training scheme to VideoLLaVA [26] and the results in Tab. Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Sec. 5 for the discussion on the broader impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: See Sec. 5 for the discussion on the safeguards.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: See Sec. B for dataset licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.Definition of compression ratio from WikipediaVQA-v2: The VQA-v2[14] dataset, containing 265,016 images, dataset containing open-ended questions about images. These questions require an understanding of vision, language and commonsense knowledge to answer. 
 
 Appendix:  


Decision: This paper presents a study on the analysis of redundancy concerning visual tokens and efficient training within large multimodal models. The paper received 2 borderline accepts and 1 weak accept recommendations from reviewers. Positive points included a clean and simple idea, thorough empirical studies on an underexplored area, and significant reductions in training costs while maintaining or improving performance. Negative points included lacking discussions and investigations on comparisons to other visual compressors, missing experiments for more benchmarks, larger input resolutions, and larger models, and marginal efficiency improvements.  There were also questions including the use of 1D average pooling instead of 2D average pooling, and other questions regarding some approach and experiment details.  Although the reviewers did not follow up post-rebuttal, the ACs feel that most of the major concerns and questions were adequately addressed by the rebuttal. Overall, after carefully considering the paper and rebuttal, the ACs feel that the paper's strengths outweigh the negatives, and thus recommend accept. Please incorporate the rebuttal points into the camera ready version.

Reviews: review: paper_summary: paper_summary: The paper presents a compelling study on redundancy of visual tokens in MLLMs and practical approaches to reduce them. The paper first verifies that one could eliminate up to 70% of visual tokens at testing time via simple average pooling with minial performance degradation. Then they experiments with several approaches for compressing the visual tokens, finding that a simple average pooling works the best. They also propose a staged training recipe, where we could save computation during early training stages and gradually remove compression.
strength_weakness: strength_weakness: - There seems to be an easy baseline missing from the discussion, such as 2D conv (as in Honeybee). 
Honeybee: Locality-enhanced Projector for Multimodal LLM
- There are some ambiguous experimental details. Please see questions.
questions: questions: 1. When doing average pooling, why choose the 1D average pooling instead of 2D average pooling over the grid features?
2. What is K in table 2? I cannot seem to be able to find discussion on tuning k v.s. stride in Section 4.2.
3. For ablation, why report only the performance on the 4 benchmarks? I suspect that the compression performance will also be quite dataset-dependent.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper presents a novel approach to reducing redundancy in visual tokens within MLLMs by introducing the Visual Context Compressor (VCC) and Stage-wise MLLM Training. The VCC uses simple average pooling to compress visual tokens during training, enhancing efficiency without sacrificing performance. The Stage-wise MLLM Training progressively reduces compression as training proceeds, ensuring no loss of information during testing. The proposed methods demonstrate significant improvements in training efficiency and performance across multiple benchmarks.
strength_weakness: strength_weakness: 1. While the paper shows improvements across several benchmarks, the evaluation is primarily focused on visual question answering tasks. Additional evaluation on other multi-modal tasks could strengthen the claims.
2. The paper mentions potential information loss due to compression but does not provide a detailed analysis/visualization of how this might affect tasks requiring dense visual information.
3. This paper lacks some essential experiments. (1) The paper does not explore the effect of using larger input resolutions. For example, evaluating the method with 448×448 input images, which contain 2.25 times more visual tokens than the 336×336 input, could provide insights into the method's scalability. Designing a compression setting with a ratio of 225% for this input size and comparing it with the original LLaVA setting (compression ratio of 100%) would be valuable. (2) Testing the method on larger models such as LLaVA-13B or models with more input visual tokens (e.g., LLaVA-Next) could solidify the experimental section and demonstrate the robustness of the proposed approach.
4. The best stage-wise MLLM training scheme looks difficult to transfer to other model & data settings. The training scheme has too many options and variables. If the model size and dataset size increase significantly, the time cost and computation cost for finding the best scheme can become prohibitively large.
questions: questions: Please see the weakness.

Minor issues:

1. What does the #Tokens mean in the tables?  Is it inversely proportional to CR?
2. How do you compute the average performance in Table 5?
3. Some suggestions for future improvements: (1) multi-stage compression: use different compressors’ strides in the different positions of the LLM. For example, layer 1-3: stride=1; layer 4-12: stride=2; layer 13-24: stride: 4. (2) extend this technique to other modalities, such as video understanding.
limitations: limitations: The authors have adequately discussed the limitations.
paper_summary: paper_summary: This paper shows that visual tokens are redundant in MLLM and can be compreseed by a large ratio without significantly hurting the model performance. Based on this observation, this paper studied several different approaches to compress visual tokens and identified that the simple average pooling method is the most effective one. Based on this, the paper further studied several different stage-wise MLLM training strategies, where the training starts with heavy compression and ends with no compression. The proposed training strategy could save the training cost by 16% while achieving even better performance than the baseline.
strength_weakness: strength_weakness: 1. It would be better to provide more discussions and investigations on why other visual compressors are significantly worse than average pooling. This may provide more insights on how to design good compressor.

2. The paper only studied LLaVA-1.5-7B. It would be better to show that if the method could scale up to larger models like 13B, 34B, or other structures such as Mini-Gemini [1]. Showing the efficiency on those large variants can better demonstrates the effectiveness of the method.

3. The 16% training efficiency improvement is marginal in practice, especially given that the four-stage training would be more cumbersome compared to the baseline.

[1] Li, Yanwei, et al. "Mini-gemini: Mining the potential of multi-modality vision language models." arXiv preprint arXiv:2403.18814 (2024).
questions: questions: Please see the weakness section
limitations: limitations: Yes

Comments: title: Invitation for Discussions and Clarifications
Invitation for Discussions and Clarifications
comment: Dear reviewers,

We wholeheartedly invite and encourage discussion. We would be grateful for any further questions or comments you might have, and are fully prepared to provide additional details or clarifications. Your feedback is invaluable to enhancing the quality of our work.

As AC mentioned, we are dedicated to maximizing the remaining time in the author-reviewer discussion period to address any additional concerns you may have.

Authors of paper 2615
Dear reviewers,

Please carefully read the rebuttal and the other reviews, then reply to the authors indicating whether your questions/concerns have been addressed. If not, please specify which questions/concerns remain unresolved so that the authors have a fair opportunity to address them before the author-reviewer discussion period ends in approximately 36 hours from now (August 13th, 11:59pm AoE).

Best,
AC
Dear Reviewers,

We wholeheartedly invite and encourage discussion. To ensure we have adequately addressed your concerns, could you please review our responses and provide any further comments or questions you may have? We are more than willing to address any additional concerns.  Your time and expertise are sincerely appreciated, and we eagerly await your continued input.

Kind regards,

Authors of Paper 2615