From Bulk to Budget: Best Practices To Compress Multimodal Large Language Models

Authors: Yiran Huang, Lukas Thede, Massimiliano Mancini, Wenjia Xu, Zeynep Akata
Primary area: foundation or frontier models, including LLMs
Keywords: Multimodal large language models, model pruning, knowledge distillation, model compression

Abstract:
Multimodal large language models (MLLMs) are increasingly developed to meet diverse deployment needs, varying in scale and computational demand. While recent research has focused on building MLLMs from Small Language Models (SLMs), these efforts remain limited in flexibility and are still data- and compute-intensive. In this paper, we present the first comprehensive study on flexibly compressing and recovering existing MLLMs in a data-efficient manner. Hence, we address a critical gap in the literature by empirically analyzing best practices for adapting to specific hardware or resource limitations. Our study investigates pruning and knowledge distillation techniques, examining their impact on downstream performance across various model compression strategies, including pruning paradigms, recovery training schemes, and data requirements. Key findings reveal that widthwise pruning is particularly effective in resource-constrained scenarios. For smaller compression ratios, finetuning the multimodal projector alone can restore most performance, while combining finetuning with hidden state knowledge distillation proves most effective across all compression levels. Notably, we demonstrate efficient model downsizing using as little as 5% of the original dataset for moderate compression. Our analysis suggests best practices for compressing MLLMs for resource-efficient deployment. With our best practices, Bunny-v1.0-3B retains over 95% of its original performance, while LLaVA-v1.5-7B maintains more than 97%, with compression ratios below 30%.

Decision: reject
Reviewer scores: 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730342006287

thread_type:
official_review

<thread_object>
creator: reviewer_zh8e

<content>
summary:
This paper provides a recipe to compress MLLM by evaluating the width-wise pruning and layerwise pruning, finding that widthwise pruning is better. Then, by recovery strategies like SFT and KD to restore the performance.

strengths:
- This paper is easy to understand and follow. 
- This paper provides concrete best practices for practitioners

weaknesses:
- The experiments are not comprehensive with just limited model selection. Please include newer MLLM architectures like InternVL, CogVLM, MiniCPM-v; 
- The title of this paper claimed "Best Practices To Compress MLLM". However, this paper only focuses on pruning and then knowledge distillation. Additionally, the pruning only focuses on LLM and there is no experiment on pruning Vision encoder. It is a little bit overclaimed when you do not involve other compression methods like quantization or low-rank factorization. This paper should dive into compression and provide more detailed insight for readers. 
- Limited novelty: pruning+knowledge distillation are the common techniques on LLM, as presented in Sheared LLaMA and Minitron. This paper has limited novelty as there is no difference. 
- Limited comparison with other compression methods: this paper proposed several techniques like width/depth pruning and KD. This paper did not involve any compression method with other methods.

questions:
See Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562592522

thread_type:
official_review

<thread_object>
creator: reviewer_j6jh

<content>
summary:
This paper presents a comprehensive study on compressing Multimodal Large Language Models (MLLMs) while maintaining performance. The authors investigate two pruning strategies (widthwise and layerwise) and various recovery methods (supervised finetuning and knowledge distillation) across different compression ratios. They conduct experiments on two MLLMs (LLaVA-v1.5-7B and Bunny-v1.0-3B) and provide best practices for MLLM compression based on their findings.

strengths:
The paper provides extensive empirical evaluations across different compression ratios, recovery strategies, and data requirements.

The experiments are well-designed and cover multiple dimensions: pruning methods, and data efficiency.

weaknesses:
1. The authors' claim of being the first to investigate general-purpose MLLM compression overlooks existing work, particularly the survey paper on efficient MLLMs [1], which contradicts the authors’ claim. The literature review could be more comprehensive, especially in acknowledging related work in MLLM efficiency. 

2. The technical contributions largely adapt existing LLM compression techniques to MLLMs without introducing significant novel methods. 

3. The findings mostly confirm expected behaviors from LLM compression research. The paper primarily combines existing techniques rather than introducing new methodological advances. The exploration could better highlight MLLM-specific challenges and solutions that differentiate it from general LLM compression


[1] Efficient Multimodal Large Language Models: A Survey

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646202856

thread_type:
official_review

<thread_object>
creator: reviewer_t1q7

<content>
summary:
This paper addresses the challenge of compressing multimodal large language models in a data-efficient way, introducing structured pruning and recovery techniques for resource-constrained deployments. The paper explores layerwise and widthwise pruning strategies, with a focus on identifying effective recovery techniques, such as finetuning and knowledge distillation. Key findings include the effectiveness of widthwise pruning in low-resource scenarios and the combination of projector finetuning with hidden state knowledge distillation for optimal performance recovery. The study provides insights into best practices for compressing MLLMs while maintaining high performance across a variety of tasks.

strengths:
1. The paper presents a unique combination of widthwise and layerwise pruning for MLLMs, complemented by targeted recovery strategies like finetuning and knowledge distillation. This distinguishes the work from typical compression techniques that rely solely on one pruning or finetuning method, offering a more adaptable framework for diverse deployment needs.
2. This ppaer emphasizes data-efficient model recovery, highlighting scenarios where only 5% of the original data suffices to restore a substantial portion of the model’s performance, making the proposed method practical for environments where labeled data is scarce or costly, potentially expanding its applicability in low-data or real-time contexts.
3. Extensive experimentation across two MLLMs with varying compression ratios, recovery techniques, and data requirements provides a detailed view of how different configurations affect model performance. The ablation studies also offer a deeper understanding of the benefits and limitations of each pruning and recovery method.

weaknesses:
1. The proposed techniques are evaluated within the context of MLLMs; however, they lack comparisons with other prevalent compression methods. This absence makes it challenging to assess their effectiveness against existing solutions, especially as structured pruning and knowledge distillation are already well-established in the field.
2. While the framework demonstrates potential at lower compression ratios, its performance gains are limited at higher compression ratios. This limitation may reduce the practical appeal of the method for applications that demand more aggressive compression while still preserving task performance.
3. Although the proposed combination of multiple pruning and recovery strategies is thorough, it increases implementation complexity. The reliance on complex data structures and recovery steps may hinder deployment. An author should clarify this.

questions:
1. What are the limitations of the chosen pruning strategies for different types of tasks within MLLMs?
2. Can the proposed data-efficient recovery techniques be generalized to other model architectures or require specific adjustments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732274326791

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Author Response

comment:
We thank the reviewers for their thoughtful feedback, suggesting several improvements to our work. 
We are glad that the reviewers recognized our contributions listed below:
* This paper provides extensive experimentation across two MLLMs with varying compression ratios, recovery techniques, and data requirements (T1Q7 and J6JH).
* The experiments are well-designed and cover multiple dimensions: pruning methods, and data efficiency (J6JH). This paper provides a detailed view of how different configurations affect model performance (T1Q7) and concrete best practices for practitioners (zH8E).
* This paper emphasizes data-efficient model recovery, highlighting scenarios where only 5% of the original data suffices to restore a substantial portion of the model’s performance (T1Q7).

In response to the reviewer’s comments, we conducted new experiments and analyses to address the raised concerns and further strengthen our contributions.

* Combining quantization with pruning. To expand the scope, we included 8-bit quantization experiments for the LLaVA model and its pruned variants. These experiments demonstrate that quantization reduces the memory footprint by up to 44.5% with minimal performance loss (0.43 percentage points), while combining quantization with structured pruning achieves further reductions of 40–44% at compression ratios of 15% and 30%, with performance losses of only 0.4 and 1.3 percentage points(pp). These findings highlight the complementarity of quantization and pruning as effective compression techniques.

* New MLLM models. We extend our study to the Mini-InternVL-Chat-4B-V1-5 model, confirming the generalizability of our methods. Widthwise pruning consistently outperforms layerwise pruning without recovery training, retaining 97.4% of original performance at a 15% compression ratio. Like the previous model, InternVL benefits from finetuning the projector at small compression ratios, but finetuning both the projector and LLM remains necessary for larger ratios. Additionally, incorporating a distillation loss on intermediate features consistently enhances recovery results, mirroring earlier findings.

* Revised presentation. We also revised the related work section to include recent studies and clarified the distinct focus of our work on compressing existing MLLMs rather than building efficient models from scratch. Finally, we emphasize the practicality of our methods for deployment, showcasing the data efficiency of our recovery strategies, which use only a fraction of the original training data.

In summary, our study systematically evaluates structured pruning and recovery methods for MLLMs, addressing unique challenges like modality misalignment and providing best practices for resource-constrained deployments. We believe these additions further strengthen the contributions of our work. 

We would be happy to address any additional comments from the reviewers in the rest of the rebuttal period.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732474453983

thread_type:
official_comment

<thread_object>
creator: area_chair_88xx

<content>
comment:
Dear Reviewers,

This is a friendly reminder that the discussion period will end on Nov 26th (Anywhere on Earth). If you have not already, please take a careful look at the other reviews and author responses, and comment on whether your original rating stands. Thank you.

Best, AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732809919426

thread_type:
official_comment

<thread_object>
creator: area_chair_88xx

<content>
comment:
Dear reviewers,

This is a friendly reminder that the discussion period has been extended until December 2nd. If you haven’t yet, we kindly encourage you to review the authors' rebuttal and messages at your earliest convenience and confirm whether your comments have been adequately addressed.

We greatly appreciate your service to this process.

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates efficient compression techniques for MLLMs, focusing on two key pruning strategies, width and layerwise.  The paper received scores of 6,3,5.  Mentioned strengths include extensive and well-designed experiments, and the emphasis on data-efficient model recovery.  Mentioned weaknesses include limited novelty and technical contribution, some overclaimed statements, and missing important experimental comparisons.  The rebuttal and discussion by the authors included new experiments (including combining quantization with pruning, and more MLLM models) and revised presentation.  While the rebuttal and discussion addressed some concerns, other including limited novelty and technical contribution remained.  After carefully considering the paper, rebuttal, and discussion, the AC does not feel that the paper is ready for acceptance to ICLR.