Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

Authors: Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Shaohui Lin
Primary area: foundation or frontier models, including LLMs
Keywords: Efficient Multimodal Large Language Model

Abstract:
Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by $\sim$75\% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the $\sim$50\% computation consumption under decoding without KV cache, while saving $\sim$50\% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://github.com/Osilly/dynamic_llava.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729022785417

thread_type:
official_review

<thread_object>
creator: reviewer_fzrr

<content>
summary:
The paper introduces Dynamic-LLaVA, a dynamic vision-language context sparsification framework aimed at enhancing the inference efficiency of mllms. The main challenge addressed is the increasing computational and memory overhead during inference, particularly as output tokens are generated, which diminishes efficiency benefits of prior vision context reduction methods. Dynamic-LLaVA reduces redundancy in both the vision and language contexts during the prefill and decoding stages, with a specialized sparsification strategy tailored for different inference modes, including decoding with and without KV cache. Experimental results demonstrate that Dynamic-LLaVA significantly reduces computational consumption—by about 75% during the prefill stage and 50% during decoding without KV cache—and saves approximately 50% of GPU memory with KV cache, all while maintaining or even improving model performance. This framework is the first to attempt simultaneous sparsification of both vision and language contexts, achieving efficient inference with negligible performance degradation.

strengths:
1. As an article focusing on efficiency, it grasps the key point very well: solving efficiency problems from the perspective of efficiency. From the numerical results, Dynamic-LLaVA performs well in reducing computation and memory consumption, especially in the pre-filling and decoding stages, reducing the computation by about 75% and the GPU memory overhead by about 50%.

2. The **SPARSIFICATION INFERENCE** part achieves dynamic sparsification of visual and language context by introducing learned predictors, effectively reducing the computational and memory burden in different reasoning modes, making the entire generation process more efficient. I think this is a highlight. Sparsification of tokens during test time is a very useful method, and I personally believe that less is more. These sparse parts should actually improve the representation ability.

3. MaskedSoftmax is a good point to note. From my previous fine-tuning experience, since the generated $logp(y|x)$ is calculated by logits under the corresponding label $y$, the appearance of 0 value in the whole chain process is absolutely harmful. This soft labeling method is very thoughtful.

weaknesses:
1. In the method section, the description of sparse reasoning and end-to-end training relies too much on symbolic expressions and lacks intuitive explanations. It could be aided by adding some simplified explanations or visual flowcharts to help a wider readership understand the complex technical details.
2. You can consider adding some evaluations. I personally feel that the current evaluation benchmarks are not comprehensive enough to explain the performance of MLLMs. And you should analyze as thoroughly as possible. You can refer to the Cambrian-1 [1] standard or try to apply the method to the Cambrian model.
3. In the results of the experimental section, the impact of sparsification on complex tasks (such as generating long texts or multi-round conversations) is not sufficiently verified.

[1] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.

questions:
1. I am curious why it is not tested on the latest versions of llava such as llava-one-vision, llava-next, etc.
2. I don’t quite understand why Table 5 uses GQA and VQAv2 as vision evaluations. I think a more vision-centric dataset such as MMVP is needed.
3. Paper mentions that different sparsification strategies are used in the pre-filling stage and the decoding stage, but the specific differences between the two and the motivation for this design are not clear. In particular, why both visual and language context are sparsified in the decoding stage, while only visual tokens are sparsified in the pre-filling stage, what is the design consideration behind this?
4. In order to avoid unstable training when there are few samples, paper only sparses the samples whose text length exceeds a certain threshold. However, does this strategy ensure the stability of the model when facing inputs of different lengths? In actual use, will the model cause significant performance fluctuations due to changes in input length? If a model behaves unstable for certain input lengths, this may significantly limit its applicability in real-world applications.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565933094

thread_type:
official_review

<thread_object>
creator: reviewer_xrrx

<content>
summary:
Dynamic-LLaVA is a framework for efficient inference in multimodal large language models, dynamically reducing vision context redundancy during the prefill and decoding stages, cutting computation by ~75% in prefill, and saving ~50% computation and memory during decoding, without significant loss in performance.  It also designs the specific training and inference strategy for parallel sparsification, making their approach more efficient.

strengths:
1. The article has a rigorous logical structure, with precise formulations for each method and complete notation. Readers who read carefully can clearly understand the methods explained in the text.

2. The experimental results are excellent; it appears that inference speed and computational cost are significantly reduced with minimal loss in performance.

3. The method design is ingenious; selective approaches are often challenging to train, but the paper provides an effective end-to-end training method that successfully alleviates the gradient flow problem.

weaknesses:
Although the writing is logically clear and the formulas are complete, this level of detail makes it somewhat challenging for readers.  There are too many formulas here. Perhaps reduce them a bit and add more diagrams to aid understanding. Some high-level schematic diagrams could help; Figure 2 aids understanding to some extent, but it’s still not sufficient.

questions:
There aren’t many issues; the method in the paper is logically clear, the experiments are comprehensive, and the performance is excellent.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710355516

thread_type:
official_review

<thread_object>
creator: reviewer_uymq

<content>
summary:
This paper proposes a novel method, Dynamic-LLaVA, which dynamically sparsifies visual context during the prefill stage and vision-text context during the decoding stage of Visual Language Model inference. Specifically, two learnable predictors are designed to identify important visual and language tokens. Additionally, masked softmax is applied during training, and Gumbel-softmax with a Straight-Through Gradient Estimator is used to address gradient flow issues. Furthermore, a batch-parallel strategy is introduced to enable efficient sparsification during inference in the Dynamic-LLaVA framework. Empirical experiments demonstrate that this method can compress approximately 75% of visual context and save around 50% of GPU memory for the KV cache.

strengths:
This paper introduces a novel design for two token predictors, formulating visual and textual context compression as an end-to-end optimization problem. Techniques such as Gumbel-Softmax and the Straight-Through Gradient Estimator are employed to enhance training stability. In addition, parallel sparsification inference optimization would provide an effective solution for model serving.

weaknesses:
1. It is unclear whether the two learnable predictors are applied to each decoder layer. If they are, do they share the same parameters or are they independently parameterized?  If not all layers, please clarify which specific decoder layers the predictors are applied to.
2. Compressing generated text is primarily beneficial for long-generation tasks. Would you please provide specific statistics on the average output length for each task they evaluated? And discuss how the benefits of their method scale with increasing output length.
3. Please quantify the additional training time and computational resources required for this method compared to direct KV cache compression approaches, as well as additional prediction overhead compared normal decoding. This would help readers better understand the tradeoffs.
4. More recent related work about KV Cache compression method should be included, such as TidalDecode (https://arxiv.org/abs/2410.05076) focusing on decoding compression. 
5. A few mathematical representations are not appropriate. For example, in section 3.1, $\mathcal{I}^I = $\{ $1, 2, \cdots, N_{l}^I $}, layer index $l$ should be set for $\mathcal{I}^I $; And in section 3.3.1, all layers share the same binary masks $\mathcal{M}^I and \mathcal{M}^{OT}$? since there are no layer index $l$. 
6. The writing could be improved to make the narrative more concise and clear.

questions:
1. Previous studies have shown that deeper layers exhibit higher attention sparsity. How is sparsification strength adjusted for different decoder layers?
2. Running predictors introduces additional latency. However, the proportion of this latency relative to overall prefill and decoding time is not disclosed. Additionally, in standard LLaVA decoding, previous KV cache is fetched, and byproduct attention scores are accumulated to assess token importance, which is quick and efficient. However, for "Online KV Cache Compression," running predictor incurs additional latency. And only the current token is used to predict all previous tokens, do you think solely performing prediction on a current token provide enough information? Have you analyzed the wrongly evicted tokens?
3. Can this method be applied to language models that utilize Group Query Attention?
4. Have you considered scenarios where long-text context is present during the prefill stage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730757693163

thread_type:
official_review

<thread_object>
creator: reviewer_ldrv

<content>
summary:
In this work, the authors propose Dynamic-LLaVA, a novel approach to enhance both prefilling and generation efficiency in Multimodal Large Language Models (MLLMs). The modified model employs two predictors: an image predictor that dynamically identifies and drops less significant image tokens during prefilling, and an output predictor that filters out non-essential output text tokens. When training these two predictors, the authors propose an end-to-end sparsification training pipeline, which added a mask to the original softmax attention to keep a fixed number of image tokens and generated text tokens. Additionally, the argmax is replaced by Gumbel-Softmax during backward to avoid the gradient flow problem. Extensive experiments show the effectiveness and efficiency of the proposed method.

strengths:
1. This work pioneers the reduction of KV cache length during long text generation for MLLMs, whereas previous research has focused solely on reducing prefilled vision tokens.
2. The end-to-end sparsification training can be viewed as a novel way to enhance the token sparsity of MLLMs.
3. The paper proposed extensive experiments to show the effectiveness of the proposed method.

weaknesses:
1. While Dynamic-LLaVA demonstrates improvements over previous approaches like FastV and PruMerge+, it requires additional post-training, which introduces extra computational overhead. In table 3, it seems that the performance of Dynmaic-LLaVA and FastV are even comparable. Therefore, for real-world deployments where computational efficiency is crucial, implementing existing methods may be more practical. However, this raises an important question: the work would benefit from a clearer justification for why a trainable approach is necessary and what specific advantages it offers over more lightweight solutions. 

2. For the output reduction, as we only need to determine whether preserving the current generated token, this pipeline is the same as some work for LLM, such as StreamingLLM [1], H2O [2]. However, the author only compare the Random and Structure strategies. Therefore, I think this is a well-defined problem for LLMs, and the current experimental results cannot convince me. 

3. The methodology section of this work requires improvement to enhance clarity and comprehension. It is currently challenging for readers to grasp the entire training and inference pipeline quickly. Additionally, several design choices related to the training process lack sufficient explanation, which may leave readers puzzled about the rationale behind them.

4. Furthermore, the name "Dynamic-LLaVA" may lead to misconceptions about the method. It implies that the system dynamically drops the KV cache at each layer; however, it should be clarified that the cache is dropped permanently and cannot be recovered in subsequent layers.

5. The training process is not aligned with the inference process for the output predictor. The former one can see all generated tokens and decide the most important ones, while the latter one can only decide whether to drop the current one based on previous context.

[1] Xiao, Guangxuan, et al. "Efficient streaming language models with attention sinks." arXiv preprint arXiv:2309.17453 (2023).
[2] Zhang, Zhenyu, et al. "H2o: Heavy-hitter oracle for efficient generative inference of large language models." Advances in Neural Information Processing Systems 36 (2024).

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731560324886

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The Preliminary Rebuttal (1/3)

comment:
Dear Reviewers,

Thank you very much for your effort on our paper. 

We are preparing a detailed rebuttal to address the valuable feedback provided by the reviewers. Currently, we have observed that some reviewers have already adjusted their scores and comments before our complete rebuttal. We have decided to release a preliminary rebuttal for several key and commonly raised concerns.  *Please note that this is not the final rebuttal.* A point-by-point response to all reviewers' comments, along with a revised version of the paper, will be submitted subsequently.

Sincerely Yours,

The Authors


> **Comparison with token reduction methods.**

We present the performance of current SoTA and recent token reduction methods. The results in the table are based on evaluations conducted with LLaVA-1.5-7B, using the performance of Dynamic-LLaVA$_{I/T}$ as the base. We report the performance differences of other methods relative to ours. The 'Free' column indicates whether a method is training-free (i.e., can be applied directly to MLLMs without training).

|                             | Free           | Token$\downarrow$ | TFLOPs$\downarrow$ | VQA$^{v2}$  | GQA         | SQA          | VQA$^{T}$    | POPE        | MME            | MMB         | SEED        |
| --------------------------- | -------------- | ----------------- | ------------------ | ----------- | ----------- | ------------ | ------------ | ----------- | -------------- | ----------- | ----------- |
| ***Dynamic-LLaVA$_{I/T}$*** | ***&#10005;*** | ***20%***         | ***25%***          | ***77.9***  | ***61.3***  | ***68.6***   | ***56.5***   | ***85.9***  | ***1501.0***   | ***64.1***  | ***65.0***  |
| (ECCV24) IVTP [1]           | ✕              | -                 | 53%                | 77.8 (-0.1) | 60.4 (-0.9) | 67.8 (-0.8)  | 58.2 (+1.7)  | 85.7 (-0.2) | -              | 66.1 (+2.0) | 56.4 (-8.6) |
| PruMerge+ [2]               | ✕              | 25%               | 25%                | 76.8 (-1.1) | -           | 68.3 (-0.3)  | 57.1 (+0.6)  | 84.0 (-1.9) | 1462.4 (-38.6) | 64.9 (+0.8) | -           |
| VoCo-LLaMA [3]              | ✕              | 22%               | 22%                | 76.9 (-1.0) | 59.8 (-1.5) | -            | -            | -           | -              | 61.0 (-3.1) | 59.1 (-5.9) |
| TRIM [4]                    | &#10005;       | 21%               | -                  | 76.4 (-1.5) | 61.4 (+0.1) | 48.1 (-20.5) | 53.7 (-2.8)  | 85.3 (-0.6) | 1461.3 (-39.7) | 67.4 (+3.3) | -           |
| (ECCV24) FastV [5]          | ✓              | 25%               | 32%                | 75.1 (-2.8) | 57.5 (-3.8) | 68.7 (+0.1)  | 56.2 (-0.3)  | 81.0 (-4.9) | 1458.9 (-42.1) | 63.5 (-0.6) | 62.8 (-2.2) |
| HiRED [6]                   | ✓              | 20%               | 20%                | 74.7 (-3.2) | -           | 66.4 (-2.2)  | 44.2 (-12.3) | -           | -              | -           | -           |
| SparseVLM [7]               | ✓              | 22%               | -                  | 73.8 (-4.1) | 56.0 (-5.3) | 67.1 (-1.5)  | 54.9 (-1.6)  | 80.5 (-5.4) | 1696 (+195)    | 60.0 (-4.1) | -           |

Our proposed Dynamic-LLaVA_I/T achieves the best performance in 34 out of 42 valid results across eight commonly used benchmarks. It is worth noting that, similar to our method, some other approaches, such as PruMerge+ [2], also require additional post-training.  For LLaVA-1.5, both Dynamic-LLaVA_I/T and PruMerge+[2] undergo one epoch of post-training. However, Dynamic-LLaVA_I/T outperforms PruMerge+ in 4 out of 6 results when LLaVA-1.5-7B is used as the baseline. Moreover, for LLaVA-1.5-13B, Dynamic-LLaVA_I/T consistently surpasses PruMerge+ across all benchmarks (see Tab.1 in the main paper). Compared to other token reduction methods, Dynamic-LLaVA_I/T not only reduces 80% of image tokens in the prefill stage but also decreases 50% of output text tokens during decoding, consistently outperforming both training-free and training-required methods in most cases.

For the generation ability benchmarks, it is important to note that Dynamic-LLaVA_I/T achieves comparable PPL and METEOR scores to FastV [5], indicating similar levels of fluency and quality in generated text [8, 9, 10]. This suggests that, while some generated tokens are discarded, the impact on MLLM's ability to generate coherent text is minimal. However, this similarity in scores does not imply equivalent overall performance (see Tab. 1 in the main paper, where Dynamic-LLaVA_I/T outperforms FastV in 14 out of 16 results on the LLaVA-1.5 experiments).  Additionally, Dynamic-LLaVA_I/T further reduces 50% decoding computation (without KV cache) and GPU memory overhead (with KV cache) compared to FastV [5].
</content>
</thread_object>

</thread>
<thread>
creation date:
1732304335870

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
[2024/11/22 AOE] Summary

comment:
Dear Reviewers,

We thank all reviewers for your insightful and constructive comments! We have submitted the point-by-point responses to all reviewers' comments, along with a revised version of the paper **(with blue text)**. If there is any other question, please feel free to let us know and we will try our best to provide satisfactory answers! ***If your concerns have been addressed, we respectfully ask the reviewer to consider raising the score.***

Please allow us to highlight the contributions of our work. **Dynamic-LLaVA is the first MLLM acceleration framework that simultaneously sparsifies both vision and language contexts while integrating inference efficiency optimization across different MLLM inference modes into a unified framework.** This design enables Dynamic-LLaVA to achieve additional inference efficiency throughout the entire generation process, surpassing both MLLM token reduction methods (additional inference efficiency in decoding with/without KV cache) and LLM KV cache compression methods (additional inference efficiency in prefill and decoding without KV cache). Moreover, Dynamic-LLaVA demonstrates significant performance advantages. Therefore, **we believe that our work provides a clear and substantial contribution to the research community.**

Menwhile, we provide a summary of the specific revisions to the paper below.

Sincerely Yours,

The Authors

> **Additional results.**

- [Reviewer LdRV] Additional sparsification training for FastV (Tab. 1 and Appendix A.4.2).
- [Reviewer LdRV] Evaluate LLM KV cache compression methods in MLLM vision understanding and generation ability benchmasks (Tab. 1, Tab. 3 and Appendix A.4.2).
- [Reviewer LdRV] Evaluate the effect on the MaskedSoftmax operation (Tab. 7 and Appendix A.2).
- [Reviewers UyMq, FzrR] Evaluation of the text context sparsification in prefill (Tab. 9 and Appendix A.3.3).
- [Reviewer FzrR] Evaluation of vision-centric vision understanding benchmarks (Tab. 1 and Tab. 5).
- [Reviewer FzrR] Evaluation of complex generation ability benchmarks (Tab. 3 and Appendix A.4.4).

> **Additional details and discussions.**

- [Reviewer LdRV] Comparison with more SoTA vision context sparsification methods (Tab. 11 and Appendix A.5.1).
- [Reviewers LdRV, UyMq] Inferency efficiency discussion of Dynamic-LLaVA and MLLM token reduction methods (Appendix A3.1).
- [Reviewers LdRV, UyMq] Detailed discussion of Dynamic-LLaVA and LLM KV cache compression methods (Fig. 4, Sec. 2.2 and Appendix A.3.2).
- [Reviewer UyMq] Benchmark statistics and inference benefit discussion (Tab. 8 and Appendix A.3.1).
- [Reviewer UyMq] Details of additional training time (Tab. 15 and Appendix A.5.3).
- [Reviewer UyMq] Details of the lantency when decoding with KV cache (Tab. 14 and Appendix A.5.2).

>  **Writing.**

- Add more explanations for the predictor mechanism (Sec. 3.3.1, Sec.3.3.2 and Fig. 2).
- Add more explanations for the end-to-end sparsification training (Sec. 3.3.3, Sec. A.2 and Fig. 3).
- Add more intuitive figures (Fig. 3 and Fig. 4).
- Move the ''Parallel Sparsification Inference Optimization'' section focused on engineering optimizations to the Appendix (Appendix A.1).

> **Code.**

The implementationsof Dynamic-LLaVA and other methods (e.g., FastV, H2O)  are available at the Anonymous GitHub (https://anonymous.4open.science/r/ICLR2025_submission4573_full_codes-0F42/README.md). We provide the complete codes to facilitate a thorough review of our implementations. Should you require additional details, such as weights or logs, please feel free to contact us. We are committed to ensuring the authenticity and reliability of our results.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Dynamic-LLaVA, a method for efficient multimodal large language models through dynamic vision-language context sparsification. The framework reduces computation significantly during both the prefill stage and decoding stage, while maintaining performance. Reviewers raised concerns about the necessity of the trainable approach, post-training overhead, and alignment between training and inference. The authors addressed these concerns, leading to scores of 6, 6, 6, 6. 

Given the promising results in computational efficiency and memory savings, the AC acknowledges the significant contribution of this work to the field of multimodal large language models, especially in terms of optimizing vision-language context sparsification. With an average score of 6, the paper is deemed competitive for this year's ICLR conference. The AC agrees to accept the paper, with the recommendation to integrate the rebuttal content into the final version.