Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification
Authors: Wenxuan Huang
Zijie Zhai
Yunhang Shen
Shaosheng Cao
Fei Zhao
Xiangfeng Xu
Zheyu Ye
Shaohui Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Efficient Multimodal Large Language Model
Primary area: foundation or frontier models, including LLMs
TL;DR: Dynamic-LLaVA is the first MLLM acceleration framework that simultaneously sparsifies both vision and language contexts while integrating inference efficiency optimization across different MLLM inference modes into a unified framework.

Abstract: Multimodal Large Language Models (MLLMs) have achieved remarkable success in vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding, directly affecting the efficacy of MLLMs. Existing methods attempt to reduce the vision context redundancy to achieve efficient MLLMs. Unfortunately, the efficiency benefits of the vision context reduction in the prefill stage gradually diminish during the decoding stage. To address this problem, we proposed a dynamic vision-language context sparsification framework Dynamic-LLaVA, which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLaVA designs a tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice, Dynamic-LLaVA can reduce computation consumption by $\sim$75\% in the prefill stage. Meanwhile, throughout the entire generation process of MLLMs, Dynamic-LLaVA reduces the $\sim$50\% computation consumption under decoding without KV cache, while saving $\sim$50\% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaVA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation or even performance gains compared to the full-context inference baselines. Code is available at https://github.com/Osilly/dynamic_llava.

Published as a conference paper at ICLR 2025
DYNAMIC-LLAVA:
EFFICIENT MULTIMODAL LARGE LANGUAGE MODELS VIA DYNAMIC VISION-LANGUAGE CONTEXT SPARSIFI-
CATION
Wenxuan Huang Zijie Zhai Yunhang Shen Xiangfeng Xu'
Shaosheng Cao
Fei Zhao
Zheyu Ye
Shaohui Lin 1,5M
1 East China Normal University
2Xiamen University "Xiaohongshu Inc.
*Nanjing University
$ "eey Laboratory of Advanced Theory and Application in Statistics and Data Science, MOE, China os osilly0616 60 com, chachuilin00 7 gmail. com
ABSTRACT
Multimodal Large Language Models (MLLMs) have achieved remarkable suc- cess vision understanding, reasoning, and interaction. However, the inference computation and memory increase progressively with the generation of output tokens during decoding. directly affecting the efficacy of MILMs. Existing meth- ods attempt to reduce the vision context redundancy achieve efficient MLLMs. Unfortunately, the efficiency benefits the vision context reduction in the pre- fill stage gradually diminish during the decoding stage. To address this prob- lem, we proposed dynamic vision- lision-language context sparsification framework Dynamic-L.LaVA which dynamically reduces the redundancy of vision context in the prefill stage and decreases the memory and computation overhead of the generated language context during decoding. Dynamic-LLavA designs tailored sparsification inference scheme for different inference modes, i.e., prefill, decoding with and without KV cache, to achieve efficient inference of MLLMs. In practice Dynamic-LLaVA can reduce computation consumption by ~55 % in the prefill stage. Meanwhile, throughout the entire generation process of MILMs. Dynamic- LLaVA reduces the ~00% computation consumption under decoding without KV cache, while saving ~5% GPU memory overhead when decoding with KV cache, due to the vision-language context sparsification. Extensive experiments also demonstrate that Dynamic-LLaWe VA achieves efficient inference for MLLMs with negligible understanding and generation ability degradation even perfor- mance gains compared to the full-context inference baselines. Code available at 1lava
https://giihus: com wttp://www........ll/../////////
INTRODUCTION
Large Language Models (LLMss) have achieved outstanding performance and made a significant impact in real-world applications (Zheng et al., 2023 Team 2023 Touvron et al., 2023a Achiam al. 2023 Jiang al. 2024). In particular, within the vision- language multimodal fields, the LLa VA paradigm (Liu et al.] 2024b Bai et al. 2023 Zhu et al. 2023 Zhao et al. 2024) has emerged as the mainstream approach of Multimodal Large Language Models (MLLMs). This paradigm involves mapping visual data, through feature extractors and projecting embeddings, into the same feature distribution of LLMs for processing. It has shown notable success enabling general capabilities in vision understanding, reasoning, and interaction.
However, L.MSs often adopt the decoder-only Transformer as the base architecture and typically contain an extremely large number of parameters. As output tokens are generated during decoding, the computational consumption progressively increases, leading to a substantial computational burden Vaswani 2017). To alleviate this issue, modern LLMs frequently employ the KV cache
"Equal contribution.
"CCoresponding authors.

---

Published as a conference paper at ICLR 2025
Eell
PLoWie (er Cerhe)
(9)
SFU Hemory (we EV Gache)
PLOPss (wrl Goche)
(b)
GPU Herrery (wlo Geche)
Luaft
16
Racdy
Opear (hann
605
1.2
GP) neteer
500
400
400-
0.4
144
41.
36
54
2000
Total tuken
07/08
Toral tokkn
20:M0
$11000
Tota token
"ottl
Figure 1: The entire generation process of MLLMs. As generation progresses, the primary resource overheads of MLLMs under decoding with and without KV cache modes are GPU memory overhead and computation consumption, respectively. Previous vision context sparsification methods achieved initial inference efficiency through vision context sparsification. However, these benefits gradually diminish as decoding continues. The results are measured in one A 100 (80G) and the batch size is fixed to 8. "OOM" means the generation process has failed due to the out of GPU memory.
technique during decoding, which accelerates inference speed by storing previously computed KV activations and reusing them in subsequent decoding steps. helps reduce redundant computations but introduces significant GPU memory overhead to store activated intermediate variables (Luohe et al. 2024 Radford et al. 2019). Moreover, the GPU memory overhead gradually increases in the decoding progresses. This problem becomes even more pronounced in MLLMs within the LLaVA paradigm, which is often combined with LLM, requiring significant resources for inference.
To achieve efficient MLLMs, recent works attempted to reduce the number of image tokens pro- cessed by LLMs, which decreases computation and memory overhead while maintaining model performance (Jin et al., 2024). One category of these methods (Chen et al. 2024a| Shang al. 2024; Ye et al. 2024| Arif et al. 2024, Song et al. (024)) selects a subset of image tokens feeding to L.Mss, e.g., FastV (Chen et al. 2024a) uses the full full-context attention matrix determine which image tokens feed to LLMs. We refer to this approach as vision context sparsification. Another category (Li al., 22044; Kar al., 2024 Li al. 2024b) modifies the vision feature extractors or projectors to generate few number of image tokens. We refer to this strategy as efficient vision encoders/projectors. However, reducing image tokens primarily accelerates the prefill stage. We argue that the efficiency benefits of the image token reduction in the prefill stage gradually diminish during the decoding stage. As shown in Fig. [ the previous sfate-of-the-arr art (SoTA) methods with image token reduction, e.g., FastV (Chen et al. (024a), significantly reduce the number of image tokens during the prefill stage, leading to substantial resource savings. Unfortunately, as decoding progresses, the actual benefits of this reduction diminish. This is because, in the decoding stage, the computation cost increasingly shifts towards the autoregressive generation of the output text tokens, where the reduced image tokens have less impact on the overall efficiency. As result, the initial speedup from fewer image tokens becomes less dominant as decoding progresses. This phenomenon discussed in detail in Sec. 3.2
To address the above problem, we propose a novel dynamic vision-language context sparsification framework, named Dynamic-LLLa VA. Specifically, we introduce two learnable predictors sparsify the vision and language contexts in both prefill and decoding of MI.MM, respectively (Sec. 1. The tailored sparsification inference scheme designed for different inference modes within MLLMS. i.e., prefill, decoding without KV cache and with KV cache (Sec. 3.3.2) During the training process, the proposed Dynamic-L LLaV employees masked softmax to isolate the influence of non-essential tokens on the important tokens and the Gumbel-Softma
(Jang et al. 2016) with Straight- Straight-Through Gradient Estimator (Bengio et al. (013)) to avoid the gradient flow problem, which enables end-to-end optimization (Sec. 3.3.3.. Last but not least, we further develop the batch-parallel strategy for the sparsification inference of the Dynamic- VL framework, which fully leverages GPU hardware advantage under batch parallel inference conditions (Sec. A.I. Our framework can be casily integrated into MLLMS (e.g., LLaVA (Liu et al. 2024b a)) and efficient vision encoder/projector methods for MILMSs (e.g., TokenPacker (Li et al. 2024a)) to achieve more efficient inference.
Extensive experiments conducted on the vision understanding benchmarks and the generation ability evaluations demonstrate that the proposed Dynamic-LLat -LVVA framework achieves efficient inference throughout the entire MLLM generation process with negligible performance degradation.Il In prac- tice, the computation consumption of the prefill stage is reduced by 75 % via the vision context
2

---

Published as a conference paper at ICLR 2025
sparsification. Meanwhile, during the entire MLLM generation process, through the comprehensive vision-language context sparsification, the proposed Dynamic-LLawe reduces computation cost by ~50% when decoding without KV cache. while saving ~50 % GPU memory overhead with KV cache. To the best of our knowledge for efficient MLLMs, Dynamic-LLaaA is the first framework which attempts simultaneous sparsification of both vision and language contexts. We believe that our work can inspire new insights for the research community.
2
RELATED WORK
2.1 TOKEN REDUCTION FOR EFFICIENT MLLMS
The existing vision context sparsification (Chen et al. 2024a; Shang et al. 2024 Ye et al. 2024 Arif
al. 2024 Song al. 2024) and efficient vision encoder/projector (Li al. 22024a] Kar et al. 2024
Li et al. 2024b Bai et al. 2023 Cha et al. 2024 Chen et al. 2024b Chu et al. 2024)) methods have attempted reduce the image token to accelerate the prefill stage. However, with the decoding of
MLLMs, the efficiency benefits gradually diminish during the decoding. Our proposed Dynamic-
LLaVA framework achieves consistently efficient inference throughout the entire generation phase via the vision-language context sparsification. The detailed discussions are presented in Appendix A
2.2 KV CACHE COMPRESSION FOR EFFICIENT LLMs
Some previous works (Zhang et al.
2024b 2023
Li et al. 2024c Xiao et al. 2023 Ge et al. 2023
Han et al., 20233 Yang et al. 2024 Zhang et al., 2024a) compress KV cache in order to reduce the GPU memory overhead of LLM inference during decoding, which has similar goal to our Dynamic- LLaVA when considering decoding with KV cache. However, these methods typically necessitate using the subset of past generated KV cache to facilitate the selection of critical activations for participation during inference. In other words, it is imperative to obtaion the generated KV cache to make the decision for which activations should be removed during LLM inference. Dynamic-LL when decoding with KV cache, can be regarded as "online KV cache compression", i.e., it utilizes the features of the current output text token to determine whether retain the generated activations without relying on past KV cache. This capability is crucial, as it enables Dynamic-LLaVA to enhance efficiency during the stages of prefill and decoding without KV cache, which do not involve KV cache utilization. Moreover, the additional efficiency gained in prefill and decoding without KV
cache modes of comparable importance for MLLMs (Liu al.) 2024b] [i et al.
2024d] Chen et al.
2024a Huang et al.) 2024a Cha et al.) 2024 Leviathan et al.) [0033] Chen et al. 2023a Liu et al.
2023a). A detailed discussion the proposed Dynamic-LLa VA framework and traditional
L.M KV
cache compression methods is provided in Appendix A.3.2
METHOD
3.1 PRELIMINARIES AND NOTATIONS
Multimodal large language model (MLLM), egg. LLaVA (Liu et al. 2024b), continues the autore- gressive model paradigm (Radford et al. 2019 Liu et al. 2024a) and typically includes prefill and decoding stages during inference. In the prefill stage, features from different modalities are mapped into the same feature distribution as the input embeddings of large language model (LLM). These multimodal features, in conjunction with text tokens, are simultaneously processed by LLM to gener- ate the initial output text token. During the decoding stage, the tokens in the prefill stage, together with all subsequently generated output text tokens, are used in autoregressive manner to predict the next output text token. In the context of L.M with I. Transformer decoder layers, we denote distinct index sets for various types of tokens processed across the stages of the model. Specifically, the index set for image tokens during prefill is denoted as I = {1, 2, ... N/), }, for text tokens during prefill as IT = {1, 2, ... .NF), J. while for output text tokens during decoding as o = {1, 2, ... NOT), }. where N/. N/T and NOT are the counts of image tokens, text tokens and output text tokens of I-th decoder, respectively. The sets of image tokens, text tokens, and output text tokens processed by the 1-th LLM decoder layer are denoted as St [tt, Vi E I'), S = [(, E In' and SoT (107 Ivi IOT), respectively, where 11,0 (T. 1, IRd denotes the i-th token of the
3

---

Published as a conference paper at ICLR 2025
different token sets for the I-th decoder. The sets of all tokens and the corresponding size defined as S = =s US,T and Ni = N2 +N7 + +NO respectively.
Considering the computation of the standard LLM Touvron al. 2023a the prefill stage of the (-th decoder simply described as follows:
S FFN(MHA , ((")),
(1)
where Sf = S Sf USF MHA(.,-. and (-F denote the Multi-Head Attention Block and the Feed-Forward Networks in 1-th LLM decoder layer (Vaswani 2017).
During the decoding stage, there typically two methods available, i.e. decoding without and with KV cache. First, we consider decoding without KV cache as an extension of the prefill stage. The one-pass decoding is computed as follows:
Sf,1 USIT FFN(MHA(S) USPT. SPuSP" US("))
(2)
For the decoding with KV cache, we further split the MHA operation as follows:
Q K, V = 1W N,pprt (W," sk = S. UK S = = UV,
SW (,Npr), {W,' 11x0r
(3)
= Wo Attention(Q sk s
OOT N+I.NOT FFN(O),
where W W.K. w/ and W/O are the linear layers to obtain the activation sets Q. K, V and O in 1-th LLM decoder layer. Attention(.,"" : .) is the Scaled Dot-Product Attention operation (Vaswani 2017). And SOT is the last output text token (NoT -th output text token) in the 1-th decoder layer N,Nor
during decoding, while s and sy are KV cache of the 1-th decoder layer.
The primary computation cost of the prefill and decoding without KV cache operations involve processing various sets of tokens and most overhead is in the activation linear layers. Specifically, the cost of (-th deocder layer during the prefill stage is dictated by the combined size of the image and text token sets, denoted as Computation (Prefill); x |S| US! |. where . denotes the number of tokens. For the decoding stage without KV cache, the computation overhead is determined by the sum of the aforementioned components plus the size of the generated output text tokens, represented as methods, decoding
Computation/ (Decodingw///oooooooooooo cache)) x US/ US!T Unlike the previous with KV cache operation requires only the last output text token to be activated. This decoding method yields lower computation overhead compared to decoding without KV cache. However, needs additional GPU memory to store the the activated intermediate variables, thereby introducing higher GPU memory cost (Luohe et al. 2024). The GPU memory overhead of 1-th KV cache is also determined by the quantity of the token sets, defined as Memory(Decodingg (Decodingw/ cache)! oc |S| USS 1 Given (-th LLM decoder layer, reducing the sizes 1. | and | results in a corresponding reduction in the number of tokens for layers beyond the (-th decoder layer (Huang et al., 2024b). This reduction decreases the computation consumption and GPU memory overhead during the prefill and decoding stage of LLM.
3.2 MOTIVATION
As mentioned above, the token number affects the computation consumption and GPU memory overhead. while the previous works (Chen et al. [2024a] (Saang et al. [2024] Li et al. 2024a Ye et al.) 2024 Arif et al. 2024 Song et al., 2024) focus on reducing the image token set S. However, the reduction in the number of image tokens often primarily accelerates the prefill stage and gradually diminishes in benefit during the decoding stage.
Computation(Prefily x |S} US!
o |S/ USF US!T |SPT) Memory (Decodingw/ )t x |s| uST uS!T R |S!T)
Compputatoon(eeoodngg..//
(4)
where, |S oc.
As shown in Eq. and Fig. [ with the generation of output text tokens during the decoding stage, the computation consumption and GPU memory overhead increase progressively. Meanwhile, the prefill
4

---

Published as a conference paper at ICLR 2025
Prefill
Decoding w/o KV Cache
Decoding w/ KV Coche
Actrotien
It LLM decader laver
14 LLM dkonda lour
decoder
Prodette
Presicte
Predicte
Toui
Teol
Caslee
Figure 2: The sparsification inference modes for MLLMs. In the prefill stage, only image tokens are dropped based on the decisions of the learnable image predictor. For decoding without KV cache, we reduce both the image tokens and output text tokens maintain consistent inference efficiency. When decoding with KV cache, the output predictor determines whether the activations generated by the current output text token should be added to KV cache and thereby discard part of the activations to reduce the size of KV cache. Note that the decision regarding the activations of the current output text token will be shared across all subsequent layers beyond the 1-th layer. Meanwhile, the "Yes" branch means the decision to keep the token or its activations to participate in subsequent calculations.
stage typically computes only once during the generation process of LLM. Therefore, the benefits gained from reducing the number of image tokens diminish over the course of the entire generation process, as their impact becomes less significant during the later stages of decoding. Inspired by this, we propose the Dynamic-LLad VA framework to sparsify both vision context S/ and language context Sp" involved in LLM generation to achieve the consistently Efficient MLLM.
3.3 DYNAMIC VISON-LANGUAGE CONTEXT SPARSIFICATION
3.3. OVERVIEW
To compress the image token set and output token set, we use two binary masks M and Mor to determine whether their tokens should be retained or discarded in I-th decoder layer, where M [m-/m;; E {0, 1} A Vi E I'} and MOT = {m.lm; E (0. 1) A Vi E IOT). } The reduced image token set and output text token set are defied as S/- = {S = 1 A E I'} and SOT+ = {St,, MMOT- = A vi E I respectively, where | < | and S < | J. In this way, the computation consumption and GPU memory overhead both the prefill and decoding stages, as described in Eq. 4 are simultaneously reduced. Such optimization contributes to efficient resource utilization during the entire generation process of L.M. In particular, Dynamic-ILLav uses two learnable predictors pl and por to generate the binary masks M and MOT respectively. The learnable predictors are applied only once, after the (-th decoder layer of the MILM. Specifically, once the predictor makes the decisions layers. This one-time token reduction strategy circumvents the need for complex adjustments to sparsity ratios of MLLM decoders. In practice, follow the work (Chen et al. 2024a) and set I to 2. The ablation study for / is presented in Tab. Note that the learnable predictors consist of lightweight, multi-layer neural networks that introduce only marginal additional computation costs (less than 1 % of the total). The detailed real GPU latency and memory of the Dynamic-LLa framework are shown in Tab. and Tab. 14] Detailed architectures are shown in Appendix A.4.1
tokens, this decisions are shared across all subsequent
In the following sections, we introduce how to perform the sparsification inference in the prefill and decoding stages of MLLMs in Sec. 3.3.2 We also introduce the end-to-end training for the learnable predictors in Sec. 3.3.3
3.3.2 SPARSIFICATION INFERENCE
As shown in Fig. 2 we apply different sparsification strategies tailored to specific generation stages of MLLMs. For the token sets S/ and ST processed by 1-th decoder layer in the prefill stage, we consider the reduction of the image tokens. The image predictor leverages the features of image tokens and predicts the binary mask to select the tokens that participate in the prefill computation.
5

---

Published as a conference paper at ICLR 2025
Specifically, we consider the set S/ E IR N/ 2D matrices and the pipeline is defined as:
D' = P' (S)) E x/2,
M = argimax (D'),
(5)
S = S...MM'-AWWEN
where (.) is the forward propagation of the image predictor, D' is a feature sampling decision generated by Pl, and argmax; indicates that sample values of DD' along the second dimension. This indicates that a token should be kept if the second value is greater than the first value in the second dimension of D' Then we replace the reduced token set 8 S! with S in Eq. to perform the prefill stage.
For the decoding stage, we first consider decoding without KV cache. This operation is similar to the prefill stage except that the last output token generates the next output token during decoding. obtain S where M Npr the NOT -th value of the mask MOT We further replace the above SP and SpT*
Thus we use the pipeline in Eq. with ST and por while keeping M
with Sf. and S in Eq. to conduct the decoding stage. Note that in both the prefill and decoding without KV cache stages, after the predictors reduce the token set Si to St, all subsequent layers use this reduced token set for computation, i.e., the length of computed token sets is smaller to introduce the computation efficiency.
For decoding with KV cache, our primary goal to reduce the sizes of KV cache, specifically S and SY to achieve GPU memory savings during the decoding process. We use the last output token NPT E {0,1) by Eq. 5 and the decoding operation in
SoT i,vor and pOT to get binary decision MOT
Eq. 3 is modified to the following operation:
Q, K, V = (W/ S/Xpr, (W sor Si,pTT) (W. Sor S(xpr)
O= = Wo Attention(Q Sk K SK UV)
S = SKUK Sy = =5 UV, Map =1,
(6)
8 UK, SY = SK, U0, otherwise,
SOT IILNOT
FFN(O),
where the binary mask Nor determines whether KV activations of the current output text token should be added to KV cache. Meanwhile, the decision to add the current token's 's activations to the KV cache is also propagated subsequent layers, and the size of the KV cache for subsequent layers is reduced accordingly. Note that we still include this token in the current attention computation, regardless of whether will be discarded in subsequent calculations. This ensures that this token contributes to the attention mechanism immediately, even if it does not persist in KV cache for future decoding steps.
3.3.3 END-TO-END SPARSIFICATION TRAINING
Unlike the sparsification inference phase, where the token sets are directly reduced, we employ the full-token sets along with binary masks to optimize the predictors and LLM in an end-to-end training
process, as inspired by (Veit Belongie 2018 Herrmann et al., 2020 Rao et al., 202T Lin et al.) 2023 This approach ensures the model dynamically adjusts and learns which tokens essential during training while maintaining the full set of tokens for comprehensive optimization. The details of the end-to-end sparsification training are presented in Fig. and Appendix A.2
Given the binary masks of the token sets, we need to use these masks to isolate the influence of non-essential tokens on the important tokens during the LLM training computation. One native idea is to directly set the values of the unnecessary tokens to zero vectors. However, this method introduces a problem: when we sparsify output text tokens during the parallel training of LLM, discarding the value of an output text token will prevent it from autoregressively predicting the next output text token, making it impossible to compute the loss. This "hard training" result is presented in Tab. 7 To address this, we apply the masks to the Softmax(() operation in Attention(-) during training. Specifically, we firstly obtain binary mask of the full-token set M MM'U11 OM

---

Published as a conference paper at ICLR 2025
Then use the full-token set mask get the binary mask matrix G wM R and let diag(G) (G) = The Softmax(-) is changed to MaskodSoftmax(( -) operation during training:
exp(Xi,) Gay
MaskeedSoftmax(... G)
A= exp( (X..)
where X NRXN is the input of Softmax(-) operation. Eq. allows our framework to maintain parallelism in training while ensuring that the non-essential tokens do not influence the output, without breaking the autoregressive process essential for loss calculation. Note that this form of masking and predictors maintains uniformity between training and inference. Specifically, during training, the presence of the causal attention mask (Brown 2020) ensures that each token focuses only on prior context information, while the of MLP the output predictor pl ensures that decisions are based solely on its own features, consistent with the process used during decoding.
In addition the issues mentioned above, have the gradient flow problem in the backward propagation of training. The argmax(() operation we performed to obtain M and MOT is non- differentiable and impedes end-to-end training. Thus use the Gumbel-Softmax (Jang al. 2016) with Thrrouhh Gradient Estimator (Bengio et al., 2013) to avoid the gradient flow problem. Taking the reduction of the image token set as an example and the argmax(-) in Eq. 5 modified to the differentiable operation during training. The forward propagation is formulated as follows:
D CumbelSoftmax,(0D ,7),
(8)
M' = argmax, (D"")
where is the temperature of the Gumbel-Softmax function. When becomes smaller, DJt smoothly approaches the discrete distribution. Following the work (Lin et al. (202)), Dynamic-LLaVA employ the exponential decay for from to 0.1 during training. Meanwhile, in the backward propagation, we propagate the gradient flow from M to D'I:
Oc
ac
(9)
AM
where C is the objective loss function.
Furthermore, incorporate constraint regularization term to constrain binary masks according to pre-defined keep rate for token sets. This constraint ensures that the predictors retains a certain proportion of tokens, adhering the desired sparsification strategy. It is worth noting that we found that applying sparsification to SOT on samples with short output tokens during training lead to instability training and performance degradation. To avoid this problem, we apply SOT sparsification only to samples where output text tokens exceed pre-defined length LENOT E No, where No denotes the set of non-negative integers. The constraint regularization term R be defined as:
sum(MM 1/1(s )/1(" Irl
|S 2 LEN
r'1l + | MM -
R =
#10
otherwise,
(10)
where sum(-) is the summation operation and ||. || denotes the Frobenius norm. r' and OT are the pre-defined keep rates of S/ and S respectively. We directly add this term to the original objective loss function, with hyper-parameter À used to control its weight. The trade-off analysis of vision context (r-) and language context (T are presented in Tab. 5, while the ablation study of the sample used for training should have minimum output text token length (LENOT) and the weight of the regularization term (A) are shown in Tab.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets, Benchmarks, and Metrics. During training, we use the instruction-tuning dataset 656K Mixture Dataset, as in VA- 1.5 et. 2024a). For vision understanding evaluations, we use the commonly used vision understanding benchmarks to evaluate the performance similar LLa
such as VQAv2 (Goyal et al. 2017). GQA (Hudson & Manning 2019). VizWiz (Gurari et al. 2018).
SciQA (Lu al. 2022) Text (Singh al. 2019) POPE al. 2023b), MMBench (en) LLu
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison with SoTA vision context sparsification methods on vision understanding bench- marks. The best results are bolded and the second best results are underlined in all following tables. The sign "I" and "I[T" mean only the vision sparsification and the vision- languaee sparsification of Dynamic-LLe LLaic VA in all following tables, respectively. "TFLOPs" is only the computation consump- tion of the image tokens. Synamic-LLLaVa achieves the best performance in most benchmarks using a similar number of image tokens. The "Free" column indicates whether a method is training-free free.
Minbed
YUNM2
D0A
5210
S00V?
Haah 5-18
11.1
76.3
40.0
65A
1917
1924 265444222
543 MAMA:
144-19903
14 34.24433
569:50
543:33
1203 (16.5) (313
585-47)
384637)
-4951
72.1-3.11
0.374.11
5701433
"Latie 5 The
57a
18.1
3013 11
Ha
146
910
013
153
148
565-35
160 143 5)
11912091
PARGEN
Froccn 2p3 de pouee
oE unttigg In Dyw.mb---... 100 11 New Bart 1120 prosile
and procecee Aww.-aa
Table 2: Comparison with SoTA efficient vision projector methods on vision understanding bench- marks. Dynamic-LLaVA uses the original vision projector and achieves competitive performance.
Projentr
Taage yovnil
N0M-NN
VQAV2
CAAA
POrE
Ang.
Trice
MLP
330
30.1
66.3
31.1
78.5
62.0
$5.
$0.0
62.0
DAndeOTo 2al Recendar TD
Recregiw:
136
174
3.5
63.1
76.3
25.1
58.1
813
$1.8
00.1
(Arorhirll dec Radi Soutte 16
Vovi Shurts
196
164
3.5
600
16.3
06-7
68.
65%
48.5
49.8
LOP-2
339
2.5
66.3
28.7
77.3
62.1
15.1
01.2
Sapaees ((ha))
50.2
Laaii I0em)
MLP
336
30.2
27.2
02.3
$3.5
21.3
(Aradee LLAVA Trlcshhttrr deTalee
330
60:1
16.3
(0CC924 LL Talee Yanl
141468)
600
32.0
26.6
$3.4
61.5
Dycanio Thaehhire IDen.
335
661
23.1
68.9
15%
21.3
LaMe Take 11h Eooo
136
1/4
600
546
75.4
67.5
674
55.6
615
(Arssónn 1L Rat 130 ortakan
ToosFater
336
66.
13.0
833
53.
CEreNs 1.. Tokes Punt 18g our
336
54.5
DynaeoLLaaa FeietPadeen 15 (Dmo)
336
23 (375))
06.5
301
78.3
62.
153
342
et al. 2023b), SEED (image) (Li et al. 2023a) and MM-Vet (Yu et al. (023). Furthermore, we also use the vision-centric vision understanding benchmarks, such as MMVP (Tong et al. (024b). RealWorI (xAI 2024) and CVBench-2D (Tong et al. 2024a). We employ the same evaluation metrics of the above benchmarks to assess the results.
Specially. to evaluate the generation ability of MLLM before and after the generated language context sparsification. We constructed VIS-VQA single- and multi-round Benchmarks based on the subset of IVIS-Instruct4V (Wang et al. 2023). Meanwhile, we constructed single-round long generation text benchmark Share VQA by ShareGPT4V dataset (Chen et al. (0036)). Details of these datasets are presented in Appendix AA.4.3 and [A.4.4] To ensure a fair comparison, none of the images selected for our benchmarks were included in the training set of the base MLLMS and Dynamic-LLave evaluate the fluency of the generated responses of MLLMs using the Perplexity Metric (PPL) (Jelinek lt al. 1977). and the similarity of the generated answers to the standard answers using the Metric for Evaluation of Translation with Explicit ORdering (METEOR) (Banerjee & Lavie 2005).
We
Implementations. All of the methods are trained on 8 NVIDIA A100 (80G) using Pytorch (Paszke et al., 2019). For fair comparisons, we utilize the open-source MILMMS weights available in the official GitHub repositories and continue one-epoch predictors. The hyper-parameters of which decoder layer to sparsify the tokens (O, the sample used for training should have a minimum output text token length (LENOT) and the weight of the regularization term (X) are set to 2, 50 and 100 in all experiments, respectively. The details are presented in Appendix A.4.2
citructtio-tuning for MLLMs and learnable
4.2
MAIN RESULTS
Vision understanding. As demonstrated in Tab. [ our proposed Dynamic-LLaVA frame- work achieves superior performance compared SoTA vision context sparsification methods in most benchmarks while reducing image tokens by approximately 80%. Additionally, the Dynamic-LLaVA framework reduces FLOPS by more than 70% for official I.LVA-11.5 base-
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of generation ability benchmarks. "Troal--Computing means the total gener- ated token number and the maximum number of tokens involved in decoding. "a/b" means the results in experiments of PPL/METEOR. "TFLOPs" and "Mem." in "Text (decoding)" are measured under decoding w/o and w/ KV cache, respectively, while "Mem." " means maximum GPU memory of KV cache. Dynamic-LLLaVA achieves significant inference efficiency during the decoding stage with negligible generation ability drop when sparsifying both vision and language.
181
2380
8.390
vOW
10.00
Seanes
246872
76.0
0.018
LM15-IIA
ES: Weni (499)
4409
156
0.50
0.13
LLark Ca
15510
4.65
6:318
3094
65 (87))
0.K3
By Syanaa du No (aar
245 ANSORES HESH
50HHB0O (009)
435
La 2-1
151
21461-2140
413M46
250
6-427
33
21145
2500
335
B-105
121
3.M 1099084
340
0.283
Thare 1100
Dwwme (Bhan
201022
437
B4455
116
305-45
326
Laat Part
(3:451
250
8-100
140
500 HORG/LLAS
400
8.302
345
B-122
Spares
36.7
13-1133-770 117912--7796
MAN 44L-417777) (495)
5.46
16/3177
18A
KNN 201
70.18
1409
RIS 1ENLE
341
kig
bookh
Table 4: Total time and maximum GPU memory overhead during MLLMs' generation. "2K/4K" means the number of the generated output text tokens. "x" means the generation process has failed due to the OOM. The results are measured in one A100 (80G) and the batch size is fixed to 8.
Metiod
Pretil)
Decesing wis KV cadho
<K
Deooding KV cache
IK
3K
4x
Mam
Men.
13:38
13565
460
11 V. Til
0.43%
1079
416
533
77G
Dyrambed 1ab
0.37c
79836
39G
We opply the Poralel Spaniticatoo Inference PestV aclieve the buxh-panlld Indrenno.
lines in the prefill stage, with only minimal performance degradation across most benchmarks. Especially, for the LLaVA-1.5 with 7B and 13B parameters, the Dynamic-LLaVA even shows performance improvements on the SciQA, POPE, MME, and MMBench benchmarks. For in- stance, on the SciQA benchmark, the 7B and 13B Dynamic-LLaVA models achieve perfor- mance gains of +2.3% and +0.8%, respectively, compared to the original V..... More comparison of SoTA vision context sparsification methods are presented in Appendix I As shown in Tab. g for efficient vision projector methods for MLLMss, Dynamic- LA exclusively utilizes the original MI.P of LLaVA-1.5 as the vision projector. This approach surpasses those that modify the vision projector across most benchmarks, achieving an average performance that exceeds the best alternative methods by 0.9%. Furthermore, our Dynamic- LL framework can be in- tegrated with other efficient vision projector methods. For instance, our combination with the TokenPacker projector, Dynamic-LLa VA- TokenPacker, achieves significant reductions in vision tokens by an additional 60% on top of the already reduced count by efficient vision projector methods, with only minimal loss in performance. Compared to the official only a minimal performance loss of 0.3% and 0.1% (for 7B and 13B models) while using fewer vision tokens (57 vs. 144). Moreover, we observe that the sparsification of the output text tokens does not impede the comprehension ability of MLLMs. This finding is supported by comparisons between Dynamic-LLavee and Dynamic- Manaamic-LLaVAA, LLa as detailed in last two rows of Tab. T and Tab. 2
Table 5: Trade-off of the keep rates of vision context (r-)) and language context (r OT)
110
10
TORAO 119 09
10
25
method, our
incurs

---

Published as a conference paper at ICLR 2025
Generation ability. To evaluate the impact of lision-language context sparsification on generation ability, we utilize the LVIS- VQA (single-round) and LVIS- VQA (multi-round)) as benchmarks.
Table 6: Hyper-parameter ab- lation for I, LEN and A.
Hoper
Vine
Benehemek
To demonstrate the advantages of Dynamic- LLaVA in dynamically sparsifying the language context, we compare it against traditional static methods, "Random" and "Structure". "Random" refers to randomly discarding ~50% output text tokens, while "Structure" involves discarding every alternate output text token.
vQ/w2
POPE
77.6
61.3
85.7
77.9
61.3
77.4
61.0
85.5
LEN
58
729
61.3
854
61.2
100
As shown in Tab. Dynamic-LLaVA maintains nearly unchanged
1000
61.
85.7
generation fluency (by PPL) and quality (by METEOR) when only the vision context is sparsified. When sparsifying both vision and language contexts, Dynamic-LLaVA exhibits a slight increase in PPL. (less than 0.3) and minor improvement in METEOR score compared to full-context baselines, while also reducing ~50% FLOPs (decoding without KV cache) and GPU memory overhead (decoding with KV cache) of the output text tokens. In comparison with the traditional static methods under the same condition of discarding ~50% output text tokens, both "Random" and "Structure" methods significantly underperform in terms of fluency and quality of generation compared to Linamic--LLVA' language context sparsification. Furthermore, compared to methods only focusing on the image token reduction, Dynamic- LLa achieves additional inference efficiency during the decoding stage while maintaining nearly consistent generation ability. Meanwhile, the LLM KV cache compression method H2O, directly discarding historical KV cache does not adapt well to mixed-modality contexts, causing significant performance degradation. We enhance the generation ability by integrating FastV with H2O, Despite these reductions, Dynamic-LLa VA still maintains slight advantage in generation ability.
Practical inference efficiency. As presented in Tab. we measured the practical inference time and GPU memory usage of MLLMs. The proposed Dynamic-LLaVA framework can perform parallel inference with mini-batch samples, due to the Parallel Sparsification Inference strategy in Appendix A. Vision token reduction methods provide noticeable inference benefits during the prefill stage and play a role in subsequent generations. However, the proportion of this efficiency gradually decreases throughout the entire generation process. Dynamic-LLa VA not only achieves significant inference speed improvements during the prefill stage but also reduces generation time by ~50 when decoding without KV cache and significantly lowers GPU memory overhead when decoding with KV cache. Moreover, we further report the practical inference lantency of the decoding stage with KV cache, which presented in Appendix A.5.2
4.3 ABLATION STUDY
Trade-off. We respectively adjust the keep rates of vision context (r*) and language context OT during training, mentioned in Sec. 3.3.3, As in Tab. 5 we observe that increasing the value of r i.e,, retaining more image tokens during inference, significantly enhances the vision understanding performance of Dynamic-LLaVA. On the other hand, adjusting the value of OT has negligible effects on vision understanding but impacts the generation ability of Dynamic- LaVA. By tuning the values of r' and OT during training. it possible to achieve a balance between performance and efficiency tailored to specific requirements. More detailed results are presented in Appendix A.5.4
Hyper-parameters. Tab. 6 provides an analysis of which decoder layer to sparsify the tokens (1) and the sample used for training should have minimum output text token length (LEN ), as mentioned in Sec. 3.3 This analysis enables the Dynamic-ILLaV framework to be easily integrated into various MLLMs. For I and X, Dynamic-LLava shows relative insensitivity in vision understanding performance. We follow the work (Chen et al. 2024a) and let / 2, while À 100. For LEN sparsifying output text tokens of all samples significantly reduces the understanding performance of Pynamic-LLaaae Incrementally increasing the threshold of OT results in diminishing returns.
CONCLUSION
In this paper, we proposed dynamic vision- language context sparsification framework, termed Dynamic-LLas LLa This framework is designed with tailored sparsification inference strategies for the inference modes of MILMS and can be integrated into MILMs training in an end-to-end manner.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is partly supported by the National Natural Science Foundation of China (NO, 62102151), the Open Research Fund of Key Laboratory of Advanced Theory and Application in Statistics and Data Science, Ministry of Education (KLATASDS23055) the Fundamental Research Funds for the Central Universities.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 0rXivv2303.08777 2023.
Kazi Hasan Ibn Arif, Jin Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Ji. Hired: Attention-guided token dropping for efficient inference of high-resolution vision-language models in resource-constrainee environments. arXiv preprint arXiv:2408. 10945, 2024.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
Shaosheng Cao, Wei Lu, and Qiongkai Xu. Grarep: Learning graph representations with global structural information. In Proceedings of the 24th ACM international on conference on information and knowledge management, pp. 891-900, 2015.
Junbum Cha, Wooyoung Kang. Jonghwan Mun, and Byungseok Roh. Honeybee: Locality-enteer eehhaney- projector for multimodal 1mm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13817-13827, 2024.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023a.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-plas and-play inference acceleration for large vision- language models. arXiv preprint 2oXiv:2403006660 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong. Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.122 12793, 2023b.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16822 2024b.
Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang. Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevim v2: Faster and stronger baseline for vision language model. arXiv preprint ar X02v:24402.03766 2024.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 11929, 2020.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive kv cache compression for llms. arXiv preprint arXiv:2310.01801, 2023.
11

---

Published as a conference paper at ICLR 2025
Yash Goyal, Tejas Khot, Douglas Summers-Star Stayy Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904-6913, 2017.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. arXiv preprint arX1v:2308.16117, 2023.
Charles Herrmann, Richard Strong Bowen, and Ramin Zabíh. Channel selection using gumbel softmax. In European conference computer vision, pp. 201-241125 Springer, 2020.
Kai Huang, Hao Zou, Ye Xi, BoChen Wang, Zhen Xie, and Liang Yu. Ivtp: Instruction guiidd visual token pruning for large vision- language models. 2024a.
Wenxuan Huang, Yunhang Shen, Jiao Xie, Baochang Zhang, Gaoqi He, Ke Li, Xing Sun, and Shaohui Lin. A general and efficient training for transformer via token expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15783-15792, 2024b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700-6709, 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax. arXiv preprint arXiv: 11111.001444 144, 2016.
Fred Jelinck, Robert Mercer, Lalit R Bahl, and James K Baker. Perplexity- a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1): S63-S63, 1977.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.0400
2024.
Yutao Jiang, Qiong Wu, Wenhao Lin, Wei Yu, and Yiyi Zhou. What kind of visual tokens do need? training-free visual token pruning for multi-modal large language models from the perspective of graph. arXiv preprint arXiv:2501.022 02268,2,,.. 2025.
Yizhang Jin, Jian Li, Yexin Liu, Tianjun Gu, Kai Wu, Zhengkai Jiang, Muyang He, Bo Zhao, Xin Tan, Zhenye Gan, et al. Efficient multimodal large language models: A survey. arXiv preprint wrXiv:240S.10739 2024.
Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vasion-language models. arXiv preprint arXiv:2404.07204, 2024.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp. 19274-19286. PMLR, 2023,
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal 1lms with generative comprehension. arXiv preprint arXiv.230..16125 2023a.
Wentong Li, Yugian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jianke Zhu, and Lei Zhang. Token- packer: Efficient visual projector for multimodal Ilm. arXiv preprint arXiv:2407.02392 2024a.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemint Mining the potential of multi-modality vision language models. arXiv preprint a 2403. 18.../88// 18814, 2024b.
12

---

Published as a conference paper at ICLR 2025
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large aisoonlanguage models. arXiv preprint arXiv:2305.10335 2023b.
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapkv: Lim knows what you are looking for before generation. arXiv preprinl arXiv:2404. 14469, 2024c.
Zhang Li, Biao Yang. Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26763-6667777 2024d.
Shaohui Lin, Wenxuan Huang, Jiao Xie, Baochang Zhang, Yunhang Shen, Zhou Yu, Jungong Han, and David Doermann. Filter pruning for efficient enns via knowledge-drine driven differential filter sampler. arXiv preprint arXhv:2307.00198 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the TEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis, Alvin Cheung, Zhijie Deng, Ion Stoica, and Hao Zhang. Online speculative decoding, arXiv preprint arXi::2310.07177 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprini arXiv:2307.00 06287, 2023b.
Meta Introducing Llama. 3.1: Our most capable models to date, 2024.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Shi Luohe, Zhang Hongyi, Yao Yao, Li Zuchao, and Zhao Hai. Keep the cost down: A review on methods to optimize I1m's ky-cache consumption. arXiv preprint 24000 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937- 2021.
Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
13

---

Published as a conference paper at ICLR 2025
Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, and Benyou Wang. Less is more: A simple yet effective token reduction method for efficient multi multi-modal Ilms. arXiv preprint 4rXiv:2409.10994, 2024.
InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities, 2023.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya lyer, Xichen Pan, et al. Cambrian-1: fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406. 16860, 2024a.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9568-9578, 2024b.
Hugo Touyron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397/ 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 092888, 2023b.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European conference on computer vision (ECCV), pp. 3-18, 2018.
Junke Wang, Lingeeen Meng. Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arXiv:2311.07574. 2023.
xAI. grok. 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309. 1745339..744,,. 2023.
Lijie Yang, Zhihao Zhang, Zhuofu Chen, Zikun Li, and Zhihao Jia. Tidaldecode: Fast and accurate Ilm decoding with position persistent sparse attention. arXiv preprint arXiv:2410.05076. 2024.
Xubing Ye, Yukang Gan, Xiaoke Huang, Yixiao Ge, Ying Shan, and Yansong Tang. Voco-llama: Towards vision compression with large language models. arXiv preprint arXiv:2406. 12275, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint ar0iv:2308. 200.22008.02490, 2023.
Xuan Zhang, Cunxiao Du, Chao Du, Tianyu Pang, Wei Gao, and Min Lin. Simlayerkv: A simple framework for layer-level kv cache reduction. arXiv preprint arXiv:2410.1384 2024a.
Yichi Zhang, Bofei Gao, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Baobao Chang, Junjie Hu, Wen Xiao, et al. Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling. arXiv preprint arXiv:2406.0 02069, 2024b.
Yuan Zhang, Chun-Kai Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gu- dovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al. Sparsevim: Visual token sparsifica- tion for efficient lasion-language model inference. arXiv preprint 17X14:2410.04417, 2024c.
14

---

Published as a conference paper at ICLR 2025
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36:34661-34710, 2023.
Fei Zhao, Taotian Pang, Chunhui Li, Zhen Wu, Junjie Guo, Shangyu Xing, and Xinyu Dai. Aligngpt: Multi-modal large language models with adaptive alignment capability, arXiv preprint 1rXi::2005.14129, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4666 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models, arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
Appendix
17
A.1
Parallel Sparsification Inference Optimization
17
A.2
Detailed Pipeline of End-to-end Sparsification Traming
17
A.3
More Detailed Discussion
19
A.3.1
Inference Efficiency of Dynamic-LLava and Token Reduction Methods
19
A.3.2
Discussion for LLM KV Cache Compression Methods and Dynamic- LL VA
19
A.3.3
Additional Discussion for Sparsifying Text Context during Prefill
A.4
Implementation Details
21
A.4.I
Details of Predictor Architecture
A.4.2
Details of Settings
22
A.4.3
Details of LVIS- VQA Benchmarks
A A.4.4
Details of ShareGPT4V- Benchmarks
23
A.4.5
Details of Calculation Equation of FLOPs
24
A.5
Additional Results
24
A.5.1
Additional Benchmark Results
24
A.5.2
Additional Practical Inference Efficiency Analysis
25
A.5.3
Training Time
25
A.5.4
Additional Trade-off Analysis
25
A.6
Visualization
27
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PARALLEL SPARSIFICATION INFERENCE OPTIMIZATION
In this section, we introduce the parallel inference optimization for the sparsification inference described in Sec. 3.3.2 We denote the mini-batch form of S and So in the 1-th decoder layer as Sf and S/ respectively. We treat these as matrices, thus S (S (S) [Vb E 2,, ... B}} E R # jxd and SpT ( lvb 21, BB} B xd where B represents the mini-batch size, and S| (() and S sT(( refer to the token set for the b-th sample within the batch.
For the prefill stage, we pad the indefinite length mini-batch image token sets and one-pass parallel inference of the predictor for the padded mini-batch image token set 8/ = {LPadding(S) (b))))))))))))))))))))))) (()))) E 21,2, BB}} E xBxmaa((X)xd to obtaion the reduced token set S for prefill, where N/ is the sizes of the mini-batch image token set, max(N/) denotes the maximum size of the image token sets within the mini-batch and LPadding(()) represents the operation of padding zero values in the left of the input token set, extending its size to match the maximum size max(N)) ). Considering the computation parallelization, we directly select the tokens with high predictor scores to retain, and the pipeline in Eq. 5 is modified
D' P' (3/) E RR Bx xx
s/ = { Vi E TopkArgmax [r' (r'(s!(01]] (D!))))
(1I)
of {L.Padding($) (S/ ( US? E 11,2, B}} E R Bxxxxx(X"
where Topk TopkArgmax (.) the top argmax opeartion with the number of l tokens and [.] is a floor represents the extraction of all the second values along the last dimension of (M),
function. D T(b)
serving the predictor score, with (((() Imax(())) It allows retain fixed proportion y' of the image tokens for each batch's image token set, while simultaneously enabling batch-parallel processing for both the predictor's predictions and the subsequent computations within LLM.
For the decoding without KV cache, we use SOT to get SDT* for computation parallelization:
DoT = por (o(²) ER (BBxmxx(N(T) )x2 ( = argmax (T (TTO)))
(.TT(()) (S ( MOTOh LAVAAAEE I
(12)
SOT- (LPadding(S, (TT-(())))))))))))))))))))))))) Ab (1,2, BB} R B Xx
Meanwhile, considering the decoding with KV cache, and the mini-batch KV cache set can be define as { (15 ss s HVb E , We apply a similar operation as in Eq. 12 each last output text token soro) I,ver: resulting in the batch-wise binary
KV cache for the each batch token set
decision M OT())
Nor E {0, I). which determines whether to add the activations to {S (()))) .s sV(M) as
outlined in Eq. 6 For the computation in the Attention(.,- .., operation, we utilize the padded KV cache sets {Sf s/} } {[LPadding($] K()) )l66 {1, 2, ..... B}}, (LPadding(S) (s)( vo ()) {1, ......... ())}) 5. In this way, we reduce the activations stored in KV cache, and use these reduced ac- tivations to participate in the computation of Attention(-.. -). Note that [ = max(N x oT SOTT due
B}}}
Eq. 10 to constrain the number of the output text token set during training,
and ensures that each batch S (b) adheres to close keep rate of OT during inference.
A.2 DETAILED PIPELINE OF
SPARSIFICATION TRAINING
The mask generation pipeline of the MaskedSoftmax operation is presented in the above figure of Fig. 3 Specifically, we use the mask generated by the predictors create matrix for the MaskedSoftmax operation, somewhat analogous to the attention mask in the Multi-Head Attention Block. When values in this matrix are zero, they engage the MaskedSoftmax to set corresponding attention scores in the attention matrix produced by Q and K to zero. This effectively isolates the influence of non-essential tokens on essential tokens during training.
17

---

Published as a conference paper at ICLR 2025
Mask for MaskedSoftmax during Training
Multi Head Affection
Insgee Predictor
-Gi-AAp
Replone Syfteae f4
((gf))))))
MarkedSifraci
Cutput Predictor
Token
Binory mask
Binory mask matrix G
Pipeline of Predictors during Training
D
(413)
Cresicn
Gradiert
x
Image Predictor
Gantelettmce
0
orpeas
O
Predictor
O
Token set
D
D
M
Andder
T
token
Text
Output
Value
Tronspase
The detailed training pipeline of Dynamic-LLaVA, Above Figure: the mask for Masked- Softmax operation during training. We utilize the predictors to generate the binary mask M and subsequently form binary mask matrix G. This generated binary mask matrix is employed in the Multi-Head Attention Block within the MaskedSoftmax operation to isolate the influence of non- essential tokens essential tokens during training. Bottom Figure: the pipeline of predictors during training. In the forward propagation, we use GumbelSoftmax function to relax the decision matrix d and DOT to obtain Duf and DOT', respectively. Then, we argmax operation generate the binary mask M for the token set. During back propagation. we utilize the STE technique Bengio al. (2013) to directly estimate the gradient of D' and DOT through the binary mask M, bypassing the non-differentiable argmax operation to avoid the gradient flow problem.
Figure 3:
Table 7: Effect on the MaskedSoftmax operation.
Method
POPE
VQAv2
GQA
I.LAVA-1.-7B
85.9
78.5
62.0
Dynamic-LLaVA-77Byy (Ours)
85.9
77.8
61.3
Dynamic-LLaVA-7B.pp //2 w/o MaskedSoftmax
84.5
76.7
59.8
As shown in the bottom figure of Fig. we display the pipeline of predictors during training. Simply put, during the forward propagation of training, we relax the decision matrix generated by the predictors. In the backward propagation, we employ the Straight- Estimator (STE) (Bengio et al. 2013) technique to circumvent the gradient problem, thus enabling end-to-end training of the predictors.
Furthermore, shown in Tab. 7 we analyzed the rationale for using the MaskedSoftmax operation instead of directly employing selective approaches during training. evident that directly setting the values of unnecessary tokens to zero vectors leads to significant performance degradation, as observed in VQAv2, GQA, and POPE, where there was performance loss of over 1%.
18

---

Published as a conference paper at ICLR 2025
Table 8: Benchmark statistics of average token length (rows 2-5) and the total number of tokens that participated in computation during inference (the last rows).
Dotasct
VQAv2
VizWiz
SeiQ
LVIS-VQA (single)
LVIS-VQA (mult))
SharoGPT/V-VQA (single)
Avg. image tnken length
576
576
S76
576
576
576
Avg. text length
10
15
205
51
Avg. output text tokon length
2
3
6
189
351
1555
Avg. token length
586
589
597
794
1132
2182
1LaVA-1-5-13B
586
589
597
794
1132
2182
LLaVA-13R- -07.
154(-74%)
157 (-7356)
165 (-7256)
359 (-55%)
700 (-38%)
1750 (-20%%
DymamieLLLVAAI333
125 (-79%))
128 (-78%)
136 (-77%)
255 (-683)
501 (-56%)
929 (-57%)
A.3
MORE DETAILED DISCUSSION
A.3.1 INFERENCE EFFICIENCY OF DYNAMIC-LLAVA AND TOKEN REDUCTION METHODS
To further quantify the improvements of inference efficiency that Dynamic-LLaVa VA brings by the output text token lengthens, we have presented statistics on the token length for three vision under- standing benchmarks and generation ability benchmarks in Table g Additionally, we compare the total token lengths of Mynamic-LLaave and FastV, including both the prefill and decoding stages.
The results indicate that as the output length increases, Synamic-LLaVe progressively exhibits a significant advantage in terms of token reduction percentage compared to FastV, which only reduces image tokens during the prefill stage.
should be noted that the vision understanding benchmarks (VQAv2, VizWiz, SciQA) generally need the model responding to multiple-choice questions providing brief answers. However, real-world scenarios often require MLLMs to provide more detailed and extensive responses, which aligns with our constructed LVIS-VOL and ChareGPT4V-VOA benchmarks. Therefore, the improvement in inference efficiency that Dynamic-LLav provides is particularly significant on these two generation ability benchmarks compared to previous MLLM token reduction methods (e.g., FastV Chen et al.) (2024a)), which have longer output text lengths. Additionally, Dynamic-LLa VA also demonstrates superior performance in terms of generation fluency and quality.
A.3.2 DISCUSSION FOR LLM KV CACHE COMPRESSION METHODS AND DYNAMIC-LLAVA
We further discuss the core distinctions between Dynamic- Dynamic-LLaVa and LLM KV cache compression methods as follows.
First, considering the complete generation process of MILMs with KV cache. The key distinction between Dynamic-LLaVA and other LLM KV cache compression methods lies in its approach. Dynamic-LLaVA implements "online" poccision-makingg activations of the current token to KV cache, rather than removing KV activations from the historical KV cache. As presented in Fig. 4 we show the differences between a commonly used LLM KV cache compression method, H2O (Zhang et al. 2023)) and method. A significant distinction is highlighted between the two methods. In the left figure, H2O computes the attention scores between the current token's query Q and all past KV cache, removing useless KV cache based their attention scores (e.g.. KV cache corresponding to tokens with an attention score of 0.05). While our method (in the right figure) does not decide how retain historical KV cache. Instead, it calculates KV activations (by Wa and W.) for the current token and applies an output predictor (on the current token's 's embedding) to decide whether to add the current token's corresponding KV activations into KV cache (as shown in the "Yes" branch) or not to add them (as shown in the "No" branch).
process to determine whether to add KV
Second, Dynamic-LLats LLai is an MLLM inference acceleration framework that considers the distinct properties of different modalities and incorporates tailored sparsification strategies accordingly. We have implemented the H2O method in conjunction with VA, and the results are presented in Tab. [ and Tab. (3 We configured the hyperparameters of H2O to retain 50% KV cache in each of the frefill and decoding stages. However, H20, performs poorly in multimodal scenarios involving vision and language contexts. Our analysis suggests that H2O's 's strategy of discarding historical KV cache based on attention scores does not adapt well to mixed- mixed-mddality contexts. To get more comparable results, in the vision understanding benchmarks of Tab. [ we modified the layer configuration for H2O's KV cache compression to enhance the performance. The first 10 layers do not conduct KV cache compression, and H20 is applied only beyond the 10-th layer. In the generation ability tasks presented
19

---

Published as a conference paper at ICLR 2025
Dynamic-LLaVA
H2O
(Decoding w/ KV Cache)
Calucate Score
Activations
0.28
0.6
p.ce
0:29
CU
Ns
besson
W.
Wx
Output Predictor
Coche
Book
W2
Wg
Wg
KV Cache
00
Image taken
Text token
Output text token
Lost output text token
KV activations
Post KV coche
New KV cache
Figure 4: KV cache compression pipeline (H2O (Zhang al. 2023) Dynamic-LLaVA (when decoding with KV cache). Left Figure: the KV cache compression pipeline H2O involves calculating the attention score between the current Q and past KV cache during the decoding stage. The KV activations corresponding to the minimal attention score is subsequently dropped from historical KV cache. Right Figure: The workflow of Dynamic-LLa VA when decoding with KV cache. Our approach evaluates each current token's 's features by an output predictor to determine whether its activations which through WK and Wy should be added to the KV cache.
in Tab. we combined H2O with FastV (Chen et al. 2024a). The enhanced H20 implementation on MILMM shows some performance improvements. However, its performance still falls short compared to Dynamic-LLaVA.
Third, Dynamic-LLaVA introduces a tailored sparsification inference scheme specifie to various inference modes. In the above discussion, we have extensively discussed the core design of our method and its distinctions from other approaches in the context of decoding with KV cache. However, it is important emphasize that we have designed efficient inference methods tailored for different scenarios, i.e., prefill, decoding with KV cache, and decoding without KV cache. Decoding with KV cache be seen "online KV cache compression". For the scenarios of prefill and decoding without KV cache, although KV cache activations are not involved, Dynamic-LLLa' can still substantially enhance the computational efficiency of MLLMs. This enhancement is an advantage that traditional KV cache compression methods do not provide, demonstrating the broader applicability and effectiveness of our method across various inference modes within MLLMs.
Moreover, enhancing computational efficiency during the prefill and decoding without KV cache stages is equally critical for MLLMs Liu et al. (2024b): Li al. (2024d): Chen al. (2024a) Huang et al. (2024a) Cha et al. (2024) Leviathan et al. (2023) ) Chen et al. (2023a): Liu et al. (2023a)
There positive correlation between the vision understanding performance MLLMs and image resolution Liu et al. (2024b); Li al. (2024d). While achieving improved performance, the increased number of image tokens significantly adds to the computation burden during the MLLM's prefill stage, which includes rise in computation budgets such as longer image token sequences and increased inference latency Chen et al. (0224)) Huang et al. (2024a); Cha et al. (2024). This challenge echoes the importance of structural information compression in graph representation learning, where methods like GraRep Cao et al. (2013) reduce complexity via low-dimensional embeddings. Similar graph-based approaches, such as compressing image tokens via graph sparsity Jiang et al.) (2025). further enable efficient MILM acceleration. Therefore, reducing the number of image tokens to accelerate MILMs crucial. However, existing LLM KV cache compression methods typically do not accelerate the computation in the MLLM prefill stage, thus limiting their applicability in MLLMs.
20

---

Published as a conference paper at ICLR 2025
Table 9: Results of Dynamic-LLaVA with text context sparsification during the prefill stage on 3
vision understanding benchmarks.
Method
POPE
VQAv2
GQA
ILAVA-115-7B
85.9
78.5
62.0
Synamic-LLaVA-7BByy (Ours)
85.9
77.8
61.3
Dynamic-LLavA-720 +30% text context sparsification
85.1
75.3
60.2
For decoding without KV cache, this inference mode still retains advantages over those using KV cache Vaswani (2017). For instance, decoding without KV cache does not require the storage of extensive KV activations for subsequent decoding use, which significantly reduces the memory overhead during the MILM inference process. Accelerating the inference speed of decoding without KV cache also has potential applications commonly used speculative sampling strategies Leviathan al.] (2023); Chen et al. (2023a); Liu al. (2023a). These strategies utilize preliminary decoding by smaller LLMs to parallelize the autoregressive decoding of larger LLMs, also enhancing the efficiency of MILMs in practical deployments. Our Dynamic-LLaVA LLLVA facilitates this by reducing the number of tokens processed in parallel during decoding, thereby accelerating the parallel decoding of large MILMS and further improving the inference benefits of speculative sampling strategies.
A.3.3 ADDITIONAL DISCUSSION FOR SPARSIFYING TEXT CONTEXT DURING PREFILL
In the current design of Dynamic- LLaVA, we only apply sparsification to the image tokens during the prefill stage. Naturally, it is also feasible to perform sparsification on text tokens during the prefil stage. Therefore, we also conduct experiments with sparsifying text tokens during the prefill stage, reducing 30% text tokens. The results on 3 vision understanding benchmarks are presented in Tab. Unfortunately, sparsifying text tokens during the prefill stage resulted in a noticeable performance degradation. Across the three benchmarks presented in the table, the performance dropped by an average of 1.45% compared to sparsifying only the image tokens during the prefill stage. These results imply that text tokens during the prefill stage are crucial in the current multimodal scenarios, necessitating a more refined sparsification design. This will also serve as an important direction for future works.
A.4 IMPLEMENTATION DETAILS
A.4.1 DETAILS OF PREDICTOR ARCHITECTURE
Output
Curput
512--2
MLP
512-2
MLP
512-512
Vision Transformer Block
Cm
Linear
Cin 5512
Linear
d
Figure 5: Overviews of the image predictor (a) and the output text predictor (b).
The architectues of the image predictor and the output text predictor are presented in Fig. Both the image predictor and the output text predictor employ a linear layer reduce the input dimension from MLLMs to 5 512, thereby decreasing computational demands. The image predictor utilizes two vision transformer blocks (Dosovitskiy] 2020: Touvron et al.) (222)) and ciimension-reducin three-layer MLP (512 + 256 + 128 + 2). The structure of the output text predictor similar to that of the image predictor, except it does not use vision transformer blocks. This design choice ensures that
21

---

Published as a conference paper at ICLR 2025
Table 10: The detailed calculation formula for FLOPs.
Model Name
Calculation
5-77
92 x (32 x X 576 4096 X 4096 X +
676
X 4086)
10.17
PuMagge
(32
4096
4096
146
4096) 2.57
LLAVA-FiVV
x [32 576 x 4096 X 4096 4-4
576 x 376
4096)
+ 29 (32 x22x1 144
4096 4096
144
4096)
3.27
VoCo
32 X (32 x x 128 X 4096
4096 X +
1
128 x 1225 4099) = 2.27
LLaVA-HRRD
32 x (32 x 1 X 115 X 4096 x 4096 x 1 +
4 X 1
X 115 X 115 X 4096) 2.077
Dysamio-ELaVA
4096
4096 X 115 X 115
X
4096)
4096)
2.5T
x (32
1-VVA-15-138
40
5120
51 20
576
5120) 19.67
PruMegee
40 X (32
146 x 0120
5120
146 X 146 5120) 4.97
LLVVA-FaNV
3. (32 X X 576 x 5120 X 5120 + 4- 1 576
X 576
5120)
+-37 x (32 X X 144 X
6120 5120
144
144
0120)
6..02
VoCo-LLaMA LLAV--HRED
40 x (32 X 11X 128 X 5120 5120 + X
128 X 122 X 51200 = =.3T
(32
X
115
3.9T
Dynamic LLaVA
2 [32 X 376 0120 6120 4-4 576 X 376
6120)
+ 1.38 X (32 X 116
5120
5120)
4.77
LLaVA ToVVV-TleePacker... 7B 19-1441o
32 x [32 x 1 X 144 X 4096 4096
144 X 144 4096) 2.5T
LLaVA-TokenPacker 78-6-788-644
82 x (32 X X G4 x 40%6 x 4096 + 4 x 1 x 64 X 64 X 4096) 1.1T
LLaVA, TokesPadker FistVi7B
(32 x 4096 x 4096 4 X 72 X 72 X 4096) = 1.47
4096 + x
144 4096)
(32
72
Dynamic-LLawe Baaami-LLaV--TaeeePaddee TnkenPacker
(32 x
4006
4
144 4096)
(32
x
4096 x 4096
4 57 x 57 x 4096) = 1.17
LLvVA TokenPacker 13B- 144Token
x (32 x x x 5120 5120 +
144 X
5120) 4.9T
LLaVA TokenPacker- 13B Tokkn
40 (32 x 1 X 64 5120 x 5120 4
64 64 5120) 2.2T
LLaVA--okkmPPciier 135t
3 x (32 x 1 x 144 x 5120 x 5120 + x 1 X 144 x 144 x 5120)
5120 6120 4
72 x 72 6120) -2.77
37 x (32 x x 72
x (32 x x 144 x 5120 x 5120 + 4 x X 144 x 144 x 5120) 38 x x
5120 5120 4 57 x 57 6120) 2.17
Table 11: Comparison with more SoTA vision context sparsification methods on vision understanding benchmarks.
Visut
A3.8
10.2
15127
78.8
643
4148
140
1:34
(756-45)
1498
55766
6.74 5166
78.
we:1 5
557(87)
de 16:1::0
$15:113 11
4916220)
73 (447)
560-42)
$-0633
855:44
100 H4155
060:43)
231 0288
(105)
09108831
550083
853088)
(0
714
773
decisions during decoding with KV cache are solely based on the features from the current output text token.
A.4.2 DETAILS OF SETTINGS
For Dynamic-LLa presented in Tab. I we use LLaVA-1.5 (Liu et al. 2024a) as the base model and performing an additional one-epoch of instruction-tuning -uning on its open-source weights. During training. we freeze the vision encoder and projector, updating only the parameters of LLM and predictors. The initial learning rates for LLM and predictors are set at 5e-6 and 2e-4, respectively, with fixed global batch size of 64. We use the same 656K Mixture Dataset of VA-15 for instruction-tunin exclusively employing data containing images to train the predictors. All the other settings remain consistent with LLa VA-1.5.
For Dynamic- Vanaami--LLVVATokeeeee VoTTecapaake presented in Tab. 2 we utilize the open-source weights of VLaVA- TokenPacker (Jin al. 2024) the base model. All settings remain identical to those described for Dynamic- -Laniic-LLLVA. VA above.
We make adjustments only to the learning rates of the original MLLMs. applying uniform learning rates across all weights and the experiments are conducted on 8 NVIDIA A100 (80G).
For LLaVA- starting from the 3-rd layer of the model, applying F to tokens only during the prefill stage.
=00.5 F
presented in Tab. 1 and Tab. we use FastV to perform token pruning
For LLL FasaaV==,.r=.r.. k=3.5=0.75 presented in Tab. 1 we further train LLa WAAFaatVA=s..-.. =33. for 1 epoch based on the open-source codes. The learning rate and batch size are same as Dynamic-LLa VA, and the discarded tokens are to 0.
22

---

Published as a conference paper at ICLR 2025
Table 12: More comparison with SoTA vision context sparsification methods on vision understanding benchmarks.
Image (pretill)
Method
IJM
VizWiz
M.VA-Wild
MM-Ver
Res.
Token
TFLOPs
LLaVA-L5--B
Vicuna-7B
336
576
10.1
50.0
65.8
31.7
Vicuna-7B
336
146 (-75%)
2.5 (-75%)
(BCCV24) LLaVA, Faa.V
Vicuna-72 7B
336
144 (-75%)
3.2 (.68%)
51.9 (+1.9)
66.7 (+0.9)
28. (33.6
(Aniv224) V6Ca-ILaMA
Vicuna-7B
336
128 (-78%)
2.2 (-78%)
(Aniv24) LLaVA-
Vicuna-77 7B
336
115 (-80%)
2.0 (-80%)
Dynamie-LLaVA- (Ours)
Vicuna-7B
336
115 (-80%)
2.5+0.01 (-75%)
50.2 (40.2)
67.3 (+1.5)
29.5
Dynamic L.L.V 78 (Ours)
Vicuna 7B
336
11.5 (-80%)
2.5+0.01 (.75%)
51.2 (+1.2)
68.7 (+2.9)
(+0.5)
LLaVA-115-13B
Vicuna-138
336
576
19.6
53.6
72.5
36.6
(Arxie24)
Vicuna-
336
(-75%)
(-75%)
(BCCV24) LLaVA I FastV 75
Vicuna 13B
336
144 (.75%)
6.0 (-69%)
54.7 (+1.))
74.2 (+1.7)
(-2.))
Dynamic-LLava, (Ours)
Vicuna-138 13B
4.740.01 (-76%)
(-0.3)
73.4 (+0.9)
(40.7
Dynamic LLaVA 13B (Ouns
Vicuna- 13B
336
115 ( (.80%)
4.7+0.01 (.76%)
$3.0 (-0.6)
70.1 (-2.5)
(-2.0))
Table 13: Additional results of Dynamic-LLavA with the TokenPacker projector on generation ability benchmarks. Dynamic-LLaVA achieves significant inference efficiency during the decoding stage with negligible generation ability drop when both sparsify vision and language.
Bendltmakk
Method
Image (peetill)
Test (decoding)
PPL.
METEOR?
TELDS
Toeal Compuiing
Mom (M)
LL.WA-TBkcoPucke-77 144Tokkm
159/155-190175
102/90
4,80
0,3114
LVIS-VQA (singlemonnd)
Dynumic LLaVA TokenPacker 7By (Ours)
Dynamic LLAVA TokeaPicker 7B,
(.1)))))
(.1336))
1591178 1598178
139182 +83990
2.75/3.08
1.311
(339 (39%)
111/101
4,91
4.72
LLNA TokenPacker- I 5SB 14-Token
4.9
1591176-1591
1501 76
5.365 96
148/142
4,40
0,3166
Dyaumiis LLa TokesPucker-3 3B, (Ours)
(:13379))
5 S.366005
152/149
4.55
0.3149
Dynomic- VA Packm 36, (Ours)
2.1 (:175)
1891 -8486
2,95/300 06
87 (-41% (-49%)
4,78
0,3131
LLaVA-Tokallatee TokenPaderr 7B: 144Tokes
951409-355444
6.12/8.7
222/257
2.97
0.4220
IVIS-VQ4 (mulli-mouaa)
Dyaumie- Miaiii--Laaaaa VA. ToleuPackkss7B. (Ours)
Dynamic LLIVA TokenPackeer 70, (Ores)
1.1 (:36%)
351/519-1181
3.3244.88
142 (:369)144 (-44%)
245/2701
3.30
0.4245
LLVA TokenPackerr H4iToken
355554-355/514
12/177755
321/409
2.89
0.4232
Dysnmiid VArTooleiisccoos- (Ours)
851/914-351051
12/1000000
5464426
2,90
014222
Dynuaiee To 3B, (Oars)
(-37))
151/517-119255
189-41982077 (-UR)
For H2O in vision understanding benchmarks presented in Tab. | Tab. means that we directly implement H2O on VL, which retains 50% KV cache in each of the prefill and decoding stages. And -...V....-7/13--1111
in
10. means that we still retain 50% KV cache in each of the prefill and decoding stages, but the first 10 layers do not conduct KV cache compression, and H2O is applied only beyond the 10-th layer.
For H2O in generation ability benchmarks presented in Tab. 3 H20,=0.5 means that we directly implement H2O on LLaVA, which retains 50% KV cache in each of the prefill and decoding stages. Fastv-7/13B =o.=..r=.. use H2O to reduce KV cache (retain ratio 50%) only for the decoding stage.
means that we use FastV to reduce tokens of the prefill stage, and
A.4.3 DETAILS OF LVIS-VQA BENCHMARKS
We use the I.VSS-Inst Dataset (Wagg al. 2023 to build the generation ability benchmarks.
The LVIS- Vnstruuuttt Dataset contains 220,000 visually aligned and context-aware instructions generated by prompting the advanced GPT-4V model with images sourced from the LVIS database. We created the LVIS-VQA (single-round) Benchmark by selecting 1,000 instances from the LVIS- Instruct4V dataset, specifically focusing on single-round Visual Question Answering (VQA) scenarios where the answer lengths exceed 100 words. Additionally, we constructed the LVIS- VQQ (multi- round) Benchmark from another subset of 1,000 multi-round VQA instances. The average answer length in this benchmark exceeds 300 words, with interactions spanning more than seven rounds. Examples from the LVIS- VQA benchmark we constructed can be found in Fig. and Fig. 7
A.4.4 DETAILS OF SHAREGPT4V VQA BENCHMARKS
To evaluate the generation ability of the model in long-text context scenarios, we use the ShareGPT4V dataset (Chen et al., 2023b), extracting VQA samples with long-text caption to construct our ShareGPT VQA benchmark. The original ShareGPT4V dataset consists of 100K VQA samples, with average caption length of 900 characters. To construct the long generation text benchmark
23

---

Published as a conference paper at ICLR 2025
Table 14: The latency of the model during the entire generation stage with KV cache. For decoding. we report the average latency per token when batch size = and the number of total generation tokens 1000. The results measured in one A100 (80G).
Method
Prefill latency (ms)
Decoding lateney (ms)
LLaVA-1.5-132
124.52
28. 28.42
111.84
37.71
Mimami--LLaVA-1BBB T|T
72.51
26.85
H20 requires an additional attention computation process to obtain the attention scores, due the efficient inference operator implicitly handling attention operations.
Table 15: Training time of Dynamic-LLaVA 8 A100 (80G).
Method
Training time (h)
Synamic-LLaVA-7B. I/T Dynamic-LLaVA-13BBy
11
24
and obtain the benchmark ShareGPT4V-VQA of the ShareGPT4V dataset (total of 178 samples), selected the samples with captions containing no less than 300 words, while the average output text length of this benchmark is than 1,000 tokens.
A.4.5 DETAILS OF CALCULATION EQUATION OF FLOPS
For the prefill FLOPs presented in Tab.
Tab.
and Tab. 3 We adopt the equation below to calculate:
FLOPs 32 + +4BN2,
(13)
where B, N and C' denote the input batch size, the number of image tokens, and the dimensionality of MLLMs (Liu et al. 2024b a) utilizing the LLaMA architecture (Touvron et al. 2023a b Llama )024), respectively. The first term in Eq. 13 represents the computation consumption of the MHA layer, while the second term corresponds to the overhead of the MLP layer. Specific calculations are detailed in Tab. TO Note that only calculate the FLOPs of the image tokens for prefill stage of MLLMs.
A.5
ADDITIONAL RESULTS
A.5.1 ADDITIONAL BENCIIARRK RESULTS
Vision understanding. Due to constraints on the length of the main text, we have included the remaining experimental results for the three vision understanding benchmarks in Tab. [2] Dynamic- LLaVA achieved competitive results on these three benchmarks. In comparison with other methods that only sparsify vision tokens, Dynamic-LLa VA is capable of significantly reducing the number of tokens involved in the computation process.
We further compare the vision understanding performance with more current SoTA and recent token reduction methods, including IVTP (Huang et al., 2024a), TRIM (Song et al. 2024), and Sparse (Zhang et al. 2024c). The results in Tab. 1...5-...-7B. Compared to other token reduction methods, Dynamic-LLat LLa not only reduces 80% of image tokens in the prefill stage but also decreases 50% of output text tokens during decoding, consistently outperforming both training-free free and training-required methods in most cases.
based on evaluations conducted with
Generation ability of Dynamic-I.LaVe with TokenPacker projector. To further demonstrate the comprehensiveness of our proposed method, we present the results of Dynamic-LLLaV VA using TokenPacker as the vision projector on the LVIS-VQA benchmarks Tab, [3 As shown Tab. [3] compared to LLaVA-TokenPaeee
Synamic-LLave that sparsifies both vision and language contexts, significantly reduces computational costs with only a slight compromise in performance. Specifically, Dynamic- Silamic-LLaVA, T.a.V---okePaackkr memory overhead on average, while the PPL increases by a maximum of 0.38 and on average only by 0)275; METEOR decreases by maximum of 0.0035, and in most cases, the results improved.
l/r reduce up ~50% TFLOPs and 37.75 37.75% GPU
24

---

Published as a conference paper at ICLR 2025
Table 16: More trade-off results of the rates of vision context sparsification and language context sparsification.
Benchmark
Context
Rate
SciQA
TexIVQA
POPE
MME
MMBench
SEED
20% %
68.6
56.5
85.9
1501.0
64.1
65.0
Vision
50%
68.8
57.2
85.5
1487.0
65.7
65.5
80%
69.7
57.3
86.1
1469.3
65.9
65.9
20%
68.7
54.9
86.3
1483.3
64.6
64.5
Language
50 %
68.6
56.5
85.9
1501.0
64.1
65.0
80%
69.4
56.9
86.6
1473.9
64.8
64.4
baseline
100%
66.8
58.2
85.9
1510.7
64.3
66.1
Table 17: off of the rates of vision context sparsification and language context sparsification on generation ability benchmarks. Baseline indicates VL-1----55-7 Note that the definition of "Total PoonallCCommutinn Coopptting" "TFLOPs", "Mem. (M)" and "a/b" same as Tab.
Context
Rate
(single-round)
(malli-round)
Toxal Comppriig
TFLOPs
Mem. (M)
PPL.
MET:
Toxal Computiig
TELOPs
Mem. (M)
PPL.
MET.1
20%
84/90
1,52/157
182/260
3.33/4.38
Vision
50%
1591173-8833 83386
1.52/1.53 55
63/202
4.88
0.3107
351478-177822
3.27/4.34
138/125
3.15
0.4264
80%
159/17)-8388
1.31/1.56
66/284
4.88
0.3090
351 119/88
3,29/140
139/124
3.16
0.4258
20%
159/170
0.840082
41/2
5.53
0,2592
351/487884/100
1.742.28
95169
3:47
04426
Language
1591181
844/0
0.3108
351/523- 182/260
80%
1591172-1299/
2.36/2.43
85770
0.3116
3511484 -281/887
190200
0.2266
haseline
100%%
1590173-159117
2.78/2.99
103/91
4.59
0.3103
351/461 -351/461 6.12/8
222/336
2.97
0.4227
A.5.2
ADDITIONAL. PRACTICAL. INFERENCE EFFICIENCY ANALYSIS
We further report the practical inference latency with KV cache in Tab. 14 We measure the average generation latency per token when the generation length is 1000. The proposed Dynamic-LLLal framework exhibits an average improvement of 1 1.57 ms per token generation latency compared LLaVA when decoding with KV cache. This result demonstrates that, the learnable lightweight predictors add negligible increase to inference latency (less than 1%). Meanwhile, for traditional "attention-basee based" KV cache compression methods (e.g., H2O), the requirement for attention scores during decoding to perform KV cache compression poses a challenge in practical engineering implementations. many cases, the attention operations of efficient inference operators are implicit, thus requiring an additional computation step to obtain attention scores during inference. This can impact inference speed, especially when dealing with excessively long KV caches. In contrast, Dynamic-LLa's VA relies solely on the features of the current token for prediction when decoding with KV cache, thereby avoiding this issuc.
A.5.3 TRAINING TIME
Same as LLaVA-PruMenns (Shang et al. 2024). Dynamic-LLaVA requires one epoch instruction- tuning based on pretrained L.aVA-1.5. We report the training time of Dynamic-LLavA in Tab. [5] our training time is similar to the original LLa Notably, Dynamic-LLaVA achieves efficient inference with superior performance compared to other token sparsification methods that require
training (Shang al. 2024 Song et al. 2024 Huang et al.) 2024a|
Ye et al. 2024).
A.5.4 ADDITIONAL. TRADE-OFF ANALYSIS
Trade-off analysis on vision understanding tasks. We further illustrate in Tab. 16 performance across more vision understanding tasks when adjusting the keep rates of vision context (r²) and language context (T during training. Across most datasets, setting higher keep rates leads to better results but also entails greater computational expense. Under the current settings, i.c., r' 20%, ,r = 50%, Dynamic-LLavi performance. Additionally, we did not observe sharp fluctuations in performance when adjusting r and OT, suggesting that Synamic-LLaV., has robustness to the keep rates variations.
achieve balance between computational costs and
Trade-off analysis of generation ability. Same as vision understanding tasks, we also analysis the trade-off of rd and of on -VIS-VQA (single-round) and L.VI-VQA (multi-round),At shown in
25

---

Published as a conference paper at ICLR 2025
Bosed the image. il oppears to represent on older era time peried. This presence of old- ashion stove top cm olong with the woman cooking ft, auggeata that phete might be from time when electric modem atiivss were not common widely used The
including weeden floor the older time senod, everall setting dis contribute mid-20th cantury, pre pelntt houssholds:
mid-22000
Discuse the possible era time period this image might represent bcood the stove cnd
the survoundings
Dynomic -LaVA-133((Omm))
The yellow ne ligen un the image is un unvll-- ushisonel, sinluge oppl lonoe, which suggests not it lisely dates back to the mid-20 201h century earlier Duning this time rei rigenitors were offen designed with more traditional and simple appearence, opponed to the sleck, modern design applion yellow color
contemporory poroory when app
indicative
painted bright
white
nurtig mid-201 contury. rigraaton were typically with materials like metal with inauattio made from foom other meteridle The design rel igenaton
Siven the design and oeatheric of the
imoge featuring freezer comport mont top rel igenston section below elso common during ara simpiictyy and whbe color the ne iggrater serve as reminder of the design fronds notemole used opp lenes during that time pertod
refrigerstor, con you provide come historical context about applicnces from this erd?
LLoVA 138(Ours)
files populo"
backsplashes from small colse appeal and durability. They
are arranged in pottemm which creates oilolly interesting and unique design element Nosol: tiles are ovailable vehous colors shapes matoricla alloxing home. serrs custom in lodk their backapl cocording to their pensonal preferences and the everall design of the spocs in
adaition their viau appeal. mosal: tiles are dis known for their duribiliiy and resisfence moist making then idee choice for areas splashes
water
auch bothrsoms kitchins This mokes them option
you arovide insights on
behind sirk
night uae mesaic backaplash?
Dynomic- -3VVA-1333(OOrr
the
image
they were The deagn culturol prefer-noss
popular choices during
mid-20 century, porticularly 1950 1960- There bethrooma
designed to functional yet lest eecclllllllyy pleasing with focus clean cas
ly line
link illss which were often mode of cenimic glass were chesen for their durability case of clean The white sink, which common feature
vintese bathmonm. considered red symbol cleon
purity, preseriee
above
batheson image festures functionality space.
farther emphos
populor sterage
during
paried.
you the significcnce
place nole
other bathesoo essorticls koeping space
vintrae
organized dummer
aumnory. the vintion both com in inago reprosente the design
preferences and cultural valuea of the mid-201 century with focue clean eas functiorality and eatherien The link illd wal white sink icon features of this era while metal adds practical touch to everall des
Figure 6: Visual representation of dynamic token reduction for LVIS-VQA (single-round), The gray color means the contexts reduced by Dynamic- LA- 13B J Note that the reduction of the language context does not imply the texts not generated. Rather, it refers to the subsequent computational exclusion of these output text tokens to improve inference efficiency. Dynamic- LLaVA-13BB u|r is able reduce the vision and language contexts that are not crucial for generating the next token.
Table 17 when adjusting r', the performance and computational costs of Dynamic-LLLat LL remain consistent; whereas, when adjusting orr a small oO leads to significant decrease in computational expenses, along with decline in generative ability. Setting , 50 achieves balance between performance and computation costs. Although a smaller y OT significantly lowers computational costs, concurrently yields a considerable decrease the model's s generative ability. While larger y OT leads to higher computational costs, it only results in slight improvement in performance.
26

---

Published as a conference paper at ICLR 2025
Wher mone objact inago?
What one the wan sigecte prezent image?
not <bject
image a colet sovered
herver nutl 4 who micigge How mecry hemees the imagen
Go desce thin preinnme ott
tile covered pltten euili io leek very tanoy sperk... light giona pazte eye-
lerage
What are the jickeys en the hemee de eg"
Ara thane other aojects the tailer?
hareea cnter opecd cling baack
thore ret velic "upe sumoording
you vermect wiech Feking plane?
Heling encrity
How mayy ondersore reage?
There infrore (ile the
covered ge yoh
In Here any sign wcter ony ser lepudd
choss
mock appe
cepty
sdoctiyy
yau doodene way'? bresit oppir bethorree
Yos Farese waciiin
race uddd hanee back
image' and actting aarautie
verie mncettin batands run
arother locatios
scarnng spedal ginng
erticientty Hondly
roo poocile space overal otnocphere ore eppcare le serving de th corte piece some
sofetty confert duriin racee
Ane thare athar vidall abjacse preaert oport frce the rackng
mia seart forrogilon more decenotiv
Sives dlmmr cavering worn kiady rothe
Yos there some people sarcching race addeine herras comp.
unquu eppaaran
aapeine to locd og the naod?
piece mnniirrian nather fancrional
Ore lif hems peotacn chhend
makuttng wonee the rowee Heveeer thraskar dkuutti
eight the sigs fumnce babindl
on oiverig
coms: opannc each
decareted tode menn cawvey woner
artunnin
ayii the ever??
parpooui
obout
three recing besch wit backs auguen rooing experine booth nerting appea cheasphere Eatn
right serv oo cu
pura
Aiteralieely
chilleg
cool
alere
lialy mooning related mi
peodo togeller (ast) of keannwin
contanperery offare
Figure 7: Visual representation of dynamic token reduction for LVIS-VQA (multi-round). The gray color means the contexts reduced by Dynamic-LLaVA-I33 yr Dynamic-LLaVA-I3B yrr is able to reduce the vision and language contexts that are not crucial for generating the next token. Note that the reduction of the language context does not imply the texts are not generated. Rather, it refers to the subsequent computational exclusion of these output text tokens to improve inference efficiency. Dynamic-LLa VA-13B Jjr is able to reduce the vision and language contexts that are not crucial for generating the next token.
A.6 VISUALIZATION
Visualization of LVIS-VQA. Fig. shows the dynamic token reduction process of Dynamic-LLaVA on LVI (single-round), while Fig. iillustrates the same process in LVIS- VQA (multi--ound). In Fig. 6 user asks question to Synamic-LLaVA, and the associated input image is shown, with regions marked in gray indicating the dynamically reduced token patches determined by Dynamic- LL Drawing from the identified mask and the provided input information, the model produces
27

---

Published as a conference paper at ICLR 2025
image
masked image
image
masked image
STOP
STOP
Figure 8:
Visualization of the vision token patches for COCO. The gray color means the contexts reduced by Dynamie-LLaaA--1BB, The first and third columns are the original images, and the second and fourth columns are the reduced vision token patches by Dynamic-LLaVA-13Bjr
textual mask, with the obscured words or affixes similarly exhibited in gray. This procedure boosts the inference efficiency of the model and ensures effective focus on relevant information for a fast response generation.
Fig. 7 further shows this process by illustrating the example of multi-round dialogue interaction in LVIS-VQA (multi-round), In this scenario, the model only considers the current user query and
28

---

Published as a conference paper at ICLR 2025
image but also incorporates context from previous rounds of dialogues. The dynamically masked areas in both the image and text are both shown in gray, emphasizing how the model dynamically adjusts its focus based the ongoing interaction. This enables the model to maintain coherence and context across multiple exchanges while boosting response generation speed.
Visualization of COCO. In Fig. we show the visualization results of the vision token patches in COCO Dataset (Lin et al. 2014). We set the instruction as "Describe this image" for Dynamic- LLa VA. Notably, the patches focus primarily on the foreground elements of the images, effectively highlighting important features and discarding irrelevant background details. This demonstrates the capability of Dynamic-LLan VL to isolate and retain essential visual features for further process.
29


Reviews:
[{"review_id": "GtmNXL0cF9", "reviewer": "Reviewer_LdRV", "summary": "In this work, the authors propose Dynamic-LLaVA, a novel approach to enhance both prefilling and generation efficiency in Multimodal Large Language Models (MLLMs). The modified model employs two predictors: an image predictor that dynamically identifies and drops less significant image tokens during prefilling, and an output predictor that filters out non-essential output text tokens. When training these two predictors, the authors propose an end-to-end sparsification training pipeline, which added a mask to the original softmax attention to keep a fixed number of image tokens and generated text tokens. Additionally, the argmax is replaced by Gumbel-Softmax during backward to avoid the gradient flow problem. Extensive experiments show the effectiveness and efficiency of the proposed method.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This work pioneers the reduction of KV cache length during long text generation for MLLMs, whereas previous research has focused solely on reducing prefilled vision tokens.\n2. The end-to-end sparsification training can be viewed as a novel way to enhance the token sparsity of MLLMs.\n3. The paper proposed extensive experiments to show the effectiveness of the proposed method.", "weaknesses": "1. While Dynamic-LLaVA demonstrates improvements over previous approaches like FastV and PruMerge+, it requires additional post-training, which introduces extra computational overhead. In table 3, it seems that the performance of Dynmaic-LLaVA and FastV are even comparable. Therefore, for real-world deployments where computational efficiency is crucial, implementing existing methods may be more practical. However, this raises an important question: the work would benefit from a clearer justification for why a trainable approach is necessary and what specific advantages it offers over more lightweight solutions. \n\n2. For the output reduction, as we only need to determine whether preserving the current generated token, this pipeline is the same as some work for LLM, such as StreamingLLM [1], H2O [2]. However, the author only compare the Random and Structure strategies. Therefore, I think this is a well-defined problem for LLMs, and the current experimental results cannot convince me. \n\n3. The methodology section of this work requires improvement to enhance clarity and comprehension. It is currently challenging for readers to grasp the entire training and inference pipeline quickly. Additionally, several design choices related to the training process lack sufficient explanation, which may leave readers puzzled about the rationale behind them.\n\n4. Furthermore, the name \"Dynamic-LLaVA\" may lead to misconceptions about the method. It implies that the system dynamically drops the KV cache at each layer; however, it should be clarified that the cache is dropped permanently and cannot be recovered in subsequent layers.\n\n5. The training process is not aligned with the inference process for the output predictor. The former one can see all generated tokens and decide the most important ones, while the latter one can only decide whether to drop the current one based on previous context.\n\n[1] Xiao, Guangxuan, et al. \"Efficient streaming language models with attention sinks.\" arXiv preprint arXiv:2309.17453 (2023).\n[2] Zhang, Zhenyu, et al. \"H2o: Heavy-hitter oracle for efficient generative inference of large language models.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XtZY02t9nu", "reviewer": "Reviewer_UyMq", "summary": "This paper proposes a novel method, Dynamic-LLaVA, which dynamically sparsifies visual context during the prefill stage and vision-text context during the decoding stage of Visual Language Model inference. Specifically, two learnable predictors are designed to identify important visual and language tokens. Additionally, masked softmax is applied during training, and Gumbel-softmax with a Straight-Through Gradient Estimator is used to address gradient flow issues. Furthermore, a batch-parallel strategy is introduced to enable efficient sparsification during inference in the Dynamic-LLaVA framework. Empirical experiments demonstrate that this method can compress approximately 75% of visual context and save around 50% of GPU memory for the KV cache.", "questions": "1. Previous studies have shown that deeper layers exhibit higher attention sparsity. How is sparsification strength adjusted for different decoder layers?\n2. Running predictors introduces additional latency. However, the proportion of this latency relative to overall prefill and decoding time is not disclosed. Additionally, in standard LLaVA decoding, previous KV cache is fetched, and byproduct attention scores are accumulated to assess token importance, which is quick and efficient. However, for \"Online KV Cache Compression,\" running predictor incurs additional latency. And only the current token is used to predict all previous tokens, do you think solely performing prediction on a current token provide enough information? Have you analyzed the wrongly evicted tokens?\n3. Can this method be applied to language models that utilize Group Query Attention?\n4. Have you considered scenarios where long-text context is present during the prefill stage?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper introduces a novel design for two token predictors, formulating visual and textual context compression as an end-to-end optimization problem. Techniques such as Gumbel-Softmax and the Straight-Through Gradient Estimator are employed to enhance training stability. In addition, parallel sparsification inference optimization would provide an effective solution for model serving.", "weaknesses": "1. It is unclear whether the two learnable predictors are applied to each decoder layer. If they are, do they share the same parameters or are they independently parameterized?  If not all layers, please clarify which specific decoder layers the predictors are applied to.\n2. Compressing generated text is primarily beneficial for long-generation tasks. Would you please provide specific statistics on the average output length for each task they evaluated? And discuss how the benefits of their method scale with increasing output length.\n3. Please quantify the additional training time and computational resources required for this method compared to direct KV cache compression approaches, as well as additional prediction overhead compared normal decoding. This would help readers better understand the tradeoffs.\n4. More recent related work about KV Cache compression method should be included, such as TidalDecode (https://arxiv.org/abs/2410.05076) focusing on decoding compression. \n5. A few mathematical representations are not appropriate. For example, in section 3.1, $\\mathcal{I}^I = $\\{ $1, 2, \\cdots, N_{l}^I $}, layer index $l$ should be set for $\\mathcal{I}^I $; And in section 3.3.1, all layers share the same binary masks $\\mathcal{M}^I and \\mathcal{M}^{OT}$? since there are no layer index $l$. \n6. The writing could be improved to make the narrative more concise and clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5yKN3LitSz", "reviewer": "Reviewer_xRRX", "summary": "Dynamic-LLaVA is a framework for efficient inference in multimodal large language models, dynamically reducing vision context redundancy during the prefill and decoding stages, cutting computation by ~75% in prefill, and saving ~50% computation and memory during decoding, without significant loss in performance.  It also designs the specific training and inference strategy for parallel sparsification, making their approach more efficient.", "questions": "There aren’t many issues; the method in the paper is logically clear, the experiments are comprehensive, and the performance is excellent.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The article has a rigorous logical structure, with precise formulations for each method and complete notation. Readers who read carefully can clearly understand the methods explained in the text.\n\n2. The experimental results are excellent; it appears that inference speed and computational cost are significantly reduced with minimal loss in performance.\n\n3. The method design is ingenious; selective approaches are often challenging to train, but the paper provides an effective end-to-end training method that successfully alleviates the gradient flow problem.", "weaknesses": "Although the writing is logically clear and the formulas are complete, this level of detail makes it somewhat challenging for readers.  There are too many formulas here. Perhaps reduce them a bit and add more diagrams to aid understanding. Some high-level schematic diagrams could help; Figure 2 aids understanding to some extent, but it’s still not sufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHTz0FCfSB", "reviewer": "Reviewer_FzrR", "summary": "The paper introduces Dynamic-LLaVA, a dynamic vision-language context sparsification framework aimed at enhancing the inference efficiency of mllms. The main challenge addressed is the increasing computational and memory overhead during inference, particularly as output tokens are generated, which diminishes efficiency benefits of prior vision context reduction methods. Dynamic-LLaVA reduces redundancy in both the vision and language contexts during the prefill and decoding stages, with a specialized sparsification strategy tailored for different inference modes, including decoding with and without KV cache. Experimental results demonstrate that Dynamic-LLaVA significantly reduces computational consumption—by about 75% during the prefill stage and 50% during decoding without KV cache—and saves approximately 50% of GPU memory with KV cache, all while maintaining or even improving model performance. This framework is the first to attempt simultaneous sparsification of both vision and language contexts, achieving efficient inference with negligible performance degradation.", "questions": "1. I am curious why it is not tested on the latest versions of llava such as llava-one-vision, llava-next, etc.\n2. I don’t quite understand why Table 5 uses GQA and VQAv2 as vision evaluations. I think a more vision-centric dataset such as MMVP is needed.\n3. Paper mentions that different sparsification strategies are used in the pre-filling stage and the decoding stage, but the specific differences between the two and the motivation for this design are not clear. In particular, why both visual and language context are sparsified in the decoding stage, while only visual tokens are sparsified in the pre-filling stage, what is the design consideration behind this?\n4. In order to avoid unstable training when there are few samples, paper only sparses the samples whose text length exceeds a certain threshold. However, does this strategy ensure the stability of the model when facing inputs of different lengths? In actual use, will the model cause significant performance fluctuations due to changes in input length? If a model behaves unstable for certain input lengths, this may significantly limit its applicability in real-world applications.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. As an article focusing on efficiency, it grasps the key point very well: solving efficiency problems from the perspective of efficiency. From the numerical results, Dynamic-LLaVA performs well in reducing computation and memory consumption, especially in the pre-filling and decoding stages, reducing the computation by about 75% and the GPU memory overhead by about 50%.\n\n2. The **SPARSIFICATION INFERENCE** part achieves dynamic sparsification of visual and language context by introducing learned predictors, effectively reducing the computational and memory burden in different reasoning modes, making the entire generation process more efficient. I think this is a highlight. Sparsification of tokens during test time is a very useful method, and I personally believe that less is more. These sparse parts should actually improve the representation ability.\n\n3. MaskedSoftmax is a good point to note. From my previous fine-tuning experience, since the generated $logp(y|x)$ is calculated by logits under the corresponding label $y$, the appearance of 0 value in the whole chain process is absolutely harmful. This soft labeling method is very thoughtful.", "weaknesses": "1. In the method section, the description of sparse reasoning and end-to-end training relies too much on symbolic expressions and lacks intuitive explanations. It could be aided by adding some simplified explanations or visual flowcharts to help a wider readership understand the complex technical details.\n2. You can consider adding some evaluations. I personally feel that the current evaluation benchmarks are not comprehensive enough to explain the performance of MLLMs. And you should analyze as thoroughly as possible. You can refer to the Cambrian-1 [1] standard or try to apply the method to the Cambrian model.\n3. In the results of the experimental section, the impact of sparsification on complex tasks (such as generating long texts or multi-round conversations) is not sufficiently verified.\n\n[1] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)