CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Authors: Qinsi Wang
Hancheng Ye
Ming-Yu Chung
Yudong Liu
Yueqian Lin
Martin Kuo
Mingyuan Ma
Jianyi Zhang
Yiran Chen
Venue: ICML 2025 poster (2025)
Keywords: Vision-language model
VLM inference acceleration
Primary area: deep_learning->large_language_models
TL;DR: We propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency.

Abstract: Vision-Language Models (VLMs) excel across diverse tasks but suffer from high inference costs in time and memory. Token sparsity mitigates inefficiencies in token usage, while neuron sparsity reduces high-dimensional computations, both offering promising solutions to enhance efficiency.
Recently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing assumption that they function independently. However, a fundamental yet underexplored question remains: Do they truly operate in isolation, or is there a deeper underlying interplay that has yet to be uncovered?
In this paper, we conduct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and reinforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse inference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical analysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the-art baselines on ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5$\times$ FLOPs reduction and a 10$\times$ overall speedup.
Code is released at [https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main](https://github.com/wangqinsi1/2025-ICML-CoreMatching/tree/main).

CoreMatching: A Co-adaptive Sparse Inference Framework with Token and Neuron Pruning for Comprehensive Acceleration of Vision-Language Models
Qinsi Wang' Hancheng Ye
Ming- Yu Chung'
Yudong Liu' Yueqian Lin
Martin Kuo
Mingyuan Ma
Jianyi Zhang'
Yiran Chen 1
Abstract
1. Introduction
Large language models (LLMs) have achieved outstand- ing performance in various applications and have exerted a significant influence on our daily life (Brown, 2020; Chowd- hery et al., 2023; Touvron et al., 2023a; Kuo et al., 2025; Qinsi et al.). As LLMs become increasingly aware of the physical world, researchers have discovered their potential for understanding visual information (Lin et al., 2023; Liu et al., 2024a; 2025b). As result, series of lasgun-language models (VLMs) such as LLaVA (Liu et al., 2024b), Blip(Li et al., 2022), and I.lamaTTuvvron et al., 2023b) have been introduced, demonstrating impressive performance on tasks like image-based question answering. However, because of the requirement of long image-token inputs, VI.Ms typically demand more time and memory for inference than L.MMs, limiting their practical adoption in real-world scenarios.
Vision-Language Models (VLMs) excel across di- verse lasks but suffer from high inference costs in time and memory. Token sparsity mitigates ineffi- ciencies in token usage, while neuron sparsity re- duces high-dimensional computations, both offer- ing promising solutions to enhance efficiency. Re- cently, these two sparsity paradigms have evolved largely in parallel, fostering the prevailing as- sumption that they function independently. How- ever, fundamental yet underexplored question remains: Do they truly operate in isolation, is there a deeper underlying interplay that has yet to be uncovered? In this paper, we con- duct the first comprehensive investigation into this question. By introducing and analyzing the matching mechanism between Core Neurons and Core Tokens, we found that key neurons and tokens for inference mutually influence and re- inforce each other. Building on this insight, we propose CoreMatching, a co-adaptive sparse in- ference framework, which leverages the synergy between token and neuron sparsity to enhance inference efficiency. Through theoretical anal- ysis and efficiency evaluations, we demonstrate that the proposed method surpasses state-of-the- art baselines ten image understanding tasks and three hardware devices. Notably, on the NVIDIA Titan Xp, it achieved 5x FLOPs re- duction and a 10x overall speedup. Code released at ICCLhCorMaahnnn/rrroiinn.
Token Sparsity, which exploits the high degree of redun- dancy among image tokens, is promising solution to this challenge(Ye et al., 2024; Huang et al.. 2024). Researchers have found that VLMs can retain strong performance using as little as 10% of the total tokens. This has led to exten- sive interest in determining which tokens are essential. For example, PruMerge (Shang et al., 2024) uses the average attention scores between image tokens and text tokens to measure token importance and retains 20% of the tokens with only minor performance loss; FastV (Chen et al., 2025) uses the total attention scores received by other token and finds that more than half of the tokens can be discarded from the second layer. Nevertheless, is worth noting that most existing methods rely on attention scores as guide for selecting important tokens, but their validity and accuracy have not been thoroughly examined.
Another effective approach to accelerating inference is model-internal sparsity, particularly adaptive Neuron Spar- sity. It leverages the highly sparse activations feed- forward network (FFN) layers to skip the computation of inactive neurons, thereby redueing the computational in LLMs. For instance, methods such as Dejavu (Liu et al., 2023) and PowerInfer (Song et al., 2023) employ MI.P based predictors to identify which neurons are activated for given input, achieving up to 93% prediction accuracy. Fur-
'Department of Electrical and Computer Engineering, Duke University, North Carolina, USA. Correspondence to: Qinsi Wang <jjanyi.zhang@ dukc.
Jianyi Zhang
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Languag, Models
Activation
"What color clothes the rabbit wearing?
Non-Activatiin
Neurons
Spocss Artivabing bung
Outpa 184,
Fully Act.
1876 (876 1100111000 1100
Match
Pick
Fore Toke Tokena
Fully Deact
All Nearmaa (4098.1 1008)
Matching Flow
Coro Neurona 10.1 10.1.0.1.0.. ....0.........))
To Tokens KV Cachi
legus (878,60 (098)
Forward
FPN Block sa
Core Neuzens
Frozen
FIELFIL
DECODING
Figure I. Schematic diagram of CoreMatching. In the Pre-filling stage, CoreMatching calculates Core Neurons in the FFN block based the activation. Core Neurons are the most frequently activated group of neurons. Afterwards, CoreMatching matches the neurons activated by different tokens with the core neurons, and selects group of tokens with the largest intersection as the Core Tokens. Only the Core Tokens are passed the subsequent layers. During decoding stage, the model only uses Core Ncurons for calculations, and there are only core tokens in the kv cache. CoreMatching achieves comprehensive acceleration for inference of VLMs.
ther, CoreInfer (Wang et al., 2024) defines core neurons as the subset neurons most frequently activated by an input sentence, demonstrating that only these core neurons are needed maintain performance. While neuron sparsity has shown great promise LLMs, it remains underexplored and underutilized in VLMs.
co-adaptive inference framework. As illustrated in Fig. 1, CoreMatching requires a single step in the pre-filling stage to jointly compute Core Neurons and Core Tokens, thereby achieving sparsity along both the token and neuron dimensions. Furthermore, we conduct a detailed analysis of the theoretical and practical benefits of CoreMatching. Theoretically, propose projection- evaluating the importance of tokens, which takes into ac- count not only attention scores but also angular information. We analyze the effectiveness and efficiency of this crite- rion and theoretically analyze the proportional relationship between core tokens and the criterion. Empirically, we im- plement and evaluate CoreMatching across various tasks and hardware, showing that it exceeds the performance of state-of sere--ffthee th-rtt-rrt token sparsity approaches. CoreMatching delivers comprehensive inference acceleration due to its multi-dimensional sparsity, achieving a 2. 1x speedup the pre-filling stage and 9.2x speedup in the decoding stage. In summary, our contributions are follows:
criterion for
Although both token sparsity and neuron sparsity can indi- vidually accelerate the model, each has limitations in prac- tical applications. Token sparsity primarily speeds up the pre-filling stage and can only provide limited acceleration during decoding by reducing key-value operations. con- trast, neuron sparsity primarily accelerates the decoding stage but cannot achieve a high speedup ratio in the pre- filling stage due to the large number of tokens and the re- sulting low level of sparsity. Hence, combining these two forms of sparsity is promising approach to achieving com- prehensive acceleration. However, an interesting question that has been overlooked in previous work is: What is the relationship between these two sparse spaces?
To answer this question, we first experimentally verify the existence of neuron sparsity in VLMs and the effectiveness of core neurons, which are the most important neurons de- termined by the activation distribution of all input tokens. Furthermore, we investigate the alignment between core neu- rons and tokens, By analyzing how core neurons influence token inference, we uncover an insightful matching pattern: tokens whose activated neurons most closely match the core neurons correspond to the most critical part for determining the output. Inspired by this, we define core tokens as the set of tokens that have the largest intersection of activated neurons with the core neurons among all tokens.
We introduce Core Tokens, which are those tokens that activate the largest number core neurons. Experi- ments show that core tokens consistently capture subset of tokens most relevant to the output.
We present Projection-guided Criterion and lever- age it to explain why Core Tokens outperform those selected by previous attention score-based methods.
We propose CoreMatching, a co-adaptive inference framework. Experimental results confirm that not only surpasses baselines on the performance of various tasks but also achieves comprehensive acceleration.
Building on these insights, we propose CoreMatching,
2

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Table 1. Verification of the importance of core neurons. We retain different proportions of core neurons on Llava-1.5-7h, and calculate the accuracy of the model on Tex
where fp(s) denotes the count of each neuron marked as the core neuron across all tokens, defined in Eq. (4).
M
Keep Ratios
0.2
0.4
0.6
0.8
1.0
fp(8) = ([p(n:)))
Σ
T(n E Cp(am))),
(4)
Accuracy
45.1%
53.2%
55.8%
56.3%
57.8%
where TI(.) is an indicator function that returns one if is in Cp(Tm) else zero. Percentile(f,()), B) denotes the 8-th percentile of fp(8).
Longth-
Teat: Length-
Text Longth-
Lo
0.0
For an input 8, core neurons are the subset of neurons that are most frequently activated and exhibit the highest activation values. In LLMs, core neurons have two key characteristics. First, they are crucial for the in- ference, as only core neurons retained can maintain the performance. Second, they exhibit predictability; when input is sufficiently long and semantically rich, the core neurons identified during the pre pre-filling stage closely align with those in the decoding stage.
Effectiveness.
az
Figure 2. Verification of the predictability of core neurons. We visualized the core neurons of the 25-th layer of Llava-1.5-77 when input text token at different lengths. p = 0.2, g = 0.4. We selected the first 256 neurons. can be seen that when input semantics are sufficient, core neurons are almost unchanged.
2. Intrinsic Relations of Two Paradigms
In this section, to delve deeper into the relationship between two sparse spaces, we propose a novel interaction paradigm between tokens and neurons. Through this approach, we investigate how important neurons and tokens mutually in- fluence and determine each other.
To validate that core neurons applicable to VLMs, we con- ducted experiments on Llava-1.5-7b (Liu et al., 2024a). As shown in Tab. and Fig. 2, core neurons also exhibit these two characteristies on Llava. Specifically, by retaining only 40% of the neurons, the performance decreases by merely 3%. This demonstrates that in VLMs, given specific input, we can accelerate inference by preserving only a small set of core neurons. Note that the calculation of core neurons takes into account the activation distribution of all tokens. There- fore, core neurons are essentially a part of neurons that capture the most critical features, which are determined by all tokens input.
2.1. Identifying Core Neurons for Tokens
In the FFN block in LLMs, there are typically two linear layers, W,, R Ni X2, Wa R. N, For single token, denote the input representation of the FFN layer x, the output y can be expressed as:
A = (xW.), =
2.2. Deriving Core Tokens from Core Neurons
where a represents the activation function, such as ReLU or SiLU. Intermediate output A } where an is the activation value of the n-th neuron in W..
In the previous section, we defined core neurons, which are selected by counting the key activations determined by all input tokens. In this section, we explore how core neurons in turn select important tokens.
Core Neurons. The concept of Core Neurons was first introduced in Corelnfer (Wang al., 2024), which repre- sents group of neurons that are most important for input to maintenance performance.
Matching of Neurons and Tokens. To explore the rela- tionship between core neurons and tokens. We first study the impact of core neurons on different tokens. Specifically, we explore the changes in token inference when only core neruons are retained compared using all neurons.
Specifically, for a single token x, the token-wise core neu- rons (z) are defined as the ncurons with the top p maxi- mum positive activation values, that is,
For a single token the set of neurons activated by it can be denoted as 1'(x)= = { n an (æ) > 0). When only the core neurons are retained, token x can only activate neurons in the intersection [(x)) (C)())). Consequently, information that token Jr able to transmit, i.e., I(xr), can be regarded as proportional the number of intersections between the neurons it activates and core neurons,
C,(x) = {nl on Percentile(At p)}}
(2)
where At = {ax | an > 0, an E A} represents the set of positively-active activated neurons, and Percentile(A* .p) denotes the p-th percentile of the positive activation.
=[x1, [T1, (2,,,.,..... . [MM] containing M tokens. The sentence-wise core neurons C²(s) are defined as the top 3 of neurons that appear most frequently in the core neurons of all tokens, U (=;), which formulated as
For sentence
(5) Therefore, to analyze how much information core neurons retain for different tokens, we can equivalently measure the number of core neurons each token activates.
I(x) x [(x) n (s))
C. (s) {n fo(n; s) 2 Percentile(,,(()),)) 8)), (3)
3

---

CorcMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
the rabiin woueen??
What color are the Vhat colour bouse backgroundd
What color ts the road purbec?
2000
3800
100
Figure 3. (Upper) Distribution of [Z))) n (s) of image token. The experiment conducted Llava-1.3 1.5-7b, and selected the 10th layer. The input image is the rabbit on the left, and the input text the text token above the image. We use red font to emphasize the key points of the text token. (Note that since regarded as random sample.) (Lower) Core token under different inputs. The left is the schematic diagram of the maximum geometric distance method to select the threshold. The right side is the core token retained under the distribution of the corresponding image above.
neurons themselves account for 40% of neurons, intersection of about 2000 can be
We conduct experiments on LLaVA-1.5-7B. focusing on the number of core neurons activated in the 10th layer for various tokens across different text inputs. The results, illus- trated in Fig. 3 (Upper), reveal intriguing patterns. Specifi- cally, image tokens that exhibit stronger correlations with the text prompt significantly activate more core neurons than other image tokens. Notably, even for the same image, the distribution of activated core neurons varies depending on the emphasis of the text prompt. Image tokens most relevant the text consistently trigger a substantially higher activation of core neurons.
tokens, core tokens are consistently the most relevant the text tokens. Moreover, the core tokens adaptively retain the most critical portion of the tokens based on the input, without being constrained by fixed number limit.
In Section 4, we empirically demonstrate that core tokens represent the most crucial set tokens for maintaining performance. By retaining only these core tokens during in- ference, the model can achieve nearly lossless performance.
3. CoreMatching
In this section, building pose CoreMatching, a co-adaptive inference framework. We further provide theoretical analysis explaining why core neurons can guide core tokens.
previous findings, we pro-
From the above analysis, we know that core neurons can preserve the information flow of important tokens while suppressing that of unimportant tokens. The subset of tokens that activate the largest number of core neurons are the most critical for the inference.
3.1. Overall Framework
As shown in Fig. 1, CoreMatching precomputes core neu- rons at each FFN layer during pre-filling, and identifies core tokens at layer and discards non-core tokens. Dur- ing decoding, the model predicts the next token using the prefetched core neurons and core tokens. CoreMatching pruned unimportant tokens to speed up the pre-filling stage. Meanwhile, the reduced number of neurons also acceler- ates the decoding stage. Our method is training-free and plug-and-play Algorithm is provided in the Appendix .B.
Core Tokens. Motivated by the fact that core neurons can guide the selection of important tokens, we define core tokens here and verify their effectiveness.
From the previous analysis, tokens that exhibit higher cor- relation with text tokens activate more core neurons. There- fore, for input s, we can define core tokens (s)) as tokens activate the largest number of core neurons,
(s) T'm
[Tx..) (s) T (6)
3.2. Theoretical Advantage
where Te the "knee" threshold derived from the distribu- tion of [C())cc) (s)) for all tokens in s using the maximum geometric distance method (Garg& & Walker, 1990). Fig. 3 illustrates how Te is obtained. Notably, since Ti is com- puted adaptively based on the data distribution, there is no need to manually fix the number of core tokens.
To analyze why core tokens are effective, first explore the question: How can we evaluate the importance of image to- kens in VLM? In existing research (Shang et al., 2024: Chen et al., 2025), the attention scores between image tokens and text tokens are widely used as a metric for evaluation. However, no work has yet analyzed its optimality. There- fore, in this section, we first propose superior criterion quantify the importance of image tokens. Furthermore,
Fig. 3 (Lower) shows the retained core tokens under dif- ferent text inputs. It can be observed that, for various text

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
|| Poo (A
Voll
OM
OM V
GjM 41
|| (0 Pll
(a) Forms of OM
(b) Attention Score (c) Projection Value
(a)W, y W
(b) W.Wa
(c)Attention Score
Figure 4. Diagram of attention score and projection value. v/ indi- cates the token is reserved under this matric. indicates discarded.
Figure 6. (a) (b) Numerical visualization of W,,W, W2 and W.WT. They are approximately I. (c) Visualization of a. It can be seen that the an (diagonally) is much higher than the others.
obtained using the projection magnitude. Obviously, the projection magnitude filters out important tokens more effee- tively, whereas the attention score alone considers only the vector magnitude and ignores angular information, which introduces more noise.
(a) Attention score (b) Projection Value
(c) Core Tokens
Figure Comparison of three metries. The input is rabbit in Fig. and "What color clothes is the rahbit wearing?". Experiment conducted on Llava-1.557h and the 10-th layer selected.
Theoretical Analysis. Although the projection value is superior evaluation metric, its computation is both complex and time-consuming Therefore, we show that it is actually related to the proposed core token.
mathematically derive and demonstrate that core tokens are proportional to this criterion.
To demonstrate this, we first introduce two observations:
Projection-guider Criterion. In the inference of VLM, the only module where tokens influence each other is the attention mechanism. And since the final output is deter- mined by the last token in the input, we only need to focus on how all image tokens affect this last token.
Observation 1. Wg and W are nearly approximately or- thogonal each other, i.e., Wg W. x OT. 0 is constant. (W W.. Wa) are approximately orthogonal matrices, i.e., W WT = XI. À is constant.
We present experimental evidence supporting this obser- vation in Fig. 6, and provide comprehensive experiment and more detailed discussion in Appendix C. This is a fas- cinating phenomenon in VLMs. In fact, further investiga- tion reveals that such matrix orthogonality proven to exist and be effective in MLPs and CNNs. For example, (Li et al., 2019; Bansal et al., 2018) have shown that this orthogonality helps improve the training stability and generalization ability of the model.
For single token, suppose its input to the Attention block is y. where y a (x: W.WWa, W.. Consider sequence of tokens [v1, .2..... ). for the last token VM, its computation in the Attention layer can be expressed as:
previously
ÿar = LayorNorm(yss). V in Wa,
0.M = Softmar (() W₂) (Nx )) OM = anm V1 02m V/2 + .... + crAsar VM:
Observation 2. For activation function output vectors A, and AM, cos( AM)) is proportional to the number of intersections of activated neurons, i.e., [(x,) n [(+M)]
where a&M is the attention score between the i-th and the M-Uh token. Om is the output vector of the M-th token.
As shown in Fig. 4 (a), when we focus on the O2, we can observe that Om is the sum of multiple vectors. This indicates that Os depends not only on the attention scores arM, but also on the angle between V; and Oи. Based on this insight, we propose that the Projection-guis guided Criterion for evaluating the influence of the i-th token on the M-th token should be the projection of aaa V, onto Om, given by
This observation is illustrated in Fig. 7. An intuitive un- derstanding is that if x; and ZM activate more of the same neurons, A, and Ay will have more positive values in com- mon positions, making (o,A,A,, larger. We provide more visualizations in Appendix C.
Building on these two observations, we provide insights into how projection values are influenced in LLMs. Following the inference order, we first analyze the impact of input vec- tor y on projection values, then examine how the activation layer affects these input vectors.
Projose (aiss 1l) = |laus Vil cos (Z(V,, OM)). (8)
where |Prooo., (auM Vl is the projection value of OaM V: on Om- And cos (Z(V,, Owr) is the cosine value of the angle between the V/2 and OM vectors. Fig. 4 pro- vides schematic illustration of this projection-basl based metric. Furthermore, in Fig. 5 (a) and (b), we compare the results of using the attention score as an evaluation metric with those
Theoretical Insight The projection values between the i-th and M- token, i.e., (Proios(aaa V.) |] is propor- tional to the cosine value of the angle between y and yas-
Justification. To analyze how y, and yas influence the pro-
5

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
LSd 1BC Rec 1200
506 251 Gearsoate Ninber
A/ 280 20 180 ANe
(a) What time is now?
(b) What's the word on the button?
(a) Layer
(b) Layer 10
(c) Layer 15
Figure 7. The distribution cos(A, AM) and [Ta,) n [(*w) of LLLVA--.557b,, clearly shows proportional relationship.
jection value, we first decompose it into form related to these two tokens. From Eq. 7, Oas is sum of vectors in different directions. Since the self-attention score aM is typically much higher than oras for other tokens (as shown in Fig. 6 (c)), we can simplify the projection by assuming that az is primarily determined by ausa VaI, i.e.
(c) What's the word on the sign?(d) What's the person wearing?
Figure Examples of CoreMatching sampled tokens.
A,, and y/4 A.Wa- Based Observation Wa is scalar multiple of unitary self-orthogonal matrix, applying the same rotation to any input while preserving the inner product and angle between any two input vectors (proof is provided in Appendix C). Thus, we can have:
P (asMs VI)ll |aaa Vill cos(/osV VM)), (9)
where CrAs- Softmazt Wa) (ŪM W.)"). Since the Softmax function is monotonic, that is, (mfm())()) x x.
cos(<(Bi, Bar)) = cos (o,,
(13)
Furthermore, based on Observation 2, cos <(A,, AM)) x T(x;) n [( Combined with Eq. 19, we can get cos (ww, ymr) x [Txx) nn [( which is consistent with Insight 2.
We can have agas x (9. Wq) (ÜMa Ws)". Therefore.
|| Vill cos( (VV;, VM))
(14)
(6s Wi) (Uas W&) [[Val| [Vs]lcos(([ cos(< VMr)
(10)
Eq. 14 shows that activation layers adjust token angles by controlling the intersections of their activated neu- rons. More shared activated neurons lead to smaller angles and greater mutual influence.
(9. W.. UM Wx) (Vi Valll
(W, W )ol) (W, /
Based on the Observation 1, W2 W OI, W, W. x AI. Therefore, combining Eq. and Eq. 10, can have
Correlation with Core Tokens. Based on the predictabil- ity of core neurons, when the semantics of the input is stable enough, the neurons activated by Z are approximately the same as those of the core neurons, I( = (s).. Therefore, by combining Insight and can get
[|Projo' (asas Villl x 0X doa) Kūs 0sx) VVVall (11)
Given that Y is the result of y after LayerNorm, á and y share the same direction, and ý ||= 1, it holds that (gi, wir)
cos(<(y), vM)). We can have
||roojo (aaM V.)Il x [T(a;) n cg))))
(15)
Projo (OaM Vi)ll x cos( (v0 UM)))
(12)
Eq. 15 shows that the number of intersections between core neurons and the activated neurons in one token reflects how much information this token contains that contributes to the next-token prediction.
which is consistent with Insight 1.
Eq. 12 shows that the angle between different tokens directly affects their mutual interaction in attention layer. The closer the angles of two tokens, the greater their mutual influence. Therefore, the angle should be an essential indicator for token importance evaluation, which has not been fully demonstrated and utilized previous work.
that using only the atten- does not consider the angular information. In contrast, core neurons are directly proportional to the pro- jection value, incorporating both the absolute influence and the angular component, thus constituting a more accurate metric. Fig. 5 shows comparison between using core to- kens and using the attention score as metrics. Fig. 8 shows the effectiveness of CoreMatching for different inputs.
From the above analysis, tion
Furthermore, we analyze how the activation influences the relative angle between y and yas.
Theoretical Insight 2. cos( (() is proportional to the number of intersections of neurons activated in the previous activation layer, i.e.,
Moreover, to the best of our knowledge, this is the first work to investigate the intrinsic relationship between token spar- sity and neuron sparsity. Our findings reveal the inherent correlation between these two sparsity patterns and high- light that the relative angle between tokens a crucial yet previously overlooked factor in token selection.
[(x:) C [(
Justification. To analyze how the activation layer affects cos( (y), yM)),, we first decompose its computation for- mula. Suppose the activation output of the i-th token is
6

---

CorcMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models
Table Comparison with SoTA vision context sparsification methods vision understanding benchmarks. The hest results bolded.
The results of the other methods are from their papers. CoreMatching achieves the best performance in most benchmarks using smaller number of image tokens. The "Free" indicates whether method training free (i.e., can be applied directly to MLLMs without training.)
Method
Free
Token
TFLOPs
VQAv2
GQA
SciQA
TextVQa
POPE
MME
MMB
SEED
VisWiz
MM-Vet
LLVVA-15-7B
576
10.1
78.5
62.0
66.8
58.2
85.9
1510.7
64.3
66.0
50.0
31.1
+
X
146
2.5
76.8
57.4
68.3
57.1
84.0
1462.4
60.9
62.8
42.5
25.0
V
/
144
3.2
75.1
57.5
68.7
56.2
81.0
1458.9
63.5
62.8
47.8
26.3
VoCo-LLAMA
X
128
2.2
76.9
59.8
61.0
59.1
LLaVA-TRIM
X
121
2.2
76.4
61.4
69.1
85.3
1461.3
67.4
65.8
48.1
28.0
Dynamic-LLaVA
X
115
2.5
78.0
61.4
69.1
57.0
85.0
1479.8
64.1
64.6
50.2
29.5
CoreMatching
64
2.1
78.5
61.9
69.5
54.5
85,9
1506.5
64.6
66.1
49.8
29.6
L1aVA-1.5-13B
576
19.6
80.0
71.6
61.3
85.9
1531.3
67.7
68.2
53.6
36.1
PruMerge+
X
146
4.9
77.8
58.2
71.0
58.6
84.4
1485.5
65.7
49.7
28.0
FastV
144
6.0
77.0
60.1
72.8
59.0
83.2
1470.3
66.9
65.4
52.3
31.3
IJAVA-TRIM
X
121
5.8
75.4
59.0
72.8
54.8
86.3
1438.0
65.7
65.9
53.2
30.3
Dynamic-LLaVA
X
115
78.8
62.5
72.4
59.6
86.5
1563.3
66.9
66.5
52.8
34.8
CoreMatching
98
4.3
79.4
63.1
72.8
58.3
87.0
1529.9
68.5
67.4
53.2
34.9
Table 3. Hardware performance comparison with the most advanced token sparse and neuron sparse methods. Experiments performed on a single NVIDIA Titan Xp (12GB). The input 610 tokens, and the output is 64 tokens. We test the latency of the baseline method based the LLaVA framework. For PowerInfer, migrate the predictor provided by Llama2-7b the LLM module of LLaVA-1.5-76
Method
Cost
Sparisity
Memory
Pre-filling
Decoding
Free
Neurons
Token
Kv Cache
FFN
Total
Ratio
TFLOPs
Lat(s)
GFLOPs
Lat(s)
LLVAA-BB
X
304.8
8.26
15.63
1.00x
10.1
2.23
0.845
44.2
PruMerge+
X
89.73
8.26
15.41
1.01x
2.5
1.16
0.822
39.5
FastV
X
V
88.98
8.26
15.41
1.01x
3.2
1.15
0.822
39.1
Conamic-LLaVa
X
V
74.40
8.26
15.39
1.01
2.5
1.15
0,820
37.6
PowerInfer
X
V
X
304.8
1.83
9.20
1.70x
2.2
2.03
0.516
11.6
CoreMatching
V
48.99
1.65
8.62
1.82x
2.1
1.05
0.486
4.81
4. Experiments
Low Cost. Unlike Dynamic-LLaVA and PruMerge+, which require post-training specific to each model, Core- Matching requires no training or fine-tuning, making it a plug-and-play solution for any model. Additionally, Core- Matching does not require pre-specification of the number of tokens to retain, it dynamically determines the opti- mal number of tokens based on the input. Consequently, CoreMatching achieves comparable performance with fewer tokens on average. Specifically, CoreMatching only re- tains 10% and 17% tokens on average for Lla5A-1.5-7b and 1.aVA-1.5-133 respectively, compared to 20% for PruMerge+ and FastV. Moreover, CoreMatching introduces neuron sparsity in addition token sparsity, further reduc- ing computational requirements during the decoding stage.
Our experiments conducted three levels. First, we evaluate the performance of CoreMatching on various tasks to demonstrate its effectiveness (Sec. 4.1), After that, we deploy CoreMatching on different hardware devices to ver- ify the improvement of hardware performance (Sec. 4.2).
Finally, we perform ablation studies further analyze the effectiveness of each module in CoreMatching (Sec. 4.4).
The experimental details are provided in the Appendix D.
4.1. Task Performance.
High Accuracy. Tab. 2 presents the performance of CoreMatching various tasks. As shown, CoreMatch- ing demonstrates minimal performance degradation across all tasks and achieves higher accuracy than sfate-of-the-art baselines on most tasks. Notably, on LlaVA-1.5-7b. Core- Matching achieves lossless performance on datasets such as VQAv2 and GQA while using only about 10% tokens. Furthermore, on SciQA and MMBench, CoreMatching even outperforms the original model. For Lla Core- Matching demonstrates clear advantage over other meth- ods, achieving near-lossless accuracy across all datasets.
4.2. Hardware Performance
Reduced Inference Memory. Tab. compares the hard- ware resources required for inference between CoreMatch- ing and other sparse inference methods on an NVIDIA Titan Xp (12GB). CoreMatching can simultaneous sparsity across both token and neuron dimensions, CoreMatching signifi- cantly reduces the memory footprint the KV cache during inference well as the memory required for neurons during

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Prejiting
Decoding
Lateacy Rodiusion D0
Length
Length 64
Leogth 512
Table 4. Peformance comparison with popular acceleration meth- ods on OCR, chart, and document understanding tasks, The basic models are LLaVA-1.55 and LLaVA-1.55 1.5--......B......... tasks are eval- uated on test set. "TF" indicates TFLOPs. "OCRB" indicates
2
OCRBench
WN WIN
Oct
W/N
Method
Token
TF
DocVQA
InfoVQA
ChariQA
OCRB
A12D
.76
10.1
22.2
22.3
18.2
31.3
55.2
prejkingg
Diccoding
PreMenget
146
17.9
21.1
15.8
47.3
Latemny Rediemion
FustV
144
3.2
20.6
22.7
16.2
28.1
51.2
CoreMaihing
48.4
22.7
55.2
LLaVA-13-138
576
19.6
24.6
25.1
18.2
33.6
59.2
PreMeeges
146
146
23.6
16.9
27.8
49.1
FudVV
144
6.0
22.9
21.5
17.1
30.1
51,9
CoreMating
563
24.8
17.5
59.3
Metboos
Table S. Performance of CoreMatching the Qwen2.5-VL-3B and Qwen2.5-VL. VL-7B models on OCR, chart, and document un- derstanding tasks. The results of the original model are extracted from the Qwen2.5-VL. technical article. Since the number of image tokens in Qwen2.5-VL dynamic, we take the average of different tasks as the number of tokens used. "Neur" indicates Neurons.
Figure 9. Latency comparison okken-oulyneurrossoollyyyyee
sparse on NVIDIA TiTAN Xp. W/T means only core tokens, W/N means only core neurons, and W/TN means CoreMatching. The number the bar means how many seconds it took.
Prefiling
Decooking
Langh
Lmngth MM
Lengrs M3
Model
Tokens
Neur
DoeVQA
InfoVQA
ChariQA
OCRB
AI2D
3
Clande-3. 3.5 Sonnec
95.2
74.3
90.8
78.8
81.2
Geminn 1.5 Pio
93.1
81.0
87.2
75.4
88.4
5
86.7
84.5
0.5
Owen2.5. 2. VL 3B
235.2
11008
93.9
77.1
84.0
79.7
81.5
CoreMatching
4403
93.2
77.1
83.6
79.2
81.6
Qwen2,5-V1 7B
234,9
11008
95.7
62.6
87.3
86.4
83.9
WT
WIN
WIN
WIN
WIIN
00 W.1
WIN
W71N
CarcMatching
23,8
4403
94.8
8233
87.0
85.6
84.2
Prefiling
Decoding
Lengta
Langih 14
532
1.2
and collected results for FastV and PruMerge on these tasks to ensure fair comparison. As shown Tab, 4, CoreMatch- ing consistently outperforms other acceleration methods most tasks while incurring only minimal performance degradation compared to the original models. Notably, for tasks such as Doc and InfoVQA, which often require understanding only small portions of text within an im- age, CoreMatching's dynamic token pruning allows it to achieve near-lossless performance using significantly fewer tokens-on average, less than 10% of the original token.
WTT
WIN
WIN
021
WTN
W7TN
W71
Metbods
Figure 10. Latency comparison of tokee--ully/nnunnn-ooolybbllll sparse on NVIDIA RTX 6000.
decoding. When the input token length is short, CoreMatch- ing requires only about half the memory of the original model during decoding, eliminating memory-bound limi- tations on resource-constraing reducing inference latency. For long input token sequences, CoreMatching also can effectively reduce primary memory usage by minimizing the KV cache size.
devices and significantly
Model Generalization. To further evaluate the generaliz- ability and effectivencss of our method on more advanced architectures, we conducted comprehensive experiments on Owen2.5-VL-3B and Qwen2.5-VL-7B. two representative LVLMS that incorporate dynamic resolution mechanisms adaptively allocate computational resources based visual content complexity. These models present more challeng- ing evaluation setting due to their sophisticated token selec- tion and multi-scale processing capabilities. As summarized in Tab. 5, CoreMatching continues to exhibit robust perfor- mance under this setting. Notably, it achieves near-lossless performance while utilizing only approximately 10% of the visual tokens, demonstrating its capability to distill and preserve essential visual information with high efficiency. Even more, on the A12D dataset, CoreMatching not only retains accuracy but actually surpasses the performance of the original full-token model, underscoring its adaptability and potential enhance model performance through better
Reduced Inference Latency. In contrast methods that achieve partial acceleration, such as token sparsity for pre- filling time reduction or neuron sparsity for decoding time reduction, Catchngg delivers comprehensive inference acceleattinn through multidimensional sparsity. As illus- trated in Tab. 3, CoreMatching achieves 2.1 x speedup in the pre-filling stage and 9.2x speedup in the decoding.
4.3. Generalization Analysis
Task Generalization. To verify the task generalization of our method, we conducted extensive evaluations on LLaVA- 1.5-7B and LLaVA-1.5-13B across additional OCR and chart/document understanding tasks. We also reproduced
8

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Table 6. Performance comparison of different LS on vidco rea- soning benchmarks. The base model for all acceleration methods Video-LLaVA. Results of PruMerge and FastV are taken from
son
the original papers.
MSVD-QA
MSRVT-QA
AetivityNet-QA
Methods
Size
Accuracy
Score
Accuracy
Score
Accuracy
Score
(a) 5-th Layer
(b) 15-th Layer
(c) 25-th Layer
HowenRiL.M
32.2
16.8
24.7
Figure 11. Number of tokens required at different layers.
VidcoChat
56.3
2.8
45.0
2.5
2.2
LaMAAAddiperr
54.9
43.8
2.7
34.2
Video-LLaMA
7B
51.6
2.5
29.6
1.8
12.4
1.1
pronounced gains as the output token length increased.
Vidco-ChanPP
64.9
49.3
2.8
35.2
Vidoo-LaVA
70.7
3.5
45.3
Token-only sparsity primarily accelerated the pre-filling stage, with minimal impact on the decoding stage (limited to reducing KV computation). Conversely, neuron-only spar- sity mainly accelerated the decoding stage, offering little improvement during the pre-filling stage. By simultaneously reducing memory and computational load, CoreMatching achieved higher acceleration rates in both stages.
FastY
7B
71.0
3.9
57.0
3.5
PruMerge
71.1
3.9
58.4
3.5
48.3
PruMerges
7B
71.1
3.9
59.3
3.6
47.7
4.4
CoreMatching
71.6
3,6
48.4
Table The average number of core tokens for different tasks,
Type
Choice
Malti
Questions und Answers
Long Text Generate
Tisks
SaQA
YOAx 63.5
GQA
VisWix 76.9
Te 64.1
POPE
MME
Met 93.9
Number
27.9
76.6
71.1
66.9
Core Tokens Across Different Tasks. Due to the input- adaptive of CoreMatching, it can automatically determine how many tokens retain based on the data distribution of each task. Tab. 7 shows the number of core tokens retained for different task types. As observed, CoreMatching adap- tively adjusts the required tokens according to task difficulty. For instance, simpler tasks such as multiple-choice question answering (c.g.. SciQA where one must choose between options A or B) require only 27.9 tokens on average. In contrast, more challenging tasks involving longer text gen- eration (c.g.. MM-Vet, which requires generating medical records from images) need around 93.9 tokens on average. Notably, even for the most difficult tasks, CoreMatching needs fewer than 20% of tokens, outperforming other token sparsity methods in terms of token reduction.
token selection. These results collectively validate that Core- Matching remains effective when applied to state-of-the-art LVLMs with dynamic resolution, reinforcing its potential as a general-purpose visual token reduction strategy that scales across model sizes and architectural paradigms.
Performance on Video Task. To verify the scalability of CareMat on Video tasks, we used Video-LLa VA as the base model and conducted experiments on multi- ple video tasks including MSVD-QA, MSRVT-QA, and ActivityNet-QA. As shown in Tab. 6, CoreMatching con- sistently outperforms existing advanced acceleration meth- ods across multiple video reasoning benchmarks. Specifi- cally, on MSRVT-QA, CoreMatching reaches an accuracy of 59.8%, outperforming origianl Video-LLaVA (59.2%). On ActivityNet-QA. CoreMatching improves accuracy (48.4%) compared to the origianl Video-LLaVA (45.3%). This sug- gests that token redundancy in video frames can be substan- tial, and strategic pruning not only reduces computational overhead but may also enhance model performance by clim- inating noisy or irrelevant tokens. These results collectively highlight the strong potential of Matchnng as a general- purpose acceleration framework for video LLMs.
Tokens Required at Different Layers. Another crucial aspect of token sparsity is understanding how many tokens are needed at each layer. Using our proposed optimal evalua- tion metric, we visualize and analyze the token requirements across different layers. As illustrated in Fig. 11, image to- kens that are closely correlated with text tokens consistently exhibit higher projection values, and as the layer depth in- creases, the number of tokens with high projection values gradually decreases. This suggests that the model focuses on a larger number of tokens in the earlier layers, and more tokens can be pruned in later layers, aligning with results from previous studies.
4.4. Ablation Study
Module Ablation. To further analyze the contributions of token sparsity and neuron sparsity to inference acceleration, we conducted ablation experiments on various hardware. As shown in Fig. 9 and 10, we compared the acceleration achieved by token sparsity alone, neuron sparsity alone, and CoreMatching (sparsity applied to both tokens and neurons) during the pre-filling and decoding stages. Across different devices, models, and output token lengths, CoreMatching consistently achieved significant acceleration, with more
5. Conclusion
This paper introduces CoreMatching, a co-adaptive sparse inference framework that accelerates VLMs by leveraging the matching relationship between core neurons and core tokens. Additionally, we also propose a more principled token measurement criterion and theoretically derive how activation layers influence token interactions, providing new insights into how VLMs process image and text information.

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Acknowledgements
Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Qinsi Wang, Jianyi Zhang and Yiran Chen disclose the support from NSF 21 2112562, ARO -- and NAIRR Pilot project NAIRR240270. We thank area chair and reviewers for their valuable comments.
Huang, W., Zhai, Z.., Shen, Y., Cao, S., Zhao, F,, Xu, X., Ye, Z., and Lin, S. Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification. arXiv preprint arXiv:2412.008777 2024.
Impact Statement
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 6700- 6709, 2019.
This paper aims to advance the field of adaptive sparse infer- ence for VLMSs. We believe our work has significant poten- tial for applications in deploying large models on resource- constrained devices. While our research may have various societal implications, we do not find any particular aspect that requires special emphasis in this context.
Kuo, M., Zhang, J., Ding, A., Wang. Q., DiValentin L., Bao, Y. Wei, W., Li, H., and Chen, Y. H-cot: Hijack- ing the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai ol/o3, deepseek-rl, and gemini 2.0 flash thinking. arXiv preprint arXiv:2502.1289) 2025.
References
Alizadeh, K., Mirzadeh, I., Belenko, D., Khatamifard, K., Cho, M., Del Mundo, C. C., Rastegari, M., and Fara- jtabar, M. Llm in a flash: Efficient large language model inference with limited memory, arXiv preprint arXiv:2312.11514, 2023.
Li, B., Wang, R., Wang, G., Ge, Y., Ge, Y., and Shan, Y. Seed- Benchmarking multimodal Ilms with gener- ative comprehension. arXiv preprint arXiv:2307.16122 2023a.
Bansal, N., Chen, X., and Wang. Z. Can we gain more from orthogonality regularizations in training deep networks? Advances in Neural Information Processing Systems, 31, 2018.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Boot- strapping language- image pre-training for unified vision- language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022.
Brown, T. B. Language models are few-shot learners. arXiv preprint 140.. 2020.
Li S., Jia, K., Wen, Y., Liu, T., and Tao, D. Orthogonal deep neural networks. IEEE transactions on pattern analysis and machine intelligence, 43(4):1352-1368 2019.
Chen, L., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer, 2025.
Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W. X., and Wen, J.-R. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305.10355. 2023b.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113,, 2023.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-llava: Learning united visual represen- tation by alignment before projection. arXiv preprint arXiv:231..01122, 2023.
Garg, A. and Walker, P. Prediction of total knee mo- tion using a three-dimensional goaputer-graphies model. Journal of Biomechanics, (3(1):45-58, 1990.
Lin, Y., Wang, Q. Ye, H., Fu, Y., Li, H., Chen, Y., et al. multimodal memory
Hippomm: Hippocampal-inssee for long audiovisual event understanding. arXiv preprint ar arxi::2504.1 X.1729904.10739, 2025.
Goyal, Y., Khot, T., Summers-Stta D., Batra, D., and Parikh, D. Making the in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 4904-6913, 2017.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Gurari, D., Li, Q.. Stangl, A. J., Guo, A., Lin, C., Grau- man, K., Luo, J., and Bigham, J. P. Vizwiz grand chal- lenge: Answering visual questions from blind people. In
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
10

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216 233. Springer, 2025a.
Wang, Q. Ke. J., Liang, Z., and Zhang, S. Mathnas: if blocks have role in mathematical architecture design. Advances in Neural Information Processing Systems, 36: 47446, 2023.
Wang. Q.. Vahidian, S., Ye, H., Gu, J., Zhang, J., and Chen, Y. Coreinfer: Accelerating large language model infer- ence with semantics-inspiree adaptive sparse activation. arXiv preprint arXiv:2410.188 18311, 2024.
Liu, Y.. Sun, J., Lin, Y., Zhang, J., Yin, M., Wang, Q., Zhang, J., Li, H., and Chen, Y. Keyframe-oriente vision token pruning: Enhancing efficiency of large vision language models on long-form video processing. arXiv preprint 1X14 2025b.
Xue, Z., Song, Y. Mi, Z., Chen, L., Xia, Y., and Chen, H. Powerinfer-2: Fast large language model inference smartphone. arXiv preprint arXiv:2406.06236 066282 2024.
Liu, Z. Wang, J., Dao. T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., et al. Deja vu: Contextual sparsity for efficient Ilms inference time. In International Conference on Machine Learning, pp. 22137-22176. PMLR, 2023.
Ye, X., Gan Y., Huang, X., Ge, Y., Shan, Y., and Tang, Voco-Ilama: Towards vision compression with large lan- guage models, arXiv preprint arXiv:2406. 12275, 2024.
Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-25211 2022.
Yu, W., Yang, Z., Li, L., Wang. J., Lin, K., Liu, Z., Wang, X., and Wang, L. Mm-vet: Evaluating large multi- modal models for integrated capabilities, arXiv preprint arXiv:2308.0244 2023.
Qinsi, W. Ke, J., Tomizuka, M., Keutzer, K., and Xu, C. Dobi-svd: Differentiable svd for Ilm compression and some new perspectives. In The Thirteenth International Conference on Learning Representations.
Shang, Y., Cai, M., Xu, B., Lee, Y. J., and Yan, Y. Llava- prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 15388, 2024.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, 2019.
pp.
Song, Y., Mi, Z., Xie, H., and Chen, H. Powerinfer: Fast large language model serving with consumer-grade gpu. arXiv preprint arXiv:2312. 1456, 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-.,, Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F,, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302. 13971, 2023a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.1397) 2023b.
Wang, Q. and Zhang, S. Dgl: Device generic latency model for neural architecture search on mobile devices. IEEE Transactions on Mobile Computing, 23(2):1954-1967, 2023.

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Organization In this appendix, we provide in-depth descriptions of the materials that are not covered in the main paper, and report additional experimental results. The document is organized follows:
A- Related Work
Algorithm
C- Assumption Explanation
D- Experiments Settings
E- Additional Experiments
F- Visualization of Results
A. Related Work
A.1. Activation Sparsity
In a single Feed-Forward Network (FFN) block in LLMs, there are typically two linear layers, W., Wa E RNN For a single token, denote the input representation of the FFN layer as x. the output y can be expressed as:
A = (xWW),
y aAW,
(16)
where a represents the activation function, such as ReLU or SiLU. Intermediate output A = [a), (2.00...... C4N]] where d, is the activation value of the n-th neuron.
Previous work (Song et al., 2023; Xue et al., 2024; Wang & Zhang, 2023; Wang al., 2023; Lin al., 2025) indicates that individual tokens in LLMs exhibit significant activation sparsity. For example, in the OPT-30B, single token activates only approximately 10% of the neurons (Alizadeh al., 2023). Therefore, if the activated neurons can be accurately predicted in advance, substantial amount of activation computation can be eliminated, thereby accelerating model inference without compromising performance. This potential has attracted considerable attention from researchers. For example, Deja Vu (Liu et al., 2023) inserts an MLP predictor into each FFN block to forecast which neurons will be activated, achieving a prediction accuracy of 93%. PowerInfer (Song et al., 2023) introduces the concepts of hot neurons and cold neurons to respectively represent frequently and rarely activated neurons, and accelerates inference by intelligently allocating hardware resources. LLM in Flash (Alizadeh al., 2023) and PowerInfer2 (Xue al., 2024) optimize this algorithm for mobile devices, thereby reducing the DRAM requirements for LLM inference mobile phones.
Notably, recent work CoreInfer (Wang et al., 2024) proposed sentence-level adaptive activation sparsity inference method without the need of predictors. Corelnfer identifies a set of core neurons that most frequently and strongly activated for each input sentence. Experiments demonstrated that for a given input sentence, LLMS require only a static set of core neurons to maintain performance. Leveraging sentence-level sparsity, CoreInfer eliminates the need for frequent neuron switching during the decoding stage, achieving a 10.33 speedup on NVIDIA Titan Xp GPU with negligible performance loss.
The effectiveness of CoreInfer highlights the ability of core neurons capture the most critical information of the input, underscoring their remarkable potential. In this paper, we further explore the characteristics of core neurons in VLMs, and propose a novel one-pass co-adaptive sparsity inference.
A.2. Token Sparsity
The notorious quadratic complexity in Transformers is well-known issue and one of the key bottlenecks in scaling input sequence lengths. In VLMs, this problem becomes even more pronounced as the length of input image tokens increases. To address this challenge, a multitude of adaptive token sparsity methods have been proposed (Huang et al., 2024) et al., 2S24)SShang et al., 2Chen et al., 2025). For instance, Prumerge (Shang et al., 2024) observed the sparsity the distribution of attention scores between class token and visual tokens, and employed attention scores as an evaluation metric to determine which tokens should be discarded. FastV (Chen et al., 2025) further explored and demonstrated the inefficiency and--lay approach that achieved 45% reduction in FLOPs on 1.5---1..-13b.
of visual attention in VLMs, proposing These methods validate that VLMs require only small subset of important tokens to achieve nearly lossless performance.
However, the acceleration and efficiency gains from purely token-level sparsity are limited, especially since the speedup benefits during the decoding phase are significantly diminished. In this paper, we thoroughly investigate the intrinsic relationship between token sparsity and neuron sparsity and propose a co-adaptive sparsity method. Our method achieves comprehensive acceleration at nearly zero cost.
12

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models
B. Algorithm
The algorithm of CoreMatching is shown in Algorithm 1. the pre-filling stage, we calculate core neurons each FFN block and calculate core tokens the I-th layer. After calculating the core tokens, we only use the core tokens in the subsequent inference process. In the decoding stage, we only use the retained core neurons for calculation.
Algorithm 1 Visonison: Co-adaptive sparse inference
Input: Sentence [1), #2, ..., war]: Tokennpruning layer C. Sparsity hyperparameters p. B.
Stepl: Pre-filling Stage
for layer | = 1, 2, ..., L do
Capture the activation of all tokens;
for token 21, 42, .... #M
|
Record neurons with the largest 9 activation as token-wise core neurons, (x))..
end
Record the most frequent 3% neurons in
sentence-wis neurons C,² (8);
Keep only neurons C (s) the device.
If then
Record the neurons activated by each token I(=);
Calculate the number of intersections [[(>) n Cg (s)]: maximum geometric distance to get threshold Tx; Select all x; that [xx;) (s)) Ts core tokens; Keep only core tokens for inference.
end end
Step2: Decoding Stage
for layer 1 1, L do
Use (s) kept the pre-filling stage for inference. end
C. Assumption Explanation
In this section, we provide comprehensive experimental proofs and discussions for the two observations proposed.
For Observation 1, we show Wo @W T, Wy@Wy.T, w,@Wp, of all different layers of LLaVA-1.5-7B in Fig. 12, 13, and 14. It can be seen that different layers have this orthogonal relationship.
In fact, the orthogonal relationship of matrices in neural networks has been studied since long time ago. In particular, (Li et al., 2019) proposed new regularization method that encourages the weight matrix of the neural network to maintain orthogonality during training by introducing a self-orthogonaliit module. This method helps to improve the training stability and generalization ability of the model. (Bansal al., 2018) explores adding orthogonal regularization to weights during training to improve training stability. The author proposed orthogonal regularization method for weights, aiming to solve the gradient vanishing and explosion problems encountered by deep convolutional neural networks during training.
can be seen that modules with orthogonality are found in various different models to improve the training stability and performance of the model. To the best of our knowledge, we are the first work to intuitively show this orthogonal performance LLM,, which can be more fully explored subsequent research.
13

---

CorcMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models
Liver3
Live
Lawer
Lewer
Lawrd
Lover
10
1
Uy 24
UNT
Low
Lyp
My
13
Figure 12, Visualization of Wa T at different layers LLa A-1.7-bb.
Lewero
Lovel
Lewe?
Layer
Use
Uy 29
Figure Visualization @W n at different layers LLaVA-L.5-7b
Tayen
Lavr
Loye
+0:2011
1aye 27
Lapa
Leer
Ugr
Lyye
Figure Visualization Wy, Ww@Wy.l T' at different layers LLaVA-1.5-7b.
14

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models
For Observation 2, show the distribution of 32 layers of LLaVA-1.5-7bb in Fig. 15. It can be seen that starting from the 4-th layer, cos( ( Ax)) is proportional the number of co-activated neurons. This is actually in line with intuition. On the one hand, the more neurons A, and Am co-activate, the more non-zero values they have in the same positions, and the larger the product. On the other hand, A, and AM have positive values in more of the same dimensions, indicating that their directions are closer.
Layer
Layer1
Layer
Leyer3
Layend
Layers
Layer
Layer?
0.30
wuns-
Coolk
6.00
c.00
2000
2006
2730
2000
2103
2003
20E6
2006
Zite
2883 2009
2003 1999
1803
3300
1890
300
Layer
Layera
1ayer11
Laye112
Layer
Laye11
Layer15
2
CostA
1890 Co-act Not Num
1893
1750
Co-act Nouron Num
Coaett Neuron Num
Colact heuron Num
Coaac: Noua: Neuron Num
Crattt Nouron Num
Coass Nouzon Num
Co-act Ne.ron Num
Layeri6
Layer 37
Layer
1ayer
Layer
Layer
Layer
Layer23
Coosaa 200
Costa
2750 3x0 1250
27e 1h0 1251
3003
3993
2153
1030
125
XXE
1500
2hE 1990 1200
1900
1891
wre
L4
Layer 25
Layer
Layer
Layer 30
Layer33
Custd JAMI
Costie 2011
Costee 2011
3
2003
2X1
1000
2053
108
3200
2093
3206
2093
1000
2250
2500
2008
Co Ne.ron Nun
Co Nouron Num
Co-act Nouron Num
Coaac houton Num
Co-ac: Nouron Num
Crest Nouron Num
Colest Nouron Num
Co-act Neuron Nun
Figure 15. Visualization of cos (<(A;, Am)) and co-act neurons number at different layers in LLaVA-1.5-7bb
cos(<(y), VMr) can be written For proof of cos cos(L VM)) = (os,L(A, (M, AM))
cos(<(yi, yar)) = (A, AM Wa) (all(lleslllll
-..WWa ) / IIlaar)
(17)
(MAA,AA)) /1llellll
where 77 is a constant based on Observation 1. Furthermore, since Wa is an orthogonal matrix, we have
|| || = || 2 (A.Wa)
(18)
= A. ) = nAA = n/1
which means ||w.ll = A,I. Substituting this into Eq. 17 we can have
cos(<(y), VM)) = A,, Am) = cos (Ai, Aw))
(19)
This shows that the orthogonal matrix Wa does not change the angles between the input vectors.
D. Experiments Settings
Models and Tasks. Our main experiments are conducted on LlaVA-1.5-70 and LlaVA-1.5-13b, using FP16 for all models. Following the same evaluation setting as LlaVA, we evaluate our methods on ten classical tasks, including VQAv2 (Goyal al., 2017), GQA (Hudson & Manning, 2019), VizWiz (Gurari al., 2018), SciQA (Lu et al., 2022), TextVQA (Singh al., 2019), POPE (Li al., 2023b), MMBench (en) (Liu al., 2025a), SEED (image) (Li et al., 2023a), and MM-Ve Vet (Yu et al., 2023). We also provide additional results on more models in the Appendix.
Hardware. We conduct experiments on two distinct hardware configurations. NNVIDIA Quadro RTX 6000 (24GB), representing perforaeform hardware scenarios. In contrast, NVIDIA TITAN XP GPU (12G), representing low- performance hardware scenarios. We also provide experimental results on more hardware devices in the Appendix.
15

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprchensive Acceleration of Vision-Language Models
Baselines. Given the multi-dimensional sparsity of CoreMatching, we compare our method against approaches using either token-level sparsity or neuron-level sparsity. For token sparsity, we compare with state-of-the-art token pruning methods such PruMerget, FastV, VoCo-LLaMA, LLaVA-HIRED, LLaVA-TRIM, and Dynamic-LLaVa For neuron-level sparsity, since there has been no prior work specifically on activation sparsity in VLMs, we adapt the predictor from PowerInfer to Llava to emulate the time required for MI.P-based activation sparsity on Llava.
Implementation Details, Core.natching uses the same hyperparameter settings across all activation layers for all models. For the computation core neurons, we follow the hyperparameter setting from CoreInfer with p = 0.2, 8 = 0.4. For core tokens, we set I 2, consistent with FastV.
E. Additional Experiments
We supplement here the experimental results that are not included in the main text.
E.1. Lateney on NVIDIA A100
Fig. 16 shows the latency comparison on NVIDIA A100. It can be seen that CoreMatching can achieve excellent acceleration
effects even without memory limit. And as the batch size increases, the acceleration effect becomes more and more obvious.
Prefilling
Decoding
Latency Reduction (P)
1.4
Length
Length
64
Length 512
1.2
0.06
0.26
0.08
2.00
0.00
35.00
0.06
622
0.20
112
3530
LLava-71 7b
0.04
1.2
0.03
1
0.04
W/T
WIN W/TN
Ori W/T WN
W/TN
Oni WT W/N W/TN
Methods
Prefilling
Decoding
Latency Reduction (X)
1.4
Length
Length 64
Length 512
1.2
0.09
0.29
0.09
2,05
2.02
30-0
70
15
62
1.50
0.30
0.44
0.04
04
0.04
70
7h
Ori
W/T
W/N
W/TN
Ori
W/T Methods
WN
W/TN
Oni
WT
W/N
W/TN
Figure 16. Latency comparison of tokken-onlynneeurns-oolly/oooooo sparse NVIDIA A100.
E.2. Experimental Results on LLaMA3.2
We show the experimental results of CoreMatching on LLaVA in the main text. In order to demonstrate the model generalization of CoreMatching, we further show the experimental results on LLaMA3.2 here. Since LLaMA-3.2 uses the crossAttention layer, the activation layer cannot be used to guide the token pruning of attention. We directly perform activation sparseness without token sparseness. The experimental results shown in Tab. 8. For LLaMA3.2-11 CoreMatching still achieve near-lossless performance. This shows the applicability of CoreMatching to models of different architectures.
Table 8. Experimental results of CoreMatching LLaMA-3.2-1Ib.
Methods
TextVQa
MME
VQAV2
VisWiz
POPE
GQA
Number
69.7
1384.5
81.4
52.5
88.7
68.6
Number
66.5
1381.2
79.8
49.8
87.8
66.2
16

---

CoreMatching: A Co-adaptive Sparse Inference Framework for Comprehensive Acceleration of Vision-Language Models
F. Visualization of Results
In this section, we visualize the experimental results of CoreMatching. shown Fig. 17, different inputs, CoreMatching can always retain the tokens that are important to the output.
What it now?"
(a)Sampled
(b) What is word on the button?
Sampled
(c)" "Wat's the word the sign?"
(e)Sampled (d)"What the person in the picture wearing?"
(d)Sampled
What colour is the Orange?"
(e)Sampled
How many dogs in the picture?"
(l) Sampled
"What colour is the cola?"
(g) Sampled
Wha colour the bus?"
(h) Sampled
Figure 17. Examples of CoreMatching sampled tokens for different inputs.
17


Reviews:
[{"review_id": "DDJ2X1WQL1", "reviewer": "Reviewer_TfEt", "summary": "This paper proposes a sparse inference framework to reduce the inference latency of Vision-Language Models (VLMs). The main method combines token compression and neural unit compression techniques, and establishes a connection between the two. Through the CoreMatching approach, it is possible to significantly reduce the inference latency and memory consumption of the LLaVA model without substantially compromising the model's performance. The method does not require additional training and has great application potential.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TfEt,\n\nWe sincerely thank you for taking the time to review our paper and for providing valuable feedback. We are pleased to hear that you found both our theoretical analysis and experimental validation to be comprehensive. We appreciate the opportunity to address your suggestions, which we believe will further strengthen our work.\n\n***Suggestion1. It is recommended to conduct further demonstrations on more tasks, especially OCR tasks that are highly sensitive to the number of visual tokens, some dense OCR scenarios, and some chart understanding tasks.*** \n\n**A1.** We greatly appreciate your constructive suggestion. In response, we have conducted additional experiments on OCR, chart understanding, and document understanding tasks. The results are summarized in Table 1 and Table 2 of the anonymous link below:\nhttps://drive.google.com/file/d/1tYVMhd12V6Qgx6aqyLEK1rG8lCPOJvcb/view?usp=sharing\n\nSpecifically, we supplemented the following two experiments:\n\n**Experiment 1. Results on the LLaVA-1.5 model on more tasks.** To facilitate comparison with existing acceleration methods, we conducted extensive evaluations on LLaVA-1.5-7B and LLaVA-1.5-13B across additional OCR and chart/document understanding tasks. We also reproduced and collected results for FastV and PruMerge on these tasks to ensure a fair comparison. As shown in Tab.1 of the linked document, CoreMatching consistently outperforms other acceleration methods on most tasks while incurring only minimal performance degradation compared to the original models. Notably, for tasks such as DocVQA and InfoVQA, which often require understanding only small portions of text within an image, CoreMatching’s dynamic token pruning allows it to achieve near-lossless performance using significantly fewer tokens—on average, less than 10% of the original token count.\n\n**Experiment 2. Results on the Qwen2.5-VL model.** To further evaluate our method on more advanced models, we conducted experiments on Qwen2.5-VL-3B and Qwen2.5-VL-7B, which are LVLMs utilizing dynamic resolution. As shown in Tab.2 of the linked document, CoreMatching continues to demonstrate strong performance, achieving near-lossless results using only around 10% of the tokens. Remarkably, it even outperforms the original model on AI2D, highlighting the effectiveness and adaptability of CoreMatching on state-of-the-art dynamic-resolution LVLMs.\n\nWe will include these experimental results in the final version of the paper. Thank you again for your insightful and constructive feedback!\n\n***Suggestion2. Another suggestion is to perform ablation studies on high-resolution images.***\n\n**A2.** Thank you for your valuable suggestion. In response, we conducted ablation studies on DocVQA, InfoVQA, and ChartQA—datasets composed of scanned documents or detailed visual charts that typically require models to process localized and densely packed text. For example, many questions in DocVQA reference specific words or lines within the document, making high-resolution input essential for accurate understanding.\n\nTo preserve image resolution during VLM inference, we used the Qwen2.5-VL model, which supports dynamic resolution. We performed ablation studies to evaluate the impact of different components of our method by testing performance under three settings: token-only sparsity, neuron-only sparsity, and combined sparsity. We evaluated both task performance and inference efficiency on NVIDIA Titan Xp, a resource-constrained device representative of real-world deployment scenarios. The results are shown in the table below.\n\n|                      | **DocVQA** | **InfoVQA** | **ChartQA** | **Pre-filling (s)** | **Decoding(s)** |\n| -------------------- | ---------- | ----------- | ----------- | ------------------- | --------------- |\n| Qwen2.5-VL-3B        | 93.9       | 77.1        | 84.0        | 2.3                 | 4.5             |\n| Neuron Sparsity Only | 93.2       | 77.0        | 83.7        | 2.3                 | 1.6             |\n| Token Sparsity Only  | 93.5       | 77.1        | 83.8        | 1.2                 | 4.3             |\n| CoreMatching    | 93.2       | 77.1        | 83.6        | 1.0             | 1.4         |\n\nAs demonstrated, token-only and neuron-only sparsity each result in negligible performance degradation, but only accelerate a single stage of inference. In contrast, our combined method maintains near-identical performance while providing significant speedups in both the pre-filling and decoding stages, highlighting its practical value in high-resolution scenarios.\n\n**We will include these results in the revised version of the paper. Additionally, we plan to add more qualitative examples on high-resolution images to further illustrate the applicability of our approach.** We sincerely appreciate your insightful and constructive feedback.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "NA", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\nThe sparse inference framework CoreMatching proposed in the paper shows good results in experiments. Both the theoretical arguments and experimental validations in the paper are quite comprehensive.\n\nWeaknesses:\nNo obvious weaknesses.\nIt is recommended to conduct further demonstrations on more tasks, especially OCR tasks that are highly sensitive to the number of visual tokens, some dense OCR scenarios, and some chart understanding tasks. Another suggestion is to perform ablation studies on high-resolution images.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kiWQxj8iOK", "reviewer": "Reviewer_LK6j", "summary": "This paper proposes CoreMatching, combining the sparsity of core neurons and core tokens for VLMs. Experiments show that CoreMatching excels in both model accuracy and inference efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LK6j,\n\nWe sincerely thank you for taking the time to thoroughly read our paper and for your positive evaluation. We are excited to have the opportunity to address your questions and concerns. These discussions and revisions will further strengthen our work.\n\n***Concern 1. Lack of accuracy on some OCR tasks.***\n\n**A1.** Thank you for raising this valuable concern. In response, we have conducted additional experiments on OCR, chart understanding, and document understanding benchmarks using both LLaVA and Qwen models. The results are presented in Tab.1 and Tab.2 of the anonymous link below:\nhttps://drive.google.com/file/d/1tYVMhd12V6Qgx6aqyLEK1rG8lCPOJvcb/view?usp=sharing\n\nSpecifically, we supplemented the following two experiments: (1) On LLaVA-1.5-7B/13B, CoreMatching outperforms FastV and PruMerge on OCR and chart/document tasks with minimal performance drop, using on average less than 10% of tokens (Tab.1). (2) On Qwen2.5-VL-3B/7B (dynamic-resolution LVLMs), CoreMatching maintains near-lossless performance with ~10% of tokens and even surpasses the original model on AI2D (Tab.2).\n\nWe will include these results in the updated version of our paper. Thank you again for your helpful suggestion.\n\n***Concern 2. Lack of accuracy comparison with PowerInfer and PowerInfer-2.***\n\n**A2.** We appreciate your constructive suggestion and would like to clarify why such comparison experiments were not included in our current submission:\n\n1. **Different Target Domains.** PowerInfer 1&2 are primarily designed for accelerating  LLMs, while our work specifically focuses on accelerating VLMs. It remains unclear whether PowerInfer-style methods can be effectively adapted to popular VLM architectures such as LLaVA. Therefore, it is unfair to directly put PowerInfer 1&2 into VLM and compare them with our method.\n2. **Different Usage Costs.** PowerInfer 1&2 are based on predictors trained specifically for each model, while our method is training-free. To conduct a direct comparison, we would need to re-implement both PowerInfer 1&2 and retrain layer-wise prediction modules from scratch for VLM architectures, which is computationally expensive and unfair to our training-free methods. \n\nWe appreciate your suggestion to include comparison with neuron sparsity methods. We will make our best effort to incorporate these experiments in the revised version. Once again thank you for your valuable suggestion.\n\n***Concern 3. The article's technical novelty is limited, as it merely combines two existing approaches.***\n\n**A3.** Thank you for bringing up this concern, though we respectfully disagree. Below, we outline our reasoning in detail:\n\n- **Methodologically, our approach is not a simple combination of existing token pruning techniques.** To the best of our knowledge, our work is the first to utilize *neurons* to guide token sparsity in VLMs. *While prior token pruning methods often rely on attention-score-based importance metrics, our work is the first to demonstrate the suboptimality of such metrics (see Section 3.2).*\n- **Theoretically**, **we propose a novel interpretation of the interaction between the neuron and token dimensions in VLMs**, offering new insights into how VLMs perceive visual and semantic content. In Section 3.2, we present two theoretical insights. *These insights shed light on how the FFN and Attention blocks operate and interact.* We believe this theoretical framework can serve as a foundation for future innovations and a deeper understanding of both VLMs and LLMs.\n- **Experimentally, our approach outperforms existing approaches that simply combine token sparsity and neuron sparsity in terms of both task performance and effectiveness.** In terms of efficiency, our method achieves double sparsity using only a single forward pass in the neuron dimension without twice computation; in terms of performance, our core tokens and core neurons have consistency guarantees: core tokens tend to activate more core neurons, and retaining only core tokens helps stabilize core neuron activation. This ensures that combining neuron and token sparsity in our method does not introduce performance conflicts - this is the main limitation of simply merging the two sparsity techniques.\n\nIn summary, we argue that our work does not merely combine neuron and token sparsity, but instead proposes a novel and unified approach that integrates the two dimensions. This comprehensive integration is crucial for accelerating VLM inference in a principled and effective manner. We will highlight these contributions more explicitly in the revised version. Thank you again for raising this important concern.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "The claims (i.e., model accuracy and deployment efficiency) are clear. Theoretical analysis and efficiency evaluations are provided to support the claim.", "methods_and_evaluation_criteria": "Yes, both the methods and evaluation criteria are clear.", "theoretical_claims": "I check all the proofs.", "experimental_designs_or_analyses": "I check the experiments. One concern is the lack of accuracy comparison with PowerInfer [1], and PowerInfer-2 [2]. Another concern is that I am curious about the accuracy of some ocr (high-resolution) tasks, e.g., DocVQA & InfoVQA.\n\n[1] Song Y, Mi Z, Xie H, et al. Powerinfer: Fast large language model serving with a consumer-grade gpu[C]//Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles. 2024: 590-606.\n\n[2] Xue Z, Song Y, Mi Z, et al. Powerinfer-2: Fast large language model inference on a smartphone[J]. arXiv preprint arXiv:2406.06282, 2024.", "relation_to_broader_scientific_literature": "This paper contributes to the efficiency improvement of deploying VLMs. Combining the sparsity of core neurons and core tokens.", "essential_references_not_discussed": "Not found.", "other_strengths_and_weaknesses": "This article provides a detailed discussion on the sparsity of core neurons and core tokens, but the technical contribution is insufficient, as it merely combines these two approaches.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. More experiments. As in the 'Experimental Designs Or Analyses' part.\n\n2. More discussions regarding technical contribution, as in the 'Other Strengths And Weaknesses' part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2DZrtDcVyL", "reviewer": "Reviewer_T5zD", "summary": "The authors explore a fundamental question on jointly leveraging token and neural sparsity to enhance the inference efficiency of vision-language models (VLMs). The paper introduces the concept of core neurons and investigates their correspondence with core tokens. Building on this observation, the authors propose CoreMatching, a co-adaptive sparse inference framework that exploits the interplay between token and neuron sparsity to accelerate VLMs. Experimental results demonstrate that the proposed method outperforms state-of-the-art baselines across ten image understanding tasks and three hardware platforms.\n\n## update after rebuttal\nI appreciate the authors’ detailed responses, which have successfully addressed my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer T5zD,\n\nThank you for your thoughtful review and encouraging feedback. We're pleased you found the paper well-motivated and well-structured, and we welcome the opportunity to address your comments to further improve our work.\n\n***Q1. How to determine these two hyperparameters to balance token sparsity and neuron sparsity?***\n\n**A1:** Thank you for your constructive question. **In fact, our method does not require manual balancing between token sparsity and neuron sparsity.** During the pre-filling stage, we first dynamically select important tokens via the parameter-free maximum geometric distance strategy. During the decoding stage, we determine core neurons based on core tokens using two hyperparameters (α, β). The key steps are summarized below:\n\n1. **Token selection strategy**: For selecting core tokens, we adopt a maximum geometric distance strategy to adaptively select informative tokens based on each input, avoiding manual sparsity settings. Tab.4 reports the average number of tokens retained across tasks.\n2. **Neuron selection strategy**: For selecting core neurons,  we selected based on core tokens, ensuring theoretical consistency between token and neuron sparsity: core tokens tend to activate more core neurons. As for the hyperparameters α and β, which determine the retained neuron ratio, we refer to the detailed analysis in CoreInfer[1]. We also conducted ablation studies on LLaVA (see Tab. 1 in the main text), and ultimately adopted α=0.2 β=0.4 as the optimal values for balancing performance and efficiency.\n\nWe will highlight this point more clearly in the revised version. Thank you again for your thoughtful question.\n\n***Q2. Could attention mechanisms also be incorporated?***\n\n**A2.** Thank you for your question. Regarding acceleration within the Attention Block, we would like to offer the following clarifications:\n\n1. **Although our method is primarily applied in the FFN block, it also contributes to reducing the computational cost of the Attention block.** This is because our approach jointly sparsifies tokens and neurons in a unified pass, significantly reducing the sequence length input to Attention layers. Since attention complexity in VLMs scales linearly with token count, our token sparsification notably accelerates attention computation.\n2. **Our method is compatible with other attention acceleration techniques.** For instance, our implementation already integrates the widely-used Flash Attention. As our sparsification occurs within the FFN block, it can seamlessly combine with other attention acceleration methods for further efficiency improvements.\n\nWe appreciate your insightful question and will highlight this point more clearly in the revised version.\n\n***Q3. In the right column of L128, “3%” should be “4.6%.”***\n\n**A3.** Thank you for your careful reading and for pointing out this issue. Upon verification, you are correct — the accuracy drop should indeed be 4.6% rather than 3%. We will correct this in the revised version of the paper. We sincerely appreciate your attention to detail.\n\n***Q4. Why did the authors use the NVIDIA Titan XP?*** \n\n**A4.** Thank you for the thoughtful question. Our experiments covered three GPUs to highlight device generality: **Titan XP** (low-performance; Fig. 9, main text), **RTX 6000** (mid-performance; Fig. 10, main text), and **A100** (high-performance; Fig. 16, Appendix). We selected Titan XP for primary comparisons as it exemplifies resource-constrained environments—where inference acceleration is critical—and notably cannot efficiently run the unoptimized 7B model due to memory limitations.  Our results on RTX 6000 and A100 also demonstrate consistent speedups across different hardware configurations. We thank you again for the thoughtful question.\n\n***Q5. Explanation about token sparsity benefiting pre-filling and neuron sparsity benefiting decoding.***\n\n**A5.** Thank you for your interest—we are happy to clarify:\n\n- **Token sparsity** primarily benefits the pre-filling stage, where many tokens (especially image tokens in VLMs) are processed simultaneously, and computational cost scales linearly with sequence length. Thus, reducing tokens significantly lowers overhead here. During decoding, however, tokens are processed sequentially with cached keys and values, limiting token sparsity's impact mainly to minor KV-cache computations.\n- **Neuron sparsity** primarily accelerates the decoding stage. Decoding processes single tokens individually, with FFNs dominating computation. Sparsifying neurons effectively reduces FLOPs in this stage. In pre-filling, batching multiple tokens makes neuron-level sparsification impractical, as full activation tracking remains necessary.\n\nWe greatly appreciate your constructive suggestion and will expand this discussion in the revised version.\n\nSincerely,\n\nAuthors\n\n[1] CoreInfer: Accelerating Large Language Model Inference with Semantics-Inspired Adaptive Sparse Activation.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The proof is technically sound.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This paper proposes a novel approach that jointly applies token sparsity and model sparsity to accelerate VLLMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well-motivated, addressing an interesting, fundamental, and important research question on VLM efficiency. It is well-structured, beginning with preliminary concepts and systematically deriving the relationship between core neurons and core tokens. The experimental results are compelling, demonstrating the effectiveness of the proposed approach.", "other_comments_or_suggestions": "Please see the questions.", "questions_for_authors": "1. In the proposed method, the authors retain the top $\\rho\\%$ most activated neurons as token-wise core neurons and identify the most frequent $\\beta\\%$ neurons as sentence-wise core neurons. A key question is how to determine these two hyperparameters to effectively balance token sparsity and neuron sparsity.\n\n2. The proposed method focuses solely on core neurons in the FFN block. Could attention mechanisms also be incorporated to further enhance efficiency?\n\n3. In the right column of L128, “3%” should be “4.6%”.\n\n4. I am curious about the choice of hardware for the throughput experiments. Why did the authors use the NVIDIA Titan XP (12GB)? Given that this GPU is relatively old and may not reflect the performance of modern hardware, could the authors clarify the rationale behind this selection?\n\n5. In Section 4.3, the authors state that token-only sparsity primarily accelerates the pre-filling stage with minimal impact on decoding, while neuron-only sparsity mainly speeds up the decoding stage with little improvement during pre-filling. This observation is interesting. Providing additional explanations or insights into the underlying reasons would further enhance the clarity and impact of this discussion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CJ8Qrofwof", "reviewer": "Reviewer_cD4w", "summary": "CoreMatching combines token sparsity and neuron sparsity through the interaction between core neurons and core tokens, achieving comprehensive acceleration of VLMs. The proposed projection-guided criterion provides a more accurate way to evaluate token importance, and the co-adaptive framework effectively reduces computational costs while maintaining high performance, which has important practical significance for deploying VLMs on resource-constrained devices.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer cD4w,\n\nWe would like to thank you for taking the time to review our paper and provide valuable feedback. We appreciate the opportunity to address your questions and concerns. We believe these discussions and revisions will help further improve the paper.\n\n***Q1. Is this technique compatible with FlashAttention?***\n\n**A1.** Thank you for your valuable question. **Our method is fully compatible with FlashAttention**, as CoreMatching sparsifies in the FFN block, orthogonally complementing FlashAttention in the attention block. *Our implementation already leverages FlashAttention* by default through the LLaVA-1.5 framework, and we will clarify this explicitly in the revision.\n\n***Q2. Why not directly use angle instead of core neurons to guide core token selection?*** \n\n**A2.** Thank you for your insightful comments.\n\n1. **Theoretically, angle-based metrics alone may cause conflicts when combined directly with neuron sparsity, due to inconsistent token-neuron interactions.** Our neuron-guided token sparsification ensures consistency—core tokens activate core neurons, stabilizing neuron activation and preventing negative interference.\n2. **Empirically, our new experiments (Tab. 4, provided link: https://drive.google.com/file/d/1tYVMhd12V6Qgx6aqyLEK1rG8lCPOJvcb/view?usp=sharing) demonstrate improved task performance from combining neuron-guided token selection with neuron sparsity.** Using angular metrics without alignment indeed degraded performance, supporting our theoretical insights. We agree angle-based metrics remain valuable for token sparsity and will expand this discussion accordingly.\n\nWe will expand this discussion in the revised paper and thanks for your question.\n\n***Q3. More experiments on video datasets.***\n\n**A3.** Thank you for the constructive suggestion. We have conducted additional experiments on video datasets, and the results are presented in Tab. 3 of the anonymous link below:\nhttps://drive.google.com/file/d/1tYVMhd12V6Qgx6aqyLEK1rG8lCPOJvcb/view?usp=sharing\n\nAs shown, CoreMatching continues to outperform advanced acceleration methods such as FastV and PruMerge on video tasks. Interestingly, due to the high redundancy in video frames, token pruning sometimes even improves performance over the original model. This highlights CoreMatching’s strong potential in accelerating video LLMs. We will include these results in the revised version. Thank you again for your valuable suggestion.\n\n***Q4. How does CoreMatching perform on LVLMs using dynamic resolution?***\n\n**A4.** Thank you for the helpful question. We have added experiments on Qwen2.5-VL across OCR, chart, and document understanding benchmarks. These results, along with additional experiments on LLaVA, are shown in **Tab.2** of the anonymous link:\nhttps://drive.google.com/file/d/1tYVMhd12V6Qgx6aqyLEK1rG8lCPOJvcb/view?usp=sharing\n\nAs shown, CoreMatching achieves near-lossless performance even after pruning approximately 90% of the tokens, demonstrating its effectiveness in state-of-the-art LVLMs with dynamic resolution.  We will include these results in the revised version. Thank you again for your thoughtful suggestion.\n\n***Q5. Can CoreMatching be extended to accelerate training?***\n\n**A5.** Thank you for the constructive question. We believe CoreMatching can significantly accelerate training, with substantial potential:\n\n- **Forward pass:** CoreMatching enhances efficiency in both pre-filling and decoding, benefiting supervised fine-tuning and RL-based training (e.g., addressing decoding bottlenecks in RL methods like GRPO).\n\n- **Backward pass:** CoreMatching reduces computational costs during gradient updates by activating only core neurons (thus updating fewer weights) and pruning tokens (reducing memory usage and computation).\n\nWe plan to explore this in future work and will add relevant discussion in the revised version.\n\n***Q6. Essential references not discussed: lack of comparison against merging-based methods.***\n\n**A6.** Thank you for pointing out this important omission. We are happy to provide the following discussion:\n\nIn fact, CoreMatching is complementary to merging-based approaches. On one hand, the angle similarity and neuron activation metrics proposed in CoreMatching can serve as alternative importance or similarity scores in merging-based methods, replacing conventional attention-based metrics. On the other hand, token merging strategies can also be incorporated into CoreMatching to better retain globally informative tokens for complex tasks.\n\nWe will cite the suggested works and add a discussion in the revised version. Thank you again for the valuable recommendation.\n\n***Q7. Will you open-source the code after acceptance?***\n\n**A7.** Thank you very much for your interest in our work. **Yes —** **we fully intend to release the code upon acceptance**.\n\nSincerely,\n\nAuthors", "claims_and_evidence": "The claims in this paper should be correct.", "methods_and_evaluation_criteria": "The used evaluation criteria is standard in token pruning / merging. The experimental results are impressive.", "theoretical_claims": "The theoretical claims seem to be solid.", "experimental_designs_or_analyses": "My main concerns are about experimental verification.\n1. **Compatibility with flash-attention**: Is this technique compatible with flash-attention? As an acceleration-focused work, it should align with mainstream acceleration frameworks. I did not see any discussion from the authors about this.\n\n2. **Necessity of Core Neurons**: The paper claims \"The angle should be an essential indicator for token importance evaluation\". While I agree with this claim, why not directly integrate the angle and attention score instead of using core neurons to guide core token selection? Is there experimental evidence supporting the superiority of this design choice?\n\n3. **Insufficient Validation of Generalization**:\n(1) **Video datasets**: Many existing methods report performance on video datasets, _e.g._, Video MME, but CoreMatching does not provide such results.\n(2) **LVLMs using dynamic resolution**: LLaVA-1.5 does not use dynamic resolution. How does CoreMatching perform on LVLMs using dynamic resolution (e.g., Qwen2.5-VL or LLaVA-NEXT)?\n\n4. **Applicability to Training Acceleration**: Can CoreMatching be extended to accelerate training?", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "Token pruning and merging are two critical technical paths for token sparsity. While the authors have thoroughly compared pruning-based methods, there may be a lack of comparison against merging-based methods, _e.g._, [1,2].\n\n[1] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. In International Conference on Learning Representations, 2023.\n\n[2] Chen Ju, Haicheng Wang, Haozhe Cheng, Xu Chen, Zhonghua Zhai, Weilin Huang, Jinsong Lan, Shuai Xiao, and Bo Zheng. Turbo: Informativity-driven acceleration plug-in for vision-language large models. In European Conference on Computer Vision, pages 436–455. Springer, 2025.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "Please refer to comments about experiments and references. In addition, will you open-source the code after acceptance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: CoreMatching combines token sparsity and neuron sparsity through the interaction between core neurons and core tokens, achieving comprehensive acceleration of VLMs. The proposed projection-guided criterion provides a more accurate way to evaluate token importance, and the co-adaptive framework can reduce computational costs while maintaining high performance, which has important practical significance for deploying VLMs on resource-constrained devices. 

After the rebuttal, the paper receives 1 accept and 3 weak accepts. All reviewers are positive about the paper.
Decision: Accept (poster)