DTP: Delta-Guided Two Stage Pruning for Mamba-based Multimodal Large Language Models
Authors: Seong Yeol Park
Min Jung Kwon
XIANGHUA PIAO
Yeong Hyeon Gu
Venue: ICLR 2026 Poster (2026)
Keywords: Mamba
Multimodal Large Language Models
Token Pruning
Efficiency
Interpretability
Primary area: foundation or frontier models, including LLMs

Abstract: Multimodal large language models built on the Mamba architecture offer efficiency advantages, yet remain hampered by redundant visual tokens that inflate inference cost, with the prefill stage accounting for the majority of total inference time. We introduce Delta-guided Two stage Pruning (DTP), a method that progressively reduces token redundancy through selective pruning at early layer and complete pruning at late layer. Unlike Transformer-oriented pruning methods, our approach derives token importance directly from Mamba’s internal parameters. The statistical distribution of these importance scores, combined with implicit attention patterns, then provides the basis for determining both the pruning layers and the tokens to be removed. Extensive evaluation across diverse benchmarks shows that DTP cuts computation by nearly 50\%, maintains higher task performance than existing pruning methods, and further achieves over a 35\% reduction in prefill latency. Beyond efficiency, our analysis reveals previously underexplored behaviors of visual tokens within Mamba layers, suggesting a principled perspective for designing future pruning techniques in Mamba-based Multimodal Large Language Models.

Published as a conference paper at ICLR 2026
DTP: DELTA-GUIDED Two STAGE PRUNING FOR MAMBA-BASED MULTIMODAL LARGE LANGUAGE MODELS
Seong Yeol Park, Min Jung Kwon, Xianghua Piao, Yeong Hyeon Gu' Sejong University
{aeongyeol, mjkwon, (s yhgu@sejong.ac. ac.krr
ABSTRACT
Multimodal large language models built on the Mamba architecture offer effi- ciency advantages, yet remain hampered by redundant visual tokens that inflate inference cost, with the prefill stage accounting for the majority of total inference time. We introduce Delta-guided Two stage Pruning (DTP). a method that pro- gressively reduces token redundancy through selective pruning early layer and complete pruning at late layer. Unlike Transformer-oriemt approach derives token importance directly from Mamba's 's internal parameters. The statistical distribution of these importance scores, combined with implicit at- tention patterns, then provides the basis for determining both the pruning layers and the tokens to be removed. Extensive evaluation across diverse benchmarks shows that DTP cuts computation by nearly 50%, maintains higher task perfor- mance than existing pruning methods. and further achieves over a 35% reduction in prefill latency. Beyond efficiency, our analysis reveals previously underexplored behaviors of visual tokens within Mamba layers, suggesting principled perspec- tive for designing future pruning techniques Mamba-based Multimodal Large Language Models.
pruning methods, our
INTRODUCTION
Multimodal Large Language Models (MLLMs) capable of jointly understanding and generating across different modalitics such as images and text, thereby handling complex tasks that are difficult
for dingle-modalle models (Liu al. 2023 Dai et al., 2023 Peng et al. 2023 Wu et al.) 2024).
These capabilities have demonstrated outstanding performance in diverse tasks such as visual ques-
tion answering (Guo et al.) 2023 Hu et al. 2024 Fang al. 2025 Wang et al.
2025
Dong al.)
2025) and reasoning segmentation LLai et al. 2024 Ren et al. 2024 Xia el al. 2024).
Most existing MILMSs built upon the Transformer architecture (Vaswani et al. (017)) which has shown strong performance in wide range of multimodal tasks through the self-attention mecha- nism. However, self-attention requires computing interactions between all token pairs, leading to O(n²) time complexity that must be repeatedly incurred at every step when generating new tokens. To mitigate this redundant computation, the KV-Cache technique has been introduced, which ef- fectively accelerates autoregressive generation during inference. Nevertheless, K KV-Cache requires storing key-value pairs at every generation step, which results substantial memory consumption. Furthermore, it does not reduce the computational cost of the prefill stage.
To overcome these structural limitations of Transformers, the recently proposed Mamba architecture (Gu & Dao 2023) leverages State Space Models (SSMs) to recurrently update hidden states, thereby achieving linear-time complexity. Figure | contrasts the inference process between Transformer- based and Mamba-based MILMs. As shown in Figure I (a), Transformers must recompute self- attention with all previous tokens whenever a new token is generated, causing decoding costs to increase linearly with sequence length. In contrast, as illustrated in Figure [ (b). Mamba generates the next token through a single-step hidden state update without revisiting the entire input sequence.
"ooresponding author.

---

Published as a conference paper at ICLR 2026
Transforner
Transformer
Mamba
Marba Sclediie Naokls
Self-Altention
Self- Alte-nittt
Modia
layerrs
layers
layers
loyers
--
++
Prefill Stage
Decode Stage x N
Prefill Stage
Decode Stage X N
(a) Transformer-bart MLLMs
(b) Mamba-based MLLMS
Figure 1: Comparison of inference structures between Transformer-basee and Mamba-based MLLMs.
This structural difference allows Mamba-based MLLMs achieve much lower memory usage and faster decoding compared to Transformer-based models, benefit that has been empirically validated in recent studies (Liu al. 2024 Qiao al. 2024 Zhao al. 2025).
Thus, while Mamba provides clear efficiency advantages in the decoding stage, the majority of in- ference time is still spent in the prefill stage, where all input tokens must be processed initially. The inefficiency of the prefill stage is particularly pronounced in multimodal settings, since the number of visual tokens far exceeds that text tokens, greatly increasing the overall input length. However, many of these visual tokens are redundant or uninformative, and often do not contribute to the final output (Chen et al. 2024). Based on this observation, various pruning methods have been proposed in Transformer based MLLMs to reduce computational cost by removing unnecessary visual tokens (Alvar et al. 2025 Yang et al. 2025] Ye et al. 2025a b Lin et al., 2025). While effective in improv- ing efficiency, these approaches rely attention scores to estimate token importance, making them inherently dependent on the Transformer architecture and not directly applicable to Mamba.
To address this limitation, propose novel visual token pruning framework tailored for Mamba- based MLLMs. The proposed DTP (Delta-guided Two stage Pruning) leverages the input-dependent parameter A, estimate token importance and performs pruning during inference without any training. Pruning is applied at two specific layers, with their positions determined from the observed distribution of token importance and the analysis of Mamba's implicit attention matrices. Based on this rationale, our design effectively removes unnecessary tokens while incurring smaller perfor- mance degradation compared other approaches.
Through extensive experiments on representative Mamba-based MLLMs, Cobra (Zhao et al. 2025 and RoboMamba (Liu et al. 2024), we demonstrate that DTP reduces computational cost by nearly half while maintaining comparatively high performance relative to existing methods. Furthermore, we identify optimal internal parameters in Mamba evaluating token importance, thereby max- imizing the effectiveness of pruning. In addition empirical results, our study provides new insights into the role and significance of visual tokens across layers in Mamba- based MLLMs, offering an effective methodology for visual token pruning.
Our main contributions are summarized as follows:
We propose DTP (Delta (Delta-guided Two stage Pruning), the novel visual token pruning frame- work designed for Mamba-based MILMs. DTP leverages the input-dependent parameter A, to estimate token importance and performs visual token pruning during inference with- out requiring additional training.
Pruning is applied at two specific layers, with their positions determined based on token im- portance distribution and the analysis of Mamba's 's implicit attention matrices. This design enables the effective removal redundant visual tokens while maintaining stable perfor- mance.
2

---

Published as a conference paper at ICLR 2026
Extensive experiments on Cobra and RoboMamba demonstrate that DTP reduces FLOPs by nearly half with less performance degradation compared to existing methods. In addition, we identify the optimal internal parameters for token importance estimation, providing further insights into the role of visual tokens across layers in Mamba-based MIL.Ms.
2 RELATED WORK
Token Reduction in Mamba-based Model. Token reduction has been studied in Transformer- based models as way to alleviate high computational cost and accelerate inference by removing unnecessary input tokens without causing significant performance degradation (Bolya et al., 2022 Kong et al. 2022 Haurum et al. 2023 Wei et al. 2023 Kim et al., 2024 2025). However, since Mamba is based on State Space Models (SSMs) rather than the self attention mechanism central to Transformers, token reduction techniques designed for Transformer architectures cannot be directly applied. Accordingly, recent studies have proposed token reduction methods specifically tailored to the structural characteristics of Mamba. [Zhan et al. (2024) proposed token pruning method for Vision Mamba (ViM) (Zhu et al. (024) and Plain (Yang et al. (2024), introducing a pruning- aware hidden state alignment approach to stabilize the neighborhoods of the remaining tokens and a token importance estimation mechanism specific to Mamba, thereby improving inference speed while minimizing performance degradation. [Shen et al. (2024) proposed Famba- V, cross-layer token fusion method for ViM that identifies and merges similar tokens across layers, improving training efficiency while maintaining a balance with accuracy. Although these works explore diverse token reduction strategies such pruning and fusion in Mamba-based models, they remain limited to unimodal vision tasks, and token reduction in Mamba-based MLLMs has not yet been sufficiently explored.
Mamba-based MILMs. Recent studies have sought to leverage the structural efficiency of Mamba by extending Transformer-basee MLLMs into Mamba-based MLLMs, aiming to achieve faster in- ference speed and improved efficiency in long-sequence processing. Qiao et al. (2024) proposed VL-Mamba, the first multimodal architecture that replaces the Transformer- basoommerr-aaedddd language model with a Mamba language model. It adopts SigLIP (Zhai et al.) (023) as the vision encoder and in- troduces the Vision Selective Scan (VSS) module within a multimodal connector to enhance repre- sentational capacity, demonstrating competitive results across variety of multimodal benchmarks. Liu et al. (2024) introduced RoboManba, which combines a CLIP vision encoder (Radford al.) 2021) with Mamba and adds it with a lightweight policy head to enable SE(3) pose prediction and lision-language-acee action modeling. In addition to its strong performance robotic manipulation tasks, it also exhibits remarkable multimodal reasoning capability. Furthermore, Zhao et al. (2025) presented Cobra, which integrates a pre-trained Mamba language model with visual encoders such as DINOv2 (Oquab et al., 2023) and SigLIP (Zhai et al.) (023). Compared to Transformer-based MLLMs, it achieves both faster inference speed and superior performance.
PRELIMINARIES
3.1
STATE SPACE MODELS AND MAMBA
State Space Models (SSMs) are the core structure of Mamba (Gu & Dao 2023). which transform an input sequence (1) E into an output sequence y(() E R through hidden state h()) E RRN and are defined as:
h'(t) = Ah(t) + Bx((), v() = Ch((),
(I)
where A governs the state transitions, B maps the input into the hidden state, and C projects the hidden state into the output sequence.
In this basic form, SSMs have the limitation of linear time invariance (LTT), in which the same fixed parameters are applied to every time step of the input sequence, making it impossible to selectively process important redundant information. In addition, while this formulation is designed for continuous systems, deep learning models generally operate on discrete systems, and therefore the continuous parameters must be discretized.
To address this issue, Mamba introduces an input-dependent parameter A,- This parameter is com- puted from the input each time step through linear transformation followed by the softplus
3

---

Published as a conference paper at ICLR 2026
function, which enables different state updates each step. By leveraging this input-dependent parameter, Mamba discretizes SSMs and proposes Selective SSMs (S6), which allow inputs to be processed selectively each time step, formulated below:
A. exp(A+A),
B. -A,A) (exp(A.A) 1) A.B
(2)
h1 A,1i-1 Bex Yt C.h.
3.2 ATTENTION MATRICES IN MAMBA
Ali et al. 2024) demonstrated that the selective SSM layer be unfolded into causal kernel that closely resembles the attention matrix in Transformers. This finding provides an interpretation of Mamba's selective SSMs implicitly incorporating attention-like behavior, even without an explicit attention mechanism. Specifically, the selective SSM formulation can be expanded into a convolutional form. yielding kernel that functions as implicit attention weights. From Equation 2 assuming the initial state ho O, the output Yr can be written as:
Σ j=1
Ce
As
(3)
k=j+1
By defining
Kr,- = C.
II Amj+1
A
B₁,
3t
Σ KiyTj
(4)
that K, represents the coefficient through which the input xy is linearly transformed and incorporated into the output at time step t.
By arranging all coefficients, obtain the following lower-triangular implicit attention matrix:
Cr
C2A2B1
C2B2
0
0
B3A
C2ABB2
CB Bs
K
(5)
Cr CI B,
3 B2 Cr Cz AB)Ba
Cr
This matrix K RLXL, where L denotes the sequence length, be interpreted in way similar to the attention matrix of Transformers, where each row corresponds to the output at time t and each column shows how input zy propagates its influence subsequent outputs.
METHOD
In this section, we propose DTP -Delta-guided Two stage Pruning). pruming strategy for Mamba- based MILMs. As shown in Figure 2 DTP follows a two stage pruning strategy, performing se- lective pruning in the early layer and complete pruning the late layer. To determine the specific layers where pruning is applied and conduct selective pruning in the early layer, leverage A,, the key parameter that enables the selectivity of Mamba, to evaluate the importance visual tokens in each Mamba block.
4.1 TOKEN IMPORTANCE FROM A:
As described in Section 3.1 LTI SSMs apply the same parameters to all input sequences and there- fore lack the selectivity to distinguish the relative importance of tokens. Accordingly, Mamba (Gu & Dao 2023) introduces an input-dependent parameter A, derived from the input sequence, and A serves as the key mechanism that enables selectivity by discretizing continuous SSMs and con- trolling the state transition matrix A, and the input mapping matrix B,. Specifically, A directly modulates both A, and B, thereby controlling the discretized state transition dynamics and deter- mining the extent which each token contributes subsequent hidden states. Tokens with larger
4

---

Published as a conference paper at ICLR 2026
Vision Encoder
Mamba Block 0
Mumbu Block
Early Layer Pruning
Mambu Block 45
Late Layer Pruning
Mumbu Block 63
x
x
Is this cat playing with string?
Text Encoder
Figure 2: Overview of the proposed DTP(DDeta-guins saiittd--gieddd Two stage Pruning) method.
As values induce stronger state updates, whereas those with smaller values yield only minor transi- tions, effectively acting as less influential inputs. Building this role, directly leverage A, to obtain token importance scores for visual token pruning in Mamba-based MLLMs. The importance score of the j-th token, denoted Sy, is defined follows:
8j =
Σ d=1
A
j.d:
where D is the dimension size. This allows token importance to be directly estimated from A, during inference, without requiring any additional training or modification. Based on the token importance scores derived from Ac, we adopt top-k selection strategy retain the most informative visual tokens. At the pruning layers, visual tokens are ranked according to their importance scores, and only the top & tokens are preserved while the others are discarded. This reduces redundant visual information and enables computation to focus on the more critical tokens during reasoning, For more comprehensive analysis, we further compared the proposed token importance parameter A, with other internal parameters such as yr, B., and Ct. and the results confirmed that A, is the most suitable criterion for evaluating the importance of tokens in pruning.
4.2 PRUNING STRATEGY
Drawing prior studies of token reduction in Transformer-based MILMSs (Chen al. 2024 Lin
et al. 2025 Ye et al., 2025a), which observed that visual tokens contain redundant information and exhibit minimal attention in deeper layers, we apply pruning specifically visual tokens. Token pruning improves computational efficiency and latency by removing redundant tokens, and in deeply stacked MILMM architectures, the process of identifying the optimal pruning point just as important as determining the parameter used to measure token importance.
Based on the token-impont loken-importance parameter A, introduced Section 4.1 we conduct preliminary analysis to determine which layers are suitable candidates for pruning. To quantify how the impor- tance of tokens varies within each layer, we compute the layer-wise standard deviation of Ay-guided token importance, defined as:
1 N
Side
Σ
80)2
where 8i.8 denotes the importance value of the j-th token at layer e, 84 the mean importance at that layer, and N is the number of tokens.
For this preliminary analysis, we randomly sample examples from each benchmark dataset used in our performance evaluation and compute the standard deviation for all 64 layers in Cobra and RoboMamba. The full results are presented in Appendix E Across all datasets, both models exhibit highly consistent trends, demonstrating that layer-wise standard deviation provides an efficient and reliable criterion for identifying promising pruning layers without the need for additional heuristics or layer-specific thresholds.
5

---

Published as a conference paper at ICLR 2026
VQAv2
Staddrd deviation
0.030
0.025
0.015
0.010
Clift
Peak
Farly Layer
Late Layer
1000
10
15
20
30
35
40
45
50
60
Layer Order
Figure 3: Layer-wise standard deviation of token-importance values for the Mamba-based MLLM Cobra Zhao et al. 2025) on VQAv2. The early layers contain a clear local peak, while the late layers exhibit cliff
Layer 5, Std 0,009
Layer 15, Std - 0.004
Layer 44, Std 0.001
Layer 45, Std 0.014
Token Impertunce
0.09
Tolen importunee
6.00
600 aoe
23 ve
Token Index
Token index
Tokon index
Figure 4: Visualization of token importance distributions at different layers of Mamba-based MLLM, Cobra (Zhao et al. 2025). Red and blue points denote tokens within and outside the top 50% by importance, respectively.
In addition to the standard deviation analysis, further examine the distribution of token- importance values derived from the parameter described in Section 4.1 Moreover, inspired by the implicit attention pattern of Mamba layers discussed in Section 3.2 incorporate an additional analysis to intuitively capture token-token interaction behaviors.
Selective pruning at the early layer. Pruning very early layers has the advantage of achieving high computational efficiency, but it carries a significant risk discarding tokens that could later serve meaningful information in deeper layers. In addition, at such early layers, the distinction of token importance is not yet clear, making reliable selection difficult. To mitigate these issues, we focus on the first 25% of layers, guided by the preliminary analysis presented earlier, and examine how token importance patterns evolve within this region.
As shown Figure 3 the standard deviation of token importance exhibits noticeable local peak in this early interval, where the standard deviation is relatively low compared to neighboring lay- ers. Figure 4 further illustrates that layers near this peak contain a cluster of tokens with similar importance values, along with small subset of tokens that hold comparatively higher importance. When examining the implicit attention pattern in Figure 5 we observe that in layer 5, which shows relatively high standard deviation before the local peak appears, there is clear token-token in- teraction pattern in the trianuular region. contrast, in layer 15 within the local-peak region where the standard deviation becomes low, gins to form in the lower-triangular area. Considering all the numerical and quantitative analyses that demonstrate consistent patterns, find that layers with low standard deviation values exhibit implicit attention patterns with strong token-token interaction and represent points where important tokens can be separated.
see that token-token interaction pattern be-
Leveraging these observations, we select the optimal early pruning layer Kearly as the layer within the first 25% of the model depth in a model with L layers that exhibits the minimum standard deviation of token importance values:
Ricarly argmin Stde,
where 0 S 0.25L.
(8)

---

Published as a conference paper at ICLR 2026
-3
-44
Layer 5, Std = 0.009
Layer 15, Std = 0.004 Layer 44, Std = 0.001 Layer 45, Std 0.014
Figure 5: Visualization of implicit attention patterns across different layers in Mamba-based MLLM, Cobra (Zhao al. 2025). Appendix C provides an overview of implicit attention patterns across all layers.
the standard deviation of token impor- tance values increases sharply beyond roughly 70% of the model depth, producing cliff-like pattern that marks the onset of a high standard deviation regime. This observation is further supported by the token importance distributions in Figure which become increasingly diffuse and irregular in late layers, lacking the coherent clustering observed earlier. When examining the implicit attention pattern Figure we observe that the cliff point where the difference standard deviation be- tween adjacent layers is the largest, layer 44, which exhibits very low standard deviation, shows clear moken-token interaction pattern the lower-triangular region. In contrast, the next layer, layer 45, displays sharp increase in standard deviation, and the token-token interaction pattern in the lower-triangular area begins to weaken. Taken together, these signals indicate that once the model enters this high standard deviation regime, no longer provide meaningful cues for selecting salient tokens. In other words, the implicit attention structure collapses to the point that partial pruning becomes unreliable. Consequently, complete pruning more effective than selective pruning in this stage of the model, as visual tokens no longer contribute stable or discriminative information to the downstream computation.
Complete Pruning at Late Layers. As shown in Figure B
Leveraging these observations, we select the optimal late pruning layer hame as the layer within the final 30% of the model depth model with L layers that exhibits the maximum inter-layer change in the standard deviation of token importance values:
haate argmax [Stde+1 Stde |+
where 0.7L <L<
5 EXPERIMENT
5.1
EXPERIMENTAL SETUP
representative Mamba-based MILMs: Cobra (Zhao al., 2025) and RoboMamba (Liu et al. 2024). These models are representative Mamba-based MLLMs and are used as baselines since their pretrained weights are publicly available and they provide strong performance across multimodal tasks. For Cobra, we apply our method to six benchmarks that cover diverse aspects of multimodal reasoning, including GQA (Hudson & Manning 2019). VQAv2 (Goyal et al.) (017), TextVQA (Singh et al. 2019). POPE (Li et al.) (023), VSR (Liu al. 2023), and VizWiz (Gurari al.) 2018). For RoboMamma, we evaluate on five benchmarks, including OK OKQQA (Marino et al. 2019), GQA (Hudson & Manning 2019), VQAv2 (Goyal et al. 2017. POPE (Li 2023). and VizWiz (Gurari et al. 2018).
We evaluate our pruning strategy
The baselines considered in this study are three representative token pruning methods originally proposed for Transformer-based MLLMs: (I) FastV (Chen et al. 2024) prunes visual tokens at designated layer & based on attention scores, with the pruning ratio controlled by r. Since Mamba- based MLLMs do not expose explicit attention scores, we replace them with our proposed A based token importance measure for benchmarking. (2) VTW (Lin al.) 2025) determines the optimal layer & for withdrawing visual tokens by sampling a small subset of the dataset, comparing the original output with the output after token withdrawal, and selecting the earliest layer where the KI.

---

Published as a conference paper at ICLR 2026
divergence between the two logits falls below predefined threshold. As this method provides general criterion that is independent of model architecture, it also serves as an appropriate baseline for our study. (3) DART (Wen et al. 2025) defines small subset of visual tokens pivot tokens and removes the remaining tokens based on their similarity to these pivots, with the reduction controlled by the keep ratio All experiments implemented in PyTTorch and executed single NVIDIA RTX 5090 GPU to ensure fair comparisons.
Table I: Comparison of pruning methods on the Cobra model. The table presents FLOPs, FLOPs ratio, evaluation scores across benchmarks, and the average score.
Method
FLOPS
FLOPS ratio
GQA
VQAv2
TextQQA
POPE
VSR
VizWiz
Avg
Baseline (Cobra)
2.01
100%
62.3
77.8
58.2
88.4
58.4
49.7
63.8
FastV (#=2,, r=0.7)
1.45
72%
62.1
77.4
56,9
87.7
58.0
49.8
65.3
(-0.3)
(-0.4)
(-1.5)
(-0.7)
(-0.4)
(-0.1)
(-0.5)
62.1
77.7
58.2
88.3
58.5
49.5
65.7
VTW (k=45)
1.43
71%
(-0.2)
(-0.1)
(+0.0)
(-0.3)
(+0.1)
(-0.2)
(-0.1)
62.0
77.1
57.0
87.4
58.2
49.7
65.2
DART (r=00.6)
1.38
69%
(-0.3)
(-0.7)
(-1.3)
(-1.0)
(-0.2)
(-0.0)
(-0.6)
62.0
58.9
Ours (r=0.9)
1.35
67%
(-0.3)
(40.5)
(+0.0)
61.7
76.8
55.0
87.4
57.3
50.1
64.7
FastV (f=22, r=0.5)
1.06
53%
(-0.6)
[-1.0)
(-5.2)
(-1.0)
(-1.1)
(-0.4)
(-1.1)
47.1
54.1
42.6
74.1
57.9
48,5
54,0
(k=32)
1.04
52%
(-18.2)
(-28.7)
(-15.6)
(-14.3)
(-0.5)
(-1.2)
(-11.8)
61.2
76.1
55.1
86.3
57.7
49.5
64.3
DART (r=0..44)
0.96
48%
(-1.1)
(-1.7)
(-9.1)
(-2.3)
(-0.7)
(-0.2)
(-1.)
61.4
77.1
56.1
87.3
57.9
49.6
64.9
Ours (r=0.5)
0.97
48%
(-0.9)
Table Comparison of pruning methods on the RoboMamba model. The table presents FLOPs,
FLOPs ratio, evaluation scores across five benchmarks, and the average score.
Method
FLOPs
FLOPs ratio
OKVQA
GQA
VQAv2
POPE
VizWiz
Avg
Baseline (RohoMamba)
0.70
100%
64.4
56.4
74.9
85.2
54.2
67.0
64.1
56.1
74.4
85.3
53.0
66.6
FastV r-0.)
0,50
71%
(-0.3)
(-0.3)
(-0.5)
(+0.3)
(-1.))
64.0
55.9
74.7
85.3
53.3
66.6
VTW (k=45)
0.50
71%
(-0.4)
(-0.6)
(-0.2)
(40.33
(-0.9)
(-0.4)
64,2
55.4
73.9
84.2
66.1
DART (r=0.63)
0.47
67%
64.2
56.1
74.7
85.2
53.2
66.7
0.9)
(-0.2)
(-0.3)
(-0.2)
(0.0)
(-1.0)
(-0.3)
63.4
55.1
73.4
84.2
52.1
65.6
FastV (k=2, r=0.5)
0.37
53%
(-1.0)
(-1.3)
(-1.5)
(-1.0)
(-2.3)
(-1.4)
40.0
44.7
55.2
82.1
45,2
53.4
VTW (l=32)
0.36
51%
(-24.4)
(-11.7)
(-19.7)
(-3.3)
(-9.0)
(-13.6)
64.1
54.0
72.8
83.4
52.6
65.4
DART (r=0.47)
0.36
51%
(-0.5)
(-2.4)
(-2.1)
(-1.8)
(-1.6)
(-1.0)
63.8
$4.9
73.6
84.4
52.8
65.9
Ours (r=0.5)
0.34
49%
(-0.6)
(-1.6)
(-1.3)
(-0.8)
(-1.4)
(-1.1)
5.2
MAIN RESULTS
We evaluate all pruning methods under two comparable FLOPs reduction regimes, a reduction of about 70% relative to the baseline and a reduction of about 50%. Table [abesentsssntts the results on the Cobra model. For the 70% FLOPs level, FastV is configured with k = 2 and r 0.7, achieving a FLOPs ratio of 72%. This configuration yields noticeable accuracy drops, including reduction of 1.3 TexIVQA and reduction of 0.7 on POPE. VTW is applied with its KL divergence based withdrawal criterion, which selects &o 45 and achieves a similar FLOPs ratio of 71% with only 0.1 decrease in average accuracy. This indicates that the optimal & determined by the withdrawal criterion of VTW indirectly supports the validity of our complete pruning at the late layer strategy described in Section 4.2 DART follows its original design, where it prunes at 2, retains eight
8

---

Published as a conference paper at ICLR 2026
pivot tokens, and uses keep ratio of 0.66. Under this setting, which results in approximately 69% FLOPs, DART exhibits clear degradation across several benchmarks, including a reduction of 1.2 on Tex,VQA, reduction of 1.0 POPE, and reduction of 0.7 on VQAv2. Our method deter- mines the pruning points using the preliminary analysis in Section 4.2, which consistently identified the early and late pruning layers Kearly 15 and Mlate = 45 across all benchmark datasets. With r == 0.9 applied at the early layer, DTP attains a FLOPs ratio of 67% while exactly matching the baseline's 's average score of 65.8, demonstrating substantially lower accuracy degradation than ex- isting methods. When FLOPs were reduced to about half of the baseline, FastV is configured with == and r 0.5 for comparison with existing methods. VTW is method for finding the optimal k. we fixed k = 32 to enable comparison under this specific FLOPs condition. In this setting, F showed significant performance degradation, including 3.2 point drop on TextVQA and 1.1 point drop on VSR, while VTW exhibited large performance losses across all datasets. DART prunes at = with a keep ratio of r = 0.47, retaining eight pivot tokens. Under this configuration, which corresponds roughly 48% FLOPs, DART also suffers noticcable accuracy degradation across benchmarks. Our method applies a keep ratio of r = 0.5 at the early pruning point Rearty = 15, followed by complete pruning at Ajsse = 45. Our proposed method showed only 0.9 point decrease on average compared to the baseline, demonstrating smaller performance degradation than the other methods.
Table presents the results on RoboMamba. Our proposed method shows only a 0.3 point perfor- mance drop even when FLOPs are reduced 66%. When FLOPs are further reduced around 50%, VTW exhibits severe performance degradation, similar to the case of Cobra. Both FastV, DART and our method experience some decrease in performance, but our method shows relatively smaller drops on OKVQA and VizWiz, achieving an average score of 65.9 with the least overall degrada- tion. This effect arises because Robo.amma employs the CLIP vision encoder (Radford al. 202T) with only 256 image tokens. Compared to Cobra, which processes 729 tokens, the smaller number of tokens makes RoboMamba more susceptible information loss under aggressive pruning.
5.3 EFFICIENCY ANALYSIS
To verify that the FLOPs reduction achieved by DTP leads to practical runtime improvements, we additionally measure wall-clock inference latency on the Cobra model using the POPE benchmark. The evaluation includes mean prefill latency, mean decode latency, total inference time, and GPU memory consumption. As shown in Table 3 DTP reduces the mean prefill latency from 98.04 ms to 61.54 ms, corresponding to a 37% improvement, and lowers the total latency from 16m 05s to 10m 35s and GPU memory usage also decreases from 8.8 GB 8.3 GB. Compared to existing pruning baselines such as FastV, VTW, and DART, DTP provides competitive or superior latency while maintaining strong accuracy and larger FLOPs reduction. These results confirm that the efficiency gains of DTP extend beyond computation savings and translate into substantial real-world speedups.
Table 3: Inference latency and gpu memory usage on the Cobra model using the POPE benchmark. The table reports mean prefill/decode latency, total inference time, and peak GPU memory consumption for different pruning methods.
Method
FLOPs
Prefill Mean (ms)
Decode Mean (ms)
Total Latency
GPU Memory
Cobra
2.01
98.04
5.04
16m 05s
8.8 CB
FastV (
1.06
60.64
4.94
10m 24s
8.5 GB
VTW (k=32)
1.04
67.31
4.99
11m 25s
8.3 GB
DART (r=0.44)
0.96
76.45
4.95
12m 44s
8.5 GB
DTP (r=0.5)
0.97
61.54
5.02
10m 35s
8.3 GB
5.4 ABLATION STUDY
All ablation studies in this section are conducted under the setting where FLOPs are reduced to approximately 50% of the baseline.
Identifying effective internal parameters for token importance. Table presents the ablation study results when different internal parameters of the selective SSM used to compute token

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study internal parameters of Mamba for token importance estimation.
Table 5: Ablation study token selection methods for Cobra and RoboMamba
Cobra
Robo
Cobra
RobuNfamka
Parameter
GQA
TeuVQA
Vizwiz
GQA
POPE
OKVQA
Methed
GQA
TesVVQA
Vizwiz
GQA
POPE
OKVQA
60.2
61.1
47.4
48.0
49.2
49.0
54.5
54.2
82.9
83.8
63.8
62.6
Top-k tokene
Random
7,00
61.4
53.7
10.3
30.7
49.4
21.2
54.1
56.8
83.6
63.2
46.5
44.9
Top- (visual only)
61.4
56.1
49.6
54.9
54.4
63.8
Ax
614
56.1
49.6
54.9
84.4
6388
Table 6: Ablation study on the effect of complete pruning at the late layer.
Cobra
RoboMamba
Condition
FLOPs
FLOPs Ratio
GQA
TextVQA
Vizwiz
FLOPs
FLOPs Ratio
GQA
POPE
OKVQA
w/o complete proning
1.26
100'6 87%
61.5
56.1
49.6
0.44
100%
55.1
84.4
63.9
complete pruning
0.97
61.4
56.1
0.34
77%
84.4
63.8
importance. Comparing the output term Yt, the input coefficient B3, the state coefficient Ct, and the temporal delta term A,, was found that A, provides the most stable and effective signal in both Cobra and RoboMamba. The output term yi shows generally competitive performance but falls short of A, on TextVQA, while B, and C, exhibit relatively lower performance, particularly on TextVQA for Cobra and POPE for RoboMamba. In contrast, A, achieves the highest comparable scores across all datasets in the ablation study, demonstrating that it serves as the most reliable criterion for distinguishing salient from redundant tokens. These results justify our choice of adopting A, as the default measure for token importance.
Exploring strategies for token selection in pruning. comppress three strategies for selecting pruning candidates. The first baseline simply selects tokens at random. The second strategy applies Top-k policy but also allows text tokens to be pruned. This strategy leads to catastrophic degradation across all benchmarks, with Cobra and RoboMamba both showing dramatic performance drops that make the results far below usable levels. These results highlight that text tokens are essential for reasoning and must not be removed. The third approach applics Top-k only to visual tokens. This strategy reduces redundancy while maintaining overall performance stably, showing that visual tokens be pruned safely without affecting model performance. These results support the validity of a ranked Top-k pruning policy that targets only visual tokens.
Effect of complete pruning at the late layer. Table 6 reports the ablation results of complete pruning at the late layer, while keeping the selective pruning in the early layer fixed at pruning ratio of 0.5. For Cobra, enabling complete pruning reduces FL.OPs from 1.26 to 0.97, which corresponds to a 23% reduction, while accuracy remains nearly identical across GQA, TextVQA, and Vizwiz. Likewise, RoboMamba achieves a reduction from 0.44 0.34, amounting to 23% decrease, with no meaningful accuracy drop on GQA, POPE, and OKVQA. These results emphasize that complete pruning the late layer offers significant computational savings without sacrificing performance, validating it as an effective strategy for improving efficiency.
CONCLUSION
In this paper, we introduced Delta-guided Two stage Pruning (DTP), a novel framework for to- ken pruning Mamba-based MLLMs. Token importance derived from the internal parameter A, in Mamba's 's selective SSM, and the statistical distribution of these scores is analyzed together with implicit attention patterns to determine where pruning should occur and which tokens should be removed. Extensive experiments revealed that, under the same FLOPs budget, DTP preserves task performance more effectively than alternative pruning approaches. Ablation studies further demonstrated that A, provides the most reliable criterion for estimating token importance, and that selecting the top-k visual tokens based on these scores is a reasonable pruning strategy. Moreover, confirmed that applying complete pruning at late layers maintains performance equivalent to taining tokens at those layers. Taken together, these findings establish DTP as an effective method for pruning visual tokens in Mamba-based M...Ms.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea govermmen(MMSII) www.RS22222-254415555 Develop- ment of human foundation model for human-centric universal artificial intelligence and training of personnel)
REFERENCES
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint urXiv:2403.01590 2024.
Saced Ranjbar Alvar, Gursimran Singh, Mohammad Akbari, and Yong Zhang. Divprune: Diversity- based visual token pruning for large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9392 22229940111 2025.
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022.
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- language models. In European Conference on Computer Vision, pp. 19-35. Springer. 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250-4925 2023.
Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9062-9072, 2025.
Wenlong Fang, Qiaofeng Wu, Jing Chen, and Yun Xue. guided mllm reasoning: Enhancing mllm with knowledge and visual notes for visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19597-19607, 2025.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6911 2017.
Albert Gu and Tri Dao. Mamba: Linear-tin timear-immeeee sequence modeling with selective state spaces. arXiv preprint ar arXiv:2312.00752. 2023.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10867-10877, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3608-3617, 2018.
Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which kens to use? investigating token reduction in vision transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 773-783, 2023.
Wenbo Hu, Yifan Xu, Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: A simple multimodal Ilm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2256-2266 2024.

---

Published as a conference paper at ICLR 2026
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer viston and pattern recognition, pp. 6700-6709, 2019.
Kwonyoung Kim, Jungin Park, Jin Kim, Hyeongjun Kwon, and Kwanghoon Sohn. Faster parameter- efficient tuning with token redundancy reduction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30189-30199 2025.
Minchul Kim, Shangqian Gao, Yen- Chang Hsu, Yilin Shen, and Hongxia Jin. Token fusion: Bridg- ing the gap between token pruning and token merging. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1383-1392 2024.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In European conference on computer vision, pp. 620-640. Springer, 2022.
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Rea- soning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9579 9579,9589, 2024.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
Zhihang Lin, Mingbao Lin, Luxi Lin, and Rongrong Ji. Boosting multimodal large language models with visual tokens withdrawal for rapid inference. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 5342, 2025.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-651, 2023.
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Pengju An, Xiaoqi Li, Kaichen Zhou, Senqiao Yang, Renrui Zhang. Yandong Guo, and Shanghang Zhang. Robomamba: Efficient vision-language- action model for robotic reasoning and manipulation. Advances in Neural Information Processing Systems, 37:40085 401--4011,, 10, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193. 2023.
Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint 2xXiv:2306.18244, 2023.
Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao Mingzhen Sun, Qi Wu, and Jing Liu. Vl-mamba: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403.1 13600, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. (6374-26383, 2024.
Hui Shen, Zhongwei Wan, Xin Wang, and Mi Zhang. Famba-v: Fast vision mamba with cross-layer token fusion. In European Conference Computer Vision, pp. 268-278. Springer, 2024.
12

---

Published as a conference paper at ICLR 2026
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Zining Wang, Tongkun Guan, Pei Fu, Chen Duan, Qianyi Jiang, Zhentao Guo, Shan Guo, Junfeng Luo, Wei Shen, and Xiaokang Yang. Marten: Visual question answering with mask generation for multi-modal document understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 14660-1477, 2025.
Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeez- ing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2092-2101, 2023.
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang. Stop looking for important tokens in multimodal language models: Duplica- tion matters more. arXiv preprint arXiv:2502.1 1494, 2025.
Jiannan Wu, Muyan Zhong. Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, al. Visionllm v2: An end-to-end generalist multimodal large language model for hundreds of vision-language tasks. Advances in Neural Information Processing Systems, 37:69925-69975, 2024.
Zhuofan Xia, Dongchen Han, Yizeng Han, Xuran Pan, Shiji Song, and Gao Huang, Gsva: Gen- eralized segmentation via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3858-3869, 2024.
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Chendi Li, Jinghua Yan, Yu Bai, Ponnuswamy Sadayappan, Xia Hu, et al. Topv: Compatible token pruning with inference time optimization for fast and low-memory multimodal vision language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19803-19813, 2025.
Chenhongyi Yang, Zehui Chen, Miguel Espinosa, Linus Ericsson, Zhenyu Wang, Jiaming Liu, and Elliot J Crowley. Plainmamba: Improving non-hierarchical mamba in visual recognition. arXiv preprint Xiv::2403.17699 2024.
Weihao Ye, Qiong Wu, Wenhao Lin, and Yiyi Zhou. Fit and prune: Fast and wrining-free visual token pruning for multi-modal large language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 22128-22136, 2025a.
Xubing Ye, Yukang Gan, Yixiao Ge, Xiao-Ping Zhang, and Yansong Tang. Atp-llava: Adaptive token pruning for large vision language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24972-24982, 2025b.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov. and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 11975-119988 2023.
Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis loannidis, Wei Niu, Pu Zhao, et al. Exploring token pruning in vision state space models. Advances in Neural Information Processing Systems, 11550095250971.. 2024.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10421-10429, 2025.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.004 00911,, 2024.
13

---

Published as a conference paper at ICLR 2026
A TOKEN IMPORTANCE IN ROBOMAMBA
We provide further analyses on token importance using another Mamba-based MLLM, Robomamba. The figures show similar trend to those observed in Cobra, supporting the generality our pruning strategy.
Layer 5, Std 0.011
Layer 15, Std 0.004
Layer 44, 0.002
Layer 45, 0.014
8.34
046 c.e
Tokee Index
Tokan trdex
Token mndex
Takee Indox
Figure 6: Visualization of token importance distributions different layers of Robomamba (Liu et al. 2024). Red and blue points denote tokens within and outside the top 50% by importance, respectively.
B FULL LAYER-WISE STANDARD DEVIATION ANALYSIS
This section provides the complete layer-wise standard deviation plots for all datasets used in our evaluation. We sample 50 examples from each benchmark and compute the standard deviation of coken-importance values across all 64 layers for both Cobra and RoboMamba. As shown in Figures 7 and & all datasets exhibit highly consistent trends. These results reinforce our findings in Section 4.2, demonstrating that wiserwwise standard deviation serves as reliable and architecture- agnostic indicator for identifying carly and late pruning layers
Figure 7: Layer-wint wise standard deviation of token importance values for the Cobra (Zhao et al. 2025 across all benchmark datasets.
14

---

Published as a conference paper at ICLR 2026
CQA
Figure 8: Layer-wise standard deviation of token importance values for the RoboMamba (Liu et al. 2024 across all benchmark datasets.
15

---

Published as a conference paper at ICLR 2026
C
FULL IMPLICIT ATTENTION PATTERNS
Laperr
Layer:3
Leswr
Layer
Layer:44
Layer 5
Layer
Layer?
Layur
Laper
Layer
Lupe
Layer
Laper
Laye
Laysr
Layer 24
Layer
Layer 24
Layer 27
Laser 20
Leyer 20
Laye-33
Layer 31
Layer
Layer
Layer
Lager
Layer
Laper
Layer
Layer
Layer 40
Layer
Layer 50
Layer St
Layer 52
Layer 33
Laye-54
Layer 55
Figure 9: Full implicit attention patterns of each layer of Cobra.
16

---

Published as a conference paper at ICLR 2026
Layer-
Layer1
Laver
Lager
Layer-4
Layer 5
Laser
Layer?
Layur
Laper
Layer
Layer
Layer
Layer
Layer
Layer 24
1ayer
Layer22
Layer 27
Layer 28
Layer
Laye-33
Layer 31
Layer
Layer
Layer
Layer
Layer
Layer
Layer48
Layer
Layer 40
Layer
Layer 50
Layer St
Layer 52
Layer 13
Laye-St
Layer 55
Figure 10: Full implicit attention patterns of each layer of RoboMamba.
17

---

Published as a conference paper at ICLR 2026
D PERFORMANCE STABILITY ACROSS VARYING KEEP RATIOS
To examine how DTP behaves under different pruning strengths, we visualize the relationship be- tween the keep ratio and the average score computed over all evaluation benchmarks. As shown in Figure [T both Cobra and RoboMamba display smooth and monotonic performance degradation as decreases from 1.0 0.3, without any abrupt drops or signs of instabilitty This confirms that the method remains robust even when large portion of visual tokens is removed.
Effect of Keep Ratio on Performance
67.0
Cobra
RaboMariba
66.5
Average Score
66.0
65.5
64.5
64.0
1.0
0.9
0.8
0.7
0.6
0.5
0.4
0.3
Keep Ratio (r)
Figure 11: Performance trends across keep ratios (r).
18


Reviews:
[{"review_id": "jQDa8exdnk", "reviewer": "Reviewer_K7EU", "summary": "This paper introduces Delta-guided Two-stage Pruning (DTP), a training-free framework for visual token pruning in Mamba-based multimodal large language models.", "questions": "How were the 15th and 45th layers chosen for selective and complete pruning? Were these empirically optimal for all models or do they depend on model depth or dataset characteristics?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[1] Adaptation of pruning for Mamba’s state-space mechanism, clearly distinct from Transformer-specific attention-based approaches.   \n[2] Two-stage pruning strategy is empirically well-motivated by variance and implicit-attention statistics.   \n[3] Extensive ablations validating design choices and demonstrating robustness.   \n[4] The method performs pruning during inference without any retraining or fine-tuning, which makes it practically deployable.", "weaknesses": "[1] The layer selection heuristic (15th & 45th) is empirical; a more formal justification or adaptive strategy could strengthen generality.   \n[2] The analysis depth of implicit attention remains qualitative; quantitative correlation with pruning behavior would improve rigor.   \n[3] While FLOPs reduction is well-documented, real-world inference latency (wall-clock time) is not reported.   \n[4] Limited baselines are included. There are many works that should be considered for comparison, e.g. PyramidKV [a], VL-cache [b], etc.  \n[5] Novelty is limited, where the methods/motivation are borrowed from the transformer-based research work. \n\n\n[a] Cai, Zefan, et al. \"Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling.\" arXiv preprint arXiv:2406.02069 (2024).   \n[b]   Tu, Dezhan, et al. \"VL-cache: Sparsity and modality-aware KV cache compression for vision-language model inference acceleration.\" arXiv preprint arXiv:2410.23317 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G73kaGLE5h", "reviewer": "Reviewer_BGBy", "summary": "This paper proposes Delta-guided Two Stage Pruning (DTP), a pruning framework designed for Mamba-based multimodal large language models (MLLMs). The method estimates token importance using the Mamba-specific internal parameter ∆t, enabling selective pruning in early layers and complete pruning in late layers. Experiments on Cobra and RoboMamba demonstrate that DTP can reduce FLOPs by nearly 50% with minimal accuracy degradation. The study further analyzes implicit attention patterns in Mamba, providing insight into token dynamics across layers.", "questions": "- Can ∆t be efficiently extracted in real inference pipelines without additional latency?\n\n- How sensitive is DTP to the choice of pruning layers (15th and 45th)? Would adaptive layer selection improve performance?\n\n- Could the ∆t-based importance be combined with other statistics (e.g., gradient-based) to further enhance robustness?\n\n- How does DTP perform under extremely high pruning ratios (>60%)?\n\n- Are the observations about implicit attention patterns consistent across all Mamba-based architectures?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Proposes a new pruning paradigm specifically designed for Mamba-based models rather than Transformer-based ones.\n- Experiments are comprehensive, covering both Cobra and RoboMamba across multiple benchmarks.\n- The two-stage pruning strategy is intuitively reasonable and empirically validated.\n- Provides interesting analysis of implicit attention patterns in Mamba, offering new perspectives on token behavior.", "weaknesses": "- Limited theoretical justification for ∆t as a universal token importance indicator; the claim is mostly empirical.\n- The computational overhead of computing ∆t during inference is not discussed; clarity on latency gain beyond FLOPs would be beneficial.\n- Comparison with non-pruning efficiency techniques (e.g., token merging, KV cache optimization) is missing.\n- Some figures and analysis (e.g., implicit attention) could be better interpreted to help readers grasp the practical implications.\n\nIf authors address my concerns, I will consider raising my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzaSqrY8Dt", "reviewer": "Reviewer_oGU1", "summary": "This paper introduces DTP (Delta-guided Two stage Pruning), a novel and effective token pruning framework designed for Mamba-based Multimodal Large Language Models (MLLMs) to reduce the high computational cost of the prefill stage, which is dominated by a large number of visual tokens. The core idea is to leverage an internal, input-dependent parameter of the Mamba architecture, ∆t, to estimate the importance of each visual token without requiring any retraining. The proposed DTP method employs a two-stage strategy, beginning with selective pruning at an early layer (the 15th), where a portion of the least important visual tokens are discarded. This is followed by complete pruning at a late layer (the 45th), where all remaining visual tokens are removed, a decision justified by the observation that their contributions become negligible in deeper layers as implicit attention patterns diminish. Extensive experiments on two Mamba-based MLLMs (Cobra and RoboMamba) demonstrate that DTP can reduce computation (FLOPs) by nearly 50% while incurring minimal performance degradation, significantly outperforming adapted Transformer-based pruning methods.", "questions": "- Regarding the choice of the 15th and 45th layers: Could you elaborate on the generality of this finding? If a researcher were to apply DTP to a new Mamba-based MLLM with a different number of layers, should they re-run the standard deviation analysis to find the new optimal \"early\" and \"late\" layers? Is there a rule of thumb, e.g., pruning at \\~25% and \\~70% of the model's depth?\n- The paper's efficiency claims are based on FLOPs reduction. Have you conducted any experiments to measure the actual wall-clock inference speedup (e.g., in terms of tokens/second or total latency)? This would be a valuable addition to confirm the practical benefits of DTP.\n- How sensitive is the DTP framework to the choice of the early-stage keep ratio r? Could you provide a brief analysis or a curve illustrating the performance-computation trade-off as r is varied continuously between, for example, 0.5 and 1.0?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a relevant and important problem: the inference inefficiency of Mamba-based MLLMs. It is intrinsically tied to the Mamba architecture by using the ∆t parameter for importance scoring, a concept not applicable to Transformers.\n- The design choices, particularly the selection of the 15th and 45th layers for pruning, are not made ad-hoc. They are convincingly supported by a careful analysis of the model's internal state, which adds a layer of interpretability and principle to the method.\n- The method achieves a remarkable balance between computational reduction and performance preservation. A nearly 50% reduction in FLOPs with an average performance drop of less than 1 point (on the Cobra model) is a very strong result. The comprehensive ablation studies further solidify the paper's claims.", "weaknesses": "- The specific layers for pruning (15th and 45th) were empirically identified for the Cobra and RoboMamba models, which appear to have around 64 layers. It is unclear how these specific layer indices would generalize to Mamba-based models of different depths (e.g., a 48-layer or 96-layer model). While the methodology for finding these layers (analyzing the standard deviation of ∆t) seems general, the paper could be strengthened by framing it as a general \"recipe\" and discussing how it would apply to other architectures.\n- The evaluation relies exclusively on FLOPs as a metric for computational cost. While FLOPs are a good hardware-agnostic proxy, they do not always translate linearly to wall-clock speedup due to factors like memory access patterns and GPU kernel optimizations. Including actual latency measurements ( ms/token or similar) would provide a more practical and complete picture of the efficiency gains.\n- The experiments show results for keep ratios r of 0.9 and 0.5. A more detailed analysis of the trade-off between performance and the keep ratio r would be beneficial. A plot showing how the average performance gracefully degrades as r is decreased would give readers a better sense of the method's sensitivity to this hyperparameter.\n- It is recommended to include a comparison and discussion with these methods [1-4].\n\n[1] Arif, Kazi Hasan Ibn, et al. \"HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 2. 2025. \\\n[2] Xing, Long, et al. \"Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction.\" arXiv preprint arXiv:2410.17247 (2024).  \\\n[3] Wen, Zichen, et al. \"Stop looking for important tokens in multimodal language models: Duplication matters more.\" arXiv preprint arXiv:2502.11494 (2025).  \\\n[4] Ye, Weihao, et al. \"Fit and prune: Fast and training-free visual token pruning for multi-modal large language models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 21. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DoSiYKaTy8", "reviewer": "Reviewer_pdx2", "summary": "This paper targets the inference-efficiency bottleneck of Mamba-based Multimodal Large Language Models (MLLMs) and proposes **Delta-guided Two-stage Pruning (DTP)**: early **top-k** retention of visual tokens and late **complete removal**, performed entirely at inference without retraining. Guided by the $\\Delta_t$ based token importance, the authors select **layer 15** for selective pruning and **layer 45** for complete pruning. Experiments on **Cobra** and **RoboMamba** indicate that DTP can reduce FLOPs by **approximately 50%** while maintaining competitive accuracy.", "questions": "Refer to Weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Architecture-aligned pruning signal:** Avoids modifying model structure or additional training; the target layers are determined via forward passes only.\n2. **Two-stage design:** Preserves sufficient early visual information while removing later-layer redundancy, substantially improving efficiency.", "weaknesses": "1. **Methodology flaws:**\n   a. The 15/45 choices hinge on layer-wise standard deviation of $\\Delta_t$ -derived token importance computed on a **calibration dataset**; the authors mention a **VQAv2 subset** but disclose neither its size nor sampling strategy. Please detail these and validate on **multiple, distinct calibration datasets**.\n   b. A natural **ablation by perturbing the pruning layers** is missing: plots show a $Std_\\ell$ valley near **layer 15** and another around **layers 30–35**, yet no experiments/discussion evaluate these alternatives.\n   c. Reporting the **post-pruning** standard-deviation profiles would further substantiate the effectiveness of the proposed selection.\n2. **Experimental clarity and completeness:**\n   a. In **Table 5**, clarify the comparison between *complete pruning* and *disable complete pruning*: in the latter, what pruning rate is used, or is it the **vanilla** model?\n   b. It is recommended that the authors provide experimental results with different pruning rates to demonstrate the effectiveness of their approach; the main results only include (r=0.9) and (r=0.5).\n   c. Beyond **FLOPs**, include **wall-clock latency** (with **prefill/decoding** breakdown) and **memory footprint**; the two-stage paradigm may affect prefill and decode differently even under the same global token compression.\n   d. The authors should provide details such as the sampling parameters of the specific experiments to improve the reproducibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)