SpecPrune-VLA: Accelerating Vision-Language-Action Models via Action-Aware Self-Speculative Pruning

## Abstract

Abstract Pruning is a typical acceleration method for compute-bound problems by effectively reducing the computation amount. Recently, pruning has been applied to the Vision-Language-Action (VLA) models and emerges as a promising acceleration method by evicting unimportant tokens. However, existing pruning methods only focus on the local information in current action generation and ignore the global information in previous actions, resulting in a reduction of more than 20% in the success rate and limited speedup in some scenarios. In this paper, we point out that the information across consecutive actions exhibits a high degree of similarity, and thus propose the novel insight that combines local information in the current action generation with global information from previous generation for token selection. Based on the insight, we further propose SpecPrune-VLA, a training-free pruning method including two-level token pruning with heuristic control. (1) Static token pruning at action level. We explore the token redundancy through global information in previous actions and the token through local information in the current action generation, statically reducing the number of visual tokens at the action level. (2) Dynamic token pruning at layer level. We exploit the relevance between the tokens and model layer, and dynamically prune tokens based on their layer-specific importance at the layer level. (3) Lightweight action-aware controller. We point out that the generated action can be categorized into coarse-grained and fine-grained based on the speed, with fine-grained actions being sensitive to the error brought by pruning. Therefore, we introduce a lightweight controller that can identify the current action granularity and adjust the pruning strategy accordingly. Extensive experiments show that, compared with the high-performing VLA model OpenVLA-OFT, SpecPrune-VLA achieves an average 1.46 × 1.46\times speedup on NVIDIA A800 GPU and 1.57 × 1.57\times speedup on NVIDIA GeForce RTX 3090 GPU with negligible loss on task success rate in the LIBERO simulation benchmark.

## 1 Introduction

Figure: Figure 1: (a) The mainstream inference dataflow of VLA models. (b) Latency breakdown in three typical VLA models in the LIBERO benchmark during each action generation. (c) The practical arithmetic intensity of three models in the roofline model of NVIDIA A800 GPU.
Refer to caption: x1.png

Figure: Figure 2: Overview of SpecPrune-VLA. We prune the image tokens at two levels with a lightweight action-aware controller.
Refer to caption: x2.png

Vision-Language-Action (VLA) models, built on the large language model (LLM), have attracted much attention for their ability to understand multimodal information and generate robotic action. Previous works like RT-1 (Brohan et al. 2022) and OpenVLA (Kim et al. 2024) exhibit remarkable performance in cross-task generalization and instruction following by learning from real-world robotic interaction. Subsequent works (Team et al. 2024; Li et al. 2024; Black et al. 2024; Kim, Finn, and Liang 2025) further propose more delicate model structures, achieving real-time performance and efficiency improvement. Figure [1](https://arxiv.org/html/2509.05614v1#S1.F1)(a) shows the mainstream inference dataflow of VLA models, consisting of three components: (1) tokenizers that convert multimodal inputs into tokens (e.g., image and text encoders), (2) LLM backbone that understands the multimodal inputs and generates the intermediate results, and (3) an action head that generates continuous low level actions based on the intermediate results. We further select three representative VLA models, OpenVLA (Kim et al. 2024), CogACT (Li et al. 2024), and OpenVLA-OFT (Kim, Finn, and Liang 2025) to profile the latency breakdown of these three components during each action generation in the LIBERO benchmark shown in Figure [1](https://arxiv.org/html/2509.05614v1#S1.F1)(b). Statistical data reveal that OpenVLA-OPT is the high-performing model to date in inference efficiency, and the LLM is the critical bottleneck, accounting for more than 70% of the end-to-end inference latency.

Therefore, most works in VLA model acceleration focus on the LLM backbone acceleration and have explored techniques such as quantization (Park et al. 2024), early exit (Yue et al. 2024), and caching (Xu et al. 2025b) to accelerate VLA model inference. However, they ignore the unique computation characteristics of VLA models, resulting in limited performance and effectiveness. Nowadays, the latest VLA models (e.g., CogACT and OpenVLA-OPT) adopt the single-step inference paradigm that the model directly predicts a sequence of low-level actions through single LLM forward with hundreds of multimodal tokens. As a result, from the perspective of arithmetic intensity (i.e., the amount of computation per byte) in the hardware roofline model, the VLA model inference is primarily compute-bound, shown in Figure [1](https://arxiv.org/html/2509.05614v1#S1.F1)(c), where latency is mainly determined by the computation amount rather than memory access.
Pruning is a typical acceleration method for compute-bound problems by effectively reducing the computation amount. However, existing token-pruning methods in VLA models only consider the local information (e.g., the layer results in current action generation) and ignore the global information across the whole model, leading to either $>20\%$ success rate loss or limited speedup in some scenarios.

In this paper, we observe that input images in consecutive action generations exhibit high similarity due to the short temporal intervals between them. Therefore, we consider that the global information from previous inference steps can be leveraged for more reliable and efficient token pruning. Based on the above insight, we propose SpecPrune-VLA, an acceleration method for Vision-Language-Action Models through action-aware self-speculative pruning. The techniques of SpecPrune-VLA can be summarized into three points as follows.

(1) Static token pruning at action level. Based on the insight, we point out that the tokens between consecutive action generation are largely overlapped (e.g., the background in environment images), leading to significant redundancy. Therefore, we reuse the attention information from the last generation to prune unimportant tokens and retain a globally highly-weighted token set. Then we enhance it with dynamic elements and task-relevant tokens by speed-based frame comparison and self-speculative token selection. By fusing tokens selected from both local and global levels, we can prune over 50% to 70% visual tokens at the beginning of LLM forward.

(2) Dynamic token pruning at layer level. As the input features propagate through the LLM backbone, the local context of each token is progressively enriched by deeper layers. Therefore, we introduce layer-wise pruning by dynamically updating tokens’ importance scores and re-evaluating token importance at different depths. This allows the model to adaptively refine its computation focus, removing redundant tokens as contextual understanding matures.

(3) Lightweight action-aware Controller. We propose that not all actions are equally sensitive to token pruning. Therefore, we categorize actions into coarse-grained (e.g., large translations or rotations) and fine-grained (e.g., grasping) types and design a controller that determines action granularity based on the speed of the end-effector and adaptively adjusts the pruning strategy. It enables a robust trade-off between speed and accuracy across diverse robotic tasks.

We implement SpecPrune-VLA on both the NVIDIA A800-80GB and the NVIDIA GeForce RTX 3090 GPUs to evaluate the performance. Compared to the high-performing model OpenVLA-OFT, our method achieves an average 1.46× speedup on the A800 and 1.57× speedup on the RTX 3090, with negligible degradation in task success rate on the LIBERO simulation benchmark. The consistent speedup gains across both platforms demonstrate that SpecPrune-VLA generalizes well to different hardware architectures.

## 2 Related Works

### 2.1 Vision-Language-Action (VLA) Models

VLA models are typically built upon Large Language Model (LLM) structure (Zitkovich et al. 2023; Liu et al. 2023b; Wang et al. 2024), and are further fine-tuned on extensive datasets from both simulated (Liu et al. 2023a) and real-world (O’Neill et al. 2024) robotic environments. The model receives multimodal information (e.g., images and text) and generates low-level robotic actions. It is investigated that a continuous action space has better performance in manipulation accuracy (Liu et al. 2025). Therefore, an action head, such as a lightweight MLP and diffusion model (Liu et al. 2024; Wen et al. 2025) is introduced to convert action hidden states to continuous actions.
Besides, to achieve high control frequency and action coherence, modern VLA models leverage ACT (Zhao et al. 2023), diffusion models (Peebles and Xie 2023), or parallel decoding to generate action chunks.

Figure: Figure 3: (a) The original image the model sees; (b) The images where unimportant tokens are pruned; (c) The tokens are randomly pruned, some important tokens are pruned; (d) Important tokens(e.g., the tomato sauce) are pruned. (e) The influence of different pruning strategies and pruning numbers of tokens on performance.
Refer to caption: x3.png

### 2.2 Token-level acceleration for VLA models

Existing works have explored token caching or pruning for VLA models. For example, VLA-Cache (Xu et al. 2025b) caches the similar and unimportant tokens’ key and value from the last generation and reuses them in the next inference. However, the effect is limited since it only reduces 50% to 75% FLOPs in attention matrix calculation, about 17% to 25% of all FLOPs in LLM. Visual-token pruning methods such as EfficientVLA (Yang et al. 2025) propose to identify task-relevant tokens using attention maps from a single LLM layer and complement retained tokens with visually diverse patches to preserve coverage. However, this approach relies on a single-layer heuristic that may not reflect global contextual information, and the diversity-driven supplementation can inadvertently introduce task-irrelevant content, limiting its reliability across diverse environments. Besides, (Li et al. 2025) aims to preserve both spatial information and semantic content by retaining tokens with high feature saliency from the vision encoder. While this improves structural integrity, it does not explicitly distinguish between semantically rich and task-irrelevant tokens, which still contribute unnecessary computational load.

### 2.3 Self-speculative decoding and lightweight predictor

Standard speculative decoding (Leviathan, Kalman, and Matias 2023) requires an additional draft model to generate candidate tokens. In contrast, LayerSkip (Elhoushi et al. 2024) proposes a self-speculative method that leverages the early layers of the same model as the draft model, eliminating the need for an external model; the remaining layers are used for verification and correction, thereby reducing memory footprint and improving inference efficiency. Besides, the draft model can also be used as a predictor to provide estimates of helpful information. In SpecEE (Xu et al. 2025a), a small model is utilized as a predictor to narrow down the candidate vocabulary. The predictor filters out unlikely candidates, allowing the model to search within a much smaller vocabulary. This significantly reduces computational overhead.

## 3 Key Insights

Figure: Figure 4: For the task “turn on the oven and put the bottle on it”. (a) Prune the tokens relying on local information(e.g., attention scores from one LLM layer). (b) Prune the tokens relying on the global attention of the LLM in the last action generation. (c) The practically important tokens after inference completion.
Refer to caption: x4.png

### 3.1 What Really Matters in the Image

We first carry out a systematic study on critical components of the images for VLA models. Our focus is on ensuring that the model can effectively recognize task-relevant objects and dynamic elements to accurately perceive the spatial context of the target. Tokens that significantly contribute to this perception are referred to as important tokens.

To validate the effectiveness of our idea, we conduct a simple experiment on some tasks. We let the model complete a generation but don’t act in the environment; then we use the output attention score of the last layer to prune unimportant tokens with low scores. After that, we input the pruned images and let the model regenerate the action and act in the environment. The Figure [3](https://arxiv.org/html/2509.05614v1#S2.F3)(e) shows that pruning a small number of tokens at random has minimal impact on performance, indicating the presence of redundant tokens in the image. However, as more tokens are pruned, the model’s performance degrades sharply. It is worth noting that more than half of the unimportant tokens can be pruned without a significant drop in success rate (compared to 93.3% in the none pruning case). This experimental result provides strong support for the reliability of the pruning approach. Besides, the model suffers drastic performance degradation (up to 80% in success rate) when important tokens are pruned. It further highlights that important tokens need to be retained.

### 3.2 Redundancy largely overlaps in images of consecutive inference

Tokens other than important tokens are considered redundant. It is challenging to identify redundant tokens before the model completes the current inference. Current methods, such as (Li et al. 2025; Yang et al. 2025), utilize local information such as the attention score of one LLM layer or the vision encoder. However, they fail to leverage global information from the whole model and are thus not reliable as shown in Figure [4](https://arxiv.org/html/2509.05614v1#S3.F4)(a) and (c). The important tokens in these two cases vary obviously. In this paper, we emphasize that in VLA models, the overall task goal remains constant throughout the entire execution, and a large proportion of the visual scene remains unchanged across consecutive frames. Therefore, tokens identified as redundant by the model in the previous generation are highly likely to remain redundant in the current step as shown in Figure [4](https://arxiv.org/html/2509.05614v1#S3.F4)(b) and (c). This temporal consistency inspires us to reuse the global attention across time.

## 4 Static Token Pruning at Action Level

### 4.1 Observation and Insight

Due to changing sub-goals and dynamic visual elements, it is necessary to incorporate information of the current generation. We analyze attention-based importance using Equation ([1](https://arxiv.org/html/2509.05614v1#S4.E1)) and observe that 85%–95% of top-k tokens in the first two layers reappear in the final layer’s top-k as shown in Figure[5](https://arxiv.org/html/2509.05614v1#S4.F5), where top-k equals 20. We define this overlap as the hit rate. The high hit rate indicates that early-layer attention provides a reliable local guidance for important token selection. Besides, the first layer alone exhibits a low hit rate, while incorporating both the second and third layers provides only marginal gains in hit rate and introduces extra latency. Therefore, we choose the first two layers for speculation.

Figure: Figure 5: (a) Comparison of the hitrate between leveraging the first one, two, and three layers. (b) LLM latency comparison between leveraging the first one, two, and three layers.
Refer to caption: x5.png

### 4.2 Method

#### Pruning based on global information

Attention score reflects the degree to which other tokens attend to a given token, and we can use attention scores to filter out unimportant visual tokens. The attention score for a visual token to the task is computed as follows: for a given layer,
let $V_{i}$ be a visual token and $T=\{t_{1},t_{2},\dots,t_{m}\}$ be the set of text sequence describing the task. Then, the task attention score of $V_{i}$ in layer $l$ is:

$$ $\text{Score}_{l}(V_{i})=\frac{1}{H\cdot m}\sum_{h=1}^{H}\sum_{j=1}^{m}A_{l}^{h}(V_{i},t_{j}),$ (1) $$

where $A_{l}^{h}(V_{i},t_{j})$ denotes the attention weight from head $h$ in layer $l$ of the cross-attention (or self-attention) mechanism from $V_{i}$ to text token $t_{j}$, $H$ is the number of attention heads, and $M$ is the number of task-relevant text tokens. We define $V_{\text{global}}$ as the set of the $\text{top-}K_{G}$ most important tokens with the highest global attention score from prior inference.

#### Supplementation of Dynamic Tokens

Visual tokens undergoing substantial changes between inference steps cannot be accurately pruned based on global information from the last inference. Therefore, in the current inference step, we explicitly identify and preserve these dynamic tokens to ensure that up-to-date information is retained during token pruning. For two images $I_{1}$ and $I_{2}$, we first divide the images into $N\times N$ patches according to the visual token size. Then, let $\mathbf{P}_{t}^{i,j}$ denote the feature vector of the patch at position $(i,j)$ in frame $I_{n}$. We compute the cosine similarity between corresponding patches as:

$$ $\text{Sim}(\mathbf{P}_{m}^{i,j},\mathbf{P}_{n}^{i,j})=\frac{\mathbf{P}_{m}^{i,j}\cdot\mathbf{P}_{n}^{i,j}}{\|\mathbf{P}_{m}^{i,j}\|_{2}\|\mathbf{P}_{n}^{i,j}\|_{2}},$ $$

where $\|\cdot\|_{2}$ denotes the Euclidean norm.

Figure: Figure 6: The selection strategy for frame comparison.
Refer to caption: x6.png

To identify dynamic tokens, we first filter patches with similarity scores below a threshold $\tau$, then select the top-$k$ patches with the lowest similarity scores from the remaining candidates. Formally, let $\mathcal{P}_{n}=\{\mathbf{P}_{n}^{i,j}\mid 1\leq i,j\leq N\}$ be the set of all patches in frame $I_{n}$. We define the candidate dynamic patches as those with significant changes:

$$ $\mathcal{C}_{n}=\left\{\mathbf{P}_{n}^{i,j}\in\mathcal{P}_{n}\,\middle|\,\text{Sim}(\mathbf{P}_{m}^{i,j},\mathbf{P}_{n}^{i,j})<\tau\right\}.$ (2) $$

The most dynamic $K_{D}$ tokens are then given by:

$$ ${V}_{\text{dynamic}}=\text{Low-}K_{D}\left(\{\text{Sim}_{i,j}\mid\mathbf{P}_{t}^{i,j}\in\mathcal{C}_{t}\}\right),$ (3) $$

Figure: Figure 7: Detailed implementation of static token pruning. We prune the tokens based on the global information from the attention scores in the last action generation, the input image comparison, and the local information from the self-speculative results in the first two layers.
Refer to caption: x7.png

Additionally, as illustrated in Figure [6](https://arxiv.org/html/2509.05614v1#S4.F6)(a), directly comparing adjacent frames can lead to inaccurate results. To mitigate these disturbances and enhance sensitivity to dynamic tokens, we propose a velocity-based frame sampling strategy. This approach selects a historical reference frame that is T frames prior to the current frame, where T is determined based on the average movement speed of the robot arm’s end effector using the equation: $T=\left\lfloor\frac{-16}{3}\cdot\frac{v}{6}+\frac{22}{3}\right\rfloor+4$, where $v_{t}$ denotes the movement speed of the end effector. The calculation of speed is discussed in Section [6](https://arxiv.org/html/2509.05614v1#S6).

This adaptive strategy ensures that we can effectively identify dynamic tokens while minimizing the influence of irrelevant variations and maintaining temporal precision.

#### Pruning based on local information

Our observation and insights suggest that the initial two layers can serve as a reliable predictor to filter task-relevant tokens for current steps. For the first two decoder layers, we compute the task attention scores for all visual tokens as defined in Equation ([1](https://arxiv.org/html/2509.05614v1#S4.E1)). In each layer, we select the $K_{base}$ visual tokens with the highest attention scores to form a candidate set $V_{(1)}$ and $V_{(2)}$ respectively, and take the union of these two sets as the local information representation:

$$ $V_{\text{local}}=V_{(1)}\cup V_{(2)}$ $$

Finally, all the pruning token set is:

$$ $V_{\text{prune}}=U-V_{\text{global}}\cup V_{\text{dynamic}}\cup V_{\text{local}}$ $$

and $U$ is the set of all tokens.

## 5 Dynamic Token Pruning at Layer Level

### 5.1 Method

To preserve the most important tokens in layers, we propose an importance scoring mechanism that leverages attention scores and layer confidence across transformer layers to prune tokens within layers.

#### Importance Score Formulation

The token importance score is initialized for the remaining visual tokens after static token pruning and subsequently updated in the target transformer layers. The importance score $s_{i}^{(l)}$ takes into account both the relative importance weight of tokens and the layer contribution. Therefore, we compute the importance score $s_{i}^{(l)}$ for each token $t_{i}$ in the remaining visual token set $T$ based on two key components:

$$ $s_{i}^{(l)}=\omega_{\text{rank},i}^{(l)}\times\omega_{\text{conf}}^{(l)}$ (4) $$

where $\omega_{\text{rank},i}^{(l)}$ denotes rank-based weight reflecting the token’s relative importance in attention ranking, $\omega_{\text{conf}}^{(l)}$ denotes layer confidence score measuring the layer’s reliability

##### Rank-based Weight

For each attention head, tokens are ranked based on their attention scores. To emphasize the contribution of the most important tokens while maintaining a smooth decay in influence, we introduce a rank-based weighting scheme. This weight is defined as:

$$ $\omega_{\text{rank},i}^{(l)}=\frac{\sigma(-k\cdot\text{rank}_{i}^{(l)})}{\sum_{j}\sigma(-k\cdot\text{rank}_{j}^{(l)})}$ (5) $$

where $\text{rank}_{i}^{(l)}$ is the attention ranking of token $t_{i}$ in layer $l$ and $\sigma(x)$ denotes the sigmoid function, which amplifies the differences between token rankings by mapping them to a smooth range, ensuring that higher-ranked tokens receive significantly more emphasis

##### Layer Confidence Score

Not all layers contribute to the global importance equally. The layer confidence measures the reliability of attention patterns in each layer:

$$ $\omega_{\text{conf}}^{(l)}=\frac{\mu_{\text{attn}}^{(l)}}{\sigma_{\text{attn}}^{(l)}+\epsilon}$ (6) $$

where $\mu_{\text{attn}}^{(l)}$ and $\sigma_{\text{attn}}^{(l)}$ are the mean and standard deviation of attention weights in layer $l$, respectively. The mean of attention weights reflects the overall strength of focus in a layer, while the standard deviation indicates the consistency of the attention pattern. By combining both, the confidence score rewards layers with strong and stable attention, while penalizing those with high variability or weak responses.

#### Dynamic Updating Mechanism

The final importance score $S_{i}$ for each token $t_{i}$ is maintained through an exponential moving average across layers:

$$ $S_{i}^{(l)}=(1-\beta)\cdot S_{i}^{(l-1)}+\beta\cdot s_{i}^{(l)}$ (7) $$

where $\beta$ is the learning rate controlling the update speed, set to 0.2, and $S_{i}^{(0)}=0$ is set for initialization.

## 6 Lightweight Action-aware Controller

### 6.1 Observation and Insight

In our experiments, when we prune more tokens, we saw a drop in success rate. We observed every frame and found that failure cases occurred during interactions involving object contact, such as manipulating or placing objects in Figure [8](https://arxiv.org/html/2509.05614v1#S6.F8)(a), and the task merely fails when those actions are successfully executed.
Therefore, we emphasize that task success is highly dependent on fine-grained actions, which require high precision and are sensitive to execution errors. In contrast, coarse-grained actions (e.g., moving to a general location) are more tolerant to inaccuracies.

Specifically, when the robot approaches an object, the subsequent interaction often demands fine-grained control to ensure stable contact and successful manipulation. Thus, the granularity of the action—whether fine or coarse—plays a critical role in determining the required level of accuracy.

Figure: Figure 8: (a) The task process consists of four stages, which can be categorized into coarse-grained and fine-grained actions depending on their respective velocity. (b) Typical failure cases in two fine-grained stages.
Refer to caption: x8.png

Inspired by this observation, we propose an action-aware pruning strategy. By identifying whether a step requires fine or coarse control, our method can selectively preserve more candidates in fine-grained phases while allowing aggressive pruning in coarse-grained ones, improving both efficiency and success rate.

Figure: Algorithm 1 Action-Aware Pruning in one layer

### 6.2 Method

In robotic task execution, we divide the overall process into four phases: targeting, approaching, transferring, and placing. Among these, the object approaching and placing phases involve high-precision operations and are thus classified as *fine manipulation stages*. During these stages, the translational and rotational velocities of the gripper are generally maintained at low levels, and the velocity along the $z$-axis is typically negative or smaller than a small positive value, ensuring stability and accuracy.

Since each action corresponds to a fixed unit time duration, the robot’s policy effectively outputs a normalized velocity command through these relative displacements. We use the displacement magnitude per step as a proxy for instantaneous velocity.

**Table 1: Performance Evaluation**
| Method | Success Rate (%) / Latency (ms) (Speedup) | Average | FLOPs |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Spatial | Object | Goal | Long | Speedup |  |  |
| OpenVLA-OFT | 97.6 / 109  (1.00$\times$) | 96.5 / 109  (1.00$\times$) | 97.9 / 109  (1.00$\times$) | 94.5 / 109  (1.00$\times$) | 1.00$\times$ | 100% |
| SparseVLM | 96.8 / 85.3 (1.28$\times$) | 94.2 / 85.3 (1.28$\times$) | 97.6 / 85.3 (1.28$\times$) | 93.6 / 85.3 (1.28$\times$) | 1.28$\times$ | 77% |
| VLA-Cache | 99.0 / 101  (1.08$\times$) | 97.7 / 102  (1.07$\times$) | 97.4 / 102  (1.07$\times$) | 93.6 / 102  (1.07$\times$) | 1.07$\times$ | 83% |
| EfficientVLA | 96.5 / 68.8 (1.58$\times$) | 91.1 / 71.4 (1.53$\times$) | 96.0 / 73.7 (1.48$\times$) | 72.1 / 68.6 (1.59$\times$) | 1.55$\times$ | 35% |
| Ours | 98.2 / 72.4 (1.51$\times$) | 96.3 / 76.2 (1.43$\times$) | 97.7 / 73.6 (1.48$\times$) | 94.0 / 78.1 (1.40$\times$) | 1.46$\times$ | 43% |

**Table 2: Ablation Study on LIBERO-Spatial**
|  | SR (%) | Latency (ms) | Speedup |
| --- | --- | --- | --- |
| None | 97.6 | 109 | 1.00$\times$ |
| Tech.1 | 97.6 | 76.6 | 1.42$\times$ |
| Tech.1&2 | 96.8 | 70.8 | 1.54$\times$ |
| Tech.1&2&3 | 98.2 | 72.4 | 1.51$\times$ |

The translational velocity $v_{t}$ is defined as the Euclidean norm of the gripper’s 3D relative displacement vector output by the policy at each step:

$$ $v_{t}=\|\Delta x,\Delta y,\Delta z\|_{2}=\sqrt{(\Delta x)^{2}+(\Delta y)^{2}+(\Delta z)^{2}}.$ (8) $$

The rotational velocity $v_{r}$ is computed as the magnitude of the angular displacement:

$$ $v_{r}=\|\Delta\alpha,\Delta\beta,\Delta\gamma\|_{2}=\sqrt{(\Delta\alpha)^{2}+(\Delta\beta)^{2}+(\Delta\gamma)^{2}},$ (9) $$

expressed in radians. The $z$-axis velocity is represented by the vertical displacement component $\Delta z$.

To formalize this behavior, we define three thresholds: a translational displacement threshold $v_{t}^{\text{th}}$, a rotational displacement threshold $v_{r}^{\text{th}}$, and a $z$-axis displacement threshold $v_{z}^{\text{th}}$ (a small positive value). The system enters the precise mode when $v_{t}<v_{t}^{\text{th}},v_{r}<v_{r}^{\text{th}}\quad\text{and}\quad\Delta z\leq 0$ .

Conversely, the system exits the mode when $v_{t}>v_{t}^{\text{th}}\quad\text{or}\quad v_{r}>v_{r}^{\text{th}}$ ,
which typically occurs during object lifting. This strategy ensures early activation of precision control before contact and timely deactivation afterward, effectively balancing operational accuracy with task efficiency in dynamic manipulation scenarios.

This strategy ensures the robot initiates fine manipulation mode early during the approach phase, prior to physical contact with the target object. This early activation provides sufficient time for precise positional and orientation adjustments to minimize positioning errors. Subsequent z-axis velocity increases upon successful object contact and lifting trigger an immediate exit from fine mode. The specific control flow is shown in Algorithm [1](https://arxiv.org/html/2509.05614v1#alg1), which demonstrates the process within one layer of the LLM.

## 7 Experiment

### 7.1 Experimental Settings

#### Benchmarks and Platforms.

We conduct evaluations on the LIBERO simulation benchmark (Liu et al. 2023a), which utilizes a simulated Franka Emika Panda robotic arm.
We employ four task suites, LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long, to evaluate the model’s capabilities in spatial reasoning, object understanding, goal-directed planning and execution, and long-horizon task completion, respectively. Each task suite consists of ten distinct tasks. All main experiments are conducted on a Linux workstation with an NVIDIA A800-80GB GPU.

#### Implementation Details.

We select the OpenVLA-OFT (Kim, Finn, and Liang 2025) as the code base for the implementation of our techniques. Due to the varying difficulty and focus of each task suite, we adopt different overall pruning ratios $\alpha$ to balance accuracy and inference speed. Specifically, $\alpha$ is set to 1.0 for LIBERO-Spatial , 0.8 for LIBERO-Goal and 0.6 for LIBERO-Object and LIBERO-Long. We also explore the influence of different ratios in design space exploration(DSE) in Table [3](https://arxiv.org/html/2509.05614v1#S7.T3).
We repeat every task of every task suite for 40 to 50 times to mitigate the impact of random errors. The latency here is defined as the time duration from when the model receives the input to when it generates the action. We report the average inference time over the ten tasks within each task suite.

#### Baselines

We select OpenVLA-OFT as our target model. It utilizes DINOv2 (Oquab et al. 2023) and SigLIP (Zhai et al. 2023) as visual encoders to extract visual features, Llama2-7B as the backbone LLM, and a four-layer MLP as an action head to generate continuous actions. The model receives two-view images: the third-person view and the wrist view. We also consider three optimization methods: SparseVLM (Zhang et al. 2024), a framework that adaptively sparsifies less important visual tokens and recycles their information to minimize performance loss, EfficientVLA (Yang et al. 2025), a visual pruning approach for VLA models, and VLA-Cache (Xu et al. 2025b) , which leverages image similarity to cache features across time steps.

### 7.2 Evaluation on Speedup and Success Rate

Table [1](https://arxiv.org/html/2509.05614v1#S6.T1) shows the end-to-end evaluation on success rate (SR), latency and speedup on four LIBERO datasets. SpecPrune-VLA achieves an average speedup of 1.46$\times$ with negligible loss ($<0.7\%$) in success rate. The speedup ratio fluctuates slightly because the task difficulty and environment of different datasets vary. VLA-Cache achieves the high success rate on four datasets, but with limited speedup. This is consistent with our analysis that VLA-Cache reduces limited computation. Besides, it also introduces KV cache access latency. EfficientVLA(L=28, T=112) achieves higher speedup by skipping four fixed layers and aggressively pruning tokens; however, this leads to a significant drop in success rate in some scenarios, as important tokens and critical action-relevant information in the hidden states are compromised. It is worth noting that the approach is originally for VLA models with a diffusion action expert, which may be more tolerant to the changing output action hidden states.

### 7.3 Ablation Study

To evaluate the effectiveness of our proposed method, we conducted ablation study on the LIBERO-Spatial dataset.
Our full model achieves a success rate (SR) of 98.2%, which is better than the baseline (97.6%). and it significantly reduces latency from 109 milliseconds to 72.4 milliseconds, resulting in a speedup of 1.51$\times$ compared to OpenVLA-OFT. This demonstrates the efficiency gains of our approach while maintaining competitive accuracy.
The ablation study further highlights the importance of each component:
Static(Tech 1) and Dynamic(Tech 2) pruning slightly affect the SR (96.8%) but reduces latency to 70.8 milliseconds, indicating that pruning contributes to the overall latency reduction.
The introduction of the action adapter increases the success rate and cause negligible latency(1.6ms). This suggests that the action adapter plays a crucial role in maintaining high accuracy.

### 7.4 Design Space Exploration

We conduct a design space exploration on the LIBERO-Object. To reduce the model latency while maintaining accuracy, we focus on the number of pruned tokens. We set a prune ratio $\alpha$ to adjust both static token pruning and dynamic token pruning. The bigger the prune ratio, the more tokens are pruned, leading to a drop in success rate and a rise in speedup. To balance accuracy and speed, we set the prune ratio to 0.6 for LIBERO-Object.

**Table 3: Design Space Exploration on Pruning Ratio**
| Prune | Pruned | SR (%) | Latency(ms) / Speedup |
| --- | --- | --- | --- |
| Ratio ($\alpha$) | Tokens |  |  |
| 1.0 | 346 | 83.7 | 71.9 / 1.52$\times$ |
| 0.8 | 329 | 95.2 | 74.8 / 1.46$\times$ |
| 0.6 | 318 | 96.3 | 76.2 / 1.43$\times$ |

Figure: Figure 9: Evaluation on NVIDIA GeForce RTX 3090
Refer to caption: x9.png

### 7.5 Evaluation on Various Computing Platforms

To validate the applicability of our method on different devices, we conduct experiments on NVIDIA GeForce RTX 3090. As illustrated in Figure [9](https://arxiv.org/html/2509.05614v1#S7.F9), our method achieves an average speedup of 2.09$\times$ in LLM inference time and 1.57$\times$ in end-to-end latency across all task categories using the same parameters as Table [1](https://arxiv.org/html/2509.05614v1#S6.T1). The results consistently demonstrate improved inference efficiency, underscoring the scalability and effectiveness of our approach under diverse computational conditions.

### 7.6 Experiment Visualization

To understand what the model actually observed when completing a task, we visualize the retained visual tokens. The colored patches are the retained tokens. We show the visualization results of three tasks in Figure [10](https://arxiv.org/html/2509.05614v1#S7.F10). The results show that retained tokens are the important tokens discussed in previous sections. Moreover, the method works when processing images from different viewpoints, demonstrating its ability to capture viewpoint-invariant representations.

Figure: Figure 10: Retained tokens during tasks
Refer to caption: x10.png

## 8 Conclusion

In this paper, we propose a novel insight that combines local information in the current action generation with global information in previous actions for token selection, and present SpecPrune-VLA, a training-free pruning method including two-level token pruning with heuristic control. Experiments show that SpecPrune-VLA achieve an average 1.46× speedup on NVIDIA A800-80GB and 1.57× speedup on NVIDIA GeForce RTX 3090, with negligible degradation in task success rate on the LIBERO simulation benchmark. The evaluation demonstrates that our method generalizes well across different hardware platforms.

## 9 Limitation

One limitation of this work is that all experiments are conducted in simulated environments. While simulation enables controlled and scalable evaluation, real-world deployment may introduce additional challenges such as sensor noise, environmental dynamics, and hardware latency. We acknowledge this gap and plan to deploy our method on physical platforms in the future when opportunities arise, to validate its effectiveness in real-world scenarios.

## References

- Black et al. (2024)
Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. 2024.
$\pi{{}_{0}}$: A Vision-Language-Action Flow Model for General Robot Control.
*arXiv preprint arXiv:2410.24164*.
- Brohan et al. (2022)
Brohan, A.; Brown, N.; Carbajal, J.; Chebotar, Y.; Dabis, J.; Finn, C.; Gopalakrishnan, K.; Hausman, K.; Herzog, A.; Hsu, J.; et al. 2022.
Rt-1: Robotics transformer for real-world control at scale.
*arXiv preprint arXiv:2212.06817*.
- Dong et al. (2025)
Dong, Y.; Miao, Y.; Li, W.; Zheng, X.; Wang, C.; and Lyu, F. 2025.
Accelerating llm inference throughput via asynchronous kv cache prefetching.
*arXiv preprint arXiv:2504.06319*.
- Elhoushi et al. (2024)
Elhoushi, M.; Shrivastava, A.; Liskovich, D.; Hosmer, B.; Wasti, B.; Lai, L.; Mahmoud, A.; Acun, B.; Agarwal, S.; Roman, A.; et al. 2024.
LayerSkip: Enabling early exit inference and self-speculative decoding.
*arXiv preprint arXiv:2404.16710*.
- He et al. (2025)
He, Z.; Yao, Y.; Zuo, P.; Gao, B.; Li, Q.; Zheng, Z.; and Wu, F. 2025.
Adaskip: Adaptive sublayer skipping for accelerating long-context llm inference.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 24050–24058.
- Kim, Finn, and Liang (2025)
Kim, M. J.; Finn, C.; and Liang, P. 2025.
Fine-tuning vision-language-action models: Optimizing speed and success.
*arXiv preprint arXiv:2502.19645*.
- Kim et al. (2024)
Kim, M. J.; Pertsch, K.; Karamcheti, S.; Xiao, T.; Balakrishna, A.; Nair, S.; Rafailov, R.; Foster, E.; Lam, G.; Sanketi, P.; et al. 2024.
Openvla: An open-source vision-language-action model.
*arXiv preprint arXiv:2406.09246*.
- Leviathan, Kalman, and Matias (2023)
Leviathan, Y.; Kalman, M.; and Matias, Y. 2023.
Fast inference from transformers via speculative decoding.
In *International Conference on Machine Learning*, 19274–19286. PMLR.
- Li et al. (2024)
Li, Q.; Liang, Y.; Wang, Z.; Luo, L.; Chen, X.; Liao, M.; Wei, F.; Deng, Y.; Xu, S.; Zhang, Y.; et al. 2024.
Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation.
*arXiv preprint arXiv:2411.19650*.
- Li et al. (2025)
Li, Y.; Meng, Y.; Sun, Z.; Ji, K.; Tang, C.; Fan, J.; Ma, X.; Xia, S.; Wang, Z.; and Zhu, W. 2025.
SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration.
*arXiv preprint arXiv:2506.12723*.
- Liu et al. (2023a)
Liu, B.; Zhu, Y.; Gao, C.; Feng, Y.; Liu, Q.; Zhu, Y.; and Stone, P. 2023a.
Libero: Benchmarking knowledge transfer for lifelong robot learning.
*Advances in Neural Information Processing Systems*, 36: 44776–44791.
- Liu et al. (2023b)
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023b.
Visual instruction tuning.
*Advances in neural information processing systems*, 36: 34892–34916.
- Liu et al. (2025)
Liu, H.; Li, X.; Li, P.; Liu, M.; Wang, D.; Liu, J.; Kang, B.; Ma, X.; Kong, T.; and Zhang, H. 2025.
Towards generalist robot policies: What matters in building vision-language-action models.
- Liu et al. (2024)
Liu, J.; Liu, M.; Wang, Z.; Lee, L.; Zhou, K.; An, P.; Yang, S.; Zhang, R.; Guo, Y.; and Zhang, S. 2024.
Robomamba: Multimodal state space model for efficient robot reasoning and manipulation.
*arXiv e-prints*, arXiv–2406.
- Oquab et al. (2023)
Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. 2023.
Dinov2: Learning robust visual features without supervision.
*arXiv preprint arXiv:2304.07193*.
- O’Neill et al. (2024)
O’Neill, A.; Rehman, A.; Maddukuri, A.; Gupta, A.; Padalkar, A.; Lee, A.; Pooley, A.; Gupta, A.; Mandlekar, A.; Jain, A.; et al. 2024.
Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0.
In *2024 IEEE International Conference on Robotics and Automation (ICRA)*, 6892–6903. IEEE.
- Park et al. (2024)
Park, S.; Kim, H.; Jeon, W.; Yang, J.; Jeon, B.; Oh, Y.; and Choi, J. 2024.
Quantization-aware imitation-learning for resource-efficient robotic control.
*arXiv preprint arXiv:2412.01034*.
- Peebles and Xie (2023)
Peebles, W.; and Xie, S. 2023.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF international conference on computer vision*, 4195–4205.
- Sapkota et al. (2025)
Sapkota, R.; Cao, Y.; Roumeliotis, K. I.; and Karkee, M. 2025.
Vision-language-action models: Concepts, progress, applications and challenges.
*arXiv preprint arXiv:2505.04769*.
- Team et al. (2024)
Team, O. M.; Ghosh, D.; Walke, H.; Pertsch, K.; Black, K.; Mees, O.; Dasari, S.; Hejna, J.; Kreiman, T.; Xu, C.; et al. 2024.
Octo: An open-source generalist robot policy.
*arXiv preprint arXiv:2405.12213*.
- Vaswani et al. (2023)
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2023.
Attention Is All You Need.
arXiv:1706.03762.
- Wang et al. (2024)
Wang, P.; Bai, S.; Tan, S.; Wang, S.; Fan, Z.; Bai, J.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Fan, Y.; Dang, K.; Du, M.; Ren, X.; Men, R.; Liu, D.; Zhou, C.; Zhou, J.; and Lin, J. 2024.
Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution.
arXiv:2409.12191.
- Wen et al. (2025)
Wen, J.; Zhu, Y.; Zhu, M.; Tang, Z.; Li, J.; Zhou, Z.; Liu, X.; Shen, C.; Peng, Y.; and Feng, F. 2025.
DiffusionVLA: Scaling Robot Foundation Models via Unified Diffusion and Autoregression.
In *Forty-second International Conference on Machine Learning*.
- Xu et al. (2025a)
Xu, J.; Pan, J.; Zhou, Y.; Chen, S.; Li, J.; Lian, Y.; Wu, J.; and Dai, G. 2025a.
Specee: Accelerating large language model inference with speculative early exiting.
In *Proceedings of the 52nd Annual International Symposium on Computer Architecture*, 467–481.
- Xu et al. (2025b)
Xu, S.; Wang, Y.; Xia, C.; Zhu, D.; Huang, T.; and Xu, C. 2025b.
Vla-cache: Towards efficient vision-language-action model via adaptive token caching in robotic manipulation.
*arXiv preprint arXiv:2502.02175*.
- Yang et al. (2025)
Yang, Y.; Wang, Y.; Wen, Z.; Zhongwei, L.; Zou, C.; Zhang, Z.; Wen, C.; and Zhang, L. 2025.
EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models.
*arXiv preprint arXiv:2506.10100*.
- Yue et al. (2024)
Yue, Y.; Wang, Y.; Kang, B.; Han, Y.; Wang, S.; Song, S.; Feng, J.; and Huang, G. 2024.
Deer-vla: Dynamic inference of multimodal large language models for efficient robot execution.
*Advances in Neural Information Processing Systems*, 37: 56619–56643.
- Zhai et al. (2023)
Zhai, X.; Mustafa, B.; Kolesnikov, A.; and Beyer, L. 2023.
Sigmoid loss for language image pre-training.
In *Proceedings of the IEEE/CVF international conference on computer vision*, 11975–11986.
- Zhang et al. (2024)
Zhang, Y.; Fan, C.-K.; Ma, J.; Zheng, W.; Huang, T.; Cheng, K.; Gudovskiy, D.; Okuno, T.; Nakata, Y.; Keutzer, K.; et al. 2024.
Sparsevlm: Visual token sparsification for efficient vision-language model inference.
*arXiv preprint arXiv:2410.04417*.
- Zhao et al. (2023)
Zhao, T. Z.; Kumar, V.; Levine, S.; and Finn, C. 2023.
Learning fine-grained bimanual manipulation with low-cost hardware.
*arXiv preprint arXiv:2304.13705*.
- Zitkovich et al. (2023)
Zitkovich, B.; Yu, T.; Xu, S.; Xu, P.; Xiao, T.; Xia, F.; Wu, J.; Wohlhart, P.; Welker, S.; Wahid, A.; et al. 2023.
Rt-2: Vision-language-action models transfer web knowledge to robotic control.
In *Conference on Robot Learning*, 2165–2183. PMLR.

## 10 Appendix

### 10.1 Complexity Analysis

Consider a Transformer model with $N$ layers. The computational cost (in FLOPs) per layer when processing $L$ tokens is approximately:

$$ $\text{FLOPs}_{\text{layer}}(L)=4LD^{2}+2L^{2}D+2LDM$ $$

where:

- $L$: sequence length (number of tokens)

- $D$: hidden dimension

- $M$: number of attention heads

##### Static Token Pruning

The static token pruning strategy reduces an average of 315 tokens from the original about 600 tokens, let $L_{r}=\alpha\cdot{L}$ denote the number of retained tokens, with $\alpha=0.48$.

**Table 4: Number of pruned tokens across different LIBERO datasets under static pruning. The pruned number slightly changes as the environment changes.**
| Benchmark | Spatial | Goal | Object | Long | Average |
| --- | --- | --- | --- | --- | --- |
| Pruned tokens | 333 | 326 | 318 | 304 | 315 |

Thus, for most layers, the FLOPs become $\text{FLOPs}(L_{r})$.

##### Dynamic Token Pruning

We apply an extra 10% pruning at the target layer:

$$ $\mathcal{S}=\{5,10,15,20\}$ $$

Therefore, the number of retained tokens after these layers becomes $L_{r^{\prime}}=0.9\cdot L_{r}$
On average, this reduces the token count by $19\%$ across all layers.

##### Overall FLOPs reduction

Hence, the overall FLOPs reduction is:

$$ $\Delta\text{FLOPs}=(1-\frac{30}{32}\times 0.48\times 0.81)\times\text{FLOPs}\approx 0.63\text{FLOPs}$ $$

### 10.2 Experiment Details

#### Implementation of Comparative Methods

We provide details on the implementation of the baseline and comparative methods. All models were implemented on one A800 GPU.

##### OpenVLA-OFT

The latency and the success rate on four datasets are similar to the data in the original paper.

##### VLA-Cache

While the original method was developed on the OpenVLA model, its authors adapted and extended it to the OpenVLA-OFT. All results reported in our experiments are obtained using the authors’ official implementation to ensure reproducibility.

##### EfficientVLA

The method focuses on VLA models with diffusion action expert and it also optimizes the action expert. Besides, it has not been open-sourced. As a result, we re-implement it according to the details of visual token pruning and layer pruning provided in the original paper. Following the reported setup, we retain 28 transformer layers and 56 visual tokens throughout inference.

#### Experimental Setup

In the static token pruning stage, the number of tokens preserved from both the global top-$K$ tokens (from the previous inference step) and the current top-$K$ tokens is controlled by a base threshold. We set the base threshold to $K_{c}=\text{ATTN_TOPK}=24$ for coarse-grained mode and $K_{p}=\text{ATTN_TOPK_PRECISE}=40$ for fine-grained mode.

The actual number of retained tokens is scaled by the pruning rate $\alpha$, such that:

$$ $K_{\text{retain}}=\alpha\times\begin{cases}K_{c},&\text{coarse-grained mode},\\ K_{p},&\text{fine-grained mode}.\end{cases}$ $$

We apply different $\alpha$ values depending on the dataset to balance efficiency and performance:

- LIBERO-Long: $\alpha=1.0$

- LIBERO-Goal: $\alpha=0.8$

- LIBERO-Object: $\alpha=0.6$

- LIBERO-Spatial: $\alpha=0.6$

#### Experiment Visualization

To understand what the model actually observed when completing a task, we visualize the retained visual tokens. The colored patches are the retained tokens. We show the visualization results of four tasks on four LIBERO datasets in Figure [11](https://arxiv.org/html/2509.05614v1#S10.F11). It shows the retained tokens are the task-relevant and dynamic tokens.

### 10.3 Further Analysis

Different camera viewpoints present distinct characteristics. In this section, we provide a systematic exposition of the first key insight.

In the fixed third-person view, the robot arm and objects it touches are the dynamic component, while the task-relevant patches (e.g., objects on the table) are relatively static, as Figure  [11](https://arxiv.org/html/2509.05614v1#S10.F11) shows. Therefore, the key is to extract the intersection of both — the regions that involve the dynamic and the task-relevant pixels.

In the wrist-mounted camera view, although all objects are in motion, the pixel patches along object boundaries exhibit more significant changes. We process and interpret the temporal signal of a pixel patch using the Fourier transform:

$$ $P(f)=\int_{-\infty}^{\infty}p(t)e^{-j2\pi ft}dt$ (10) $$

Here $p(t)$ represents pixel intensity at time $t$ , $P(f)$ denotes the frequency-domain representation of the signal, $f$ stands for frequency. Since objects have different colors from the background, the boundary patches will change color at a certain time. Therefore, in these regions, the resulting signal $p(t)$ contains more abrupt changes, which correspond to higher energy in the high-frequency components of $P(f)$. This indicates that boundary patches carry richer temporal dynamics.
As a result, task-relevant patches and dynamic patches can be complementary to some extent, and their intersection can also be used to identify the most critical regions.

Figure: Figure 11: Retained tokens during tasks across four datasets and different views.
Refer to caption: x11.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. How do you ensure that reusing global attention maps from previous steps does not propagate outdated or erroneous importance information, especially in tasks with dynamic camera motion or sudden object appearance?
2. Many parameters (τ, α, β, K-values, pruning thresholds) are hand-tuned. Have you tested the robustness of performance under different hyperparameter configurations? How sensitive is your approach to these values?
3. The velocity-based controller seems tailored to a specific robot arm setup. How general is this to other robotic systems or action spaces?
4. For 4.1.1, the attention from the 15th and 32nd layer are selected to indicate the performance. What is the rationale? Have you tried other layers?
soundness: 2
strengths: 1. Improving the efficiency of VLA is an important research topic. 
2. The writing of the paper is quite clear and easy to follow.
3. The method is training-free, making it easily integrable into existing VLA pipelines.
summary: The paper introduces SpecPrune-VLA, a training-free two-level token pruning framework designed to accelerate inference in VLA models. It leverages global context reuse across consecutive action steps and local speculative attention to select important visual tokens, while a lightweight action-aware controller dynamically adjusts pruning aggressiveness based on robot end-effector velocity. Experiments on the LIBERO benchmark and real-world robotic platforms show that SpecPrune-VLA achieves up to 1.57× speedup (A800 GPU) and 1.7× real-world speedup with negligible task success rate degradation (<1%) compared to OpenVLA-OFT.
weaknesses: 1. The mathematical formulation of global/local fusion and the entropy-confidence weighting is heuristic.

2. Too many hyperparameters are involved in the proposed method. The pruning heavily depends on multiple manually tuned thresholds (α, τ, β, K values).

3. Experiments are limited to OpenVLA-OFT.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: - Sec 3
  - How many tokens per image in total in Fig.3?
- Sec 4
  - In Eq.1, how are the layers selected?
  - In Eq.1, what does the subscript  $i$ in $V_i$ denote?
  - In $T=\lfloor b + k\cdot v \rfloor + 4$, both translational velocity and rotational velocity seem to be included in $v$. Are these two types of velocity handled in the same way, or is there a difference in how they are applied?
  - For sec.4.1.2, could you discuss the computational complexity and the overhead introduced by the method?
  - Have you tried different similarity metrics?
soundness: 2
strengths: - The efficiency for VLA is essential, as it highly impacts the performance of real robot controlling systems.
- The proposed two-level pruning, combining global and local information, is well-motivated and beneficial.
- The coarse/fine-grained switching based on real scenarios is helpful to improve the performance.
summary: The paper presents SpecPrune-VLA, a training-free acceleration scheme for Vision-Language-Action (VLA) models that prunes visual tokens using action-level static pruning that reuses global attention statistics from the previous action step and supplements them with local early-layer attention and motion-based dynamic tokens. They also adopt payer-level dynamic pruning driven by rank-weighted attention and per-layer attention-entropy and a lightweight action-aware controller that reduces pruning during fine-grained manipulation and increases it during coarse motion.
The evaluation on LIBERO tasks shows 1.46$\times$ average end-to-end speedup over OpenVLA-OFT and 1.70$\times$ on real robot.
weaknesses: - The paper lacks an analysis of the importance/confidence scores. How are the layer confidences distributed?
- Ablation in Tab.2 is incomplete. e.g., there is no ablation on global only vs local only, entropy-based layer confidence vs rank-only weighting.
- Some annotations are potentially confusing. For example,  line-227 $I_1,I_2$ might be clearer if written as $I_t,I_{t+1}$

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness section.
soundness: 3
strengths: 1. The diagnosis is intuitive, especially the demonstration that now accelerations are bound by compute. Thus pruning image tokens is really intuitive, especially since there are redundancies in images.
2. The results are convincing as higher speedup and better success rate are shown in the experiment section.
3. The paper is well written, and the explanation of the method is clear.
summary: The paper proposes a pruning method for VLA models. The authors diagnose and show that current VLA acceleration is compute-bound, so it’s intuitive to prune tokens in LLM, thus redundant visual patches as nearby images in the query are often similar in background and importance. They also incorporate global information from the previous step to guide pruning and introduce an action-aware controller adapter. Experiments show that their approach achieves higher speedup than prior methods while achieving better task success rates.
weaknesses: 1. The method mostly relies on hand-tuned heuristics (velocity thresholds, entropy weighting, pruning ratios), it would be great to see extended ablation studies.
2. Do experiments in VLA pruning usually not report success rates and confidence intervals? This can be hard to judge statistical significance.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - How sensitive is the performance to the choice of temporal window T in the dynamic frame comparison? Would an adaptive method (e.g., using visual change detection) yield better results? Also, the method reuses attention scores from layers 15 and 32 of the prior step. Why these specific layers? Is there a systematic way to select “global” layers, or is this also empirical?

- The ablation in Table 2 implies the controller adds 1.5ms (72.3ms vs 70.8ms). What is the total latency cost of calculating $V_{local}$ (requiring partial forward passes), $V_{dynamic}$ (patch similarity), and the layer-wise entropy scores?

- EfficientVLA is noted to be originally designed for **diffusion-based action heads**, yet it’s compared against OpenVLA-OFT (which uses an MLP head). While the authors acknowledge this, the performance gap (e.g., 24.4% SR drop in LIBERO-Long) may partly stem from architectural mismatch rather than pruning quality alone.
soundness: 3
strengths: - The paper is built on clear insights specific to the robotics domain: VLA inference is compute-bound, visual information is temporally redundant across action steps, and not all actions are equally sensitive to pruning.

- The lightweight action-aware controller is practical. It uses end-effector velocity to heuristically differentiate between coarse- and fine-grained actions, reducing pruning during high-precision phases. The ablation study confirms that this controller is critical for recovering the success rate lost due to aggressive pruning.

- The method is evaluated thoroughly in both simulation on the LIBERO benchmark and on a physical robot. Demonstrating consistent speedups (1.46x-1.70x) with minimal success rate loss across multiple hardware platforms (NVIDIA A800 and RTX 3090)  strongly supports the claims of effectiveness and generalization.
summary: This paper addresses the inference latency bottleneck in Vision-Language-Action (VLA) models, primarily due to the LLM backbone. The paper presents a well-motivated, comprehensive pruning strategy tailored to VLA models. The primary strengths are its training-free design and the novel action-aware controller, though the complexity and tuning of the multi-stage pruning process are minor weaknesses.
weaknesses: - Concern about the reproducibility and generalizability of the pruning configuration. The method relies on several hyperparameters ($K_G$, $K_D$, $K_L$, $\beta$, $L_{prune}$, etc.), notably the overall pruning ratio $\alpha$, which is set manually for each of the four LIBERO task suites. It is unclear how $\alpha$ should be determined for a new, unseen task without iterative tuning.

- The complexity of the two-level pruning interaction might be high. Tokens are first pruned statically at the action level ($V_{retain} = V_{global} \cup V_{dynamic} \cup V_{local}$), and this retained set is then pruned dynamically at the layer level. This multi-stage filtering, combined with the action-aware controller, makes the final token count difficult to track and analyze.

- The velocity-based frame sampling strategy appears heuristic, with constants chosen empirically. While effective, the paper lacks analysis of how sensitive performance is to these hyperparameters or whether this strategy generalizes across robot morphologies or task distributions.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: The paper is an interesting read. I have the following questions:

A>The paper focuses on different varieties of a single dataset: LIBERO. It would be great if the authors could explain the scalability of the approach. I understand from the website of the dataset - that every variety has 10 tasks. Is there a possibility that the accelerations are limited to the lower number of tasks?

B>How do you calculate “rank-based weight” in equation 4?
soundness: 3
strengths: A> The method is training free

B> The method proposes a multi-level pruning strategy, thereby helping the user to choose the computational requirements.

C> The pipeline combines local and global context  and uses that to make the end to end pipeline efficient.
summary: The authors propose SpecVLA - an acceleration method for Vision-Language-Action Models. The acceleration is obtained by A)  Action-level static pruning: the authors compute an attention based score on the visual and text tokens from the middle and the last layer, then the top K visual tokens are chosen, based on the score values. The authors add top-k dynamic tokens, if the corresponding cosine similarity is greater than a threshold value. This combination of tokens is also carried forward to the next step, if two frames are quite similar. B) Layer pruning: The attention scores are modified to include a metric for token confidence. Attention entropy is calculated and is modified to consider a metric for layer confidence. A final importance score for the tokens is calculated to help dynamic pruning. C) Action aware controller: The model shifts tasks based on the rotational and translational velocity.
weaknesses: A> The novelty of the method is limited. The proposed contributions have appeared in other contexts earlier. For instance: Attention entropy in LLMs to ensure faster inference [1], choosing transformer layers and tokens based on context [2], calculating token importance score [3], combining local and global features for inference [4], using translation and rotation velocities to determine tasks [5]. 

B> In equation 4, the term “rank-based weight” is vague. It is not clear if it is a concatenation of the earlier scores or is a smaller network.

C> There are a lot of threshold and design choices that need clarifications to understand the optimal choice, for instance: the value of top-K tokens, velocity thresholds, dynamic token threshold. The appendix includes instances of tokens pruned - but it does not show a trade-off between performance and latency, to understand the optimal values.

D> The paper has limited experimental results - the pipeline has been evaluated on a single dataset, but multiple devices. Since there is an intersection of scope with a lot of other efficient transformer works, the paper lacks comparison with other contemporary efforts.


[1] Zhang, Zhisong, et al. "Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models." CoRR (2024).

[2] Meng, Lingchen, et al. "Adavit: Adaptive vision transformers for efficient image recognition." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[3] Xu, Xuwei, et al. "GTP-ViT: efficient Vision transformers via graph-based token propagation." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[4] Norouzi, Narges, et al. "Algm: Adaptive local-then-global token merging for efficient semantic segmentation with plain vision transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[5] Qu, Delin, et al. "SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model." CoRR (2025).


Decision: Reject