Vulcan: Crafting Compact Class-Specific Vision Transformers For Edge Intelligence
Authors: Ziteng Wei
Qiang He
Feifei Chen
Ranjie Duan
Xiaodan Li
Bin Li
YueFeng Chen
Hui Xue
Hai Jin
Yun Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Class-specific model derivation
Vision Transformer
structured pruning
edge intelligence
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We introduce Vulcan, a pruning-oriented post-training method that follows a novel train-then-prune paradigm to derive compact class-specific Vision Transformers (ViTs) from pre-trained models.

Abstract: Large Vision Transformers (ViTs) must often be compressed before they can be deployed on resource-constrained edge devices. 
However, many edge devices require only part of the *all-classes* knowledge of a pre-trained ViT in their corresponding application scenarios. This is overlooked by existing compression methods. Lightweight models produced by these methods retain a substantial amount of class-irrelevant knowledge and suffer suboptimal performance on target classes. To address this, we analyze the knowledge distribution of ViT and reveal a knowledge disentanglement within it: neurons in the feed-forward network (FFN) modules encode class-specific knowledge, while the multi-head attention (MHA) modules capture class-agnostic patterns. Building on this insight, we introduce Vulcan, a pruning-oriented post-training method for deriving compact class-specific models from a pre-trained ViT under given resource budgets. Vulcan follows a novel *train-then-prune* paradigm, which introduces redundancy into ViTs deliberately by collapsing FFN neurons onto those with the highest class-specific activations and by enforcing low-rankness in MHA weights. This design mitigates the irreversible knowledge loss of direct pruning, so that the post-trained model can be compressed into a compact one with negligible performance loss. Notably, the derived edge ViTs not only achieve significant reductions in size and computation but also even surpass the original ViTs in performance on specific classes. Comprehensive experiments with five base ViTs covering three representative visual tasks on four datasets demonstrate that Vulcan-derived ViTs outperform the base ViTs on class-specific tasks by up to 15.12\% in accuracy, with only 20\%–40\% of their sizes. Compared with state-of-the-art structured pruning methods, Vulcan improves class-specific accuracy by up to 13.92\%. Code is available at [Vulcan](https://github.com/CGCL-codes/Vulcan).

Published as a conference paper at ICLR 2026
VULCAN: CRAFTING COMPACT CLASS-SPECIRIC VI- SION TRANSFORMERS FOR EDGE INTELLIGENCE
Ziteng Wei', Qiang He Feifei Chen", Ranjie Xiaodann Li
Xiaodan
Bin Li Yuefeng Chen", Hui Xue", Hai Jin', Yun Yang*
National Engineering Research Center for Big Data Technology and System,
Services Computing Technology and System Lab, Cluster and Grid Computing Lab, Huazhong University of Science and Technology
2 Deakin University Alibaba Group
Swinburne University of Technology
(wei ziiininnn, hqiang,
ABSTRACT
Large Vision Transformers (ViTs) must often be compressed before they can be deployed on resource-constraine edge devices. However, many edge devices require only part of the all-classes knowledge of pre-trained ViT in their corre- sponding application scenarios. This is overlooked by existing compression meth- ods. Lightweight models produced by these methods retain a substantial amount of class-irrelevant knowledge and suffer suboptimal performance on target classes, To address this, we analyze the knowledge distribution of ViT and reveal a knowl- edge disentanglement within it: neurons in the ferd-forward network (FFN) mod- ules encode class- spasifsc knowledge, while the multi-head attention (MHA) mod- patters.. Building on this insight, introduce Vulcan, a pruning-oriented post-training method for deriving compact class-specific mod- els from pre-trained ViT under given resource budgets. Vulcan follows novel train-then-prumee paradigm, which introduces redundancy into ViTs deliberately by collapsing FFN neurons onto those with the highest class-specific activations and by enforcing low-rankness in MHA weights. This design mitigates the irre- versible knowledge loss of direct pruning, so that the post-trained model can be compressed into a compact one with negligible performance loss. Notably, the derived edge ViTs not only achieve significant reductions size and computation but also even surpass the original ViTs in performance on specific classes. Com- prehensive experiments with five base ViTs covering three representative visual tasks on four datasets demonstrate that Vulcan-derived ViTs outperform the base ViT' on class-specific tasks by up to 15.12% in accuracy, with only 20%-40%% of their sizes. Compared with state-of-the-are structured pruning methods, Vulcan improves class-specific accuracy by to 13.92%. Code is available at Vulcan.
ules capture
INTRODUCTION
Vision Transformers (ViTs) have achieved remarkable success in diverse visual tasks, including image recognition (Lu et al., 2025; Fixelle, 2025), object detection (Singh, 2023; Wang al., 2025a), and instance segmentation (Yang et al., 2022: Ravi et al., 2025). Recent advances largely come from scaling up ViTs, which enhances their representation and generalization ability (Wang al., 2024; 2025b; Han et al., 2025). While such scaling trends have pushed performance boundaries on various benchmarks, they inevitably result in ViTs that are computationally expensive and memory- intensive (Papa et al., 2024; Saha & Xu, 2025). As a result, these oversized ViTs are typically deployed on cloud servers with sufficient computing resources (Jiang al., 2025).
As illustrated in Figure 1, cloud deployment often fails to guarantee real-time performance, security, and reliability, while edge deployment can address these issues via local inference (Wen et al., 2023: Liao et al., 2024; Ding et al., 2024; Bonazzi et al., 2025). This highlights the urgent need to unlock the potential of VITs on edge devices such as drones and autonomous vehicles through model
"Corresponding Author.

---

Published as a conference paper at ICLR 2026
Cloud Deployment
Edge Deployment
Clase-Spacifcc Edge VITe
Rase VIT
Birds
Flowere
Vahicine
Data
Procoasing
Derivation
Model Distribution
13 High latency
Privacy laakage
Figure 1:
Comparison between cloud and edge ViTs deployment. 1) Cloud Deployment: users access models via cloud APIs, suffering from high latency, privacy risks, and poor reliability due to communication and network dependence. 2) Edge Deployment: local inference reduces latency, preserves privacy, and improves reliability. Class-specific models are provided for users.
compression (Tuli & Jha, 2023; Ye et al., 2024). However, existing compression methods ignore the fact that edge ViT', i.e., ViTs deployed edge devices, typically require only class-specific knowledge in their own application scenarios, rather than the all-classes knowledge embedded in large-scale pre-trained ViTs (Yao & Abdelzaher, 2023; Zhuang al., 2024). For example, an in- vehicle sensor tasked with recognizing traffic-related classes such as vehicles, street signs, and traffic lights does not require knowledge about flowers or insects. The presence of irrelevant knowledge distracts a model from focusing on target classes, leading to suboptimal performance. This raises a key question: how to derive compact class-specific edge VITs from a generul-purpose pre-trained base VIT for lightweight deployment? (Its necessity is discussed in Appendix A.)
To address this question, we adopt structured pruning (Cheng et al., 2024), an edge-friendly tech- nique ($2), for lightweight deployment. However, existing pruning methods (Zhang et al., 2024; Sun et al., 2025) lack pruning strategies tailored to class specificity. Simply replacing calibration datasets with class-specific data during pruning and retraining is insufficient, as it results in models that still fail to focus on target classes. Moreover, these methods follow the conventional prune- prune-then- train paradigm, which often incurs irreversible knowledge loss, particularly high pruning rates, since pruned weights may be unimportant but not dispensable. More fundamentally. achieving class- specific model derivation requires an understanding of how class-specific knowledge is distributed across ViT modules-4 a question that remains largely unresolved, as existing interpretability studies offer only limited insights (Choi et al., 2024: Li et al., 2025). We investigated the knowledge distribution within ViTs (Geva et al., 2021; Dai et al., 2022) and found a disentangled distribution: FFNs primarily encode interpretable class cass--pe knowledge, while MHAs capture class- agnsoagi patterns. Building on this insight, this paper presents Vulcan, a pruning- pruning-oriented oriented post-training method that can derive compact spaeiic ViTs from a pra-trained VIT. Unlike conventional pruning methods, Vulcan follows novel train-then-prutt paradigm that ensures near-lossless pruning after post-training and minimizes knowledge loss during model com- pression. Specifically, Vulcan employs class-centric neuron collapse to aggregate FFN neurons onto those with the highest activations, deliberately introducing redundancy while enabling class- relevant neurons to dominate feature extraction. Meanwhile, Vulcan applies truncated nuclear-nom regularization to enhance the low- low-rankness of projection matrices in MHA, enabling near-lossless pruning via singular value decomposition (SVD). Given a specified resource budget (e.g.. #Param, GFLOPs), Vulcan integrates these two strategies under an augmented Lagrangian framework (Birgin & Martínez, 2014) derive edge VITs. We summarize the key contributions of Vulcan follows:
We provide fundamental insights into the disentangled knowledge distribution of ViTs, where FFNs encode class knowledge while MHAs capture alass-agnostic patterns.
To the best of our knowledge, Vulcan is the first technique for deriving compact class-specific ViTs. As a pruning-uricon oriented post-training method, is also the first to introduce the novel train- then-prune compression paradigm, which minimizes knowledge loss during pruning.
2

---

Published as a conference paper at ICLR 2026
Class Contric Neuron Collapse (3.21
Base VIT (3.11
Praned
Caloraate Schottr boad oniciiation
W ar
(eM
KM
We ghi baued Clamtrring
Network3TM
W. gesd
Layer Horm
Post Training and Pruning (3A)
Interprebailiity Analyses (3.2, 3.3)
Truncated Nuclear Norm Regelar riation [B.3)
Ad
Calodate Fruving
Low Rani Strechue
Vuican
Matt-Wedd (Mi4))
LayerNorn
Desempaation (oro)
(WPA 24-1
(W7? g0r4)
With
gghnn
kw
TraskrrerBBa Bock
Figure 2: Overview of Vulcan. 1) Class-Centris Neuron Collapse (CCNC): neurons in FFN modules clustered, and all neurons within a cluster collapse into the one with the highest activation for the target classes. 2) Truncated Nuclear Norm Regularization (TNNR): low-rank structures are introduced into matrices in MHA modules to support near-lossless SVD-based compression.
Extensive experiments with five base ViTs covering three typical visual tasks and four bench- marks demonstrate that Vulcan-derived edge ViTs achieve significant reductions in model size while outperforming both the base ViT's and models derived by state-of-the-ar structured prun- ing methods in class-specife performance.
BACKGROUND AND RELATED WORK
Edge Model Deployment. Recently, a series of increasingly large ViTs have been developed (Zhai al., 2022; Dehghani et al., 2023; Wang al., 2024), contrasting with the growing demand for deploying models on edge devices for real-time responsiveness, privacy preservation, and reliable service. To address this, several edge-friendly architectures have been proposed, such as Mobile- ViT (Mehta & Rastegari, 2022), EfficientViT (Liu et al., 2023), and Flatten Transformer (Han et al., 2023). While effective, these approaches rely on manual architecture design and require training from scratch. Moreover, such specialized architectures do not naturally scale with the rapid advances of large ViTs, limiting their ability to deliver increasingly powerful models for edge deployment. In contrast, Vulcan focuses on deriving compact ViTs from pre-trained base ViT, allowing edge ViTs to inherit knowledge from the base ViT and enabling efficient model development.
Model Compression. Various compression methods enable edge deployment of ViTs, including quantization (Choi & Kim, 2025; Zhong et al., 2025), knowledge distillation (Yang et al., 2024b; Cao et al., 2025), as well as structured (Zhang et al., 2024; Sun et al., 2025) and unstructured prun- ing (Chen et al., 2021; Liao et al., 2023). However, not all are edge-friendly (Appendix B). Quan- tization and unstructured pruning typically rely specialized infrastructure for acceleration, which limits their applicability on diverse edge devices (Yang et al., 2024a; Cheng et al., 2024). Knowledge distillation transfers knowledge only from the feature space, incurring substantial training cost (He et al., 2025). In contrast, structured pruning extracts knowledge from the parameter space and yields regularly shaped models that can deployed across diverse edge devices, which serves as the foundation of Vulcan. Different from the conventional train-then-prune paradigm that avoids irreversible knowledge loss caused by direct weight removal and achieves near-lossless pruning. is worth noting that Vulcan is orthogonal to other compression techniques and can be scamlessly combined with them for even lighter deployment (Appendix J.9).
seine-then-train paradigm, Vulcan follows a
Interpretability of VITs. Extensive research has investigated the interpretability of ViTs from dif- ferent perspectives, such as attention visualization (Chefer et al., 2021; Choi et al., 2024), decision pathways (Komorowski al., 2023; Brinkmann al., 2024), and robustness to perturbations (Fu et al., 2022; Hu et al., 2024). However, the distribution class-specific knowledge across different
3

---

Published as a conference paper at ICLR 2026
(1,1800) (a) mi
(3.790) (b) n1
(c) mi
(5,2801)
(4,14 (d) 11
(9,2154) (c) n4
(f) m
(11,1644)
Figure 3: Top-25 activated images for random FFN neurons in DeiT-Base. (a)-(f) correspond to pat- terns of cold color tones, vertical stripes, monotone backgrounds, grid textures, stacked objects, and
snakes, illustrating the strong interpretabillty of FFN neurons. See Appendix C for more examples.
modules of ViTs remains largely unexplored. This missing insight is particularly critical for sup- porting class-specific model derivation, which essential to meet the eustomized requirements of edge users. In this paper, we shed light on this issue by providing an analysis of how class-specine and class-agnostic knowledge are structurally disentangled within ViTs ($3.2-$3.3).
VULCAN: CLASS-SPECIRE MODEL DERIVATION
This section introduces Vulcan, a pruning-oriented post-training method for class-specific model derivation. We begin with notations and preliminaries ($3.1), then detail two key components: class- centric neuron collapse for FFNs ($3.2) and truncated nuclear norm regularization for MHAs ($3.3). Finally, we introduce the post-training procedure and the pruning strategy for constructing compact models from the post-trained base ViT (83.4), An overview of Vulcan is illustrated in Figure
3.1 NOTATIONS AND PRELIMINARIES
Vision Transformer. A VIT (Dosovitskiy, 2021) consists of stack of Transformer blocks, each containing an MHA and FFN module. Given patch token sequence X E R. xd with tokens and embedding dimension d, each MHA head computes and aggregates contextualized representations:
Hi
Hi
XW-) T
W (1.h) X
XW T W
MHA (1) (X)
Σ Attn (1.b) (X)
Σ
softmax
(1)
h=1
h=1
where W Ruxd xd w.f.)) E Rooxd and W...). E Rd are the query, key, value, and out- put projection matrices for the h-th head in the I-th block. For simplicity, bias terms are omitted. Following MHA, the FFN module applies a two-layer multi-layer perceptron:
(FN (X) (XW(
)W!"
(T)
Σ
er
(XW
(1 (t) s w
[d T =
o
(1.)) (1.0)) @ n2
(2)
where W R and W Rdxe are the FFN projection matrices in I-th block, a(-) is nonlinear activation (e.g., GELU), nlf2 Rd is the i-th neuron in WiO. and @ is the outer product.
Sub-Task. Let MB be the pre-trained base ViT trained on Dy for classes y = {y....., yyl }- Vulcan aims to derive compact class-specific edge ViT ME from MB. specialized for subset S V. We refer to S as sub-task of MM, and denote its corresponding dataset by Ds.
3.2 CLASS-CENTRIC NEURON COLLAPSE FOR FFNS
Insight. As shown in Eq. (2), an FFN computes via weighted aggregation, where the activation of each neuron nef(.)) serves as weights and n((++) as values. Thus, activation magnitudes decisively de- termine neuron contributions FFN outputs. Inspired by the concept of "knowlldgge neurons" (Geva et al., 2021; Dai et al., 2022), we conduct an activation- 2021). Specifically, for each neuron re(4.i), we sum its activations over all patch tokens for each image in the ImageNet- validation set (Russakovsky et al., 2015), and visualize the top-25 images with the highest activations to inspect what kind of knowledge encoded in the neuron. Figure shows that these neurons exhibit remarkably strong human-recognin coming more semantic in deeper blocks. For example, shallow neurons capture simple patterns such color tones, textures, backgrounds, while deeper neurons specialize in semantic concepts such as specific classes like snakes. This suggests that FFN modules serve as reservoirs of class-specific knowledge, since different classes can be distinguished by shallow and semantic patterns.
analysis on DeiT-Base (Touvron et al.,
interpretability, with knowledge be-
4

---

Published as a conference paper at ICLR 2026
Query-tay Dimerstos 1003 E/l
Value-Getput Bimendinn
vorea ouh
TAXSR HEAA
0000 shocove
SAMpENt
282
110
91
012
Figure 4:
Comparison of accuracy between SVD-based and score-based pruning for DeiT- Base on ImageNet-IK. (2)) represents sub- task Tj (TTj==N) with pruning rate R. SVD out- performs other methods by significant margin.
Figure 5: Effective rank (Frank) distribution of QK and VO dimensions across different blocks of DeiT-Base. The QK/VO Erank represents the sum of the Erank values across all heads in these two dimensions for each block.
Class-Centre Neuron Collapse (CCNC). Building on this insight, we propose CCNC to introduce redundancy into the FFN intermediate dimension (e) for pruning. Given a sub-task S, Vulcan use Dss Higher activation implies stronger relevance
to compute the activations (4.3) = Σ, (X(j)-n((.)) of n((1.) to target classes in S. To guide the model to focus on class-specific knowledge, Vulcan performs k-means clustering on n((x.)) in each block based on their weights, and collapses all neurons within the same cluster C( into the neuron with the highest activation at") termed the anchor neuron nl As Figure 3 shows that neurons in different blocks specialize in distinct functions, Vulcan avoids uniform pruning rates across all blocks (Appendix J.2). Instead, it adaptively determines the number of clusters per block based on activation distribution and overall pruning rate R e (0, 1). Formally, the number of clusters in the FFN of the I-th block K(O) is calculated as follows:
KD) Σ (1.))) > (A 12 ()) E
R1))
A = Utaliili -I e 1=1
j=1
where ((A, k) returns the k-th smallest element in set A. To realize cluster-level collapse, Vulcan enforces the weights or activations of all neurons within the same cluster contract toward the anchor neuron. This is achieved by introducing a collapse regularization term during post-training:
K
Coollapse Σ
10
(1.)
D =
(112) 1" arg max O k
(L.)
(4)
xelle("1
where ctO) is the k-tth cluster in the (-th block, and v/(.0) denotes either the weight nf.0) or the activation afl-3) of the i-th neuron in C(, with a unique choice applied in practice. Experiments show that weight collapse converges faster and performs better than activation collapse (Figure 10). Through Eq. (4), Vulcan guides the model to focus on class-specific knowledge and enables each cluster to be represented by single anchor neuron, which achieves effective compression of the FFN modules. Appendix D further shows that CCNC can also be applied to other FFN architectures.
3.3 TRUNCATED NUCLEAR NORM REGULARIZATION FOR MHAS
Insight. For MHA, Vulcan prunes the query-key (QK: g) and value- dimensions (VO: v), en- forcing identical sizes across heads in each block (Appendix E), As shown in Eq. (1), the QK and VO dimensions are intermediate dimensions of two matrix multiplications (i.e., W. WK. W., W E Rdxd), indicating that they can be pruned through singular value decomposition (SVD). Surpris- ingly, we observe that this data-free SVD-based method consistently outperforms data- inta-indeeenddnt score-based pruning methods (Yu et al., 2018: Chen et al., 2021; Yang et al., 2023), as shown in Figure 4. It suggests that effective compression of MHA can be achieved even when the sub-task is unknown. This method-driven analysis indicates that the QK and VO dimensions in MHA store alass-agnostic knowledge. We provide theoretical explanation for this observation in Appendix F.
Truncated Nuclear Norm Regularization (TNNR). To leverage SVD for near-lossless MHA prun- ing, we propose TNNR to introduce low-rank structures into W!I.A) and Wil.*), with their low-rank properties shared by W(A and Wi(.)), respectively. We use effective rank (Roy & Vetterli, 2007) to measure the knowledge capacity of each attention head and observe that the knowledge in MHA is also unevenly distributed across different blocks and dimensions, as shown in Figure 5. Based on this observation, Vulcan adaptively determines the pruning rate Rill R2Kivo for the QK and VO
5

---

Published as a conference paper at ICLR 2026
dimensions according to the dimension-level effective ranks and the overall pruning rate R:
2r (1)
RQKIVO
(
VOOOK R [1]
R
L
R
(5)
(1)
E 1/(rQK
10)
+ r11o)
'QK + (1)
[1]
QQK
+ ''vo
'vo
H
! QKIVO
Σ
E(W
@kIvo), E(W) exp(- Σ pr log
Pe Σ, 0g
(6)
where W = -W
W = W.I.. Wi E(W is the effective rank of matrix
W and o, the i-th singular value of W. Then, Vulcan applies truncation to the nuclear norm of W and Withh) according to Eq. (5) and constructs regularization term:
H
Lrakk ΣΣ 1=
(.h.s) Q
. (u(1-RQk)] (mm = (7) (Lh.)) Σ ov
i=
where obi1") is the i-th singular value of W With Eq. (7), Vulcan effectively extracts class-
agnostic knowledge from the MHA and supports near-lossless pruning after post-training.
3.4 TRST-TRAINING AND PRUNING
Objective. Vulcan post-training aims to focus the model on specific classes while introducing re- dundancy in the three ViT dimensions (e, q. U) for pruning. Eqs. (4) and (7) serve as redundancy constraints that guide the compression process. To strictly enforce these constraints during post- training. Vulcan uses the augmented Lagrangian framework to construct the final objective function. Specifically, it extends Eqs. (4) and (7) into linear and quadratic functions, and introduces two learn- able Lagrange multipliers d and X2 to control convergence. The final loss function is as follows:
(Aale - -101 - )2)
L -
1=1
(8)
(Nio (liiit) + Noo0) + Σ (Nrk + 220 (..01)))
=1
imqf+1
imof+1
where LT is the loss term for visual tasks, well a and X2 are initialized to zero and updated using gradient ascent with penalty parameter p as the learning rate.
Pruning. After post-training. Vulcan can directly derive a compact class-specific edge VIT from the post-trained base ViT with negligible performance loss (Section 3.5). Specifically, the pruned FFN module contains two new weights, Wi" e R xd and W30" Rdd which satisfy:
WhTh [k] =
W
[.
Σ 1=1
A EE (KO)
where (1.i) denotes the neuron in WiO corresponding to (L.)) This process is essentially the inverse of network expansion (Ding al., 2023; Yao al., 2024) and adheres to the function- preserving principle (Chen et al., 2016). For the pruned MHA module, its four new weights, WiLN) E l Witk) E Reix and WiLKY E Raxef can be directly obtained through SVD:
W(LAy' (U(L)) : of (Lh)
T Val/a.
(V-(L)) W (V [,, T
(10)
Wilhh (U l of W o/.: wf)T wilkay ((( :f
(11)
where 1 = and DV WVO). It is worth noting that con- ventional pruning methods remove weights based importance scores, which may discard useful knowledge. In contrast, Vulcan ensures that the pruned weights are entirely redundant and contain valuable knowledge. Pseudocode and notation tables provided in Appendix K and I..

---

Published as a conference paper at ICLR 2026
3.5 PROOF OF LOSSLESS PERFORMANCE AFTER PRUNING
As discussed in Section 3.4, once the constraints in Eq. (4) and Eq. (7) are fully satisfied, i.e., Loollapse = and Lrank = O, Vulcan ensures lossless pruning for the post-trained model. In this section, we provide a formal proof of this property.
ProofI: Lossless Performance after Pruning. After post-training, the base VIT satisfies the fol- lowing two key conditions:
1. All neurons within the same cluster CDO in the FFN module in I-th block collapse the anchor neuron ng of that cluster.
2. For the h-th attention head in l-th block, the matrices WgS.A) and W contain exactly of and of non-zero singular values, respectively. This means rank (W/g.))) S of and rank rank(W (W: S e/-
FFN Case. According to Eq. (2), the output of the FFN in the 1-th block can be expressed as:
KO
1
FFN (X)=
(Xn)
(110)
n2
(1.
Σ
oXnn
(Xn
(1.))
nk.2
(1.)
(12)
K00 (c
K
1c
Σ
Σ
a (Xing
(4)
222
(1.)
Σ
XX
(1)
k
(
where nlf denotes the neuron in WiS corresponding to (4.)) the pruned FFN in the 1-th block can be expressed
m
According Eq. (9), the output of
ej
K
(FN pruned
X)=
=
(XW
&
[M]]
Σ
xxW
1 [k)) & [;
k=1
1=1
(13)
K
(Xng
(1)
T2
(LJ) 22,
FFN (X)
MHA Case, According to the second key condition, we can conclude that: rank(W((.))) w(l.A)) S min((ank(W<(())) Q rank(W (w())) s di
(14)
rank(w w S (in((ran((:(((h))) rank( (W(l))) S of Therefore, the pruning process based SVD is lossless, i.e., the pruned matrices WILLY
Vik satisfy:
w-l.hyy T w (-((kh) W-(.)) w(("")"" w-llhy = -w(l.)) w(()))
Q Substituting these equalities into the Eq. (1), we obtain:
(15)
Xw (LA))
MHA (1)
x
softmax
XW T W(LAy
pruned
V
h=1
(16)
H.
XW
(.N.) K
XW
w-(lh) MHA (
softmax
V
Thus, both the FFN and MHA modules of the pruned model produce identical outputs to those of the post-trained base ViT, which completes the proof.
In practice, since Loollapse and Lrank only asymptotically approach zero, model accuracy before and after pruning exhibits minimal fluctuations, indicating that Vulcan achieves near-lossless pruning.
4 EXPERIMENTS AND ANALYSIS
Models and Datasets. We evaluate Vulcan on two widely adopted VIT families: DeiT- Base/Small/Tiny (Touvron et al., 2021) and Fast/Mask R-CNN (Swin-T) (Liu et al., 2021). For recognition tasks, we use ImageNet-1K (Russakovsky et al., 2015) and CIFAR-100/10 (Krizhevsky

---

Published as a conference paper at ICLR 2026
Table 1: Overall performance of Vulcan and baselines on DeiT-Base with ImageNet-IK sub-tasks of different sizes (25, 50, 100 classes) under pruning rates of 0.60 and 0.80. The best top-1 accuracy (%) in each column is highlighted in bold, while the second best is underlined. '(FT)' denotes fine-tuning on the class-specific data. Results for other pruning rates are provided in Appendix J.
Method
T1/25
12/25
13/25
Ang
14/50
T5/50
T6/50
Avg
17/100
TW/10
T9/10
Avg
DefT-Base
79.92
82.56
80.67
81.05
80.48
79.64
84.96
81.69
80.49
84.12
78.58
81.06
DeiT-Base (FT)
96.40
96.96
96.77
93.36
94.35
91.56
93.29
92.56
Pruning Rate 0.60
Random
91.76
92.72
89.69
91.39
86,16
85.60
87.56
86.44
83.11
86. 10
80.42
83.21
NVIT
83.91
87.12
85.62
85.55
81.64
81.20
80.72
81.19
78.31
80.31
78.12
78.91
X-Pruner
91.68
93.04
93.21
02.64
90.24
88.28
90.16
89.56
84.51
87.12
86.46
86.03
DC-NTT
71.60
83,18
76.92
77:27
69.48
67.20
67.88
60.06
64.98
65. 18
MDP
91.04
94.80
92.09
92.64
90.32
85.00
91.16
88.83
82.61
84.33
82.86
83,27
Vulcan
95.04
96.24
95.53
95.60
92.16
91.72
92.44
92.11
88.03
90.70
89.02
89.25
Pruning Rate 0.80
Random
71.36
78.48
82,90
77.58
71.92
71.16
73.28
72.12
65.28
67.71
60.98
64.66
NVII
64.16
58.00
60.38
60.85
52.76
53.44
49.04
51.75
41.56
47.62
48.74
45.97
X-Pruner
85.60
86.00
85.62
74
76,24
74.00
79.72
76.65
72.62
76.74
75.52
74.95
DC-VIT
54.72
66.24
64,46
61.81
47.52
53.40
50.84
50.59
35. 36.66
35.12
34.66
35.15
MDP
83.68
88.08
84.66
85.47
74.04
74.20
79.88
76.04
61.72
67.81
67.54
65.69
Vulcmn
92.24
94.32
92.57
93,04
88.36
88.
88.52
88.33
81.82
85.20
82.84
83.29
al., 2009); for detection and segmentation tasks, we use COCO (Lin al., 2014). Sub-tasks of varying scales are constructed by randomly sampling classes from each dataset.
Implementations. GFLOPs (Appendix G) is used as the metric to compute pruning rates. During post-training, we set the batch size to 256, the learning rate to 10 -4, the penalty parameter to 1.0, and use the AdamW for optimization. For CCNC, we apply z-score normalization to the activation values across blocks before computing K(O,, Anchor neurons are updated per batch. To ensure that the derived models can achieve accelerated inference on wide range of edge devices, Vulcan aligns the pruned dimensions to multiples of 8 (NVIDIA, 2023). See Appendix H for more details.
Baselines. We compare Vulcan with five state-of-the-art structured pruning methods: Random Prun- ing (Gadhikar et al., 2023), NVIT (Yang et al., 2023), X-Pruner (Yu & Xiang, 2023), DC-ViT (Zhang et al., 2024), and MDP (Sun et al., 2025) (Appendix I). To ensure fairness, we retrain the models pruned by these methods until they converge.
4.1 RESULTS
Class-Agnostic vs. Class-Specificc. We compare class-agnostic and class-specift methods to high- light the advantages of the latter. Baselines derive class-agnostic models from DeiT-BBsee for all Im- ageNet classes, while Vulcan derives class-specific models tailored to the target sub-task S. Figure shows that Vulcan consistently outperforms base- lines in class-specific accuracy at the same prun- ing rate. This demonstrates Vulcan's 's advantage in meeting edge devices' customized needs.
Pruning Rate 0.20
Pruning Rabe n 0.40
Figure 6: Comparison between class-agnostic and class-specific model derivation. Tj/N rep- resents sub-task Tj with N random classes
Overall Performance. For fair comparison, we replace the calibration datasets originally used by the baselines with the sub-task dataset Ds, which adapts them to the setting of class-specific model derivation. As shown in Table 1, Vulcan-derived edge ViTs achieve up to 15.12% higher accuracy than the base ViT and up to 13.92% over the models derived by state-of-the-art baselines across different sub-tasks and pruning rates. At pruning rate of 0.60, Vulcan improves accuracy by 11.05% over the base ViT, and surpasses the two best-performing baselines, X-Pruner and MDP, by 2.91% % and 4.07%, respectively. Even at high pruning rate of 0.80, where all methods suffer from significant accuracy degradation, Vulcan still delivers strong performance, yielding improvements of 6.95%, 9.11 1%, and 12.49% over the base ViT, X-Pruner, and MDP, respectively. It can be seen that the advantages of Vulcan become more pronounced as the pruning rate increases and the sub-task size grows. Meanwhile, Vulcan able to retain 97.63% and 03.30% of the accuracy of the fine- tuned base ViT at pruning rates of 0.60 and 0.80, respectively. These results demonstrate Vulcan's strong capability to specialize models for target classes.
8

---

Published as a conference paper at ICLR 2026
CIFARIDO: 1:40 1533 TA33
CIFAR10: 710
COCO:
(OORSTAAR (94)
ICC OIE
S
DuttTey
Inildeat
Part EVM (eviso))
Mart Swetel
Praning Rate 199
13
Pruning Race 1861
Pruning Falt (94)
Figure 7: Performance of edge ViT' derived by Vulcan across different base models and datasets.
Table 2: Comparison of computational efficiency between DeiT-Base and edge ViTs derived by
Vulcan under different R, evaluated on Jetson Orin NX (bz=1) and NVIDIA RTX 4090 (bz=256).
Latency (ms)
Throughput (Imagels)
Memory (GB)
#Param
#FLOPs
Methods
Orin NX
RTX 4090
Orin NX
RTX 4090
Orin NX
RIX 4090
(M)
(G)
DefT-Base
45.45
274.27
23.00
933,39
0.34
2.21
86.57
17.57
Vulcan (0.20)
16 (1.23x)
27.14
1173.43
2.14
67.22
13.56 (121,51%)
Vulcan (0.40)
176.01 (1.53x)
(1.56x)
33.63
1454.42
0.21
1.82
51.00
10.24 (14).
Vulcan (0.60)
21.81 (2.16x)
136.67 (2.01
45,86
1873.11
0.15
1,66
34.09
6.77 (161.47%)
Vulcan (0.80)
15.06 (3.02x)
99.59 (2.75x)
66.41
2570.61
0.08
1.40
16.96
3.26 (181 45%%)
Generality Across Models and Datasets. To further validate the generality of Vulcan, we extend our evaluation to different base models (DeiT-SmalllTiny and Fast/Mask R-CNN with Swin-T back- bone) and datasets CLFFAR-100/10 and COCO2017). As shown in Figure 7, for recognition tasks, the derived edge ViT's consistently outperform the base ViT on sub-tasks when the pruning rate is below 0.60. Even pruning rates higher than 0.80, the accuracy of edge ViTs does not degrade severely enough to make the models unusable. For some small-scale sub-tasks, edge ViTs even significantly outperform the base ViT at pruning rates as high as 0.90. For detection and segmenta- tion tasks, Vulcan also effectively extracts class-specific (i.e., person, car, and cat) knowledge from Swin Transformer backbones and yields edge ViTs that rival the performance of the base ViT. See Appendix 1.7.77 for experiments on generalization, robustness, and wall-clock cost.
Computational Efficiency. We evaluate the efficiency of edge ViTs derived by Vulcan on both a representative edge device, Jetson Orin NX, and an NVIDIA RTX 4090. Considering that requests typically arrive sequentially in real-time edge scenarios, whereas servers process requests in batches, we use batch sizes of and 256 to measure the speedup on Orin NX and RTX 4090, respectively. All results are averaged over the models for nine sub-tasks listed in Table As shown in Table 2. when the pruning rate ranges from 0.20 to 0.80, Vulcan-derived models achieve 1.23 x33.02 speedup on Orin NX and 1.28 1.26 x -2.55 x on RTX 4090, while reducing memory consumption by 20.59%-76.477 and 3.17%-36.66 36.65% ,, respectively. See Appendix J.3 for comparisons with baselines.
4.2 ANALYSIS
We further analyze the effectiveness of Vulcan from multiple perspectives. Unless otherwise speci- fied, all experiments are conducted on DeiT-Base at pruning rate of 0.60, with the sub-tasks TI-T3 in Table constructed by sampling classes from ImageNet-IK.
Understanding the tracking 1) the accuracy of the derived edge ViT (Acc-Pruned) and the base ViT (Acc-Base) as well as their accuracy gap (AAcc), and 2) the overall loss L in Eq. (8). As shown in Figure 8, the post- training process can be viewed as a gradual alignment between the edge ViT and the base ViT. At the early stage, the task loss Lr dominates, leading to an improvement in the accuracy of the base VIT. As the Lagrange multipliers λ1 increase, the begin to dominate. Consequently, the accuracy of the base ViT slightly decreases while that of the edge VIT increases substantially. In the later stage, the two models converge, achieving equivalence.
Process. We visualize the post-training process of Vulcan by
-effrrcin losses Leollapse and Lrank
Penalty Parameter p. As discussed in Section 3.4, the penalty parameter p. as the only hyperpa- rameter of Vulcan, serves the learning rate for updating 21,2. It determines when Leollapse and Crank dominate post-training and align edge ViTs with base ViT. However, Vulcan is largely insen- sitive p. As shown in Figure 9, when p is set to 0.1, 1.0, or 10.0, both the convergence speed and the final accuracy of the edge ViT remain almost unchanged, which demonstrates the robustness of Vulcan. We further analyze the effects of batch size and learning rate in Appendix J.8.
Weight vs. Activation Collapse. As discussed in Section 3.2, the term v((.)) in Coollopce can be defined in two ways: using the weight vector ((..)) (weight collapxe) or the activation value (1.b)

---

Published as a conference paper at ICLR 2026
All Rone pol
110
2018
332
200
282
800
500
300
120
120g
280
200
Figure 8: Mean accuracy and loss curves with standard deviation bands on sub- tasks T1/25, T2/25, and T3/25 when deriving models from DeiT-Base using Vulcan at pruning rates 0.60 and 0.80.
Figure 9: Vul-
can is insensitive to the choice of p.
Cal.TT3
Table 3: Ablation study of Vulcan. At prun-
A.Col/C7 (T3)
ing rate of 0.60, the edge ViTs derived from DeiT- Base for classes in sub-tasks TI-T3 exhibit sig- nificant accuracy degradation when any of CCNC, TNNR, or anchor neurons are removed.
Ce ITT)
Cal.l (T2)
W Col. (T31
500
1000
Setting
T1/25
T2/25
T3/25
Arg
1500
Stop
Vulcan
95.04
2002
Figure 10: Accuracy trajectories of derived edge ViTs during post-training with weight collapse (W Coll.) and activation collapse (A Coll.).
w/o CCNC
8.08
13.68
12.30
11.35 (184.25%)
w/o TNNR
79.36
81.60
77.38
79:45 (116.115%
w/o anchor
90.00
92.32
90.81
91.04 (14.566%
(activation collapse). As shown in Figure 10, weight collapse achieves both faster convergence and higher final accuracy compared activation collapse. We attribute this two main reasons: 1) weight collapse directly enforces equality among weights, which better aligns with the pruning process of Vulcan: and 2) the Coollapse under activation collapse is smaller in magnitude, resulting in weaker constraints. Moreover, we observe that simply increasing P does not alleviate this issue.
Ablation Study. Class-centric neuron collapse (CCNC) and truncated nuclear norm regularization (TNNR) are Vulcan's 's core components, responsible for extracting class-relevant knowledge from the FFN and MHA modules, respectively. Table 3 summarizes their effectiveness. Removing CCNC causes an 84.25% accuracy drop due to irreversible knowledge loss in the FFN. Without TNNR, accuracy drops 16. 15%, smaller since MHA matrices are inherently low-rank. We further compare collapsing neurons toward the anchor neuron versus random neuron within each cluster, and find that the absence of anchor guidance leads to an average accuracy drop of 4.56%. These results demonstrate that all designs in Vulcan are indispensable for effective class-specific model derivation.
Visualization. To further show that the models derived by Vulcan indeed specialize in the target classes, we de- rive an edge ViT MMg/dos from DeiT-Base at a pruning rate of 0.60 for the Stanford Dogs (Khosla et al., 2011), subset of ImageNet consisting of 120 dog classes. Fol- lowing the same procedure as Figure 3, we visualize the 680 neurons in the last block of Mg/dogs and iden- tify 229 neurons specialized in recognizing dog-related classes. Representative examples are shown in Figure 11. This demonstrates Vulcan's ability to effectively guide the model toward target classes.
Figure 11: Dog-related neurons in the
last block of Vulcan-derived Mgjdogs-
5 CONCLUSION
This paper presented Vulcan, a novel pruming oriented post-training method that derives class- specific models from pre-trained ViTs for deployment on edge devices. Motivated by the insight that the FFN modules of VITs primarily encode class-specific knowledge while the MHA modules cap- ture class-agnostic patterns, Vulean adopts a novel train-then-prume compression paradigm, lever- aging class-centric neuron collapse and truncated nuclear norm regularization introduce redun- dancy into the FFN and MHA modules, respectively. This design allows Vulcan to derive compact class-specific edge ViTs from the post-trained base ViTs under resource constraints with negligible performance loss. Extensive experiments demonstrate its effectiveness, generality, and robustness.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We sincerely thank the AC and reviewers for their constructive and valuable feedback. This research was supported by the National Key R&D Program of China under Grant No. 2023YFB4502400. is also supported by the Alibaba Group.
REFERENCES
Ernesto G Birgin and José Mario Martínez. Practical augmented Lagrangian methods for con- strained optimization. SIAM, 2014. 2
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of
neural network pruning? Proceedings of Machine Learning and Systems, 2:129-144 146, 2020. 21
Pietro Bonazzi, Christian Vogt, Michael Jost, Lyes Khacef, Federico Paredes-Valle and Michele Magno. Towards low-latency event-based obstacle avoidance on fpga-drone. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938-4946, 2025.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mini-ig discriminative com- ponents with random forests. In European Conference Computer Vision, pp. 446-461, 2014. 29
Jannik Brinkmann, Abhay Sheshadri, Victor Levoso, Paul Swoboda, and Christian Bartelt. mech- anistic analysis of transformer trained on symbolic multi-step reasoning task. Association for Computational Linguistics, 2024.
Jiajun Cao, Yuan Zhang, Tao Huang, Ming Lu, Qizhe Zhang, Ruichuan An, Ningning Ma, and Shanghang Zhang. Move-kd: Knowledge distillation for vlms with mixture of visual encoders. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19846-19856, 2025.
Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 782-791, 2021.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988 2021. 3.5
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. International Conference on Learning Representations, 2016. 6
Hongrong Cheng, Miao Zhang, and Javen Qinfeng Shi. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 3,3
John Chipman. "proofs" and proofs of the eckart-young theorem. Stochastic processes and functional analysis, pp. 71-83. CRC Press, 2020. 22
Dahun Choi and Hyun Kim. GradQ-ViT: Robust and efficient gradient quantization for vision transformers. In AAAI Conference on Artificial Intelligence, volume 39, pp. 16019-16027, 2025.
Hoyoung Choi, Seungwan Jin, and Kyungsik Han. Icev2: Interpretability, comprehensiveness, and explainability vision transformer. International Journal of Computer Vision, pp. 1-18, 2024. 3
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In 60th Annual Meeting of the Association for Computational Linguisties, pp. 8493-8502, 2022. 2,
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner. Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International conference on machine learning, pp. 7480-7512, PMLR, 2023. 3

---

Published as a conference paper at ICLR 2026
Ning Ding, Yehui Tang, Kai Han, Chao Xu, and Yunhe Wang. Network expansion for practical training acceleration. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20269-20211 2023. 6
Yucheng Ding, Chaoyue Niu, Fan Wu, Shaojie Tang, Chengfei Lyu, and Guihai Chen. Enhancing on-device Ilm inference with historical cloud-based Ilm interactions. In 30th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 597-608, 2024.
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. 9rd International Conference on Learning Representations (ICLR 2021), 1) 2021. 4
Joshua Fixelle. Hypergraph vision transformers: Images are more than nodes, more than edges. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9751-9761, 2025.
Yonggan Fu, Shunyao Zhang, Shang Wu, Cheng Wan, and Yingyan Lin. Patch-fool: Are vision transformers always robust against adversarial perturbations? International Conference on Learn- ing Representations, 2022. 3
Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz. Why random pruning is all need to start sparse. In International Conference on Machine Learning, pp. 10542-10570. PMLR, 2023. 8, 17
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021. 2,
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. Flatten transformer: Vision transformer using focused linear attention. In IEEE/CVF International Conference on Computer Vision, pp. 5961-5977 2023.
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan. Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15733-15744, 2025.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for
efficient neural network. Advances Neural Information Processing Systems, 28, 2015. 16
Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong, Haotong Qin, and Xianglong Liu. DA-KD: Difficulty-wware knowledge distillation for efficient large language models. In Forty-second In- ternational Conference on Machine Learning, 2025. 3
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In IEEE/CVF International Conference Com- puter Vision, pp. 8340-8349, 2021. 29
Geoffrey Hinton. Distilling the knowledge in neural network. arXiv preprint arXiv: 150.00533 2015. 17
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. 9rd International Conference on Learning Representations (ICLR 2021), 2021. 31
Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Improving interpretation faithfulness for vision transformers. In International Conference on Machine Learning, 2024.
Linyi Jiang, Silvery D Fu, Yifei Zhu, and Bo Li. Janus: Collaborative vision transformer under dynamic network environment. In JEEE INFOCOM 2025-IEEE Conference on Computer Com- munications, pp. 1-10. IEEE, 2025. 1
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In CVPR Workshop on Fine-Gre Grained Visual Categorization (FGVC), volume 2, 2011. 10
12

---

Published as a conference paper at ICLR 2026
Piotr Komorowski, Hubert Baniecki, and Przemyslaw Biecek. Towards evaluating explanations of vision transformers for medical imaging. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3726-3732, 2023. 3
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine- ane-grained categorization. In IEEE International Conference Computer Vision Workshops, pp. 554-561, 2013. 29
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7, 17
Ken Lang. Newsweeder: Learning filter netnews. In International Conference on Machine Learning, pp. 331-339. Elsevier, 1995. 29
Lu Li, Jiale Liu, Xingyu Ji, Maojun Wang, and Zeyu Zhang. Self-explainable graph transformer for link sign prediction. In AAAI Conference on Artificial Intelligence, volume 39, pp. 12084-12092, 2025. 2
Song Liao, Mohammed Aldeen, Jingwen Yan, Long Cheng, Xiapu Luo, Haipeng Cai, and Hongxin Hu. Understanding gdpr non-compliance privacy policies of alexa skills in european market- places. In ACM Web Conference, pp. 1081-1091, 2024.
Zhu Liao, Victor Quétu, Van-Tam Nguyen, and Enzo Tartaglione. Can unstructured pruning reduce the depth in deep neural networks? In IEEE/CVF International Conference on Computer Vision, pp. 1402-1406, 2023.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014. 8
Xinyu Liu, Houwen Peng, Ningxin Zheng, Yuqing Yang. Han Hu, and Yixuan Yuan. Efficientvit: Memory efficient vision transformer with cascaded group attention. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14420 14430, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In /EEE/CVF Interna- tional Conference on Computer Vision, pp. 10012-10022, 2021. 7,25
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 5rd International Con- ference Learning Representations (ICLR 2017), 2017. 24
Zhiying Lu, Chuanbin Liu, Xiaojun Chang, Yongdong Zhang, and Hongtao Xie. DHVT: Dynamic hybrid vision transformer for small dataset recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Sachin Mehta and Mohammad Rastegari. MobileViT: Light-weight, general-purpose, and mobile- friendly vision transformer. International Conference on Learning Representations, 2022. 3
P Molchanov, S Tyree. T Karras, T Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, 2019. 21
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp- 722-729, 2008. 29
NVIDIA. Train with mixed precision, 2023.
URL
com/ html.
deeplearning/ wwwtarninngg/rrfeeeeee welppperrrrrcccc http://ddd.rrrciiioootttaiaaaaaa
Lorenzo Papa, Paolo Russo, Irene Amerini, and Luping Zhou. A survey efficient vision trans- formers: algorithms, techniques, and performance benchmarking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):6622-770, 2024.
13

---

Published as a conference paper at ICLR 2026
Nikhila Ravi, Valentin Gabeur, Yuan-Timee Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. International Conference Learning Representations, 2025. 1
Benjamin Recht, Rebecca Roelofs,
Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference Machine Learning, pp. 5389-5400. PMLR, 2019. 29
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 15th European Signal Processing Conference, pp. 606-610. IEEE, 2007. 5
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115:211-252, 2015. 4, 7
Shaibal Saha and Lanyu Xu. Vision transformers on the edge: A comprehensive survey of model compression and acceleration strategies. Neurocomputing, pp. 130417, 2025.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 20
Apoorv Singh. Training strategies for vision transformers for object detection. In IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 110-118, 2023.
Xinglong Sun, Barath Lakshmanan, Maying Shen, Shiyi Lan, Jingde Chen, and Jose M Alvarez. MDP: Multidimensional vision model pruning with latency constraint. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20113-20123, 2025, 2, 8, 26
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357, 2021. 4, 7,
Shikhar Tuli and Niraj K Jha. EdgeTran: Device-aware co-search of transformers for efficient inference on mobile edge platforms. IEEE Transactions on Mobile Computing, 23())70122-029, 2023. 2
Tim Van Erven and Peter Harremos. Rényi divergence and kullback-leibler divergence. 1EEE Transactions on Information Theory, 60(7):3797-338) 2014. 17
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deep- net: Scaling transformers to 1, 1,000 layers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6761 -6774, 2024. 1,3
Xiao Wang, Yu Jin, Wentao Wu, Wei Zhang, Lin Zhu, Bo Jiang. and Yonghong Tian. Object detec- tion using event camera: A moe heat conduction based detector and a new benchmark dataset. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 29321-29330, 2025a.
Zhaoqing Wang, Xiaobo Xia, Runnan Chen, Dongdong Yu, Changhu Wang, Mingming Gong, and Tongliang Liu. Lavin-dit: Large vision diffusion transformer. In IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 20060-20070, 2025b. 1
Hao Wen, Yuanchun Li, Zunshuai Zhang, Shiqi Jiang, Xiaozhou Ye, Ye Ouyang, Yaqin Zhang, and Yunxin Liu. AdaptiveNet: Post-deployment neural architecture adaptation for diverse edge environments. 29th Annual International Conference on Mobile Computing and Networking,
pp. 1-17, 2023.
Haocheng Xi, Yuxiang Chen, Kang Zhao, Kai Jun Teh, Jianfei Chen, and Jun Zhu. Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per- block quantization. International Conference on Machine Learning, 2024. 16
Huanrui Yang. Hongxu Yin, Maying Shen, Pavlo Molchanov. Hai Li. and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18547-18557, 2023. 5, 21,
14

---

Published as a conference paper at ICLR 2026
Shusheng Yang, Xinggang Wang. Yu Li, Yuxin Fang, Jiemin Fang, Wenyu Liu, Xun Zhao, and Ying Shan. Temporally efficient vision transformer for video instance segmentation. In 1EEE/CVF
Conference on Computer Vision and Pattern Recognition, pp. 2885-2895, 2022. 1
Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. Conference on Empirical Methods in Natural Language Processing, 2024a. 3
Zhendong Yang, Zhe Li, Ailing Zeng. Zexian Li, Chun Yuan, and Yu Li. ViTKD: Feature-baser knowledge distillation for vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1379-1388, 2024b.
Shuochao Yao and Tarek Abdelzaher. Model compression for edge computing. In Artificial Intelli- gence for Edge Computing, pp. 153-195. Springer, 2023. 2
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster lan- guage model pre-training. International Conference on Learning Representations, 2024.
Shengyuan Ye, Jiangsu Du, Liekang Zeng, Wenzhong Ou, Xiaowen Chu, Yutong Lu, and Xu Chen. Galaxy: A resource-efficient collaborative edge ai system for in-situ transformer inference. In IEEE INFOCOM 2024-IEFE Conference on Computer Communications, pp. 1001-1010. IEFE, 2024. 2
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 24355-24363, 2023. 8, 26
Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad Morariu, Xintong Han, Mingfei Gao, Ching-
Yung Lin, and Larry S Davis. Nisp: Pruning networks using neuron importance score propagation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9194-9190000 2018. 5
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12104-12113, 2022.
Hanxiao Zhang, Yifan Zhou, and Guo-Hua Wang. Dense vision transformer compression with few samples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15825- 15834, 2024. 2, 8, 26
Yunshan Zhong, You Huang, Jiawei Hu, Yuxin Zhang, and Rongrong Ji. Towards accurate post- training quantization of vision transformers via eтror reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 3
Yan Zhuang, Zhenzhe Zheng, Yunfeng Shao, Bingshuai Li, Fan Wu, and Guihai Chen. Nebula: An edge-cloud collaborative learning framework for dynamic edge environments. In 53rd Interna- tional Conference on Parallel Processing, pp. 782-791, 2024.
15

---

Published as a conference paper at ICLR 2026
A NECESSITY OF MODEL DERIVATION
When customizing models for specific edge devices, one may consider directly fine-tuning small models to meet resource and task requirements. However, this approach is often impractical in real-world settings. In contrast, deriving small model from large model offers several distinct advantages:
Flexibility. Edge devices exhibit highly diverse resource budgets, and unrealistic to assume that suitably sized pretrained lightweight model always exists for every deployment profile. Training a lightweight model from scratch is also undesirable, as poor initialization often leads to suboptimal convergence, shown in Table 4. In contrast, deriving small models from large base model avoids these restrictions. Vulcan can automatically determines the target architecture and transfer useful knowledge directly from the large model into the derived one.
Higher Accuracy. Large pretrained models possess richer representational capacity and encode high-level knowledge that small models do not contain. Models derived through Vulcan inherit this knowledge and therefore achieve substantially better decision boundaries.
Scalability. The paradigm of deriving small models from a large model is forward- compatiile:: as foundation models continue to improve, the derived models will naturally become stronger as well, without requiring redesign or retraining from scratch.
Table 4: Accuracy comparison on three ImageNet sub-tasks (T1/25-T3/25) between DeiT-Small fine-tuned with and without initialization as well models of the same size derived from pretrained DeiT-Base using Vulcan.
Setting
T1/25
T2/25
T3/25
Avg
DeiT-Small (w/o init)
58.24
60.24
61.58
60.02
DeiT-Small (w/ init)
93.96
94.64
93.61
94.07
DeiT-Base (Vulcan-0.73)
94.80
96.08
94.65
95.18
B COMPARISON OF MODEL COMPRESSION TECHNIQUES
Quantization and Unstructured Pruning. As discussed in Section 2, quantization and unstruc- tured pruning cannot deliver practical acceleration on edge devices without specialized hardware or software support. To empirically validate this point, we conduct experiments on Jetson Orin NX, widely used embedded device in robotics, industrial inspection, and autonomous driving. Specifically, we apply INT8 quantization (Xi et al., 2024) and magnitude pruning (Han et al., 2015) representative techniques of quantization and unstructured pruming, respectively, DeiT-Small, and DeiT-Tiny (Touvron et al., 2021). We then measure inference latency with a batch size of since edge devices are typically deployed in real-time scenarios where inputs arrive quentially. As shown in Table 5, INT8-quantized models cannot be executed on the GPU of Jetson Orin NX due to the lack of INT8 support, and are instead forced significantly slower inference compared to the original base ViTs. For magnitude pruning, although the pruned models can still run on the GPU, the parameter count remains unchanged, leading to vir- tually no difference in inference latency before and after pruning. This result suggests that relying on quantization or unstructured pruning would undermine the latency advantage of edge deployment.
DeiT-Base,
the CPU, which results in
Table Inference latency and throughput of VITs before and after INT8 quantization and magnitude pruning. Quantized models run CPU due to lack of TensorRT support, resulting in much slower inference, while magnitude pruning does not reduce parameter count and thus yields no speedup.
Base VIT (GPU)
INTS Quantization (CPU)
Magnitude Pruning (GPU)
Methods
Latency (ms)
Throughput (imagols)
Latency
Throughpur (image/s)
Latency
Throughput timago/s)
(ms)
(ms)
DeiT-Base
45.45
22.00
3429.48
0.29
45.47
21.99
DelT-Small
15.58
64.16
1497.11 11
0.67
15.80
63.28
DeiT-Tiny
14.52
68.87
666.43
1.50
14.88
67.21
16

---

Published as a conference paper at ICLR 2026
Knowledge Distillation. As discussed in Section 2, knowledge distillation (KD) incurs substantial training overhead. This because KD transfers knowledge primarily in the feature space, where the student model is trained from scratch to fit the outputs of teacher model. Given student architecture, it is often infeasible to obtain a pre-trained model with the exact same architecture for initialization. The absence of parameter-space knowledge transfer results in slow convergence. In contrast, structured pruning naturally enables parameter-space knowledge transfer by inheriting weights from the pre-trained model, which provides strong initialization and significantly acceler- ates convergence compared to K.D.
100
Accuracy (%)
KD (0.6)
Pruring (0.6)
Stap- 100 (50g)
Stop=1000 (5x)
Pruring (0.8)
2000
4000
Step
Figure 12: Comparison of convergence speed between logit-based knowledge distillation (KD) and random pruning (Pruning) across different pruning rates with DeiT-Base on CIFAR-10.
To illustrate this, compare the convergence speed of logit-based KD (Hinton, 2015) with random pruning (Gadhikar et al., 2023) with DeiT-Base on CIFAR-10 (Krizhevsky et al., 2009). Specifically, random pruning uniformly selects a proportion of query-key and value-output dimensions MHA modules as well as neurons in FFN modules according to the target pruning rate, and removes them construct smaller model. In the KD setting, the same student architecture derived from random pruning is instead trained from scratch, with the original pre-trained model serving as the teacher. KL divergence loss (Van Erven & Harremos, 2014) between the student and teacher logits is used to perform distillation. As shown in Figure 12, the pruned models converge significantly faster than their KD counterparts, highlighting the effectiveness of parameter-space knowledge transfer. Nevertheless, it is worth noting that KD be effectively combined with structured pruning to further improve the accuracy of pruned models.
C INTERPRETABILITY OF NEURONS FFN
As discussed in Section 3.2, neurons in the FFN modules of ViTs exhibit remarkably strong terpretability. In this appendix, we provide additional experimental details and supplementary re- sults to further support this finding. We begin by clarifying how the activation of a specific neuron (2..)) E Rd is computed for given image. For input image, it is represented patch tokens X R xx xd fed into the FFN of the (-th block. The activation neuron n((..) for this image is then defined as:
sequence of
000 Σ (((i),n()))))
(17)
We use Eq. (17) to measure the relevance between image and neuron, and then rank all images in the dataset accordingly. For each neuron, we select the top-25 images with the highest activations examine the type of knowledge encoded in that neuron.
17

---

Published as a conference paper at ICLR 2026
1066
(a) n (1,2640). Dark color tones.
(b) m
(2.718); Repetitive textures.
(c) (3,1718) Grid textures.
(d) nl (4,343) Grid textures.
(c) (5,573), Close-up up shots.
(f) 11
(6,1848) Honeycomb textures.
Figure 13: Additional visualizations of top-25 activated images for randomly selected FFN neurons from the first six blocks of DeiT-Base.
18

---

Published as a conference paper at ICLR 2026
(a) n 17.1)))) Green backgrounds.
(h) n
(8,1732)
Paired objects.
(c) m
(9.801) Chocolate drizzles.
(d) (10,1 Urban landscapes.
(e) 12 (1111)))), Water birds.
(1) m1 (12.17); Dogs.
Figure 14: Additional visualizations of top-25 activated images for randomly selected FFN neurons from the last six blocks of DeiT-Base.
As shown in Figures 13 and 14, we provide additional visualizations of randomly selected neurons from blocks of DeiT-Base. Consistent with the observations Section 3.2, these neurons ex- hibit strong interpretability, and the knowledge stored in deeper blocks tends to be more semantic. Interestingly, as illustrated in Figure 13(e), we even observe neurons that capture patterns such as "closc-up shots," which are unrelated to the actual content of the image. This further demonstrates that FFN neurons not only encode class-specific knowledge but also capture interpretable patterns beyond classes.
m
(1,399)
(b) n
(2.1
(c) n (5.513)
(7,296)
(e)
(9,686)
( (1,475)
Figure 15: Top-25 activated images (sampled from CIFAR-10) for random FFN neurons DeïT- Tiny fine-tuned CIFAR-10.
19

---

Published as a conference paper at ICLR 2026
(1,288) m1
(b) n1 (3,7)
(c) n1
(5,860)
mi
(7,674)
(e) ni
19,1311)
mi (11,687)
Figure 16: 25p-25 activated images (sampled from CIFAR-100) for random FFN neurons in DeïT- Small fine-tuned CIFAR-100. 100.
H
m1
(1,932)
(b) n
(3,3855)
(c) 11
(5913)
nj
17,205)
(c) n
(9,11
(f) m
(11,2007)
Figure 17: Top-25 activated images (sampled from ImageNet- for random FFN neurons VIT- Large/16.
(1,12188) (a) m1
(2.1,299) (b) ri
(c) m1
(3,1,1008)
(d) m (3,3,221)
(e) T2
(3.6.918)
Ti
(4.1,2612)
Figure 18: Top-25 activated images (sampled from COC02017) for random FFN neurons in the Swin-Tiny backbone of Mask R-CNN. n (s.b,i) denotes the i-th neuron in the FFN of the b-th block in the s-th stage of the Swin Transformer.
To demonstrate that the interpretability of FFN neurons based on their activation patterns is general phenomenon across ViTs, conducted additional visualizations different models and datasets. Specifically, for each model, DeiT-Tiny, DeiT-Small, ViT-Large, and Mask R-CNN (Swin-T), we randomly sampled FFN neurons and visualized the patterns they respond to CIFAR-10, CIFAR- 100, ImageNet, and COCO. As shown in Figures 15-18, these visualizations consistently indicate that the FEN modules of VIT models encode class-specife knowledge.
D EXTENDING NEURON COLLAPSE TO ALTERNATIVE FFN ARCHITECTURES
Our class-centric neuron collapse (CCNC), originally designed for canonical FFN architecture, representative and generalizable, making it applicable to other mainstream Transformer FFN vari- ants. Taking SwiGLU (Shazeer, 2020), a widely used design in LLMs, as an example, its computa- tion is follows:
SwiGLU (X)
(o(XW(1T) o (XW:"IT) W
((XW())) (1) (xw( s W
(18)
Σ
(XXm (1.0) o (xnll.i)) 8 nn
20

---

Published as a conference paper at ICLR 2026
where W E Raxd and WiD E Raxe are the up- and down-projection matrices of the FFN in the 1-th block of ViT, W E R is the gating matrix, and (1.1] f E IR denotes the i-th neuron in ||210- To prune the intermediate dimension et of the FFN, one only needs to extend Eq. (4) with additional constraints on Ng. Specifically, the anchor neuron of each cluster for nı and ny is selected as the one that maximizes the following value:
W
((1.)) Σ( (o ((x(i, (10)) (Xlilin(..)) -=1
(19)
Notably, in most Transformer architectures, the neurons associated with different intermediate di- mensions of the FFN (e.g., nf4-",, n5!..), and n(l.)) at the i-th dimension in SwiGLU) are mutually independent, which implies that CCNC can be readily applied these architectures as well.
E COMPARISON OF PRUNING SETTINGS IN MHA
Table 6: Comparison of head-level and dimension-leved pruning on DeiT-Base with ImageNet- IK sub-tasks under pruning rates of 0.40 and 0.60. Dimension pruning achieves consistently higher accuracy, showing its advantage in preserving knowledge compared to coarse-grained head pruning.
Methods
0.40
0.60
0.40
0.60
0.40
0.60
Head-level
4.56
1.12
1.68
0.92
14.11
3.47
Dimension-level
78.08
59.76
78.76
62,20
77.29
61.46
Head or Dimension? The Multi-Head Attention (MHA) module can be pruned at two different granularities: head-level pruning, which removes entire attention heads, and dimension-level prun- ing, which reduces the dimensionality of the query, key, or value vectors. To determine the appro- priate pruning granularity for Vulcan, we conduct comparative study between these two strate- gies. For head-level pruning, we follow prior works and adopt the Taylor expansion approxima- tion (Molchanov et al., 2019; Yang et al., 2023) criterion to evaluate the importance of each attention head and perform block-uniform pruning (Blalock et al., 2020). For dimension-lewe pruning, we apply singular value decomposition (SVD) to compress the query-key (QK) and value-output (VO) dimensions, motivated by the clear advantage of SVD observed in Figure 4.
As shown in Table 6, we compare these two strategies across different pruning rates (0.40, 0.60) and sub-task (25, 50, 100) sizes with DeiT-Base on ImageNet-IK. IK. The results show that dimension-level pruning consistently outperforms head-level pruning, indicating that coarse-grained head pruning leads to substantial knowledge loss. In contrast, Vulcan leverages the computational structure of MHA and applies SVD-based adaptive pruning to the QK and VO dimensions, enabling more effi cient and effective knowledge extraction.
Table 7: Inference latency of DeiT-Base, DeiT-Small, and DeiT-Tiny under uniform vs. uneven QK/VO dimensions Jetson Orin NX (bz=1) and NVIDIA A40 (bz=256). In the uneven setting, the dimensions of each head in each block are randomly assigned while keeping the total parameter count unchanged.
DeiT-Base (ms)
DeiT--Small (ms)
DeiT-Tiny (ms))
Methods
bzwl
bom256
bzwl
bo=256
bawl
hz=256
Uniform Uneven
45.45
600.52
84.10
629.62
50.66
200.63
31.50
82.51
Uniform Ps. Uneven QK/VO Dimensions. One key design of Vulcan is to enforce identical query- key (QK) and value-output (VO) dimensions across all heads within the same block, which is crucial for efficient inference. When head dimensions
uniform, the MHA module can be computed in parallel across heads; otherwise, attention must be evaluated sequentially, introducing slicing operations and memory copies that significantly increase latency. As shown in Table 7, we measure
21

---

Published as a conference paper at ICLR 2026
inference latency on both Jetson Orin NX (batch size 1) and NVIDIA A40 (batch size 256) for DeiT-Base, DeiT-Small, and DeiT-Tiny under two settings: uniform vs. uneven QK/VO dimensions. Despite identical parameter counts and GFLOPs, the uniform meiformmheedd setting achieves consistently lower latency, with the gap particularly pronounced on edge devices.
F THEORETICAL ANALYSIS OF SVD FOR MHA COMPRESSION
As shown in Figure 4, pruning the query-key and value-output dimensions with singular value de- composition (SVD) achieves significantly better performance than score-based pruning methods. In this section, we first introduce the fundamentals of SVD, and then provide a theoretical explanation for this seemingly counterintuitive observation.
SVD. Given a matrix W E Rmx", SVD factorizes it into a set of orthogonal singular vectors and their associated singular values, which capture the intrinsic low-rank structure of the matrix. Formally, SVD expresses W as:
W UEVT, UER ma
DER R V E R
(20)
where U is the left singular matrix, E is the diagonal singular value matrix, and V is the right singular matrix. The columns of U. referred to as the left singular vectors, are eigenvectors of WWT and provide orthogonal basis for the input (row) space. The columns of V, referred to as the right singular vectors, are eigenvectors of W W and provide an orthogonal basis for the output (column) space. The diagonal entries of Σ are non-negative real numbers known as singular values, arranged in descending order. Each singular value quantifies the "stretching factor" of the matrix along the corresponding singular direction, with larger values indicating more informative directions.
Why SVD is Better? SVD not only reveals the low-rank structure of W but also allows for con- structing its best rank-k approximation by truncating the top-k singular values and their associated singular vectors. Taking Wq E IRax and WK E R1xd as an example, when pruning the query-key dimension, the goal to preserve the intermediate representations as much as possible. In other words, the optimal pruned matrices W/2 Q
R. xd and W TRyy' xd should satisfy:
W Wk arg min| W W W WKlF
(21)
W.
According to the Eckart- Eckart-Young theorem (Chipman, 2020), SVD guarantees the minimization of the truncation error in the Frobenius norm. Specifically, for a given pruned query-key dimension q' < q. the optimal pruned matrices W/2 and W' are obtained by applying SVD to W Wx:
W WK = U@K Eox VOK
(22)
w = (UQkl,; ::ll:::::///xx/.....
W = (Vokls;:41
(23)
In contrast, score-based pruning methods remove certain row vectors from WQ and Wx directly, which leads to a truncated error the Frobenius norm that no smaller than that obtained by SVD. This inevitably results in greater loss of information. Therefore, SVD maximizes knowledge preservation under the same pruning rate, which explains its significant advantage over score-based methods.
G PROOF OF PRUNING RATE SATISFACTION IN DERIVED MODELS
The pruning rate R is determined according to the resource constraints of the target edge device. To ensure that the derived models can be successfully deployed for inference on such devices, Vulcan must strictly satisfy the specified pruning rate. In this section, we first introduce the computation of FLOPs for ViT, and then provide a formal proof that the size of the models derived by Vulcan exactly matches the pruning rate R.
FLOPs. FLOPs represent the number of Soating--pon operations required for a model to perform inference single image. For a VIT, let the embedding dimension be d, the query-key dimension, valuc-output dimension, and the number of heads in the MHA module of the I-th block be qı vi and H₁, respectively, and let the intermediate dimension of the FFN module in the 1-th block be ex-
22

---

Published as a conference paper at ICLR 2026
Then, for VIT Mg consisting of L blocks, the total FLOPs can be formulated as:
FLOPs(MB) = Σ
(2Ndg + 2Nder + N2 + N2 Hi + (2Nde))
1=1
(24)
(2Nd+ N2) Σ Hi v1) + 2Nd Σ E3
where N is the number of patch tokens. It can be observed that the three dimensions pruned by Vulcan, i.e., Us vs and Eg, are directly associated with the FLOPs of a VIT.
Proof2: Pruning Rate Satisfaction. We denote the actual pruning rate of the derived model ME as R'. Then, proving that Vulcan strictly satisfies the target pruning rate amounts to showing that: (L
R' == FLOPs(MB)
N
(25)
From Eq. (3). it follows that the intermediate dimension of the pruned FFN module in the I-th block,
denoted e/, satisfies:
L
KTO
a (1i)
(A4,
e₂)
R1)
1=1
j=1
L
Σ
el
et) R
el
a1) R
(26)
1=1
1=1
1=1
1=1
Σ
x (1 -1-R R)
Under the assumption that gi 1t and gi qj. according Eq. (5) and Eq. (6), the query-key dimension of and valuc-output dimension of in the pruned MHA modules satisfy:
Σ dal +1) Σ 1=1
(la(1 R@K ) + [1r(1 - RBoo))
1=1
(9((1 Bxx) v1(1 BPPOO)
OOK rvo 21 R!!)
(1)
(1) (1)
(0)
Tyo 2r R())
(1) (1)
OOK
+
(1)
(1)
(1)
TQk
+ +vo
TQK + vrv
2(r + +Tvo)= - 2((())
(1)
+ R
QK
(1)
(I)
rQk + Tvo
(27)
L
-2 Σ
(1 RR( -2n1 Σ
(1
R
1=1
1=1
L
R
2Q1 Σ
Σ 1/(r (1)
+ (8)
(1)
(1)
rv))
TQK +
Vv
LR L 1/(10k + (1)
rvo)
-2Lgi 241
2L41 2L
Σ/= 1/(70k +r(1))
7
=2Lq (1 R.) = (L(g) + (z)) - R) =
Σ
(qt + vi)
(1 - R)
23

---

Published as a conference paper at ICLR 2026
Under the assumption that H; = H substituting these relations into Eq. (25), obtain:
1 (2Nd + No) Σ Hilg/ + +/)++ 2Nd
Ei=1
(2Nd+
Hilyr + 2N
1 (2Nd+ 2 Σ 1= Hi +0)
2Nd E ed -
R)
(28)
(2Nd + No) E +(( + 2Nd 21=1
=1- - (1) R This establishes that the models derived by Vulcan strictly satisfy the specified pruning rate R.
H IMPLEMENTATION DETAILS
We summarize the hyperparameter settings used during the post-training process in Table
Table 8: Hyperparameters for the post-training process of Vulcan. Here, R denotes the target pruning rate and represents the Lagrange multipliers.
Hyperparam
Value
Steps
6250R + 1250R, R (0,1)
Optimizer
AdamW (Loshchilov Hutter, 2017) 256 (Recognition), 8 (Detection/Segmennttion)
Batch Size
Learning Rate (LR)
le-4 (Recognition), 5e-5 (DetectionnSSgnneetatiinn))
LR Scheuler
Constant
Weight Decay
0.05
Penalty Parameter p
1.0
Penalty Update Rule
A1+1 + p(OC/OX,
Seed
Normalization. As discussed in Section 4, Vulcan will normalize the activations of each neuron before computing (0)) The choice of normalization strategy is closely related to the derived model architecture. Therefore, we compare three commonly used normalization strategies. For activation vectors (!) (a(1.1), al.22) ,.... a(l.e1) E R their normalization processes follows:
L.2 Normalization:
(1.1)
(.11
(att.)))
V
Min-Max Normalization:
(1+1) - min(a!))
(1.1)
max(a(!) m))(
Z--core Normalization:
(1.0)
(L.)) 0 homm
(.))
(a05) - 11)²
a
==1
24

---

Published as a conference paper at ICLR 2026
Intermediate dimensien
12
dimension
Mi-Maa
2393
2000
2302
80
15c8-
2020
2083
Accuracy (%)
Block
Block
Intermediats aimenciinn
Z-Score
dimension
Unitorm
2010
2383
L2
2020
2083
Min-Mar
Z-Score
Block
Block
Stop
Architectures of models de- rived under different normalization strate- gies. Since the architectures corresponding to TI-T3 are highly similar, this figure shows their averaged architectures.
Figure 19:
Figure 20: Convergence curves of models with three normalization methods and the uniform architecture. Z-score normalization and the uniform architecture achieve faster convergence and higher accuracy than L2 and Min-Max normalization.
At a pruning rate of 0.60, we derive a series edge VITs from DeiT-Base for the three ImageNet sub-tasks (TI-T3) in Table 1. In addition, we introduce uniformly structured models bascline for comparison with the activation-adaptive architectures. As illustrated in Figure 19, the architec- tures resulting from different normalization methods exhibit substantial differences. Specifically, L2 normalization produces architectures with fewer parameters in the intermediate blocks, min-max normalization leads to architectures with fewer parameters in the deeper blocks, while z-score nor- malization yields relatively uniform architectures, showing an overall trend where the number of parameters decreases as the block depth increases. Figure 20 further demonstrates that the archi- tecture derived with 7-score normalization achieves the best performance, and thus Vulcan adopts for normalizing activation values of DeiT. At the same time, we also observe that the effective- ness of normalization strategies is model-dependent. For Swin-Transformer, Vulcan instead adopts min-max normalization.
Adaptive Pruning Rate Allocation for Non-Uniform Architectures. Eq. (3) and Eq, (5) implicitly assume that Transformer blocks share an identical architecture, which does not hold for non- uniform models such Swin-Transfowe Transformer (Liu al., 2021), where different blocks may contain heterogeneous structures. To address this limitation, we extend Eq. (3) and Eq. (5) to support non- uniform architectures. First, let the number of parameters in the FFN module and the MHA module of the 1-th block be denoted as Pf and P(l) respectively. By incorporating P()) and P()) into the importance evaluation, generalize the computation of K and RU) as follows:
KO)
a (1.)
(AA (
e))xx x))x
A=
(1,3)
li 1 L....,er} (29)
log( (!
log(P!))
[=1
EL (()
P()
R(
R
(30)
ΣL
(
(P / + + vr)))
TQK
(1)
+ Tvo
(
It is worth noting that in certain extreme cases (e.g., when the importance of a particular block significantly lower than that of others while the pruning rate very high), the values of R()) computed by Eq. (5) and Eq. (30) may exceed 1. In such situations, Vulcan sets the query query-key and valuc-output dimensions of the overflowing blocks to 1, and iteratively recalculates the R!)) of the remaining blocks until all blocks satisfy R e (0, 1).
I BASELINES
In this section, detail the five baselines used for comparison with Vulcan:
25

---

Published as a conference paper at ICLR 2026
Random. The random pruning method operates the same dimensions Vulcan, ie., the intermediate dimension (é) of the FFN and the query-key and value value-output dimensions (q, v) of the MHA. Given a pruning rate. it uniformly prunes dimensions by randomly selecting neurons to delete, without considering the specific sub-task. We adopt this class-agnostic pruning method lower-bound benchmark to evaluate the effectiveness of class-specific model derivation.
NVTT (Yang et al., 2023). NViT prunes all dimensions of a VIT, including the intermediate dimension (e) of the FFN. the query-key value value-output dimensions (g. v), and the number of attention heads (H) of the MHA, and the embedding dimension (d). It groups the weights along each dimension and evaluates their importance using Hessian- -aaee importance score. The pruning process is performed iteratively by removing the groups with the lowest scores until the resource constraints of the target device are satisfied.
X-Pruner (Yu & Xiang, 2023). X-Pruner prunes the intermediate dimension of the FFN (e) and the number of attention heads (H) in the MHA. introduces learnable masks into the ViT, and pruning is achieved by applying sparsity regularization gradually drive parts of the mask values to zero.
DC-VIT (Zhang et al., 2024). The goal of DC-ViT to achieve the target pruning rate while pruning as few blocks as possible. It designs an importance metric to evaluate each block, and for the selected blocks, it removes the entire MHA module and uniformly prunes the intermediate dimension (e) of the FFN at random. Among all the baselines, DC-VIT is the only method that performs pruning at the module level.
MDP (Sun et al., 2025). Similar to NVIT, MDP jointly prunes multiple dimensions of ViTs, including the embedding dimension (d), the number of attention heads (H), the query-key and outuc-output dimensions (q. v), and the FFN intermediate dimension (e). It formulates pruning as mixed-integer nonlinear program (MINLP) problem under latency budgets, solved with bessian-based importance scores and precomputed latency lookup table (LUT).
is worth noting that although Random pruning intuitively expected to perform the worst, ac- tually outperforms some score-based methods. We attribute this to the fact that it adopts the same pruning dimensions and granularity as Vulcan. In contrast, DC-ViT suffers from overly coarse prun- ing granularity, making it difficult to recover model accuracy through retraining when the pruning rate becomes high.
J ADDITIONAL EXPERIMENTAL RESULTS
In this section, we present additional experimental results and analyses. Appendix J.1 compares Vulcan with five baselines at lower pruning rates to further demonstrate the effectiveness of Vul- can. Appendix J.2 visualizes the model architectures of edge ViT' derived by Vulcan and further investigates how class-specific knowledge is distributed across different blocks. Appendix J.3 com- pares the computational efficiency of Vulcan-derived models with five baselines, demonstrating the edge-friendliness of Vulcan. Appendix J.4 further demonstrates the scalability of Vulcan larger ViTs and BERT. Appendix J.5 shows that Vulcan enhances model generalization and robustness. Appendix J.6 evaluates Vulcan in complex open-domain scenarios. Appendix J.8 investigates the trade-off between derivation performance and derivation overhead. Appendix J.9 illustrates how Vulcan be combined with other compression techniques achieve extreme lightweighting.
26

---

Published as a conference paper at ICLR 2026
J.1 PERFORMANCE UNDER MODERATE PRUNING
Table 9: Overall performance of Vulcan and baselines on DeiT-Base with ImageNet- 1K sub-tasks of different sizes (25, 50, 100 classes) under pruning rates of 0.20 and 0.40.
Method
T1/25
T2/25
T3/25
Aug
T4/50
T3/50
T6/50
Avg
T7/100
T8/100
T9/10
Avg
DefT-Base
79.92
82.56
80.67
81.05
80.48
79.64
84,96
81.69
80.49
84.12
78.58
81,06
DeiT-Base (FT)
96.40
96.96
96.96
96.77
95.00
93.36
94.68
94.35
91.56
93.29
92.84
92.56
Pruming Rate 0.20
Random
95.60
96.48
96.40
96.16
93.84
93.36
94.60
93.93
91.41
93.07
91.44
91.97
NVTT
94.96
96.12
96.01
95.76
92.96
92.72
94.32
93.10
91.27
91.98
X-Pruner
96.56
96.48
96.62
94.40
93.00
94.26
91.16
92.65
DC-NTT
95.52
95.76
94.57
95.28
91.40
93.16
94.00
92.85
84.73
88.34
90.84
87.97
MDP
95.44
96.56
94.73
95.55 95.58
92.60
88.00
92.48
91.03
84,50
92.79
90.74
89.34
Valcan
96 06.48
97.12
97.04
96.88
95.00
93.24
94.60
94,28
91.49
92.91
91.74
92.05
Pruning Rate 0.40
Random
94.64
95.68
94.48
94.93
91.84
90.84
92.48
91.72
87.94
90.41
89.12 12
89.10
NVIT
92.08
94.24
94.17
93.50
90.60
90.00
91.52
90.71
87.23
X-Pruner
94.76
94.68
95.01
94.82
92.34
91.60
93.28
92.41
88.95
91.67
89.04
89.89
DC-VIT
86.80
89.12
88.66
88.
83,84
80.60
87.52
83.99
80.91
83.73
83.16
82.60
MDP
95.36
96.40
93.69
95.15
93.40
89.76
90.60
91.25
88.89
88.12
89.36
88.79
Vulcan
94.80
95.68
94.97
95.15
92.76
91.32
93.60
92.63
89.71
91.39
89.84
90.38
We first compare Vulcan with five baselines at pruning rates of 0.20 and 0.40 to evaluate its perfor- mance under moderate pruning conditions. As shown in Table 9, we observe that at pruning rates of 0.20 and 0.40, Vulcan consistently outperforms the base ViT and five baselines in terms of accuracy across different sub-task sizes. Specifically, at a pruning rate of 0.20, the model derived by Vulcan outperforms the base VIT by 13.13% on average, and further surpass the best pest-performing baselines, X-Pruner and MDP, by up 0.64% and 6.99%, respectively. At the higher pruning rate of 0.40, Vulcan achieves average accuracy improvements of 11.45% 0.35%, and 5.99% over the base ViT, X- X-Pruner, and MDP, respectively. These results further confirm the effectiveness of Vulcan, showing that it delivers consistent advantages even under moderate pruning conditions.
J.2 MODEL ARCHITECTURES OF EDGE VITS
3000
330e
1200
2080
B0.k
0000 Direndin
133520
1011
Figure 21: Visualization of the architectures of edge ViTs derived by Vulcan from DeiT-Base for sub-tasks T1-T3 on ImageNet under pruning rates of 0.60 and 0.80.
We visualize the architectures of edge ViTs derived from DeiT-Base for the sub-tasks T1-T3 in Table 1, as shown in Figure 21. From the visualization, we observe that class-specific knowledge in the FFN modules is more concentrated in the middle blocks, while class-agnostic knowledge in the MHA modules is more concentrated in the value-output dimensions.
27

---

Published as a conference paper at ICLR 2026
J.3 COMPUTATIONAL EFFICIENCY COMPARISON
Table 10: Comparison of computational efficiency between DeiT-Base and edge ViTs derived by Vulcan and five baselines under different pruning rate R, evaluated on Jetson Orin NX (bz=l) and
NVIDIA RTX 4090 (bz=256).
Latency (ms)
Throughput (image's)
Memory (GB)
#Param
#FLOPs
Acc.
Methods
Orin NX
RTX 4090
Orin NX
RTX 4090
Orin NX RTX 4090
(M)
(G)
(%)
DeiT-Base
45.45
22.00
2.21
86.57
81.27
Pruning Rate
0.20
Random
42.97 (1U55) 278.21 (0.99))
23.27
920.15
0.28
2.75
69.47
14.06 (119.98%)
94.02 (2.75)
NVIT
(1.21x)
227.53 (1.21
26.68
1125.15
0.27
2.15
68.71
14.05 (1.9.98%)
93.61 (21334
X-Pruner
(1.21))
223.87 (1.23x)
26.67
1143.51
0.28
2.15
68.58
13.94 (120.66%%
94.24 (112.97)
DC- VII
36.98 (1.23x) 225.19 (1.22X)
27.04
1136.83
0.28
2.15
69.62
12.49 (128.91%)
92.03 (110.76)
MDP
33,78 (1.35x)
202.48 (1.35xx
29.61
1264.31
0.25
2.12
62.21
14.05 (1.9998%)
91.98 10.71)
Vukcan
(1.23x)
(1.266)
27.14
1173.43
0.27
2.14
67.22
13.56 (121.51%)
94.40
13.15)
Praning Rate 00.40
Random
(1.33x)
229.04 (1.20x)
29.20
11177.69
0.21
2.57
52.39
10.55 (1.39.95%)
91.94 (1.66)77
NVIT
182.20 (1.55xx)
(1.51x)
34.19
1405.02
1.99
50.86
(1))000))))))
(79.60)
X-Pruner
183.33 (1.54x)
(1.49%)
33.86
1396.38
0.21
2.09
52.61
10.71 (139.049))
92.37 (111.10
DC-VIT
21.59
175.33 (1.56x)
34.94
1460.09
0.21
2.08
52.38
10.54 (139.00%%)
84.93 (33.66)
MDP
159.68 (1.33x)
(1.72x)
38.03
1603.25
0.19
2,06
45.76
9.25 (147.35%%
91.73 (110.46)
Vulcan
176.01 (1.33x)
(1.56x)
33.63
1454.42
0.21
1.82
51.00
10.24 (141.725))
92.72 <1 (114.45)
Praning Rate
0.60
Random
(1.70x)
(1.49x)
1388.29
2.39
35.29
7.04 (139.9318)
(15.74)
NVIT
24.55 (1.85x) 137.03
(2.00x)
40.73
1868.23
0.14
1.44
33.13
7,02 (46000018)
81,85 (70.61)
X-Pruner
21.79 (2,09x)
135.53 (2.002)
45.89
1888.93
0,15
2.02
35.56
7.27 (158 62%)
89.41 (78.14)
DC-VIT
20.04 (2.27x)
126.84 (2.16x)
49.90
2018.22
0,15
2.02
35.43
7,03 (1.5.999%)
69.85 (111.42)
MDP
19.35 (2.35x)
122.99 (2.23
51.67
2081
0.13
1.43
31.08
6.20 (164.71%)
88.2 ('6.98)
Vulean
21.81 (2.16x)
136.67 x2
45.86
1873.11
0.15
1.66
34.09
6.77 (161 (718))
92.32 (1 (1.5)
Pruning Rate
Random
101.99 (2.99x)
65.85
2510.1
0.08
1.06
18.21
3.53 (179.9118)
71.45 (.9.82)
NYTT
(2.37x)
98.94 (2.77 x)
52.28
2587.47
0.08
1.37
16.77
3.49 (1800144%)
52.86 (1 (122841)
X-Pruner
90.92 (2.30x)
(3.02x)
50.61
2815.58
0.09
1.33
19.19
3.80 (1788.395)
79.11 (.2.16)
DC-ViT
14.76 (3.08%)
79.56 (3.44
67.75
3213.83
0.08
1.95
18.19
3.51 (180.02%))
49.18 (1 (132.09)
MDP
82.79 (2.48x)
(3)31
54.64
3092.11
0.08
1.09
17.73
3.35 (180.93%)
75.73 (L5.54)
Vulcan
(3.02x)
(2.75x)
0.08
(J81 45%)
88.22 ('6.55)
Given that some pruning methods are not hardware-friendde we compare the computational effi ciency of Vulcan with five baselines under different pruning rates. As shown in Table 10, under the same pruning rate, Vulcan-derived models achieve comparable or better inference efficiency, ranking in the top-3 speedups in most cases, demonstrating Vulcan's 'ss hardware friendliness. It is worth noting that although DC-ViT attains significantly higher speedup at pruning rate of 0.80 by removing part of the MHA modules, the resulting models suffer from overly degraded accuracy, making this trade-off unreasonable.
J.4 SCALING TO LARGER MODEL AND NEW DOMAIN
Table 11: Performance of Vulcan on larger ViT-L/16 and BERT-Base from the NLP domain. '(FT)' denotes fine-tuning the class class-specific data.
VIT-L/16 (ImageNet)
BERT-Bare (NewsGroups)
Methods
T1/25
T223
T3/25
T1/2
124
T3/8
Base VIT
82.64
84.72
83.76
77.60
68.34
72.71
Base VIT (FT)
97.52
97.28
98.16
94.43
87.95
81.52
Vulcan (0.60)
96.72
97.10
97.58
94.56
86.41
81.66
(0.80)
To further demonstrate the scalability of Vulcan, we conduct experiments on both a larger vision model, ViT-L/16, and language model, BERT-Base, from different domain. For ViT-I/16,
28

---

Published as a conference paper at ICLR 2026
derive edge ViTs for the sub-tasks T1-T3 in Table 1. For BERT-Base fine-tune the model on the 20 NewsGroups (Lang, 1995) dataset and derive edge ViTs for three sub-tasks with 2, 4, and target classes. As shown in Table 11, Vulcan achieves results comparable to those in Table 1, confirming its strong scalability across both larger architectures and new domains.
J.5 GENERALIZATION AND ROBUSTNESS OF DERIVED MODELS
Table 12: Accuracy comparison on Distribution-shifii datasets between DeiT-Base and Vulcan- derived models for three ImageNet sub-tasks under pruning rates of 0.60 and 0.80. The models derived from Base by Vulcan exhibit stronger generalization and robustness on the sub-tasks.
ImageNet-V2
ImageNet-R
Methods
T1/25
12/25
13/25
T1/25
T2/25
T3/25
DeiT-Base
2.40
0.40
0.00
0.00
0.00
0.00
Vulcan
11.20
8.00
1.63
Vulcan (0.80)
12.40
7.60
10.40
2.17
2.36
0.26
To evaluate the generalization and robustness of the edge ViTs derived by Vulcan, we further com- pare DeiT-Base and its derived edge ViTs on two datasets whose distributions differ from ImageNet:
V2 ImageNet-Ve V2 (Recht et al., 2019): A re-collection of ImageNet validation data designed to test distribution shifts. contains images sampled from the same class labels but gathered under different conditions, serving as benchmark for evaluating model generalization.
ImageNet-R (Hendrycks et al., 2021): A dataset consisting of various renditions of ImageNet classes (e.g., sketches, paintings, cartoons), which introduces significant appearance variations and is commonly used to assess robustness against domain shifts.
As shown in Table 12, Vulcan can improve the generalization and robustness of base ViTs on sub- tasks to some extent. We attribute this improvement Vulcan's ability encourage edge VITs to focus on specific classes, enhancing their capacity to extract and recognize class-related features, An interesting observation that models with higher pruning rates exhibit even stronger generalization, suggesting promising direction for future exploration.
J.6 VULCAN FOR TRANSFER LEARNING
Vulcan designed to derive compact models from a pre-trained base VIT for a sub-task S, where S SC and Y denotes the class set recognized by the base ViT. This corresponds to a closed-domain scenario. In practical applications, however, users often encounter open-domain settings in which the target classes of sub-task are not contained in y. Such cases fall within the scope of transfer learning. To investigate Vulcan's effectiveness in this context, we employ DeiT-Base pre-trained on ImageNet the base ViT and evaluate Vulcan-derive three downstream benchmarks: Stanford Cars (Krause et al., 2013), Oxford Flowers- 102 (Nilsback & Zisserman, 2008), and Food- 101 (Bossard et al., 2014).
models under different pruning rates
Table 13: Overall performance of Vulcan under different pruning rates on downstream tasks. The classes and data distribution of downstream tasks are different from the pre-training data of the base VIT. '(FT)' denotes fine-tuning on the downstream task data.
Methods
Stanford Cars
Oxford Flowers-102
Foud-101
Arg
DeiT-Base
86.81
Volca (0.60)
79.62
86.40
80.60
82.21
Vulcan (0.80)
66.70
78.
74.25
73.04
As shown in Table 13, at pruning rates of 0.60 and 0.80, the edge ViTs derived by Vulcan achieve 94.34% and 83.82% of the accuracy of models obtained by directly fine-tuning the base ViT. This performance is somewhat lower than the results in Table 1, the three selected downstream tasks
29

---

Published as a conference paper at ICLR 2026
all belong to fine-gr the-grained image recognition (FGVC), which are inherently more challenging and complex than the ImageNet sub-tasks. Nevertheless, the results clearly demonstrate the strong gen- eralizability of Vulcan and its applicability to more difficult open-domain scenarios.
J.7 WALL-CLOCK COST ANALYSIS
In this section, we present several implementation optimizations and provide a comprehensive anal- ysis of Vulcan's wall-clock cost and scalability.
To begin with, our implementation incorporates several efficiency-oriented design choices:
CCNC: For each FFN block, concatenate the indices of all clusters into a single tensor,
allowing all difference and aggregation operations to be computed one pass.
TNNR: Instead of performing SVD directly on W, we compute the eigen decomposition of WWT, which is significantly more efficient. We also process all attention heads jointly through batched matrix operations.
With these optimizations, Vulcan does not introduce heavy computation in practice. To quantify the actual overhead, we systematically evaluated how sub-task, pruning rate, and model size affect the time required to compute the CCNC and TNNR regularizers. The wall-clock durations were measured on an NVIDIA A40, and the detailed results are provided in Table 14-16 (all numbers in these tables are measured in seconds).
Table 14: Wall-clock time per training step for
Table 15: Wall-clock time per training step for computing the CCNC and TNNR regularization terms at different pruning rates when Vulcan de- rives models from DeiT-Base for the ImageNet sub-task T1/25.
computing the CCNC and TNNR regularization terms when Vulcan derives models from DeïT- Base for ImageNet sub-tasks T1/25-T3/25 pruning rate of 0.60.
Loss Term
T1/25
T2/25
T3/25
Avg
Loss Term
0.20
0.40
0.60
0.80
CCNC
0.07
0.07
0.07
0.07
CCNC
0.06
0,07
0.07
0.06
TNNR Total
0.50
0.49
0.49
0.49
TNNR
0.50
0.49
0.50
0.49
0.57
0.56
0.56
0.56
Total
0.56
0.56
0.57
0.55
Table 16: Wall-clock time per training step for computing the CCNC and TNNR regularization terms when Vulcan derives models from different-sized base ViTs for ImageNet sub-task T1/25 at pruning rate of 0.60.
Loss Term
DeiT-Tiny (5M)
DeiT-Small (22M)
DeiT-Base (86M)
ViT-Large (307M)
CCNC
0.02
0.04
0.07
0.10
TNNR
0.12
0.25
0.50
1.23
Total
0.14
0.29
0.57
1.33
For example (Table 14), when deriving compact models from DeiT-Base on an NVIDIA A40, Vul- introduces only 0.56s of extra time per step compared to fine-tuning with the original task loss alone. From these results, combined with Vulcan's 's design characteristics, we summarize the follow- ing observations:
The cost of CCNC and TNNR is largely insensitive to the choice of sub-task or pruning rate.
The overhead grows sub-linearly with model size.
The computation depends only on model weights, and is therefore independent of the batch size. This implies that in real deployment scenarios, the relative overhead of Vulcan becomes smaller for larger batch sizes and larger models.
30

---

Published as a conference paper at ICLR 2026
J.8 TRADE-OFF BETWEEN PERFORMANCE AND OVERHEAD
Effeets of Batch Size and Learning Rate. Considering that practical applications often impose varying constraints on memory usage and derivation efficiency, we investigate how batch size (bz) and learning rate (Ir)-two two critical training configurations directly related these factors affect the performance of Vulcan. Specifically, for the three ImageNet sub-tasks T1-T3 defined in Table 1, Vulcan derives models from DeiT-Base under different batch sizes and learning rates. We record memory usage and the accuracy of the derived models across varying batch sizes, well as the convergence speed of the derivation process under different learning rates.
Accuracy
Memory
100
92.92
80
90
Memory (GB)
Accuracy (%)
80
60
Accuracy
70
60
50
40
40
30
Ir=le-4
5e-4
Ir-1e-3
32
64
128
256
0
1000
2000
3000
Batch Size
Step
Figure 22: Effects of batch size and learning rate on edge ViT's derived by Vulcan. The asterisks in the right figure mark the step-accuracy points at convergence for each learning rate.
As shown in Figure 22, Vulcan exhibits robustness to batch size: memory usage is reduced by 86.34% with only 4.59% average drop in accuracy. For the learning rate, convergence is defined as accuracy fluctuations within 0.1% over 100 steps. When the learning rate is set to x 10-4 and 10-3 derivation efficiency improves by 57.14% and 65.44% with accuracy losses of 3.02% and 6.05%, respectively, indicating that higher learning rates can accelerate derivation at modest accuracy cost.
Table Compatibility of Vulcan with LoRA (r=256). The edge VTTs are derived from DeiT-Base with pruning ratio of 0.60, tailored for the three ImageNet subtasks T1-T3.
Acc. (%)
Methods
Memory (GB)
T3/25
Avg
Vulcan
1.67
95 95.04
96.24
95.53
95.60
Vulcan (LoRA)
1.10 (334.139%)
88.32
89.75
89.05
89.04 (16.569%)
In addition, we can also extend Vulcan to LoRA (Hu et al., 2021) to reduce memory usage. We observe that post-training with LoRA requires a relatively larger learning rate to achieve stable convergence. Accordingly, we set the learning rate to 1 x 10-3 As shown Table 17, when adapted to LoRA (r=256) with batch size of 4 and gradient accumulation steps of 64, Vulcan incurs an average accuracy drop of 6.56% while reducing memory consumption by 33..... We must acknowledge that LoRA inevitably introduces degree of accuracy degrada- tion, optimizing two low-rank matrices alone cannot fully satisfy the constraints associated with Loollapse and Lrank- Consequently, these losses do not monotonically decrease during the later stages of post-training: instead, they tend to increase monotonically with larger values of Y and X2. Nev- ertheless, the derived model still substantially outperforms the base ViT (DeiT- BaiTe, achieving an average improvement of 8.01% % across the three sub-tasks.
Compatibility of Vulcan with LoRA.
J.9 EXTREME MODEL COMPRESSION
As discussed in Section 2, Vulcan is orthogonal to other model compression techniques and can be combined with them enable even lighter deployment. To demonstrate this, take the edge ViT
31

---

Published as a conference paper at ICLR 2026
MÆ/TI derived from Dese-Base at a pruning rate of 0.80 for the ImageNet sub-task T1/25, and further compress it using knowledge distillation (KD) and INT8 quantization.
Table 18: Results of combining Vulcan with depth pruning, knowledge distillation, and FP16 quan- tization on ImageNet T1/25. With Vulcan as the core, integrating other compression techniques enables effective lightweighting.
Latency (ms)
Throughput (imagels)
Storage (MB)
#Param
#FLOPs
Aec.
Methods
(M)
(G)
(%)
DeiT-Base
45.45
274.27
22.00
933.39
330.23
2.21
86.57
17.57
Vulcan (T1-0.80)
15.04 (3.02x)
99.14 (2.77x)
66.48
2581.97
69.48 (178.9666)
18.21
3.49
92.24
Depth Pruning
(3.45%)
(3.33x)
75.96
3112.8
60.24 (181.76%)
15.79
3.06
92.32
Quantization (FP16)
11.35 (4.00xx
28.34 (9.68x)
88.07
9033.06
30.12 (490.88%)
15.79
92.32
Specifically, we sample 5,000 samples from the T1/25 training set to evaluate the accuracy change after removing each block of MM/TTI- Following greedy strategy, at each step we remove the block that incurs the smallest accuracy drop. Through this depth pruning process, three blocks are removed from ME/TI to construct student model for knowledge distillation. Next, we take the DeiT-Base fine-tuned on the T1/25 training set as the teacher model and fine-tune the student model for 5,000 steps using logit-based knowledge distillation with a learning rate of le-5, As shown in Table 18, the distilled model achieves additional 0.08% accuracy gain. We then apply FP16 quantization to the distilled model, further reducing storage cost and improving inference efficiency. Overall, this pipeline reduces the model's storage cost by 90.88%, improves accuracy the sub- task by 12.40%, and achieves 4.00x and 9.68 inference speedups on Orin NX and RTX 4090, respectively, substantially enhancing the deployability of the model.
32

---

Published as a conference paper at ICLR 2026
K
PSEUDOCODE
Algorithm Vulcan Class-Specific Model Derivation
Input: pre-trained base ViT MA, sub-task S, class-specific dataset Ds, pruning rate R, Lagrange multiplier Åι X2, penalty parameter p
Output: compact class-specific edge VIT ME
1: Determine the target model architecture
2: (all)) hiigunsise t MBsD))
b obtain the activation values of the neurons in W((A)) 3: (K JisIs +- Compute ((()) according to (a(t.)) hsts,,iiss and Eq. (3)
4: {c hSISLISKKKK + Perform clustering based on {K JSsILL
5: (8xx dissh. (1(0) (Rrohsiss + Compute 12(1) @o according Eq. (5)
6: # Post-training
7: ^1 A2 < 0.0,0.0
8: step 0
9: total step + 6250 + 1250R
D empirical relationship between total step and R
10: while true do
11: for t-th batch B, in Ds do
(al-") hisss.iiiss
Forward propagation MB (B.)..)
(!) Update the anchor neuron n of each cluster (!)) to the neuron with the max activation t Compute L according to X1, A2 and Eq. (8)
13:
15:
Backward propagation on L and update parameters of MB using optimizer &c
16:
A1, + \1,2 + Paxi.
b update the Lagrange multipliers via gradient ascent
17:
if step 2 total step then break
end for
19: if step 2 total step then break
20: end while
# Pruning
22: Me U' w U . wilhh) w + MB
23: for I-th block in Me do
24: Create the weight matrix wi E R xd whll e kdxx
(' wr(0)/
Pruning according Eq. (9)
26: for fi-th head in /-th block in MM do
df v/ + Le R@*x )1, [vx R())) 28: Create the weight matrix (((h)) RRixd wilkkiy Ruix wilkhy Rdxe
27:
(LA))) w-( w.l.yy ((b)) + Pruning according Eq. (10) and Eq. (11)
end for
31: end for
ME (W w 1 (
(Waa)) AAA w withh w w((kh)
return Mg
33

---

Published as a conference paper at ICLR 2026
L NOTATION TABLE
Table 19: Notation Table
Symbol
Description
N
Number of patch tokens the sequence, obtained by partitioning the input image signal into fixed-size patches.
d
Embedding dimension of each patch token, indicating the length of the feature vector associated with every token. Patch token sequence sented by d-dimensional embedding vector.
X
IR Nxd, consisting of N tokens, each repre-
MHA (
Multi-Head Attention module in the I-th VIT block.
Atm (+)
The attention operation of the h-th head in the (-th block.
Hi
The number of attention heads in the 1-th block.
q
Dimensionality of the query and key vectors (query-key dimension) in the I-th block.
Dimensionality of the value vectors (value-output dimension) the 1-th block.
w
Query projection matrix w-(l.) E Rexd xd for the h-th attention head in the I-th block.
Key projection matrix (t(.k) E Rexd for the h-th attention head in the I-th block.
w(..)
Value projection matrix (.(.N) Roid for the h-th attention head in the I-th block.
w().))
Output projection matrix w!l.h) Rdxx for the h-th attention head in the I-th block.
FF)
Feed-Forward Network module in the 1-th ViT block.
e
Intermediatelexpaasion dimension of the FFN in 1-th block, correspond- ing to the number of neurons in the (FN
w
First FFN projection matrix Wr(O) E Reex in the I-th block.
w
Second FFN projection matrix W;O)) igdxe in the (-th block. The i-th row of W.
w
W
The i-th column of W.
W[:
Submatrix of W consisting of its first rows (row slice).
W[::
Submatrix of consisting of its first columns (column slice).
x())
Nonlinear activation function applied within the FFN module (e.g..
GELU).
Outer product operator.
(1.0)
The i-th neuron (1.5) w (0 E TRd (row vector) in W(".
n2
(1.))
The i-th neuron 72
((.3) [ Rd (column vector) in W
(1.5)
Activation value (1.1) E/ o ((b)), of the i-th neuron
(1.4) in the FFN the 1-th block. Overall pruning rate R (0,
R
(AA,))
Function that returns the k-th smallest element in the set A. Indicator function that outputs if its condition is true and 0 otherwise.
I(-)
K
Number of clusters formed in the FFN of the 1-th block.
c(l)
The k-th cluster c { ) in the FFN of the the block.
(..) d())
The activation value of the i-th neuron within cluster e!l)
ag
[M]
The collection of the first M positive integers 21, ... ,N..
34

---

(1) Published as a conference paper at ICLR 2026
Table 20: Notation Table2.
Symbol
Description
The index of the neuron within cluster c that attains the max- imum activation value, i..., 1" arg max (1,3)
dk
xellok
(1.0)
The i-th neuron ni n1 E F in cluster & The anchor neuron f = A
(1.0) ((,)
ng.
(1,4") of cluster c()
n
(1,0)
The value associated with the i-th neuron in cluster C repre-
senting either its weight vector n
(1,3) or activation 0k
(1.3)
The value associated with the anchor neuron in cluster C rep
resenting either its weight vector f1 (1,s") activation dx
(1.²*)
w
The query-key composite matrix for the h-th head in the 1-th block, defined W w (
w.l..) E Rdxd
w/o
The value-output composite matrix for the h-th head in the 1-th block, defined v4 w ((l.)] w-ll.k) Rdddd
E(W)
The effective rank of matrix W.
(1)
TQK
Dimension-level effective rank of the query-key composite the I-th block, computed as the sum over all Hi heads of the effective ranks E(WQX)
(
Tvo
Dimension-level effective rank of the value-outpin composite in the I-th block, computed as the sum over all Hi heads of the effective ranks ((W(())).
VO
R0)
Pruning rate of the MHA module in the (-th block.
B
Pruning rate of the query-key (QK) dimension in the MHA module of the 1-th block.
Ryo
Pruning rate of the ouluu (VO) dimension in the MHA module of the I-th block.
(1,h,)
The i-th singular value of W (I.))
dQ
Q
(.h.a))
The i-th singular value of W (1.A)
oy
V (1.b)
QQK _Q Qk
(L.)) (L.))
(L.)
The left singular vectors U
E Rdxd, singular values (L.)) gdxd, and right singular vectors ((th) dxdd of
the wilk)) obtained by its singular value decomposition.
((hh) ((L)) (((A)
The left singular vectors ((l.k) Rdxd, singular values x((hh) e Raxd and right singular vectors v((hh) Rdxd of
vo .VO
the wi(Lk) obtained by its singular value decomposition.
35


Reviews:
[{"review_id": "rQDDlW5yKR", "reviewer": "Reviewer_pANN", "summary": "The paper targets class-specific edge deployment by compressing general ViTs into compact, per-class models. It hypothesizes FFN layers are more class-specific while MHA is class-agnostic, and proposes a train-then-prune pipeline: CCNC for FFN and TNNR+SVD for MHA under FLOPs/parameter budgets. Across five ViTs, three tasks, and four datasets, the approach keeps only 20–40% of parameters while matching or improving base accuracy, outperforming structured pruning baselines by up to 13.92%, with hardware-friendly dimensional alignment.", "questions": "I see you have covered many experimental results in the appendix. I don't have more questions. Good luck.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper systematically formulates class-specific ViT compression for edge, supports the FFN/MHA functional split with analyses, and pairs CCNC (FFN) with TNNR+SVD (MHA) under explicit budget constraints.\n2. Introducing redundancy and then pruning by effective rank/activation clustering mitigates irreversible knowledge loss typical of prune-then-train; results show consistent gains at 0.6/0.8 pruning ratios over strong structured-pruning baselines.\n3. Budgets are enforced in GFLOPs/params, post-pruning dimensions are aligned to hardware-friendly multiples (e.g., ×8), and SVD on attention heads is shown competitive with saliency-based pruning.", "weaknesses": "1. It could be very interesting if this framework could be extended as a streaming service platform which can publish tuned weights to edge models in real time.\n2. Although some hardware cannot support numerical types, the combination of the proposed method and quantization could be important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "06QSOoQbW7", "reviewer": "Reviewer_QM9G", "summary": "The paper proposes Vulcan, a framework for class-specific pruning of pre-trained Vision Transformers (ViTs). In ViTs, the FFN layers primarily capture class-related features, while the MHA layers tend to preserve class-agnostic information. Based on this property, Vulcan applies different compression strategies to the two components: redundant neurons in FFN layers are merged through neuron clustering and collapse, and attention matrices in MHA layers are regularized with truncated nuclear norm (TNNR) to induce low-rank structures. After a post-training stage, the framework directly derives a compact ViT specialized for a subset of target classes. Experiments on multiple datasets demonstrate that Vulcan improves classification accuracy under high pruning ratios and achieves notable speedups.", "questions": "1. It is not specified whether the baseline models were also retrained on the same class subsets—could this affect the fairness of the comparisons?\n2. The paper does not compare Vulcan with lightweight models designed for limited-class scenarios; would Vulcan still provide a clear advantage under such settings?\n3. The FFN collapse and MHA low-rank compression losses are directly added together—could there be potential interactions between the two, and is such joint optimization theoretically justified?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The experimental evaluation is extensive, covering multiple datasets (ImageNet, CIFAR-10/100, COCO) and tasks (classification and detection). Results are reported under various pruning ratios with comprehensive metrics including accuracy, FLOPs, parameter count, and hardware latency.\n2. Implementation details are clearly presented, making the method reproducible. The post-training behavior, hyperparameter sensitivity, and neuron visualization provide additional insights into the framework.", "weaknesses": "1. The overall idea builds on the known observation that FFN layers in ViTs capture class-related information, and both the neuron clustering and low-rank decomposition techniques are well-established; thus, the level of novelty is limited.\n2. The application scenario involves only a small number of target classes, which could often be addressed by lightweight models trained on smaller datasets, leaving the necessity of this approach unclear.\n3. The method section contains an excessive amount of symbols and equations, making the presentation somewhat dense and difficult to follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FCIi34GkzL", "reviewer": "Reviewer_J15G", "summary": "This paper focuses on the class-specific requirements of deploying ViT on edge devices. Many edge scenarios require only a small number of target classes to be recognized, but the large amount of irrelevant class knowledge carried by pre-trained ViT consumes resources and can interfere with target class performance. The authors claim that by analyzing the knowledge distribution of ViT modules, they find that FFNs favor class-specific knowledge, while MHAs favor class-independent patterns. Based on this, they propose a train-then-prune post-training framework Vulcan. Vulcan introduces redundancy into ViTs deliberately by collapsing FFN neurons onto those with the highest class-specific activations and by enforcing low-rankness in MHA weights.", "questions": "1.\tThe paper uses visualization and the phenomenon that \"data-free SVD is better than score-based pruning\" to support the claim that \"MHA is more class-agnostic \", but lacks more rigorous verification. Can some other models or datasets be used to further support this observation?\n2.\tCan the authors report the relative training time/GPU memory overhead of TNNR (relative to fine-tuning only the task loss) on different platforms/datasets?\n3.\tThe random subtask is only divided into 3 categories. Can you provide more mean ± variance or confidence intervals of the random divisions, especially under high shear rates of 0.8/0.9, to prove stability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe problem studied in this paper is meaningful, namely, achieving lightweight model deployment at the edge. The authors also have a good insight that not all types of information are useful.\n2.\tCCNC clusters FFN neurons by \"activating the highest anchor point\", and TNNR adaptively distributes pruning rates by effective rank. Both are consistent with the observation of \"FFN class specific/MHA class independent\"; they are unified in the augmented Lagrangian objective, which facilitates direct implementation of structured pruning.\n3.\tThe authors conducted sufficient experiments to demonstrate the effectiveness of the proposed framework.", "weaknesses": "1.\tThe paper uses visualization and the phenomenon that \"data-free SVD is better than score-based pruning\" to support the claim that \"MHA is more class-agnostic \", but lacks more rigorous verification. Can some other models or datasets be used to further support this observation?\n2.\tTNNR introduces additional computations of singular values/effective ranks, which may incur significant overhead; the paper does not provide data on wall clock duration and scalability for large models/big data scenarios.\n3.\tThe author can provide a rough sketch of the theoretical analysis in the main paper to improve the completeness of the entire work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K3XtkFLbKL", "reviewer": "Reviewer_Z9iw", "summary": "The paper introduces Vulcan, a pruning-oriented post-training framework for deriving class-specific compact Vision Transformers for edge deployment. The key insight comes from that FFN neurons encode class-specific knowledge, while MHA layers capture class-agnostic patterns. Based on this, Vulcan proposes two complementary techniques: \n(1) CCNC clusters FFN neurons by activation and collapses them toward anchor neurons, and \n(2) TNNR, which induces low-rank structure in MHA weights to enable near-lossless SVD pruning. \nA unified train-then-prune optimization via augmented Lagrangian enforces these redundancies.", "questions": "1. Could CCNC overfit to small subtasks if activations are dominated by a few classes?\n2. How sensitive is the neuron clustering step to dataset imbalance or class overlap?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Edge deployment with individualization is an interesting and important problem in edge cloud cooperation, and the authors provide a practical and effective pipeline solution.\n2. The insight on knowledge disentanglement between FFN and MHA modules is convincing and well-supported by visualization and interpretability experiments.\n3. Experimental evaluation is comprehensive, covering multiple architectures (DeiT, Swin), tasks (classification, detection, segmentation), and hardware setups (Jetson, RTX 4090).\n4. The ablation studies and visual analyses are comprehensive.", "weaknesses": "1. The writeup is somewhat complicated. Although the steps and concepts are strictly formalized, there is little intuition or explanation, making the formulations and notations hard to follow. For example, in the term $a^{(l,i)} = \\sum_i \\sigma(X[i] \\cdot n^{(l,i)}_1)$ , it seems  $i$ is summed out on the right-hand side, but the left-hand side still includes $i$ , which is confusing.\n2. While conceptually clear, the method section is dense and overly mathematical, which makes it difficult to grasp the main ideas of the augmented Lagrangian optimization.\n3. The methodology feels more like a collection of engineering techniques combined into one pipeline rather than a focused research exploration, with too many tricks applied at once.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)