Elastic ViTs from Pretrained Models without Retraining
Authors: Walter Simoncini
Michael Dorkenwald
Tijmen Blankevoort
Cees G. M. Snoek
Yuki M Asano
Venue: NeurIPS 2025 poster ()
Keywords: Elastic Inference
Elastic ViTs
Pruning
Primary area: deep_learning
TL;DR: We propose a method to enable elastic inference for pretrained ViTs via structured pruning. We do not require labels, generalize to models without a classification head and do not need re-training. We improve over the existing state of the art.

Abstract: Vision foundation models achieve remarkable performance but are only available in a limited set of pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: single-shot network approximation for pruned Vision Transformers, a new post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via an evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining-free. Experiments on DINO, SigLIPv2, DeIT, and AugReg models demonstrate superior performance over state-of-the-art methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an efficient pruning strategy for pretrained Vision Transformers, a novel evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: https://elastic.ashita.nl/

Elastic ViTs from Pretrained Models without Retraining
Walter Simoncini!
Michael Dorkenwald"*
Tijmen Blankevoort
Cees G.M. Snoek?
Yuki M. Asano'
University of Technology Nuremberg
University of Amsterdam
NVIDII
Abstract
Vision foundation models achieve remarkable performance but are only available in a limited set pre-determined sizes, forcing sub-optimal deployment choices under real-world constraints. We introduce SnapViT: single-shot network approxi- mation for pruned Vision Transformers, new post- post-pretraining structured pruning method that enables elastic inference across continuum of compute budgets. Our approach efficiently combines gradient information with cross-network structure correlations, approximated via evolutionary algorithm, does not require labeled data, generalizes to models without a classification head, and is retraining- -rree. Experiments on DINO, SigLIPv2, Del', and AugReg models demonstrate supe- rior performance over state-of-the-an methods across various sparsities, requiring less than five minutes on a single A100 GPU to generate elastic models that can be adjusted to any computational budget. Our key contributions include an effi- cient pruning strategy for pretrained Vision Transformers, novel evolutionary approximation of Hessian off- off-diagonal structures, and self-supervised importance scoring mechanism that maintains strong performance without requiring retraining or labels. Code and pruned models are available at: httpss//clastic.ashiaaa
1 Introduction
Recent advances in model architectures and training recipes have enabled the training of vision foundation models with several billion parameters [12 53 16 60), achieving sfate-of-the-art results across a wide range of tasks. However, these models must operate under strict compute, latency, and cost constraints when deployed in real-world settings. Yet, only limited
model sizes e.g., 21M, 29M, 86M, 300M, 840M and 6.7B parameter vision transformers (ViTs) for the DINOv3 family [15 [9] are made available, forcing users select the largest model that still fits their requirements, a choice that can often be sub-optimal.
Traditionally, challenges related to model flexibility have been tackled using knowledge distillation 27]. However, this strategy mandates a predetermined target architecture and relies on often non- public pretraining datasets, which can in turn undermine robustness and limit flexibility. In parallel, methods for elastic inference 14 multiple sub-networks inference time. Yet, these methods necessitate networks designed with a predefined structure, such as nested Matryoshka [34], and require such structures to be present during pretraining. This dependency restricts their applicability existing or proprictarily pretrained models.
29 70 have emerged to enable dynamic selection among
An attractive alternative is structured pruning 37 23
technique that reduces memory and computational requirements, enabling models to be adapted to diverse deployment settings. Despite their promise, most pruning techniques tailored to specific compute constraints and tasks [35 83
24
Denotes equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and typically require retraining, leaving a significant gap for developing universally adaptable models. To bridge this gap, we propose novel structured pruning method, SnapViTI that operates in a post- pretraining setting and enables elastic inference. By that, we can extract continuum of sub-networks from a single pretrained model, thereby enabling users to precisely tailor state-of-the-art models to their computational budget and task.
To this end, we introduce a prunability score that enables effective pruning across varying sparsity levels in a single shot that is extremely fast (less than five minutes on one A GPU). This score facilitates selective pruning of transformer components (e.g., row-column combinations within blocks (14)) and larger structures, such as entire attention heads. Specifically, our component, in line with previous works (37 35 [0], and (ii) a novel cross noossne correlation score that approximates parameter sensitivity, as captured by the off-diagonal elements of the Hessian. While prior approaches have largely ignored this component due to its quadratic scaling with the number of parameters, we propose an efficient approximation using an evolutionary algorithm to estimate these correlations. Moreover, by basing the gradient term on self-supervised loss, our method works on any pretrained model without requiring a classification head and generalizes well across diverse downstream target datasets.
prunability score composed of two terms: (i) gradient-
We evaluate pruned models across eight datasets, including ImageNet- Ik, via k-nearest neighbor classification (k-NN), linear probing, and linear semantic segmentation using small and large ViTs from the DINO [9] [59, AugReg [61], DelT [64. [6], and SigLIPv2 [66] model families. Our method consistently matches or outperforms state-of-the-arr approaches such as LAMP 38 the LLM Surgeon 67 FPTP B5 SparseGPT [8]. SNIP Magnitude [31, and NViT (76 across various sparsity levels, while generating all sparsities in single shot. particular, our method can prune DINOv1 ViT-B/16 to 40% sparsity, accelerating inference by 1.58x while keeping the accuracy degradation below 5%. We also show that our method is compatible with post-pruning weight correction and full fine-tuning, and outperforms or is competitive with state-of-the-art approaches in these setups. Finally, we demonstrate the importance modeling cross-network correlations through multiple ablations and visualize the sparsity distribution across the network. Our contributions can be summarized as follows:
We introduce an effective, fast pruning strategy for pretrained ViTs, yielding elastic models that can adapt to any computational constraints.
We propose novel strategy to approximate the off-diagonal components of the Hessian for network structures using genetic algorithm.
We obtain of-e-of-theear performance under considerable pruning without retraining requiring any labels.
2 Related work
Network efficiency To improve model efficiency, several techniques have been proposed, including pruning [22). quantization [52], and knowledge distillation (27). Pruning, in particular, aims at eliminating the "unimportant" bits of the network while preserving model performance. Most pruning research [25] 2 771 39 has focused on CNNs for image classification. Pruning methods can be classified into unstructured [22], removing individual weights yield irregular sparsity and high compression, often requiring specialized hardware to realize speedups, or as structured [39], eliminating entire filters, channels, or other structures to enable practical acceleration on standard hardware. Finding the "unimportant" parts of the network has been done based on weight magnitudes [3
23
84
43
activations [62 81 82), gradients 777 (56). or the model's 's Hessian [37 24 67
68
The latter is the most accurate, as it accounts for all second-order dependencies however, computing the full Hessian infeasible. Several tractable approximations have been introduced, such as diagonal [37, 63), block diagonal [36] 18) and block diagonal with K-FAC
35
63
45
69
37
67 68). While these approximations efficiently capture local and intra-layer interactions, they inherently disregard inter-layer dependencies. To overcome this limitation, we introduce black-box evolutionary algorithm that circumvents the need for explicitly computing the Hessian and can model intra-layer dependencies.
Elastic inference The idea of extracting multiple smaller models from a single larger model
has been widely explored [79 78,, 5 20 6), mostly in the context of CNNs. OFA [5] trains
2

---

Local Hessian
Global Hessian
Prunability Score
Elastic Pruning
Estimate curvature from self supervised gradients
Model cross-netword correlations using xNES
Fuse local and global scores in a unified importance measure
Rank once to derive subnetworks for any arbitrary sparsity
Figure 1: Overview of our pruning method. We decompose the Hessian into local and global components: the local Hessian is approximated from self-supervised gradients, while the global Hessian models cross-block correlations learned via evolutionary algorithm. Combining both yields a unified prunability score that ranks parameters once, allowing single-shot generation of sub-networks at any desired sparsity. The pseudocode for our algorithm is listed in Appendix [D22
teacher CNN model and employs distillation to fine-tune randomly sampled, non-nested submodels within a universal student model. Slimmable networks [79] jointly optimize models but offer only a limited set of predefined widths. Universal Slimmable Networks [78] extend this concept by allowing sampling from a continuous search space of submodels and jointly optimizing them. HAT [70] trains a universal network solely to learn the relative performance of different architectures; however, requires NAS to identify the optimal architecture and trains it from scratch before serving. DynaBERT [29) jointly trains fixed set of submodels but lacks search strategy, limiting its approach to explicitly trained granularities. Matryoshka representations [4 can adapt to diverse downstream tasks while accommodating varying computational constraints through nested representational structure. Various works have leveraged such a nested structure for multi-modal [7], encoder-only decoder-only (14), diffusion (21). and state-space (58) models. Notably, Matformer (14] trains transformers with this specific structure for the feed-forward part of transformer blocks from scratch, yielding versatile model.
In contrast to these works, we explore how to derive an elastic Vision Transformer model from any pretrained network without specialized pre-training retraining. Our method can prune both feed-forward blocks and attention heads, and requires less than five minutes on a single A100 GPU,
3 Method
Our method enables single-shot generation of sparse subnetworks from pretrained models, regardless of computational budget. To this end, assign prunability score P to each network structure, e.g., row-column combinations within feed-forward blocks (14) or entire attention heads, and remove the least important ones meet the computational constraint. Our method is summarized in Figure
3.1 Blockwise Hessian decomposition
The importance of a parameter can be expressed by the change it induces in the objective function when perturbed or removed. While directly measuring this effect ideal [42), infeasible for large-scale networks with billions of parameters N. Following (37 24, we approximate the loss variation under a small perturbation 80 using a second-order Taylor expansion
8L = VoL 80
150 H50 ((1801)))
where H is the Hessian of L with respect to 0. Assuming the model is near a local minimum 37 24
351. the first-order term vanishes (VoL 50 0), leaving the Hessian as the dominant indicator of sensitivity and parameter coupling. Each entry H.j = 80,00; quantifies how parameters 0 and 0g interact; The off- diadoiagonal elements thus capture correlations and redundancy across parameters.
a²c
However, computing the full Hessian intractable since it contains 2 entries. Practical approxima- tions, such as diagonal 337. block-diagonal [35], or Kronecker-factored (KFAC) [67] 68), reduce cost but capture only local dependencies, e.g., within single transformer block. We instead approximate composition of local term (!), capturing intra-block structure, and global the Hessian (99), which modulates sensitivities across B functional units (e.g., attention heads correlation term or MI.P blocks). This formulation provides a scalable, data-driven representation of both local and inter-layer dependencies without explicitly forming the full 22 Hessian.
3

---

We next approximate the local curvature term HO using self-supervised gradients, which yields an efficient diagonal estimate of parameter sensitivity before learning global correlations in Sec 3.3
3.2 Local Hessian approximation using SSL
We first estimate the local curvature of the loss surface following 24
80
as
1 No
IVo.
ND
computed over a dataset D with ND samples while retaining only the diagonal entries of the Hessian (see Fig. [ bottom). This diagonal approximation captures parameter-wise sensitivity within each block and provides efficient proxy for the local curvature H To obtain gradients in a model- agnostic way, we adopt the self-supervised DINO objective [9], which removes dependence on a classification head and allows pruning of both supervised and foundation models. For an input image, we sample ny global and ny local crops, compute their normalized embeddings 8 and 2 and minimize the cross-viee consistency loss
ng
12g
SSI.
(C 2/m))
where LCE denotes the soft cross-entropy between teacher and student embeddings.
The resulting diagonal curvature H serves baseline measure of local parameter sensitivity. In the next stage, xNES learns structure- wise scaling factors e TR that rescale these sensitivities across network structures based inter-block correlations.
3.3 Global Hessian estimation via xNES
Even with structure-wise grouping of (9)), computing all structure-level Hessian entries remains infeasible. Instead, we employ the Exponential Natural Evolution Strategy (xNES) 19 to model these interactions implicitly, without explicitly forming the Hessian. By doing so, we can simulate pruning and measure sensitivity directly, which has been shown be more reliable than pure analytic approximations 42.
The diagonal Hessian estimate H (1) from Sec. 3.2 provides baseline sensitivity for each param- eter. During the xNES optimization, we combine these local scores with sampled global factors N(u, Σ), which represent structure-wis reweightings. Each candidate rescales the local sensitivities to produce trial pruning mask; this mask evaluated using a label-free fitness metric, allowing the covariance Σ to evolve toward the inverse of the true inter-block Hessian. This coupling ensures that the global correlations learned by xNES are grounded in the local curvature of the pretrained model.
We parameterize the search distribution as multivariate Gaussian W(,, Σ) over potential solutions, with exponential parameterization of the covariance matrix Σ = BB where B e xNES performs natural-gradient updates on both the mean and the covariance
Ay = The V nat [u, E),
(4)
# D not E),
AA
(5)
where (1,, Σ) = Ee-N(pp.) [F(v) and F' denotes the fitness score. To measure the fitness for each sampled c, we prune the model according the rescaled local sensitivities and evaluate the resulting representation quality. We compute embeddings 2 from the original model and 2p. from the pruned model sparsity S, compress them via PCA 192 dimensions, and measure cosine similarity
F
Σ sim((C)()), PCA(Ep.)) SES
PCA
(6)
|S
The natural-gradient update then adjusts Σ such that its inverse reflects which blocks can be pruned jointly without degrading this similarity.

---

K-NN AugReg VIT-B/16
Linear AugReg VIT-8/16
02
72
40
52
315
318
K-NN DelT VIT-B/16
Linear DelT VIT-B/16
Avg.
75
72
55
42
34.1
GFLOPs
GFLOPs
LLM Surgean
FPTP
SparseGPT
Magnitude
NVIT
(SSL)
Figure 2:
Our method matches improves upon the state-of-the-art in retraining-free
setup, while not using labels. Top-1 accuracies k-nearest neighbor and linear classification averaged across 7 datasets for supervised AugReg and DelT ViT-B/16 models. Our label-free method outperforms or matches baselines that utilize labels, especially at high sparsity ratios.
We interpret the evolving covariance data-drive proxy for the inter-block curvature,
H (9) a-
where a is scaling constant that does not affect the relative importance of correlations. Although this is an approximation, prior analyses [57 I show that evolution strategies naturally adapt their covariance to the inverse Hessian on locally quadratic landscapes. In practice, the off-diagonal terms of Σ evolve to mirror cross-block dependencies, providing a tractable estimate of the global Hessian that complements the diagonal local scores.
Intuition. xNES contracts variance along steep directions and expands it along flat ones, repeated updates drive Σ approximate the underlying curvature structure. This makes Σ useful Hessian surrogate for capturing both intra- and inter- block sensitivities, improving pruning robustness.
3.4 Elastic pretrained ViT pruning
Combining the local and global Hessian approximations yields unified prumability score for each parameter. We define as
ND Σ
= diag
Vo C SSL
O Mc.
(8)
ND
where the diagonal term captures local parameter sensitivity and the blockwise scaling vector learned through xNES, encodes global inter-block correlations. The membership matrix M e {0, 1 expands each block factor c to all parameters within its corresponding block, ensuring dimensional consistency with the parameter vector of size N. This reweighting amplifies or suppresses local sensitivities depending on the block's global importance: blocks whose parameters co-vary strongly with others (high off-diagonal curvature) receive larger effective scores, whereas isolated or redundant blocks are deig-weighted.
After computing P, we globally rank all parameters to determine the pruned subset at any desired sparsity level S
Oş = {1) E e rrank(P)) < <001 (1 -S)}
(9)
5

---

K-NN DINO VIT-B/16
Linear DINO VIT-B/16
82
Avg. Accuracy
72
62
50
32
40
GFLOPS
GFLOPs
LAMP SNIP Magnitude (SSL) Ours (SSL)
Figure 3: Our method outperforms baselines for DINO VIT-B/16. Top-1 accuracy in k-nearest
neighbor and linear classification averaged across 7 datasets for models pruned with our method,
LAMP, and SNIP Magnitude. Our method can prune DINO to 40% sparsity with an accuracy degradation under 5%.
k-NN DINOV3 VIT-B/16
Linear DINOv3 VIT-B/16
k-NN SigLiPv2 VIT-B/16
Linear SigLIPv2 VIT-B/16
60
Avg. Accuracy
00-
60-
60
50
40
60-
20
30
70-
35.1
25.3
25.3
31.
25.1
31.8
GFLOPS
GFLOPs
GFLOPs
GFLOPs
LAMP
SNIP Magnitude (SSL)
Ours (SSL)
Ours (SSL) sparsities, 500 GA Iber.
Figure 4: Large-scale pretraining complicates pruning. Top-1 accuracy in k-nearest neighbor and linear classification for pruned DINOv3 (59] and SigLIPv2 ViT-B/16 [66] models. We find that self-supervised models trained on large datasets are harder prune, benefit from longer optimization horizons, and optimizing for more sparsities.
where θ the complete parameter set and P. denotes the score of parameter 0,. This global ranking enables single-shot pruning, as any target sparsity S E [0, can be realized without retraining, Hessian storage, or additional optimization. The same evolutionary run therefore produces a full continuum of compute-adaptive sub-networks.
4 Experiments
We evaluate models pruned at six evenly spaced sparsity levels between 10% and 60%, using k-nearest neighbor (k-NN) classification, linear probing, and linear semantic segmentation. We compare our single-shot pruning protocol against state-of-the-ar multi-shot approaches under the same conditions. Additionally, we assess our method with post-pruning refinements, such as weight correction and full fine-tuning, and benchmark it against prior methods in this setting. We prune hidden neurons in feed-forward blocks (equivalent to a row-column combination) and whole attention heads. We report speedups in GFLOPs. The complete experimental details are described in Appendix [ We evaluate our method on supervised DelT (4) and AugReg (61 models and self-supervised DINO [9] DINOv3 [59) and IIP2 [66] models, comparing it to saate-of-the-art methods such as the LLM Surgeon (67]. LAMP [38]. FPTP 35), NViT (76). SparseGPT [18., and SNIP Magnitude [3I]
All baselines were evaluated using their official implementation, except for LAMP and SNIP Magni- tude, which we implemented in our framework. Notably, all existing methods 38 35
67
76
[8] 31
optimize for a single predetermined sparsity level, whereas our approach optimizes over all sparsity levels simultaneously. We evaluate pruned models on 7 image classification datasets described in Appendix [C] and report the averaged top-1 accuracy, unless otherwise specified. For k-nearest neighbor classification, we utilize the scikit-learn implementation (50) with majority voting and 20 neighbors; for linear classification, we train linear head using stochastic gradient descent for 100 epochs, following the same recipe as DINO and for linear semantic segmentation, we use the same recipe as NeCo [48).

---

AugReg VIT-B/16 K-NN
AugReg ViT-B/16 Linear
63
73-
62
Avg. Accuracy
35.2
31.8
26.3
24,5
21.3
14.3
35.1
31.8
24.3
24.5 GFLOPs
21.3
17.
143
GFLOPs
DINOvI vi1-
SigLIPV2
SigliP2 VIF-8/16
80
60
60
69
40
50
20
24
24
21g
21
27 22,
LLM Surgeon (5 Shot)
SparseGPT
Ours (SSL)
Ours (SSL, Correction)
Figure 5: Weight correction helps retaining post-pruning performance. A single SparseGPT- style weight correction step greatly improves performance high sparsity levels while preserving efficiency. Our method matches or surpasses state-of-the-ar baselines across pruning ratios and preserves self-supervised model accuracy even under extreme sparsity (bottom row).
4.1 Single-Shot Structured Pruning
We evaluate our method in the single-shot structured pruning setting, without any post-processing such weight correction fine-tuning, and compare against statc-of-the-ard basclines. Unlike our approach, most existing methods (67 18 35 [5] assume the availability of a classification head; therefore, we use AugReg ViT-B/16 backbone for fair comparison. In addition, produces all sparsity levels within a single run, whereas each baseline must be executed once for every target sparsity. For NViT, instead use DeiT VIT-B/16 model, their codebase is not easily extensible to other backbones. For self-supervised models, we benchmark against methods that do require a classification head, i.e., LAMP and SNIP-Magnitude, using self-supervised gradients for the latter to ensure consistency.
method
Supervised backbones Figure reports the accuracy in k-nearest neighbor and linear classification for supervised models pruned using our label-free method (Ours SSL) versus other state-of-the-a pruning techniques that make use of labels. The results show that method all baselines, especially at high sparsity ratios, where it improves by 7% and 12.3% over SNIP and FPTP, respectively, at 50% sparsity. Notably, we often outperform the LLM Surgeon, which prunes models to a target sparsity in 5 shots.
match outperform
Self-supervised backbones In Figure and Figure 4 we evaluate our approach on foundation models, including DINOv1 [8] DINOv3 [59], and SigLIPv2 (66) ViT-B/16, and compare it to LAMP and SNIP-Magnitude Since other pruning methods depend classification head, they cannot be applied to these models. Our method prunes DINOv VIT-B/16 to 40% sparsity with less than 5% drop accuracy, achieving 15.1% and 53.2% improvement in linear classification over SNIP Magnitude and LAMP, respectively. We further observe that foundation models trained on large-scale datasets, such as DINOv3 and IigL.Ivv,, with 1.7 and 10 billion training samples, respectively, are harder to prune, benefit from longer optimization horizons (500 iterations versus baseline of 50) and optimizing for six sparsity levels rather than four. Despite this, our method outperforms both LAMP and SNIP Magnitude, improving over the second-best method in linear classification by up to 21.7% and 34.3% for SigLIPv2 and DINOv2 ViT-B/16, respectively.
Semantic segmentation In Figure 6 we benchmark our method in linear semantic segmentation on Pascal VOC 2012 [17] for AugReg and DelT ViT-B/16 backbones, reporting the best performances on the validation set. The results show that our method matches or outperforms the state-of state-of-the-arr, especially at high sparsity ratios, for example, improving by 9.1% over SNIP Magnitude 50% sparsity for AugReg and by 15.3% over NVIT at 60% sparsity for DelT.
7

---

Pascal VOC mlou, AugReg VIT-B/16
Pascal VOC m DelT VIT-B/16
miou
20
25.2
22.3
28.2
24.1
22.3
17.0
1433
29.1
51,8
28.3
24.3
22.3
17.9
143
GFLOPs
GFLOPs
LLM Surgean (5 Shot)
SparseGPT
SNIP Magnitude
Ours (SSL)
Figure 6: Our method retains segmentation performance best. We report the mloU on Pascal VOC 2012 for linear semantic segmentation for AugReg and DeIT ViT-B/16 models.
SigLIP 2 VIT-G/16 K-NN
SigLIP VIT-G/16 Linear
DINOV3 VIT-H+/16 K-NN
DINOV3 VIT-H+/16 Linear
50-
Avg. Accuracy
10
75
60
60
50
50
50
40
Ours (SSL)
60
37
592g
2013
402.9
421
202.8
201.5
20
207.3
2011
2015
222
105.4
12
327.3
2012
212.
1064
1954
GFLOPs
GFLOPs
GFLOPs
GFLOPS
Figure 7: Huge models are not necessarily more prunable. Top-1 accuracy in k-nearest neighbor and linear classification averaged across 7 datasets for DINOv3 V/T-H+/16 and SigLIPv2 ViT-G/16 models pruned with our method. Contrary a DeIT-IIII ViT-H/14, shown in Figure 12 performance quickly degrades beyond 30% sparsity.
4.2 Structured Pruning with Post- Post-Processing
While not a core component of our method, we evaluate performance after a single SparseGPT- style weight correction step or full fine-tuning, the latter performed using the same setup NViT. We compare the performance of our models after weight correction or fine-tuning to comparable state-of-the-ard methods for supervised backbones.
Weight Correction We compare our approach with methods that include a weight-correetion step. Note that we apply weight correction to a single sparsity and thus do not produce clastic models in this setup. We extend our method by applying SparseGPT-style weight correction after pruming with our original algorithm. For each pruned weight matrix, we compute a layer-wise Hessian approximation using input activations collected from 1000 random ImageNet-I4 1k training samples. The inverse Hessian is then used to update the remaining weights to minimize the reconstruction error of the corresponding output activations. This process is applied sequentially, layer by layer, starting from the first Transformer block. The results are shown in Figure 5) where we compare the performance of our method, with and without weight correction, which performs five correction steps instead of one. Our method is either competitive or outperforms the state-of-the-an at all pruning ratios. We also show that single weight-correction step can largely preserve the performance of self-supervised models at extreme sparsity levels. In particular, we can prune SigLIPv2 to 50% sparsity with negligible performance loss in linear classification.
SparseGPT and the L.M Surgeon, the latter of
Full Fine-tuning We also compare our method with prior pruning and model adaptation approaches that rely on extensive fine-tuning, Similar to NViT and SAViT (83), we fine-tune a DeIT ViT-B/16 pruned 50% sparsity for 300 epochs on ImageNet-Ik, using the same recipe as NVIT. We compare our results to the author-reporte linear classification performance on ImageNet- 1k in Table alongside the average linear and k-nearest-neighbor classification accuracies for open- open-weight models. The results show that our pruned model outperforms the unpruned model on ImageNet- 1k after full fine-tuning and is competitive with the state-of-the-art, matching or outperforming it. While NVIT achieves the highest ImageNet- lk accuracy, it generalizes less effectively: on average across our seven benchmark datasets, its k-nearest neighbor and linear classification performance are 1.7% and 3.9% lower, respectively, than that of model pruned with method and fine-tuned.
8

---

Table I: Image Nk full fine-tuning recovers performance for 50% pruning. Our method fully recovers pre-pruning performance on ImageNet- and is competitive with other state-of-the of-the-art approaches on ImageNet-Ie Ik, while generalizing better in k-nearest neighbor and linear classification.
Method
Avg. k-NN
Avg. Linear
ImageNet-
Fine-tuning Epochs
Unprumed
75.8
78.5
81.8
SN-Net 46
70.2
71.4
80.0
100
NViT 775
73.7
72.0
83.3
300
LPViT 774
80.6
300
SAVIT
83
82.6
300
SnapVi1 VIT (Ours)
75.4
75.9
82.6
300
Feed Forward Network
Attention Heads
0%
Sparsity
1.0%
50%
60%
Block
Block 3
Block
Block 7
Block Block 10
Block 11
Block
Block
Block
Block 8
Block 12
Figure 8: Deeper blocks are heavily pruned in DINO ViT-B/16: visualization of the normalized parameter allocation for DINO VIT-B/16 models pruned increasing sparsity levels (0-60%). Feed- forward blocks, especially deep ones (8-12), are pruned before attention heads, while earlier blocks maintain their parameter density, revealing the network's s inherent structural redundancy.
4.3 Pruning big ViTs
Given the efficiency of our method, which only approximates the Hessian, can apply it to prune large ViTs. As shown in Figure 7 we prune SigLIPv2 ViT-G/16 and DINOv3 ViT-H+/16 models, containing 1.2B and 840M parameters, respectively. While both models retain performance up 30% sparsity, their accuracy drops sharply beyond this point. In contrast, our results on DelT-III ViT-H/14 (Figure 12) show stable performance for up to 50% sparsity. We hypothesize that this difference arises from the pretraining regime: DelT--III is pretrained on ImageNet-21k (13M samples), whereas SigLIPv2 and DINOv3 are pretrainedit on substantially larger datasets with 10B and 1.7B samples, respectively. Large-scale pretraining likely distributes representational knowledge more evenly across parameters, making it less obvious which units can be pruned. Nonetheless, combining our method with simple weight correction techniques can recover performance even for models that undergo large-scale pretraining, demonstrated in Figure 5 for SigLIPv2 ViT-B/16.
4.4 Ablations
Sparsity allocation In Figure 8, we visualize the sparsity allocation across blocks for DINO ViT- B/16. The visualization reveals two key patterns: (i) pruming initially favors slimming ferw-fo blocks, leaving attention heads largely intact, though they too undergo pruning at higher sparsity ratios, but lesser extent; (ii) blocks to 12 demonstrate higher pruning susceptibility, suggesting these layers contain more redundant information. As sparsity increases, pruning progressively concentrates in these blocks while earlier blocks maintain relatively stable parameter density.
Importance of global interactions. Table 3 illustrates the impact of approximating the global Hessian (Φ) using more netsskk interactions. particular, we compare the average k-NN accuracy of DINO ViT-B/16 models pruned to 50% sparsity while modeling either interactions (B = 0), equivalent to not using the genetic algorithm, only the interactions between feew-forward blocks (B 12), and interactions between all pairs of feed-forward blocks and attention heads (B = 156). The results show that the performance of pruned models increases as more global interactions modeled, by up to +6.9% in average k-nearest neighbor accuracy.
Sparsity-specifie vs continuous optimization. Figure we compare the performance models pruned with our method while optimizing for each individual target sparsity against our one-shot

---

Table 2: Performance improves with more genetic algorithm iterations. Average accu- racy in k-NN and linear classification versus the number of iterations for an AugReg ViT- B/16 backbone pruned to 50% sparsity.
Table 3: Modeling more cross-network interac- tions improves performance. Average top-1 accu- racy in k-NN versus the interactions modeled and optimized using the genetic algorithm, for DINO ViT-B/16 backbone pruned to 50% sparsity.
Iterations
Avg. k-NN
Avg. Linear
Interactions Modeled
Avg. k-NN
50
39.3
42.2
None (0)
56.6
250
39.9
42.2
FFN (12)
60.1
500
40.9
44.0
FFN and heads (156)
63.5
Target Sparsities
SSL CE Gradients
Avg. K-NN Accuracy
72-
70
60
Target Sparsity
66-
2 Target Sparsities
64
Target Sparsities
Ours (SSL)
62
60
6 Target Sparsities
Ours (CE)
31.8
2133
14.3
GFLOPs
GFLOPs
Figure 9: Ablations for the number of target sparsities and loss. Average accuracy in k-nearest neighbor classification for AugReg ViT-B/16 models pruned by optimizing for sparsities (left), and for models pruned using gradients from either self-supervised or cross-entropy loss (right).
to clastic model approach, and show that both produce models with similar performance, with the advantage that our approach only needs to be run once. Furthermore, we compare optimizing the genetic algorithm for two, four, and six sparsities, and find that optimizing for more sparsities can significantly improve performance.
Function evaluations. In Table we ablate the number of iterations used for the genetic algorithm optimization and show that running the algorithm for 500 iterations improves the performance of an AugReg ViT-B/16 pruned 50% sparsity by up 1.8% on average in linear classification.
Supervised gradients. In Figure we ablate the choice of self-supervised loss to guide pruning, and observe that, for an AugReg ViT-B/16 backbone, using a cross-entropy loss only performs marginally better.
5 Conclusion
In this work, we presented a novel and fast post-training structured pruning method that enables elastic inference across continuum of sparsity levels. Our approach combines gradients and cross-structus correlations, approximated via genetic algorithm, to produce efficient ViTs with strong performance across several tasks without retraining. Furthermore, we have shown that it is possible to effectively prune models without requiring labeled data or classification head via a self-supervised loss.
Acknowledgment This work has received financial support from Qualcomm Technologies Inc., the University of Amsterdam, and the Top Consortia for Knowledge and Innovation (TKIs) allowance from the Netherlands Ministry of Economic Affairs and Climate Policy. The authors gratefully acknowledge the scientific support and HPC resources provided by the Erlangen National High Seeehh-lleeane--nniieee
Performance Computing Center (NHR @ FAU) of the Nürnberg (FAU) under the BayernKI project 15be. BayernKI funding is provided by Bavarian state authorities.
Erlangen-
10

---

References
[1] Youhei Akimoto, Yu Nagata, Isao Ono, and Shigenobu Kobayashi. Bidirectional relation between ema-es and nes. In International Conference on Parallel Problem Solving from Nature (PPSN XI), 2010.
[2] Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Emerg, Technol. Comput. Syst., 13())33:1-322:18, 2017.
[3] Ivana Balazevic, David Steiner, Nikhil Parthasarathy, Relja Arandjelović, and Olivier Henaff. Towards in-context scene understanding. Advances in Neural Information Processing Systems, 36, 2024.
[4] Manel Baradad, Richard Chen, Jonas Wulff, Tongzhou Wang, Rogerio Feris, Antonio Torralba, and Phillip Isola. Procedural image programs for representation learning. Advances in Neural Information Processing Systems, 35:6450-6462, 2022.
[5] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint 1908,091,
[6] Han Cai, Chuang Gan, Lin, and Song Han. Network augmentation for tiny deep learning. arXiv preprint arXiv:2110.08890, 2021.
[7] Mu Cai, Jianwei Yang. Jianfeng Gao, and Yong Jae Lee. Matryoshka multimodal models. CoRR, abs/2405. 17430, 2024.
[8] Mathilde Caron, Ari Morcos, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Pruning convolutional neural networks with self-supervision. arXiv preprint NoKiv:2001.0355: 2020.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self superviie vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9950669 2021.
[10] Adam Casson.
hwwp://s/wwaaacccsssoooo wwwpenn...oooppooooeoooe posts/ transformer- flops 2021. Accessed: 2025-09-24.
Transformer flops.
[11] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.
Describing textures in the wild. In CVPR, 2014.
[12] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pp. 7480-7512. PMLR, 2023.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icce, 2009.
[14] Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit S. Dhillon, Yulia Tsvetkov, Hanna Hajishirzi, Sham M. Kakade, Ali Farhadi, and Prateek Jain. Matformer: Nested transformer for elastic inference. In NeurIPS, 2024.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16
words: Transformers for image recognition at scale. In ICLR, 2021.
[16] Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Ángel Bautista, Vaishaal Shankar, Alexander T Toshev, Joshua M. Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. In Proceedings of the 41st International Conference on Machine Learning, pp. 12371-12384 2024.
[17] M. Everingham, I.. Van Gool, C. K. I. Williams, J. Winn, and Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010.

---

[18] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned one-shot. In International Conference on Machine Learning (ICML), 2023.
[19] Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionnar computation, pp. 393 400, 2010.
[20] Matteo Grimaldi. Luca Mocerino, Antonio Cipolletta, and Andrea Calimera. Dynamic convnets on tiny devices via nested sparsity, IEEE Internet of Things Journal, 10(6):5073-5082, 2022.
[21] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M. Susskind, and Navdeep Jaitly, Matryoshka diffusion models. In ICLR. OpenReview.net, 2024.
[22] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. pp. 1 1135-115111 113,, 2015.
[23] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In NIPS, pp. 1135-1143, 2015.
[24] Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NIPS, pp. 164-171. Morgan Kaufmann, 1992.
[25] Yang He and Lingao Xiao. Structured pruning for deep convolutional neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell., 26(5):2000-2) 2919,, 2024.
[26] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. JST-AEORS, 2019.
[27] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. 2015.
[28] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Jacob Menick, Rewon Child, Adhiguna Kuncoro, Tomás Kociský, Phil Blunsom, Chris Dyer, Oriol Vinyals, Jack W. Rae, and Erich Elsen. Training compute-optimal large language models. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 30016-30030, 2022. httph:tt wtw.b/p.........op.ope.t..oo ctw.f//////oo////////ee////oo/ce conf http:///www///ooooooooooaoaaoaa bib
[29] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9793, 2020.
[30] Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. Matryoshka query transformer for large vision-language models. In NeurlPS, 2024.
[31] Hirokazu Kohama, Hiroaki Minoura, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Single-shot pruning for pre-trained models: Rethinking the importance of magnitude pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
1433-1442, 2023.
[32] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519-3529. PMLR, 2019.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder Kaifeng Chen, Sham M. Kakade, Prateck Jain, and Ali Farhadi. Matryoshka representation learning. In NeurIPS, 2022.
[35] Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-training pruning framework for transformers. In NeurIPS, 2022.
12

---

[36] Woosuk Kwon, Schoon Kim, Michael W Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami. A fast post-traiming pruning framework for transformers. Advances in Neural Information Processing Systems, 35:24101-241 16, 2022.
[37] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[38] Jaeho Lee, Sejun Park, Sangwoo Mo, Sungsoo Ahn, and Jinwoo Shin. Layer-adaptive sparsity for the magnitude-based pruning. In ICLR. OpenReview.net, 2021.
[39] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. 2017.
[40] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1600,0398.. 2016.
[41] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aireraft. Technical report, 2013.
[42] Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning, 2019. URL httpptt///axxxi.orrr org/ wttp://wwwwwr/////9////// wtbb////..... 10771
[43] Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian. One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In NeurIPS, pp. 4933-4943, 2019.
[44] Sharan Narang, Greg Diamos, Shubho Sengupta, and Erich Elsen. Exploring sparsity in recurrent neural networks. In ICLR (Poster). OpenReview.net, 2017.
[45] Max Nonnenmacher, Thomas Pfeil, Ingo Steinwart, and Dominik Reeb. Sosp: Efficiently capturing global correlations by second-order structured pruning. In International Conference on Learning Representations (ICLR), 2022. URL httpp:://opeeeeeee...eeeeeeeeeee H1B8bA9tvr
[46] Zizheng Pan, Jianfci Cai, and Bohan Zhuang. Stitchable neural networks. In CVPR, 2023.
[47] Valentinos Pariza, Mohammadreza Salehi, and Yuki Asano. Hummingbird evaluation for vision encoders, 4 2024. UtLLAtttoe///gghhbb..ooo///rrrrp www.ooonnnnnnininnnnnnooooeoe eval
[48] Valentinos Pariza, Mohammadreza Salehi, Gertjan J Burghouts, Francesco Locatello, and Yuki M Asano. Near, far: Patch-ordering enhances vision foundation models' scene understand- ing. In The Thirleenth International Conference on Learning Representations, 2025.
[49] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
[50] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Seikit-learn: Machine learning in Python. JMLR, 2011.
[51] Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11428-10436, 2020.
[52] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. pp. 525-542. Springer, 2016.
[53] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
[54] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 10684-10695, 2022.
13

---

[55] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJC,, 2015.
[56] Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning. In NeurIPS, 2020.
[57] Ofer M. Shir and Amir Yehudayoff. Covariance matrix adaptation and inverse hessian convex quadratic functions. In Proceedings of the 15th ACM/SIGEVO Conference on Foundations of Genetic Algorithms (FOGA). 2019.
[58] Abhinav Shukla, Sai Vemprala, Aditya Kusupati, and Ashish Kapoor. Matmamba: A matryoshka state space model. CoRR, abs/2410.067188 2024.
[59] Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
[60] Mannat Singh, Quentin Duval, Kalyan Vasudev Alwala, Haoqi Fan, Vaibhav Aggarwal, Aaron Adcock, Armand Joulin, Piotr Dollár, Christoph Feichtenhofer, Ross B. Girshick, Rohit Girdhar, and Ishan Misra. The effectiveness of MAE pre-pretraining for billion-scale pretraining. In ICCV, pp. 5461-5471. IEEE, 2023.
[61] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106. 102700 2021.
[62] Mingjie Sun, Zhuang Liu, Anna Bair, and J. Zico Kolter. A simple and effective pruning approach for large language models. In ICLR. OpenReview.net, 2024.
[63] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. CoRR, abs/1801.055 2018.
[64] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
[65] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European conference computer vision, pp. 516-533. Springer, 2022.
[66] Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alab- dulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip Multilingual vision- language encoders with improved semantic understanding. localization, and dense features. arXiv preprint arXiv: 14786, 2025.
[67] Tycho F. A. van der Ouderaa, Markus Nagel, Mart van Baalen, and Tijmen Blankevoort. The LLM surgeon. ICLR. OpenReview.net, 2024.
[68] Chaoqi Wang, Roger Grosse, Sanja Fidler, and Guodong Zhang. FigenDamage: Struc- tured pruning in the factered eigenbasis. In Proceedings of the 36th Interna- tional Conference on Machine Learning, volume 97, pp. 6566-6575. PMLR, 2019. URL html
http://proceedding. mlr. press/v97/wange
[69] Chaoqi Wang, Guodong Zhang, Shunhua Fidler, and Roger Grosse. Eigendamage: Structured pruning in the kronecker-factored cigenbasis. International Conference on Machine Learning (ICML), 2019. http:tts::
patess/v97//e mlr. press/ /v7 wang html
[70] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005. /4/87, 2020.
[71] Huan Wang, Qiming Zhang, Yuehai Wang, and Haoji Hu. Structured probabilistic pruning for convolutional neural network acceleration. In BMVC, pp. 149. BMVA Press, 2018
14

---

[72] Zijie Wang, Evan Montoya, David Munechika, Haoyang Yang. Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image genera- tive models. arXiv preprint arXiv:2210. 14896, 2022.
[73] Ross
Wightman.
Pytorch image models.
https://ggttaabb.ommrrrggggaga
pytorch- image- Baagg-moode models 2019.
[74] Kaixin Xu, Zhe Wang, Chunyun Chen, Xue Geng, Jie Lin, Xulei Yang. Min Wu, Xiaoli Li, and Weisi Lin. Lpvit: Low-power semi-structured pruning for vision transformers. In European Conference on Computer Vision, pp. 269-287. Springer, 2024.
[75] Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian- saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18547-18557, June 2023.
[76] Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian- aware saliency. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 18547-18557, 2023.
[77] Ziqing Yang, Yiming Cui, Xin Yao, and Shijin Wang. Gradient-based intra- attention pruning on pre-trained language models. In ACL (1), pp. 2775-2790. Association for Computational Linguistics, 2023.
[78] Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. Proceedings of the IEEF/CVF international conference on computer vision, pp. 1811-1811, 2019.
[79] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprini urXiv:181 18/2009928 2018.
[80] Ben Zandonati, Adrian Alan Pol, Maurizio Pierini, Olya Sirkin, and Tal Kopetz. Fit: A metric for model sensitivity, 2022.
[81] Kaiqi Zhao, Animesh Jain, and Ming Zhao. Adaptive activation-based structured pruning. CoRR, 0tss/2201.10520. 2022.
[82] Kaiqi Zhao, Animesh Jain, and Ming Zhao. Iterative activation-based structured pruning. CoRR, abs/2201.09881, 2022.
[83] Chuanyang Zheng, Kai Zhang, Zhi Yang. Wemming Tan, Jun Xiao, Ye Ren, Shiliang Pu, et al. Savit: Structure- aware vision transformer pruning via collaborative optimization. Advances in Neural Information Processing Systems, 35:9010-9023, 2022.
[84] Michael Zhu and Suyog Gupta. To prune, not to prune: Exploring the efficacy of pruning for model compression. In ICLR (Workshop). OpenReview.net, 2018.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: throughout experiments show that DeIT, AugReg, DINOvI, DINOv3, and SigLIPv2 backbones pruned with our label-free one-shot, all sparsities method perform on-par or better than models pruned using existing state-of of--hh--rtttt- methods, furthermore, in [Table 3e we show the importance of modeling the Hessian off off-diagonal components of network structures, in Table 6 we show that our algorithm can prune models up to ViT-L/16 in under 5 minutes on a A 100 GPU, and in Figure 9 we show that supervised gradients only perform marginally better compared to self-supervised gradients.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
we discuss the limitations of our approach in Appendix B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: we describe our method extensively in Section provide pseudo code in Appendix [D.2] and report the experimental details plus the complete set of hyperparameters used for pruning, weight correction, fine-tuning and evaluation in Appendices D.3 D.5 D.4 and respectiivlll. We also release code https hhttps///itthu/ //github. com/ Wwwteeriimmncinnn SnapViT
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
evaluate our pruning method using only publicly available models and datasets. Furthermore, we release the codebase used to run the experiments presented in this paper hhttts://githui: instructions on how to retrieve data and models, as well as scripts to replicate the main experiments presented in the paper.
The repository includes
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https //nips.cc/ ltliiss.cc/publicguuuueee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we have, to the best of our knowledge, provided all the experimental details required to reproduce results. In particular, hyperparameters for our method, baselines, weight correction, fine-tuning, and evaluation are described Appendix D and datasets and their splits described in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
18

---

Answer: [Yes]
Justification: In Section we report the mean accuracy and standard deviation in k- nearest neighbor and linear classification of an AugReg ViT-B/16 backbone pruned with our method. We observe that at high sparsity ratios, pruned models can exhibit high variance, particularly on certain datasets. The standard deviation computed using numpy. .ttdyy.tddd... std and is reported only for this experiment, as calculating it for all experiments in the paper would computationally prohibitive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: the hardware used the experiments described in this paper and their runtimes are documented in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http:::tttttuttteeee http:r////eee/eeee http:///.......ccetcccccccccaaea
Answer: [Yes]
Justification: read the code of ethics and ensured that our paper conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
19

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the potential broader impacts of our method are discussed in Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymentt of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
20

---

Answer: [Yes]
Justification: we properly credited authors datasets, models, and code implementations used in this paper and adhered to their licenses and usage guidelines. The licenses and citations for the datasets, models, and baselines used in the paper are listed Table 4 Table 5| and Appendix D.3 respectiiv.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, papprssii throde com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: we release code implementation that can be used replicate our experiments and produce sparse models, available at https: //gi thub. com Ser terSimonciniiii SnapViT, The repository includes license and proper documentation on setting up an appropriate environment, downloading data and models, and producing pruned models. Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: we did not conduct research with human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
21

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: we did not conduct research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: we did not use LLMs to develop the core methodology of this paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
22

---

A Broader impact
Our method accelerates the inference speed of vision transformers, reducing the computing require- ments and power usage of these models. Thus, our method could have positive consequences, such as lowering the CO2 emissions generated by inference and enabling users with limited compute resources to benefit from the abilities of large (pruned) models. We believe our method should not have a direct negative impact.
B Limitations
We have shown that approach can yield elastic models in lahel- and retraining-free fashion that perform on par or better than the current state of the art, especially at high sparsity ratios, but some limitations remain:
Self-supervised models trained on large-scale datasets are difficult to prune effectively, and performance quickly collapses. Yet, we have found that longer optimization horizons and initializing with cross- strss-structure CKA scores, as shown in Appendix G can improve performance. Thus, we believe that there is way prune these models effectively, but we leave this for future work.
The re-weighting of prunability scores alters the ranking of attention heads and MLP blocks, but does not take into account individual MLP hidden neurons. Optimizing the ranking of individual neurons could further enhance the performance of sparse models.
C Data and models
We investigate the performance of pruned models on 7 image classification datasets, namely ImageNet- 1k (55), FGVC Aircraft [41) Oxford-IIT Pets [49]. DTD Textures [1I]. EuroSAT (26) and CIFAR 10/100 [33] plus Pascal VOC 2012 [17] for semantic segmentation. Table 4 lists all the datasets used in this paper alongside their license and citation.
We follow the standard evaluation protocol for each individual dataset and report the top-1 accuracy in k-nearest neighbor and linear classification for image classification datasets and the mean intersection over union (mIoU) for Pascal VOC.
We use the train/test splits defined by the dataset authors where possible, except for EuroSAT, for which we use an 80/20 stratified split as indicated by the dataset paper. We always report the performance on the test split, except for ImageNet-Ie 1k and Pascal VOC, for which we report performance on the validation split. For the linear classification experiments we use the validation split defined by the dataset authors if available, and otherwise create one using an 80/20 random split.
Table 4: Datasets. Summary table of the datasets used in the paper.
Dataset
License
Citation
ImageNet-Ik FGVC Aircraft
Research Only
55
Research Only
Oxford-IIT Pets
CC BY-SA 4.0
49
DTD Textures
Research Only
EuroSAT
MIT
26
CIFAR 10/100
Unknown
33
Shaders21k lk
Unknown
DiffusionDB
CCO 1.0
Pascal VOC 2012
Unknown
Table 5 lists the models pruned and evaluated in this paper, alongside their citation and license.
23

---

Table 5: Models. Summary table of the models used in the paper.
Model
License
Citation
DINO
Apache 2.0
DINOv3
DINOv3 License
[59
SigLIPv2
Apache 2.0
666
73
AugReg
Apache 2.0
61
73
DelT
Apache 2.0
64
DefT-III
Apache 2.0
65
Table 6: Our algorithm scales sub-linearly with respect to the number of parameters. Runtime, as measured on an NVIDIA A100, of our algorithm for the entire DeiT-Ill family.
Model
Layers
Attention Heads
Parameters (M) Runtime
DelT-III ViT-S/16
12
6
22.1
2m 35s
DeIT-III ViT-B/16
12
86.6
2m 55s
DeIT-III ViT-L/16
24
16
304.4
4m 58s
DeIT-III ViT-H/14
32
16
632.1
11m 4s
D Experimental details
D.1
Complexity Analysis
The total computational cost of pruning algorithm is
O(NDF) O(TASNsF) O(TB²) O(Plog P),
(10)
where
FF) computes the local diagonal Hessian via one barkwdbd pass on No samples, where F denotes the cost per pass.
o covers the xNES search phase: in each of iterations, candidates are drawn and each is evaluated across S sparsity targets using Ns images. F" represents forward-only pass (feature extraction + PCA), requiring no back- back-propagation.
O(TB²) accounts for updating the B B covariance matrix Σ in xNES, which models global off-diagonal dependencies between functional blocks (e.g., FFN and attention heads).
O(P P) sorts the P prunability scores once obtain elastic subnetworks for any desired sparsity level.
Compared with SOSP 45 and EigenDamage [69] our approach eliminates all explicit curvature computations. SOSP-H requires one Hessian-vector product per structure, and SOSP-1 constructs dense S Gauss-Newton matrix. In contrast, our algorithm performs only forward inference and lightweight covariance update B²), avoiding any backward curvature passes.
The runtime is thus dominated by the forward feature-extraction term. xNES iterations and four sparsity targets, a single A100 GPU prunes the entire DeiT-III family in only few minutes, shown in Table 6) The quadratic covariance term remains negligible even for the largest models, demonstrating excellent scalability and making our approach one of the most efficient second-order-aware pruning frameworks to date.
In practice, with T 50
D.2 Pseudo Code
Algorithm outlines our single-shot pruning procedure. Given a model fo. dataset D, a maximum number of iterations T, set of target sparsities S and population size X, which we initialize as + 3log( (d) as described in the xNES paper [19), where is the problem dimensionality, e.g.. 156 in the case of ViT-B/16, we proceed follows:
Compute the sule-supe gradients, obtain the local prunability scores, and initialize the xNES mean μ and covariance Σ.
24

---

Algorithm Single-shot pruning with xNES
Require:
Model fe: dataset D; blocks {I,..., B3: iterations T'; population A; sparsity grid 8 Global ranking masks for arbitrary sparsity
Ensure:
1: S f diag
CSL (x))1
> datasct-averaged diagonal Hessian proxy
Np
ZED
2: (J,, )) + (0.I)
xNES mean covariance
3:
for = 1 to T do
4:
for = to do
5:
c(k) ~N,, Σ)
blockwise reweighting factors expand parameters and combine with local scores
6:
(k*) (M (Mc os
7:
for s E do
8:
(as
(u (s)
9:
+ fo(z),
2p. (oomack (2)
forward-only minibatch
F(h,)) t cos (CC((()))) (CA(2p.))
11: end for
F(M)
[S] Exes F(k..)
13: end for
14: (11, + XNES-UPDATE((CCH (c( ((
15: end for
16: Cfinal arg max; ((R)
best-performing sample (global correlation vector) > final prunability scores derive masks for any target sparsity by thresholding
17: P Cfinal Øs
18: return argsort(P)
2. Sample À individuals for the current generation. For each individual, combine the local and global prunability scores, produce the pruning masks for each target sparsity E S, and, for each sparsity s, measure the fitness as the average post-PCA cosine similarity between pruned and original embeddings. The individual's fitness F is then computed as the average of fitnesses across the sparsity targets S.
3. Update y and Σ, and continue from step 2.
The algorithm terminates after T steps, and the best ranking is derived from the individual with the highest fitness.
D.3 Pruning
We prune models to six target sparsities, namely 10, 20, 30, 40, 50, and 60% in one shot. To do so, first estimate gradients using either DINO cross-entropy loss and 1000 random samples from the ImageNet- Ik training set (unless specified otherwise) and batch size 16. Gradients are averaged over each batch and summed across batches. We do not any data augmentation for the cross-entropy loss, and for the DINO loss, we only use random cropping to generate global and 10 local crops, with scales between (0.25, 1.0) and (0.05, 0.25), respectively.
and produce a single score for each attention head and hidden feed-forward neuron by averaging. Then, we optimize the sparsity allocation using the xNES for 50 iterations. For each iteration, we generate models pruned at 10, 30, 50, and 60% sparsity and measure the cosine similarity between embeddings produced by the pruned models and the original model, using 1000 fixed samples from the ImageNet- Ik training set. We then average the cosine similarity across sparsity ratios and select the configuration that maximizes this metric and, by consequence, minimizes divergence. Before computing the cosine similarity, project embeddings to 192 dimensions using PCA model trained using the same 1000 images, as embedded using the original model.
After approximating the gradients, we compute prunability scores using Equation 2,
For each block, we constrain our algorithm to prune most 80% of the attention heads and 95% of the feed-forward neurons, leaving at least 2 attention heads and 154 neurons for each individual block in the case of a ViT-B/16 model.
25

---

We combine the two losses as D.4 Post-pruning processing
Weight correction. We apply SparseGPT-style post-pruning weight correction to models pruned with our method as follows: first, apply our pruning algorithm to obtain binary mask M for given target sparsity s, where M, = 0 indicates that the weight at position (i.j) is pruned. Then, for each layer to be pruned, we collect input activations in a matrix E RRdix and compute the damped Hessian
0.01 dn
= XX" + +T,
À
H
(11)
dm
We then invert the Hessian using a Cholesky decomposition, obtaining H Following SparseGPT, we then process the weight matrix column-wise in blocks of 128 columns. For each column in block i ++ B, we mask out the pruned weights and compute the reconstruction error as:
W
E.g- = M..) O
(12)
We then update the unpruned weights of subsequent columns as
W W) W.j-(i+B) E.j-v H JJ:(+BB)*
(13)
After applying the weight correction to all pruned layers in a Transformer block, rearrange the attention heads and MLP hidden neurons according to our ranking and remove the pruned structures in retraining-free experiments.
Full fine-tuning. We fine-tune DelT VIT-B/16 model pruned to 50% sparsity with our method and a self-supervised loss using the fine-tuning scripts from NViT, closely following their recipe. In particular, we fine-tune the model in float16 for 300 epochs, using 8 GPUs, a per-device batch size of 144, and an initial learning rate of 0.0002. We use hard distillation with a 0.5 and soft distillation with = 20.0 from a RegNerY-16GF f. = Lhard + 10000
[1 teacher.
D,5 Baselines
We adapt the official implementation! released under the BSD 3-Clause Clear License, to prune ViT', disable weight correction and LoRA fine-tuning and closely follow the configuration recommended by the paper authors, except for the number of samples used estimate the curvature (1000 versus the default 128 to match our method) and the number of shots, 5 in our experiments compared the recommended 40, as did not notice significant differences in our preliminary runs. While we prune the hidden neurons of feed-forward blocks and whole heads, the LLM Surgeon prunes independent rows and columns in weight matrices, making a 1:1 comparison hard, as it has more degrees of freedom compared to our method. Moreover, due to their pruning strategy, the speed improvements of the LLM Surgeon are not easy to realize practice.
LLM Surgeon 677.
We evaluate NVTT using the code and configuration available the official GitHub repository released under the NVIDIA Source Code License-NC, with the exception that prune 1024 structures per step rather than 32 due to computational reasons. In contrast to other methods, we do not disable fine-tuning during pruning, as doing so causes the algorithm to fail. For the full fine-tuning experiment, we evaluate the fine-tuned checkpoint made available by the authors.
NVIT 76
We adapt the official code implementation to ViTs, disable mask tuning and prune models using the default parameters, except for the number of samples used for estimating gradients, for which we use 1000 instead of the standard 2048 for a fair comparison with other methods.
FPTP' 35
http://ttttbb.ccom//aa/aaaoaaaae httpe///gichab..c////laaa////
btte://ggtaabbbce/oooooo//eeeeee pruning
26

---

Table 7: Theoretical FLOPs formulas for ViTs. ViT components, sub-components, individual computations, and the formula used to estimate the corresponding theoretical FLOPs. The formula to estimate the FFN FLOPs assumes that the hidden dimensionality is 4dmooell-
Component
Sub- Tub-Component
Computation
FLOPs
Embeddings
pasch patch Tcchanett maddel modeelPelassee
d
Logits
QKV QK Logits
roooe 3dmodel (dkey (heaadss
2n lokens (heen))
2
Attention
Softmax
37 inkens
Block
Reduction
2n
Thheads Theaas)
Projection
Booess (dxey Theaads moddel 167h moneel
FFN
SNIP Magnitude 31.. We implemented the SNIP Magnitude score in our framework following its official implementation Score aggregation and pruning are done in the same way as for our method.
We adapt the official code implementationf] released under an Apache 2.0 License, to ViTs and to perform structured pruning by masking entire columns. We use the default parameters, except for the number of samples used to estimate the Hessian, which is set to 1000 for a fair comparison with other methods.
Sp [88]
LAMP 38
We implemented the LAMP score in our framework, closely following the formulas and pseudocode from the original paper. We aggregate scores and perform pruning as for our method.
D.6 Evaluation
k-nearest neighbor classification. We evaluate pruned models in k-nearest neighbor classification using the implementation from scikit-learn [50]. In particular, we report the classifier performance using majority voting across 20 neighbors and Lo-nommllized features.
Linear classification. We evaluate pruned models in linear classification following the DINO recipe (9. For each dataset, we train a linear classification head for 100 epochs using SGD with a 0.9 momentum, a learning rate of 0.001, no weight decay, a batch size of 256, and a cosine annealing learning rate scheduler (40) with Mmin
We then select the best classifier on the validation set and report its performance on the test set. No data augmentation is applied to the training samples.
Semantic segmentation. We evaluate models semantic segmentation on Pascal VOC 2012 17 by training a convolutional head for 25 epochs using SGD with 0.9 momentum, a 0.01 learning rate, further reduced 0.001 after 20 epochs, a 0.0001 weight decay, and batch size of 128 following the recipe from [8]. We select the best model on the validation set, and reports its average mean intersection over union (mloU). Training images are augmented via random crops with a scale
between 80 and 100% of the original image, resized to (224, 224), and flipped horizontally with 50% chance.
D.7 GFLOP definition
We use the term GFLOPs to indicate the number of theoretical floating-pons point operations required
for a single forward pass. We adopt the formulas from (28 10]. displayed in Table 7 where paathh indicates the total number of patch tokens for an input image (c.g.. 196 assuming patch size of 16 and an input size of 224 x 224), dpauch the side length of a single image patch, Mechnnels the number of channels of the input image (e.g., 3 for RGB image), dinatel the embedding size, Tookens the total
httpp://githhb: wwt.ihhu.....i....... wwwwwso www.--DDaaaabbbbeeeaeeee The official code implementation was released without an explicit license.
27

---

Table 8: GFLOPs for VIT models. Theoretical GFLOPs measurements for ViT models of increasing size and their architectural configuration parameters that contribute to the computation.
Model
Npatch
Opucch
dgnodel
Thokens
dkey
Taaaerrs
Theads
nelssses
GFLOPs
VIT-S/16
196
16
3
384
197
64
12
6
1000
9.2
VIT-B/16
196
16
3
768
197
64
12
12
1000
35.1
VIT-L/16
196
16
3
1024
197
64
24
16
1000
123.2
Table The image classification evaluation has high variance at high sparsity ratios. Average top-1 1 k-nearest neighbor and linear classification accuracies and their standard deviation across three seeds (0, 13, and 42) using an AugReg ViT-B/16 backbone pruned to 10, 20, and 30% sparsity.
Eval.
Sparsity
DTD
FGVC
EuroSAT
CIFAR 10
CIFAR 100
Pets
INIK
Avg.
10%
61.0
24.7
91.7 =
92.7 = 0.3
73.5
89.7 0.2
76.6
728 : 0.1 59.0
K-NN
30%
5122
18.3
7 9.
7..9 4.3
55.4
2.0
50%
37.1 + 2.0
11,5 1.8
89.7
56.8 = 0.4
29.900.33
28.3 5.2
22.9 2.3
39.5
1.4
10%
38.4 0.3
942
03
8.12
92.2
77.3
78.3 0.1
6
29.4
922
79.8
81.2
60.6
66.3
50%
52.2 + 2.3
18.1 + 2.1
8
51.5 = 0.8
21.6 2.4
43.7 2 5.3
20.9 1.9
41.8 1.2
number of tokens including the [CLS] token (and distillation token for DeITs), daey the attention head size, Melasses the number of output units for the classification head, Mayyers the number of transformer blocks and Thead the number of attention heads. The total number FL.PPs is computed as:
Total FLOPs embeddings Mlayers (attention + FFN) + logits.
(14)
For a ViT-B/16, this results approximately 35.1 GFLOPs. Some papers report multiply-accumulaa operations (MACs) instead of FLOPs, equivalent to FLOPs/2, i.e., 17.6 GMACs for a ViT-B/16. Theoretical measurements for other model sizes, alongside the relevant architectural details, are illustrated in Table 8
E
Statistical significance of results
In Table we report the mean accuracy and one standard deviation computed across three seeds (0, 13, 42) in k-nearest neighbor and linear classification for AugReg ViT-B/16 models pruned using method to 10, 30 and 50% sparsity. Accuracies for models pruned to 10% sparsity are consistent across seeds. In contrast, sparser models have higher standard deviations average and certain datasets, such as Oxford-IIT Pets, on which the k-nearest neighbor accuracy at 30% sparsity has standard deviation of 6.5%.
F Compute resources
The pruning experiments were run using a NVIDIA A100 GPU with 40GB of VRAM, 16 CPU cores, and 40 GB of RAM. While the pruning runtime is negligible, evaluating each (model, sparsity) pair in k-nearest neighbor and linear classification requires approximately one GPU hour in float16. Given this, we estimate that reproducing the main experiments presented in this paper would require approximately 275 GPU hours, covering the one-shot pruning experiments (with and without weight correction) and all ablations. The full fine-tuning experiment required an additional 2.5 days on GPUs (480 GPU hours), bringing the total compute budget to roughly 755 GPU hours. Preliminary exploratory runs required less than 50 GPU hours in total.
G Additional experimental results
Importance of data. We ablate the pruning performance with respect to the data used to estimate gradients and the genetic algorithm optimization for DINO ViT-B/16. In particular, we compare DiffusionDB 772.. dataset of synthetic images generated via Stable Diffusion (54). Shaders 21
4. dataset abstract images generated via shared programs, ImageNet- 1k [33 which was used for pretraining and strongly aligns with some of the evaluation datasets, including ImageNet- Ik itself,
28

---

Genetic Algorithm
K-NN, 50 Steps
Linear, Steps
0.8
I=l
0.9
I- CKA
Avg. Accuracy
0.3
0.88
263
352
21
283
Genelic Algorithim Steps
GFLOPs
GFLOPs
K-NN, 100 Steps
Linear, 100 Steps
K-NN, 250 Steps
Linear, Steps
Accuracy
38-
GFLOPs
GFLOPs
GFLOPs
GFLOPs
Figure 10: Initializing Σ using CKA scores improves performance for SigLIPv2 ViT-B/16. The genetic algorithm loss over 250 steps, smoothed using exponential moving average with 0.95, when is initialized either with an identity matrix or the CKA scores between structures, plus the model performance when pruned up 30% sparsity after 50, 100 and 250 genetic algorithm steps.
Table 10:
The CKA initialization significantly improves performance across datasets. Top-1 accuracy in k-nearest neighbor and linear classification of SigLIPv2 V/T-B/16 pruned to 30% sparsity using our method with either identity or CKA initialization for Σ and 250 genetic algorithm steps.
Initialization
DTD
FGVC
EuroSAT
CIFAR 10
CIFAR 100
Pets
INIK
Σ I
51.4
13.3
85.1
49.0
24.5
43.0
35.8
k-NN
Y = CKA
63.5
24.6
88.2
75.7
48.0
60.1
56.9
Σ = I
4.9
1.0
74.8
40.4
18.5
38.2
33,9
Linear
D CKA
11.3
2.3
76.7
68.3
41.9
58.3
54.9
CIFAR 10 and 100 33 and Oxford-IIT Pets 49 and a dataset obtained by sampling a total of 1000 images in equal parts from the training split of each of the evaluation datasets, which call "Merged Data". The results, shown (Figur 11, demonstrate that alignment between pruning and task data heavily affects performance, improving by up to 6.4% when comparing ImageNet- -i-lett---bbseee pruning to Shaders-21k at 40% sparsity. Furthermore, while the Merged Data dataset performs similarly to ImageNet-Ik Ik at shallow sparsity ratios, it can improve by up to 5.3% 60% sparsity, suggesting that a data-centric view of pruning can help produce sparse models that generalize better.
Genetic algorithm initialization. By default, xNES initializes Σ T, where 7 is the identity matrix. We compare this strategy to initializing each entry Di, using the centered kernel alignment (CKA) 32] score between the activations of structures and j (e.g. attention heads, one attention head and feed-forward block or two feed-forward blocks), estimated using 2500 random samples from the training set of ImageNet- 1k for a SigLIPv2 ViT-B/16 backbone. We run the genetic algorithm for up to 250 steps for both initializations and evaluate the performance of pruned models at 50, 100, and 250 steps. The results, shown in Figure 10, demonstrate that the CKA initialization improves both convergence and performance, as the initial loss is lower, and significant gap persists even after 250 steps. Regarding performance, the CKA initialization matches or outperforms the baseline across all pruning ratios at 50, 100, and 250 steps, with a gap of up to 16.4% in k-nearest neighbor at 250 steps and 30% sparsity. Table 10 reports the model performance at 30% sparsity on a per-dataset basis, showing that CKA initialization can improve performance by up to 26.7% in k-nearest neighbor and 27. 9% in linear classification on CIFAR 10.
Pruming across model sizes. In Figure 12 we plot the average accuracy in k-NN and linear classification across the seven image classification datasets for DefT-IIII (65] models, trained using ImageNet-224 and fine-tuned on ImageNet- 1k, ranging from VIT-S/16 to H/14, pruned to up to
29

---

K-NN
K-NN
Avg. Accuracy
05
52
Shaders-21k
DiffusionDS
ImageNet-14 1
13-
35
ImageNet-1k 1k
52 Merged Data
50
3L.
GFLOPS
Figure 11: Alignment between pruning data and target tasks improves performance. Average top-1 accuracy in k-nearest neighbor and linear classification of DINO ViT-B/16 models pruned using our method and data from ImageNet- Ik, DiffusioDB, Shaders 21k, and a 1000 samples dataset built by sampling equally from the training set of each of the evaluation datasets, named "Merged Data".
DelT-III VIT-S/16
DefT-IlI ViT-B/16
DelT-III VIT-L/16
DefT-I11 VIT-H/14
Avg. K-NN Accuracy
60
72
50
65
11g
14.
122
112
202.
200
202.2
Avg. Linear Accuracy
70-
72
50
65-
52
28
11g
122
312
136
302
200
210
202.2
188
105
GFLOPs
GFLOPs
GFLOPs
Ours (SSL)
Ours (SSL, Correction)
Figure 12:
Larger supervised models can be pruned more aggressively. Top-1 accuracy in k- nearest neighbor and linear classification, averaged across 7 datasets, for models of various sizes belonging to the DelT-III [65] family, pruned to up to 60% sparsity using our method with and without weight correction. Weight correction consistently improves performance for VÝT-B/16 and larger models, by up to 25.4% for ViT-L/16 pruned to 60% sparsity.
60% sparsity using our method. We find that larger models from this family can be pruned more aggressively with minimal loss in performance. For example, the ViT-H/14 model can be pruned to 50% sparsity, equivalent to removing approximately 316M parameters, while losing only 3.7% and 3.4% on average in k-nearest neighbor and linear classification, respectively. The results per-dataset basis are shown in Table IT where we observe that for some datasets, such as EuroSAT and Oxford-IIT Pets, the performance drop remains below 1.5% for both lincar and k-nearest neighbor classification. Interestingly, accuracy even improves on FGVC Aircraft. Finally, the model maintains strong performance on ImageNet-Ik, with at most 7.6% decrease in accuracy. When post-pruning weight-correction is applied, performance is mostly restored, with an average degradation only 0.5% in k-nearest neighbor, and average improvement of 0.9% in linear classification 50% sparsity. On a per-dataset basis, DTD Textures, FGVC Aircraft, and ImageNet- k benefit the most from weight correction, improving by 7%, 5.8%, and 2.4%, respectively, in linear classification. Interestingly, weight correction improves performance for all models except for the ViT-S/16 at high pruning ratios. We hypothesize this might be due to the limited remaining representational capacity of the model, as ViT-S/16 pruned to 50% sparsity has only 1 1M remaining parameters.
we qualitatively analyze quality of dense represen- tations for pruned DINO ViT-B/16 models via the HummingBird in-context semantic segmentation evaluation R. as implemented in [47, for Pascal VOC 2012. We use the default parameters except for the memory bank size and input image size, which are 196 x 10 and 224 x 224, respectively,
Dense representation quality. In Figure 13
30

---

Figure 13: Our method preserves the quality of dense representations. Visualization of seg- mentation masks for Pascal VOC 2012 produced via in-context semantic segmentation using DINO ViT-B/16 models pruned from 0 to 60% sparsity using our method (top rows) and SNIP Magnitude (bottom rows). The ground truth is shown the left-most image.
in our experiments. We prune models using our method and SNIP Magnitude, and visualize results for 10 linearly spaced pruning ratios between and 60% sparsity. Similarly the results for global understanding tasks shown in Figure 3] the representations of models pruned with SNIP Magnitude start to collapse 40% sparsity, while representations of models pruned with method are more robust, producing sensible segmentation masks even at 60% sparsity.
Table 11: DIIT-III ViT-H/14 retains performance across datasets at 50% sparsity. Top-1 accuracy in k-nearest neighbor and linear classification of DeIT-III ViT-H/14 model pruned to 50% sparsity with our method using a SSL loss, with and without weight correction, versus the original model.
Eval.
Sparsity
Cor
DTD
FGVC
ESAT
CIFAR 10
CIFAR 100
Pets
INIK
0%
62.4
30.1
89.9
96.8
86.1
92.4
86.6
k-NN
50%
x
58.9
35.5
88.6
91.9
73.2
91.7
79.0
50%
/
60.7
35.0
90.8
93.3
74.8
92.3
82.0
0%
69.2
22.0
92.9
97.4
90.2
93.9
86.5
Linear
50%
60.9
26.5
93.0
94.4
79.6
93.4
80.4
50%
67.9
32.3
93.7
95.2
82.3
93.8
82.8
31


Reviews:
[{"review_id": "FHFCQV7Ifg", "reviewer": "Reviewer_e4hx", "summary": "Vision foundation models are limited by fixed architectures that force suboptimal deployment decisions under real-world computational constraints. The paper presents a post-pretraining structured pruning method that enables elastic inference across different compute budgets by combining gradient information with cross-network structure correlations. These are approximated through evolutionary algorithms, requiring no labeled data or retraining. The approach significantly outperforms existing methods on DINO and AugReg models across various sparsity levels. It generates flexible models in under five minutes on an A100 GPU that can be dynamically adjusted to any computational budget.", "questions": "* It is clear that at 50% pruning you nicely improve over SOTA. But why does 50% matter? In my understanding, a key goal of novel pruning methods is to move the drop point, i.e., the point at which the model drastically loses in terms of performance. In this critical region, the paper on shows very minimal or even no improvements over the state of the art for some benchmark setting (Figure 2 a and b). I lack a discussion of this. \n\n* For others (e.g. Figure 3), they show nice improvements over but only over simple baseline (activations, LAMP) and do not compare against SOTA baselines. This makes it very hard for me to judge the contributions. Please close these gaps. \n\n* I suggest showing both the baselines and the authors' approach with and without post-pruning fine-tuning. Fine-tuning is affordable in many use-cases, and I think it is important to evaluate how \"fine tunable\" an approach is. Especially, as most baselines are designed for fine-tuning. \n\n* In some settings the authors report throughput, in others not. Why?\n\n* \"Our method improves over baselines, while not using labels:\" In my understanding, the baseline also do not use labels\n\n* \"Our method retains performance for downstream tasks:\" This statement feels a bit too strong, at least for the high pruning rates. \n\nThis paper is very interesting, but I lack:\n* motivation on the need for approaches without any fine-tuning\n* a focus on the critical regions of pruning\n* fully clean evaluation", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n* well-written paper\n* Important field: elastic models\n* nice: elastic models without the need for fine/tuning / retraining. \n* detailed supplementary material / appendix.\n\nWeaknesses:\n* limited improvements or even none in the critical accuracy regions in some settings\n* in other settings: lacks of SOTA baselines\n* strong focus on pruning without any finetuning, while many SOTA works include fine-tuning.", "was_revised": true, "final_justification": "I thank the authors for their quick response and the new insights. After also rechecking the other reviews, I increase my score to reflect the quality of their new results.", "rebuttal": "Thank you for the insightful feedback and for recognizing the importance of elastic models, specifically in their ability to operate without the need for finetuning, and for the well-presented paper and appendix. Below, we address each of the raised questions in detail:\n\n---\n## Motivation for pruning without the need for finetuning:\nOur work focuses on post-training structured pruning without finetuning to address increasingly common deployment and research needs:\n- Generalist, multi-task use cases: Many models today (e.g., CLIP, DINO, or AugReg) serve broad roles across diverse tasks, as we also show in our evaluation across seven bechmarks. However, finetuning for each downstream target might be infeasible due to proprietary constraints, latency budgets, or lack of labeled data. Robust, general-purpose pruning is thus essential.\n- Preserving robustness and generalization: We observe that pruning+finetuning on a narrow dataset (e.g., CIFAR-10) can reduce generalization (see e.g., accuracy drop of 3.1% from 64.2 to 61.1% via k-NN). This highlights that pruning **without** retraining can better preserve the inherent robustness encoded in foundation models.\n- Fast iteration & model churn: In rapidly evolving domains, frequent model updates make repeated fine-tuning costly. Our one-shot, retraining-free method offers a scalable alternative with strong performance across benchmarks.\n- Scientific insight: By avoiding finetuning, we can more directly analyze how pruning alters the learned representations. This links to ongoing foundational work on interpretability and the role of structural representations.\n\nWe will clarify these points more explicitly in the future version to help readers better understand the impact and scope of our work.\n\n---\n## Drop point analysis:\nWe apologize if the plots (e.g., Figure 2) were not sufficiently clear in illustrating our improvements near the drop point. To clarify, we provide a detailed table below that demonstrates consistent gains over FPTP and LLM surgeon in the 30, 40, and 50% sparsity range, on average across the seven benchmark datasets, while also using label supervision for pruning an AugReg ViT-B/16 backbone:\n\n| Sparsity | Method       | k-NN Avg. | Linear Avg. |\n|----------|--------------|-----------|-------------|\n| **30%**  | **Ours**         | **63.7**      | **70.4**        |\n|          | FPTP         | 60.7      | 68.3        |\n|          | LLM Surgeon  | 54.6      | 59.8        |\n|----------|--------------|-----------|-------------|\n| **40%**  | **Ours**         | **51.2**      | **57.7**        |\n|          | FPTP         | 44.7      | 51.5        |\n|          | LLM Surgeon  | 34.0      | 36.3        |\n|----------|--------------|-----------|-------------|\n| **50%**  | **Ours**         | **41.0**      | **43.6**        |\n|          | FPTP         | 34.5      | 36.3        |\n|          | LLM Surgeon  | 32.5      | 28.5        |\n\nThese results show that our method significantly outperforms FPTP and LLM surgeon, already before the 50% mark, thus delaying the drop point. In the response below, we show that we can recover from such drops via finetuning. We will better visualize and provide these results in the updated version.\n\n---\n## Baselines & full finetuning results:\nThank you for the suggestion. Our work focuses on pruning **without any retraining or finetuning**, motivated by the growing need to deploy large foundation models efficiently across diverse downstream tasks **without the computational cost or complexity of finetuning** (see earlier response).\n\nWhile finetuning can help recover accuracy after pruning, [1, 3] have shown that full retraining often not only recovers the original performance but even improves it. However, finetuning must be repeated for every sparsity level and each downstream task, which can be computationally expensive. For example, finetuning ViT-B for 100 epochs on ImageNet takes roughly 3 days on a single A100 GPU, and ViT-H requires more than 12 days. This poses a significant bottleneck in multi-task or dynamic deployment scenarios, or when models are frequently updated. Our work targets this more challenging but practically relevant scenario, requiring only about 5 minutes for pruning on a single A100 and no label supervision. Because of this, fewer directly comparable baselines exist, as most prior work assumes finetuning.\n\nThat said, we also demonstrate our method’s **finetunability**: pruning 50% of DeiT ViT-B and finetuning on ImageNet-1k achieves 81.5% top-1 accuracy, close to the unpruned baseline of 81.8% while delivering a 76% inference speedup. Our results are competitive with state-of-the-art pruning methods. We will include a comprehensive table with all sparsity levels in the updated manuscript. \n\n| Method  | ImageNet Top-1 | Average k-NN | Average Linear | Finetuning Epochs | Speedup  |\n|---------|----------------|--------------|----------------|--------|------------------|\n| Unpruned |  81.8   |    75.8         | 78.5     | –  | 0%     |\n| Ours    | 81.6          | 75.0         | 75.1          | 100     | 76% (A100)     |\n| NViT [1]   | 83.3           | 73.7         | 72.0           | 300    | 75% (A100)     |\n| LPViT [2]  | 80.6          | –         | –            | 300    | 79% (V100)     |\n| SAViT [3]  | 82.6          | –          | –            | 300    | 55% (V100)     |\n| SN-Net [4] | 80.0           | 70.2        | 71.4          | 100    | 83% (RTX 3090) |\n\nThank you for encouraging this experiment which demonstrates strong performance gains when finetuning is employed. We will add this experiment for all sparsities to our updated version.\n\n[1] Huanrui Yang et al. Global Vision Transformer Pruning with Hessian-Aware Saliency. CVPR 2023.\n\n[2] Kaixin Xu et al. LPViT: Low-Power Semi-structured Pruning for Vision Transformers. ECCV 2024\n\n[3] Chuanyang Zheng et al. SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization. NeurIPS 2022\n\n[4] Zizheng Pan, Jianfei Cai, Bohan Zhuang: Stitchable Neural Networks. CVPR 2023\n\n---\n## Throughput vs GFLOPS:\nIn Figure 2 (a, b), we report GFLOPs as floating-point operations per inference pass, as the LLM Surgeon and FPTP mask out pruned structures rather than effectively removing them, making it non-trivial to realize the theoretical speedups in practice. Therefore, we do not report throughput for these baselines and compare them to our method only in terms of GFLOPs. In contrast, our method and NViT produce pruning patterns that are hardware-friendly and straightforward to realize, so we report throughput for these methods as it better reflects actual runtime performance.\n\n---\n## Labels for pruning:\n\nWe appreciate the reviewer’s comment and would like to clarify. While we do offer a cross-entropy (CE) variant of our method for comparison, our primary approach does not require any labels, instead relying on a self-supervised loss (Eq. 5) during pruning.\nIn contrast, most baselines, including FPTP, LLM surgeon, and NViT, rely on supervised cross-entropy losses and assume access to a linear classification head during pruning. This limits both their flexibility and generalization, especially for pretrained models where such heads are unavailable; thus, we cannot report a comparison with those methods for DINOv1, DINOv2, and CLIP.\nWe outperform these baselines even without using labels, showing that meaningful structure-aware pruning is possible in a fully unsupervised setting (see Table 1). \n\n---\n## Retaining performance claim:\nThank you for pointing this out. We agree that the phrasing “retains performance for downstream tasks” may be too strong, especially at very high sparsity levels. While we do observe strong retention even at high sparsities for big models (e.g., Table 9 in the appendix; DeIT-III ViT-H/14), the drop in performance at extreme pruning rates is indeed dataset- and model-dependent.\nWe will revise the wording in the future version to more accurately reflect this nuance, emphasizing that our method preserves downstream performance well in the practical sparsity range (e.g., 30-40%), which is also where it most consistently outperforms baselines."}, {"review_id": "msAU5k2wbx", "reviewer": "Reviewer_ZyhL", "summary": "This paper introduces a structured pruning method for Vision Transformers (ViTs) that enables the creation of elastic inference models directly from pretrained models, eliminating the need for retraining. The proposed approach combines second-order sensitivity measures with evolutionary algorithm-based approximations. The authors demonstrate substantial improvements over several baseline methods across a range of image classification and segmentation tasks.", "questions": "- How sensitive is the performance of the evolutionary algorithm to the choice of initial parameters or random seeds?\n\n- What impact would including first-order terms have on pruning performance, given that these terms are typically non-zero?\n\n- How does the proposed pruning approach compare to random pruning when evaluated using KNN-based methods?\n\n- The authors mention challenges in applying their pruning method to large models trained on large-scale datasets. What specifically were these challenges—reduced performance, computational inefficiency, or something else?\n\n- Are there any limitations to applying this approach to Vision Transformers (ViTs), or is it equally effective for other model architectures?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n-To the best of my knowledge, the combination of gradient-based pruning with an evolutionary approximation of Hessian correlations is a novel and innovative approach.\n\n-The method demonstrates strong performance gains across multiple benchmark datasets and various Vision Transformer architectures, including DINO, AugReg, and CLIP.\n\n-It achieves effective pruning of Vision Transformers with minimal computational overhead, making it well-suited for practical deployment scenarios.\n\n-The paper includes a thorough analysis of performance in relation to throughput and FLOPs, offering valuable insights into the trade-offs involved.\n\nWeaknesses:\n\n-The paper lacks an analysis of computational complexity, like done in SOSP (Second-order Structured Pruning). Additionally, SOSP and similar approaches are not mentioned in the related work section as viable alternatives for approximating the full Hessian in pruning (line of Approach using Gauss-Newton-Approximation also done in Eigendamage). Including this comparison (in terms of complexity) would help clarify the scalability of the proposed method.\n\n-There is no comparison with supervised pruning methods that incorporate fine-tuning after pruning. Including such comparisons would strengthen the evaluation and address a current weakness of this work, which is that it only benchmarks against three other methods.", "was_revised": true, "final_justification": "This paper presents a novel structured pruning approach for ViTs that combines second-order sensitivity with evolutionary search, enabling elastic inference without retraining. The rebuttal addressed my key concerns by adding a complexity analysis, comparisons to finetuning-based baselines, and robustness studies, which significantly strengthen the contribution. Overall, I find the paper technically solid, well-motivated, and impactful for practical deployment, and I recommend acceptance.", "rebuttal": "Thank you for the valuable feedback and for recognizing the novelty of our approach, its strong performance across benchmarks, the effective pruning of Vision Transformers with minimal computational overhead, and our analysis of performance for throughput and FLOPs. Below, we address each of the raised questions:\n\n---\n## Analysis of computational complexity:\nWe thank the reviewer for requesting a formal complexity analysis and for pointing us to SOSP and EigenDamage. Both papers will be added to the related‑work section as well as the analysis.  \nThe algorithm’s total cost is\n$$\n\\mathcal{O}(N_D F)\n\\+\n\\mathcal{O}(T \\lambda S N_S F')\n\\+\n\\mathcal{O}(T B^{2})\n\\+\n\\mathcal{O}(P \\log P),\n$$\n\nwhere  \n\n* **\\\\(\\mathcal{O}(N_D F)\\\\)** computes the local diagonal Hessian via one forward–backward pass on \\\\(N_D\\\\) samples (\\\\(F\\\\) per pass). \n* **\\\\(\\mathcal{O}(T \\lambda S N_S F')\\\\)** in every generation of the xNES, we draw \\\\(\\lambda\\\\) candidate masks, test each one at \\\\(S\\\\) sparsity targets on \\\\(N_S\\\\) images; \\\\(F'\\\\) is a forward‑only pass (feature extraction + PCA), so no back‑propagation is involved.\n\n* **\\\\(\\mathcal{O}(T B^{2})\\\\)** updates the \\\\(B \\\\times B\\\\) covariance matrix \\\\( \\Sigma \\\\) in xNES (FFN + head blocks), capturing global off‑diagonal structure. \n* **\\\\(\\mathcal{O}(P \\log P)\\\\)** sorts \\\\(P\\\\) prunability scores once to yield elastic subnetworks. \n\nThe key speed‑up over SOSP stems from **zero backward curvature calls**: SOSP‑H requires one Hessian–vector product per structure and SOSP‑I builds a dense \\\\(S \\times S\\\\) Gauss–Newton matrix, whereas we rely exclusively on forward inference and a lightweight covariance update \\\\(\\mathcal{O}(T B^{2})\\\\) (contributing <0.01%).\n\nRuntime is therefore dominated by feature extraction.  With 50 xNES iterations and four sparsity targets, a single A100 prunes the entire DeiT‑III family in 2–11 min: ViT‑S(2 : 35), ViT‑B(2 : 55), ViT‑L(4 : 58), and the 32‑block ViT‑H(11:04).  The quadratic covariance term remains negligible throughout, demonstrating efficient scalability even for the largest ViTs.\n\n---\n## Finetuning comparison:\nThank you for highlighting this. Our work intentionally targets the supervision-free, no-finetuning pruning setting, motivated by real-world deployment constraints (e.g., multi-task inference, edge applications, or frequent model updates), where finetuning is often infeasible or expensive. In such cases, retraining must be repeated for each sparsity level, dataset, and task. For instance, finetuning a ViT-B on ImageNet takes ~3 days on a single A100 GPU, whereas finetuning a ViT-H requires over 12 days, making scaling challenging.\n\n\nThat said, we agree that comparing with supervised finetuning-based methods is important to analyze the effectiveness of our approach. We have now added comparisons to several state-of-the-art pruning methods that assume full supervision and post-pruning finetuning, including SN-Net [4], NViT [1], LPViT [2], and SAViT [3]. As shown below, our method is highly competitive even in the finetuning setting: at 50% sparsity, we achieve 81.6% ImageNet top-1 accuracy after finetuning, nearly matching the unpruned baseline (81.8%) while offering a **76% inference speedup**.\n\n| Method     | ImageNet Top-1 | Avg. k-NN | Avg. Linear | Finetuning Epochs | Speedup     |\n|------|---|-|--|--|-------------|\n| Unpruned   | 81.8   | 75.8      | 78.5        | –      | 0%          |\n| Ours       | 81.6   | 75.0      | 75.1   | 100    | 76% (A100)  |\n| NViT [1]    | 83.3 | 73.7      | 72.0     | 300         | 75% (A100)  |\n| LPViT [2]   | 80.6     | –         | –     | 300        | 79% (V100)  |\n| SAViT [3]   | 82.6  | –         | –      | 300   | 55% (V100)  |\n| SN-Net [4]  | 80.0   | 70.2  | 71.4    | 100   | 83% (3090)  |\n\nWe will include the full finetuning results for all sparsities in the updated manuscript. While our primary contribution is enabling **fast, label-free pruning without retraining**, we show that our method also remains **finetuning-compatible and competitive**.\n\n[1] Yang et al., *Global Vision Transformer Pruning with Hessian-Aware Saliency*, CVPR 2023  \n\n[2] Xu et al., *LPViT: Low-Power Semi-structured Pruning for Vision Transformers*, ECCV 2024  \n\n[3] Zheng et al., *SAViT: Structure-Aware Vision Transformer Pruning*, NeurIPS 2022  \n\n[4] Pan et al., *Stitchable Neural Networks*, CVPR 2023\n\n---\n## Method robustness to initialization and random seeds\n\nWe investigate the sensitivity of our pruning approach to variations in random seeds and the initialization parameters of the evolutionary algorithm.\n\n#### **Random Seeds**  \nTable 7 in the Appendix reports k-NN and linear classification accuracy (mean ± standard deviation) across three random seeds for an AugReg ViT-B/16 backbone pruned to 10%, 30%, and 50% sparsity. We observe that performance variance is minimal at low sparsity levels (e.g., an average of 0.2% at 10%) and increases modestly at higher sparsity levels (up to 1-2%). Some datasets, such as EuroSAT, remain stable across seeds, while others, like Oxford-IIT Pets, are more sensitive, particularly in the k-NN setting. This suggests that our method is generally stable, with expected variability under more aggressive pruning.\n\n#### **xNES Iterations (\\\\( T \\\\))**  \nAs shown in Table 3, increasing the number of xNES iterations from 50 to 250 and then to 500 results in an approximate 2% performance improvement, indicating that the method converges effectively even with relatively few update steps.\n\n#### **Initialization of Covariance Matrix (\\\\( \\Sigma \\\\))**  \nWe found that initializing the xNES covariance matrix \\\\( \\Sigma \\\\) using CKA similarity scores between architectural components (e.g., attention heads and MLP blocks) leads to faster convergence and significantly improved pruning outcomes. As shown in Table 8, appendix, this initialization improves both k-NN and linear accuracy across all datasets, for instance, k-NN accuracy on ImageNet-1k increases from 19.5% to 40.4%, and on Oxford-IIT Pets from 27.6% to 58.0%, highlighting the effectiveness of structure-aware initialization.\n\nOur pruning method thus shows strong robustness across random seeds, xNES iteration counts, and initialization of \\\\( \\Sigma \\\\).\n\n---\n## Impact of including First-Order term\nThanks for pointing this out. We explored adding explicit first-order information to the pruning score in addition to the local Hessian (the diagonal approximation of the Hessian). To ensure compatibility with our genetic algorithm, which constrains weights to be non-negative , we use the absolute values of the gradient terms.\n\nBelow, we report k-NN and linear classification accuracy (%) on an AugReg ViT-B/16 model across sparsity levels:\n\n| Sparsity | k-NN (Local Hessian) | k-NN (Local Hessian + Gradient) | Linear (Local Hessian) | Linear (Local Hessian + Gradient) |\n|---|-----|-------|---|-----|\n| 0%       | 73.6                 | 73.6        | 73.6    | 73.6     |\n| 10%      | 72.8                 | 72.7     | 78.1     | 78.6       |\n| 20%      | 68.2                 | 65.7      | 74.5     | 71.8   |\n| 30%      | 60.7                 | 57.4             | 67.3      | 64.8        |\n| 40%      | 47.3                 | 42.6           | 53.2                   | 46.9      |\n| 50%      | 39.3                 | 38.3         | 42.5       | 40.4           |\n\nThe gradient term introduces noise from first-order optimization dynamics, which appears to distort the fitness signal used during pruning. In contrast, the local Hessian alone provides a more stable and reliable criterion, which is why we retain it as our default.\n\n---\n## Comparison to Random Pruning \nThe table below reports average k-NN accuracy (%) across seven benchmark datasets for an AugReg ViT-B/16 model pruned using our method versus random pruning. Both methods start from the same unpruned baseline.\n\n| Sparsity | 0%   | 10%  | 20%  | 30%  | 40%  | 50%  | 60%  |\n|---|-------|-------|-------|-------|-------|-------|-------|\n| Ours     | 73.6  | 72.8  | 68.2  | 60.7  | 47.3  | 39.3  | 35.9  |\n| Random   | 73.6  | 66.9  | 47.7  | 37.6  | 33.3  | 30.2  | 26.9  |\n\nOur method consistently outperforms random pruning at all sparsity levels, with particularly large gains at moderate-to-high sparsity (e.g., +21% at 30% sparsity). These results demonstrate the value of structure-aware pruning: unlike random pruning, our approach preserves semantically meaningful representations that remain effective for downstream tasks without requiring finetuning.\n\n---\n## Challenges with large models:\n\nLarge Vision Transformers like ViT-H are pretrained on hundred-million–scale datasets, which leads to a much flatter loss landscape. As a result, the xNES optimizer requires roughly 75–100 warm-up iterations just to explore the landscape before the loss begins to decrease. To obtain a pruning mask of comparable quality, we therefore need to **extend the search horizon \\\\( T \\\\)**.\n\nWhile this increases the number of optimization steps, we find that larger models are better to prune, as they contain more redundant parameters. For example, ViT-H can be pruned to reduce **40% of GFLOPs with virtually no performance loss** (see Table 9 in the appendix). \n\n---\n## Extendability of our approach\nWhile our paper focuses on ViTs, the method is not architecture-specific. It requires only (1) a gradient-based loss, (2) structural units to prune (e.g., layers, blocks, filters), and (3) a fitness function for the genetic algorithm. This general framework makes it applicable to other architectures like CNNs, and domains such as language modeling or audio. For instance, in language models, one could prune attention heads and MLP blocks using a next-token prediction loss."}, {"review_id": "ulcggW0nL8", "reviewer": "Reviewer_69Bj", "summary": "This paper presents a novel one-shot pruning method for large-scale, pretrained neural networks, with a focus on Vision Transformers (ViTs). The proposed approach enables the creation of sparse subnetworks within a specified computation budget by assigning a prunability score to each structural unit (such as attention heads or feed-forward blocks) and removing the least important elements.\nThe core of the method is a blockwise Hessian decomposition that approximates the loss sensitivity to parameter removal, capturing both intra-block (within each component) and inter-block (between components) dependencies. The local Hessian (intra-block) is efficiently estimated per block using diagonal approximations and self-supervised learning losses to ensure the approach can be applied regardless of the model’s output head. For the global Hessian (inter-block), the algorithm employs a natural evolution strategy (xNES) to implicitly model the cross-block dependencies via a learned covariance structure without computing the full Hessian.", "questions": "1) The use of xNES to approximate inter-block dependencies is the most novel part of your work. However, the connection Equation (9), where Σ is the covariance matrix from xNES, is presented as a given. Could you please provide more intuition or a brief theoretical justification for this relationship? For example, why is the inverse of the covariance of optimal sensitivity parameters a good proxy for the global Hessian? A clearer explanation here would significantly strengthen the paper's theoretical grounding and my assessment of its originality and quality. Specifically, how the fitness function F (based on PCA-compressed embeddings) directly translates into an optimal Σ that accurately represents the Hessian's global structure requires deeper elaboration. \n\n2) The paper note that pruning massive models like DINOv2 and CLIP was \"challenging\" and required tuning the xNES algorithm (e.g., more iterations). This raises a question about the method's scalability. Could you elaborate on the computational complexity of the xNES stage, particularly how it scales with the number of prunable blocks B? Furthermore, the fitness function (Eq. 8) relies on a discrete set of sparsity levels S. How sensitive are the final results to the choice and number of these sparsity levels? \n\n3) This method prunes FFN neurons and entire attention heads. Could the framework be extended to other structures (e.g., individual attention weights, LoRA-like low-rank components)? More importantly, the fitness score is based on the cosine similarity of PCA-projected embeddings. How was the dimensionality (192) chosen, and how does this choice impact the final pruning quality? For instance, does using a higher-dimensional representation in the fitness evaluation lead to better-performing pruned models, perhaps at the cost of more computation? \n\n4) While the results show competitive performance, the figures (especially Figure 4) suggest a significant accuracy drop at higher sparsity levels . Could the authors provide a more explicit analysis of the performance decay curves at extreme sparsity levels for these challenging models? Quantifying the acceptable sparsity range for \"strong performance\" and discussing the inherent limitations or trade-offs when pushing sparsity very high would be valuable. For instance, a comparison of accuracy loss vs. GFLOP reduction at extreme sparsities would be informative.", "limitations": "For further improvement, it might be beneficial to also briefly mention the computational scaling of the evolutionary search itself as a potential limitation, which connects to the challenges observed with larger models.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nThe paper addresses a critical and timely problem: the difficulty of deploying large, fixed-size foundation models under varying real-world computational constraints.  The proposed approach of using xNES to find a block-wise sensitivity vector $c$ that captures these complex correlations is a creative and principled departure from prior work. It cleverly reframes a difficult analytical problem (computing off-diagonal Hessian blocks) as a black-box optimization problem.\n\nWeaknesses:\n1) The paper's most novel component is also its most conceptually dense. Taking Equation (9) as an example , and the paper would be stronger if it provided a more detailed explanation of why the inverse of this covariance is a suitable approximation for the global Hessian.\n2) The authors commendably acknowledge in the limitations section that pruning very large-scale models is challenging and required tuning the xNES algorithm. For future, even larger models with hundreds or thousands of blocks, this search could become a bottleneck. A brief discussion on the computational complexity of the xNES stage and how it scales with model size (B) would be a valuable addition.", "was_revised": true, "final_justification": "I am still concerned about the time cost associated with applying evolutionary algorithms to pruning, as it may limit its applicability.", "rebuttal": "Thank you for the valuable feedback and for recognizing our paper’s focus on the timely and important challenge of deploying large foundation models. We appreciate the acknowledgment of our method as a creative and principled departure from prior work. Below, we address each of the raised questions:\n\n---\n## Relationship between Sigma and Hessian: \nThank you for pointing this out. We agree that the xNES–Hessian relation in Eq. (9) needs a clearer rationale. We will add the following justification to the paper.\nDuring optimisation, xNES samples the block‑sensitivity vector \\\\(c\\\\) from a Gaussian search distribution \\\\( \\mathcal{N}(\\mu,\\Sigma) \\\\).  When the fitness surface is *locally well‑approximated by a positive‑definite quadratic*,  \\[1\\] prove that for a \\\\((1,\\lambda)\\\\)-ES with isotropic mutation, the selected‑population covariance converges to a scalar multiple of the inverse Hessian\n$$\n\\Sigma \\ \\propto\\ \\bigl(\\nabla^{2}F(c)\\bigr)^{-1}.\n$$\nNatural‑gradient analyses of xNES and CMA‑ES \\[2\\] predict the same tendency, because steep directions (large eigenvalues of \\\\( \\nabla^{2}F \\\\)) receive **low variance** while flat directions receive **high variance**.  Taking the inverse yields our working rule\n$$\n\\nabla^{2}F(c) \\\\approx\\ \\alpha\\\\Sigma^{-1}, \\qquad \\alpha>0. \\tag{9 revisited}\n$$\n*Intuition.* xNES narrows its sampling ellipse in high‑curvature directions to avoid overshooting and widens it in flat directions to explore; repeated updates push \\\\( \\Sigma^{-1} \\\\) toward the Hessian, capturing both local and inter‑block sensitivity.\n\nOur fitness \\\\(F(c)\\\\) is not the raw loss \\\\( \\mathcal{L} \\\\); it is a cosine‑similarity score on embeddings compressed to 192 PCA components and averaged over several sparsity levels.  PCA removes high‑frequency noise, and sparsity averaging further smooths the surface, so \\\\(F\\\\) is locally smooth near good pruning vectors even though it is globally non‑convex.  Thus, the second‑order approximation remains reliable.  We therefore treat \\\\( \\Sigma^{-1} \\\\) as a local Hessian proxy and use it to re‑weight first‑order pruning gradients (Sec. 3.4).  This hybrid score captures both intra‑ and inter‑block sensitivity. It consistently improves pruning robustness on large Vision Transformers.\n\n\\[1\\] Shir, O. M., & Yehudayoff, A. (2019). *Covariance Matrix Adaptation and Inverse Hessian on Convex Quadratic Functions*. FOGA 2019.  \n\n\\[2\\] Akimoto, Y., Nagata, Y., Ono, I., & Kobayashi, S. (2010). *Bidirectional Relation between CMA‑ES and NES*. PPSN XI.\n\n---\n## Scalability of the xNES stage\nThe evolutionary algorithm consists of two cost terms.\n\n$$\n\\mathcal{O}\\bigl(T \\lambda S N_{S} F' \\bigr)\n+\n\\mathcal{O}\\bigl(T B^{2}\\bigr)\n$$\n\nwhere the first term is the **forward‑only fitness evaluation**  \\\\(T\\\\) iterations × \\\\(\\lambda\\\\) population × \\\\(S\\\\) sparsities × \\\\(N_{S}\\\\) samples, each with cost \\\\(F'\\\\)and the second term is the **covariance update** for the \\\\(B \\times B\\\\) matrix maintained by xNES.\n\n* \\\\(T\\\\):  xNES iterations (generations)  \n* \\\\(\\lambda\\\\): population size per generation  \n* \\\\(S\\\\): \\# sparsity levels optimised jointly  \n* \\\\(N_S\\\\): images per fitness call  \n* \\\\(F'\\\\): cost of a forward pass (feature extraction + PCA)  \n* \\\\(B\\\\): count of prunable blocks (FFN slices + heads)\n\nThe quadratic term touches only the \\\\(B \\times B\\\\) covariance and is negligible; runtime is dominated by forward passes whose cost grows with the **parameter count** of the ViT backbone.\n\n| Model | Params | Layers | Heads | Runtime on one A100 (50 iters, 4 levels) |\n|--|----:|--:|------:|----|\n| **DeiT‑III ViT‑S/16** | 22 M | 12 | 6  | **2 min 35 s** |\n| **DeiT‑III ViT‑B/16** | 86 M | 12 | 12 | **2 min 55 s** |\n| **DeiT‑III ViT‑L/16** | 304 M | 24 | 16 | **4 min 58 s** |\n| **DeiT‑III ViT‑H/14** | 632 M | 32 | 16 | **11 min 4 s** |\n| **EVA ViT-G/14 [3]** |  1B   | 40     | 16    | **20 min 5s**            |\n\n[3]: EVA: Exploring the Limits of Masked Visual Representation Learning at Scale, CVPR 2023\n\nThe near‑linear increase from S → B → L → H→G confirms that the model size (feature dimension and parameter size, hence forward cost) is dominant. Emperically, the \\\\(TB^{2}\\\\) term contributes \\< 0.01% even for ViT‑H. Models trained on hundred–million–scale corpora have flatter curvature.  Empirically, ViT‑L and ViT‑H spend their first \\\\(\\approx 75\\\\)–100 iterations exploring before the xNES loss declines steadily.  Increasing \\\\(T\\\\) improves mask quality but leaves the per‑iteration cost unchanged.\n\n## Sensitivity to sparsity levels \nOur default setting uses a grid of four target sparsities: 10%, 25%, 40%, and 60%. We find that increasing the number of sparsity levels in \\\\(S\\\\) can lead to improved pruning performance with minimal overhead, as the fitness evaluations are forward-only and parallel.\nFor example:\nOn AugReg ViT‑B/16 (86M params), expanding \\\\(S\\\\) from 4 to 6 yields a +1.7% average accuracy gain.\nOn DINOv2 ViT‑B/14 (88M params), the same change results in gains of up to +12.9% on certain tasks.\n\n\nThis shows that pruning quality is moderately sensitive to the choice of \\\\(S\\\\), and practitioners can easily trade off evaluation cost for accuracy by adjusting grid resolution.\n\n\n---\n## Framework Extendability\n\nThank you for the suggestion. Our framework is naturally extendable to prune other structures beyond FFN neurons and attention heads. Since the genetic algorithm learns real-valued weights and is agnostic to the specific architectural units being pruned, it generalizes to any modular network component. This includes individual attention weights and even low-rank modules like LoRA.\n\n\nWe conducted an experiment where we pruned individual attention weights rather than entire heads. For that, we use the same weighting from the genetic algorithm, but when creating subnetworks from the prunability score, we pruned individual attention weights rather than whole heads. However, while this is technically feasible, the practical speed-up is limited, as hardware accelerators cannot exploit sparsity at the fine-grained level of individual weights. In contrast, pruning entire heads allows structural simplifications that lead to realized speedups. Below, we report the linear classification accuracy for an AugReg ViT-B/16 backbone across different sparsity levels, comparing attention head pruning vs. individual attention weight pruning:\n\n| Sparsity | Ours (Attention Weights) | Ours (Attention Heads) |\n|----------|-----------------------|----------|\n| 0%       | 73.6          | 73.6        |\n| 10%      | 71.6        | 71.8  |\n| 20%      | 64.2          | 65.1  |\n| 30%      | 49.3            | 52.5 |\n| 40%      | 38.7        | 41.2   |\n| 50%      | 35.1     | 36.3   |\n\nWe observe that pruning entire heads consistently outperforms finegrained pruning. This aligns with evidence that attention heads function as coherent computational units [4, 5], and pruning individual weights inside them can disrupt their effectiveness. Previous work that finds finegrained pruning to outperform structured pruning typically relies on finetuning to recover performance after pruning, which is not part of our method’s setup.\n\n[4] Michel, P., Levy, O., & Neubig, G. (2019). Are Sixteen Heads Really Better than One? NeurIPS 2019.\n\n[5] Voita, E., Talbot, D., Moiseev, F., Sennrich, R., & Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019.\n\n\n### PCA dimensionality. \nWe ablated PCA dimensions to assess their effect on pruning quality. Increasing the PCA size from 192 to 384 led to a modest improvement in average linear classification accuracy (42.5% → 42.9%) at 50% sparsity on AugReg ViT-B/16, though with increased computational cost. Not using PCA at all resulted in a notable drop to 41.5%, indicating PCA helps both reduce computation and regularize the embedding space for better fitness evaluation. We chose 192 dimensions as a practical balance between performance and efficiency, making it the default setting for our experiments.\n\n---\n## Extreme sparsity levels:\nThe trade-off between sparsity and performance is highly dependent on both model type and model size. We found that large-scale models—such as ViT-H DeiT-III with 630M parameters—can be pruned effectively with minimal performance loss at 40% sparsity (Table 9). Figure 10 further illustrates the relationship between performance degradation and GFLOPs for ViT-S, ViT-B, ViT-L, and ViT-H models from the DeiT-III family, pruned up to 60%. Notably, larger models tend to have more redundant parameters, making them more amenable to pruning.\n\nTo quantify pruning efficiency, we introduce an efficiency score—defined as the percentage of GFLOPs that can be reduced while retaining at least 70% of the original performance. According to this metric, ViT-S allows a 40% GFLOPs reduction, while ViT-B and ViT-L achieve 56%, and ViT-H up to 60%. Additionally, as shown in Figure 4, different model architectures exhibit varying degrees of prunability. For instance, CLIP models demonstrate better tolerance to high sparsity levels compared to DINO models.\n\nOur primary focus is on pruning without any retraining or finetuning, driven by the increasing demand for efficient deployment of large foundation models across diverse downstream tasks. Under this constraint, we typically observe a significant performance drop beyond 50% sparsity. However, as demonstrated in our responses to Reviewers e4hx and ZHY4, this degradation can be nearly fully recovered through finetuning—bringing our method in line with state-of-the-art approaches in retraining settings.\n\nWe will include a more detailed discussion of the trade-off between performance and compute reduction in the revised version of the paper. For additional insights, please refer to our response to Reviewer e4hx regarding performance prop point analysis."}, {"review_id": "viTZnxBnov", "reviewer": "Reviewer_ZHY4", "summary": "This paper introduces a post-pretraining structured pruning method that enables elastic inference across a continuum of compute budgets. Specifically, the authors propose a evolutionary approximation of Hessian off-diagonal structures, and a self-supervised importance scoring mechanism that maintains performance without requiring retraining nor labels.", "questions": "1. The “Hessian matrix” and “evolutionary algorithm” are widely used across various domains (for example, in neural architecture search). How are these two components effectively integrated within the proposed pruning method? In other words, what is the motivation for employing parameter sensitivity in this work?\n2. This paper performs pruning on a large pretrained model to accommodate different deployment scenarios. Existing works—such as the SN-Net series [1]—also leverage pretrained models to handle resource-constrained environments. Could the authors provide a detailed discussion comparing their approach to SN-Net? More generally, what advantages do pruning-based methods offer over alternative strategies?\n3. In the experimental section, the term “GFLOPs” is reported. What is the precise meaning of “GFLOPs” in this context? Does it refer to the use of models of varying scales, or does it denote another specific metric?\n\n**Reference**\n\n[1] Stitchable Neural Networks", "limitations": "yes.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The multiple technical modules employed in this paper—such as the evolutionary algorithm for Hessian off-diagonal structures—are all well-established and mature components.\n\n\n**Weaknesses**\n1. **Unclear Integration of Hessian Analysis and Evolutionary Search**\n   The paper does not adequately explain how Hessian matrix computations and evolutionary algorithms are synergistically combined in the proposed pruning method. Specifically, it remains ambiguous how parameter sensitivity—derived from Hessian off-diagonal structures—is calculated and utilized within the evolutionary search process. This lack of clarity obscures the method’s underlying motivation and makes it difficult to assess its novelty or effectiveness.\n\n2. **Insufficient Comparison with Existing Pretrained-Model Adaptation Methods**\n   Although the paper aims to prune a large pretrained model for various deployment scenarios, it does not offer a rigorous comparison with existing approaches like SN-Net[1]. The authors fail to elucidate what unique benefits their pruning-based strategy provides relative to SN-Net’s methodology or other resource-adaptation techniques. Without a detailed comparison, readers cannot gauge the relative strengths and weaknesses of the proposed approach.\n\n3. **Ambiguity in the Definition**\n   The term “GFLOPs” is used throughout the experimental results but is not explicitly defined. It is unclear whether GFLOPs refers to floating-point operations per second, the computational cost of different model scales, or another metric altogether. This ambiguity hinders the reproducibility of the experiments and prevents a clear interpretation of the reported efficiency gains.\n\n\n**Reference**\n\n[1] Stitchable Neural Networks", "was_revised": true, "final_justification": "Thank you! I keep my current rating.", "rebuttal": "Thank you for the valuable feedback and for recognizing that the technical modules are well-established and mature components. Below, we address each of the raised questions.\n\n----\n## Integration of Hessian and Evolutionary:\nOur approach decomposes parameter sensitivity into local and global components. The local Hessian, approximated via squared gradients, provides parameter-level sensitivity within blocks, capturing the diagonal Hessian terms efficiently. The global Hessian, which encodes off-diagonal dependencies between blocks, enables the xNES evolutionary algorithm to approximate implicitly. Our algorithm can be summarized with the following pseudo-code\n\n```text\n# Pre-compute local importance\ns ← diag-Hessian scores\n\n# Initialise xNES mean and covariance\nm ← 0; Σ ← I\n\n# for T iterations and λ population size\nfor t = 1 … T: \n  for k = 1 … λ: \n    # Sample global re-weighting factors\n    c ← sample 𝒩(m, Σ)\n\n    # Combine local and global terms\n    score ← c ∘ s\n\n    # Prune K lowest-score blocks\n    mask ← TopK(score, K)\n\n    # Measure label and backward-free fitness via embedding similarity (Eq. 8)\n    fit ← cos(PCA(z), PCA(z▹mask))\n\n  # Natural-gradient update of mean and covariance\n  (m, Σ) ← natural-grad update(fit)\n```\nBecause the covariance matrix \\\\( \\Sigma \\\\) is updated by a natural-gradient step, it gradually aligns with the inverse block-Hessian (see response to R69Bj). Once the search stabilizes, \\\\( c \\sim\\mathcal{N}(m,\\Sigma) \\\\) re-weights the fixed local Hessian scores, yielding a prunability vector \\\\( P \\\\) (see Eq. 10). Sorting \\\\( P \\\\) defines a universal parameter ranking so that a single cutoff can obtain any desired sparsity—no extra optimization, Hessian storage, or retraining. A lone evolutionary run thus produces a full continuum of compute-adaptive subnetworks. We will add the pseudo-code to the updated paper to improve clarity. \n\n\n\n---\n## Comparison with previous works:\n\nThank you for drawing attention to this. SN‑Net [4] achieves adaptability by bridging between several fully pretrained backbones from the same family (e.g., ViT‑S/B/L of DeiT) via additional stitching layers that must be trained and stored alongside the original model. By contrast, our method performs one‑shot structural pruning: a single pass over a pretrained model produces a family of subnetworks that require no extra layers and no further finetuning. We will add SN-Net and others to our related work section.\n\nWe follow previous works [1,2,3,4] and finetune our method after pruning on ImageNet-1k. We prune 50% of DeiT ViT-B and finetune for 100 epochs. We achieve 81.5% top-1 accuracy—close to the unpruned baseline of 81.8%—while delivering a 76% inference speedup. Our results are competitive with state-of-the-art pruning methods while outperforming the SN-Net baseline by 1.6%. Our method generalizes well, outperforming other works on average performance for k-nn and linear across six datasets. \n\n| Method  | ImageNet Top-1 | Average k-NN | Average Linear | Finetuning Epochs | Speedup  |\n|---------|----------------|--------------|----------------|--------|------------------|\n| Unpruned |  81.8   |    75.8         | 78.5     | –  | 0%     |\n| Ours    | 81.6          | 75.0         | 75.1          | 100     | 76% (A100)     |\n| NViT [1]   | 83.3           | 73.7         | 72.0           | 300    | 75% (A100)     |\n| LPViT [2]  | 80.6          | –         | –            | 300    | 79% (V100)     |\n| SAViT [3]  | 82.6          | –          | –            | 300    | 55% (V100)     |\n| SN-Net [4]  | 80.0           | 70.2      | 71.4        | 100               | 83% (RTX 3090)  |\n\nWe will add full finetuning experiments to the revised manuscripts covering all sparsity levels.\n\n[1] Huanrui Yang et al. Global Vision Transformer Pruning with Hessian-Aware Saliency. CVPR 2023.\n\n[2] Kaixin Xu et al. LPViT: Low-Power Semi-structured Pruning for Vision Transformers. ECCV 2024\n\n[3] Chuanyang Zheng et al. SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization. NeurIPS 2022\n\n[4] Zizheng Pan, Jianfei Cai, Bohan Zhuang: Stitchable Neural Networks. CVPR 2023\n\n---\n## GFLOP Definition:\nWe use the term GFLOPs to indicate the number of theoretical floating-point operations required for a forward pass, following [5, 6]. We will add the following calculation to the manuscript:\n\n\nLet \\\\( n_p \\\\) be the number of image patches of size \\\\( d_p \\times d_p \\\\) pixels,  \\\\( c \\\\) input channels, \\\\( d \\\\) hidden width, \\\\( n = n_p + 1 \\\\) tokens (CLS + patches),  \\\\( h \\\\) attention heads, \\\\( L \\\\) layers, and \\\\( C \\\\) classes.  With the usual FFN ratio 4 (\\\\( d_{\\text{ff}} = 4d \\\\)), the forward-pass FLOPs are\n\n$$\n\\text{FLOPs} =\n2 n_p d_p^{2} c d\n+\nL \\bigl[ 8 n d^{2} + 4 n^{2} d + 3 h n^{2} + 16 n d^{2} \\bigr]\n+\n2 d C .\n$$\n\nThe bracketed terms contain the self-attention and FFN term; the outer terms are patch embeddings and output logits.  \n\nFor DeiT ViT-B/16 (\\\\( n_p = 196,\\\\, d_p = 16,\\\\, c = 3,\\\\, d = 768,\\\\, h = 12,\\\\, n = 197,\\\\, L = 12,\\\\, C = 1000 \\\\)) this evaluates to\n\n$$\n\\text{FLOPs} \\approx 35.2\\ \\text{GFLOPs},\n$$\n\nor \\( 17.6 \\) GMACs (multiply–accumulates).\n\n[5]: Training Compute-Optimal Large Language Models, NeurIPS 2022\n\n[6]: Adam Casson, Transformer FLOPs"}]

Author rebuttal:

Meta-review: The authors propose a structured pruning method to enable elastic inference across a range off pre-defined sparsity levels, the primary novelty of the method is to use an approximate decomposition of the Hessian, both intra-block and inter-block, to inform the saliency criteria for the pruning method. The reviewers all agreed the proposed methodology was novel and well motivated. Most of the initial concerns of the reviewers were resolved in a robust discussion between the reviewers and authors. The main points of concern for the reviewers that appear to remain post-rebuttal are the scalability/computational efficiency of the method (and specifically if it is a constant cost), and the applicability of the proposed method to LLMs. 

Having read through the discussion I believe the authors did address both of these concerns adequately, and I do not see these concerns as being sufficient to reject what appears to otherwise be a robust method, evaluation and writeup.
Decision: Accept (poster)