SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Regularization and Augmentation. For augmentation, similarly to [6], we adopt known best-practices from the literature: smaller models such as DeiT-Tiny benefit from lower levels of data augmentation during fine-tuning as in [41], whereas larger models such as DeiT-Base behave best with more complex augmentation and regularization [43]. This is intuitive, since fine-tuning sparsified small models with high augmentation may exceed model capacity, rendering the optimization process unstable. We provide detailed parameter values and ablations in Appendix C.
Augmentation for the Empirical Fisher. The choice of augmentation is of great importance not only for the model training but for the accurate estimate of the Empirical Fisher as well. We observed that without proper augmentation the sparse solution obtained overfits to the training data even when finetuned with the same augmentation and regularization setup. We provide details in Appendix P.
Efficient Sparsity Sweeps. We propose a simple iterative pruning framework, which takes a set of target sparsity configurations and produces models which match these configurations in a single run. Specifically, we start from a standard gradual pruning setup, which prunes in a sequence of steps of increasing sparsity, followed by sparse fine-tuning. We then set the intermediate values in such a way that all intermediate target sparsity levels are achieved. For example, if one wishes to obtain checkpoints with sparsity levels  $40\%$ ,  $50\%$ ,  $75\%$ ,  $90\%$ , one can set the lowest sparsity level on the gradual pruning schedule to  $40\%$ , the highest sparsity level to  $90\%$ , and  $50\%$ ,  $75\%$  as intermediate points. Between any two such pruning steps, we apply the cyclic retraining schedule above, which ensures that all intermediate points are sufficiently optimized.
We emphasize that the accurate CAP pruner is key for efficient pruning: virtually all previous high-accuracy pruning methods in this setting, e.g. [25, 39, 6] redo the entire training run for each sparsity target. In our experiments, we also examine the impact of additional fine-tuning applied to each checkpoint, and show that it induces small-but-consistent improvements.
4 Experimental Setup and Results
Setup and Goals. We consider the ImageNet [37] image classification benchmark, and aim to examine how sparsity impacts accuracy for different model variants. We consider three scenarios: one-shot, single-step pruning of a pretrained model, where performance is clearly tied to the quality of the second-order approximation, one-shot + fine-tuning, in which we follow one-shot pruning
by a short period of fine-tuning, and, finally, iterative gradual pruning, where one applies pruning periodically, with some retraining interval, gradually increasing sparsity.
4.1 One-shot pruning, without and with fine-tuning
We start by examining the quality of existing one-shot pruners relative to CAP. We compare against carefully-tuned variants of Magnitude Pruning (Magn), First-Order Gradient times Weight (GrW) [38], and the SOTA second-order methods WoodFisher [39, 23] and M-FAC [14]. Our tuning process, optimal hyper-parameter choices, and ablations are detailed in Appendices J and K. WF-1 represents the diagonal approximation of the Hessian proposed by the original OBD [26], scaled via the WoodFisher implementation. For Magnitude and CAP we present results for both uniform layer-wise sparsity and global sparsity. For all other methods, we present results for global sparsity, which yields better results, as the methods can adjust sparsity globally. We only investigate global sparsity for the other methods in Figure 2.
Figure 2: One-shot pruning for DeiT-Tiny.
Figure 3: Validation loss surface.
The full comparison is presented in Figure 2 for DeiT-Tiny. Notice that all first and second-order methods outperform Magnitude, and that all 2nd-order methods are better than the 1st order saliency. WoodFisher with small block size is better than both the diagonal approximation and large-block M-FAC. This suggests that it is beneficial to take weight correlations into account, but attempting to incorporate dependencies between large groups of weights may lead to noisy estimates detrimental to performance. We also present a comparison on other models (ViT and ResNet) in Appendix E.
CAP outperforms all other methods, by a large margin. Remarkably, the gap is so large that CAP with uniform sparsity still outperforms WoodFisher with global sparsity, which can re-distribute sparsity across layers. The computational cost of WF is approximately the same as for CAP: CAP pruning step on DeiT-Small takes 23 minutes, compared to 20 minutes for WF. (The majority of the cost for both methods comes from the collection of gradients, not Hessian estimation.)
To investigate the performance difference between CAP and WF, we pruned several variants of ViT models to  $50\%$  and compared the mask overlap between WF and CAP via their IoU (intersection over union). We observe that the sparsification masks differ significantly: the IoU between the WF and CAP mask is  $82.7\%$ ,  $80.5\%$ ,  $73.3\%$  for DeiT-Tiny, DeiT-Small, and DeiT-Base, respectively, suggesting that the weight correlations taken into account by CAP lead to significantly different pruning decisions. The same trend is visible in the validation loss surface, projected on the plane in weight space, for the dense model and the  $50\%$  sparse models via WF and CAP. (We adopt the approach from [32] for loss surface visualization. Specifically, the unit vector in the horizontal direction is chosen to be  $w_{WF} - w^{*}$  and the vertical direction is defined by the component of  $w_{CAP} - w^{*}$  orthogonal to the horizontal axis. Above  $w^{*}$ ,  $w_{WF}$ ,  $w_{CAP}$  are the weights of the dense model and sparse solutions found by WF and CAP, respectively.) Figure 3 shows that CAP chooses a significantly less steep direction in the loss basin compared to WF.
Compressing highly-accurate models. Modern pretraining methods [7, 5, 34, 51] in conjunction with large vision backbones achieve extremely high accuracy on standard benchmarks such as ImageNet-1K. We leverage the scalability and accuracy of CAP to investigate, for the first time, sparsity in such highly-accurate models, in the absence of finetuning. Specifically, we start from the
ConvNext-Large checkpoint pretrained via CLIP on the LAION-2B dataset and finetuned sequentially on ImageNet-21k and ImageNet-1k [47].
Table 1: Accuracies of CLIP-pretrained ConvNext-L on ImageNet-1k, following one-shot pruning.
Model	Method	Sparsity (%)	Top1-Accuracy (%)
ConvNext-L	Dense	0	87.8
GM	80.2
WF	50	86.6
CAP	87.5
GM	38.7
WF	60	85.1
CAP	87.1
GM	0.5
WF	70	73.7
CAP	86.8
In Table 1, we compare our method with WoodFisher (WF) and Global Magnitude (GM) for one-shot pruning via several sparsity targets. Results show that 1) CAP can induce  $50\%$  sparsity with relatively low  $(0.3\%)$  accuracy loss, and  $60\%$  with less than  $1\%$  accuracy drop; 2) CAP significantly outperforms other methods. In addition, we find that these results can be improved with a limited amount of fine-tuning; please see Appendix E for full results.
4.2 Gradual Pruning Results
Finally, we execute gradual pruning with the sparsity schedule, augmentation choices, and cyclic linear learning-rate scheduler discussed above. The whole gradual pruning procedure lasts for 300 epochs, as in [43]. We aim to obtain accurate sparse checkpoints for  $50\%$ ,  $60\%$ ,  $75\%$ ,  $80\%$ , and  $90\%$  sparsity. For this, we prune to  $40\%$  in the initial step, and increment sparsity every 20 epochs, until reaching  $90\%$ , with fine-tuning in between. (See Appendix J for full results and ablations.) We select the accuracy of intermediate models which match the target sparsities; to examine the impact of fine-tuning, we trained each of the resulting sparse checkpoints for an additional 100 epochs, marked with  $(\uparrow)$ . We compare with global magnitude (GM) following the same schedule as CAP, as well as the state-of-the-art SViTE [6] paper, which trains the sparse model from scratch using a variant of RigL [10], but for a total of 600 epochs. The results are in Table 2.
Table 2: Results for gradual pruning of DeiT-Tiny and Small models on ImageNet-1k. CAP achieves  $50\%$  sparsity without accuracy loss, and  $80\%$  sparsity with less than  $1\%$  relative error.
Model	Method	Sparsity (%)	FLOP Reduction (%)	Top1 Accuracy (%)
DeiT-Tiny	Dense	0	0	72.2
GM	43.9	73.5
CAP	50	43.9	73.7
SViTE-Tiny	43.9	69.6
GM	60	52.6	73.1 (73.2 ↑)
CAP	52.6	73.3 (73.6 ↑)
GM	65.8	71.4 (71.9 ↑)
CAP	75	65.8	72.3 (72.6 ↑)
SViTE-Tiny	65.8	63.9
GM	80	69.7	70.5 (70.9 ↑)
CAP	70.2	71.7 (72.0 ↑)
GM	79.0	66.2 (66.6 ↑)
CAP	90	79.0	67.4 (68.0 ↑)
SViTE-Tiny	79.0	49.7
Model	Method	Sparsity (%)	FLOP Reduction (%)	Top1 Accuracy (%)
DeiT-Small	Dense	0	0	79.8
GM	46.7	79.3 (79.8 ↑)
CAP	50	46.9	79.4 (79.9 ↑)
SViTE-Small	46.3	79.7
GM	56.1	79.0 (79.5 ↑)
CAP	60	56.2	79.3 (79.8 ↑)
SViTE-Small	55.4	79.4
GM	70.1	78.0 (78.7 ↑))
CAP	75	70.2	78.5 (79.0 ↑)
SViTE-Small	70.3	77.0
GM	74.2	77.3 (77.9 ↑)
CAP	80	74.9	78.0 (78.6 ↑)
GM	84.0	74.1 (74.7 ↑)
CAP	90	84.1	75.2 (75.8 ↑)
SViTE-Small	84.1	70.1
For DeiT-Tiny and  $50\%$  sparsity, we achieve significant improvements upon SViTe, and even manage to improve test accuracy relative to the dense model. We believe this is due to the choice of augmentation during fine-tuning, and possibly due to regularizing effects of sparsity. At  $75 - 80\%$ , we recover the dense model accuracy. We observe a significant accuracy drop only at  $90\%$ . GM pruning also benefits from the choices made in our schedule, outperforming SViTE at  $50\%$  sparsity; yet, there are significant gaps in favor of CAP at higher sparsities, as expected.
On the 4x larger DeiT-Small model, SViTE performs remarkably well at  $50\%$  sparsity  $(79.7\%)$ , almost matching the dense model, but CAP outperforms it slightly after fine-tuning  $(79.9\%)$ . In terms
of total training budget, SViT uses 600 epochs to produce each model (and so, the  $50\%$  -sparse one as well), whereas we use a total of 40 epochs for gradual pruning to  $50\% +$  initial fine-tuning, and 100 additional epochs for sparse model fine-tuning. Even if we take into account the original 300 epochs for training the publicly-available dense DeiT checkpoint [43], our approach is significantly more efficient (440 vs. 600 epochs). The cost savings compound across sparse models, since we obtain all our pruned models from the same end-to-end gradual run. At  $75\%$  sparsity, CAP drops  $\sim 1\%$  of accuracy relative to dense post-finetuning, with a significant gap of  $1\%$  Top-1 relative to GM, and  $2\%$  Top-1 relative to SViT. The trend continues for higher sparsities, where we note a remarkable gap of  $5.7\%$  Top-1 vs SViT at  $90\%$  sparsity. We obtain similar numbers for DeiT-Base in Table 3; generally, we achieve  $\geq 99\%$  recovery at  $\geq 75\%$  sparsity, showing for the first time that ViT models can be pruned to such sparsities with marginal accuracy loss.
Model	Method	Sparsity (%)	FLOP Reduction (%)	Top1 Accuracy (%)
DeiT-Base	Dense	0	0	81.8
CAP	50	48.5	81.6
SViTE-Base	48.0	81.5
CAP	60	58.2	81.5
SViTE-Base	57.5	81.3
CAP	75	72.8	81.1 (81.2 ↑)
80	77.7	80.8 (81.1 ↑)
90	87.4	79.7 (80.1 ↑)
Table 3: Accuracy results for gradual pruning of DeiT-Base model on ImageNet-1k.
Figure 4: Accuracy vs. real-time throughput for dense  $(\star)$  and sparse ViT-models.
Sparse Speedups. The results in Figure 4 examined the speedups obtained by CAP from unstructured sparsity for  $\{50\%, 75\%, 90\%\}$ -sparse ViT-family (base, small, tiny) models, when executed on a sparsity-aware CPU inference engine [24]. Specifically, we executed the models from Table 2 using 4 cores of an Intel(R) Xeon(R) Gold 6238R CPU, at batch size 64. We find it interesting that sparse ViTs build an almost-contiguous Pareto frontier from  $82\%$  to  $68\%$  Top-1 accuracy (Y axis), with a  $25x$  span in throughput (from 11 ims/second to 260 ims/second, X axis). Notably, the DeiT-Tiny model obtains a speedup of  $2.4x$  without any accuracy loss, while Base and Small ViTs show  $1.5x$  speedups with minimal loss of accuracy. Thus, these results show that unstructured sparsity can be a very promising approach to speeding up ViTs. We note that competing methods (e.g. SVITE) would provide similar speedups at the same sparsity levels, but significantly lower accuracy, as shown in the figure, as well as in Table 2.
Additional Results and Details. Due to space constraints, full ablations and several additional experiments have been deferred to the Appendix. We present the full pseudocode for CAP in Appendix A, and a faster approximate version of CAP in Appendix B. We describe all parameters and perform additional ablations in Appendices C and J. We present results for one-shot pruning across several other models, as well as with short amounts of fine-tuning, in Appendix E. We present gradual pruning results for several other vision models (EfficientFormer, ResNet50D, and EfficientNetV2) in Appendix F and demonstrate that one can compress them to high sparsity as well, using our approach. In Appendix H we show that CAP comes with little additional computational cost compared to WoodFisher [23]. In Appendix I we show that CAP can also be applied in conjunction with other types of compression, in particular token pruning [35], quantization-aware training, and semi-structured (2:4) sparsity. In Appendix M, we show that CAP also outperforms AC/DC pruning [33], whereas Appendix N contains results for the DeTR detection model. In Appendix G we study the scaling behavior of sparsity solvers with respect to the ConvNext2 model family.
We presented a new correlation-aware pruner called CAP, which sets a new state-of-the-art sparsity-accuracy trade-off. We have shown for the first time that ViT variants can support significant weight pruning  $(\geq 75\%)$  at relatively minor accuracy loss  $(\leq 1\%)$ , inducing compression trade-offs that similar to those of CNNs, and that CLIP-pretrained, highly-accurate models can also support sparsity accurately. Thus, despite weaker inductive biases, next-generation vision models do not require over-parametrization, and can be competitive with CNNs in terms of accuracy-per-parameter. Our approach is complementary to other compression approaches, leading to significant speedups.
This project has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 805223 ScaleML). The authors would also like to acknowledge computational support from the ISTA IT department, in particular Stefano Elefante, Andrei Hornoiu, and Alois Schloegl, as well as Amazon EC2 for research credits. D.K. was supported by Russian Science Foundation, grant 21-11-00373.
[1] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
[2] Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251-276, 1998.
[3] Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6):629-654, 2008.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021.
[6] Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988, 2021.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020.
[8] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
[10] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pages 2943-2952. PMLR, 2020.
[11] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611, 2019.
[12] Elias Frantar and Dan Alistarh. Optimal Brain Compression: A framework for accurate post-training quantization and pruning. arXiv preprint arXiv:2208.11580, 2022.
[13] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot, 2023.
[14] Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-fac: Efficient matrix-free approximations of second-order information. Advances in Neural Information Processing Systems, 34, 2021.
[15] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv:1902.09574, 2019.
[16] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
[50] Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer, 2021.
[51] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models, 2022.
[52] Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-aware pruning and optimal neural implant. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3880-3891, 2022.
[53] Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
[54] Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning, 2021.
A CAP algorithm description
The section below illustrates in the CAP pruning algorithm step-by-step. Prunable model weights  $\mathbb{R}^d$  are partitioned into blocks of fixed size  $B$ . Below  $\rho_i^{(B)}$  denotes the saliency scores for weight  $i^{\mathrm{th}}$  inside a block it belongs to and  $\rho_{i}$  is the score across the whole model. The steps of the algorithm are listed below:
Algorithm 1 CAP pruning algorithm
1:  $\rho_{i}$  - saliency scores for weights  
2: Accumulate Fisher inverse blocks  $\mathbf{F}$   
3: for each block do  
4: err = 0  
5: for element in a block do  
6: Select the weight  $w_{i}$  with smallest score  $\rho_{i}^{(B)}$  (using the (2) for  $\rho_{i}$ )  
7: Prune  $w_{i}$   
8: Update remaining weights in the block via (2)  
9: err + =  $\rho_{i}^{(B)}$   
10:  $\rho_{i} \gets$  err  
11: Save current state of the block for later merging  
12: Update Fisher inverse block  
13: end for  
14: end for  
15: Sort the scores  $\rho_{i}$  in ascending order  
16: Mark the weights with smallest scores  $\rho_{i}$  as pruned  
17: for each block do  
18: Load the saved state of the block with the weights marked pruned and all remaining alive.  
19: end for
B FastCAP: A faster version of the algorithm
CAP can scale to models with  $< 1$ B parameters, covering most standard vision models. Its key computational and memory cost comes because of computing and maintaining the inverse Fisher approximation, similarly to WoodFisher/Optimal BERT Surgeon.
To address the scalability issue we present a much more scalable approximate variant of CAP, called FastCAP, which is inspired by the recent SparseGPT scalable pruner [13]. Specifically, FastCAP reduces compute and memory cost by  $\sim 100\mathrm{x}$  by leveraging a cheaper Fisher approximation and relaxing the optimal greedy order of pruning. FastCAP can be scaled to billion-parameter vision models, i.e. we can prune ViT-Giant in a few minutes, while achieving reasonable performance in one shot-setting for  $50 - 70\%$  sparsity.
The main two components of the CAP/FastCAP algorithm are:
- Empirical Fisher estimate. The original CAP has specific Fisher block for each output channel, whereas the FastCAP version averages blocks across the output channel dimension. This is a further approximation, but significantly reduces storage complexity.
- Weight elimination and update iteration. If the original CAP eliminates weights in greedy optimal order, FastCAP prunes weights in a fixed order, similar to SparseGPT [13]. This reduces computational complexity.
The full pseudocode for FastCAP is provided in Algorithm 2. The same procedure is applied for all weights. We use the following notation:
-  $B$  - Fisher block size
-  $b$ -block size in iterative pruning algorithm
-  $N$ -number of gradients
-  $N_B$  - number of blocks per parameter
The Fisher matrix is the same for all blocks, reducing space complexity to  $\mathcal{O}(B^2)$ . We adapted the trick of maintaining the Fisher inverse in Cholesky form from SparseGPT [13], to account for iterative elimination of the weights in a fixed order. This allows running the iterative process efficiently.
Algorithm 2 Fast CAP pruning algorithm
1: 1. Fisher accumulation  
2:  $F \gets$  matrix with zeros with shape  $(B, B)$   
3: for  $i = 1:N$  do do  
4: reshape  $\nabla_w\mathcal{L}_i$  to  $(N_B, B)$   
5:  $F = F + \frac{1}{N}\nabla_w\mathcal{L}_i^T\nabla_w\mathcal{L}_i$   
6: end for  
7: 2. Iterative pruning  
8:  $M \gets 1_{N_B \times B}$  // binary pruning mask  
9:  $E \gets 0_{N_B \times b}$  // block pruning errors  
10:  $F \gets \text{Cholesky}(F^{-1})$  // efficient form of the Fisher inverse  
11: for  $i = 0, b, 2b, \ldots$  do  
12:  $M_{:,j:(j+b)} \gets \text{mask of (1 - p)% weights } w_k \text{ in } W_{:,j:(j+b)} \text{ with largest } w_k^2 /[F^{-1}]_{kk}^2$   
13: for  $j = i, \ldots, i + b - 1$  do  
14:  $E_{:,j-i} \gets W_{:,j} / [F^{-1}]_{jj}$  // pruning error  
15:  $E_{:,j-i} \gets (1 - M_{:,j})E_{:,j-i}$  // freeze weights  
16:  $W_{:,j:(i+b)} \gets W_{:,j:(i+b)} - E_{:,j-i}F_{j,j:(i+B)^{-1}}$  // update remaining weights to correct output  
17: end for  
18: end for  
19:  $W \gets W \cdot M$  // set pruned weights to 0
To validate performance of FastCAP we applied it for pruning of large ViT models in a one-shot setting, specifically eva_giant_batch14_224.clip_ft_in1k from the TIMM library [48]. This model has  $>1\mathrm{B}$  parameters. We compared the top-1 accuracy on ImageNet of FastCAP vs Magnitude pruner. One can see that FastCAP significantly outperforms the baseline. At the same time, FastCAP has a reasonable runtime and memory footprint at this scale. The pruning step for the whole model takes 400 seconds on a single A100 GPU.
Table 4: Performance of large ViT after one-shot pruning.
Model	Sparsity	Method	Accuracy (%)
EVA ViT-G	0	-	88.7
50	Magnitude	87.9
FastCAP	88.1
60	Magnitude	85.5
FastCAP	86.3
70	Magnitude	64.3
FastCAP	76.1
C Training details
Augmentation/regularization recipe
For the gradual pruning experiments (with 300 epochs) we have used cyclic learning schedule, with high learning rate directly after the pruning step with gradual decrease up to the next pruning step. For both DeiT-Tiny and DeiT-Small model during the additional fine-tuning for 100 epochs we've applied cosine annealing schedule with  $\eta_{\mathrm{max}} = 5\cdot 10^{-5}$ ,  $\eta_{\mathrm{min}} = 1\cdot 10^{-5}$  and all other parameters the same as in the Table 6.
D Post-Pruning Recovery