The Need for Speed: Pruning Transformers with One Recipe
Authors: Samir Khaki
Konstantinos N Plataniotis
Venue: ICLR 2024 poster (2024)
Keywords: transformers
pruning
re-training
latency
throughput
token reduction
one-shot
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We introduce the $\textbf{O}$ne-shot $\textbf{P}$runing $\textbf{T}$echnique for $\textbf{I}$nterchangeable $\textbf{N}$etworks ($\textbf{OPTIN}$) framework as a tool to increase the efficiency of pre-trained transformer architectures $\textit{without requiring re-training}$. Recent works have explored improving transformer efficiency, however often incur computationally expensive re-training procedures or depend on architecture-specific characteristics, thus impeding practical wide-scale adoption. 
To address these shortcomings, the OPTIN framework leverages intermediate feature distillation, capturing the long-range dependencies of model parameters (coined $\textit{trajectory}$), to produce state-of-the-art results on natural language, image classification, transfer learning, and semantic segmentation tasks $\textit{without re-training}$. Given a FLOP constraint, the OPTIN framework will compress the network while maintaining competitive accuracy performance and improved throughput. Particularly, we show a $\leq 2$% accuracy degradation from NLP baselines and a $0.5$% improvement from state-of-the-art methods on image classification at competitive FLOPs reductions. We further demonstrate the generalization of tasks and architecture with comparative performance using Mask2Former for semantic segmentation and cnn-style networks. OPTIN presents one of the first one-shot efficient frameworks for compressing transformer architectures that generalizes well across different class domains, in particular: natural language and image-related tasks, without $\textit{re-training}$.

THE NEED FOR SPEED PRUNING TRANSFORMERS WITH ONE RECIPE
Samir Khaki
Konstantinos N. Plataniotis
Department of Electrical and Computer Engineering University of Toronto
Toronto, Canada
samir. khaki @mail. 11. utoronto.comm ca
ABSTRACT
We introduce the One-shot Pruning Technique for Interchangeable Networks (OPTIN) framework as tool to increase the efficiency of pre-trained transformer architectures, across many domains, without requiring re-training, Recent works have explored improving transformer efficiency. however often incur computation- ally expensive re-training procedures or depend on architecture-specine isties, thus impeding practical wide-scale adoption across multiple modalities. To address these shorteomings, the OPTIN framework leverages intermediate feature distillation, capturing the long-range dependencies of model parameters (coined trajectory). to produce state-of-the-ar results on natural language, image classifica- tion, transfer learning, and semantic segmentation tasks. Our motivation stems from the need for a generalizable model compression framework that scales well across different transformer architectures and applications. Given FLOP constraint, the OPTIN framework will compress the network while maintaining competitive accuracy performance and improved throughput. Particularly, we show S 2% accuracy degradation from NLP baselines and a 0.5% improvement from state- of-the-art methods on image classification at competitive FLOPs reductions. We further demonstrate the generalization of tasks and architecture with comparative performance on Mask2Former for semantic segmentation and enn-style networks. OPTIN presents of the first one-shot efficient frameworks for compressing transformer architectures that generalizes well across multiple class domains, particular: natural language and related tasks, without re-training. Code is available at: https: /g thub. com/ Sahak/ 1t/kiii//ooo //optinn---i/
character-
INTRODUCTION
The inception of transformer architectures (Vaswani et al., 2017) marked the beginning of a new era in deep learning, since affecting various domains including natural language processing (Kenton & Toutanova, 2019), and vision-related tasks (Dosovitskiy al., 2021). The transformers' straightfor- ward design has enabled extensive applications to variety of challenging problems, but it also brings a major drawback: high computational costs (Yu & Xiang. 2023). The computational resources often quite significant and pose real
required for training and inferencing with transformer impediment to wide-scale adoption, especially in resource-constraing devices (Wang et al., 2020a). Recent works have proposed methods including quantization (Xiao et al., 2023), pruning (Ma et al., 2023), and knowledge distillation (Hao et al., 2022) to address this bottleneck, similarly explored in convolutional neural networks (CNN) compression (Li et al., 2017).
environments, such as edge
Despite much success in compressing CNNs, transformer architectures contain significant differences their structure, often causing impediments for methods that work well the former domain (Kwon et al., 2022; Yu & Xiang, 2023; Yang et al., 2023). Due to the massive size of Transformer models, some works have introduced various methods of compression, which can be loosely divided into one-shot and iterative (Zhang et al., 2022). One-shot methods generally consist of pruning phase followed by re-training to recover the lost generalization performance, meanwhile, iterative processes
Projecc Page: http: http:///wwwwsa.eeeeeee www.rkaaki.icommmmmammmmmm com http://ppiii-teeettt ott.aaaw..cccooooottaaatttttttee

---

account for the training dynamics in model compression (Zhang al., 2022). Unfortunately, in the past, both methods have often been limited to a particular architecture/task or required significant resources in the pruning and re-training processes. For user models that have already endured the expensive cost of training, there exists limited options for fast model compression (Kwon et al., 2022) that can be easily realized on standard hardware for different types of transformer architectures. The lack of a general approach to transformer pruning across multiple tasks and modalities provides sufficient motivation for the introduction of unified framework; hence we introduce one of the first one-shot model compression techniques that generalize well over multiple tasks and architectures without incurring the cost of re-training.
In this work, we introduce the One-shot Pruning Technique for Interchangeable Networks (OPTIN) framework to efficiently compress modern transformers. The novelty is in generalizability across domains and tasks, and its ability to produce competitive models without requiring re- re-training, thus enabling future application to larger models across many tasks. We apply OPTIN to natural language, and vision-related tasks, showing competitive performance with SoTA in these cases.
Our primary contribution rests on the ability of our OPTIN framework to produce transformers with competitive performance at reduced computational loads (FLOPs) across various task domains and architectures, that can be realized on standard hardware. In particular, we demonstrate superior performance variety of tasks in Language 4.1), Vision (Sec 4.2), and Application tasks (Sec 4.3), while maintaining competitive compression rates, without incurring the cost of re-training. Finally, execute several extensive experiments from waneewwwrk-seeiiff... settings applications on transfer- learning and CNN networks to demonstrate OPTIN's robustness and generalizability over the task and architecture (Sec 3- 4.3).
2 RELATED WORKS
Due to the diversity of architectures and tasks discussed in our work, the following review of state-of- the-art methods provides an overview of efficient transformer design followed by recent developments in both language and vision domains.
Domain Specific Design of Efficient Transformers Transformers have enabled significant progress in the field of NLP (Vaswani al., 2017: Kenton & Toutanova, 2019) and Computer Vision (Dosovit- skiy et al., 2021; Liu et al., 2021a). Efficiency improvements in transformers have stemmed from variety of approaches including exploring hybrid architectures (Liu et al., 2021a), quantization techniques (Kim et al., 2021), knowledge distillation (Hao et al., 2022), and model pruning (Pan et al., 2021). Recently, TorchPruning (Fang et al., 2023) explored the application of multi-domain pruning by creating an inter-architecture dependency map, while UPop (Shi et al., 2023) introduced a unified pruning method for combined vision- Laigual models. However, these methods have limitations. including architecture-specifi dependencies and expensive re-training policies generally impeding wider-scale industry use (Fang et al., 2023). In contrast, our approach leverages intermediate feature distillation to compress pre-trained transformers in one shot across both language and vision-related tasks. Notably, our method operates effectively without re-training and scales well over a variety of complex architectures and task domains.
Compressing Language Transformers Several struetured pruning methods have been introduced to compress models in the language domain. Attention Head pruning (Michel et al., 2019) explored the dynamics of attention heads across a transformer architecture to determine their individual impact on performance. Block-wise pruning (Lagunas et al., 2021) was motivated by removing block structures from weights under the movement pruning (Sanh et al., 2020) paradigm. DynaBERT (Hou et al., 2020) used distillation transfer knowledge from a width-adaptivv network onto a repth-adaptive smaller network. CoFi (Xia et al., 2022) explored the joint pruning between coarse and fine- ane-grained modules in the transformer architecture. A recent work, namely TaanTTrrnnn-FFammwooot Framework (PTF) (Kwon et al., 2022), was introduced to prune BERT in one-shot for NLP tasks, however, it leverages Nomain-related tricks to boost performance with a particular architecture and application. However, these methods have limitations, including dependence on architecture (Kwon et al., 2022; Lagunas et al., 2021; Hou et al., 2020) and expensive re- re-training procedures (Lagunas et al., 2021; Sanh et al., 2020; Hou et al., 2020; Xia et al., 2022). Focusing on the challenge of developing efficient transformers, we overcome these shortcomings by introducing a one-shot framework that produces competitive results at significant FLOPs reductions across several application domains.
2

---

Compressing Vision Transformers There have been several approaches to compressing vision transformers by focussing on different compute-intensive modules. V (Chen et al., 2021) explored structured sparsity by modifying first-order importance approximations enabling the dynamic sizing of attention heads in the VIT. SAVIT (Chuanyang et al., 2022) developed collaborative pruning scheme that analyzes component interaction to learn individual pruning ratios. Another stream of research introduced token reduction methods to accelerate both the training and inferencing throughput by gradually removing tokens from propagating forward in a Transformer (Kong et al., 2022; Fayyaz et al., 2022). EVIT (Liang et al., 2022) builds Top-K approach by creating a fused token at each reduction stage to minimize the information lost from pruning. Dynamic ViT (Rao al., 2021) introduced lightweight prediction module to derive the importance scores of each patch per input. ToMe (Bolya et al., 2023) was introduced as one of the first one-shot methods in token reduction and leveraged bipartite matching to merge fixed number of tokens at each transformer block regardless of input patches. TPS (Wei et al., 2023) furthered token reduction and merging, by identifying a pruned subset and squeezing the informative regions into reserved subset of kept tokens. However, these methods still have limitations that prevent their widescale use including architecture specific design (Song et al., 2022: Chuanyang et al., 2022: Bolya et al., 2023) and expensive re- training policies (Chen et al., 2021; Chuanyang et al., 2022: Liang et al., 2022; Rao et al., 2021; ;; Wei et al., 2023). In contrast, the OPTIN Framework leverages a one-shot approach to compress vision transformers across classification and semantic segmentation achieving competitive performance amongst state-of-the-art The granularity and number of prunable components in the domain of vision transformers widely differ across state state-of-the-ar methods (Song et al., 2022). Similarly, the OPTIN Framework increases the base prunable components by allowing for the incorporation of token reduction methods through generating an optimal reduction policy as discussed in Sec. 4.2.
3 MEASURING TRAJECTORY
We aim to compress pre-trained transformer models by removing prunable parameters with minimal importance scores as determined by our salience metric without re-training. By analyzing the effects of parameter removal on deeper layers in the network, our trajectory metric is able better select important parameters by leveraging long-term inter-layer dependencies in the model.
Problem Statement. Given model f (with N layers) expressed by its collection of weights [0. 81 --- 0N] 0n] R Nxd, a pruned subset has weight collection [6% 86. ... -.y E Nxx, such that 0' = m @ f where m E {0. 1 is a binary mask and @ is the element wise product operator. We detine this pruned subset to be optimal if satisfies the cost constraint and results in the minimum decrease in validation error from the base model expressed with Lerr. Formally, we express this as:
argmin [6.01..-000) Lerr subject to C(10) 02, (xl.)..)'x)==>>>))) S C.
10.. .1.00.1-000000 (l..--ON)), ((, 04 -.. -.,)))))
where, the optimal selection of weights [0% 81.-. 8......%% 8yy] meets the cost requirement C while retaining the minimum drop in validation performance on the dataset X.
Approach In general, for each transformer block we define the prunable weights as the collection of attention heads and fully connected neurons, individually denoted by 0t,, where the parameter is located in layer at an arbitrary index j. The exact prunable components for each task are described in Appendix A.8. We progressively mask each prunable parameter, and compute the importance score by executing a forward pass that originates from layer i and propagates forward to the logit prediction. In particular we express the masking of weight j in layer as MASK, @ 0, where M MAKKi is the instance of m with a single zero at location j, as used in Algorithm 1. This masked forward pass yields subsequent layer-wise activations and output logits, which are both used in computing the trajectory of parameter, 0sgj We denote the cumulative importance of parameter, 0.g- as I.,. Referring to the optimization problem in Eq. 1, we use our importance metric as a proxy for determining which parameters will least affect the validation error, Lerr, on the testing dataset, X. Upon computing all importance scores Ir,5. we employ an expedited mask- policy, from (Kwon et al., 2022), which computes the optimal configuration in a faster polynomial-time by sequentially adding parameters in descending importance. Further details on the search method are discussed in Appendix A.I. We introduce the OPTIN Framework algorithm in Algorithm and Diagram in Fig. 1.
3

---

Algorithm OPTIN Framework for Model Compression
1: Inputs: FLOPs Constraint (C), Importance Scores (I
[], model, batch
2: (IFo.... FN.. ; +- model(balch)
> Pre-Compute Forward Pass
:.00,...))) 0n) do 4: for j E (d) do
3: for 0, in 00,,01,...0
layer:
D. weight:
5: model[i]. weight
MASK, 6: (1F. ... Fl logits') ) t model (batch)
Apply Mask to Weight (i.j)
Compute Masked Forward Pass
7: Iij + E 25=it1 (MD ACRD
b Apply Eq. and 3
8: end for
9: end for
10:
Reduced Model + SEARCH(I,C) C)
Fixed Pre-Computed Feature Embeddings ([Fo, ... FN))
F
F..
Fx
Batch
MASK,
C LMD
CKD
Data
Classi fire
Layer
Lager,
Lager
Lagery
Logits
Legend:
FP
Embeddings
I(i LMD A. LED
Layer-wise MD
LRD
Logit-based KD
Figure 1: Illustrates the computation of the OPTIN Frameworks trajectory metric weight 0tg. By applying mask to weight O,g in Layer; and executing a forward pass, the OPTIN framework can measure the effect on future layer embeddings (trajectory), as an indicator of weight importance. LMD is the manifold distillation loss computed between layer embeddings at each transformer block, while LKD D is the KL--Divereecce computed between the original logits and those due to the masked weight. The combination losses are further detailed in the Weight Importance heading under Sec. 3
Parameter Importance Prior to assigning an importance score to each parameter, we define what it means to be "important". While many prior works have coined the importance of parameters by analyzing their intrinsic structure and error dynamics (Kurtic et al., 2022), these are not necessarily the most intuitive. For instance, magnitude-bant based metrics (Li et al., 2017) capture the intrinsic dominant property of individual weight structures, however, fail to capture their interactions with the data, meanwhile, activation methods (Lin et al., 2023) can capture in in-place reconstruction errors, however, they may obscure the global impact on deeper layers in the model. Motivated by capturing the long-term effects of weights, frame the problem as identifying which weights more important based on how much they affect subsequent layer embeddings, hence we coin the measure trajectory.
Efffet on Trajectory To compute the trajectory of weight, 0z.; follow 2-step procedure. Firstly, we measure layer-wise activation errors prior to the LayerNorm operator at each block subsequent to the layer interest. We conducted an ablative study in la showing the effect of using pre-layer norm embeddings. We first define the feature output of layer i as F, R #xTxX where B is the batch size, 7 is token length and the embedding dimension. of the masked network using the prime' symbol. Inspired by distillations works (Sajedi et al., 2023a; Peng et al., 2019), we compute the layer-wise error by adopting fine- mee-grained manifold distillation (Hao al., 2022). A reshaping operator v)(-) R BT>D leverages patch and batch level information, defining the relational map and associated metric as:
similarly express the feature output
M(F)) = v (F.))
LMD F.) [LM(F)) -M(R)
(2)
However, unlike previous works, we do not use this loss guide training distillation, rather, express it as an in-place metric to help understand parameter importance throughout the network.
If the masked weight is at position j in layer i, the computed error is accumulated over both the dimension D and at each layer the range li + 1, N]. The choice error aggregation was explored in Tab. 1 and clearly demonstrated the benefit of the operator. Additionally, explored the
4

---

Dataset
Emb.
Acc.
Datasct
Temp.
Acc.
Dataset
Aggregate
Acc.
MNLI
L-Norm
81.92
MNLI
82.01
MNLI
sum
81.90
MNLI
FFN
81,90
MNLI
2
82.
MNLI
mean
80.75
MNLI
IM-Dense
81.83
MNLI
4
81.90
MNLI
82.14 8
ImageNet L-Norm 71.27
ImageNet
1
70.54
ImageNet
sumn 71.25
ImageNet FFN 71.25
ImageNet
2
70.77
ImageNet
mean
71.15
ImageNet
IM-Dense
70.90
ImageNet
4
71.25
ImageNet
8
71.00
(a) Embedding Choice. The dense output layer best informs weight performance when com- pared laser-
(b) Temperature Choice. The best captures the effect on the logits by removing weights.
(c) Layer Error Aggregation. The cumulative error over the layer's manifold distribution best captures weight importance.
choice
Dataset
CMD
LKD
A
Acc.
Dataset
Type?
Acc.
MNLI
81,71
MNLI
[
78.89
MNLI
80.91
MNLI
[
80.07
MNLI
81.74
MNLI
N]
81.65
Language
MNLI
81.86
MNLI
[ 1, N
81.90
MNLI
0.1
81.90
0.01
82.12
ImageNet
70.34
ImageNet
68.91
ImageNet
68.85
ImageNet
69.55
ImageNet
10
70.82
ImageNet 11, N
70.04
Vision
ImageNet
70.85
ImageNet
N,
71.25
ImageNet
0.1
70.99
ImageNet
0.01
71.25
(d) Component Analysis Combining both LMD and CK cap- tures the best information (see Eq. 3). Based the value of X, can see LMD should have stronger weight in parameter selection. Further details on the hyperparameter choice are in Appendix A.2.
(e) Layer Trajectory Depth. Ac- cumulating LMDD over deeper lay- ers performs best. indicates which layer(s) to measure L relative to current layer i and final layer N.
Table 1: Ablative Experiments on Trajectory using BERT BASE (Kenton & Toutanova, 2019) on the GLUE benchmark MNLI dataset (Wang et al., 2019), and DeiT-Ti (Touvron et al., 2021) on the ImageNet-1K dataset (Deng et al., 2009) to explore the effect parameters on model performance. We measure one-shot post-pruning accuracies vision datasets. In particular (a) explores locations to extract features for distillation loss: L-Norm (After Layer Normalization), FFN (After Dense output layer), IM-Dense(Afice Dense Embedding Layer). (b) examines the effect of temperature in the KL- KL-Divergence Formulation, (c) explores the effect of summing or averaging over the layer distillation error, (d) explores the effect of metrics LMD and LKD as well their balancing paramter X, (e) explores which layer to accumulate the distillation error in relation current layer Compression rates remain consistent with Tab. and Tab The baseline performance of BERT BASE MNLI is 84.53 %, meanwhile DeiT-Ti ImageNet- 1K 72.20 % %. Our default settings are marked in green
various configurations both the language and
effect of modifying which layers were relevant to the trajectory see Tab. 1c overall it was evident that using subsequent layers yielded the best result, correctly aligning with the original motivation.
Effect on Logits Next we compute the effects on the logit prediction as shown in Fig. with LKD- D- Several works have shown the effects of using logit predictions guide the training process with distillation (Hao et al., 2022; Zhao et al., 2022) or correlation (Sajedi et al., 2024; 2023b). However, in-place metric quantify
in this work, we use the LKD loss (defined (Hinton al., 2015)) the importance of a particular weight. We ablate the temperature value in Tab. 1b. Thus, if masking particular weight produces larger CKD. we would hypothesize that a more important weight.
We can formalize the importance of a particular weight j at layer with iterator z as:
N
I5j =
LMD(F ACRD
(3)
5

---

where I, is defined for a particular weight in a particular layer, À controls the contribution effect of KD, and LMD compares the pruned and original resulting embeddings in deeper layers. We ablate the effect of different À values well as the contribution of each loss individually in Tab. 1d. Further, we show that applying greater importance on LMD results in better parameter selection. Further details the contribution hyperparameter are expressed in Appendix A.2. We also provide the algorithm for applying OPTIN on generic model instance in Algorithm 1.
4 EXPERIMENTAL DESIGN
In this section, we demonstrate the effectiveness of the OPTIN Framework, at improving model performance and throughput given strict FLOP reduction ratios. We introduce implementation and evaluation details to ensure reproducibility and benchmark our method with state of art in natural language and image classification to illustrate the potential of our one-shot framework. We further investigate the applications in transfer learning, alternate architectures, and downstream tasks to show the generalizability of our method across tasks and architectures.
Experimental Setup We implement our method using transformers from the HuggingFace Library (Wolf al., 2020) and infrastructure from PyTorch (Paszke al., 2019). The majority of our experiments explore using the OPTIN Framework to improve off-the-shelf models without re- training. The exceptions include select experiments in the Applications Section, see Sec. 4.3. The OPTIN Framework computes parameter importance on the basis of training data in a gradient-free forward pass. The amount (batch) of data used compute the scores is ablated in Appendix A.6. Finally, details on the prunable components under each setting are described in Appendix A.8.
The OPTIN Framework is tested against a variety of network architectures and datasets to ensure generalizability over both the task and model domains. For Natural Language Processing, OPTIN is evaluated on the GLUE Benchmark (Wang et al., 2019) using the BERT BASE (Kenton & Toutanova, 2019) architecture. For Image Classífication, both ImageNetl-K (Deng et al., 2009) and CIFARTO O (Krizhevsky al., 2009) were used with the DeiT-Ti/S/B (Touvron et al., 2021), ViT-B (Dosovitskiy al., 2021), and VGGNet (Simonyan & Zisserman, 2014) architecture to demonstrate the OPTIN Framework's ' robustness on model type/size, image datasets, and transfer learning. For Semantic Segmentation, the Cityscapes Dataset (Cordts et al., 2016) was used with the Mask2Former (Cheng et al., 2022) with a Swin-Ti Ti backbone (Liu et al., 2021a) to show how the OPTIN Framework could be used to maintain competitive performance and throughput at constrained FLOPs. Further details on dataset selection are in Appendix A.3
Datasets Networks
Evaluation Metrics With the goal of model compression, we evaluate models based their accuracy (or mloU for segmentation) given a FLOP reduction. Details regarding the metric choice for the corresponding task can be found in Appendix A.4. In select cases, we include either a measurement of throughput speed (images/data per second) or latency expressed as ratio of the improved inferencing speed to that of the baseline. All time measurements are captured over 300 iterations on an Nvidia RTX 2080 using 100-iteration warmup.
4.1 LANGUAGE EXPERIMENTS
Performance on NLP Benchmarks In Tab. 7 we investigate the OPTIN Framework for compressing language models on the GLUE dataset using BERTH ASF. Measuring performance and throughput speeds, we show relatively low average decline in baseline accuracy (< 2%) at a 40% FLOPS compression rate. Similarly, we benchmark our performance with leading one-shot SoTA method: Post- Tost-iraining-P Truning-Framewee superior performance. In particular, we compare with the mask search results from PTF, as subsequent phases in their method could be stacked other post-training pruning methods (refer to Appendix A.7 for exetended comparisons). We demonstrate robustness over various compression ratios in Fig. 2 where we benchmark OPTIN against pipelines that incorporate re-training, including CoFi (Xia et al., 2022), DynaBert(Hou et al., 2020), SLIP (Lin et al., 2020b), EBERT(LiU et al., 2021b), BMP (Lagunas al., 2021) and FLOP (Wang et al., 2020b). Despite the added re- re-training phase in other methods, the OPTIN Framework is able to retain competitive test performance over a variety of compression ratios thus establishing compelling argument for retraining- free pipelines.
(KTF))Kwon et al., 2022) at the same compression rate and show
4.2 VISION EXPERIMENTS
Extending to Image Classification Transitioning the OPTIN Framework from the language domain to the vision domain, we were required to increase the number of prunable components. Comparable

---

Method
MNLI
QQP
QNLI
SST
STS-B
MRPC
BERT
84,53
91.00
91.41
93.57
88,90
86,27
PTF
81.21
89.99
88.38
92.13
87.10
83.14
OPTIN'
81.90
90.06
88.49
92.24
87.25
85.13
Table 2: Natural Language Benchmarks. Comparing OPTIN performance on the GLUE (Wang et al., 2019) benchmark (refer to A.7 for additional results). The relative FLOP constraint is set to 60% for fair comparison.
omn
Dyadee
Col
EXPT
Accuracy Latienyy MNL varousa FLDP Compremsinn Flates *Dal Accusscy Cuave
CML
597/2
VMALI Laterce Cuave
0.00
3:35
-1X
Coaelits
1.20
-.40
3.50
-00
0.00
1.55
4.00
Gasiira
Auatin TCCP Compress
Niaore TLOP Comprastie
DNEAS LLT Comprisiic
Figure 2: Natural Language FLOPs vs Accuracy. We directly benehmark the OPTIN Framework against leading sfate-of-the-art methods in natural language model compression. Due to the numerous different baselines reported in each work, plot the relative performance drops for each method. On the right, we compare the performance gap with latency showing that with an average drop of < 1.75% we can achieve a 1.25x speedup in throughput purely from static model size reduction.
works have used a larger number of components including the pruning of Q-K-V layers in each attention head, tokens& & patches, and final embeddings in each transformer block (Zhu et al., 2021; Wei et al., 2023; Pan et al., 2021). Thus the state of the art in the field of transformer pruning widely differs in the granularity and consistency of the pruned components. With priority on reducing real- world inference time, we extend OPTIN to include a variant of token reduction; a similar adaptation was made in CP-ViT (Song et al., 2022). In particular, we derive a modified trajectory formulation to rank tokens between each transformer block as described in Appendix A.9. By incorporating ranking, the OPTIN framework can deduce the optimal
the trajectory metric for layerwise- number of tokens preserve between each transformer block, and can thus create an informative token reduction schedule that can be leveraged by any token reduction method, In particular, we were inspired by a recent work, ToMe (Bolya al., 2023) which features an efficient token merging technique based on bipartite matching that removes tokens between transformer blocks either at a constant or linearly decreasing schedule. We incorporate ToMe as a method of merging tokens based on the optimal number of reduced tokens per layer determined by the OPTIN framework search. Since our framework produces the reduction schedule, we can leverage the benefit of batching as the number of tokens per image will be constant, and the methods of merging or reducing can be selected by any user we ablate the bipartite matching scheme with a random pruning scheme in Appendix A.6 and show similar improvements when using the OPTIN Framework.
Image Classification Results Tab. we benchmark our proposed re-training free method on the ImageNet- KK dataset with the baseline, and SOTA results to show competitive performance at given FLOPs reductions. We offer two configurations: 3 (base) denotes the base OPTIN Framework without the additional prunable components (directly shifted from the language domain), 7 (expanded) denotes the incorporation of token reduction into our search space. Compared with methods that perform re-training, the OPTIN Framework produces competitive performance, particularly with 0. 0.5%improvement at a 5% lower FLOPs with respect to SAViT(Chuanyang et al., 2022). Comparing with methods that have removed re-training, we note that VTP (Zhu et al., 2021) still includes additional aaarsityregularizaatoo training. POWER (Goyal et al., 2020) still includes the auxiliary network training with soft-extract, and HVT (Pan et al., 2021) still reduces FLOPs via re-training the architecture with a pooling method. However our method is considered a fundamentally one-shot design and despite lacking these additional pruning artifacts & components, still able to outperform outperforming CP-ViT(Song et al., 2022) by
the current SoTA, with the best result on 0.4% at a ~ 10% higher FLOPs reduction. To further benchmark our performance in perspective of a wider FLOPs spectrum and more model compression methods, we introduce Fig 3 which includes: X-Pruner (Yu& & Xiang, 2023), WDPruning (Yu et al., 2022a), 22VITE/SSP(Cher et al., 2021), SCOP (Tang et al., 2020). HVT (Pan et al., 2021), SAViT(Chuanyang et al., 2022), VTP (Zhu et al., 2021), PoWER(Goyal et al., 2020), CP- PP-ooion et al., 2022) and UVC (Yu et al., 2022b). Despite our lack of re-training, the OPTIN framework produces competitive results over various flop ratios.
7

---

DeiT Tiny
DeiT Small
Method
OFTN GCOP
HST
SOUTE SAAT
PoweR CeNT
FLOPs(G)
Acc(%)
FLOPs(G)
Acc(%)
Baseline
1.3
72.2
4.6
79.8
ImageNet Prurting PLOPS
Re- Traineined
0.0
SSP
0.99 123.7%
68.59
3.15 3.15441.6x 6%
77.74
ViTE
0.99
70.12
3.15 11.51.8%%
79.22
SAVITI
0.98 424.4%44
70.72
Not Re-Trained
VTP
1.00
69.37
3.65 12.
77.35
POWER!
120.33
69.56
3.61 121-5%
77.02
HVT
1.01 121.2%
68.43
3.66 20.596
76.72
CP-VIT
1,00 123.0%
71.06
3.64 .2..0%
78.84
OPTIN 8
1.1 1.1416.46%
67.51
4.11 4.11411:235
77.01
OPTIN,
0.91 129.7%
71.25
3.15
79.24
0.50
0.55
0.60
0.55
0.70
0.75
Compressis
Table 3: Pruning ImageNet-IK. Benchmarking the per- formance of OPTIN using DeiT-Tiny/Small. methods reproduced in (Song al., 2022) without re-training. il DeīT-S result from (Chuanyang et al., 2022) is ex- cluded it performs superior to the available baseline. OPTIN framework runs without re- re-training producing both the 3 and 5 configurations.
Figure 3: DeïT-Ti FLOPs vs Accuracy Benchmarking OPTIN, over a range of FLOP reductions ImageNet-IK. The OPTIN Framework shows strong robust- over various FLOP constraints with- out re-training.
Method
FLOPs(G)
1AAcc(%)
ImageNet-IK
Transfer-
VIT-B
17.47
Model
Method
FLOPs(G)
Acc(%)
FLOPs(G)
Acc(%)
ToMe
11.50
+. L.88
Baseline
4.6
79.8
4.6
97.13
OPTIN
11.45
I 0.71
DeiT-S
OPTIN,
12377%
79.01
150.0%
96.60
DeiT-B
17.6
Baseline
17.47
75.40
17.47
98.01
Dyn-Vit Vyn-VIT
11.81
1.17
VIT-B
OPTIN,
13.33
72.98
8.77450.05
8.77
150,0%
97.82
Top-K*
11.81
0.9.4
EViT"
11.81
0.886
Table 4: Transfer Learning on CIFAR Dataset. Bench- marking the performance of OPTIN on the CIFAR-10 Datasets. Models were pre-trained on ImageNet-IK, pruned through the OPTIN Framework configuration, and trans- ferred learned at more aggressive pruning rate onto the CIFAR-10 10 (C-10) Dataset.
ToMe
11.81
0.80
TPS
11.51
4.0.1
OPTIN
11.75
$ 0.52
Table 5: Token Reduction Bench- marking OPTIN,(oo) configuration. t&tt detailed in the main text.
For completeness, chose introduce third configuration T(oo) which only applies OPTIN to creating token reduction schedule, while leveraging ToMe for merging. Under this constraint, we evaluate method with state-of-the-art token reduction methods: including Dynamic (Dyn-ViT) (Rao et al., 2021), Top-K (Haurum et al., 2023), EVIT (Liang et al., 2022) and TPS (Wei et al., 2023) in Tab 5 and show superior performance under our framework without re-training. i methods follow setup & produced results in (Haurum et al., 2023), we convert token percentage to FLOPs reduction to benchmark our method. it estimated from (Wei et al., 2023). We further complement this with more detailed comparison against the schedules using constant and linearly decreasing reduction schedules in ToMe over a wide variety FLOP constraints in Appendix A.6. Ultimately this provides a compelling case for OPTIN's ability to effectively determine average token importance transformer architectures.
Transfer-Leernin for Image Classification To demonstrate the transferability of our compressed models, we obtain the pruned networks from ImageNet- IK and apply transfer learning to the CIFAR- 10 dataset. We choose to include DeiT-S and ViT-B for model size diversity. Benchmarking against baseline models, in Tab we show significant recovery performance when transferring learning onto CIFAR-10 10 extensive FLOPs reduction ratios. Although show re-training is not necessary when it comes to pruning on a specific dataset & task, we evidently show that the method works well under the transfer learning paradigm for different downstream purposes.
4.3 APPLICATIONS
We further explore downstream tasks and architectures that can similarly benefit from the OPTIN Framework. Particularly, high-resolution (HR) semantic segmentation is compute-intensing task,
8

---

and explore how OPTIN maintains competitive performance and increases throughput speeds in Tab 4a and Fig 4b. Finally, we explore brief experiments on CNN pruning to show generalizability beyond standard transformer modules.
Method
FLOPs(M)
Top-1(%)
Epochs
Baseline
313.73
93.96
T,!
206.00
93.40
HRank
131.17
93.73
200-300
CFDP
131.17
94.10
200-300
OPTIN
131.17
94.10
100-150
Figure 4: Pruning on CNN. Benchmarking the performance of OPTIN on the CIFAR- 10(Krizhevsky et al., 2009) dataset using VGG-
16-BN. OPTIN outperforms previous model com- pression techniques.
Method FLOPs| Params] ml)U(%) (a)ency(4)
Baseline
78.81
OPTIN; 24.2% 46.6%
74.57
13%
(a) Fask2Former (Swin-Thh Ti): Semantic Segmentation. Benchmarking OPTIN; on the CityScapes (Cordts et al., 2016) FLOPs. Params.. and Latency reduction mea- sured relative to SWIN encoder.
(b) High-Resolute Segmentation (a) Baseline Mask2Former; (b) OPTIN Framework a FL.OPs reduction of 25%. The minimal observable discrepancy encireled white rectangles.
Figure 5: Evaluated OPTINg on HR (1024x2048) Segmentation ((a) Quantitative; (b) Qualitative).
Exploring Semantic Segmentaion To demonstrate the OPTIN framework's 's generalizability to complex architectures and downstream tasks, we apply model compression to the Mask2Former Architecture with the Swin-Tiny backbone on the Cityscapes dataset. We specify the selected prunable components in Appendix A.8. In Tab. 4a we show impressive performance despite roughly a 24% reduction in FLOPPs and 47% reduction in parameters of the endocer. Qualitatively can strong resemblance between the original and compressed network, with a small discrepancy in predictions towards the bottom right of the frame in an already difficult-t--segment region (as evidenced by the unclear segmentation in the original prediction) and on the traffic sign towards the top left in Fig. 4b.
Exploring CNN Architectures To demonstrate the potential applications of OPTIN architectures, we extend our trajectory measure as described in Appendix A. In Tab. 4a we compare the model compressed through the OPTIN Framework with the baseline on the VGG-16-BN architecture, heuristic approach (L') (Li et al., 2017) and two leading (Lin et al., 2020a) and CFDP(Khaki & Luo, 2023). Following previous works (Lin et al., 2020a), fine-tuning has been shown to be required post-compression. However as evident in Tab 4a, following the same training procedures as HRank, we were able outperform all methods much faster convergence speed given comparable FL.OPs reductions.
CNN
HRank
5 CONCLUSION
In this work, we introduced OPTIN as one-shot technique to enable efficient compression of modern pre-trained transformer architectures without re-training. OPTIN exploits the trajectory of prunable components in the transformer architecture to enable a smarter criterion for parameter selection. In this work, we've explored several domains including natural language processing, image classification, transfer learning and semantic segmentation tasks. We additionally show how method can work in concert with existing token reduction modules to produce even stronger competitive results in the image domain. We further expanded our method to show robustness on prior CNN-style architectures opening future avenues of research into fused architectures. In all cases, we' ve shown robustness against compression rates and competitive performance including against methods that perform re-training. We complement our performance improvements with synonymous improvements in throughput speed enabling the practical use of our framework. In the future, we plan to explore more complex architectures and tasks in addition to expanding the number of prunable components to further the cause in an efficient design of transformer models.

---

REPRODUCIBILITY STATEMENT
The attached supplemental code contains a framework with the algorithms and metrics behind our main results. All of our adapted L formulations are described in the main paper: Sec Appendix A.9, A0 and are implemented in the supplemental code. By our innate one-shot structure, there are no training augmentations applied we don't re-train. The exception for transfer learning and re-training on the CNN architectures. For these, we adopt the standard augmentations from HRank (Lin et al., 2020a). Our datasets and evaluation metrics are described in Appendix A.3, A.4.
10

---

REFERENCES
Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. The Eleventh International Conference on Learning Representations, 2023. URL https: h http:///pppeeeeeeeeee/eeee
Tianlong Chen, Yu Cheng,
Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems, 34:19974-19988, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299, 2022.
Zheng Chuanyang. Zheyang Li, Kai Zhang, Zhi Yang, Wenming Tan, Jun Xiao, Ye Ren, and Shiliang Pu. SAVit: Structure-aware vision transformer pruning via collaborative optimization. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL w5DacXWzQ-Q.
www.vviw.neeeffuuunn net/foummmid
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL https://openreet net/forum?id=-iicbFiN
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 16091-16100) 2023.
Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Juergen Gall. Adaptive token sampling for efficient vision transformers. In European Conference on Computer Vision (ECCV), 2022.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pp. 3690-3699. PMLR, 2020.
Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine-grained manifold distillation. In Advances in Neural Information Processing Systems, 2022.
Joakim Bruslund Haurum, Sergio Escalera, Graham W. Taylor, and Thomas B. Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV) Workshops, October 2023.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. Associates, Inc., 2020. URL file/6f521 6f 010.55/60888800111014444444eeee0
Curran neurips.cc/papet cc/paper files/ https//pppr/2222
nww.ips.

---

Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, pp. 2, 2019.
Pre-training of deep
Samir Khaki and Weihan Luo. Cfdp: Common frequency domain pruning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 4714-4723, June 2023.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 1-bert: Integer- only bert quantization. In International conference on machine learning, pp. 5506-5518 PMLR, 2021.
Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency-aware soft token pruning. In Computer Vision-ECCH 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XI, pp. 620-640. Springer, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 4163-4181, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. 10.18653/v 118633/w12022.emmmlll mainnlpmmii..9...... URL https://aclanthollogg org/2022. emnlp-main.22 279.
Woosuk Kwon, Sehoon Kim, Michael W. Mahoney, Joseph Hassoun, Kurt Keutzer, and Amir Gholami, A fast post-training pruning framework for transformers. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL hts http:///ppeeeeeeeeeeeeeeeeeee wwwpppeeeeeeeeeeeeeeeeeeeee www.forumtrmeeeeee (idommmnLdB1O3B1111
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander Rush. Block pruning for faster transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10619-10629, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 100.0///w112211mmmllpm //aclanthology.org org//021 wwm.lp-main.n..........mmmmm 829.
URL https:
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017. URL https:
//openreview.ere wtw.eeewweee//orreeeee www.:nnn//ooumiiddrffofoft.o
Youwei Liang, Chongjian GE, Zhan Tong. Yibing Song, Jue Wang, and Pengtao Xie. EVit: Expe-
diting vision transformers via token reorganizations. In International Conference on Learning Representations, 2022. URL
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv, 2023.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1529-1538, 2020a.
Zi Lin, Jeremiah Liu, Zi Yang, Nan Hua, and Dan Roth. Pruning redundant mappings transformer models via
identity prior. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 719-730, Online, November 2020b. Association for Computational Linguistics. doi: 10. 10..18653.00 118653/v1/20200 http://w//22...nnnnnneeeee findings-emnlp.64 URL wtthhtts://aaa htttps://aclaathhll.ooo or orgg 2020. findings-emnlp. 64.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVF international conference computer vision, pp. 10012-10022, 2021a.
12

---

Zejian Liu, Fanrong Li, Gang Li. and Jian Cheng. EBERT: Efficient BERT inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. pp. 4814-4823, Online, August 2021b. Association for Computational Linguistics. doi: 10. 18653/v1/2021 http://.2.21....n......ee .iindings-acl .25.. URL https: /aw.a/oooootoooyo.oooooomooo org org/2021. findings--cl., aaces-s.c..255..
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Zizheng Pan, Bohan Zhuang. Jing Liu, Haoyu He, and Jianfei Cai. Scalable vision transformers with hierarchical pooling. In Proceedings of the IEEE/cvf international conference on computer vision,
pp. 377-386, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: / http://pooeeennn..eoo newreeerdinnneniiiieeeeeeeeeeeee File/bdbca288fee.77
neur meurips.ce wwwpppppeaapapp/ppppppppeeeeeeee ptppe//ee///////////
Baoyun Peng. Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5007-5016, 2019.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https:
Ahmad Sajedi, Samir Khaki, Ehsan Amjadian, Lucy Z Liu, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Datadam: Efficient dataset distillation with attention matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17097-17107 1077 2023a.
Ahmad Sajedi, Samir Khaki, Konstantinos Plataniotis, and Mahdi Hosseini. End-to-end super- vised multilabel contrastive learning. arXiv preprint arXiv:2307.03967, 2023b.
Ahmad Sajedi, Samir Khaki, Yuri A Lawryshyn, and Konstantinos N Plataniotis. Probmel: Simple probabilistic contrastive learning for multi-label visual classification. arXiv preprint 2rXiv:2401.01445 2024.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 20378 20388. Curran Asso- ciates, Inc., 2020. URL https: hwwpseeeeddinnnnniiiss..ommm wwwppppp.ccppppr.ffllle....... (aee5aaaaaeee449999944444446-000
.dff.
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. UPop: Unified and progressive pruning for compressing vision-language transformers. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 31292-3133 11311. PMLR, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556. 2014.
Zhuoran Song, Yihong Xu, Zhezhi He, Jiang, Naifeng Jing, and Xiaoyao Liang. Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction, 2022.
Yehui Tang, Yunhe Wang. Yixing Xu, Dacheng Tao. Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 33:10936-10000 10947, 2020.
13

---

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Iliia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https: ptttpp:://pperriiiie forum?2d=r34km22557,
Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. HAT: Hardware-aware transformers for efficient natural language processing. In Proceedings of the 58th
Anmual Meeting of the Association for Computational Linguistics, pp. 7675-7683 Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/v1/2020.cam haw.:////220000m0mmmmm.m.2-6..3 URL https://acllanthollog orgo 2020. acl maain. maim.................
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2020b. doi: 10.18653/v wtl.1866612220.coom 1/t0://0000000.0mmmmaaaaa.m 496. URL https::/doi.org// 10. 18653%2Fv 12222200 1x222000..mmmmmmmm.. 496.
Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2092-2101, 2023.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. https://acll http:://aclanthology org 2020. mamlp-demos.6
URL
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL.), 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 38087-38099. PMLR, 23 Jul 2023. URL https://proceedinn mlr www..ppss//eee. wtw.prrsssvv22/xxeeepp https///2//xxxxxxeee// html.
Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian-aware saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18547-18557, 2023.
Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width & depth pruning for vision transformers. In AAAI Conference on Artificial Intelligence, 2022a. URL https: //api. semanti sesanticscoollrrr..... www.oolarrrrrrr.orooeeeeeeeoeeee orgg/CorpusID 250294994.
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24355-24363. 2023.
Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang, Ji Liu, and Zhangyang Wang. Unified visual transformer compression. In International Conference on Learning Repre-
sentations, 2022b. URL https: /
14

---

Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference Machine Learning, pp- 26809-26888 PMLR, 2022.
Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation, 2022.
Mingjian Zhu, Yehui Tang, and Kai Han. Vision transformer pruning, 2021.
15

---

A APPENDIX
The appendix is structured to provide details that matches elicitation from the main text. Experiments and discussions included in the appendix serve as supplemental information to provide greater context to claims and experiments stated in the main text.
A.1 DISCUSSING THE OPTIN ALGORITHIM
Algorithm demonstrates the base structure for computing and assigning importance to each of our printable parameters. Once our importance scores were computed we directly leveraged the mask search algorithm from PTF (Kwon et al., 2022) as it searches to maximize scores in the partitioned search space. Their policy partitions the search space by incrementally adding attention heads in order of importance, and at each step, adding the maximum number of rank-ordered neurons that will satisfy the cost constraint C(1%. 0' ... (%1)= S C. By computing the cumulative importance each step, easily deduce that the step with the maximum cumulative importance must be optimal, as any other configuration would yield a cumulative score less than or equal to that of the best.
A.2 DETAILS ON HYPERPARAMTER À
In Tab 1d, we use the sweep to express relative magnitude differences between LMD and LKD- Based on the reported results, we can conclude that larger importance should be weighed on the distillation loss, in particular, we expect that (og {10, 100} * [logio(CkD)] (i.e. the order of magnitude of LMD should be 10- 100 times larger than that of LKD)
A.3 DETAILS ON DATASET CHOICE
In this work, we evaluated the strength of the OPTIN framework on natural language processing benchmarks, image classification and semantic segmentation. Across this wide variety domains, we used different datasets for the various tasks following from previous works to bolster the performance of our method.
GLUE Benchmarks The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) contains a variety of NLP-related tasks of which we included: Similarity and Paraphrase Tasks (MRPC STS-B, QQP) and Inference Tasks (MNLI, QNLI). The dataset distribution widely vary per task. From the included selection, STS-B is regression task, MNLI has three classes, and the remaining tasks include two classes.
ImageNet 1K Benchmarks The ImageNet-I K dataset (Deng et al., 2009) is a widely used benchmark for image classification, as cited in several works (Zhu et al., 2021: Goyal al., 2020: Pan al., 2021). It contains 1.2 million training images and 50K validation images across the 1000 classes. Due its difficulty, stemming from the number of classes and images, presents a perfect benchmarking medium for our one-shot pruning approach.
CIFAR 10 The CIFAR 10(Krizhevsky et al., 2009) datasets are common benchmark across both convolutional neural network pruning (Khaki & Luo, 2023; Lin et al., 2020a) CIFAR 10 contains 50K training and 10K validation images spread over 10 classes. Due the prominent use of this dataset in benchmarking tasks, we decided to benchmark our method for fair comparison with SOTA.
Cityscapes The Cityscapes dataset (Cordts et al., 2016) is heavily used for semantic segmentation tasks, and in particular, contains high-resolution images of (1024x2048), with roughly 3K training and 500 validation images. In this paper, our goal was to demonstrate the effects of pruning downstream network under the OPTIN framework, and due to the high resolution of Cityscapes data, were able to demonstrate improvements in throughput speed for our pruned segmentation model.
A.4 DETAILS ON EVALUATION METRIC CHOICE
The two main metrics reported in this work are FLOP(s) and Accuracy (or equivalently mIOU in the of segmentation). Given the target domain of this paper, these metrics best express the tradeoff between high accuracy and computational complexity, and further how the OPTIN framework is better able to make this distinction. The selected metrics have further been reported in similar
16

---

previous works (Kwon al., 2022; Bolya et al., 2023; Wei et al., 2023). Finally, also report im/s throughput to better illustrate the real-world implications the OPTIN Framework, especially in resource time-constrained environments.
A.5
AVERAGE TIME ANALYSIS
Task
Datasel
Model
Avg. Pruning Time (Hours)
Natural Language
GLUE Benchmark
BERT
0.4
Image Classification
ImageNet-IK
DeiT Tiny
0.3
Image Classification
ImageNet-IK IK
DeiT Small
0.3
Semantic Segmentation
Cityscapes
Mask2Former(Swn
0.5
A.6
ABLATIVE EXPERIMENTS
Onfi
Cenace
Take Movenn Endyy Tokeles Stitain Foodbe
Dataset
Batch Size
Acc.
MNLI
82.12
MNLI
32
81.90
64
81.73
MNLI
128
82.20
ImageNet
16
70.53
ImageNet
32
71.25
ImageNet
64
71.01
ImageNet
128
70.82
(a) Effect of Batch Size. The compu- tation of Z appears to be more robust to changes in the hatch-size.
(b) Ablating the Token Reduction Schedule. On ImageNet- IK for DeiT-S, our (PTIN, Framework produced more optimal reduction scheme as opposed to the default in ToMe (Bolya et al., 2023)
Token Merging Alg.
Scheduler =4Aoo(%)
FLOPs(G)
Baseline (N/A)
17.6
Random Prune
default
4 8.47
11.81
Random Prune
OPTIN)(oo)
1 7.11
11.75
bipartite merge
default OPTIN 1. 0.52
4. 0.80
11.81
bipartite merge
11.75
(c) Ablating the Token Merging Algorithm. On ImageNet-IK for DeiT-B, the bipartite matching default configuration from ToMe was used. However based on the results, is evident that OPTIN improves the performance of arbitrary token pruning methods well. Configuration Ablated: OPTIN.
Table 6: Additional Experiments Here we evaluate three components: (a) the ablative effect of batch size in computing the distillation loss, (b) a comparison of the optimal reduction schedule from OPTIN compared to the constant and decreasing schedules from ToMe. (c) the effect of different patch reduectonn/meringg techniques using the OPTIN Framework. Our default settings are marked in green.
17

---

A.7 ADDITIONAL LANGUAGE EXPERIMENTS
Method
MNLI
QQP
QNLI
SST
STS-B
MRPC
BERT BASE
84.53
91.00
91.41
93.57
88.90
86.27
PTF
81.21
89.99
88.38
92.13
87.10
83.14
OPTIN:
82.12
90.08
88.54
92.36
87.19
85.21
PTF
82.51
90.35
90.06
92.49
88.00
85.27
OPTIN=
82.74
90.43
90.35
92.73
88.21
85.68
Table 7: Natural Language Benchmarks. Augments the main table with two additional exper- iments. OPTIN 00 runs the OPTIN algorithm with A 0.01 to display the best results achieved using our standard framework. Further, OPTINES compares with PTFft which includes the mask tuning/scalinn from PTF (Kwon et al., 2022) to discover a non-binary mask that helps to reduce in-place reconstruction errors. Latency is estimated at B = 32 and ranges between 1 1.35-1.38 x improvement. results are averaged over 5 different seeds.
Task
Attention Heads
Hidden Neurons Patches&Tokens
Output Channels
Natural Language
Processing
Image Classification (CNN)
Image Classification (TF)
Semantic Segmentation
Table 8: Identifying the prunable weights that OPTIN uses accelerate the model for various downstream tasks
A.8 EXTENDING OPTIN TO VARIOUS DOWNSTREAM TASKS
When moving from the language domain to other applications, competitive methods leverage ad- ditional pruning components in order to spread the compression over larger search space. In response, we too apply this with OPTIN. Tab identifies the search space used OPTIN for various downstream tasks.
A.9 ADAPTING THE TRAJECTORY FORMULATION TO TOKENPATCH INFORMATIVENESS
As detailed in Sec 4.2, the OPTIN framework allows
select the best token reduction technique for their task to create an expanded prunable search space. The OPTIN framework adapts to image datasets by further producing optimal reduction schedule for tokens that can be leveraged by any reduction or merging technique. In the main paper, we use ToMe with bipartite matching, however, we ablate the metric choice and merging strategy Appendix A.6.
In order to obtain the optimal token reduction, we apply the trajectory estimation to patches in the vison-transformer models, by simply modifying the reshaping operator and the dimension upon which we compute the importance. We adopt the inter-sample representation (Hao et al., 2022) and since we are determining patch-level importance, it follows that we should compare our base and pruned embeddings along said dimension. We note that the use of the term putches in the context of frooo-transfommrss would be represented by the same dimension as the token sequence length the language domain. We redefine the manifold distillation loss according to the index j which ranges up to the number of patches for the co-responding model. We begin by redefining the manifold structure relational map based index where Fi,(..s] R Bx1xD as:
M(F.p.44 (F..
In particular, modify the L inter-image patch distillation loss from (Hao et al., 2022) by replacing the student input with that of the masked patch, and the teacher input with the precomuted embeddings from the network. For two given feature embeddings from layer for base and pruned model, the
18

---

sample manifold reconstruction enror would present
LMD(F Fi)
1 T
Σ
[IM(F (1.5a))))) M(F.A.Ja))l
This error is accumulated with standard KI. Divergence resulting in a similar Equation
After determining the importance score for each patch, we can derive the average number of patches required per layer for maximum information throughput by simple rank elimination given an FLOPs constraint. After executing the mask search with attention heads and neurons, we alleviate the remaining FLOP reduction required by removing tokens in order of ascending importance (i.e remove the lowest importance first). This ultimately produces the number of tokens per layer which can be extracted as token reduction schedule that can be leveraged on run-time with the ToMe bipartite matching scheme. We have further shown that the OPTIN Framework reduction scheme is much more informed than the standard constant or decreasing schemes commonly used - See Appendix A.6.
A.10 ADAPTING THE TRAJECTORY FORMULATION TO OUTPUT CHANNELS
To adapt the trajectory estimation to output channels in CNN-style networks, we reduce the relation to a simple mean-square error computed between the feature embeddings along the length of the model. In particular, we average the embeddings along the batch dimension and compute the sum of the mean squared error between the base and pruned model along each layer deeper in the network:
B
B
LMD Pa)
1 B
Σ
P.I
Once again, we are able to plug this into Equation with standard KI. Divergence to determine overall channel importance.
19


Reviews:
[{"review_id": "Sb8YAhHlix", "reviewer": "Reviewer_RsPn", "summary": "The paper introduces OPTIN, a framework designed to prune neural networks without the necessity of additional training. It prunes weights that have least impact on the intermediate embeddings and final logits within neural networks. The authors have tested OPTIN across a variety of tasks in both vision and language domains, showcasing its potential effectiveness.", "questions": "1. In Table 2, the results for PTF are directly cited from the original paper. Could the authors confirm that the pre-trained checkpoints and fine-tuning settings are consistent across all experiments to ensure a fair comparison?\n2. Could the authors provide details on the estimated time required for OPTIN to prune different models across various datasets? This information would be helpful to evaluate the practicality of implementing OPTIN in different scenarios.\n3. Could the authors give further explanations on 'IM-Dense (After dense embedding layer)'. What does it mean?\n4. How does OPTIN determines layer-wise sparsity? Does OPTIN rank weights globally? \n5. In Table 1.(d), why $\\lambda_c$ is chosen to be 0.1 even though 0.01 provides better results?\n\nI am willing to adjust my rating if my questions are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. **Comprehensive Experiments**: The authors have conducted a wide array of experiments, covering domains such as vision, language, and semantic segmentation tasks, ensuring a comprehensive evaluation of OPTIN.\n2. **Strong Performance**: OPTIN demonstrates robust results across the tasks it was tested on, showcasing its effectiveness in network pruning.", "weaknesses": "1. **Performance on Vision Models**: As highlighted in Table 3, the performance of OPTIN (without token reduction) appears to be inferior to that of VTP and PoWER for both evaluated models. It seems that the impressive performance of $OPTIN_\\tau$ can be mainly attributed to token reduction. Does it mean OPTIN is more suitable for language models rather than vision models?\n\nMinor issues:\n1. Table 1: \"(a)Explores\" -> \"(a) explores\".\n2. A.6: \"Tab ??\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJULM22op1", "reviewer": "Reviewer_oBzK", "summary": "This paper introduces One-shot Pruning Technique for Interchangeable Networks (OPTIN) to prune networks with re-training. With intermediate knowledge distillation, the proposed method saves remarkable computational costs. Experiment results prove the effectiveness of the method.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper targets on a practical problem, model compression in a more efficient way. \n2. Overall this paper is well-written and easy to follow. It presents the proposed method very clearly.", "weaknesses": "1. I think this paper is a ok paper. For discussion, I want to see the method performance on more complicated network architecture or datasets. \n2. I shall mention that I am not very familiar with the baseline methods. Therefore, if the other reviewers points out the missing related works, please add the comparison results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yo4VpncFje", "reviewer": "Reviewer_meSa", "summary": "The authors propose a post-training pruning method that operates iteratively. The idea is to approximate each weight's impact on the validation error. To achieve this, the authors mask out weights, then run the forward pass and compute the change in logits.", "questions": "- typo \"Perforamnce\" in 4.1\n- Every instance of \"without re-training\" is italicized; it's a bit excessive to do that, particularly given there is previous work that already introduced the idea.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a significant number of experimental results and ablations.  The ablations in Table 1 are convincing evidence that the design choices made were fully vetted.\n- Overall, the study seems very thorough. I don't have any strong reasons to reject or to accept. (suggestions below)", "weaknesses": "- It seems like PTF is the most relevant and competitive baseline. Per 4.1, the authors compare with an abridged version of PTF however. If you use the same tricks that PTF used (e.g., mask rearrangement -- maybe mask tuning), would your method outperform PTF?\n- Figure 1 is a bit complicated to look at. One possibility is to break up this figure into two (the left vs. the right). The details are nice, but they detract from the main point (e.g., the activation difference and the final difference in logit distribution). The logits could be illustrated as two histograms, highlighting their difference.\n- The double axis in Figure 2 right is really confusing. Maybe just add a 5th plot? Might not be so easy to see for anyone with color blindness.\n- How long does the search take to run? On the order of minutes? Hours? Days? PFT said it took minutes, so that seems like the baseline. Is there significant latency overhead in storing and fetching large activations in RAM, especially for vision models?\n- The paper's motivation could be stronger. The abstract and title focus heavily on \"without retraining\" but it seems like many previous papers already focus on the post-training regime (as the paper cites). The more important motivation is then: What is PFT lacking that is present in this method? *Why is that difference important, intuitively?\n\n\nSummary: In short, I'm very lackluster about the paper. The results seem reasonable, but the motivation and the presentation (figures for example) can be improved to make a more convincing case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)