Specialized Transformers: Faster, Smaller and more Accurate NLP Models
Authors: Amrit Nagarajan
Sanchari Sen
Jacob R. Stevens
Anand Raghunathan
Venue: ICLR 2022 Submitted (2022)

Abstract: Transformers have greatly advanced the state-of-the-art in Natural Language Processing (NLP) in recent years, but are especially demanding in terms of their computation and storage requirements. Transformers are first pre-trained on a large dataset, and subsequently fine-tuned for different downstream tasks. We observe that this design process leads to models that are not only over-parameterized for  downstream tasks, but also contain elements that adversely impact accuracy of the downstream tasks.
We propose a Specialization framework to create optimized transformer models for a given downstream task. Our framework systematically uses accuracy-driven pruning, i.e., it identifies and prunes parts of the pre-trained Transformer that hinder performance on the downstream task. We also replace the dense soft-attention in selected layers with sparse hard-attention to help the model focus on the relevant parts of the input. In effect, our framework leads to models that are  not only faster and smaller, but also more accurate. The large number of parameters contained in Transformers presents a challenge in the form of a large pruning design space. Further, the traditional iterative prune-retrain approach is not applicable to Transformers, since the fine-tuning data is often very small and re-training quickly leads to overfitting. To address these challenges, we propose a hierarchical, re-training-free pruning method with model- and task- specific  heuristics. Our experiments on GLUE and SQUAD show that Specialized models are consistently more accurate (by up to 4.5\%), while also being up to 2.5$\times$ faster and up to 3.2$\times$ smaller than the conventional fine-tuned models. In addition, we demonstrate that Specialization can be combined with previous efforts such as distillation or quantization to achieve further benefits.
For example, Specialized Q8BERT and DistilBERT models exceed the performance of BERT-Base, while being up to 3.7$\times$ faster and up to 12.1$\times$ smaller.    


Under review as a conference paper at ICLR 2022
SPECIALIZED TRANSFORMERS: FASTER, SMALLER AND MORE ACCURATE NLP MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers have greatly advanced the state-of-the-art in Natural Language Pro- cessing (NLP) in recent years, but are especially demanding in terms of their com- putation and storage requirements. Transformers are first pre-traue pre-trained on large dataset, and subsequently fine-tuned for different downstream tasks. We observe that this design process leads to models that are not only over-paramettrized for downstream tasks, but also contain elements that adversely impact accuracy of the downstream tasks. We propose a Specialization framework to create optimized transformer models for given downstream task. Our framework systematically uses accuracy-driven pruning, i.e., identifies and prunes parts of the fine-tuned Transformer that hinder performance the downstream task. We also replace the dense soft-attention in selected layers with sparse hard-attention to help the model focus the relevant parts of the input. In effect, our framework leads models that are not only faster and smaller, but also more accurate. The large number of parameters contained in Transformers presents a challenge in the form of a large pruning design space. Further, the traditional iterative prune-retrain ap- proach is not applicable Transformers, since the fine-tuning data often very small and re-training quickly leads to overfitting. To address these challenges, we propose a hicrarchical, re-training pruning method with model- and task- specific heuristics. Our experiments GLUE and SQUAD show that Specialized models are consistently more accurate (by up to 4.5%), while also being up to 2.5x faster and up to 3.2x smaller than the conventional fine-tuned models. In addition, we demonstrate that Specialization can be combined with previous ef- forts such as distillation quantization achieve further benefits. For example, Specialized Q8BERT and DistilBERT models exceed the performance of BERT- Base, while being up to 3.7x faster and up to 12.1 x smaller.
INTRODUCTION
Transformers models such BERT (Devlin al. 2019), GPT-2 (Radford al. 2019) and GPT- (Brown et al., (2020) have revolutionized the field Natural Language Processing (NLP), greatly advancing the state-of-the-art in many NLP tasks. Models that achieve good performance these tasks are of high practical significance, finding their place commercial applications such as social media monitoring (sentiment analysis), AI chat assistants (question answering), automated summa- rization tools (analyzing sentence similarity), etc. Therefore, there is a strong interest in creating more accurate and efficient Transformer models for these tasks.
Transformers are first pre-trained on very large datasets, and subsequently fine-tuned for different downstream tasks. However, the current method of pre-training and fine-tuning Transformers has two major drawbacks. First, fine-tuning the pre- preitreined Transformers leads to models that are highly over-parameteriine models for the downstream tasks, especially since many of these tasks have very limited training data. This could lead to unstable models (Dodge et al. 2020) with sub-optimal generalization ability (Michel et al., 2019). Second, these large fine-tuned models present high computation and storage requirements for inference. This problem is further exacerbated by the trend towards larger and more accurate models over time. For instance, increasing the number of parameters from 1.5B to 175B enabled reduction in perplexity for Language Modelling (on the Penn Treebank dataset) from 35,8 in GPT-2 (2019) to 20.5 in GPT-3 (2020). In this work, we address both these challenges, taking advantage of the over-parameterized nature of pre-trained models to

---

Under review as a conference paper at ICLR 2022
create individually Specialized models for the different downstream tasks that are smaller, faster and more accurate than the conventional tne-ttuned models.
Prior research efforts have explored approximation techniques, such quantization, pruning and distillation, for improving the inference efficiency of various classes of neural networks. However, these techniques invariably involve trade-off between accuracy and efficiency. In addition, the vast majority of these techniques require re-training or additional fine-tuning. This becomes especially problematic for Transformers, since these large models require significant time and energy to train and fine-tune. Fine-tuning is usually performed on a very small data set, and as a result, conventional iterative grune-and-retren methods quickly end up overfitting when applied Transformer models. In contrast, our Specialization framework utilizes the unique characteristics of Transformer training and deployment to enable substantial gain in both accuracy as well as efficiency. Our framework also does not require any additional re-training or fine-tuning, and can be applied in a plug-and-play manner to any Transformer model that is fine-tuned for any downstream task.
During pre-training, Transformers capture rich linguistic knowledge and gain deep understanding of the structure of the target language. Fine-tuning refines this knowledge for a specific downstream task by training task-specific final layer. We observe that, due to the nature of their design process, Transformer models are not only wverpparameterized but also contain parts that are, in fact, harmful to performance of downstream tasks. In order to exploit this observation in a principled manner, we introduce a framework to identify and prune the harmful elements a Transformer (parameters, grouped different levels of granularity i.e., self-attention blocks, feed-forward neural network blocks, attention heads and neurons), with the goal of maximizing accuracy on the downstream task. In contrast with prior pruning methods that prune elements with little-or-no impact on the network output, the proposed method prunes elements that have considerable impact on the output, leading to the highest positive impact on accuracy.
In order to reduce the large pruning space, we analyze the different elements of the fine-tuned Trans- former in hierarchical manner, starting with entire self-attention or feed-forward neural network blocks, followed by attention heads, and neurons, and prune the harmful elements. The core of the Transformer self-attention, where each token in the input builds its representation based on the extent of attention it places on all the other tokens. However, ing the attention span of each token to only focus on the relevant tokens (in certain layers) leads to better information flow inside the model. Hence, our framework is also equipped with the abil- ity to identify the appropriate layers and replace the "soft" self-attention with "hard" self-attention in these layers. We also introduce Transformer-speciff heuristies to minimize the run-time of our framework, thereby enabling it to scale to large Transformer models.
observe that in some cases, restrict-
We summarize main contributions as follows:
We introduce a Specialization framework that optimizes Transformer models for specific downstream tasks through the use of accuracy-drive pruning and selective hard attention.
We incorporate multiple heuristics in the framework, such hierarchical processing, model-driven insights, and run-time based ordering of elements, in order to minimize the overheads.
We propose significance analysis technique to identify the importance of each element of the fine-tuned Transformer for given downstream task. We use this technique to prune elements that are harmful to performance on the downstream task.
We propose the selective replacement of the "soft" self self-attention with hard attention in the appropriate layers, helping the model focus only on the relevant parts on the input to build better representations.
Across suite of different Transformer networks, we demonstrate that Specialized models are consistently more accurate and stable, while also being significantly faster and smaller than their conventional fine-tuned non-Specialized counterparts.
2 RELATED WORK
Task-agnostic optimizations. Given the effectiveness and popularity of Transformer models, sev- eral techniques have been proposed to overcome their computational and memory challenges, and
2

---

Under review as a conference paper at ICLR 2022
to accelerate inference using these models. A vast majority of these works introduce task-agnostic optimizations, using popular approximation techniques such as knowledge distillation (Sanh et al.
2019
Sun et al., 2020 Wang et al. 2020), early exit/ depth modulation (Elbayad et al. 2020 Xin
et al. 2020: Zhou et al. 2020). quantization (Zafrir et al. 2019). attention head pruning (Zhang et al.,
2020) and parameter sharing (Lan al. 2020). In addition, Faaaittoon,FFnnnaa al. (2020) randomly drop layers during pre-training, thereby enabling their dropping during inference; Khetan & Karnin (2020) learn the optimal sizes of the BERT elements during pre-training, and WW al. (2020) use Long-Short Range Attention to speed up the aelf-attention operation. Using DistilBERT and Q8BERT as ex- amples, we demonstrate that our techniques are complementary to these works, and can be applied while fine-tuning for specific task to create accurate ultra-efficient models for specific downstream tasks.
Tpec-ip optimizations. Hou et al. (2020), Shen et al. (2020). Diao et al. (2019), Wang et al. 2020). Lagunas al. (202T) and Sajjad et al. (2020) introduce task specificiiiic optimizations, but the gain in efficiency comes at the cost of degradation in accuracy on the downstream task. In contrast, our framework improves both accuracy as well as efficiency, and hence appeals to a wider range of users. We also demonstrate that task-specific optimizations more effective when applied to Specialized models compared to conventional fine-tuned models.
Pruning techniques. Finally, Structured Pruning has been applied to various classes of neural networks (Anwar et al. 2017 Molchanov et al. 2017). and greedy pruning strategies have also been explored to identify weights and parameters that the output is least sensitive (Zhuang al.) 2018 Ye et 2020). In contrast, our method is designed to identify and prune parameters that have most detrimental effect on the output.
METHODOLOGY TO SPECIALIZE TRANSFORMERS
We propose a framework for producing Specialized Transformer models that are optimized for a specific downstream task, illustrated in Algorithm Our framework performs two main optimiza- tions: (1) It identifies and prunes elements that hinder performance on the downstream task at hand. (2) It selectively replaces soft self-attention with hard self-attention to help the model focus only on the relevant parts of the input.
3.1 ACCURACY-DRIVED PRUNING
The problem of identifying an optimal set of elements to prune is challenging, and this is especially true for Transformers. In order optimize a given model, we would ideally want characterize the significance of each and every parameter in the model, rank them in order of importance, and finally prune only the least significant parameters. However, Transformers have billions of param- eters, making this process computationally infeasible. In addition, previously proposed techniques that can efficiently estimate the importance of each parameter, such as using Taylor expansion, are not applicable. This is because the (approximate, fine-tune, approximate] cycle does not work for Transformers during fine-tuning, since they very quickly overfit the limited training data for the downstream tasks (usually within 5 epochs). We address both these issues through the use of hier- archical greedy algorithm that does not require any additional training or fine-tuning. To determine the significance of each Transformer element, the given downstream task to obtain the baseline loss. Then, for the element under consideration in each iteration of the framework, we compute the loss of the current Transformer model with the el- ement removed. We prune the element under consideration if the validation loss when it is removed less than the minimum loss seen thus far during the optimization process. since the goal is find a model with minimum loss. Also, in order to prevent overfitting to the validation set, we introduce a generalization constraint in addition to the aforementioned loss condition. This constraint ensures that an element is pruned only if it decreases (or retains) the loss of at least a certain number (N) of samples the validation set over the current best solution (computed by reum_samplee heeppp function in Alg. D. where N is the number of elements in the validation dataset whose loss is less than the average loss of the misclassified samples (we consider samples whose loss is greater than the average loss of the misclassified samples to be outliers). Therefore, elements pruned only if a vast majority of the samples in the validation set benefit from their removal, resulting in improved generalization performance. If the loss with the element removed is greater than the minimum loss
first fine-tune the original Transformer model for
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Traanformer Specialization
Input : Fine-tuned (for the given downstream task) Transformer T, Validation Output: Specialized Transformer for the given downstream task TT'
Function onalyze_element (element E):
T prunned =
ew WVew_oow Loss = Evaluate([ D) if LowsLoss
Loss < i and paaamppee_helpee > N then Min_Loss Loss
T= T= prrnned
Function nam_samples__eeleeee
computes the number of samples in the validation set that benefit from the pruning of an element; must be N for element to be pruned
Baseline Loss = Evaluate (T,D)
M Baseline._............aae. Loss
Q= Order O wtt.eeeeeeett.....mmstttooooo... D)
for each layer
do
Replace soft self-attention in L with hard self-attention
New _oos D)))
if New Loss < Min and rum.ssample.hhlped > N then
L Min_Loss New-Los
else
L Restore soft self-attention in L
while Q is not empty do
TrialElement = Q.pop
analyze
TrialElement has not been pruned from and New Loss < Baseline Loss then if Tlemett is an attention block then
for each attention head h in TrialElement do analyze amalyze_elemene element(h)
else if TrialElement a feed-forward block then
for each neuron w in TrialElement do amalyze_elemenn(g eeeeeen(())
return T
seen so far but less than the baseline loss, we inspect the element at a finer granularity, and prune only parts of the element that hinder performance (rather than pruning the entire element).
Hierarchical processing of elements. is computationally prohibitive to analyze every single parameter in large Transformers using the method described in [ Since the framework iterates through the entries of the queue sequentially, its efficacy is dependent on both the total number of elements under consideration, and the time required to analyze each element. We take advantage of the inherently hierarchical structure of Transformers and consider the elements manner, ordered by increasing granularity. Specifically, we analyze entire feed-forward and self- attention blocks first, and inspect them at finer granularity (attention heads and neurons) only when required. Through this ordering, we are able quickly eliminate large numbers of parameters from further consideration. In addition, due to the over parameterized nature of Transformers, it is likely that time-consuming blocks are pruned from the Transformer earlier in the process, thereby speeding up future iterations of the framework. For example, eliminating a single feed-forward block in the BERT-Base model removes 5.6% of all parameters under consideration, and speeds up future iterations by 1.15x. To further reduce the number of elements under consideration, dynamically remove elements they are encompassed by a high-importance block. For example, if given self-attention block is determined to be of high importance (the validation loss with the block removed is greater than the baseline loss), we remove heads within that block from further consideration.
hierarchical
also
Creating an ordered queue of elements for inspection. Since our framework performs greedy pruning of highly peeepearaeteri models, it is essential to know where the harmful elements
4

---

Under review as a conference paper at ICLR 2022
likely to be. Some elements may appear to be harmful for the downstream task (especially in early iterations of the framework), but this often ends up being an artifact of over- parapaaattteiiiiiionn As result, when elements are not carefully ordered for inspection, our framework lands in local minima of the validation loss function, leading to inefficient models with sub-optimal generalization ability. Our solution to this problem utilizes the unique linguistic properties captured by the different Transformer layers (Jawahar et al., (2019) to guide the ordering of elements for inspection. For example, it was found that BERT captures phrase-level information in the lower layers, mapping related tokens together. The lower layers also capture surface features, while the middle layers capture syntactic features and higher layers capture semantic features. It was also observed that BERT requires deeper layers only when long-range dependency information is required. Different tasks require different types of linguistic knowledge. For example, sentiment analysis requires only local context, and long-range information often ends up confusing the model, since sentiments often change rapidly: it also unlikely that syntactic and semantic information are needed. Hence, we place the final layer at the front of the queue, and work way backwards towards the first layer, since blocks in the final layers are more likely to hinder performance on sentiment analysis. This ordering of elements ensures that elements that are pruned early in our framework (when the model is most over- ) do not lead the system into bad local minima.
3.2 SELECTIVE USE OF HARD SELF-ATTENTION
In traditional Transformer architectures, the self-attention operation computes the attention scores of each token in the input sequence with all the other tokens. These attention scores, after passing through the softmax operation, are used to build the new representation for the token based on the extent of attention it places on the other tokens. The use of this "soft" attention enables end-to-end training of the Transformer. However, we observe that replacing the softmax operation on all at- tention scores with softmax only on the attention scores above a certain threshold T. in selected layers after training and fine-tuning is completed helps the Transformer focus only on the relevant parts of the input sequence (Fig. D This leads better information flow inside the model, improv- ing performance on many downstream tasks (especially those involving sequence classification and question answering). This also introduces large amount of activation sparsity that can be exploited to analyze the information faster, and helps alleviate the critical memory bottleneck in Transform- ers. Ta is set to a fraction of the maximum attention score of each token, and tokens on which the selected token places significantly less attention than it places on its most relevant token are consid- ered irrelevant. The exact value is tuned for the different downstream tasks (based on maximizing the validation loss), since observe that different tasks require different attention spans for opti- mal performance. We note that replacing soft attention with hard attention in all layers (especially the deeper layers) leads loss of important information in the model and hence, loss of accuracy, necessitating its selective use. In order to identify the layers that benefit from hard attention, we replace the soft attention with hard attention in each layer (one by one), and inspect which type of attention leads to smaller validation loss.
John
John
John
John
skipped
skipped
skipped
skipped
school
school
school
school
since
since since
since
he
he
was
was
was
was
sick
sick
sick
sick
(a)
(b)
Figure I: Illustrations of (a) Traditional "soft" self-attention and (b) "Hard" self-attention for the word "he". In hard self-attention, "he" concentrates all of its focus on "John", while in soft self-attention, "he" places a small amount of attention on the other irrelevant words also. Therefore, hard self-attention helps build better representation for "he".
5

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS AND RESULTS
We implement our techniques within Huggingface's Transformers library in PyTorch (Wolf et al. 2019). We use Intel Al's NLP Architect for experiments on QSBERT. The experiments were per- formed on a GeForce RTX 2080 Ti GPU with 1IGB memory. All results are reported on the test set, averaged across 10 runs with random seeds after 3 epochs fine-tuning, unless otherwise specified. We randomly sample 15% of the training set with class balance, and use it as the validation set to guide the Specialization process. We present results on the GLUE dev set in Appendix A
Specialization leads to more accurate models that are also faster and smaller. We present results on GLUE (Wang et al. 2019), a set of Language Understanding tasks, and SQUADvL. (Rajpurkar et al., 2016), Question Answering task, in Table I For GLUE, we present results on the test set using the GLUE evaluation server to obtain the scores al. and for SQUAD, we present up to 4.5% more accurate, while also being up to 2.5x faster and up to 3.2x smaller than their baseline counterparts. In addition, Specialization is able to obtain substantial improvements over Q8BERT-bass (already 4x smaller than BERT-base due to the use of 8-bit integer quantization) and DistilBERT-bass (60% faster and smaller than BERT-base). Specialized Q8BERT-base and DistilBERTT-bae models exceed the accuracy of non- Specialized BERT-base models, while being up to 3.7x faster and 12.1x smaller than BERT-base Therefore, Specialization can be used in conjunction with other approaches that improve the effi ciency of Transformers to create highly efficient accurate models for different downstream tasks.
results the dev set. Specialized models
Table 1: Results on GLUE and SQUAD v1.1. We report Matthews correlation for CoLA, Pearson Correlation for STS-B and accuracy for all other tasks. We report only "matched" accuracy for MNLI and the Exact Match score for SQUAD. Speedup and Compression are reported over the non-Specialized baselines for DistilBERT and Q8BERT, and not over BERT-base.
SQUAD
CoLA
MNLI
MRPC
QNU
QQP
RTE
SST-2
STS-B
WNLI
Average
v1.1
BERT-Base
81.97
51.38
85.37
83.11
90.74
89.76
62.65
95.94
83.8
64.1
78.88
Specialized
BERT-Base
83.88
54.43
86.63
85.88
91.17
91.12
65.81
96.7
85.08
65.1
80.58
Speedup/
1.38x/ /
1.98x/ /
1.21x/
1.61x/
1.16x/ /
1.49x/ /
2.2x/ /
1.32x
1.71x /
2.51x /
1.67x/
Compression
1.52x
2.88x
1.6x
2.02x
1.31x
1.48x
2.69x
1.44x
1.38x
3.18x
1.95x
Q8BERT-Base
81.55
50.5
84.73
82.05
90.19
89.65
61,26
94.32
83.09
61.21
77.86
Specialized
82.52
54.12
84.77
91.08
91.44
65.8
96.02
84.14
63.1
79.73
Q8BERT-Base
Speedup/
1.31x/ /
2.28x
1.08x/
1.57x/
1.3x
1.26x /
2.28x
1.15x/
1.58x /
2.44x /
1.63x/
Compression
1.41x
3.02x
1.24x
1.47x
1.2x
1.38x
2.74x
1.21x
1.46x
2.82x
1.8x
DistilBERT-Base
79.94
50.5
83.04
81.07
89.72
88.92
62.01
93.88
83.49
61.3
77.39
Specialized
DistilBERT-B.. Base
81.91
54.08
85.47
85.02
91.01
89.2
64.5
95.48
84.16
63.3
79.41
Speedup/
1.21x/ /
1.84x/
1.09x
1.34x
1.07x// /
1.21x/ /
1.94x
1.08x
1.31x /
2.31x /
1.44x
Compression
1.28x
1.96x
1.14x
1.41x
1.08x
1.14x
1.7x
1.17x
1.28x
1.98x
1.41x
XLNee-Baae
82.28
51.98
85.93
83.68
91.01
89.83
62.01
95.88
84.12
64.8
79.15
Specialized
83.93
54.28
85.92
91.78
91.12
65.78
96.98
85.46
65.1
80.71
XLNet-Bas Base
Speedup/
1.48x
2.06x/
1.38x/
1.98k/
1.47x./ /
1.78x/ /
2.11x/
1.59x
1.93x /
2.46x /
1.83x/
Compression
1.55x
2.84x
1.72x
2.12x
1.56x
1.6x
2.48x
1.71x
1.65x
2.98x
2.02x
Specialization reduces sensitivity to random seed initialization. Previous research (Dodge et al., 2020) has shown that the random seed (which determines the initialization of the task-specific layer and the order of training data for the downstream task) has a significant impact on the quality of the fine-tuned models. When the amount of training data small (which is the case with most downstream NL. tasks), this effect becomes more pronounced, evidenced by the fact that WNLI, which has the least amount of training samples (634 training samples), exhibits highest variance across runs (Table 2). Therefore, in order to find model with high accuracy a downstream task, multiple models need to be created (using different random seeds), and evaluated. Here, we demonstrate that Specialized models exhibit significantly less sensitivity to random seeds than their non-Specialized counterparts (Table thereby greatly increasing the odds finding "good" models in fewer iterations. In particular, find that even if two models have vastly different accuracies

---

Under review as a conference paper at ICLR 2022
on the validation set before Specialization, they converge to similar accuracies after Specialization. This, in turn, reduces variance on the test set.
Table 2: Sensitivity to random seeds. Results reported are averaged across the GLUE tasks and SQUAD on the Base models of each Transformer.
Maximum
Average of maximum task
Average of minimum task scores
Average variance across tasks
Transformer
variance for
single task
scores
BERT
1.48
4.86 (WNLI)
80.05
76.53
Specialized BERT
0.86
1.95 (WNLI)
82.02
79.93
Q8BERT
1.61
4.62 (WNLI)
79.01
75.29
Specialized
1.1
2.02 (WNLI)
81.14
79.58
QSBERT
DistilBERT
1.29
3.85 (WNLD
78.78
74.89
Specialized
0.83
1.78 (WNLI)
80.96
79.59
DistilBERT
Larger models Specialize better. Current state-of-the Transformer networks, such as T5 (Raffel et al. 2019) and GPT-3 (Brown et al.] 2020), have hundreds of billions of parameters. Model sizes are also expected to grow further in future as increasing the number of parameters has been shown to improve performance. This makes it computationally challenging to train Transformers as well as perform inference using them. Recent research ([i et al. 2020) has shown that larger models converge in significantly smaller number of training iterations than smaller models, and hence they train faster in spite requiring more time per iteration. However, larger models are significantly slower than smaller models at inference time. Here, demonstrate that Specialized larger models achieve much higher accuracy while being comparable in speed to Specialized smaller models (Table 3). In addition having higher accuracy ceiling, we also demonstrate that Specialized BERT- Large (pruned beyond the max accuracy point, by relaxing the constraints to trade-off accuracy for efficiency), is 1.5x faster than Specialized BERT-base at iso-accuracy. This indicates that larger models capture greater linguistic knowledge, but also contain more redundant parameters for the different downstream tasks, further motivating the development of larger and better language models that can be Specialized for fast and accurate inference.
Table 3: Specialization of BERT-base and BERT-large. Results reported are averaged across the GLUE tasks and SQUAD.
Transformer BERT-Basc
Average accuracy
Average samples/second
78.88
0.21
Specialized BERT-Base
80.58
0.12
BERT-LLarg
80,98
0.45
Specialized BERg-Large
82.89
0.15
Previously proposed techniques to improve inference efficiency are more effective when applied to Special- ized models. Specialization identifies and prunes harm- ful elements of the Transformer for the downstream task hand. Techniques that prune elements that have mini- mal impact on the output are complementary to our tech- niques, and we demonstrate this using the popular Lottery Ticket Hypothesis (Frankle & Carbin 2019). which finds sparse sub-networks that can be trained match the per- formance of the large network. Lottery Ticket Hypoth- esis has been successfully applied to Transformers also (Prasanna et al. 2020 Liang et al.) 2021 Chen al. 2020), and using these techniques, we demonstrate that winning tickets of the Specialized model are consistently more accurate than the iso-efficient winning tickets of the conventional fine-runed models (Figure
87
Accuracy
84
81
78
20%
40%
80%
100%
Inference Time
BERT
Specialized BERT
Figure 2: The winning tickets of con- ventional fine-tuned and Specialized BERT-Base on MRPC.

---

Under review as a conference paper at ICLR 2022
Specialization provides insights into the working of Transformers. We analyze which elements of the Transformer are pruned for different downstream tasks using different Transformer models (F We find that the differences in importance of elements are more pronounced across different tasks than across different models. For example, for sentiment analysis, long-range dependency information is not required, and often ends up confusing the model. Hence, for all models fine- tuned for sentiment analysis, we observe that components in later layers (closer to the output) are more likely be pruned. This is not the case with Language Modelling tasks (predicting masked words in text), where longer attention spans are required. Across models, we only observe subtle differences. For example, we find that XLNet regressvgress is able to learn important task- specific information earlier than BERT (non auto-regressive), similar to the observation made in (Sajjad al. 2020). Hence, we are able to drop more components (in earlier layers) in XLNet than in BERT, leading to more efficient models for inference. In DistilBERT (a distilled model), we find that there is a clear demarcation in linguistic knowledge across layers due to the reduced capacity of the model. This is evidenced by the fact that elements in the top four layers are never pruned across all Language Understanding tasks, while the boundaries are more soft in the original models. We also observe that Hard Attention is most useful in the lower layers, where phrase-level information is captured, for all the models and tasks. As future work, our framework can be combined with previously proposed techniques (such probing classifiers) gain deeper understanding of the working of Transformers, especially at finer levels of granularity.
Number of times
Sequence Classificaiion
Short Question Answering
Language Modelling
40
ATTN-Apppox = ATTH--rune Appos FFtoPrune
30
20
1
3
6
9
10
31
12
3
4
5
8
9
10
12
3
5
10
[al
Number of times
24
36
0
Layer
Figure 3: Elements pruned for (a) downstream tasks and (b) Transformer architectures across GLUE and SQUAD in our most accurate models for each task. Here, ATTN-Approx means only certain attention heads inside the attention block are pruned, and FFN-Approx means only certain neurons inside the feed-forward block are pruned.
Specialization using previously proposed importance estimation techniques leads to less accu- rate models. L1-norm and magnitude based pruning techniques are designed to prune elements that have minimal impact on the output, and hence, they are not effective identifying and prun- ing elements that have most detrimental impact on output. Gradient-based methods such as Taylor expansion are not reliable for increasing accuracy on the downstream tasks, since Transformers can- not be repeatedly fine-tuned to recover the accuracy losses from approximating the model (they very quickly overfit the limited training data for the downstream tasks). In addition, while knowledge dis- tillation is complementary to our method (as evidenced by our results DistilBERT)), we show that the accuracy gains from Specialization are significantly higher than those from regularized knowl- edge distillation proposed in (Hou et al.) 2020). Super-tickets (Liang et al. 202 demonstrates that accuracy can be improved by finding winning lottery tickets at low levels of pruning. However, we find that alleviating the peer-aaramtterzzaioon issue by pruning elements that hinder performance (and the selective use of hard attention) leads to more accurate models than by pruning elements that have least impact on output in the original model [T..le
Specialization incurs minimal overhead. Our Specialization framework does not require any ad- ditional training or fine-tuning iterations. It only requires multiple passes through small validation
8

---

Under review as a conference paper at ICLR 2022
set. Our hierarchical processing ensures that the number of elements to inspect is greatly reduced. In addition, each pass is expected to become progressively faster, since the framework potentially prunes element in each iteration, and our ordering of elements ensures that large. time-consumiin blocks are pruned early in the process, making future iterations faster. Therefore, the overheads of Specialization negligible compared the time required for re-training additional fine-tuning. We find that the average (for GLUE and SQUAD) wall-clock time for Specialization is only 7.6 minutes for BERT-base and 18.3 minutes for BERT-large on single GPU.
We also evaluate the heuristics used in our framework (Table 4. When elements are not carefully ordered for inspection. we find that elements that are pruned early in the Specialization process are pruned due to wver-parameterization and not because the linguistic knowledge contained in these elements is harmful/not useful for the task hand. This leads the system into bad local minima, where the validation loss cannot be further reduced by pruning other elements. We observe that starting at the layer granularity leads to worse models than starting at the block granularity. This is because the attention and fced-forward blocks in each layer have vastly differently functionality, and hence, the effect removing an attention block of relatively high significance is often masked by removing the ferwad block that greatly hinders performance in the same layer (and vice- versa), causing the entire layer to be pruned. This destabilizes the Specialization process, and also leads the system into bad local minima, thereby creating inferior models. To drive home the fact that our greedy approach combined with a global error bound does not lead to inferior models, we also experiment with an adaptive loss threshold. In particular, we use very tight constraint when analyzing elements at coarse granularities, and relax the constraint as we move towards finer granu- larities. We again find that there negligible change in the quality of the final model produced (the accuracy is slightly lower, possibly due to over- fittiinig to the validation set), but the Specialization process is significantly slower. We hypothesize that a single global error bound is sufficient because we order the elements in such way that for the given task at hand, we intuitively expect that the elements at the head of the queue are likely to removed using the linguistic knowledge different layers.
Table 4: [Left] Results of Specialization with previously proposed methods (MRPC with BERT- Base). For L1-norm, magnitude and Taylor (absolute gradient of loss), we prune 5% of the least important weights at a time and record the test accuracy. We report the highest test accuracy seen in this process. [Right] Results of Specialization with different heuristies. All heuristics use hier- archical processing of ordered elements with Selective Hard Attention, unless otherwise specified.
Pruning Method
Accuracy
Baseline (No
Wall-clock Specialization
83.11
Specialized Model Accuracy
Pruning)
Heuristic
LI-Norm
83,16
Time (minutes)
Magnitude
83.18
Taylor Expansion
Randomly ordered
(with additional
83.19
83.42
7.2
fine-tuning)
elements
Taylor Expansion (no
Start by inspecting
additional
83.32
83.44
5.8
fine-tuning)
layers
Regularized
No Selective
84,94
6.3
Knowledge
Hard Attention
83.46
Distillation
Adaptive
85.82
29.3
(DynaBERT)
Threshold
Super-ticks
84.62
Ours
85.88
Ours
85,88
CONCLUSION
We proposed Specialization framework optimize fine-tuned Transformers for the different down- stream tasks. The framework identifies elements that are hinder performance on the downstream task at hand, and prunes these elements. We also demonstrated the advantage of selectively using hard self-attention in selected layers to improve information flow. Using this framework, we produced models that guaranteeably more accurate, while also being up to faster and smaller.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM J. Emerg. Technol. Comput. Syst., 13())32:1-32:188 2017. doi:
10.1145/3005348. URL //cii /trg//0.114//30000
Tom B. Brown, Benjamin Mann, and Nick Ryder et al. Language models are few-shot learners. CoRR, abs/2005. 14166, 2020.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained BERT networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL htttt:::spp::teeeeeeetst nggrroceeedings.......oo neur ips cccppaper/222//hastte b6a f f2c 203.022f22022222 (al35a222299000034444 f2e3 Abst htm
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653//1/119-44433. URL https://doi.org org/10........ 1et.11166/wwiiti
1423
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations. data orders, and early stopping. CoRR, abs/2002.063333 06600, 2020. URI. htt.::ttts///cc /ttps:////axiii wttp://wwwrgg///////ee/// 06305
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URLhttps: 3Jg7KhVKPHF
net fortt/foum?me
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with struc- tured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net/fozee fort/form??2 id= wow.oooouudulyllol. SyloyyttDr
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neu- ral networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL hhttts:://ppeneeeeett
met/forun
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic BERT with adaptive width and depth. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Had- sell, Aaoiia-Foorna Balcan, and Hsuan- - Lin (eds.), Advances in Neural Information Process- ing Systems 33: Anmual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020. URL http http:::://///ooceeeeeee nggsseddeinggg.. nwwrips....... cc/ paper/2020/hase (w1.665526666611 www.889966666688088888
Abst Abbsttttaa-a-aaarrrr .html
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David R. Traum, and Lluís Marquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3651-3657, Association for Computational Linguistics, 2019, doi: 10. 10.18653/v1/p19-111 13366. URL https: //t::///ooiiio org /0. wt...//./.....1111111..1 18653/v1/ 619-1356
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tenybert: Distilling BERT for natural language understanding. CoRR, (bss/1999,1035, 2019. URL htt.:http//aaatt www.vvorrrggaoo. abs 1909. 10351
10

---

Under review as a conference paper at ICLR 2022
Ashish Khetan and Zohar S. Karnin. schubert: Optimizing elements of BERT. In Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel R. Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2807- 2818. Association for Computational I Linguistics, 2020. URL https::////..... www. ac web. orgg
aathology/2022 acl -main 250/
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M. Rush. Block pruning for faster Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 Novem- ber, 2021, pp. 10619-10622 Association for Computational Linguistics, 2021. URL https:
transformers. In
emnlp ceammnppmainn.... 829
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, net
April 26-30, 2020. OpenReview.net, 2020. URL pesttps://operreeee http://wwweee.rrurereeee wof.urrriiioo #1eA7ABEvS
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transform- ers. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5958- html
5968. PMLR, 2020. URL
mlr. htt.:nng..........ssss.s... watesss/vv19/1e1
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compres- sion to improving generalization. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- tics and the 1th International Joint Conference on Natural Language Processing. ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 6524 6544. Associa- tion for Computational Linguistics, 2021. doi: 10.18653/v aol--oog...111 URL https:
7/doi. ooo.i.oor/0.... /1186533v//220
acl-long..55500.. 510
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 14014-14024, 2019. URL http::tt:///pppe nips. cc/ www.realllleeettte-------a-eeeee
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. net./ formt/////rmoooommmmmmmmmmmm wtt.ormmmmdsscsiisso...
OpenReview.net, 2017. L
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When BERT plays the lottery, all tickets winning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, On- line, November 16-20, 2020, pp. 2208-3229. Association for Computational Linguistics, 2020. 259. doi: 1111653/v12222.emmpp... emn emm.pp mom.lp-maiin m.....nn.9
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR,
2019. URL htt:: //tp:i//aaxiv.a/aa org abs /1t//.99..0000 10683,
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Careeas, and Kevin Duh (eds.), Proceedings
11

---

Under review as a conference paper at ICLR 2022
of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA. November 1-4, 2016, pp. 2383-2392 The Association for Computational Linguisties, 2016. doi: 10.18653//1/d16-111 1264. URL bttp::///bi.org//D... d1 6- 1264
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov, Poor man's bert: Smaller and faster transformer models, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL http:// arxiv. org/ http//bs///99 1910. 01108
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: hessian based ultra low precision quantization of BERT. In The Thirty-Four Poerty-Fourthhhoo AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020. The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 8815 8821, AAAI Press, 2020. URL https: /atpss:/ai aaai .rgg o js/ index.idxx...... php/ AAAI CeAT/article///
6409
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic BERT for resource-limited devices. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2158 2170. Associa- tion for Computational Linguistics, 2020. wttpt:tt////. /www ac web.
2020. a0000acclmmin.. 195/
Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:://pperevvee net. fwr.morrmmmmmoo
FJ4 fJ4km25577
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 3E5ee243547deee hat.eeeeeeee00000001111134883330 Abst .ttml
wew.:eeeeedddd.................. CC wtcc/paper//ee http:/ppprrr//////2/ 2200e//2220/ashh.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's 's transformers: tft natural language processing. CORR, abs/1910.03771,, 2019. URL
http:://zze http://axxvvv./// org/ awwpaasss...............bbb.tw.a 1910.030000 03771
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL 1HKPH
net vet/Eozze
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating BERT inference. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2246-2251. Association for Computational Linguistics, 2020. URL www. www.wacllla.. acw.es org www.rrrrggtttgogooooooooooeeee/e
acl main 204/
Canwen Xu, Wangchumshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing BERT by progressive module replacing. CoRR, abs/2002.02925,, 2020. URL htt.s:tppp//..mm........ ///a/////xx..... org http/bb/220022. 02925
12

---

Under review as a conference paper at ICLR 2022
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good sub- networks provably exist: Pruming via greedy forward selection. In Proceedings of the 37th In- ternational Conference Machine Learning. ICML 2020, 13-18 July 2020. Virtual Event, vol- ume 119 of Proceedings of Machine Learning Research, pp. 10820-10830. PMLR, 2020. URL prwss / vv 9/ye20b. ht.ml.
http:://rrocedding ml www...lww.....pp.mmmmm.........m
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: quantized 8bit BERT. CoRR, abs/1910.06188, 2019.
http:://axxiv...roo
11t./1910... 06188
Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Qun Liu, and Maosong Sun. Know what you don't
need: Single-shot metanng for attention heads. CoRR, abs/2011.03377 2020. URL https:
wttp:iiv.rrrbb///000100 03770
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. BERT loses patience: Fast and robust inference with early exit. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httaa//eeess3222ccccdbaaaaaaaaaa
neuripp cc http:paprr/222////0
Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang. and Jin-Hui Zhu. Discrimination-wwarr channel pruning for deep neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 883-894, 2018 URL https: /poceedding.oooeeggsee www.ongsnnnurirr.omo B99277/0777/sssssscccccccccttttr Att.taaattt-stttttttt.oa.. .tml
13

---

Under review as a conference paper at ICLR 2022
A
ADDITIONAL EXPERIMENTS AND STUDIES
A.1 RESULTS ON THE GLUE DEVELOPMENT SET.
We present additional results on the GLUE dev set using the same models used to present results on the test set in Table We use 15% of the training set as the validation set for Specialization. Similar to results on the test set, Specialized models are consistently more accurate, faster and smaller than their conventional fine-tuned counterparts. In addition, Specialization is able to obtain substantial improvements over already optimized models (Q8BERT and DistilBERT).
Table 5: Results on GLUE. We report Matthews correlation for CoLA, Pearson Correlation for
STS-B and accuracy for all other tasks. We report only "matched" accuracy for MNLL Speedup and
Compression are reported over the non-Specialized baselines for DistilBERT and Q8BERT, and not
over BERT-base
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
WNL
Average
BERT-Base
53.21
86.11
91.06
64.25
88.13
78.58
Specialized
55.45
85.24
88.89
91.63
91.77
64.8
89.41
58.82
80.06
BERT-Base
Speedup/
1.98x /
1.21x/
1.61x/
1.16x
1.49x
2.2x
1.32x
1.71x /
2.51x/
1.67x /
Compression
2.88x
1.6x
2.02x
1.31x
1.48x
2.69x
1.44x
1.38x
3.18x
1.95x
Q8BERT-Base
52.85
83.73
85.05
91.19
90.65
64.26
92.32
88.09
56.11
78.25
Specialized
55.45
84.32
86.79
91.96
91.57
64.8
93.16
88.97
57.12
79.35
Q8BERT-Base
Speedup/
2.28x
1.08x /
1.57x/
1.3x/ /
1.26x
2.28x/ /
1.15x
1.58x /
2.44x /
1.63x
Compression
3.02x
1.24x
1.47x
1.2x
1.38x
2.74x
1.46x
2.82x
1.8x
DistilBERT-Base
50.25
82.04
84.07
88.72
89.92
61.01
90.48
86.49
56.33
76.59
Specialized
54.1
83.48
86.24
89.65
90.98
63.94
92.1
87.98
57.08
78.41
DistilBERT-BE
Speedup/
1.84x/ /
1.09x /
1.34x
1.07x /
1.21x
1.94x /
1.08x/
1.31x /
2.31x /
1.44x /
Compression
1.96x
1.14x
1.41x
1.08x
1.14x
1.7x
1.17x
1.28x
1.98x
1.41x
XLNet-Base
57.89
86.97
87.61
90.18
91.11
59.88
93.62
87.93
56.48
79.08
Specialized
57.89
87.81
88.96
91.84
92.08
64.8
94.74
88.71
58.8
80.61
XLNet-Base
Speedup/
2.06x/ /
1.38x /
1.98x
1.47x/ /
1.78x
2.11x /
1.59x
1.93x /
2.46x /
1.83x/ /
Compression
2.84x
1.72x
2.12x
1.56x
1.6x
2.48x
1.71x
1.65x
2.98x
2.02x
A.2 SIGNIFICANCE ORDERING ELEMENTS FOR INSPECTION
We find that the order in which elements of the fine-tuned model are inspected during accuracy- driven pruning has a large impact on the generalization performance of the Specialized model. Prior research (Jawahar al.) 2019) has shown that the linguistic knowledge contained in different Trans- former layers can be demarcated into three main functional regions: phrase-level information in bottom layers (the first one-third of transformer layers closest the input), semantic and syntactic information in the middle layers, and long-range dependency information the top layers. When elements are not carefully ordered for inspection. we find that elements that are pruned early in the Specialization process are pruned due to over-parameterization and not because the linguistic knowledge contained these elements is harmful/not useful for the task at hand. This leads the system into bad local minima, where the validation loss cannot be further reduced by pruning other elements (Table(6). The quality the Specialized model is worst when the blocks containing the most important linguistic knowledge is inspected first (bottom layers for sentiment analysis, Appendix A.3), since these blocks may get pruned simply to alleviate the over-parameterization In addition, find that the ordering of elements within the same functional block (for example, blocks in layer 10 before layer 11 or vire-versa), has negligible impact on the quality of the Specialized model, further demonstrating the existence of regions with different kinds of linguistic knowledge.
14

---

Under review as a conference paper at ICLR 2022
Table 6: Specialized aceuracy from inspecting elements in different orders on SST-2 using
BERT-Base. The bascline validation and dev accuracy of the fine-tuned model are 93.98 and 93.23, respectively.
Order in which elements are
Validation accuracy of Specialized model
Dev accuracy of Specialized model
inspected
Bottom, Middle, Top
94.06
93.28
Bottom, Top, Middle
94.38
93.41
Middle, Bottom, Top
94,44
93.52
Middle, Top, Bottom
95.22
94.28
Top, Bottom, Middle
95.18
93.88
Top, Middle, Bottom
95.24
94.56
A.3 ANALYZING THE LINGUISTIC KNOWLEDGE REQUIRED FOR DIFFERENT DOWNSTREAM TASKS
Different downstream tasks require different types of linguistic knowledge. We find that the most important functional block of the Transformer model for each task (corresponding to the most im- portant kind of linguistic knowledge required to effectively solve the task) agrees with our intuition about the task. For example, we expect sentiment analysis to require only local context, since long- range information often ends up confusing the model (sentiments often change rapidly); it is also unlikely that syntactic and semantic information are needed. This is experimentally validated by the fact that inspecting the blocks in top layers, followed by the middle and finally the bottom layers leads to most accurate models syntax and semantics to be most important for a task that tests the linguistic acceptability of a given sentence (CoLA). We also find that this holds across all Transformer architectures that study in this work (BERT, DistilBERT, Q8BERT and XI.ett. We leverage this intuition about the different NLP tasks in our framework reduce the overheads of Specialization without compromising on the quality of the Specialized model. If there is no prior knowledge of the task, we need to inspect elements in multiple orders obtain the best model.
SST-2 (Table Z}. Similarly, we expect knowledge about language
Table The order of inspected elements that provides maximum accuracy for different down- stream tasks using BERT-Base, DistiBBERT-Bas Q8BERT-Base and XLNet-Base. We find that the same ordering provides best performance on all the studied Transformer architectures.
Most important type of linguistic knowledge for solving this task
Dataset
Task
Best ordering
CoLA
Linguistic acceptability
Top, Bottom, Middle
Semantic/ Syntactic
MNLI
Entailment
Top, Middle, Botton
Phrase-level
MRPC
Semantic equivalence
Top, Middle, Bottom
Phrase-level
QNLI
Question answering
Top, Middle, Bottom
Phrase-level
QQP
Semantic equivalence
Top. Middle, Bottom
Phrase-level
RTE
Entailment
Top. Middle, Bottom
Phrase-level
SST-2
Sentiment Analysis
Top, Middle, Bottom
Phrase-level
STS-B
Sentence similarity
Top. Middle, Bottom
Phrase-level
WNLI
Reading comprehension
Top. Middle, Bottom
Phrase-level
SQUAD
Question answering
Top, Middle, Bottom
Phrase-level
15

---

Under review as a conference paper at ICLR 2022
A.4 ANALYZING THE BENEFITS SPECIALIZATION
Transformers are pre-trained on large text corpus on a difficult NLP task, such as predicting the word in given all the preceding words. When pre-trained models fine-tuned for different downstream tasks, they contain lots of task-irrelevant information that add noise and confuse the model. addition, since the pre-trained models are highly overparameterized, they severely overfit the small fine-tuning datasets. Intuitively, Specialization helps the model focus on the task at hand by eliminating the irrelevant information in the fine-tuned models.
A.4.1 IMPROVED GENERALIZATION
Accuracy-driven pruning can be seen form of training, since the objective of accuracy-driven pruning minimizing loss on the (validation) dataset - exactly the same as the objective of train- ing. When large, over-parameterize models are trained on very limited data, the constraints of accuracy-dric driiven pruning (1) loss can be reduced only be setting weights to 0 (cquivalent to prun- ing elements), and no other changes to weights are allowed, and (2) a majority of samples in the validation set must benefit from each weight update; reduction in loss is necessary but insufficient condition for weights to get updated - help it learn more effectively than SGD. As result, we find that training with SGD on subset of the training data, followed by accuracy-driven pruning on the remaining unseen data, produces models with better generalization performance than training with SGD the entire dataset (Figure 4). As added benefit, accuracy-driven pruning also leads to smaller and faster models at inference time.
Fine-tuned model
After applying selective Hard Attention
Specialized model
Accuracy 85.78
Accuracy 86.27
Accuracy 88.24
Figure 4: Change in class boundaries during Specialization on MRPC with BERT-Base. We use T-distributed Stochastic Neighbor Embedding (TSNE) on word embeddings right before the final classifier, using the 3 main components of each embedding. Here, and 1 refer to sentence pairs that are semantically non-equivalent and equivalent, respectively. Specialized models show better class separability, and hence,
accurate.
A.4.2 REDUCED VARIANCE
Accuracy-driven pruning and selective Hard Attention filter out noise in the model by removing irrelevant information that ends up confusing the model. Accuracy-driven pruning reduces noise by removing zask-irrelevant information. Selective Hard Attention further reduces noise by ensuring that the model focuses only the relevant parts of the input. We find that this leads to reduced variance on the dev/test sets, where Specialized models from different seeds acquire similar word representations, even if the initial fine-tuned models have vastly different word representations (and hence accuracies). (Figure 5). However, the reduction in variance is limited by the effects of random initialization of the specifcc final layer.
16

---

Under review as a conference paper at ICLR 2022
Specialized model with selective Hard Attention
Fine-tuned model with Soft Attention
Seed 42
Accuracy = 32.4
Accuracy 54.93
Seed 100
Accuracy = 50.1
Accuracy 56.34
Figure 5: Attention patterns Specialized models on WNLI using BERT-Base, shown the final ATTN block in the model (layer 12 in the fine-tuned model, layer 9 in the Specialized model). WNLI has the smallest training set among all of our studied tasks, and hence, exhibits highest sensitivity to random seeds.
17


Reviews:
[{"review_id": "WuKEuBejIFs", "reviewer": "Reviewer_Hc1r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The proposed method does not require further fine-tuning.\n2. It is simple enough, easy to apply on the general transformer-based network.\n3. Speed, size, and performance could be improved at the same time. That is interesting. \n\nCons:\n1. The pruning method is not very novel. \n2. Here I quote from the paper, \"where N is set to be greater than at least three-fourths the number of samples in the dataset. In order to reduce the effect of outliers, the exact value of N is tuned for the different datasets, based on the validation loss of each sample.\" Could you introduce the details of tuning N? \n3. Lack mechanism analysis. For example, how does the attention to each word change after the pruning of each element?\n4. Not enough ablation study. Not sure what really matters. The pruning of which element matters the most? \n5. \"For example, sentiment analysis requires only local context, and long-range information often ends up confusing the model, since sentiments often change rapidly; it is also unlikely that syntatic and symantic information are needed. Hence, we place the final layer at the front of the queue, and work our way backwards towards the first layer, since blocks in the final layers are more likely to hinder performance on sentiment analysis\". Here I have two questions: 1. How do you get this conclusion about the final layers and sentiment analysis? I am not convinced by the short reasoning process. 2. Considering the conclusion is correct, then \"Place the final layer at the front of the queue\" is based on some prior knowledge about the task. Therefore, can I assume that without these inductive biases, the performance will be worse? Should I introduce the inductive bias for each downstream task?\n6. It does not compare with some of the latest work. Like \"Know what you don't need: Single-Shot Meta-Pruning for attention heads\". Also, could it be combined with other works like compression? For example, your method then compression? Or does the implementation of your method hinder the implementation of other pruning/compression/quantization methods?\n7. (Minor) Some tables are using figures. I think these tables can be generated via latex.\n", "summary_of_the_paper": "This paper introduces a framework to prune parameters in transformer-based structures. The authors claim that the proposed method leads to a smaller model with faster and more accurate performance. It first replaces soft self-attention with hard self-attention by checking whether the replacement leads to a smaller loss and benefits more than N samples in the validation set. Using a similar way, it further prunes attention block, feed-forward block, and their neurons. \n ", "summary_of_the_review": "Overall, it is an interesting paper. An easy method to reduce the model size and increase the performance in the downstream tasks. Not sure whether it can be generalized to general transformer-based models. It looks more like some technology that is not so novel. There is also not enough detailed research analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tv27kPw15fk", "reviewer": "Reviewer_bSmL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n1. The paper is overall well-written and easy to follow.\n2. The evaluation is thorough with SQUAD and GLUE. The method shows effectiveness on both benchmarks and with different models by outperforming the fine-tuned counterparts. \n3. The paper proposes a progressive and hierarchical strategy that provides a solution for both self-attention and FFNN in a Transformer, which has the potential to be adopted in more tasks (e.g., generation, translation).\n4. Figure 3 is quite interesting - it looks like DistilBERT is more \"dense\" than an uncompressed model and it warrants further exploration.\n5. The method is practical since it does not require re-training and works on downstream tasks thus enabling the reuse of nearly all off-the-shelf pretrained models.\n\n### Cons\n1. My main concern is novelty - the pruning technique is not new and pruning a neural network progressively is also not a novel idea. That being said, the paper is still informative in my opinion.\n2. Although specialization does not require a complete re-training, it does introduce extra overheads for training such a model. The authors mentioned this problem on page 8 but I'd like to see more discussion with analysis. \n\n### Missing References \n- https://arxiv.org/abs/2002.10957\n- https://arxiv.org/abs/2005.07683\n- https://arxiv.org/abs/2006.04152\n- https://arxiv.org/abs/2109.04838", "summary_of_the_paper": "This paper proposes a new framework for pruning a pretrained Transformer on downstream tasks for better performance and efficiency. The resulted model is 2.5x faster and 3.2x smaller than a fine-tuned model. The authors test the effectiveness on GLUE and SQUAD with BERT-base, Q8BERT, DistilBERT and XLNet.", "summary_of_the_review": "This paper proposes a new pruning framework for pretrained Transformers with good evaluation and analysis. Although the idea is not completely new, I would like to recommend weak acceptance for this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LZhc7OvnRr9", "reviewer": "Reviewer_ccHS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The proposed method is interesting, and it can be adopted by different models with significant improvement.\n2. The accuracy-driven pruning is reasonable and different from previous pruning-based methods.\n3. Leveraging hard attention is an interesting idea. \n4. The paper is easy to read and the experiments are solid. The authors compare the method with multiple pruning methods including regularization and lottery ticket based methods. And the experiment shows that the proposed method Is better.\n\nWeakness:\n1. It would be better to have some more analysis on hard attention, such as some statistical comparison between soft-attention and hard-attention. Also, more analysis on training and inference speed between two attention mechanisms would be helpful.\n", "summary_of_the_paper": "The authors propose a Specialization framework to create optimized transformer models for a given downstream task. The framework systematically uses accuracy-driven pruning. The authors proposed two ways to reduce model parameters, 1) Hierarchical pruning. Start from analyzing entire feed-forward and self-attention blocks, and inspect them at finer granularity (attention heads and neurons) only\nwhen required. 2) Replacing soft-attention with hard-attention. The proposed method significantly improves benchmark models, BERT, Q8BERT and DistillBERT.", "summary_of_the_review": "Overall, the proposed method is novel and reasonable. The experiments are quite solid by adopting the method to multiple structures and comparing it with SOTA pruning methods.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E9HOWXHxKT_", "reviewer": "Reviewer_GDsL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method is a *greedy* method with unjustified heuristics. However, how can it perform well without getting stuck in a local minimum and have low variance? I am pretty suspicious about the results because many essential details and explanations are missing.\n\nAs far as I understand, it prunes a transformer fine-tuned for a downstream task. (I’m a little bit confusing because the abstract mentions pruning a pre-trained transformer.) Then, it is already overfitted in some sense. I am not sure how pruning can improve the generalization.\n\nI am curious whether similar pruned architectures are obtained with different random seeds. Is the pruning process deterministic or stochastic? If it is deterministic, is it measuring with differently fine-tuned transformers? Do you have any explanation about low variance?\n\nAre pruning and hard self-attention separate each other? What is the order of them?\n\nThe method does not perform any retraining. Simply thinking, eliminating one layer or even a smaller portion of parameters could change the representation distribution of the following layers. How can it perform well?\n\nThe paper should be revised further. The related work section is just a listing of works that improve the efficiency of the transformer. The authors should categorize them and provide clear relation with their work. Method and result sections include many long paragraphs only divided by headings.\n\nI don’t understand how to use Taylor expansion to estimate the importance of each parameter. Is it more than a first-order gradient?", "summary_of_the_paper": "This paper proposes Specialized Transformer that identifies harmful parts and prunes a pre-trained transformer in a greedy and hierarchical manner to boost accuracy in a downstream task. According to the paper, the proposed Specialized Transformer is faster, smaller, and more accurate than a standard fine-tuned model without any retraining.", "summary_of_the_review": "The results look very promising, but the presentation is bad. The method lacks crucial details for implementation. Their methods seem too naive to be effective, but the authors do not fully explain why they result in a good performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a simple and effective technique for task-specific pruning of transformer models that identifies which model components to prune by minimizing validation loss. Weaknesses of the paper include (1) related work reads more like a list and doesn’t compare and contrast the proposed approach with related work, (2) authors don’t compare to other structured pruning methods (that use different objectives) (3) lack of novelty — main difference with existing work is using validation loss to optimize and (4) one reviewer was unconvinced that the results should be possible given the approach. I share these concerns, and, in particular, I think they might be related. Given that the models are pruned using the development set (essentially equivalent to training on the development set), it seems infeasible that this approach could have been developed without looking at the testing data, and I’m concerned that this explains the unprecedentedly high accuracy compared to previous pruning approaches. At the very least, comparing to a baseline that trains on development data would be prudent in order to understand the result.
Decision: Reject