Optimizing Transformers with Approximate Computing for Faster, Smaller and more Accurate NLP Models
Authors: Amrit Nagarajan
Sanchari Sen
Jacob R. Stevens
Anand Raghunathan
Venue:  (2021)
Keywords: BERT
Transformer
NLP
Efficient
Faster
Smaller
Accurate

Abstract: Transformer models have garnered a lot of interest in recent years by delivering state-of-the-art performance in a range of Natural Language Processing (NLP) tasks. However, these models can have over a hundred billion parameters, presenting very high computational and memory requirements. We address this challenge through Approximate Computing, specifically targeting the use of Transformers in NLP tasks. Transformers are typically pre-trained and subsequently specialized for specific tasks through transfer learning. We observe that pre-trained Transformers are often over-parameterized for several downstream NLP tasks and propose a framework to create smaller and faster models with comparable accuracy. The key cornerstones of the framework are a Significance Analysis (SA) method to identify important components in a pre-trained Transformer for a given task, and techniques to approximate the less significant components. Our framework can be adapted to produce models that are faster, smaller and/or more accurate, depending on the user's constraints. We apply our framework to multiple Transformer models and different downstream tasks, including previously proposed optimized models like DistilBERT and Q8BERT. We demonstrate that our framework produces models that are up to 4$\times$ faster and up to 14$\times$ smaller (with less than 0.5% relative accuracy degradation), or up to 5.5% more accurate with simultaneous model size and speed improvements of up to 9.8$\times$ and 2.9$\times$, respectively.

Under review as a conference paper at ICLR 2021
OPTIMIZING TRANSFORMERS WITH APPROXIMATE COMPUTING FOR FASTER, SMALLER AND MORE ACCURATE NLP MODELS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Transformer models have garnreed lot of interest in recent years by delivering state-of-the-art performance in a range of Natural Language Processing (NLP) tasks. However, these models can have over a hundred billion parameters, present- ing very high computational and memory requirements. We address this challenge through Approximate Computing, specifically targeting the use of Transformers in NL.P tasks. Transformers are typically pre-trained and subsequently specialized for specific tasks through transfer learning. Based the observation that pre- trained Transformers are often over- veer-paaameerized for several downstream NLP tasks, we propose a framework to create smaller, faster and in some cases more accurate models. The key cornerstones of the framework are Significance Anal- ysis (SA) method that identifies components in a pre-trained Transformer that are less significant for a given task, and techniques to approximate the less signifi- cant components. Our approximations include pruning of blocks, attention heads and weight groups, quantization of less significant weights and a low-complexity sign-mater matching based attention mechanism. Our framework can be adapted to pro- duce models that are faster, smaller and/or more accurate, depending the user's s constraints. We apply our framework to seven Transformer models, including op- timized models like DistilBERT and Q8BERT, and three downstream tasks. We demonstrate that our framework produces models that are up to 4x faster and up 14x smaller (with less than 0.5% relative accuracy degradation), or up to 5.5% more accurate with simultaneous improvements of up to 9.83 x in model size or 2.94x in speed.
INTRODUCTION
Transformer networks with hundreds of billions of parameters, such as T5 (Raffel et al. (2019) ).
Megatron (Shoeybi al. (2019)). BERT (Devlin et al. (2019)), GPT-2 (Radford et al. (2019) and GPT-3 (Brown et al. (2020)), have achieved state-of-the the--e--thrrr art performance in several Natural Lan- guage Processing tasks. Model sizes expected to grow further in the future increasing the number of parameters has been shown to improve performance. For instance, increasing the number of parameters from 1.5B 175B enabled reduction in perplexity for Language Modelling (Penn Treebank) from 35,8 in GPT-2 to 20.5 in GPT-3. This makes it computationally challenging to train Transformers well as perform inference using them. The challenges associated with training these models are alleviated through the rre-)use of pre- pre-trained models that are subsequently fine-tuned for different tasks. Consequently, these models incur a major one-time cost in computational resources, time and energy during the pre-training process, but the repeated fine-tuning for individual down- stream tasks is performed a considerably lower cost.
However, performing inference using fine-tuned Transformer models continues to remain a chal- lenge because of the large amount storage and compute operations required. Prior research efforts have explored different techniques for improving the efficiency of Transformer inference. However, several of the proposed approaches either require training the network completely from cause significant degradation in
scratch (which is extremely compute and memory-intensivel accuracy on the downstream task. In this work, we overcome these limitations by exploiting the transfer leamning step in Transformers produce individually optimized models for the different

---

Under review as a conference paper at ICLR 2021
downstream tasks, using techniques that do not require training from scratch and maintain im- prove accuracy levels.
From the runtime and memory breakdown of Transformers (Fig. D. observe that the most time- consuming and which are used to identify and form relationships between the different tokens in text, and the feed- forward neural network blocks (FFN blocks) in the Transformer layers. These blocks together ac- count for more than 85% of the inference time (and more than 75% of the model's parameters). We accordingly optimize the execution of these two components in our approach. The self-attention component dominates the execution time and memory size for longer context lengths as its opera- tion scales quadratically in time and memory with sequence length. Some previous works (Kitaev) al. (2020). Ye al. (2019) have addressed this issue, accelerating training and inference of Trans- formers when large context lengths are used. However, they suffer from significant overheads and slowdowns in applications with smaller context lengths, such as question answering, where ques- tions and answers are usually short, in the order of a few hundred tokens. Our approach, on the other hand, performs well across context lengths, size of hidden layers, number layers and other network characteristics.
operations in a Transformer are the self-attention (ATTN) blocks,
The pre-training of Transformer models with some initial objective (most commonly predicting masked words in a large text corpus) and the subsequent fine-tuning on downstream task leads to highly over-parametterized models for many downstream tasks (Michel et al. (2019). providing am- ple opportunities for approximations. As these models grow larger, such opportunities are expected to increase even further. We observe that for a given downstream task, some parts of the pre-trained Transformer more significant to obtain good accuracy, while other parts are less important or unimportant. In order to exploit this observation in a principled manner, we introduce a framework to introduce approximations while fine-tuming a pre-trained Transformer network, optimizing for either size, latency, or accuracy of the final network. We perform and apply significance analysis in a hierarchical manner, first pruning entire blocks, followed by attention heads, and finally pruning weight groups. We achieve further gains by also allowing elements that cannot be pruned to be ap- proximated by other techniques. We specifically apply two forms of approximations, depending on the element type. For weights, we utilize quantization. For the self-attention operation, we replace the scaled dot product attention mechanism with a novel sign matching- based attention mechanism.
We summarize main contributions as follows:
We introduce framework for creating fine-tuned models from pre-trained Transformer models that are optimized for various metrics (size, latency, accuracy).
We incorporate multiple heuristics in the framework, such as hicrarchical processing, model-driven insights, and run-time based ordering of elements.
We propose significance analysis technique identify the importance of each element of the pre-trained Transformer for a given downstream task. We use this technique to prune entire blocks, attention heads, and weight groups and to guide the quantization of low-importance weights.
We propose cowplexxl attention mechanism, sign matching, in order to approximate dot product attention in the less significant attention layers.
Across a suite of different Transformer networks, including previously proposed optimized networks, we demonstrate that our techniques produce models that are up to 4x faster and up to 14x smaller (with less than 0.5% relative accuracy degradation), or up to 5.5% more accurate with simultancous size and latency improvements.
RELATED WORK
Given the effectiveness and popularity of Transformer models, several techniques have been pro- posed to overcome their computational and memory challenges, and to accelerate inference using these models. Most of these works directly pre-train efficient models from scratch. For exam- ple, DistilBERT (Sanh et al. (2019)), MobileBERT (Sun et al. (2020)) and TinyBERT (Jiao et al. (2019)) use knowledge distillation to train smaller and faster networks using the original network as teacher. LayerDrop (Fan et al. (2020) randomly drops layers during pre-training. thereby enabling
2

---

Under review as a conference paper at ICLR 2021
their dropping during inference. SchuBERT (Khetan & Karnin (2020)) learns the optimal sizes of the BERT elements during pre-training. Lite Transformer (Wu et al. (2020) uses Long-Shor Soog-Shott Range Attention to speed up the self-attention operation, with different attention heads attending to local and global context. Depth-adaptive Transformer (Elbaayad et al. (0200) and DeeBERT (Xin et al. (2020) modulate Transformer depth depending the complexity of each input sample using gat- ing functions that are trained along with the model. AIBERT (Lan et al. (2020) uses factorized embeddings and cross-layer parameter sharing. These works are orthogonal to ours, as the models that they produce are still subsequently fine-tuned for downstream tasks. We demonstrate using Dis- tilBERT, AIBERT and LayerDrop as examples that these optimized networks still have significant opportunities that our techniques can take advantage of.
Other works (including ours) aim improve the inference efficiency Transformers using tech- niques that do not require training new models from scratch. Among these, PoWER-BERT (Goyal al. (2020)), which eliminates redundant word vectors from the model without removing any pa- rameters, and QSBERT (Zafrir al. (2019)), which quantizes all weights and activations in the model to 8-bit integers through the use of Quantization- Aware Training at fine-tuning time, are or- thogonal and complementary to our work. Poor Man's BERT (Sajjad et al. (2020)) evaluates several layer-dropping techniques that do not require re-training. Compared to layer-dropping techniques that do not require re-training, at comparable inference speed, and this is especially true when working with highly optimized baselines such as Q8BERT. Our framework can also be adapted satisfy wide range of user constraints.
techniques produce models that are up 20% more accurate
3 PRELIMINARIES
Runtime/ Parameters Breakdown
Tosk-
Input
Embedding Loyer
Transformer 1
Transformer loyer
Transformer
Speofic
Output
100
Final Layer
80
x of total
60
40
ATIN BLOCK
FEN BLOCK
ÇATTN Hoade
LAVER NORM
LAYER NORM OUTTPUY LAYER
INPUT LAVER
Key
BERT
BERT
GPT-2 language Modelling
Number of Parameters
language
language
Volue
Understanding (n:128)
Modelling [e=512]
(e=1024)
Wright groups
groups
Final Layer
Layer blocks
Feed forward MLP blocs
Multi- haadhhaaddd self attentitnn blocks
and position Embeddings
Figure 1: [Left] A typical Transformer architecture and its elements. [Right] Runtime/parammttr-comm breakdown of Transformers. At smaller context lengths, the feed- forward neural network is the time-dominant operation. As context length increases, self-attention becomes the time-dominant operation. In both cases, ATTN and FFN blocks together account for >75% of total parameters, and ~85% of the total runtime on an NVIDIA GTX 1080 Ti GPU.
Transformer (Fig. T consists of an embedding layer, followed by multiple transformer layers stacked together, and a task-specific final layer. A transformer layer consists of the multi-headed self-attention operation (ATTN block), followed by a feed-forwart neural network (FFN block) with layer norm operations at the input and output of the layer. In this work, we define the elements of Transformer to include different levels granularity, i.e., ATTN blocks, FFN blocks, Attention Heads and Weight Groups. We define Weight Groups only along dimensions that do not impact the shape of the output of the block when these groups are removed.
The self- atffaattentonn operation takes input sequence of vectors X, and computes three matrices, Query X x Wg. Key = X Wt and Value X x W.. Then, the output of the self-attention x Key")) + attention mmnkk) x Value. For operation is computed as Y = so not allowed to attend to future tokens. Hence, attention mask regresseee models, tokens applied before the softmax operation, setting attention scores with future tokens to very large negative number, which becomes after the softmax operation. This operation has multiple
3

---

Under review as a conference paper at ICLR 2021
"attention heads" working in parallel the input sequence, where each head has its set of parameters to compute the query, key and value matrices. The independent attention outputs are concatenated and transformed into the expected output dimensions. The self-attention operation scales quadratically in time and memory with sequence length n since Query x K Key" has n? entries.
DESIGN METHODOLOGY
Accuracy
Downstream
Optimization
Pre-trained
Creating TransElement
Constraint,
Metric,
Transformer,
Qfencompessing.
element gramufortt
Speed Focus:
Size Focus:
Lower loyers more important: by
Optimization
T
Per.....
Top fayers more importont:
D(enc Joment,granularitn haannlllrttt.pppplll. ppopl)
enc element: None granularity
Create TransElementuueuss Q
trialllement
Framework
altone.0
Block
(T:: Exccution time, P. of parameters, &,i Layer)
Block
(T,
2
Block
(To Par Ga
Significance Analysis
trialElement
W
H, d Ho..
Model, Loss Finetune(T,,
O.deletettrialEee ment,il;
1/
FFN Block Weight
ATTN Block
WG
WG N/ Head N
AxThresh (A.F
enc. element:
Groun
None
[W]: Starting indeo Weight
[W,
H..
PruneThresh (A,f)<
Group, Hj: Starting Index Head
Loss AuThresh (A.F)
AW, S.W,
Prume Thresh(AA)
Granularity
clement:
granularity
ATTN Block Weight
AWG
AWG
trialElement gronularit>>
maxGronularity
(AIN): Starting Index Weight
(AW,)
(AW))
Prune trialElement; ;
maxGranularity
Group]
modifieeTrialElementt NULL;
Head Queue
Approximate trlaiElement:
modifiedTrielE TrifiedddiilEEeeeent Approximated trialElement;
Sign- Matching based Appraximate Attention
Query Matris
Key Matrix
Hamming
Compute
0.5
granularity:
enc
05
0.4
2.5
0.7
-16
1.8
trialElement modiffedTrralllemenn
IQ emppy(l
15
0.8
1.6
0.2
11
1.7
Q.mmpty()
Build sign representation Query vectors
Select vectors with smallest H.D
FinalModel, FinalLoss Finetune(T)
0.4
Figure 2: Illustration of the Transformer Approximation Methodology. Sign Matching is illus- trated with K=1
We propose a framework for producing fine-tuned Transformer models that are optimized for specific metric (speed, model size, accuracy). Fig. presentts overview of the proposed frame- work. As shown in the figure, the inputs to the framework are pre-trained Transformer model, the fine-tuning dataset, the goal of optimization (speed, size accuracy) and acceptable accuracy loss (when optimizing for speed or size). The framework has three major components: (i) a set of heuristics used to build ordered queue of elements (TransEElmments) significance analysis method to identify insignificant elements in a pre-trained Transformer and (iii) set of techniques prune or approximate the insignificant elements. The framework proceeds in an iterative manner. That is, we first start with the original Transformer. We then remove an element from the TransElements queue, analyze its significance, and apply pruning/approximatior techniques to the element. This results in new Transformer, where the element is replaced by the pruned or approximated version. This modified Transformer is then used as the baseline for the next iteration. After processing all of the identified elements, we fine-tune on the downstream task for the same number of epochs as the baseline model to obtain the final, optimized model. A de- tailed description of our methodology for approximating Transformers is presented in Fig. 2 and in Algorithm 4 In the following subsections, we further describe our techniques for generating the ordered queue TransElements, followed by the significance analysis method, and finally the pruning and approximation techniques for different Transformer elements.
be considered, (ii)
Trannelement Ordered Queue. In order to optimize given model, would ideally want to char- acterize the significance of each and every parameter the model, rank them in order of importance, and finally prune/approximate only the least significant parameters, in Molchanov et al.) (2017)
4

---

Under review as a conference paper at ICLR 2021
However, Transformers have billions of parameters, making this process computationally infeasi- ble. In addition, previously proposed techniques that can efficiently estimate the importance of each parameter, such using Taylor expansion expansion, are not applicable. This is because the (approximate, fine-tune, approximate) cycle does not work for Transformers during fine-tuning, since they very quickly overfit the training data for the downstream task (usually within 5 epochs). We take advan- tage of hierarchical structure of Transformers and consider them in hierarchical manner, ordered by increasing granularity. Specifically, we place entire FFN and ATTN blocks earlier in the queue, followed by heads, and finally weight groups. Through this ordering, we are able to quickly elim- inate large numbers of parameters from further consideration, speeding up future iterations of the framework. For example, eliminating a single FFN block in the BERT-Bass model removes 5.6% of all parameters under consideration. To further reduce the number of elements under consideration, we also dynamically remove elements from the queue if they are encompassed by a high-importance block. For example, if a given ATTN block is determined to be of high importance, we remove all heads and weight groups within that block from the TransElement queue.
Since the framework iterates through the entries of the TransElement queue sequentially, its efficacy is dependent the ordering of the elements at each level of gramularity. In order to minimize the run-time of the framework, we provide two additional heuristics to guide the ordering of elements. First, we use the unique linguistic properties captured by the different Transformer layers (Jawahar) eet al.] (2019). These properties depend on both the Transformer and the downstream task under consideration, since different tasks require different types of linguistic knowledge. For example, top layers usually have low significance for Language Understanding tasks, since long-range depen- dency information is not required for most tasks (for example, sentiment analysis requires only local context). Hence, we place the final layer at the front of the queue, and work our way backwards to- wards the first layer, since blocks in the final layers are more likely to be removed, thereby speeding up future iterations. Second, we use run-time (or parameter-count) aware ordering of the TransB- lements, such the most time consuming blocks (or blocks with the most parameters) are likely to be removed carlier in the algorithm. For example, at large context lengths, we start with the ATTN blocks in all layers before moving on to the FFN blocks, and vice-versa at small context lengths. This has the dual benefit of producing highly optimized models for inference, as well speeding up the significance analysis process by eliminating time-consuming blocks early and making further iterations faster. Algorithm I and Fig. [ describe the process of creating the TransElement Queue. The utility of this framework and the heuristics used are discussed in Appendix C
Significance Analysis. To determine the significance of each Transformer element, we first fine- tune the original Transformer model for the given downstream task to obtain the baseline loss. We then use this baseline loss, along with the provided acceptable accuracy degradation, to generate a set of loss thresholds that determine whether given element is of low importance and therefore can be pruned/aprroximated. This a one-time step and performed globally for all elements in the TransElements queue. Then, for the element under consideration in each iteration of the framework, we compute the loss of the current Transformer model with the element removed. We then com- pare this loss to the thresholds determined above. The exact thresholds used are dependent on the optimization metric: speed, size, or accuracy. If we are optimizing the network for speed or size, we prune the element under consideration if the training/validation loss upon removing it from the Transformer is less than the pruning threshold. If we are optimizing for accuracy, we prune the ele- ment only if the training/validation far during the optimization process, since the goal is to find a model with minimum loss. Similarly, we apply approximations if the loss with the element removed from the Transformer is greater than the pruning threshold but lower than the approximation threshold. Algorithm 2 describes Signifi- cance Analysis.
loss when it is removed is less than the minimum loss seen thus
Pruning and Approximating. As evident from Section 3. the structure and functionality of ATTN blocks differ significantly from that of FFN blocks in Transformer. We accordingly adopt different strategies for approximating them, as described below. But pruning an entire ATTN or FFN block is effectively the same as it simply involves using the skip connection to bypass that block. The pruning strategies for the FFN and ATTN blocks are illustrated in Fig. 4 and Fig. 5
Pruning Weight Groups within approximable FFN Blocks. Consider an approximable FFN block that performs the transformation Rxxd x Øaxy - Raxy, with weight groups defined along the d dimension ((/W)) weight groups of (W) weights each, where W is a hyperparameter that defines the granularity of approximations). When optimizing models for accuracy, we remove weight groups
5

---

Under review as a conference paper at ICLR 2021
only if doing so results in reduction in the model loss. When optimizing for size, we remove weight groups that maintain loss within the pruning threshold when removed. When optimizing for speed, however, removing weight groups with low significance from arbitrary locations does not help, since it introduces unstructured sparsity in the weight matrix that can be difficult to exploit to achieve speedups. Instead, we impose structure algorithm that finds the largest number of weight groups that can be removed while maintaining loss below the threshold, such that the weight groups that remain in the model form a contiguous block. We first start from the bottom (weight group 0), work our way up and remove as many weight groups as possible while staying within the loss threshold. We then start from the top (weight group d/W), work our way down and remove many weight groups as possible while staying within the loss threshold. When this process is completed, the weight groups that remain form contiguous dense block, enabling speedups on all hardware platforms. Since weight groups are removed along the "hidden" dimension d, our methods do not change the shape of the output of this block; instead, are simply "shrinking" the effective hidden dimension size through structured pruning.
our pruning. Specifically, we use a "greedy shrinking"
Quantizing Weight Groups within approximable FFN and ATTN Blocks. When optimizing the Transformer for size, we also quantize weight values within weight groups for which the loss lies between the pruning and approximation thresholds. We use uniform quantization with Quantization- Aware Training proposed in Q8BERT (Zafrir al. (2019) within our hierarchical framework to quantize insignificant weight groups to lower precisions. This reduces the memory requirements of those weight groups but does not improve the execution time as the computations are still performed at the baseline precision.
Pruning ATTN heads and Weight Groups within approximable ATTN Blocks. We divide the multi-headed self-attention operation into two main steps. In the first step, we compute the Query, Key and Value matrices by multiplying the input to this layer with the corresponding weight matrices (Raxd x R + R=**), and then reshape them into multiple attention heads (RR*y + (n Our approach to pruning this step is exactly the same as for the FFN blocks, where we iteratively prune weight groups along the dimension using our shrinking algo- rithm. In the second step, we compute the "attention output" as Y = softmar( (Query x Key") )+ attention_mask x Value. To optimize this step, we apply two techniques. Firstly, we identify insignificant attention heads, and prune them from the model. However, removing attention heads changes the shape of the output of this layer. We overcome this by keeping track of the pruned heads, and padding the output with zeros in the corresponding locations. In spite of this overhead, we still manage to achieve significant speedup from this approximation technique since pruning heads makes multiple downstream operations (computing the attention scores, applying softmax to the attention scores, and computing the final score) considerably faster. Therefore, we do not use our greedy shrinking method, but rather rely unstructured pruning it allows for greater pruning which further benefits the downstream operations. Secondly, we dynamically reduce the size of the key and value matrices by pruning weight groups from the same location along the n dimension in both matrices, based on sign matches with the query vectors. This again makes multiple downstream operations considerably faster and does not change the shape of the output of the pruned block.
Approximating self-attention within approximable ATTN Blocks. We observe that the "atten- tion scores" matrix is highly sparse, especially after the softmax operation. This sparsity implies that most of the dot products between the query and the key are unnecessary. Thus, we would ideally like to perform the attention operations for the query vectors that give highest dot-product with each key vector efficiently without explicitly performing all of the dot products. To this end, we propose replacing the Onn²) dot product-based attention mechanism with a linear-time nign-matching-bae mechanism in approximable ATTN blocks. Sign-matching attention (SM) is based on the idea that key vectors whose signs match with the largest number of query vectors will have high dot-products with maximum number of query vectors. However, it is expensive to compute sign match for all pairs of query-key vectors, as this will grow quadratically. Instead, we employ a low-cost approx- imation. For each column of the query matrix, we identify if more number of vectors will have positive or negative number in that column. This becomes the representative sign in that column for all the query vectors. Each key vector is then scored by how well the sign of each of its elements matches with the sign of the representative query vector by computing the Hamming distance be- tween the two sign vectors. This score is used to select the top K key vectors. As a result, we reduce the number of computations required to score each key vector (and the overall complexity) from O(n²) to O(n). Sign matching is illustrated in Fig. and explained in detail in Appendix B As this

---

Under review as a conference paper at ICLR 2021
approximation does not increase the accuracy of the models nor decrease the number of parameters, they are only applied when optimizing the fine-tuned models for speed.
5 EXPERIMENTS AND RESULTS
We implement our techniques within Huggingface's 's Transformers library in PyTorch (Wolf et al. (2019). ). We use Intel AT's NLP Architect for experiments on Q8BERT. The experiments were per- formed on a GeForce RTX 2080 Ti GPU with 11GB memory. All results reported are the average of 10 runs with random seeds after 3 epochs of fine-tuning on the dev set, unless otherwise specified. When reporting results on the development set, for the larger datasets (like MNLI), we create a vali- dation set using a small subset 15%) of the training data. We use the loss on this set to characterize the significance, On the smaller datasets (like WNLD), there isn't 't enough data to get a meaningful training- validation split. Hence, we directly use the loss on the training set. When reporting results on the test set, we use loss on the development set to characterize significance. Detailed descriptions of the tasks and Transformers used in our experiments given in Appendix E.
trininn-validdaioon
Additional results on
the GLUE test set are presented in Appendix F
Primary Results. We
present results on GLUE
CoLA
MNUI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
WNL
Average
(Wang et al.] (2019) in
XLNet
Table I SQUADvI.1 (Ra-
Baseline
55.2
86.76
87.05
90.12
90.78
54.42
93.01
87.69
56.34
77.93
jpurkar et al. (2016)). and
Accuracy Focus
57.89
86.97
87.61
90.18
91.11
59.88
93.62
87.98
56.48
79.08
Penn Treebank (Marcus)
Speed Focus
54.91
86.54
86.87
89.79
90.65
54.15
93.38
87.52
56.33
77.79
jet al.] (1994)) in Table 2
(Speedup)
[3.81X]
(1.35X]
(1.63K)
(1.35X)
[2.01X]
[3.11X]
(1.68X)
(1.9%)
[4.22X)
[2.34K]
Focus
54.93
86.49
89:91
90.68
54.32
93.46
87.51
56.32
77.83
for speed, we aim to reduce inference time as much as possible while main- taining at least 99.5% of baseline accuracy. While the focus in these experi- ments is on speed, we find that our framework still leads to models that are 1. 1.29x-10. 65xx 65x smaller due to TransElements being dropped from the pre-trained model. On the other hand, when optimizing for size, we focus on reducing the model size as much as possible while maintaining the <0.5% accuracy degra- dation constraint. We use uniform quantization with Guantization-Awarr Train- ing proposed in Q8BERT (Zafrir et al.] (2019) within our hierarchical framework quantize insignificant blocks, heads and weight groups to lower precisions. This leads to models that are smaller than and at least as fast as uniform 8-bit integer quantized model such as QSBERT (Table []. Our results are competitive with QBERT (Shen et al. 2020),, while maintaining the advantages of uniform 8-bit quantization over the group-wise quantization proposed in QBERT. The compression is lowest for AIBERT since its parameters are shared across layers, and most of the compression is from quantization. While the focus these experiments is on size, we find that our framework still leads to models that are 1. 07 x-3 26x faster due to elements being dropped from the pre-trained model, with potential for much greater speedups on optimized 8-bit integer kernels. When optimizing for accuracy, the goal is to maximize the accuracy of the pre-trained Transformer model for any given downstream task. While the focus in these experiments is on accuracy, we find that our framework still leads to models that are 1. 28x-9. 83x smaller and 1. x-x-2. 94% x faster due to TransElements being dropped from the pre-trained model.
When optimizing
(Compression)
[14.06X]
(5.89X)
[6.04X]
(5.98K)
(9.5X)
[9.96X]
(9:1X)
(4.88K)
[14.98X)
[8.85X]
QSBERT
Baseline
52.85
83.73
85.05
91.19
90.65
64.26
92.32
88.09
166.1
78.25
Accuracy Focus
55.45
83.92
85.89
91.26
91.19
64.8
92.76
88.97
56.72
78.99
Speed Focus
53.91
90.79
63.91
87.88
56.34
78.23
(Speedup)
(1.45X)
[1.43X]
[1.33X)
(1.4X)
[1.66X]
[1.41X]
(1.43X)
(1.51X]
[2.85X]
[1.61X]
Size Focus
53.57
83.55
85.03
90.91
90.48
63.97
92.42
87.94
56.28
78.24
(Compression)
[13.84X
(5.64X)
[5.56K)
(5.52X)
(8.44X)
(10.48X]
(8.12X)
(5.52x)
(14.08X)
(8.6K)
DistilBERT
Baseline
50.25
82.04
84.07
88.72
89,92
61.01
90.48
86.49
56.33
76.59
Accuracy Focus
52.12
82.28
84.84
88.91
94.22
62.14
90.6
86.94
56.82
77.39
Speed Focus
50.06
61.84
86.83
89.86
61.73
90.25
86.34
56.33
76.57
(Speedup)
(1.46X)
[1.34X]
[1.34X]
(1.34X)
[1.46X]
[1.44X]
(1.29X)
(1.43X)
[1.54X]
[1.41X]
Size Focus
50.14
81.82
84.03
88.79
89.88
61.58
90.29
86.31
56.33
76.57
(Compression)
[700k)
(4.62x)
(4.66x)
[4.59x]
[6.12x)
(6.02x)
(4.71x)
[6.36x)
(7.12x)
(5.6x)
BERT
Baseline
53.21
84.43
86.11
90.68
91.06
64.25
93.23
88.13
56.11
78.58
Accuracy Focus
55.45
84.69
88.89
90.83
91.77
64.8
94.06
89.01
56.82
79.59
Speed Focus
53.91
84.28
90.41
90.87
64.32
93.04
87.99
56.33
78.57
(Speedup)
[2.92k]
[1.29x)
(2.21x)
[1.28k]
[1.84x]
[3.29k)
[1.33x]
[1.8kk]
(3.51x)
(2.17k)
Size Focus
53.91
84.25
85.05
90.32
90.81
64.32
92.99
88.02
56.33
78.56
(Compression)
(13.16x
(5.19k)
(6.61x)
[5.22k]
(8.36x)
[10.88x]
(8.01x)
(5. (5.08k]
(14.08x)
(8.5x)
Table 1: Results on GLUE. We report Matthews correlation for CoLA, Pearson Correlation for STS-B and accuracy for all other tasks. We report only "matched" accuracy for MNLI.
7

---

Under review as a conference paper at ICLR 2021
Table 2: [Left] Results on SQUAD v1.1. We report the Exact Match score. The compression is lowest for AIBERT since parameters are shared across layers, and most of the compression is from quantization. [Right] Results on Penn Treebank. We report perplexity (lower is better).
Speed
Accuracy Focus
Size Focus
Network
Baseline
(Speedup)
Focus
(Compression)
Network
Baseline
Accuracy Focus
Speed Focus (Speedup)
80.66
80.71
(Compression)
80.97
(1.42X)
(1.28X)
12.08
12.09
81.26
81.24
BERT-Large
12.04
11.92
(2.47X)
(13.16X)
(1.39X)
[7.72X)
14.24
14.23
Q8BERT 80.28
80.14
80.11
GPT-2
14.21
14.08
(2.08X)
(7.64X)
(1.27X)
(6.61X)
Tuning the Approximation Knobs:
In this work, we considered tight accuracy constaint of <0.5% accu- racy degradation while optimizing the model, and determined the hy- perparameter values Threne'Thresh- old and Thproxlhreshold) empiri- cally for that constraint. How- ever, users of different applications and platforms may be willing relax the accuracy constraint for obtaining faster or smaller models. In view of this, we demonstrate the ability of our framework to operate at differ- ent points in the preed-size-accurae tradcoff curve (Fig. 3 through differ- ent values of hyperparameters. We note that directly using optimized pre-trained Transformers for infer- ence works best when there is a need
Average GLUE Accuracy
79
78
77
Size
76
Inference Time
0%
20%
40%
60%
80%
100%
Model Size Inference Time
BERT
Q8BERT
QABERT with Size Focus
Q8BERT with Speed Focus
QBBERT
Figure 3: Tuning the SA Approximation Knobs with Size and Speed Focus. The average GLUE scores across the tasks using Q8BERT are reported for different acceptable accuracy loss levels.
for significant speed/size improvement with negligible loss accuracy (<2%). or if there is a need for more accurate models. When significant degradation in accuracy (>3%) is acceptable, tech- niques that distil knowledge into simpler networks that no longer maintain the structure of Trans- formers may be more beneficial. Even in these situations, our techniques are still useful, since they serve as better teachers/baselines during distillation/archiieetuue search.
Comparison to previously proposed compression techniques: A majority of previous works for improving efficiency of Transformers directly pre-train efficient models from scratch. Using repre- sentative subset of these networks (covering the most commonly used techniques used to create effi- cient models), we demonstrate that our techniques are complementary, since these efficient networks still fine-tuned for different downstream tasks, providing opportunities for optimization. In addi- tion, we show that our techniques are also complementary to Q8BERT, a post-training quantization method. Poor Man's 's BERT (Sajjad et al. (2020)) evaluated several layer dropping strategies that do not require pre-training, and found top-layer dropping to produce least accuracy degradation across tasks. Comparing our framework to top-layer dropping, we observe greater speedups/compression at iso-accuracy across all tasks and networks, and the largest benefits are observed on Q8BERT, where the use of quantization greatly reduces the resilience of the network. making it unsuitable for drastic changes such as dropping entire layers. However, by approaching the problem of improving hierarchical manner and with finer granularity, we are able to exploit redun- dancies in the model missed by a layer-only strategy, achieving greater benefits without significant loss in accuracy. In fact, in our experiments, we observe that starting on the layers as a first step leads to worse models than starting with blocks. We find that the effect of removing an ATTN block of relatively high significance may be masked by removing the FFN block of very low significance in the same layer (and vice-versa), leading to low significance for the entire layer. This has sequences further along in the process, since removing a high-significance block greatly reduces further opportunities for pruning and approximating the model. For experiments with Layerdrop
inference efficiency
8

---

Under review as a conference paper at ICLR 2021
Fan al.
).20)).
experiment on RoBERTA (Luu et al. (2099) using fairseq (Ott et al. (2019)
pre- tre-trained with a layer drop rate 0.3, and then drop every other layer at fine-tuning time. For QBERT, we directly the results reported by the authors (Table
Table Comparison to previously proposed compression techniques. Results reported are aver- aged across the GLUE tasks, unless otherwise specified.
Network
Optimization Strategy
Accuracy
Speedup
Compression
Pruning/ Optimization Strategy
Network
Accuracy
Speedup
Compression
BERT-Base
None
78.58
1x
Integer
78.25
1x
Top Layer
Q8BERT
Quantization
Dropping
75.15
1.39x
Layers
Size Focus
78.24
1.19x
8.6x
dropped]
Knowledge
Poor Man's
Q8BERT
Distillation
BERT (applied to QBBERT)
Speed Focus
DistilBERT
Speed Focus
76.57
2.26x
3.2K
(iso acturacy point]
75.16
5.08x
8.68k
Focus
Q8BERT
Parameter
Size Focus
75.15
2.92x
10.94x
81.2
1.7x
9.09x
(ssoaacuraac point]
Factorized
AIBERT
Embeddings
Hesslan- based mixed
Speed Focus
81.03
2.35x
9.2k
Focus
precision Quantization
81.75
9.01x
Layer Dropping Layers
QBERT MNU
RoBERTa
92.81
1.89x
1.64x
Q8BERT + Size Focus (Iso- accuracy point]
LayerDrop
dropped)
81.71
2.12x
9.06x
MNLI
Speed Focus
82.79
2.33x
1.92x
Impact on fine-tuning time. Un- like the baseline models, frame- work requires multiple fine-tuning passes to optimize the model (Table 4). We minimize this overhead in ways. First, since iterative method potentially eliminates com- ponent in each pass and our order- ing of elements ensures that time- consuming components are elimi- nated early, each subsequent opti- mization fine-tuning pass takes less time. Second, for the optimization fine-tuning passes, we do not use the entire dataset for large datasets. Instead, we compute the thresholds based smaller subset of the target data. Specifically, randomly sample small subset of the training data (20%) to fine-tune the model, and a validation set (15% of the training set) to char- acterize significance. We find empirically that doing so results in the same elements getting pruned and approximated as when the entire training data is used. We further see that this subsampling robust across models; if the reduced dataset works for one model, works for all other models. Thus, by both greedily reducing the size of the model to be fine-tuned as well as reducing the amount of work performed each optimization fine-tuning pass, we can quickly explore the search space.
Fine Fine-tuning tuninu Passes Optimization
Optimization Time (minutes)
Final Fine- Total time (minutes)
Transformer
Time (minutes)
BERT-Base
MNLI
118
356
154
510
BERT-Base
WNLI
82
6.5
0.16
6.68
DistIBERT-
MNLI
93
187
Table 4: Optimization and fine fine-tuning time. The baseline final fine-tuning times are 189, 0.3 and 114 minutes respec- tively. For MNLI, each pass during optimization. For WNLI (which has only 635 training samples), we the entire training data.
only 15% of the training data for
CONCLUSION
We proposed approximate computing framework to optimize pre-trained Transformers. The framework identifies elements that are insignificant for the downstream task at hand, and applies techniques approximate these elements. We demonstrated that this framework can be adapted to produce models that are faster, smaller or more accurate, depending on the user's 's constraints. Us- ing this framework, produced models that up to 4.22x faster, up 14.08 x smaller (with less than 0.5% relative accuracy degradation) and up to 5.46% absolute points more accurate with simultaneous speed and size benefits.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Tom B. Brown, Benjamin Mann, and Nick Ryder et al. Language models are few-shot learners. CoRR, abs/2005. 11100005..111.,.. 2020.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/v 1423. URL
1423
Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In 8th International Conference on Learning Reprexentations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL SJG7KhVKPH
net ftrrm/ormm/// ?id-
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with struc- tured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpps://openeeviee:
hett/Eorun21dd=5y02225535
Saurabh Goyal, Anamitra Roy Choudhury, Venkatesan T. Chakaravarthy, Saurabh ManishRaje, Yo- gish Sabharwal, and Ashish Verma. Power-bert: Accelerating BERT inference for classification tasks. CoRR, abs/2001 108950, 2020. UtRlhttss://axxiy/
www.siiiivrrrooooooooo aab//aa//////aaaaaaa///r///raaar abs/ 2001. 08950
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. What does BERT learn about the structure of language? In Anna Korhonen, David R. Traum, and Lluís Marquez (eds.). Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 3651-3657. Association for Computational Linguisties, 2019, doi: 10. 10.18653/ URL htttss://co..rgg// wtt..o.................. p19-1356
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. CoRR, htss/999..10351 2019. URL httt:://aaatt..
org/ abs/ 1909 10351
Ashish Khetan and Zohar Karnin. schubert: Optimizing elements of BERT. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguiitiiss ACL 2020, Online, July 5-10, 2020, pp. 2807- www. ac wee...e.... org/
2818. Association for Computational Linguistics, 2020. htLhstp:::/
anthology/2000 /20ggg//000000 acl main. 2507]
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
net for.mmirrmm
vk
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htt.h:ttps//// //tpnr/ve///peeeeee.... net ftt.mmorummmmm H1eA7ABEVa
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CORR, hbs/1907.11692 2019. URL //axxii wwwpsiiwiorrrroooooooo.o abs 1907. 11692
Mitchell P. Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In Human Language Technology. Proceedings of a Workshop held at Plainsboro. New Jerey, USA, March 8- 11, 1994, Morgan Kaufmann, 1994. URL https: / www. aclweb. org/ enthology/HBt 1020/
10

---

Under review as a conference paper at ICLR 2021
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 14014-14024, 2019. URL http:h/tppppppeeeeeee nips nips.cc/ saper/9551-azee sixteen---eeee heads really better taaetter-hhante -hen-one
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. http:t http:///ppeeeeeieee /ttphttps////eeeeei.ee.. wttp:/ewweeeeetoommmmm/aeaa....e
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-IILT 2019: Demonstrations, 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683, 2019. http::tttp:::ttt www.aawwwwww...aaaaoooaaaaoaaaaa wtw.b///19...11..111
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 2383-2392. The Association for Computational Linguisties, 2016. doi: 10.18653/v1/d16-1110 1264. d16 -2264
Hassan Sajjad. Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's bert: Smaller and faster transformer models, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. URL http://
arxiv. abs//9/s/11900 01108
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Q-BERT: hessian based ultra low precision quantization of BERT. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligenee, EAAl 2020, New York, NY, USA, February 7-12, AAAI Press, 2020. URL https: //t::///aaaaa wtt. .ptpd/
2020, pp. A 6409
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro, Megatron-Im: Training multi- parameter language models using model par- 2019. URL http::ttt//// / /ttp://aaxvvvvv.. org. abs /109 08053
allelism. CoRR,
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact fask-agnostic BERT for resource-limited devices. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 2158-22888 2170. Associa- tion for Computational Linguistics, 2020.
/tt:://wwwwww http://wwwwwww...cc/.. wwwp://wwww.../..oo.ooooo//
2020.. socl-main... 1957
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL btttp:://ppenreeeeeeeee
net/ ffo.umiiddd
rJ4km2R5t7
11

---

Under review as a conference paper at ICLR 2021
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface transformers: aft natural language processing. CoRR, abs/1910.03771, 2019. URL
http:///rxiv.orre
/1lp0///91..00/ 003771
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httts:://ppenreviitt
net/forumt By= НXPH
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating BERT inference. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pp. 2246 2251. Association for Computational Linguistics, www. ac web. org/ htt.rr/gaannoolo///////o/// acl ac ma ii. 2047
2020. URL SttLhhtps::///
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing BERT by progressive module replacing. replacigg,, CoRR, abs/2002.02925, 2020. URL httpp://arxxv.
arg/abs/2002 0292.5
Zihao Ye. Qipeng Guo, Quan Gan, Xipeng Qiu, and Zheng Zhang. Bp-transformer: Modelling long- range context via binary partitioning. CoRR, 0os//1911.04071 2019. URL http:
org/abs/191 1911. 04070
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8BERT: quantized 8bit BERT. CoRR, abs/1910.061888 2019. httttttpp///xxi/ http:p///wxxxx..oorra////////010
12

---

Under review as a conference paper at ICLR 2021
A DETAILED DESCRIPTION OF THE OPTIMIZATION FRAMEWORK
A.1 ALGORITHMS AND ILLUSTRATION OF PRUNING STRATEGIES
Algorithm 1: Creating TransElement Queue
Input: Pre-trained Transformer T, Focus of optimization F, Downstream task D
Output: TransElement Queue Q, containing ordered elements of T for Significance Analysis empty -ueuue(
FFN Blocks are more time-cone httrr--oocnnnnngnanneeeeee-eeeee
then
Knowledge bottom layers is more important then
for layer to romeeoyers do
Qgrramularty=0))puss(( F N blocklayerr
Qg bllokkllayert www.Wettg...rpppp.
for layer = to reum_uyyers
Qgraamlariy==0)ppshhA/7777 _llock layer)
Qgraaularttyy||ppusSAAAAYY _boock layer CiatentionHHadeee
Qgrranlarriy=2.ppssS(A1777 block layer]
W Weight Groups)
else if Knowledge in top layers is more important then
for layer = num loyers do
Olgranularty==).ppss((PFFF block (loce))
Qgraanlarit==IlppussFFPyYY F block [layer] Weight Groups)
for layer = num to / do
O|granllariy=OO]psss(((T7T block layer!
Qggramlarity=11lppshhAAAYY ATTTN block layer Allention Heads)
[ggraulariiy=2)pposh((47771 block layer]
W eight,Groups)
else if ATTN Blocks are more tinee-coosniinnguuuueeerr-mntntt then
if Knowledge in bottom layers is more important then
for layer to numm .laye do
(yyranularit=O00,ppuh(AYY7 Nblock layer!)
Oggrauularty==I.pushhAA7777 block layer Attention_HHeae
Qgrraudlriyy=2l.pphhh/77777X blook layer
W wiight
for layer= to num nwm_layerss do
Qgraau/aarty==O)ppuhhFFFFFF block(layer)
Oggramlarriy=II.oss((FFFF block [layer] W (rooups
else if Knowledge top layers is more important then
for layer num_loyers to / do
Aggranulartt=OO)pushhh ATTN N block layer))
ggramulariy==1,ppss(A77/)
_block layer Eirenttio_HHaadee
Algaauulrrtss22jppss((AAAA _bock [ayer]
W Weight.Groupp
for layer = numeee 1 do
Olgranlarriy=O))ppshh(EEE (locer
[ggram/arrty==|/pssh(PFFVV NoloNNooookappa (lock(layeri] S.rtayrr..Wee. Weiih)
return Q
13

---

Under review as a conference paper at ICLR 2021
Algorithm/ 2: Significance Analysis
Input: Current state of the Transformer model T leie--tuning-datee (or its reduced subset)
D. TrintElement E. Thresholds (Pruning Threshold, Approximation Threshold}
Output: Action be performed on (whether to prune E, approximate E, retain E as-is)
action = NULL
T1 T
T TransElence Loss = Jume(TI.D if ElennnileenssLsss < Pruning.Threshhacn Threshold then
L action = "Prune"
else if Errannteemett..l. Loss >= Truming-Thresheee and
ElansElement_Lie < ApproacimationTTrreshool then
action = "Approximate"
return action
Algorithm 3: Transformer Optimization
Input: Pre-trained Transformer T Frne-tuninggddttset D (and its reduced subset D' for large datasets, else DD=D), Focus of Optimization F, Acceptable Accuracy Loss A Output: Optimized and fine-tuned Transformer for the given task
T. Baseline Loss Fine-tune(T.D')
Pruning Threshold, Approrimation Threshold=
Compute_Thrrssoolds Baseline Loss, F, A)
Q rae Llerrnnteeeemmeetuuuuueeeeeee
granularity =0
while Q is not empty do TrialElement = Q[granularity.ppop(
action =
SignfficanceAnnahssssII..IT. D'irrealllemnnt
CntrooiimationTThreshh
if action "Prune" then
modi p TranddTrreeeemmee = None
Q-pop(All elements encompassed by TrialElement)
else if action = "Approximate" " then
if granularity max grw.ularity then
if Focus "Accuracy" then fodd Element trialElement
else if Focus "Size" then modifies frn_ Element quantiize_onceettiillllleeeee
else if Focus "Speed then
TrialElement is encompassed by a FFN block then modi f TransElement = trialElement
else if TrialElement encompassed by an ATTN block then
modi Teane Trane Sign.MMathhinngttaaaEeeeeel
else
modifiee Element = TrialElement
else
Q-pop(Al elements encompassed by TrialElement) modi fied TransTaanseeeeee = TrialElement
Ol granularity] empty then
I granularity++
T TrialElement + modi Tran
T, Final Loss Fine JunelT, D)
return T
14

---

Under review as a conference paper at ICLR 2021
Baseline
Size Accuracy Focus
Speed Focus
D'
W
D
A
W
D
N
W
Figure 4: Illustration of pruning techniques used in FFN Blocks. 'A' denotes activations and "W" denotes weights, transposed for clarity. When optimizing for accuracylsize, unstructured pruning used. When optimizing for speed, our greedy shrinking method is used to prune weight groups, so that the remaining weight groups form contiguous dense block.
Baseline
Accuracy
Speed Focus
D
W Wowky
W puoy
Q/K/V
W//h/V
Y
Q/K/V
A'
W
Q/K/V
W. pxoy
Q/K/V
Q/K/
N
N
Q/K/V
Q/K/
Q/K/
N
Q/K/V
Q/K/ V
Q/K/
Y/h
v/h
Y/h
Sign Matching
Softmax
XN
Y=
Softmax
v/h
Y/h
w/h
1/h
Y/h y/h y/h
v/h
V
K
Y/h
Y/h
Softmax
Y/h
Y/h
Figure 5: Illustration of pruning techniques used in ATTN Blocks. "A' denotes activations and "W" denotes weights, transposed for clarity. Pruning heads in the second step is illustrated this way for the sake of clarity. In the actual implementation, heads are pruned by pruning the corresponding weight groups along the y dimension of Wo K/V in step 1. Greedy shrinking is not used when optimizing for speed for pruning heads.
15

---

Under review as a conference paper at ICLR 2021
B SIGN MATCHING DETAILED DESCRIPTION AND ABLATION STUDIES
B.1 ALGORITHM
Algorithm 4: Sign Matching
Input:
Set of query vectors Query [q 92, .2. gn set key vectors Key = [k1, h2 ..., kn.,
set of value vectors Value = [U1, u2,..., E,J, number of key vectors to select K
Output: Set of key vectors with highest sign match with query vectors Key" [K] k2g ..., kg
set of corresponding value vectors Value'
/* Build sign representation of Query vectors
i 661 1
count +- 0
while <= do j +
while j <= do
43.3 then
count[i] count [+] +
j j+1
count
| >=
(n/2) then
L vall
1
else
L
[al[i] + --1
i+1
Compute sign matches of Key vectors with the representative Query vector
i
malches +- 0
while do
Distli +
val)
Lititl
matches
(sort.ascenddnnggg (ist))
matches + matches[ [1: K]
K" + gather (matches)
V + gather(V (matches))
B.2 SIGN MATCHING IN AUTO-REGRESSIVE MODELS
In Auto-regressive models (XL-Net, GPT-2, Transformer-Xl. etc.), tokens are not allowed attend to tokens in the future, and an attention mask is applied to set the corresponding weights to large negative value. This is problem because the key vectors selected by SM may be such that vectors at the start of the sequence (first few query vectors) may not be able to attend to any of the key vectors (i.e.. their attention outputs will be 0), leading to significant loss of information and degradation in accuracy. We avoid this by selecting the top-scoring (K/4) vectors from the top quarter of the key matrix, and the top-scoring (3K/4) vectors from the overall key matrix and not included in the (K/4) vectors initially selected, instead of directly selecting top K vectors from overall key matrix. This reduces the probability of vectors having no vectors from their past to attend to.
B.3 COMPARISON TO OTHER DYNAMIC KEY-SELECTION TECHNIQUES
We compare our Sign-Matching based attention with other intuitive dynamic Key-selection that do not require training from scratch and provide speedups at small context lengths. We find that Sign Matching provides the best trade-off between accuracy and speed. The techniques considered described below:
Norm-based Selection (NBS). NBS is based on the idea that the "most important" key vectors (corresponding the "most important" tokens in the dataset) will have highest norm, and hence
16

---

Under review as a conference paper at ICLR 2021
highest attention (dot-product) with the query vectors. The key vectors ranked in descending order of their norm, and the top K vectors are selected. Attention is then computed only between the query vectors and the selected key vectors.
Value-based Selection (VBS). One of the disadvantages of NBS the fact that a vector with only very large value will have high norm, but it is unlikely to produce high dot-produets with other vectors. VBS overcomes this by using distribution of values in a vector, rather than the norm, as the selection criteria. In particular, specified "value threshold". We then select the K vectors with maximum number of elements with absolute values greater than the "value threshold".
count the number of elements each vector greater than
Grouped Attention (GA). GA places vectors that are likely to have high dot-product with each other the same group, and vectors likely have low dot-product with each other in different groups with high probability. The concept of GA was previously explored in Reformer (Kitaev) al. (2020), where Locality Sensitive Hashing was used to group vectors. However, since we apply this approximation only to resilient layers, we a simpler and faster grouping criterion, the position of maximum and minimum value. addition, there is no need for multiple iterations of grouping and computing attention scores that was used in Reformer to ensure that query-key pairs with high attention scores were placed in the same group in at least one of the iterations. Both of these factors together greatly reduce overheads, enabling speedups even at small context lengths. Our grouping criteria is hased on the intuition that vectors that have highest positive/negative values in the same position will have high dot- dot-products with each other. Attention scores then computed only between query-key pairs in the same group, since they are most likely to have high dot-products with each other. We limit the number key vectors in each group to K vectors with the highest absolute value in that position, and therefore GA scales linearly in time and memory with context length instead of quadratically.
B.4 SPEEDUP WITH INCREASING CONTEXT LENGTH
Since Sign Matching is linear-time approximation of the quadratic self-attention operation, speedup increases significantly with increase context length. As context length increases, ATTN Blocks become time-dominant, and hence work. In addition, the memory requirements increase quadratically with context length due to the self-attention operation, making Transformers extremely semooryhootllenecke.. Sign Matching helps alleviate this bottleneck. Through the combination of these factors, we find a large increase in speedup as context length increases from our Sign Matching technique (Table 5).
emphasis is placed on these blocks by frame-
Table 5: [Left] Comparing Sign-Matching with other dynamic Key-selection algorithms. Re- duction in accuracy and time are reported on MRPC using BERT-Base (context length of 128). They are measured the difference in accuracy and inference time when all approximable ATTN Blocks are replaced with approximate versions of the self-attention operation. [Right] Speedup from Sign Matching at increased context lengths.
Reduction in time
Speedup Context Length
Approximation
Reduction in Accuracy
Norm-Based
0.22%
31%
X
4
Selection
Speedup
Value-Based
0.14%
32%
2
Selection
Sign Matching
0.06%
32%
Grouped
128
512
1024
4096
Attention 0.05%
6%
Context Length
17

---

Under review as a conference paper at ICLR 2021
C ANALYSIS OF THE OPTIMIZATION FRAMEWORK
C.I PREVIOUSLY PROPOSED METHODS FOR ESTIMATING IMPORTANCE OF ELEMENTS ARE NOT APPLICABLE
Due to the enormous size of Transformer models, brute-force approaches to estimating impor- tance are not feasible, addition, previously proposed techniques for efficient importance estimation not well-suited due to the fact that Transformers cannot be re- peatedly fine-tuned recover the accuracy losses from approximating the model, since they very quickly overfit the training data for the downstream tasks (usually within epochs). Therefore. Taylor expan- sion, which uses gradient of loss to estimate importance, is not reliable, as evidenced in Table 6 We observe that in addition to providing greater control over the accuracy of the final model (and the ability increase accuracy), our framework also provides better speedup and compression at similar accuracy.
Model Speeduu
Model Compression
Method
Accuracy
Baseline
84.06
1x
1x
Taylor Expansion (negalive gradient of loss)
80.65
2.64x
4.53x
Ours (lso-Accuracy point)
80.66
3.28x
6.06x
Taylor Expansion (emallest absolute
80.69
2.51x
4.48x
gradient of loss)
Ours ((so-Accuracy point)
80.7
3.3x
6.09x
Ours (Accuracy Focus)
84.63
1.14x
1.98x
Table 6: Comparison of our method of estimating impor- tance with previously proposed methods. We show results on MRPC using DistilBERT with Accuracy Focus.
C.2 EVALUATION OF HEURISTICS USED
We compare different possible heuristics the ones used in our framework (Table on MRPC using DistiBBERT. When we remove the element with the lowest loss in each iteration (with loss characterized using our method), there negligible change in the quality of the final model pro- duced, but the fme-tuning-oopiiizzaii process is an order of magnitude slower if elements are still considered in the order of coarser to finer granularity, and two orders of magnitude slower otherwise compared to our approach. If the loss is characterized using Taylor expansion, it greatly destabi- lizes the system, leading to models that do not meet the accuracy constraints. To drive home the fact that our greedy approach combined with a global error bound does not lead to inferior models, we use an adaptive loss threshold. In particular, we use a very tight constraint when analyzing el- ements coarse granularities, and relax the constraint again find that there is negligible change in the quality of the final model produced, but the fine- uutinntoptimization process is significantly slower. We hypothesize that single global error bound is sufficient because we order the elements in such way that for the given task at hand, we intu- itively expect that the elements at the head of the queue are likely to be removed using the linguistic knowledge in different layers. Therefore, it is reasonable to expect that if an element at the head of the queue identified by our framework as prunable, can pruned without using up a large portion of our error budget.
move towards finer granularities. We
C.3
GAINS FROM DIFFERENT OPTIMIZATION TECHNIQUES
The gains obtained from different optimization techniques for different tasks and models depends on two factors: Number of elements to which each technique has been applied, and the gain from applying each technique to single element. In general, we observe that the largest gains are ob- This means
tained from pruning entire blocks that are most consuming/////// parameter-intensin that at small context lengths (such as BERT-Bas Base on MRPC in Fig. 6). pruning entire FFN Blocks produces maximum gain. And at large context lengths (such as GPT-2 Base on Penn Treebank in Fig. (6). pruning entire ATTN Blocks provides maximum gain. Our analysis also demonstrates that all techniques are vital for producing highly optimized models, since no single strategy can provide drastic gains with minimal accuracy degradation.
18

---

Under review as a conference paper at ICLR 2021
Table 7: Comparison of different heuristics. denotes strategy where it examines all elements (of the same granularity), and then prunes the element causing lowest loss. If marked as Taylor, Taylor expansion used estimate importance with single pass of the validation set. Otherwise, importance is estimated by our method with multiple fine-tuning passes before an element is pruned. 2 denotes greedy-strateeyy where if an element is found maintain aceuracy within the respective thresholds when removed, it is pruned/approximated right away. denotes hierarchical processing of elements. denotes non-hin non-hierarchical processing of elements (all elements are considered to be of same granularity for the purpose of importance estimation).
Optimized Model
Optimized
Optimized Model Speedup
Optimization
Heuristic
Model
Time (minutes)
Accuracy
Compression
Drop element with lowest loss in each iteration Taylor
83.22
2.64x
2.03x
61
Drap element with lowest loss in each iteration
84.05
1.95x
1,36x
1081
Random
84.06
1.92x
1.34x
66
Non-hierarchial (Min Loss) 1b
84.03
2,01x
1.37x
13822
Non-hierarchial (Greedy)20
84.03
2.01x
1.37x
3634
Adaptive 2a
84.05
1.95x
1.36x
143
TransElement Ordered Queue (Qurs)2a
84.06
1.92x
1.34x
39
Contribution to Speedup
BERT-Base MRPC (n=128)
BERT-Large Penn Treebank (n=512)
GPT-2 Base Penn Treebank
(n=1024)
Sign Matching
Pruning Weight Groups inside FFN Blocks
Pruning Heads/ Weight Groups inside ATTN Blocks
Pruning Entire FFN Blocks
Pruning Entire ATTN Blocks
Figure 6: Gains from different optimization techniques using Speed Focus.
19

---

Under review as a conference paper at ICLR 2021
D ANALYSIS OF IMPORTANT ELEMENTS ACROSS DOWNSTREAM TASKS AND TRANSFORMER MODELS
We study which blocks are pruned and approximated for different downstream tasks using dif- ferent Transformers (Fig. 7 We find that the differences in importance of blocks are more pro- nounced across different tasks than across different models. For example, for sentiment analy- sis, long-range dependency information is not very important. Hence, for all models fine-tuned for sentiment analysis, we observe that components later layers (closer the output) are more likely to be pruned. Across models, we only observe subtle differences. For example, we find that (Legessato-reegresiive) is able to learn important task task-specific information earlier than (ERT auto-regressive), similar to the observation made in (Sajjad al. (2020)) Hence, we are able drop more components (in earlier layers) in XLNet than in BERT, leading to more efficient models for inference. In DistilBERT (a distilled model), find that there is clear demarcation in linguistic knowledge across layers due to the reduced capacity the model. This is evidenced by the fact that components in the top four layers the boundaries are more soft in the original models. At a high level, these trends agree with pre- vious works on understanding how Transformers process language (Jawahar al. (2019) that use probing classifiers to discern the linguistic knowledge captured by the different layers. For example, on GLUE tasks, we expect that long-range dependency information not required for most tasks, since most of these tasks depend on local contexts. This is confirmed by the fact that blocks in layers are more likely be pruned/approximated than earlier layers using our framework. Similarly, we expect that this is not the case for I Language Modelling, since long-range dependency information vital to fully understand the text. This also confirmed by the observed trends using our frame- work. As future work, our framework can be combined with previously proposed techniques to gain deeper understanding of the working of Transformers, especially at finer levels of granularity.
never pruned across all Language Understanding tasks, while
Number of times
Sequence Classification
Short Question Answering
Language Modelling
40
ATTN Skp
30
10
12
5
6
10 11
12
2
6
10
(a)
Number of times
XUNet
BERT
DistiBBRT
16
16
10
11
5
11
12
Layer
[b]
Figure 7: Distribution of unimportant blocks across (a) downstream tasks and (b) Transform- ers.
20

---

Under review as a conference paper at ICLR 2021
E EXPERIMENT DETAILS AND HYPERPARAMETERS
E.1 DESCRIPTION OF TASKS AND TRANSFORMERS USED IN OUR EXPERIMENTS
Table 8: [Left]] Transformer models and [Right] downstream tasks used in our experiments
and studies.
Auto-
Layers
Parameters(M)
Regressive?
Optimizations
BERT-Base
12
110
None
BERT-Large
24
340
No
None
Uses factorized embeddings and cross-layer parameter sharing Faster and has less parameters
Task
Dataset
used
Context Length
AIBERT-Base
12
QBBERT-
General
Base,
Uses Knowledge Distillation the pre-training phase
Language Understanding
GLUE
DistilBERT- XLNET- Base
128
DistilBERT Base
Has half the number layers with <3% accuracy loss compared to
Question Answering
SQUAD v1.1
AIBERT-Base
384
Quantizes BERT to 8 8-bit integer weights and activations
512(BERT-
Uses Fale Quantication and Quantitation Aware Training in the fine toittunnngg phase.
Modelling
Penn Treebank
BERT-Large, GPT-2 Base
Large)/1024
(GPT-2)
QUBERT
12
4x sma ller then BERT with potential greater speedups optimized intä kernels with <1% accuracy
GPT-2 Base
12
117
Yes
None
E.2 HYPERPARAMETERS USED IN OUR EXPERIMENTS
Table 9: Hyperparameters used in our experiments.
Hyperparameter
Accuracy
Speed
Fine tuning Epochs (baselinne
3
3
Number used represent the TransElement
Fine-tuning Epochs lapprox)
TransElemnnt Loss
Learning
2,00E-05
2.00E-05
2,00E-05
Warmup steps
<1.005"baseeine
32 [fine-
(fine-tuning// (inference)
(fine-tuning)/ (inference)
1.005 "beseiinee 1..006"osseline_le loss
Batch size
tuningl 8 (inference)
(size one weight group in SA]
256
256
256
1.006 "baeeiin loss Lo.sellle loss
16, nc:128
elements selected
in Sign Matching)
N/A
64, 1128cnc1024
N/A
1.007"bseeline joseeineeeosss-- 1.008 baseebsee_eeee
128, ffn==1024
Skip, Threshoid (+0.5% 5%5 accuracy loss)
min_los Joss
1.005 1.005*baseire baasebbieeneea Joss 1.005 basseline_eoo Joss
1.008 1008@bbeelin__ii Josss-ee_oss--see
1.009"saseline, Joss
Approx, Threshoid (c0.5% accuracy
loss)
1i02*min,ih loss
100"baseline_ loss
1.02" baseline Joss
>1.009* b..09"bacellee Joss
los_ refers to the smallest loss seen during the Significance Analysis process. We use this as the threshold when optimizing for accuracy since the goal is to produce model with the smallest possible loss.
baseline loss refers to the loss when the pre-trained transformer used as-is, without any TransElements skipped or approximated. We use this to compute the threshold when opti- mizing for speed and size since small accuracy loss is acceptable.
Table & [Right] shows how we quantize insignificant TransElements lower precisions when operating with Size ApproxFocus.
21

---

Under review as a conference paper at ICLR 2021
F RESULTS ON THE GLUE TEST SET
While our main results are on the dev set following standard practice, we report results on the test also using the BERT (base) model in Table I0 We use the GLUE evaluation server obtain the scores, and make use of code from Xu et al. (2200) to prepare the data for submission.
Table 10: Results on GLUE Test Set. We report Matthews correlation for CoLA, Pearson Correla- tion for STS-B and accuracy for all other tasks, averaged across 10 random seeds. We report only "matched" accuracy for MNLI.
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST-2
STS-B
WNLI
Average
Baseline
50.38
84.37
82.11
89.74
88.76
61.65
94.94
82.8
58.19
76.99
Accuracy 51.21
84.59
83.15
89.8
89.44
63.82
83.17
65.14
78.37
Speed Focus (Speedup)
84.21
81.97
61.32
94.68
82.62
59.33
76.95
(2.81x)
(1.35x)
(2.12x)
(1.32x)
(1.98x)
(3.03x)
(1.37x)
(1.96x)
(3.49x)
(2.16x)
Size Focus
50.18
84.03
81.72
89.43
88.24
61.4
94.39
82.42
59.13
76.77
(Compression)
(13.13x)
(5.12x)
(6.28x)
(5.16x)
(8.08x)
(10.9x)
(5.58x)
(7.12x)
(14.14x)
(8.39x)
22


Reviews:
[{"review_id": "n2WSI51tDpQ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After reading the rebuttal, some of my concerns are addressed by the additional experiments. But I also agree with other reviewers that the result is not very surprising. As R4 mentioned, the proposed method depends on the a specific downstream task where the \"small\" \"general\" BERT can be further pruned. For a fair comparison to previous work, baselines that are applied to a specific fine-tuning task need to be compared. \n\n=====\n\nThis paper presents a new framework for creating small fine-tuned pre-trained language models. \nThe framework has 3 components: \n1. a set of transformer components to be pruned\n2. a significant analysis for identifying unimportant elements. \n3. a set of techniques to prune or approximate the transformer element.\n\nPros:\n1. The framework is very adaptive by considering different basic elements of the transformer. \n2. The framework is very efficient by removing large components (e.g., layers, attention blocks, ffd layers) at first and small components (e.g., weight group) later.\n3. The framework gathers multiple different pruning/approximation techniques and tries to explore the limit of pruning pre-trained models, which is appreciated. \n\nCons/Questions:\n1. Is the loss used in significant analysis computed using the development set? If the validation loss is used, the experiment results in Table 1 are not reliable. \n2. There are many BERT pruning papers. Providing comparison to these papers is very important to evaluate the proposed method. Can the model prune more weight at the same performance level? or  Can the model perform better at the same pruning ratio? \n3. It is also helpful to present how much computing resource is needed to prune the network. E.g., how many prune-finetune cycles are needed. \n4. Lack of results of pruning BERT-base on GLUE, which is a very standard and common setting. \n5. In Figure 3, why Q8BERT + Speed Focus is even larger/slower than Q8BERT? With the same speed, Q8BERT + Speed Focus is significantly worse than Q8BERT.\n\nMinor:\nPage 5: less the minimum loss seen ==> less 'than' the minimum loss", "extra_scores": {}, "extra_text": {}}, {"review_id": "dUiDo7LcMUU", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for improving a fine-turned Transformer in terms of a specific metric such as size, speed, or accuracy. The candidates of removed elements are considered hierarchically with some heuristics and are evaluated in terms of training and validation loss to determine whether they should actually be removed from the model. The authors apply their method to several state-of-the-art Transformer models and show that they can produce fast and compact models without losing much accuracy.\n\nAlthough the individual techniques employed to realize the whole pruning process are not particularly novel, the paper presents a well-thought-out approach to combine those and reports very promising experimental results. I think this is a nice contribution to the community, given that the computation cost is increasingly important in dealing with BERT-like models. \n\nIt seems to me that the authors used transformers whose weights are shared between different layers like Universal Transformers or ALBERT. Maybe I missed something, but I think the authors should clarify if this is really the case in the manuscript.\n\nThe entire process of pruning is a bit vague and hard to replicate. Would it be possible to describe the whole process in pseudo code? (Is Algorithm 1 the whole process?)\n\nI think the authors should also describe the computational cost (or maybe wallclock time) required to perform the proposed pruning processes. It seems to me that the search space is rather large and requires a considerable amount of computation.\n\n> p.5 … we prune the element only if the training/validation loss\n\nI think you should be more specific here. How did you actually use both the training and validation loss?  Why do you need to look at the training loss when you are interested in the generalization error?\n\n> p.5 … weight groups of (Wn) …   \n\nWhy is this Wn? I thought this should be W.\n\nMinor comments: \np.5 less the -> less than the?\np.6 doesn’t -> does not\np.6 ’’attention -> ``attention\np.7 second order -> second-order?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "IEIp0CXjtV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main goal of this paper is to introduce a simple methodology for optimizing transformer based models for efficiency and effectiveness. \n\nThe paper introduces two main ideas:\n\n1)A top-down strategy for pruning components of a transformer model: Given a specific focus, say speed, the strategy is to consider pruning large coarse-grained components first followed by smaller finer-grained components. The pruning decision is made based on a “significance analysis” -- a component is considered significant for pruning if it from the model does not result in a substantial increase in the model’s loss (as decided by a pruning threshold).\n\n2) Pruning and approximating techniques for different components: For example feed-forward networks are pruned by removing weights in groups (determined via a hyperparameter). For approximating self-attention a sign-matching technique for deciding which top K keys to use for computing Query x Key dot products. \n\nThe main strengths of this work are as follows:\n\n1) The techniques do not require training networks from scratch and can be applied directly during fine-tuning. \n\n2) The techniques are simple and should apply widely to most transformer-based models. \n\n3) The empirical results support the claim that the technique can yield significant speed-up and memory-reductions while maintaining accuracy and even provide improvements in accuracy if that is the pruning goal. They show that technique is orthogonal to other models explicitly designed for speed and memory footprint (Q8BERT, DistillBERT) and can provide further improvements in both efficiency and effectiveness. \n\n4) This is a practical and useful approach that should be widely applicable along with many useful insights about optimizing transformer-based systems. \n\nI appreciate that the experimental results are reported with averages across multiple runs!\n\nI don’t see any major weaknesses in the paper. Here are some areas that can be improved:\n\n1) The description of the pruning strategies was hard to follow and needed to be tightened up. Possibly adding equations and some pseudo-code to the description should help.\n\n2) I am curious to know what components get pruned cross the different models that were optimized. I wonder if there are systematic differences between original and distilled models and between auto-regressive (GPT) and auto-encoding style models.\n\n3) Also some level of ablation analysis on the strategies used will be helpful. For example if the elements were not ordered based on the granularity would the results be any different? Since this is an iterative strategy the order should play an important role in selection and utility of the subsequent pruning steps. Same goes for the set of pruning strategies. A related question would be what gives the biggest gains. \n\n4) What is the impact on the fine-tuning time? The baseline only requires one fine-tuning pass. Does this method require multiple fine-tuning passes? Or can the loss thresholds be computed on a smaller subset of the target data? This may be a good future work to look into for tasks where the training data is relatively large, where one cannot afford to exhaustively search through all the pruning strategies. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FfT4VQpGaPd", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Thanks to the authors for the detailed feedback! I still have concerns about the clarity of the presentation, and some contributions of the papers are not strong enough, so I'll keep my score.\n\n===\n\nSummary:\n\nThis paper presents a framework to systematically perform pruning and layer approximation. The framework includes a queue of potential elements for compression. At each time step, the framework will evaluate the head element of the queue, try to prune the whole element or perform approximation (quantizing, pruning attention heads, and approximating with sign-matching attention), and keep the transformation only if the loss in performance is acceptable. The paper performs experiments with various models on GLUE and shows speedups or compressions compared to the original model.\n\nReasons for score: \n\nThe techniques used in the paper are not novel, and the choices on how to apply multiple compression techniques need more justification. The experiment results are okay but not surprising. The presentation of the paper needs to be polished. See below for more details.\n\nPros: \n\n1. I like the insight that {approximate, fine-tune, approximate} cycles doesn’t work for fine-tuning.\n2. I like the insights used to determine which elements to be examined first: start from the larger blocks and later layers. I hope this point can be emphasized more and compared with more brute-force and less-efficient algorithms. For example, for each round, one can choose a layer that causes the least loss of performance to prune. You can compare your greedy algorithm with this algorithm to show the gain of using a less efficient algorithm is not significant.\n3. The sign-matching attention proposed in the paper is new. I would like to see more emphasis and ablation studies on the effectiveness of this special module.\n\nCons: \n\n1. It is well-known that compressing the model is easier during the fine-tuning phase [1, 2]. I don’t think this should be a contribution to emphasize for the paper.\n2. The whole compression framework has a single global error bound. Combining this with the greedy layer-by-layer approach taken by the framework, will the following case be possible: a layer that is early in the queue causes a huge drop of accuracy and thus makes all the future layers impossible to remove because the global error bound has been reached. A better way is to only remove the layer with the lowest loss reduction. It will be better to justify this point with an ablation study, or at least show the final pruned model doesn’t have this issue in the paper.\n3. At the end of page 5: “When optimizing for speed, however, removing weight groups with low significance from arbitrary locations does not help, since it introduces unstructured sparsity in the weight matrix that can be difficult to exploit to achieve speedups.” It’s true that if you remove random entries in a matrix will not help for the actual speedups, but you can remove an arbitrary set of rows of the matrix, and then restructure the weight matrix (i.e. concatenate all the remaining rows to form a new matrix) to make it efficient for modern parallel hardwares. \n4. I don’t really understand the point of using accuracy as the final goal. If the framework is for compression, the goal should be about speedup or size. If accuracy really matters, it should be enforced as the threshold instead of as the final goal. Also, I don’t see the difference in the framework between using speedup or size as the goal, since all the thresholds are defined by loss.\n5. The results in the paper are okay, but compared to previous works in computer vision [3], it seems that the model size can be further compressed.\n6. There are multiple places where the presentation can be improved:\n\n    a. It’s more clear to use a pseudo-code instead of a diagram in Figure 2. \n\n    b. It should be more clear to present Table 1 as multiple tables.\n\n    c. It’s better to put the results comparing with previous works in a table (in the middle of page 8).\n\nMinor comments: \n\n- On page 5, 3rd paragraph from the bottom, “less the minimum loss” -> “less than minimum loss”\n\nReferences:\n\n[1] Jiao, Xiaoqi, et al. \"Tinybert: Distilling bert for natural language understanding.\" arXiv preprint arXiv:1909.10351 (2019).\n\n[2] Shen, Sheng, et al. \"Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT.\" AAAI. 2020.\n\n[3] Han, Song, Huizi Mao, and William J. Dally. \"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.\" arXiv preprint arXiv:1510.00149 (2015).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a set of techniques that can be used to obtain smaller models on downstream tasks, when fine-tuning large pre-trained models such as BERT. Some reviewers have noted the limited technical novelty of the paper, which can be seen more as a combination of existing methods. This should not be a reason for rejection alone, but unfortunately, the results in the experimental section are also a bit weak (eg. see [1-4]), there are not very insightful analysis and it is hard to compare to existing work. For these reasons, I believe that the paper should be rejected.


[1] DynaBERT: Dynamic BERT with Adaptive Width and Depth

[2] Training with quantization noise for extreme model compression

[3] MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices

[4] SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
Decision: Reject