Data-aware Low-Rank Compression for Large NLP Models
Authors: Patrick CHen
Hsiang-Fu Yu
Inderjit S Dhillon
Cho-Jui Hsieh
Venue:  (2021)

Abstract: The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings the efficiency challenges, including the inference speed and the model size when deploying the model on devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space. Based on this observation, we propose DRONE (data-aware low-rank compression), a provably optimal low-rank decomposition of weight matrices, which has a simple closed form solution that can be efficiently computed. DRONE is generic, could be applied to both fully-connected and self-attention layers, and does not require any fine-tuning or distillation steps.  Experimental results show that DRONE could improve both model size and inference speed with limited loss of accuracy. Specifically, DRONE alone achieves 1.92x faster on MRPC task with only 1.5%loss of accuracy, and when combined with distillation, DRONE achieves over 12.3x faster on various natural language inference tasks.

Under review as a conference paper at ICLR 2021
DATA-AWARE LOW-RANK COMPRESSION FOR LARGE NLP MODELS
Anonymous authors Paper under double-bling review
ABSTRACT
The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings the efficiency challenges, including the inference speed and the model size when deploying the model on devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, observe that the learned representation of each layer lies in low- dimensional space. Based on this observation, we propose DRONE (data-aware low-rank compression), provably optimal low-rank decomposition of weight matrices, which has simple closed form solution that can be efficiently computed. DRONE is generic, could be applied to both fully-connected and self-attention layers, and does not require any fine-tuning or distillation steps. Experimental results show that DRONE could improve both model size and inference speed with limited loss of accuracy. Specifically, DRONE alone achieves 1.92x faster on MRPC task with only 1.5% loss of accuracy, and when combined with distillation, DRONE achieves over 12.3x faster on various natural language inference tasks.
INTRODUCTION
The representations learned by large-scale Natural Language Processing (NLP) models such BERT have been widely used in various tasks (Devlin et al. 2018). The pre-trained models of BERT and its variations used feature extractors for the downstream tasks such question answering and natural language understanding (Radford et al. Howard & Ruder 2018). The success of the pre-trained BERT relies on the usage of large corpus and big models. Indeed, researchers have reported better results of models with more parameters (Shazeer et al. 2018) and number of layers (AI-Rfou et al. 2019). The increasing model size of the pre-trained models inhibits the public user from training model from scratch, and it also brings the efficiency challenges, including the inference speed and the model size when deploying the model on devices.
To deal with the efficiency issue, most existing works resort adjusting the model structures or distillation. For instance, Kitaev et al. (2020) uses locality-sensitive hashing to accelerate dot-product attention, Lan et al. (2019) uses repeating model parameters to reduce the size and Zhang al. (2018) applies a pre-defined attention pattern to save computation. A large body of prior work focuses on variants of distillation information has also been explored (Sanh et al. 2019 Jiao al. 2019 Sun ctt al., 2020; Liu et al.) 2020; Xu et al., 2020) Sun et al. 2019). However, all these methods either require specific design of model architecture which is not generie, require users to train the proposed structure from scratch which greatly reduces its practicality.
In this work, we try to explore acceleration method that is more generic. Note that as shown in Figure [ matrix multiplication (Feed-forward layer) is fundamental operation which appears many times in the Transformer architecture. In fact, the underlying computation of both multi-head attention layers and feed-forwart layers matrix multiplication. Therefore, instead of resorting to the complex architecture redesign approaches, we aim to investigate whether low-rank matrix approximation, the most classical and simple model compression approach, can be used accelerate Transformers. Despite being successfully applied to CNN (Yu et al. 2017] Sindhwani et al. 2015 [Shim et al.) 2017] You et al. 2019), the first glance low-rank compression cannot work for BERT. We could see in Figure 2 that regardless of layers, matrices in feed-forward layer, query and key transformation
of attention layer are not low-rank. Therefore, even the optimal low-rank approximation (e.g., by

---

Under review as a conference paper at ICLR 2021
Parameter Size
Inference Time
(0.0%)
Feod fereadd
7 768
104 mi (0.24)
Add & Narm
Fece-Fomord2
30724 1E8
22.6 (29.0%)
Feel ward
768 3012
140.4 ms (52
422.5
Feed Lmarr
28.8 ms (6.1%)
Nalli fead Mitention
Ercedes huat
78
132 ns (8.2%)
Figure 1: Illustration of the BERT-base computational model. |V| denotes the number tokens in the model. # Claasses denotes the number of classes in the down-stream classification task. Input encoding, Feed-forward 3 and ferd-forward are computed only once in the inference and thus do not contribute to overall computational time much.
Layors
Query Key Layers
Accumalated Rabio of Sinquler Velues
Layer-1-FF2
Layer-1-FF1
Accumulated Ratto of Singular Values
Layer-1-Query
Layer-1-Key
Layer-12-ff1
Layer-12-Query
Layer-12-FF2
Layer-12-Key
64
02
Percentile
Percentile Total Ranks
(a)
(b)
Figure Illustration of the empirical observation that weight matrices in BERT model are not low-rank. The x-axis represents what percentage of the ranks are selected; the y-axis represents sum of singular values connected to the selected ranks divided by sum all singular values. Ideally, low-rank structure will have a larger area under curve, meaning that a small percentage of the ranks could explain most of singular values. We observe that the sum of top 50% of the ranks only account about 60% of all singular values for matrices in the BERT model. This shows that the matrices do have clear low-rank structure.
SVD) will lead to large reconstruction error and empirically the performance limited. This is probably why low-rank approximation has not been used in BERT compression.
In this paper, we propose a novel low-rank approximation algorithm to compress the weight matrices even though they are not low-rank. The main idea is to exploit the data distribution. In NLP applications, the latent features, indicating some information extracted from natural sentences, often lie in subspace with lower intrinsic dimension. Therefore, in most of the matrix-vector products, low-dimensional subspace,
even though the weight matrices are low-rank, the input vectors lie allowing dimension reduction with minimal degraded performance. We mathematically formulate this generalized low-rank approximation problem which includes the data distribution term and provide a closed-form solution for the optimal rank-k decomposition of the weight matrices. We propose DRONE method based this novel Data-awaRe 1Ow-raNk comprEssion idea. Our decomposition significantly outperforms SVD under the same rank constraint, and can successfully accelerate the BERT model without sacrificing too much test performance.
2

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
The inference speed is important for NLP models when deployed in various applications. Generally speaking, inference efficiency could be enhanced by hardware (Shawahna et al. 2018 or lower-level instruction optimization (Ning, 2020). On the other hand, the main focus of the current research is on using algorithmic methods to reduce the computational complexity. It could be mainly categorized into two aspects: Attention Complexity Reduction and Model Size Reduction.
Attention Complexity Reduction
Attention mechanism is the building block of transformer model and it attracts most attentions of researcher recently in NLP field (Vaswani et al. 2017). Pre-training on large courpus of BERT, performance on various tasks after fine-tuning (Devlin et al. 2018). Attention on sequences of length L is O(L2) in both computational and memory complexity. This would take long inference time when the sequence is long. Thus,
transformer-base based model, has contributed to state-of-the
researchers have focused on reducing the complexity of the attention module. Kitaev et al. (2020)
uses uses locality-sensitiv hashing to reduce the complexity to O(LlogT.) I.). Zhang et al. (2018); Child al. (2019) pre-defined an attention map to have a constant computational time. (Goyal et al. progressively eliminates the redundant context vectors within the attended sequence to improve the efficiency of attention in last few layers of the model. |Wan et al. (2020) proposes
train the
low-rank attention by choosing a rank << L. This is similar with our work in the sense of leveraging low-rank structures. But our method do not require retraining the model and could be applied to different modules other than attention. In fact, most of these methods require special modules and thus we need retrain the proposed models from scratch. This prohibits the usage of large body of publicly available open models for faster research progress. More importantly, these methods only focus the long sequence scenario. We found out that attention module is actually not the bottleneck of inference time in common usage as shown in Figure [] In most if not all models of common usages, two layers of large feed-forward layer are appended after attention module which incurs much more computational time. Attention complexity reduction only works when a large sequence is used but in current practice this is unusual. Thus, accelerating attention module itself is not contributing to a significant reduction of overall inference time.
Model Size Reduction
Efficiency of inference time also related to model compression. In principle, smaller models would lead to smaller number of operations and thus faster inference time. [anh et al. (2020) has explored pruning methods on BERT models to eliminate the redundant links, and there is line of research on pruning methods (Han et al. 2015a b. Chen et al., 2020). Quantization methods (Zafrir et al. 2019; |ubbar al., [0166] Lin et al.) 2016) could convert the 32 bits float models into lower-bits fixed-point representation, and theoretically make model prediction faster with fixed point accelerator. Lan et al. (2019) reduces the model size by sharing of encoder parameters. A large body of prior work focuses on variants of knowledge distillation (Sanh et al. 2019; Jiao et al. 2019 Sun et al.) These methods use different strategies to distill information from teacher network and reduce the number of layers (Sanh et al. 2019) or hidden dimension size (Jiao et al. 2019). Further, A hybrid compression method by combining matrix factorization, pruning and knowledge distillation is proposed by Mao et al. (2020). Among the above mentioned methods, Quantization requires hardware accelerator to reduce the inference time which is not applicable to general scenario. Pruning methods could only reduce the model size, but the inference speed might not be reduced due to the limitation of sparse operations. Only algorithmic method such as distillation could serve as a generic inference time accelerating method. We want to emphasize that our method is orthogonal to these distillation methods. In fact, the proposed method a generic acceleration method applicable all components in most NLP models. In section 4 we show that DRONE can be combined with the distilled models to further improve the performance.
Lin
2020 Liu et al. 2020 Xu et al.) 2020 Sun et al.
3 METHODS
We now introduce a generic algorithm for improving efficiency of matrix multiplication. The computation of Feed-Forward (FF) Layer in the attention models can be described as:
h Wa + b.
(1)
3

---

Under review as a conference paper at ICLR 2021
where W E Rd2xd and b E Rd are model parameters, R is the latent representation of token, and h E Rdda the output. Assume the sequence length L, all the token representations al....,, .T E IRd will pass through this same operation, so in practice the whole FF layer can be computed by a matrix-matris product W [[=i.... + and the computation of bias term b would be broadcasted to all T. input tokens. In practice we will normally have T. << max(d1, d2) (e.g..
128, d2 = 3, 072).
A standard way to accelerate the computation is to perform low-rank approximation over W. A low- rank approximation can be acquired by using singular value decomposition (SVD), which achieves the best rank-k approximation in terms of Frobenius norm and we could write W as:
W USVT Uw,& Vw:""
with unitary matrices U E Rddx V [Rd xdi and diagonal matrix S dR Uw. E gd
and Vw, E 12dh xk are the rank-k approximation matrices by taking Uw, US} Vw, S where 5t is the square-root of the first & entries of the diagonal matrix S. Given such approximation, we could simplify the computation in equation 1 by writing it as:
h Wa b = Uw, Vw, T b.
After the rank-k low-rank approximation, the computational complexity reduces from (d2d1) to O(dd, + kk). When is small enough, low-rank approximation could not only accelerate the computation (Shim al. (017) but also compress the model size (Sainath et al. 2013). However, as we showed in Figure 2 matrices in FF layer of BERT models do not show obvious low-rank structures. Ideally, want small percentage of the ranks which containing all large singular values such that sum of singular values connected to the selected ranks divided by sum all singular values is large. But we could observe that choosing rank & to be larger than 50% of the ranks (e.g.. about 0.5 times min(di, d2)) could only accumulate 60 percent of the total singular values. This will lead to large approximation error. In the meantime, the complexity is still about (d2d1) and there is no enhancement of speed.
Despite the matrices in the model are not low-rank, here we provide an illustrative example to show that low-rank computation could still exist when data distribution lies in lower intrinsic dimension. Suppose we have a W defined as below and the input x lies in a subspace:
7
0
2
3
1
2
7
5
0
2
1
W
1
8
0
3
I E span(
5
2
)
3
2
1
4
2
1
2
2
1
2
4
6
In this case, W is a full-rank matrix so there won't 't be lossless low-rank approximation on W. On the other hand, input data x lies in a 2-dimensional subspace such that we could construct the following computation:
7
0
2
3
2
1
43
23
2
0
[a]
90
39
-1
-1
0.5
0.5
0
a
6
8
0
3
5
2
66
41
5
2
b
0.5
0
0
0.25
4
45
37
1
2
2
1
2
4
29
21
VT
4
which gives rank-2 matrix UVT where W A UVT but Wx UVT for any in the low dimensional space. This shows that even we can't 't approximate the W matrix, it is still possible to construct good low-rank decomposition, and the key will be to exploit the space of input vectors.
3.1 DRONE: AWTRE LOW-RANK COMPRESSION
Assume the input of the FF Layer follows some distribution, instead of minimizing approximation error of weight matrix (for which SVD is optimal), we want to minimize the approximation error of the outputs. Denote X as the Rd matrix where columns of X capture the empirical distribution of
4

---

Under review as a conference paper at ICLR 2021
imput, goal is to find pair of projection matrix Vx Rdiik and recovery matrix Ux,* E Rd such that the output result well approximated, and we could rewrite equation | as:
W + b = 2U,, V2. x + (WUz.a) "x + where Wx, = WU.... Intuitively, when lies in a fower-dimensional space, we could find such pair by PCA decomposition on X to project X into subspace that explains most variance of X. In this way, instead of considering the decomposition of W, we leverage the distribution of X to complete the low-rank approximation. Certainly, the best way is to consider the properties of both W and X simultaneously, and we could mathematically present this desideratum by the following optimization problem:
W., b.
min W - WM s.t. rank(M) = k,
where M is the desired rank transformation which could maximally preserve the results of matrix multiplication in the computation. In the theorem below, we will show that there exists a closed-fom formm optimal solution for the above optimization problem. Before stating the theorem, we first introduce some notations. Assume rank(W) = and rank(X)) = t, we can write W = Uw Sw VW and
X" = UxSxx VX such that
Swr 0
Uw- [Uw.r Üw,r SS
Vw = -Visr Vw.rl
0
Sx,1 0
00
Ux = [UX, Ux,]] SSx
Vxx [Vx,, Vx,,].
In other words, the decomposition Sw VW and Sx are the full-SVD decomposition of W and XT. Uw.r.W..r. Vx,, denote corresponding spaces and column spaces. Uw.r, Vw.r. Ux, and Vx,t null spaces. With these notations, are ready to state the theorem.
Theorem 1. Assume rank(W)) = r and rank(X) = t. The closed form solution M of the optimiza- tion problem in equation 2
M = Vw., W.r 7x. x, X,t=
where Zg is the rank-k truncated SVD of 7 Sw, VW,, Vx Sx,t
The proof of Theorem T is postponed to the Supplementary A. We want to note that since Za. is the rank-k truncated SVD of Z, could also write 7& U22,V2, by distributing top-k singular values of Z into left right singular matrices. Thus the original computation could be rewrote as:
U where we U* WVW, S. U2 and V V2 x VX, are two rank-k matrices, and we will replace W by U*V*
WX Re (WVw,r VV
(4)
3.2 EXTENSION TO DOT-PRODUCT ATTENTION
Although the optimization problem in equation 2 is proposed for feed-forward computation, in this section we show that it can also be applied to dot-product part of the attention module too. The most important computation in Attention layer is compute pairwise similarity between query and key of the sequence, and this could be described as:
O (QP)"
(5)
where Y E IRd xn is the batch query data, Q 1adaxd; the query transformation matrix, Y Rd Xm is the batch key data, K E idda xd, is the key transformation matrix and n. m are query and key batch size. We could again see that the desired low-rank approximation the solution of following optimization problem:
((Y)), s.t. rank(M) k. min [IQ))" M
(6)
With QV = W and KÝ = X, we get the following corollary from Theorem I directly.
Corollary 2. Assume rank(QÝ) and rank(KY) = t. Denote QÝ = Uw Sw ViT and (KY)* = Ux V' the SVD decomposition of QÝ and (KY)T respectively. The closed form solution M of the optimization problem in equation i is M = Sw Sx1 vf, where Zy is the rank-k truncated SVD of 7 Sw.r W Vx,
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Data-Aware Low-rank Compression of Feed-forward layer.
Input:
rank k; training data Dirain: Original weight matrix W; Prediction Model M Output: Low-rank Approximation U*, V*
for x = -, in Dirain do
Feed the training data x into M and extract the representation <(x). 6(x) is the representation which will be multiplied with W.
Append ((x) to X.
Given X,k and W, solve the optimal low-rank matrices U*,V by equation 4
3.3 OVERALL ALGORITHM
We have shown that the proposed DRONE method is generic acceleration module applicable to all parts of neural language models. We summarize the DRONE on feed-forward layer in Algorithm| [ Since in practice we don't 't have the exact distribution of X, would use training dataset to calculate
the low-rank approximations as described in Algorithm| I Attention map could be calculated by the same procedure with W (QY)" and X KY introduced above. To accelerate the whole model, we need to select appropriate ranks for each components. However, since the approximation of one component will affect the distribution of overall representations, the optimal rank for the model requires a complete search of all possible combinations of rank values, which is infeasible in practice. We thus resort to an intuitive simplification as shown in Algorithmm2 [ listed in the supplementary B. In short, as the changes of lower layer parameters will cause the distribution of representation shifts in upper layer, we will approximate each component in ordered sequence. In other word, we will approximate the model from the lower layers toward higher layers. Within each layer, we follow the computational sequence of underlying modules. There is a total budget parameter r as an input to the Algorithm 2 The total allowed budget depends on the efficiency and efficacy trade-off which users are willing to pay. We will distribute r into each module Rú, (allowed loss increase ratio of i-th module of 1-th layer in Algorithm 2). For each module, if the approximation with certain rank used won't 't increase the loss over the ratio (1 + Ra,s), we will use that rank to approximate the module and move to the next module. The distribution from each R is based on empirical inference time of each module. The longer a module takes to compute, the more budget would be allocated such that total allowed loss increase (1 + r) = II (1, + Rt,x). ). A sample of pseudo code is provided in the supplementary to illustrate the process.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate DRONE on both LSTM model and transformer-basee BERT model. For LSTM, we train a 2-layer LSTM-based language model on PTB from scratch with hidden sizes 1, 500 on Penn Treebank Bank (PTB) dataset. For BERT models, we evaluate the pre-trained BERT models on GLUE tasks. Various pre-trained models are offered in the open source platform (Woll et al. 2019). For BERT models, we use BERT-base models and it contains 12 layers of the same model structure without sharing parameters. Each layer contains an attention module with hidden size 768 and 12 channels, a small 768 x 768 Feed-forwal (FF) layer followed by 2 larger FF layers (768 x 3, 072 and 3, 072 768). As shown in Figure T these four components contribute to the most computational time in the BERT-base models.
To the best of our knowledge, are the first work to discuss the generic matrix computation acceler- ation on NLP tasks. Therefore, our baseline comparison will be the SVD low-rank approximation. We will also include those state-of-the comparison and show that the proposed method could be combined with to further improve the performance. TERyBERT reduces the model into 4 layers of attention dimension 312 with 12 channels, and the FF layers are downsized to 312 x 1, 200.
distillation methods TinyBERT (Jiao et al. 2019) in the
As we mentioned above that all the approximation methods need to consider efficiency and efficacy trade-off. In principle, we would have a plot of accuracy versus speedup ratio as shown in Figure 3 for MRPC and SST-2 tasks. The allowed accuracy loss is up to tolerance of individual users. In this paper, we will report the approximation results with about 3% loss of accuracy. The inference

---

Under review as a conference paper at ICLR 2021
Methods
MNLI
QQP
SST-2
QNLI
MRPC 89.5
RTE
CoLA
STS-B
Original
84.3
90.9
92.3
91.4
72.6
53.4
87.8
SVD
74.4
50.8
73.1
52.2
63.8
47.3
26.0
0.13
DRONE
82.0
89.4
90.0
88.5
70.0
$2.5
85,8
BIKONE--Retaile
82.6
90.1
90.8
89.3
88.0
71.5
53.2
87.8
Speedup Ratio
1.60k
1,25x
1.64x
1.20x
1.92x
1.31x
1.33x
1.52x
Table 1: The experimental results of natural language inference tasks on Glue dataset.
Tasks
Self-Attention
Feed-Forward
Feed-Forward
Feed-Forward
Others
Total Time
MNLI
122.7
19.5
78.5
46.1
4.2
268.0
QQP
131.5
29.9
99.2
66.3
5.8
333.0
SST-2
100.5
24.7
79.3
54.5
4.5
263.6
QNLI
128.3
28.4
111.0
79.0
5.9
352.6
MRPC
82.6
12.8
89.4
38.2
2.4
225.3
RTE
116.0
25.6
85.4
62.3
3.4
292.7
COLA
108.2
22.7
93.1
70.8
3,4
298.3
STS-B
109.1
19.3
90.8
53.0
276.2
Table 2: The detailed average inference time of each component in the model by retrained DRONE method. The unit in millisecond.
Tasks
Models
Self-Anention
Feed-Farwand (
Feed-Forward
Feed-Foeward
Others 4.4
Toral Time
speedup
Accuracy (%)
BERT
139.7
32.5
134.7
109.0
420,2
1x
87.8
DRONE
0.8
29.5
14.2%
87.0
BERT
124.3
21.7
129.0
101.5
3.1
383.5
1x
72.6
RTE
TRyBERT
210.2
DRONE
2.8
BERT
131.0
29.9
120.4
127.6
530
452.5
1x
89.3
MRPC
TayBERT
18.6
1.9
9.9
6.1
0.7
37.2
11.62
86.3
DRONE
0.9
35.3
BERT
146.3
36.1
133.6
1100
533
431.3
1x
92.3
SST-2
TRnBBRT
18.1
1.9
9.9
6.2
0.7
35.8
11.7%
90.7
DRONE
33
28.1
15.3
90.7
Table 3: The average inference time of comparison to distilled models. The unit is in millisecond.
speed is measured on an Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz chip with single-thread. Inference are performed in a batch fashion with size 100 of the whole evaluation dataset. Average single sequence prediction inference time in millisecond is reported in the results. To perform the approximation, we randomly sub-sample 10% of the training data to the training distribution used, After the proposed data-aware low-rank distribution, we could slightly fine-tune the model to make the performance better. We use relatively smaller learning rate 10-7 and retrain epoch the sub-sampled training data to complete the fine-tuning procedure.
4.2 RESULTS OF BERT MODELS ON GLUE DATASET
We summarize the results the DRONE on GLUE tasks in Table We could observe that each task exhibits different diffieulty. The best acceleration we could achieve is 92% faster with less than 2% accuracy loss after retraining (MRPC). In addition, DRONE achieves 1.52x acceleration without accuracy loss on the STS-B task. By applying the same selected rank for each module with SVD method, we could observe that the performance drops significantly. This shows that the matrices within the model is generally not low-rank; thus the direct low-rank approximation without considering data distribution won't work. The detailed average inference time of the approximated models are listed in Table 2 We could observe that the FF2 layer could be accelerated most. A plausible reason could be that the input dimension to the FF2 layer in larger dimension (3072) than all the other layers (64 or 768). When the input distribution actually lies in a lower- diwee--imeniionll space, there much more room for FF2 layer to be compressed and accelerated by the data-aware low-rank method.
4.3 COMBINATION WITH MODEL SIZE REDUCTION METHODS
Distillation is a competitive method to compress the underlying model into a smaller one without losing much accuracy. Distilled models much smaller in number of layers hidden dimension, resulting a smaller model size and faster inference time. As shown in the Table TERYBERT, one of the most competitive distillation methods, indeed achieves good performance within 3% accuracy
7

---

Under review as a conference paper at ICLR 2021
Tasks
Models
Total Time
speedup
Accuracy (%)
BERT
3.17 ms
1x
87.8
STS-B
TinyBERT
0.37 ms
8.6x
86.9
DRONE
0.29 ms
10.9x
87.0
BERT
3.34 ms
1x
72.6
RTE
TinyBERT
1.73 ms
1.9x
70.8
DRONE
1.51 ms
2.2x
71.7
BERT
3.28 ms
1x
89.5
MRPC
TinyBERT
0.42 ms
7.8x
86.3
DRONE
0.38 ms
8.6x
86.7
BERT
3.29 ms
1x
92.3
SST-2
TinyBERT
0.39 ms
8.4x
90.7
DRONE
0.34 ms
9.7x
90.7
Table 4: The average inference time of comparison to distilled models on GPU. The unit in millisecond.
loss for some of the GLUE tasks. We want emphasize that DRONE is orthogonal distillation methods. Since DRONE is generic. it could be combined together with other methods to further improve the performance. Due the fact that the computation inside the distilled model is still full matrix computation, DRONE could then be applied to find the data-aware low-rank approximation of these smaller matrices. Results are summarized in Table 3 As we can see that combined with the distillation method, DRONE could further reduce the inference time without sacrificing accuracy. In particular, SST-2 task DRONE speedups the inference time from 11.7x to 15.3x while achieving the same accuracy as the TinyBERT. These results again show that the proposed method is generic and has the potential to be applied under various scenarios.
4.4 RESULTS ON LSTM MODELS
We demonstrate that DRONE could also work accelerating matrices in the LSTM model. As shown in Table 5 in the Supplementary C. DRONE could accelerate 2-layer LSTM models about 3.4x PTB dataset. And the result is slightly better than SVD methods. After the fine-tuning, DRONE could achieve less than 1% accuracy loss. This shows that DRONE is generic. As long as the underlying computation is matrix multiplication, DRONE could leverage the data distribution to obtain a better low-rank approximated computation.
4.5 COULD THE LOW-RANK STRUCTURE LEARNED BY END-TO-ENT TRAINING?
One natural question to ask if once the rank is decided, the same optimal low-rank structure could be learned by end-to-end fine-tuning We have conducted the experiments on MRPC to verify this, and the results are summarized in Table i the Supplementary D. We start the fine-tuning from the SVD results, and use the fine-tuning hyper-parameters as in (Wolf et al., 2019 After fine-tuning, accuracy goes from 63.8% 85.8% which is still slightly worse than DRONE. This shows that fine-tuning on the SVD result might not achieve the best low-rank result. The proposed method under the the optimization problem (equation 2 indeed provides good initial solution the data-aware low-rank problems.
CONCLUSION
In this work, we propose DRONE, data-aware low-rank approximation, to achieve better low-rank approximation. DRONE leverages the fact that data distribution in NLP tasks usually lies in lower-dimensional subspace. By including the data distribution into consideration, we propose data-aware low-rank approximation problem and provide an closed-form solution. Empirical results validates that DRONE could significantly outperformed the vanilla-SVD method. It could achieve least 20% acceleration with less than 3% accuracy loss. When combined with the distillation methods, DRONE could achieve 15.3 times acceleration with less then 2% accuracy loss.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rami Al-Rfou, Dokook Choc, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3159-3166, 2019.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. arXiv preprint arXiv:2007. /2000./ 2020.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint
2019.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint vrXiv:1810.04805. 2018.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan T Chakaravarthy, Yogish Sabharwal, and Ashish Verma. Power-hert: Accelerating bert inference via progressive word-vector elimination.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. CoRR, abs/1510.00149, 2015a. URL http: /wwpvvaaivvvv..a///ii org/ abs/ 11100 .00149
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. CoRR, abs/1506.02626 2015b. URL, htt.tttttp://arx/.oorrgg///// 1506. 02626
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 2018.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv 2016.
preprint wrrivv:609.0000
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: /909. 1909.1031,, 2019.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04455 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:/909, 1942, 2019.
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization deep convolu- tional networks. In International Conference on Machine Learning, pp. 2849-28555 2016.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self-distiling bert with adaptive inference time. arXiv preprint arXiv:2004.02178, 2020.
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, and Jing Bai. Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv preprint arXiv:2004.04124 2020.
Emma Ning. Microsoft open sources breakthrough optimizations for transformer inference on gpu and cpu. htt.:///e..coccoooo.....coccoooe
mi siiccooott-oo onnx open ort.acaccoo....oooooooooooooooar 2020. Accessed: 2010-09-30.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners.

---

Under review as a conference paper at ICLR 2021
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6655 6659. IEEE, 2013.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01106 2019.
Victor Sanh, Thomas Wolf, and Alexander M Rush. Movement pruning: Adaptive sparsity by fine-tuning. arXiv preprint arXiv:2005.0768; 2020.
Ahmad Shawahna, Sadiq M Sait, and Aiman El-Malch. Fpga-based accelerators of deep learning networks for learning and classification: A review. IEEE Access, 7:7823-7859, 2018.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Pemporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pp. 10414- 10423, 2018
Kyuhong Shim, Minjae Lee, Iksoo Choi, Yoonho Boo, and Wonyong Sung. Svd-sofumax: Fast softmax approximation on large vocabulary neural networks. In Advances in Neural Information Processing Systems, pp. 5463-5473, 2017.
Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for amall-footprint deep learning. In Advances in Neural Information Processing Systems, pp. 3088-3096, 2015.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 190:: 09355, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang. and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02298 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang. and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006. 04768, 2020.
Thomas Wolf, Debut, V Sanh, J Chaumond, Delangue, A Moi, P Cistac, T Rault, Louf, M Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ArXív, abs/1910.03771, 2019.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. Bert-of-theseus: Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
Y. You, Y. He, S. Rajbhandari. W. Wang, C.-J. Hsich, K. Keutzer, and J. Demmel. Fast Istm inference by dynamic decomposition on cloud systems. In ICDM, 2019.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7370-7379, 2017.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.061 2019.
Biao Zhang, Deyi Xiong, and Jinsong Su. Accelerating neural transformer via an average attention network. arXiv preprint 18x5v:1805.00633 2018
10

---

Under review as a conference paper at ICLR 2021
Supplementary Material for:
Data-Aware Low-Rank Compression for Large NLP Models
A PROOF OF THEOREM
Theorem 1. Assume rank(W) = r and rank(X)) The closed form solution M of the optimiza- tion problem in equation 2 is
M = Vw.,.S s-1 W.r 7x X,
where Zg the rank-k truncated SVD of = Sw. VW, Sx,
Proof. We firstly consider the unconstrained problem:
M argmin [W - WMXX 1|
argmin
U WXUx WU
M
argmin
|[ Vi Vx
- Sw MVX
M
where the second equality holds due to the fact that Uw and Ux are orthonormal matrices. Note that we could expand the term Sw Vi Vx Sx as:
Sw.r 0
Sx,:
Sw V/ Vx5x=
Vx.a
vx,]
0
W..
Sw. Wir
Vx, 0
0 0
[Sw Vit, Sx.
0
Similarly, we will have
Sw VT V
MVxx5Xx 07
Sw VW
Therefore, we could continue above unconstrained problem as:
M argmin
Sw V Sx
Sw
M
Sw. VT W.r Vx, - Sw NV MVX
0] 1
argmin
||
M
argmin |w VV. Vx,5xx, - Sw Vit,,. MV 5x.11
argmin | Sw,, WV
M
The above minimization problem obtains the optimal value if Sw,- VW,, MV, Sx,t equals the rank-k truncated SVD of Z by the fundamental property of SVD decomposition. Thus, we will have:
Z₁ Sw., VTT VW. M" Vx. M Vw, Z.S Z
11

---

Under review as a conference paper at ICLR 2021
B AN ALGORITHM TO SEARCH OF RANKS UNDER DRONE
In Algorithm we illustrate how to select the rank of each module by applying DRONE illustrated in Algorithm The input to Algorithm 2 consists of training data, the model with all parameters of weight matrices and original training loss. In addition, pre-defined search grid also necessary. example, we can perform grid search for proper low rank k over [1, 768] such as {96, 192, 288, 384, 768}. The finer the grid, the more compressed model we could get at the cost of longer running time of the DRONE method. With these input parameters, firstly distribute the total allowed loss into each individual module. We then iteratively apply Algorithm [ following the computational sequence illustrated in Figure [ each module, we search the rank & by going through the grid. the approximated result will increase the allowed loss increase ratio of the component, we will end the search and tie the found rank to the component and move on. The procedure will continue until all components are compressed. The whole process could guarantee us that the final loss L' of the compressed model M would not be greater than (1 + r) L, where L is the original loss before approximation.
Taking W E 12766x766 x768
For the compression of
Algorithm 2:
Overall Algorithm of Grid Search of Low-rat rank Model Approximation
Input: training data Drrain's : Original weight matrix W; Prediction Model M, total allowed loss increase ratio r, Search grids ranks for each module G, original Training loss L,
Output: Low-rank Model M
R < Distribute allowed ratio into each module.
for [ = 1.... total layers do
foreach module m, Mi do Wi,i f- [-th layer parameter of module m, (c.g., 2nd feed-forward matrix in first layer.)
for i = 1...., , do
1: 4- Gi,i
U, V < Algorithm (ks Derain, WL.; M)
M M with Wi, replaced by U,
Evaluate new loss Lnew ((Draii))
if Lnew / L < 1 + Ri, then
M M
break
12

---

Under review as a conference paper at ICLR 2021
Effiziency and Efficary Tradooff of DRONE on MAPC
Efficency and Efficacy Tradeoft of DRONE on SST-2
8.5
250
Prodiction Accuracy
Prodiction Accuracy
31.5
411-
Inference Time Speadup Ratio
Inferonce Time Speadup
(a)
(b)
Figure Illustration of efficiency and efficacy trade-off. Each point in this graph represents specific
ratio of training loss increase after approximation.
C EFFICIENCY AND EFFICACY TRADE-OFF GRAPH
D
LSTM RESULT
Models
LSTM-1 1.27ms
LSTM-2 1.30ms
Softmax 1.09ms
Others 0.13ms
Total Time
Perplexity
PTB-Large PTB-Large-SVE SVD TTB-Large-DRONT PTB-Large-DRONE Retrain
3.79ms
78.32
81.09
80.87
0.24ms
0.34ms
0.42ms
0.1 lms
1.1
79.01
Table 5: The average inference time of each component in the model of 2-layer LSTM model. Both proposed methods and SVD use same ranks so the inference time is approximately the same. The unit is in millisecond and the number in parenthesis shows the ratio respective to the overall inference time.
E RESULTS OF SVD-BASED RETRAINING
Models
Accuracy (%
BERT-MRPC
89.5
DERT-MRPC-DRONT BERT-MRPC-SVD
86.8
63.8
SETT-MRRC-SVD-Retraaa
85.8
Table 6: Illustration SVD fine-tuning. Using the same rank as the proposed method, SVD accuracy will drop significantly, After fine-tuning the SVD-based approximation, the accuracy could be recovered. But it's still less competitive than the proposed method.
F PYTHON PSEUDO CODE OF SOLVING EQUATION 2
import numpy np
def OPTsolver (x, (x,y,k)
...
compute the best rank k projection m such that M xay" xxMxy' \ L is minimized
x Vin shape d
y Vin shape
13

---

Under review as a conference paper at ICLR 2021
10
xSS np.matmul (x. transpose x))
11
kSs np.matmul (.. ttw.stossssse y))
12
01, V1 = np.1inalg. svd (x38, Fals)
13
$1 S1 ** 0.5
I1 (S1. [0])
82, - np.linalg. (kss, False)
S2 S2 **0.5
17
I2 np.eye (S2.
18
YK np.dot (np. (11+51, v1), ,p.,, det (V2. tv2.transpose () 12+82))
19
U, S, V (VI..raaspose (1/S1))
20
L = np.det (V1.
R np.dct (I2* (1/S2),
np.dot : Vl:k,,)))
23
I,, U,
Listing The python function to solve equation 2
G PYTHON PSEUDO CODE OF RANK SEARCHING
import os
import numpy as np import torch
import subprocess
cuda__i num
h_heads totalllayer 12
10
prev_ loss 11159391902588609 # Initial Loss
12 the_model_name beerts122'
14 time_attn 117.5 Empirical Inference Time on Attention Module
15 time_0 0 34.27 Empirical Inference Time on Attention FFL Module
16 time_1 133.11 # Empirical Inference Time on Feedforward 1 layer
17 time_2 128.84 # Empirical Inference Time on Feedforward 2 layer min
18 minimal_timm
ti tim,timm,,,iin time___)
multiplier (time attn-timeee Setee_tttimmmm Ottimm_ettiimee 1+tiitteem__. 2) (minimal time)
20 tolerant 2. allowed loss increase ratio. SrS Algorithm
21
22 Code to Distribute the $xş into individual Modules.
23 The distribution depends on empirical inference time of each module and namber of layers.
24 basic_tolerance exp (np peggttolerant//mlttiplleerr tol_attn ex.exp (np. log bat.ee/eccccccccca...ittaattttta
n n_layer)
26 tol 0 ep.exp (np. (baaec_tole http::/e.ceeeeeeeeeeccoeeeee/ana http://meeeecciiimmmmmmmmmmaaaae //_]ayyr1
27 tol 1 = np. ep.exp (np. Saaiicttolerance.........r. (time 1rsime__/miniiml_iiie //n_layerr
28 tol_2 - ep.exp
(time_ (tiie__/minimma_ccin)) (annmmaa___iieeee_eaeeeee /n_layer)
#*## Omitted Code ###
32 This part of the code is to change some parameters of the underlying hugginface framework in order to extract the training distribution X of each module from the model.
33 ### Omitted Code ###
36
37
for range (total_aaaal
38
for each module in the layer: This line is pscudo code for clarity reason.
$ This part of the code extracts (R_(1,1 11$ (named tol here) in Algorithr. 2. if save save_symbol "
14

---

Under review as a conference paper at ICLR 2021
the tol tol attn elif save_symbol "F0";
42
45
the_tol tol_0
44
clif Lave_symbos = "F1":
45
the_tol tol_1
46
else:
the_tal - tol_2
Update the allowed increase of loss prev_lose prev loss tol
49
50
51
rank = 16 if have_symbe = "E" else 96 & initial search rank for Atteation (16) and FFL Layers(96)
tps = 64 if save_symbol == "g" else 768 4 Maximal rank speci fied in the original modeis.
while rank tps:
54
### Omitted Code ###
55
4 write the tried rank into hugginface framework#F MA Omitted Code AAA
56
This line the inference in the command 1ine ss.system (' DIIALVLIIILL__ (cuda_ (cuda_nur)) + python run_glue. --model_type bert --mode__ram__o__pttee /dtta//inyyerrss/
38T7//origina1181/7/ -taakssstt/-tsssaaaaaaaa SST-2 --do_eval --data_dir /data/
glue iive_data/SST-22 -ooupput_ii /tmp/sst-2 --pe__gpu_ee__ bhtch_tttht_ha_ee size
100
100 --max_seq_iengee 128 > /tmp/tmp0'
62
with open httpp//pppmttt///mmmm/ (rr) as file:
63
data file....errre readlinss))))
64
new_r = float (data[-1)) if new_r < prev_lcss: break
65
66
if save_symbol -- "g": # Attention module, increase search rank 16 time.
68
rank 16
89
else:
70
#rank += 96 & For FFL layer, we increase search rank 96
time.
if rank == 384:
rank 768
73
break
74
else:
15
rank += 96
76
### Omitted Code ### This part of code update the model
Listing A mixed of real code and pseudo code illustrate the search algorithm.
15


Reviews:
[{"review_id": "uMdTUK-2RHt", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper studies a technique to increase the inference speed and decrease model sizes of pretrained NLP models such as BERT. Since most operations in BERT consist of matrix multiplications, the authors conduct empirical experiments to show that while matrices themselves are not low-rank, the learned representation of each layer lies in a low-dimensional space. These empirical insights lead the authors to propose an approach based on data-aware low-rank compression of pretrained weight matrices and can be applied to fully-connected and self-attention layers. Their approach is able to improve both model size and inference speed\nwith limited loss of accuracy on the Glue dataset\n\nStrengths:\n1. The paper studies an important problem and is well-motivated. The writing is generally clear and supported by figures and experimental results.\n2. The algorithm is presented very clearly and is very intuitive. Their algorithm is also supported by examples and proofs when necessary.\n3. The proposed approach is simple to implement and can be applied for different network architectures. This is good and the results are also promising, but there are some issues with evaluation (see below).\n4. Pseudocode is provided in the appendix and the approach seems easy to implement and quite flexible.\n\nWeaknesses:\n1. Figure 3 tradeoff between performance and speedup is not very useful unless the plots are also made for existing baselines and compared with the author's proposed approach.\n2. The only empirical comparison is with TinyBERT but it seems like there are many recent papers studying similar compression algorithms on BERT:\nhttps://arxiv.org/abs/1909.11687\nhttps://arxiv.org/abs/2001.04246\nhttps://arxiv.org/abs/2002.02925\nhttps://arxiv.org/abs/2002.08307\nthat I think should be mentioned and possibly compared to as well.\n3. What are the additional pre-processing time/space requirements in solving for low-rank compression? Some analysis of this would be good, and further comparing the tradeoffs between performance, pre-processing time, and inference time/space.\n4. Tables should include some bolding or highlighting otherwise it's hard to tell what the reader is supposed to look at.\n5. Figure and table captions can be made more informative and provide standalone arguments.\n\n================Post rebuttal================\n\nI thank the work done by the authors during the rebuttal. However, my main concerns regarding novelty and experimental comparisons still stand - I think it is insufficient to say that your method is 'complementary to most if not all existing methods, so we could easily combine the proposed method to the others to further accelerate' without showing this or comparing to current approaches. The other reviewers have also brought up important concerns regarding experimental details which I concur with.", "extra_scores": {}, "extra_text": {}}, {"review_id": "X_MB0qeOe8", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to compress large-scale neural language models (e.g., BERT) without losing too much downstream performance. In contrast to prior work that seeks to compress just the parameters W in the standard linear layer h = Wx + b, this work incorporates the input by approximating Wx instead. The authors also extend this idea to approximations of attention, which enables them to experiment with Transformer architectures. Experiments across the GLUE benchmark confirm that the method does not significantly harm downstream performance, and it also seems to result in inference speedups. However, the biggest flaw with this paper is that its timing experiments are reported on a single CPU thread, while the majority of practical scenarios rely on GPUs. There are other issues with the choices made for the timing experiments (large batch size despite using CPU?) which make me skeptical that the method works as well as claimed.  As such, I cannot recommend the paper's acceptance.\n\ncomments:\n- i'm not quite sure how the proposed method is more \"generic\" than existing related work. why does having to \"train the proposed structure from scratch\" (pg 1) reduce the practicality of a method? in general, these types of applications prioritize inference speedups over training time, so i don't get the criticism. i feel like touting the method as \"generic\" is a little misleading; the proposed method here is complementary to prior work on compression / distillation and should be presented as such.\n- the related work comparisons are similarly exaggerated / misleading (sec 2). implying that pruning methods \"might not\" reduce inference speed is strange given that much prior work on pruning BERT-like methods yields significant inference speedups. \n- this method assumes that the data lies in a low-dimensional subspace, which might not be a valid assumption for many tasks / datasets. i would have liked to see more discussion of this, although the paper does evaluate over many different tasks. \n- the GLUE speedup ratios are interesting; do they correlate to something like the complexity of a particular dataset? \n- a HUGE concern with the timing experiments is that they were done on a single thread of a CPU, not a GPU. especially with a large batch size (100 is fairly large), this definitely disadvantages the CPU computation for larger models. DRONE is not very relevant if the same speedups don't occur on a GPU; no one is serving BERT models on CPU. this makes me skeptical of the value of the proposed method, and is by itself enough to reject the paper. \n- it's even more strange because \"DRONE-retrain\" performs 1 epoch of fine-tuning, presumably on a GPU, so why would a CPU be used for timing?", "extra_scores": {}, "extra_text": {}}, {"review_id": "IyBbbPn3Yum", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this paper is to accelerate large-scale NLP models. This paper reduces the computational complexity by exploiting the data distribution. They claim that exploiting the data distribution enables us to perform low-rank approximation on feed-forward networks. Furthermore, they use that idea to reduce the complexity of the dot-product of attention modules. They experimentally show that they achieve faster inference time while retaining original accuracy. In addition, they show that their method can be combined with distillation methods.\n\nStrength\n- This paper is well organized. We easily understand the goal, challenges, and main idea. In addition, they provide an example of their observation which leads to their main idea.\n- They achieve faster inference time with similar accuracy to a given BERT model; they combine low-rank approximation with a distillation method. They show the experimental results of BERT models as well as LSTM models.\n\n\nWeakness\n- Why not report the preprocessing time? It would not be a fair comparison to compare only the inference time.\n- The accuracy loss seems to be non-negligible.\n- In the last line of proof of Theorem 1, how can we get the right equation from the left equation? U and V are column-orthogonal matrices (not orthogonal matrix) so that UU^T and VV^T are not equal to identity matrices. If Theorem 1 is incomplete, need to re-investigate the method and experiments.\n- They sub-sample 10% of the training data to perform low-rank approximation. The ratio affects the performance of a compressed model. I would like to see the performance change with respect to the sampling ratio.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "74bi8yPrzN0", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work introduces a low-rank based compression method, called DRONE, to accelerate the inference of large NLP models. Instead of decomposing weight matrices in a model, DRONE proposes to exploit the low-rank decomposition by considering the input vectors, which can be in a low-rank space. To compress the whole model, DRONE performs low-rank decomposition from the lower layers to the upper layers in a sequential order. DRONE also proposes a way to extend the low-rank decomposition to dot-product attention. In the evaluation, the paper evaluates their approach against BERT-base and LSTM, and shows that their approaches can obtain better latency-vs-accuracy trade-off versus SVD-based compression method.\n\nStrengths: \n- The paper aims to address an important problem in large NLP model inference.\n- It is an interesting idea to apply matrix decomposition by exploiting the low-rank in the input data rather than the weight matrix. \n\nWeakness:\n- The evaluation is inadequate as the baseline does not seem to represent real inference scenarios. \n- Important references are missing, making it not clear the advantage of this work as compared with existing approaches that also compress Transformer networks. \n\nComments:\n\nThe idea of doing data-aware low-rank compression is certainly interesting. However, the evaluation is largely inadequate to make it convincing that the data-aware low-rank compression will bring a benefit in NLP inference. \n\nFirst, it is not clear why a large batch size (e.g., 100) is chosen for inference. In real online inference scenarios, queries come in one-by-one, so the batch size is often just 1 or at most a few in order to meet latency constraint. Is large batch size chosen because DRONE cannot obtain much speedup with small batch sizes? What is the latency when batch size equals to 1? It would be better to add discussions on the choice of batch sizes and show speedup ratios using batch sizes that are close to real settings.  \n\nSecond, the evaluation is done \"with single-thread on CPU\", which does not represent real scenario either and makes the whole evalution less convincing. Modern servers often have multi-core CPUs and GPU with thousands of cores, and DL frameworks often support parallel execution on different hardwares. One caveat is that reducing the width of matrix-multiplication, e.g., using low-rank decomposition, may not necessarily lead to significant latency reduction, as general matrix-multiplication can execute quite efficiently through highly optimized GEMM kernels (e.g., in cuBLAS and MKL) on GPU and CPU. Therefore, it is important to report the performance of DRONE with parallelism enabled. It is important because if the latency reduction is relatively small on real hardware setting, it becomes difficult to justify the 3% accuracy loss on downstream tasks using compression. \n\nFinally, it does not seem to be accurate to claim \"the first work to discuss the generic matrix computation acceleration on NLP tasks\", given that low-rank decomposition has been used quite a bit for model compression (e.g., SVD and its variants), and there is at least some work on applying low-rank decomposition for compressing Transformer networks[1] and LSTM[2,3]. \n\nQuestion:\n\nThe empirical observation that weight matrices in BERT models are not low-rank is interesting. Do you have an explanation on why this is the case? \n\nDoes DRONE require to pre-decide a fixed batch size to apply the closed-form solution as in Equation 3? How does the speedup ratio look like when the batch size is 1? \n\nMinor:\n[3] applies low rank decomposition for LSTMs, not CNN.\n\n[1] Mao et. al. \"LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression\", https://arxiv.org/pdf/2004.04124.pdf, April 2020\n\n[2] Winata et. al. \"On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression\",  https://arxiv.org/abs/1908.09982, 2019\n\n[3] Yang et. al. \"Fast LSTM Inference by Dynamic Decomposition on Cloud Systems\", https://ieeexplore.ieee.org/document/8970702/, ICDM, 2019\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for compressing weight matrices in large scale pre-trained NLP encoders (like BERT) through low-rank decompositions of both fully connected and self-attention layers. The method is used to compress and speedup pre-trained models. Experiments measure timing on a single CPU thread and demonstrate speedups with small loss of accuracy. Reviewers noted the that goal of this paper is potentially impactful. Some reviewers viewed the resulting loss in accuracy as marginal, while others viewed it as more substantial -- a potential downside. Reviewers also raised concerns about the methodology used to measure inference speedup, a critical measure of success. Specifically, timing experiments were done only on a single CPU thread -- while most practical scenarios would almost certainly rely on GPUs -- as a result, positive experimental results are less impactful. Authors updated the paper to include GPU timing experiments, which did show speedups -- though only marginal speedups over the baseline, TinyBERT. Further, reviewers pointed out that there are several other relevant baselines on compression approaches that are not compared with, and that further analysis should be done on the timing/accruacy tradeoff of baseline methods. Finally, reviewers felt that the contribution of the proposed method relative to other approaches that also attempt to compress transformers is not clearly outlined. Weighing these concerns, I agree with reviewers that the paper is not ready for acceptance in its current form. 
Decision: Reject