DRONE: Data-aware Low-rank Compression for Large NLP Models
Authors: Patrick CHen
Hsiang-Fu Yu
Inderjit S Dhillon
Cho-Jui Hsieh
Venue: NeurIPS 2021 Poster (2021)
Keywords: Acceleration
low-rank

Abstract: The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices are not low-rank and thus canonical matrix decomposition could not find an efficient approximation. In this paper, we observe that the learned representation of each layer lies in a low-dimensional space. Based on this observation, we propose DRONE (data-aware low-rank compression), a provably optimal low-rank decomposition of weight matrices, which has a simple closed form solution that can be efficiently computed. DRONE can be applied to both fully connected and self-attention layers appearing in the BERT model. In addition to compressing standard models, out method can also be used on distilled BERT models to further improve compression rate. Experimental results show that DRONE is able to improve both model size and inference speed with limited loss in accuracy. Specifically, DRONE alone achieves 1.92x speedup on the MRPC task with only 1.5% loss in accuracy, and when DRONE is combined with distillation, it further achieves over 12.3x speedup on various natural language inference tasks.

DRONE: Data-aware Low-rank Compression for Large NLP Models
Patrick H. Chen
Hsian-fu, Yu Amazon
Inderjit S. Dhillon UT Austin & Amazon
UCLA Los Angels, CA
Palo Alto, CA rofu. yofu.yu@@mail com
Palo Alto, CA inderjit@cs wtw.us. edu
patrickchendg ucla. aela.edu
Cho-jui, Hsieh UCLA Amazon Los Angels, CA chohsiehêcs ucla. edu
Abstract
The representations learned by large-scale NLP models such as BERT have been widely used in various tasks. However, the increasing model size of the pre-trained models also brings efficiency challenges, including inference speed and model size when deploying models on mobile devices. Specifically, most operations in BERT consist of matrix multiplications. These matrices not low-rank and thus canonical matrix decompositions do not lead to efficient approximations, this paper, observe that the learned representation of each layer lies in low- dimensional space. Based on this observation, we propose DRONE (data-aware low-rank compression), provably optimal low-rank decomposition of weight matrices, which has simple closed form solution that can be efficiently computed. DRONE can be applied to both fully-connected and self-attention layers appearing in the BERT model. In addition to compressing standard models, our method can also be used distilled BERT models further improve the compression rate. Experimental results show that DRONE is able to improve both model size and inference speed with limited loss in accuracy. Specifically, DRONE alone achieves 1. 1.92x speedup on the MRPC task with only 1.5% loss in accuracy, and when DRONE is combined with distillation, further achieves over 12.3x speedup various natural language inference tasks.
Introduction
The representations learned by large-scale Natural I Language Processing (NLP) models such as BERT and its variations have been widely used in various tasks [8, 2, 25, 3, 24). The successes of these large NLP models rely on the usage of large corpus and big models. Indeed, researchers have reported better results with models that have more parameters [31] and number of layers [1]. The increasing model size of the pre-trained models inhibits public users from training a model from scratch, and
also brings forth efficiency challenges, including inference speed and model size when deploying models on mobile devices.
To deal with efficiency issues, most existing work resorts adjusting the model structure or distilla- tion. For instance, [17] used locality-sensitive hashing to accelerate dot-product attention, [18] used repeating model parameters reduce the size and [44] applied pre-defined attention pattern save computation. A large body of prior work focused on variants of distillation has also been explored
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Model Architecture
Parometer Size
Inference Time
FeeéForeerd
&Classes
0. (0.01%)
Feed Ferward 3
768 16a
104 ms (0.24%
Add Karm
Inté-tomedd
3072x 168
'26 ms [29MM)
Fool forward
768 3012
140.4 ms (52.5%)
432.5
12x
Add Karm
Fece-Fomord 0
768
25.3 ms (631%)
Nall Head Attestion
1x 768
1210 (33.4%)
Sxceder Iou!
M6
ns (8.42%)
Figure 1: Illustration of the BERT- computational model. [V] (at bottom of the Parameter Size column) denotes the number of tokens in the model. #Classes (at top of the Parmeter Size column) denotes the number of classes in the down-stream classification task. Input encoding, Feed-forward and Ferdaforwwrd are computed only once and thus do not contribute much to overall time. The inference time (in milliseconds) listed here based on the inference time measured on CPU.
[27, 15, 35, 20, 34, 35, 41, 45, 4]. These methods require specific design of model architecture, or a long training stage and thus it is less straightforward to combine these methods with each other.
In this paper, explore a simpler acceleration method to speed up inference time which can be applied to most existing architectures. As shown in Figure 1, matrix multiplication (feed-forward layer) is fundamental operation which appears many times in the Transformer [36], the backbone architecture of the BERT model. In fact, the underlying computation of both multi-head attention layers and feed-forward layers is matrix multiplication. Therefore, instead of resorting to the complex architecture redesign approaches, we aim to investigate whether low-rank matrix approximation, a classical and simple model compression approach, can be used to accelerate Transformers. Despite its successful application to CNNs [42, 33, 32), at first glance, low-rank compression does not appear to work for BERT since the matrices in both feed-forward layers and attention layers are not low rank (see Figure 2). Therefore, even the optimal low-rank approximation (e.g., by SVD) will lead to very large reconstruction error. This is probably why low-rank approximation has not been successfully used in BERT compression.
In this paper, we propose novel low-rank approximation algorithm to compress the weight matrices even though they are not low-rank. The main idea is to exploit the data distribution. In NLP applications, the latent features (features fed into each matrix mulitplication layer) usually indicate some information extracted from natural sentences, and they often lie a subspace with low intrinsic dimension [5, 32, 22). Therefore, in most of the matrix-vector products, even though the weight matrices are not low-rank, the input vectors lie in low-dimensional subspace, allowing dimension reduction with minimal degraded performance. We mathematically formulate this generalized low- rank approximation problem which includes the data distribution term and provide a closed-fone solution for the optimal rank-k decomposition of the weight matrices. By leveraging the data distribution idea, we propose DRONE (data-aware low-rank compression). Our decomposition significantly outperforms the SVD under the same rank constraint, and can successfully accelerate the BERT model without sacrificing too much test performance. In addition to compressing standard models, DRONE can also be used on distilled BERT models to further improve the compression rate. For example, DRONE alone achieves 1.92x speedup on the MRPC task with only 1.5% loss in accuracy, and when combined with distillation, DRONE achieves over 12.3x speedup on various natural language inference tasks.
2 Related Work
Fast inference is important for deploying NLP models in various applications. Generally speaking, inference efficiency can be enhanced by hardware [30] or lower-level instruction optimization [23]. On the other hand, the main focus of the current research is on using algorithmic methods to reduce computational complexity. These methods can be mainly categorized into two aspects: attention complexity reduction and model size reduction.

---

Feed-Forward Layers
Query and Key Layers
Accumul ated Radio Singular Volues
1.
Layer-14F2
Layer-1ffl
Accumulatte Ratio of Singular Values
10
Layer-1-Key
Layer-1-Query
Layer-12.F11
66
Layer-12-Query
Layer-12FF2
Layer-12-Key
153
182
Percentage of Singular Values
Parcantage of Singular Values
Figure 2: Illustration of the empirical observation that weight matrices in BERT model are not low-rank. The X-axis represents what percentage of singular values; the Y-axis represents sum of singular values connected to the selected ranks divided by sum of all singular values. Ideally, low-rank structure will have a larger area under the curve, meaning that a small percentage of the singular values can explain their total sum. We observe that the sum of the top 50% of the ranks only accounts about 60% of all singular values for matrices in the BERT model. This shows that the matrices do not have clear low-rank structure.
Attention Complexity Reduction
Attention mechanism is the building block of transformer models and has attracted the most attention of researchers recently in the NLP field [36]. Pre-training on large corpus of BERT, transformer- based model, has contributed to state-of of-the-at art performance on various tasks after fine-tuning [8]. Attention sequences of length T. (T,2) in both computational and memory complexity, which yields long inference time when the sequence long. Thus, researchers have focused on reducing the complexity of the attention module. [17] used locality-sensitive hashing reduce the complexity to ( L). [44, 7] pre-defined an attention map to have a constant computational time. [11] progressively eliminated the redundant context vectors within the attended sequence improve efficiency of attention in the last few layers of the model. [38] proposed to train the low-rank attention by choosing rank r < This similar to our work in the sense leveraging low-rank structures. But our method does not require training the model from scratch and can be applied to different modules other than attention. In fact, most of the above methods require special modules and thus need to train the proposed models from scratch. This prohibits the usage of large body of publicly available open models for faster research progress. More importantly, these methods mainly focus on the long sequence scenario. As shown in Figure 1, we have found out that attention module is actually not the main inference bottleneck of inference time in common usage. most, if not all, models of common usages, two layers of large need-forward layer are appended after the attention module which incurs much more computational time. Attention complexity reduction only works when long sequence is used but in current practice this unusual. Thus, in many tasks accelerating the attention module itself does not contribute to significant reduction overall inference time.
Model Size Reduction
Inference speed is also related model compression. In principle, smaller models lead reduction in the number of operations and thus faster inference time. [28] explored pruning methods on BERT models eliminate redundant links, and there is line of research
pruning methods [12, 13, 6, 10]. Quantization methods [43, 14, 19, 9] convert the 32 bits float models into fewer-bitt fixed-point representation and make model prediction faster with fixed point accelerator. [18] reduce the model size by sharing encoder parameters. A large body of prior work focused on variants of knowledge distillation [27, 15, 35, 20, 34, 35, 41, 45, 4]. These methods
different strategies distill information from a teacher network and reduce the number of layers [27] or hidden dimension size [15]. Further, a hybrid compression method by combining matrix factorization, pruning and knowledge distillation is proposed by [21]. Notice that [21] performed SVD for some components and in this paper we propose an improvement over SVD by leveraging input distribution to each layer. Idea of using input distribution to compress model has also been explored in PCN method

---

[37], which is perhaps the closest to our work. However, DRONE differs from PCN in following three aspects. First, PCN only considers input distribution but not weight matrix and thus it's special case of DRONE (i.e., W be an identity matrix in equation (3)). Second, PCN merely does dimension reduction whereas our formulation achieves dimension reduction and low-rank approximation simultaneously. Last, PCN does not guarantee the obtained transformation is the optimal; whereas, DRONE formulates an approximation optimization problem and we provide the optimal solution. Other forms of low-rank learning strategies including initialization and structure pruning were also explored in the literature [39, 16], and we will compare to these baseline methods. Among the Sbooee-mentione methods, quantization requires hardware accelerator to maximally reduce the inference time. Pruning methods can only reduce the model size, but the inference time might not be reduced due to the limitation of sparse operations. Only algorithmic methods such as distillation serve as a more generic inference time accelerating method. We want to emphasize that method is orthogonal to these distillation methods. In fact, the proposed method is an acceleration method that is applicable to all components in most NLP models. In Section 4, we show that DRONE can be combined with the distilled models to further improve the performance.
3 Proposed Method
We now introduce an algorithm for improving efficiency of matrix multiplication. The computation of feed-forward (FF) layer in the attention models can be described
= Wa + b, 0 (h))
(I)
(2)
where W E Rdaxd: and b E Rd are model parameters, x E Rd the latent representation of token, and h Idaa is the intermediate representation before the activation functioo, or-) is the activation function, and E Rda is the output. Assuming the sequence length is L, all the token representations 11... Tr, E Rd will pass through this same operation, so in practice the whole FF layer can be computed by matrix-matrix product V[ term b would broadcast to L input tokens. In practice we will normally have < max (di, d2) (e.g., L 128, d2 = 3072). Notice that applying a--) on h element-wisely costs (LL2)), which is much smaller than the cost of computing W x (O(Ldzdz) )). Therefore, in this paper we focus on reducing the cost of computing Wz to accelerate the computation. A standard way to accelerate this computation is to perform low-rank approximation on W. low-rank approximation can be obatined by using singular value decomposition (SVD), which achieves the best rank-k approximation in terms of Frobenius norm and we can write W as:
b. and the computation of the bias
W USVT UW.& "
with orthogonal matrices E Rd2xxd,, V Rddi and diagonal matrix S E R:daxd,, Uw.
Rd and Vw.& E dR are the rank-k approximation matrices by taking Uw, = US Vw. = vss where S is the square-root of the first & entries of the diagonal matrix Given such an approximation, we can simplify the computation in (1) by
h = Wa b = Uw Vw, Ta b.
After conducting rank-k approximation, the computational complexity reduces from O(22d) to (d2))))))) When is small enough, low-rank approximation not only accelerates the computation [32] but also compresses the model size [26]. However, as shown in Figure 2, matrices in FF layer of BERT do not show obvious low-rank structures. We observe that choosing top 50% rank (e.g.. k = 0.5 min(d1, dz)) can only achieve around 60% of the accumulation ratio of singular values. which implies large matrix approximation error. In the meantime, the complexity is still about (d2di) ) and there is no enhancement of speed.
Even though the matrices in the model are not low-rank, we provide illustrative example to show that a low-rank computation could still exist when data distribution lies in a lower intrinsic dimension. Suppose we have matrix W defined as below and the input æ lies in subspace:
7
0
2
3
17
2
6
7
5
2
1
W
6
1
8
0
3
x E span
2
3
2
5
2
1
2
2
1
2
4
6

---

In this case, W is a full-rank matrix so there is no lossless low-rank approximation of W. On the other hand, the imput data lies a low-rank approximation:
a 2-dimensional subspace so that we could construct the following
0
2
3
17
2
17
9
6
7
5
2
1
aa
6
1
8
0
5
2
b
4
3
2
5
2
1
2
2
1
2
6
43
23
2
17
90
39
2
66
41
-1
0.5
0.5
5
0.5
0
0
0.25
45
37
5
29
VT
which gives rank-2 matrix UVT where W * UVT but Wx UVT x for any x in the low dimensional space. This shows that even if W cannot be approximated, it is still possible to construct a good low-rank decomposition, and the key to exploit the space input vectors.
3.1 DRONE: Data-aware Low-rank Compression
Assuming the input x of the FF layer follows some distribution, instead minimizing the approxima- tion error of the weight matrix (for which SVD is optimal), we want to minimize the approximation error of the outputs. Denoting as the Rd. x matrix where columns of X capture the empirical distribution of the input (when is large), our goal is find projection matrix Vx R and recovery matrix Ux,t RRda such that the output is well approximated. We rewrite (1)
h = WX + WUS, v. N T +
= =WWU,A) Tx + W Wx,V 2x+b
where Wx,A: = WU,, Intuitively, when X lies a lower-dimensional space, we could find such pair by PCA decomposition X to project X onto the subspace that explains the most variance. In this way, instead of considering the decomposition of W, we leverage the distribution of to complete the low-rank approximation.
However, the best way is to consider properties of both and X simultaneously, and we can mathematically present this desideratum by the following optimization problem:
min | - WMX s.t. rank(M) = k, M
where M is the desired rank-k transformation which maximally preserves the results of the matrix multiplication. In the theorem below, we show that there exists closed-form, optimal solution for the above optimization problem. Before stating the theorem, we first introduce some notation. Assuming write W Uw Sw VW
rank(W)) and rank(X)
and X Ux V such that
Sww. 0
Uw
[Uw.r
Üw.]] SS =
Vi
(Viir V..r
0
.x Sx.t
0
Ux [Ux.
Ux.]]
Vx = [Vx.
Vx.e].
0]
In other words, the decomposition Uw Sw ViT and SX are the full-SVD decompositions of W and X, respectively. The matrices Uw.r. Vw.r. Ux,.Vxx, denote corresponding row spaces and column spaces, while Uw.r Vw,,, Üx, and Vx, null spaces. With this notation, we ready state the theorem.
Theorem 1. Assume rank (A) and rank( X) )- = t. The closed solution M of the optimization problem (3)
M Vw,, Sw ZuS
(4)
where Zy is the rank-k truncated SVD of Sw, Ux,SSx
5

---

The proof of Theorem I is provided in Appendix A. We note that since 2₁, is the rank-k truncated SVD of Z, we could also write Z as Uz. V by distributing the top-k singular values of Z into left or right singular matrices. Thus the original computation can be rewritten as:
WX x (WVW,, WW = U'V
(5)
where U = WV w.r Un, and V-T = V2 are two rank-k matrices, and we will replace W by UV
3.2 Extension to Dot-product Attention
Although the optimization problem in (3) is proposed for feed-forward computation, in this section show that it can also be applied to the dot-product part of the attention module. The key computation in the attention layer is to compute pairwise similarity between queries and keys of the sequence:
O = (KY))
(6)
where Y E Rdd x is the batch query data, Q Re is the query transformation matrix, Y e Red xm is the batch key data, K E Rd is the key transformation matrix and n. m are query and key batch sizes, respectively. We again see that the desired low-rank approximation is the solution of the following optimization problem:
min [IQQ))))))))))) (KY) (QY)" s.t. rank(M) = k. M
With QÝ = W and KY = X. we get the following corollary from Theorem I directly.
Corollary 1. Assume rank(QÝ) = and rank( = t. Let QY = Sw Vi and KY Ux be the SVD decomposition of QY and KY respectively. The closed form solution M of the optimization problem (7) is given by
X,r, where Zy is the rank-k truncated SVD of 2 Sw. Vit,, Ux,
M = Vw. W.r 7x5 ZUS UK.
(8)
3.3 Overall Algorithm
We have shown that the proposed DRONE method is generic acceleration module applicable to all parts of neural language models. We summarize the DRONE on feed-forward layer in Algorithm 1. Since in practice we don't 't have the exact distribution of X, we use training data to caleulate the low-rank approximations as described in Algorithm 1. The attention map calculation can be done by the same procedure with W (QY)² and X KY as given by the Corollary
To accelerate the whole model, we need to select appropriate ranks for each component. However, since the approximation of one component affects the distribution of overall representations, the optimal rank for the model requires a complete search of all possible combinations of rank values, which is infeasible in practice. We thus resort simplified approach shown in Algorithm 2. A more detailed description is provided in the Appendix B. In short, as the changes of lower layer parameters will cause the distribution of representation shifts in upper layers, we approximate each component one-by-one in their topological order of the model. In another words, we approximate the model from the lower layers toward the higher layers. Within each layer, we follow the topological order of underlying modules. We provide a total allowed increase of loss ratio r as an input to the Algorithm 2. The syper-parameter r depends on the efficiency and efficacy trade-off which users are willing to pay. The larger the value r, the faster approximation we get at the cost of lower accuracy. We then distribute r into each module Rt,, (allowed loss increase ratio of i-th module of (-th layer in Algorithm 2). The distribution from to each RI, is based on the observed inference time of each module Er, (observed empirical inference time of i-th module of I-th layer in Algorithm 2). The longer a module takes to compute, the more budget is allocated. Overall, total allowed loss and the distributed loss ratio for each module Ri, fulfil the equality (1 r) = II [1 + Rt,).. For each module, if the approximation with certain rank used won't it increase the loss over the ratio (1 + Ri.)). we will use that rank to approximate the module and move on to the next module. The pseudo code also provided in the Appendix to illustrate the process.

---

Algorithm Data-Aware Low-rank Compression of feed-forward layer.
Input:
rank k, training data Dirain. Original weight matrix W, Prediction Model M. Output: Low-rank Approximation U*, V*.
X=
for all batches 26 in Drasn do
Feed the batch of training data 2ú into M and extract the representation æ. z is the representation which will multiplied with W as in (1).
Append to X.
end for
Given X,k and W, solve the optimal low-rank matrices U**V** by (5).
Algorithm 2 Overall Low-rank Model Approximation Algorithm
Input: training data Dirain, original weight matrix W. prediction Model M, total allowed loss increase ratio Observed inference time E, Search grids of ranks for each module G, original Training loss L.
Output: Low-rank Model M.
Distribute allowed ratio r into each module by Emin arg min Er,
E1i + Emin
Br..
En & expl (og(1+r)
Σi, Er.
R4, - Eb Fi,i
for [ 1.... total layers do
for each module m, E Mi do
Wi,i (-th layer parameter of module m, (e.g., 2nd feed-forward matrix in first layer.) for 1.... .Gr,sl do
k + G1,;
U, V +- Algorithm (k, Dtrain, Wi,5, M) M + M with Wi,; replaced by V.V. Evaluate new loss T.new ((trrann)
Lnew L < 1 + RA then
M + M
break:
end if end for
end for
4 Experimental Results
4.1 Experimental Setup
We evaluate DRONE on both LSTM and transformer-based BERT models. For LSTMs, we train a 2-layer LSTM--aased language model from scratch with hidden sizes 1500 on Penn Treebank Bank (PTB) dataset. For BERT models, we evaluate the pre-trained BERT models on GLUE tasks. Various pre-trained models are offered in the open source platform [40]. For BERT models, we use BERT-base models and it contains 12 layers of the same model structure without sharing parameters. Each layer contains an attention module with hidden size 768 and 12 channels, small 768 x 768 Feed-forward (FF) layer followed by 2 larger FF layers (768 x 3072 and 3072 x 768). As shown in Figure 1, these four components consume the most computational time in the BERT-base models.
For the baseline methods, most of the existing work pertaining to low-rank approximation [21, 32] leverages SVD part of the compression procedure. Therefore, our baseline comparison will be the SVD approximation, and our work aims to provide improvement over SVD. We also include the state-of-the-art distillation methods TinyBERT [15] in the comparison and show that the proposed method can be combined with it to further improve the performance. TinyBERT reduces the model

---

Table I: The experimental results of running pret-rained BERT-base model natural language inference tasks (Glue dataset). Each task has its own metric for performance measurement. Accuracy (SST-2, QNLI, RTE and WNLI), F1/Accuracy (MRPC and QQP), Matthew's correlation (CoLA), Matched accuracy/Mismatchedd accuracy (MNLI) and Person/Spearman correlation (STS-B) are used respectively. All the DRONE results are within 3% accuracy loss and show that DRONE can accelerate the whole BERT model across different tasks and devices.
Methods Original
MNLI
QQP
SST-2
QNLI
MRPC
RTE
CoLA
STS-B
84.3
90.9
92.3
91.4
89.5
72.6
53.4
87.8
SVD
74.4
50.8
73.1
52.2
63.8
47.3
12.4
33.6
DRONE
82.0
89.4
90.0
88.5
86.7
70.0
52.5
85.8
DRONE-Retran
82.6
90.1
90.8
89.3
88.0
71.5
53.2
87.8
CPU Speedup Ratio
1.60x
1.25x
1.64x
1.20x
1.92x
1.31x
1.33x
1.52x
GPU Specdup Ratio
1,28x
1.38%
1.45x
1.28x
1,56x
1.33x
1.29%
1.57x
into layers of attention dimension 312 with 12 channels, and the FF layers are downsized to 312 x 1200. As we mentioned above, all the approximation methods need to consider efficiency and efficacy trade-off. In this paper, we follow previous literature [15,, 5] and report the approximation results with about 3% loss in accuracy to compare the performance of all methods.
In real-world applications, NLP models are mostly evaluated on mobile devices or servers with multiple hardware accelerators. Thus, we measure the inference speed on both CPU (Intel(R) Xeon(R) CPU E5-240 v4 @ 2.40GHz) and GPU (GeForce GTX 1080 Ti) devices. All the experiments are repeated 10 times. The average single sequence prediction inference time in milliseconds is reported in the results. We want to emphasize that unlike many of the literature [38, 17, 7]. which reported speedup only in the attention layers, our results reflect end-to-end speedup including both attention module and feed-forward layers. To perform the approximation, empirically we found randomly sub-sample 10% of the training data suffices to provide good results. Using more data can only provide limited performance boost but comes at a higher cost of longer preprocessing time. Thus, we will use 10% random sample of the training data to perform the experiments. After the proposed data-aware low-rank distribution, we slightly fine-tune the model to further improve the performance. We use a relatively smaller learning rate 10-7 and retrain I epoch on the sub-sampled training data to complete the fine-tuming procedure.
4.2 Results of BERT Models on GLUE Dataset
We summarize the results of DRONE GLUE tasks in Table 1. Detailed inference time of each component of the compressed Transformer model is listed in the Appendix. Detailed inference time of an uncompressed BERT-BASE model can be found in Figure 1. We observe that each task exhibits different difficulty. The best acceleration we can achieve is nearly twice as fast (1.92x) with less than 2% accuracy loss after retraining (on the MRPC). In addition, DRONE achieves 1.52x acceleration without accuracy loss on the STS-B task. By applying the same selected rank for each module with SVD method, we can observe that the performance drops significantly. This shows that the matrices within the model is generally not low-rank; thus the direct low-rank approximation without considering data distribution does not work. On GPU, we see that the acceleration is more or less the same as on CPU except MNLI and MRPC tasks. This is due to the fact that GPU uses massive parallelism and low-rank approximation introduces a sequential computation which might hinder the speedup depending on the size of the matrices and ranks used. To resolve this problem, low-level cuda code optimization is needed and system researchers have studied the problem [29], which out of the scope of the present work. Despite this, we can still observe that DRONE performs better on QQP, RTE and STS-B and it provides about 1.5x acceleration for various tasks on GPU. An example of ranks used in SST-2 is listed in Appendix F.
4.3 Combination with Model Size Reduction Methods
Our proposed DRONE is a general low-rank approximation technique, and it is complementary to many other model compression methods. To illustrative its power, we now demonstrate that DRONE can be combined together distillation. The discussion of combination with Quantization is left in the Appendix. Distillation methods compress the underlying model into a smaller one without losing much accuracy. Distilled models are much smaller in number of layers or hidden dimension, resulting in smaller model size and faster inference time. As shown in the Table 2, TinyBERT, one of the most competitive distillation methods, indeed achieves good performance within 3% accuracy loss for some of the GLUE tasks. Due to the fact that the computation inside the distilled model is still
8

---

full matrix computation, DRONE can be applied to find data-aware low-rank approximation of these smaller matrices. Results are summarized Table 2. As we can see combining DRONE with the distillation method further reduces the inference time without sacrificing accuracy. In particular, on the SST-2 task DRONE + TinyBERT speeds the inference time from 11.7x to 15.3x on CPU while achieving the same accuracy as the TinyBERT. Similarly, DRONE TinyBERT speedups GPU results with 10.9x STS-B and 9.7x on SST-2 with competitive performance. These results again show that the proposed method has the potential to be applied under various scenarios and hardware devices to achieve a better model inference time speedup.
Table 2:
The average inference time (in milliseconds) in comparison to distilled models on CPU and GPU. The unit is in millisecond. The results show that DRONE can be combined with distillation to further improve the performance. Compared to the state-of-the distillation method, the speedup ratio increases from 11.4x to 14.2x STS-B and from 11.7x 15.3x SST-2.
Tasks
Models
CPU speedpe
GPU-speedup
Accuracy (%)
BERT
1x
1x
87.8
STS-B
TInyBERT
11.4x
8.6x
86.9
DRONE + TinyBERT
14.2x
10.9x
87.0
BERT
XX
1x
72.6
RTE
TIR
L88X
1.9X
70.8
DRONE +inyBERT
2
2.2x
71.7
BERT
89.5
MRPC
TinyBERT
11.6X
7.8x
86.3
DRONE +TinyBERT
12.3x
8.6x
86.7
BERT
92.3
SST-2
Tiny BERT
11.7x
90.7
DRONE + TinyBERT
15.3x
9.7x
90.7
Table 3: Illustration of SVD fine-tuning MRPC, RTE, CoLA and STS-B. Using the same rank as the proposed DRONE method, SVD accuracy will drop significantly after the approximation. After fine-tuning done the SVD approximation, the accuracy could be recovered for some tasks (e.g.. MRPC), but SVD + Retrain still perform much worse than DRONE across all the tasks.
Models BERT 89.5
MRPC
RTE
CoLA
STS-B
72.6
53,4
87.8
OROON-Reetain
88.0
71.5
53.2
87.8
SVD
63.8
47.3
12.4
33.6
SVD-Retrain
85.8
63.5
24.4
66.3
4.4 Comparison to Structured Pruning Methods.
Instead of compressing model after training to accelerate inference time, another line of research called Structured Pruning tried learn the low-rank structure during training the model save both training and inference time simultancously. This raises the question if post- post-processing such as DRONE is necessary no further compression is required once we can get a small model after training. Thus, it's worth comparing DRONE with the state-of-the art Structure Pruning method [39]. In [39], attention modules are not approximated by low-rank matrices. To make the comparison fair, we apply DRONE on base models except the attention module and keep others the same. For MRPC, DRONE achieves 1.58x speedup with performance drop from 89.5 89.4. [39] achieves 1.43x with performance 88.61. For SST-2, DRONE achieves 1.41x speedup without sacrificing performance (92.3). [39] also achieves about 1.41x speedup with the performance 92.09. Thus, we can see that DRONE performs better than structured pruning.
One further question is that if we can use the idea of DRONE do fast training? We conducted DRONE on the pre-trei RTE task before fine-tuning to get low-rank structure, and then apply the regular end-to-end training over this compressed model. We found out this procedure with directly using the same rank as in our experiments ( with 1.38x speedup) can only achieve 68.2 accuracy. But if we reduce the compression ratio into 1.2x training time speedup, this procedure can give us 72.9 accuracy. On the other hand, the same procedure with SVD as the initilialization of low-rank structure can only get 52.1 accuracy. This preliminary experiment shows that in addition inference time acceleration, DRONE also has the potential to be applied in the training, but directly transport

---

DRONE into training can not lead to the optimal result. How to improve low-rank training is an interesting future direction.
4.5 Additional Experiments on Large-Scale Models and Language Generation Tasks
Concerns might be raised that if DRONE can also be generalized to other scenarios such as larger models or other NLP tasks. To validate DRONE on larger models. We conduct experiments on RTE dataset with BERT-LARGE model. BERT-LARGE doubled number of layers and the dimension is increased from 768 to 1024. Average inference time for a data increased to 1405ms and achieves accuracy 74. The overall result is 72.9 with inference time 101 8ms (1.38x speedup). This result is comparable to our BERT-BASE result (1.3lx speedup). To validate DRONE on other NLP tasks, We conducted the method on the machine translation task via OpenNMT. It provides 2-layer transformer model on en-de translation. On the transformer part, DRONE achieves 1.76x speedup with BLEU from 33.47 to 33.26. Through these two additional experiments, can validate that the proposed DRONE is generic in the sense that so long as the underlying model is composed of matrix computation, DRONE can compress the model regardless of model sizes and target tasks.
4.6 Can we directly learn low-rank structures by end-to-end training?
From an optimization perspective, a natural question to ask is whether the same optimal low-rank structure could be learned by end-to-end tin-tuning once the rank is decided. We conduct experiments on 4 tasks to verify this, and the results are summarized in Table 3. We start by performing DRONE on the task to achieve the desired accuracy, and perform SVD with the same of ranks. Accuracy of SVD drops significantly for all tasks. We then fine-tune peper-parameters as in [40]' to fine-tune the above SVD results. After fine-tuning, the accuracy improves across all tasks, but none of it can reach the same performance as DRONE. This shows that due to the difficulty of optimizing a non-convex objective function, fine-tuning the SVD result may not achieve the best low-rank result. On the other hand, the proposed DRONE method under the the optimization problem (3) can obtain the provably optimal low-rank approximation at a much lower computational cost than the fine-tuned SVD.
4.7 Pre- pre-processing of DRONE not too costly
DRONE accelerates inference speed at the cost of a pre-processing step. Thus, It's natural to ask DRONE will take long pre-processing time. Given the rank, prep-rocessing of DRONE has a one-time distribution extraction plus low-rank solving of equation (3). Depending on training data size, first stage takes 2 mins (RTE) 20 mins (MNLI). Second stage has SVD computations and is about 5-10 mins. The matrix size involved is limited as we subsample training data. SVD costs about mins and retrain costs from 5 mins to hours depending on training data size. Thus, pre- pe-processing of DRONE is about the same order as SVD but with much better performance. Distillation such as TinyBERT firstly has general distillation of BERT-base on large data used train original BERT, followed by task-specific distillation. Despite task-specific one is rather fast (15 mins to 2hrs), first stage takes a few days for a single GPU, Overall, DRONE is not costly compared to other methods.
5 Conclusions
In this paper, we propose DRONE, a data-aware low-rank approximation, to achieve a better low-rank approximation in BERT models. DRONE leverages the fact that data distribution in NLP tasks usually lies in a lower-dimensional subspace. By considering the data distribution, we propose a data-aware low-rank approximation problem and provide a closed-fome formm solution. Empirical results validate that DRONE can significantly outperform the vanilla-SVD method, and can achieve at least 20% acceleration with less than 3% accuracy loss. When DRONE combined with distillation methods, it further achieves up to 15.3 times acceleration with less than 2% accuracy loss.
https:/hwwggggggee....oommmmmmee 1/e.aplless /ww./pple..tmmttaap......eee html#glue
10

---

6 Acknowledgement
We would like to thank the anonymous reviewers for their helpful comments. This work is supported in part by NSF under IIS-1901527, IIS-2008 173, IIS-2048280.
References
[1] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages
2019.
[2] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Europeun Conference Computer Vision, pages 213-229. Springer, 2020.
[4] Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang. Wei Lin, and Jingren Zhou. AdaBERT: Task-adaptive BERT compression with differentiable neural architecture search. arXiv preprint arXiv:2001.04244 2020.
[5] Patrick H Chen, Si Si, Sanjiv Kumar, Yang Li, and Cho-Jui Hsieh. Learning to screen for fast Softmax inference on large vocabulary neural networks. arXiv preprint arXiv:1870.22400, 2018
[6] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang. and Michael Carbin. The Lottery Ticket Hypothesis for Pre-trained BERT Networks. arXiv preprint arXiv:2007.12223 2020.
[7] Rewon Child, Scout Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.0 10509, 2019.
[8] Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arxiv:1810.04805, 2018
[9] Zhen Dong. Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEF/CVP International Conference on Computer Vision, pages 293-302, 2019.
[10] Mitchell A Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the effects of weight pruning transfer learning. arXiv preprint arXiv:2002.08307, 2020.
[11] Saurabh Goyal, Anamitra Roy Choudhury, Saurabh M Raje, Venkatesan T Chakaravarthy, Yogish Sab- barwal, and Ashish Verma. POWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination.
[12] Song Han, Huizi Mao, and William J. Dally. Deep Compression: Pruning, Trained Quantization and Huffman Coding. CoRR, Compressing Deep Neural Network with 2015.
[13] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both Weights and Connections for Efffcientt Neural Networks. CoRR, abs/1 506.02626, 2015.
[14] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint 1mXvv:1609.070/1 2016.
[15] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. TinyBERT: Distilling BERT for natural language understanding. arXiv preprint arXiv: 10351, 2019.
[16] Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. arXiv preprint arXiv:2105.0102, 2021.
[17] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The Efficient Transformer. arXiv preprint arXiv:200J.0444 2020.
[18] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. AIBERT: A lite BERT for self-supervised learning of language representations. arXiv preprint 10Xi4,9909.1194,2220 2019.
11

---

[19] Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy, Fixed point quantization of deep convolutional networks. In International Conference on Machine Learning, pages 8849-2858. 2016.
[20] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. FastBERT: a distilisg BERT with Adaptive Inference Time. arXiv preprint arXiv:2004.0211 02178, 2020.
[21] Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang. Yaming Yang. Quanlu Zhang, Yunhai Tong, and Jing Bai. LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression. arXiv preprint arXiv:2004.044 04124, 2020.
[22] Jeff Mitchell and Mirella Lapata. Language models based semantic composition. In Proceedings of the 2009 Conference on Empirical Methods Natural Language Processing: Volume I-Volume 1, pages
430-439. Association for Computational Linguisties, 2009.
[23] Emma Ning. Microsoft open sources breakthrough optimizations for transformer inference on GPU and CPU. tinyurl. cot.://ddeommm 2020. Accessed: 2010-09-30.
[24] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018.
[25] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint Ar1iv:1910. 10683, 2019.
[26] Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with nigh-dimensional output targets. In Acousties, Speech and Signal Processing (ICASSP). 2013 IEEE International Conference on, pages 6655-6659, IEEE, 2013.
[27] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
[28] Victor Sanh. Thomas Wolf, and Alexander M Rush. Movement Pruning: Adaptive Sparsity by Fine-Tuning. arXiv preprint arXiv:2005. 07683, 2020.
[29] Achal Shah and Angshul Majumdar. Accelerating bonkrank matrix completion gpus. In 2014 Interna- tional Conference on Advances in Computing, Communications and Informatics (ICACCI), pages 182-187. IEFE, 2014.
[30] Ahmad Shawahna, Sadiq M Sait, and Aiman El-Malch. FPGA-based accelerators of deep learning networks for learning and classification; A review. IEEE Access, 7:7823-7859, 2018.
[31] Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koamantakool, Peter Hawkins, HyoukJoong Lec, Mingsheng Hong, Cliff Young, et al. Mesh-tensorflow: Deep learning for supercomputers. In Advances in Neural Information Processing Systems, pages 10414-10423, 2018.
[32] Kyuhong Shim, Minjae Lee, Iksoo Choi, Yoonho Boo, and Wonyong Sung. SVD-sofimax: Fast softmax approximation on large vocahulary neural networks. In Advances in Neural Information Processing Systems, pages 5463 1463-5443, 2017.
[33] Vikas Sindhwani, Tara Sainath, and Sanjiv Kumar. Structured transforms for small-fnotprint deep learning. In Advances in Neural Information Processing Systems, pages 3088-3096, 2015.
[34] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compres- sion. arXiv preprint arXiv: 090550935,,0.. 2019.
[35] Zhiqing Sun, Hongkun Yu. Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task agnostii BERT for eesource-limited devices. arXiv preprint arXiv:2004.02984 2020.
[36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998 6008, 2017.
[37] Ruger Waleffe and Theodoros Rekatsinas. Principal component networks: Parameter reduction early in training, arXiv preprint arXiv:2006. 13347, 2020.
[38] Sinong Wang, Belinda Li, Madian Khabsa. Han Fang, and Hao Ma. Linformer: Self Attenttoon with Linear Complexity. arXiv preprint 200.:2006604768, 2020.
12

---

[39] Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruming of large language models. arXiv preprini 9rX1v:1110.0732,, 2019.
[40] Thomas Wolf, I. Debut, V Sanh, Chaumond, C. Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. Huggingface's 's transformers: State-of-the-art natural language processing. ArXiv, abs/19 2019.
[41] Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou, thehessu::: Compressing BERT by progressive module replacing. arXiv preprint arXiv:2002.0229) 2020.
[42] Xiyu Yu. Tongliang Liu, Xinchao Wang, and Dacheng Tao. compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370-7379, 2017.
[43] Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat, Q8BERT: Quantized 8bit BERT. arXiv preprint 1rXi::910000688,, 2019.
[44] Biao Zhang, Deyi Xiong, and Jinsong Su. Accelcrating neural transformer via an average attention network. arXiv preprint arXiv: 1805. 00631, 2018.
[45] Sangiang Zhao, Raghay Gupta, Yang Song, and Denny Zhou. Extreme language model compression with optimal subwords and shared projections. arXiv preprint arXiv: /909. 1687, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]. See Appendix A.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] Results are mostly in Table format it's not appropriate to show error bars.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "kwlc_si1-Je", "reviewer": "Reviewer_i8zU", "summary": "The paper proposed a data-aware low rank compression for BERT. It is stated that data-aware low rank compression could perform better than standard low rank compression like SVD since weights for matrix multiplications in BERT are not low-rank and SVD will fail. It seems that the motivation makes sense, and experiments evidenced its effectiveness.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nIt is not fair to say 12.3 times speed up in the Abstract, because  DRONE is used on TinyBERT but it seems that the speedup is calculated upon standard BERT. The speedup mainly comes from TinyBERT.\n\nThe example in Sec.3 (between the line 135 and 136) assumes that X is low rank. I am wondering whether this is true in real-world applications.\n\nIt seems that data-aware low rank compression is not specific to pre-trained language models. Is there any previous work that evidences its effectiveness in some simple scenarios like MLP or CNN, especially models are not needed to be pre-trained? By doing so, it may better investigate the behavior of  \"data-aware low rank compression\".\n\nCan the authors show some information about the ranks {k} in different weights?\n\nIn Algo.2 both m_i and G are called 'module'. What is  G like? In the second for-loop, we find a subscript i in M and we also find i in the third for-loop.\n\n\nIn Table 2, why speedup in tinyBERT varies in different tasks? And RTE, why is tinyBERT only 1.8x times faster than BERT? but in other tasks, 10 more times speedup is achieved.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is nothing related to Societal Impact. The Limitation is that the speedup is not notable while the performance is a little bit worse. This limits its applications in the real world. But the idea itself is intertesting.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "d_EgaC-Z11K", "reviewer": "Reviewer_AXUP", "summary": "This paper proposes DRONE, a data-aware low-rank compression algorithm for BERT, achieving a speed-up with marginal accuracy loss.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "DRONE effectively reduces the computational cost with the observation that intermediate feature vectors lie on a low-dimensional manifold. DRONE significantly outperforms the vanilla SVD method.\n\nI presume how to distribute r into each module R_{l, i} is important for the final performance. Algorithm 2 is duplicated in Section 3.3 and Appendix B. Although there are explanations about Algorithm 2, it would be better to describe its rationale. I think the method is somewhat heuristic. Do you have any results on other design choices? Small errors in a lower layer can be propagated to a higher layer resulting in a big difference to the final representation. Is there any consideration for this effect in the proposed method? Also, in this sense, I wonder whether DRONE is also effective in deeper networks (e.g., BERT-Large).\n\nThe combination of DRONE with other compression methods is practically valuable. The authors show that DRONE can be used orthogonally with knowledge distillation and quantization. DRONE could be compared with Linformer which also leverages low-rank structures.\n\nDetailed analysis on ranks for each position (self-attention or feed-forward / layer) would give an insight for a future design of new architectures or compression methods.\n\nFigure 2 only shows singular values at the very first and last layer. I am curious whether the same trends appear at intermediate layers. Considering a space limit, the authors may report the accumulation ratio of singular values choosing the top 50% rank, for instance.\n\nMinor: It is strange that the body of the appendix is aligned to the center.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Developing a better algorithm for the distribution of loss ratio would be an interesting research direction. Instead of equally distributing in a deterministic manner, dynamic programming based method can be used for optimal distribution.\n\nInspired by many works on iterative pruning, I am curious whether DRONE can be performed iteratively (e.g., DRONE → finetune → DRONE → ...) for better final performance despite the increased computational cost.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}, {"review_id": "Z6Np7aR4seG", "reviewer": "Reviewer_Sxsk", "summary": "This paper introduces DRONE, which is a data-driven low-rank decomposition method. Experiments are conducted using LSTM on PTB dataset and BERT on GLUE tasks. The method performs better compared to naive SVD.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The method is simple and orthogonal to other methods.\n- Writing is clear and straightforward.\n- Authors report the performance of DRONE combined with distillation and quantization methods, which demonstrate the potential for production.\n\nWeaknesses & Questions:\n- For the detailed time cost (of BERT), I only can find the numbers of DRONE, can you also provide the costs of baseline?\n- How do you measure speed-up? What batch-size is used? Line 231-232 looks not very clear. I guess you use large batch-size for evaluation, then use the average time per sample? If yes, can you also provide the speed-up of single sample inference? (As single-sample inference time is important to real-world)\n- In real-world products, we often need to re-train / re-finetuning the model by the latest data, due to distribution changes. As DRONE is data-depend, will distribution-change cause more performance loss?\n- Did you try DRONE on language generation tasks? How does it perform?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "OulEuVl7Ikn", "reviewer": "Reviewer_o4hz", "summary": "This paper proposes DRONE, a data-aware low-rank model compression method designed specifically for large language models, e.g., BERT. The method is motivated by an observation that the weight matrices in a pre-trained BERT model are generally not low rank, but the data/intermediate results usually lie in a low-rank space. DRONE leverages the above observation and arrives at a close form solution to approximate the layer weights using low-rank factorization. Theoretical analysis is provided and experimental results are shown to demonstrate that DRONE successfully accelerate the inference speed of BERT model on a various range of down stream tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well-written, and the research direction of compressing the large language models and improving the inference speed is promising.\n2. The experimental results are promising. I am convinced that DRONE can work well in practice.\n\nCons:\nMy major concern on the proposed method is about its novelty. The data-aware low-rank compression method has been explicitly proposed in PCN [1] (which is not even cited by this paper). In my view, DRONE is almost identical to PCN apart from the fact that PCN has not been deployed in language models yet. The authors are expected to elaborate the novelty of DRONE further. Other concerns are summarized as follows:\n1. Decomposing one layer into two small low-rank layers can effectively increase the depth of the model (which reducing width), which is known to make the model harder to train. In such a sense, how dose the training behavior of DRONE look like? And how does that compare to the vanilla BERT training.\n2. On the memory perspective of view, deploying low rank approach reduces the model size. But the activation values for each factorize layers need to be saved. How does the memory footprint look like?\n3. It has been studied that starting from a “warm-up” stage of full-rank model can be helpful for the low-rank model [2-3]. Does it also help DRONE?\n4. In the current draft, DRONE is not compared with any other pruning/sparsification based methods. How does DRONE compare to the structured pruning method, e.g., [3]?\n5. The current experiments only study BERT models, whose scales are not that large. How does DRONE perform on GPT-2/ViT/MLP-Mixer models?\n6. It is easy to imagine that one can use DRONE to speed up the BERT pre-training stage. Adding even an preliminary result on such a direction can be really helpful.\nMinor issues:\nMissing references: [1-2], [4].\n\n[1] https://arxiv.org/abs/2006.13347\n[2] https://arxiv.org/abs/2103.03936\n[3] https://arxiv.org/pdf/1910.04732.pdf\n[4] https://openreview.net/forum?id=KTlJT1nof6d", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "My major suggestions are summarized in the “Cons” of “Main review”. Please take a close look on it. Moreover, as the state-of-the-art architectures in computer vision and NLP fields tend to be unified, e.g., Transformers/pure MLP networks. It would be important to include both NLP and computer vision results, e.g., ViT/MLP-Mixer on CIFAR and ImageNet to show that the method generalize well.\n\nI am happy to increase my overall evaluation scores if my concerns are addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not believe that this paper raises any ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes DRONE, a data-aware low-rank compression algorithm for BERT, achieving a significant speed-up with marginal accuracy loss. The authors show that DRONE can be used orthogonally with knowledge distillation and quantization. In this sense, the paper has its clear value to the community. The reviewers raised some concerns on the technical novelty, the precise quantification of the speed-up, and the experimental details. Overall speaking, the authors did a good job in their rebuttal, and some of the reviewers’ concerns were successfully addressed. The reviewers made several rounds of discussions, and the conclusion is that it would be good to accept the paper as a poster. 
Decision: Accept (Poster)