Exploring extreme parameter compression for pre-trained language models
Authors: Benyou Wang
Yuxin Ren
Lifeng Shang
Xin Jiang
Qun Liu
Venue: ICLR 2022 Poster (2022)
Keywords: pre-trained language models
tensor decomposition
compression
BERT

Abstract: Recent work explored the potential of large-scale Transformer-based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language processing. This raises many concerns from various perspectives, e.g.,  financial costs and carbon emissions. 
Compressing PLMs like BERT with negligible performance loss for faster inference and cheaper deployment has attracted much attention. In this work, we aim to explore larger compression ratios for PLMs, among which tensor decomposition is a potential but under-investigated one. By comparing existing decomposition methods, Tucker decomposition is found to be parameter-efficient for compression.  Two decomposition and reconstruction protocols are further proposed to improve the effectiveness and efficiency of Tucker decomposition in parameter compression.
Our compressed BERT with ${1}/{7}$ parameters in Transformer layers performs on-par with,  sometimes slightly better than the original BERT in GLUE benchmark. A tiny version achieves  96.7\%  performance of  BERT-base with $ {1}/{48} $ encoder parameters (i.e., less than 2M parameters excluding the embedding layer) and  \textbf{$2.7 \times$} faster on inference. To show that the proposed method is orthogonal to existing compression methods like knowledge distillation, we also explore the benefit of the proposed method on a distilled BERT. 

Published conference paper at ICLR 2022
EXPLORING EXTREME PARAMETER COMPRESSION FOR PRE-TRAIE LANGUAGE MODELS
Benyou Wang" University of Padua wang@dei. unipd.it
Yuxin Ren"
Tsinghua University
ryx200mai 1a. tsinghua. edu. en
Lifeng Shang, Xin Jiang, Qun Liu
Huawei Noah's 's Ark Lab Shang. feng, Jiang. Xin, qun www.:liihhwwei.ccom
ABSTRACT
Recent work explored the potential large-scale Transformer-ban based pre-trained models, especially Pre-trained Language Models (PLMs) in natural language pro- cessing. This raises many concers from various perspectives, e.g., financial costs and carbon emissions. Compressing PLMs like BERT with negligible performance loss for faster inference and cheaper deployment has attracted much attention. In this work, aim explore larger compression ratios for PLMM, among which tensor decomposition a potential but under- investigated one. Two decomposition and reconstruction protocols are further proposed to improve the effectiveness and efficiency during compression. Our compressed BERT with 1/7 parameters in Transformer layers performs on-par with, sometimes slightly better than the origi- nal BERT GLUE benchmark. tiny version achieves 96.7% performance BERT-bas base with 1/48 encoder parameters (i.e., less than 2M parameters exclud- ing the embedding layer) and 2.7x faster on inference. To show that the proposed method is orthogonal to existing compression methods like knowledge distillation, we also explore the benefit of the proposed method on distilled BERT.
INTRODUCTION
Pre-trained Language Models such as BERT (Devlin el al. 2018) and ALBERT (Lan et al. (019) have significantly improved various NLP tasks with significant improvement. Much recent work (Brown al. 2020 Narayanan al. 2021 Fedus al. (021) explores the potential of super large-scale PLMs. However, such large-scale PLMs are both economically and ecologically unfriendly (Bender al. 2021, Patterson et al. 2021). Furthermore, deployment of large-scale PLMs is also challenging since (I) model cannot be fully deployed or stored in single GPU server, model parallelism would consume extra time for network communication among many servers; (2) edge devices may not have enough space for storing models; (3) the long inference time cannot support real-time feedback.
Scaling down model with negligible performance drop would facilitate the real-world applications of PLMs in smaller size, faster inference time, and less network communication cost. For example, recent work explores quantization (Zhang et al. 22000; Bai et al. 2000)), weights pruning (Hou et al., 2020), and knowledge distillation (Jiao al. 22200) [anh et al. 2020) for BERT (one of the most popular PLMs). We argue that existing methods cannot largely compress large-scale PLMs stated in Sec. 2 In this paper, we aim to explore extreme parameter compression (i.e., bigger compression ratios) although they are by definition challenging.
The parameter redundancy in PLMs demonstrated by (Kovaleva al. 2019: Michel et al. 2019; Voita et al. 2019, Cordonnier et al. 2021), for which we divide into two groups: intra- matrix redundancy and inter-matrix redundancy. The former happens in different heads that are
Benyou and Yuxin contributed this work equally.
Lifeng is the corresponding author.
ht tps: / / thub. tw nkk ee033' /xcorpresston

---

Published a conference paper at ICLR 2022
calculated separately, e.g., attentions among heads act on a similar subspace and are therefore low- rank (Cordonnier et al. (021) we relate this phenomenon to the so-called decomposaaiiltty" detined in this paper. Like self-attention layers, decomposability also holds in FFN layers each FFN layer could be decomposed to many independent sub-FFNs (as explained in Appendix BB. One example of inter-matrix redundancy happens across different layers, e.g., attention maps among layers might be
similar (Clark et al. 2019 Vig 2019,
Rogers et al. 2020).
Exploration of main weight matrices in Transformer layers finds that these weight matrices are possible to be approximated in low-rank manner evidencing the possible intra-malrix redundancy and inter-matrix redundancy. We comprehensively analyze and compare different decomposition methods for parameter compression including matrix decomposition (denoted I1), tensor train decomposition (Oseledets) (01T)) (denoted as III) and Tucker decomposition (De Lathauwer et al. 2000) (denoted as IV), The fundamental difference between them is as below. Il conducts matrix factorization (c.g.. SVD) for each weight matrix thanks Intra-matria redundancy. Regarding inter- matrix redundancy, III shares the head and tail matrices while keeping the core matrix individual; IV introduces 'matrix bank' to make parameter scale being nearly constant w.r.t. the number of layers. It is concluded that Tucker decomposition (IV) is more parameter-efficies than others in terms of compression ratios. ALBERT (Lan et al. 2019) and can be considered as special cases IV.
The practical challenges of matrix/tensor decomposition for compression are twofold. First, the decomposition may result in a discrepancy between the raw weights and approximated weights, and exact decomposition impossible with large compression ratios. Instead, Knowledge Distillation (KD) is used on the compressed model to simulate the predictions of the raw model in a loss-aware manner. Second, reconstruction may lead to additional computation costs. An efficient reconstruction protocol is implemented by reordering multiplication operations that also preserve the results.
The contributions of this work are (1) we propose formal framework with standardized terminology to comprehensively discuss matrix/tenso decomposition methods to compress Transformer- language models; (2) we adopt tensor decomposition for compressing PLMs which is also faster, while existing work (Ma et al. 2019 Liu al. 2021) did not show the potential for speedup in PLMs; (3) our compressed BERT with 1/7 parameters in Transformer layers performs on-par with the original BERT in GLUE benchmark. Also, tiny version achieves 96.7% performance of BERT-base with only 1/48 parameters in Transformer layers and 2.7x faster on inference. We directly use the proposed methods on TinyBERT (Jiao et al. (020) that is purely based on KD, since our work is complementary existing compression methods like KD.
2 RELATED WORK
Compressing PLMs Although various work was proposed to design new efficient Trans- former (Tay et al. 2020), e.g., (Ma et al., 2019 Choromanski et al., 2020 Wang et al.] 2020 Kitaev et al. 2020 Zaheer et al. 2020; Cao et al. 2020). in this paper, we are focusing on the compression of Transtormer-asser pre-trained language models. The difference is that the latter expects to reuse well-trained models, e.g., BERT and GPT (and even GPT3 (Radford et al. 2019) and PanGu-a (Zeng et al.] 202I)) with manageable computing resources, which typically does not change the original Transformer architecture. Taking BERT, one of the most commonly-used pre-trained language models, as an example. Existing work explored quantization (Zhang et al. 2020 [ai et al.
2020), weights pruning (Lagunas et al. (20T)) knowledge distillation (Tiao et al. [2200] Sanh et al. 2020), progressive module replacing (Xu et al.) (020). neural architecture search (Xu al. 2021
Yin et al. 202I) and matrix decomposition (Noach & Goldberg 2020).
We argue that existing compression methods (see Tab. [ may be inadequate for extreme parameter compression, which is under-investigated The reasons are manifold, first, the knowledge distillation- based method generally Tearns new student model from scratch, which cannot inherit too much knowledge from the teacher model before distillation. Second, some methods have upper bounds of compression ratio. For example, layer-sharing ALBERT (Lan et al. 2019) shares parameters in L layers with maximum L times compression. Quantization replaces existing 32-bit parameters with binary parameters with a maximum of 32 times reduction. Moreover, quantization needs further hardware support, which is usually ad hoc to specific platforms. Weight pruning arguably cannot achieve a big compression ratio (McCarley al. 2019 Michel al. 2019).

---

Published conference paper at ICLR 2022
Table 1: The comparison between compression methods for BERT. L is the number of layers and is the dimension of hidden states. In the "Compressing ratio' column in Quantization, '32' refers that one usually uses 32-bit floating precision for a real number.
Hardware support
Compressing
Methods
Examples
ratio
Knowledge distillatioo
Sun al.
2019
Jino al
2020 Sanh et
2020
0( Td
Parameler sharing
Lam al
2019
as upper bound
Quantization
A020
Bai al.
2020
32 as upper bound
Weight pruning
Matrix decompasition
Lan
2019
Noach Goldberr
2020
0[ T LD 0f 1g
Tensor (Tucker/TT) decompasition
Matrix/tensor decomposition for compression Tensor/matrix decomposition aims approxi- mate given tensor using set of smaller tensors/matrices. It has been investigated to compress and speed up CNNs, RNNs, and Transformers for many years (Lebedev et al. 2014 Yang et al. 2017 Ye eet al. 2018 Denton et al. 2014] Winata al., (2019). Matrix decomposition (e.g., ALBERT (Lan et al. 2019) in embedding layer and (Noach & Goldberg 2020)) could decrease parameter scale with a linear factor depending on the selected rank. More advanced tensor decomposition approaches be implemented by tensor network, which has recently been used to compress general neural networks (Gao et al., [2000] Novikkvv et al. 2015), compress embedding layer (Khrulkov et al., 2019 Hrinchuk et al. 2020; Panahi et al.) 2019).
Recently, Ma et al. (2019) redesigned new 'Self-Attention Network' (SAN) in Transformer architec- ture inspired by block-term tensor decomposition. The compression ratio is limited since the majority of parameters in Transformer comes from another module called 'Feed Ferd-Forward Network' (FFN)
instead of SAN; moreover, the model does not have the potential for speedup regardless of compres- sion ratios since the time complexity is closed vanilla Transformer. Noach & Goldbergi (0020) reparameterized each weight matrix using matrix decomposition and further distill the compressed models, which nearly achieves a compression ratio of 1.7. [Lu et al. (202T) adopted matrix product operators reparameterize each group of weight matrices in embedding. SAN, and FFN, and only a small part of tensors of MPO (called 'auxiliary tensors') are fine-tuned in downstream tasks. The compression ratio of the total parameter is negligible and the inference might be slow. Those works inspire to explore in depth extreme parameter compression for large-scale PLMs.
We argue that compression ratio of existing work using matrix/tensor decomposition (Ma ct al 2019 Liu et al.] 2021 Noach & Goldberg, (2200) for PLMs is relatively-small; most of them do not have speedup effect, limiting their applications in large-scale PLMs. The potential to compress PL.Ms with matrix/tensor decomposition is under-inve investiiitt.....d.. In this work, we adopt tensor decomposition, cubically compress the parameters of PLMs.
MOTIVATIONS FOR PARAMETER COMPRESSION
Pre-trained language models are typically stack of multiple Transformer Vassann et al. 2017) layers that consist of Self-Attention Network (SAAN) module and a Feed-Forward Network (FFN) module, see App. A for Transformer. Sec. [3. will introduce an important property called 'decomposability for SAN and FFN, which indicates each sub-component in SAN or FFN is independently calculated without interactions between sub-components and they may therefore be redundant.
3.1 DECOMPOSABILITY IN TRANSFORMER
computing module is decomposable if its sub-components (g1, 92. (0..9g....9a ..HH)) could be indepen- dently calculated without interactions: f(x) = 8(g1 (æ), 92 (x). ""' (x)). Usually, 8 a simple operation that has negligible computing cost compared to (9: ). Especially, backpropagation between sub-components independent if J is concatenation or addition. Sub-components in could be calculated parallel without interactions. We will examine SAN and FFN are decomposable.
Decomposability in SAN Following (Hou et al. 2020), SAN could be decomposed as a sum of the output of every head. For the query/key/value/oune transformations parameterized by

---

Published conference paper at ICLR 2022
caphardd
0.7
0.0
288
120
256
504
700
PCA compor
(a) PCA for each single weight matrix
(b) PCA pair matrices along columns
Figure 1: PCA for existing weight block matrices in BERT- base. We got nearly similar results Fig. for paired matrices along rows and columns, as shown in App.
T / / / we divide them as N₁ heads: (W Nn (W (W,) (W A single head is calculated as
Alle, (X) = Softmax( XW W x jxw wo
(I)
Then SAN(X)= = ΣN Att, (X). indicating SAN is decomposable, Cordonnier et al. (002 argued that the attentions among heads learn redundant key/query projections due to independent calculation.
Decomposability in FFN The two weight matrices in FFN, denoted as 4 and WoOt
FFN(X) = Σ GeLU(XW In [") Out by
Wh, + Out
(2)
A=1
As Geva al. (0200) points out, FFN also operates key-value mechanism similar SAN: can consider the input of FFN as query vector, and two linear layers of FFN as keys and values, respectively. There may exist some similar key-value pairs that may introduce redundancy.
Remarks decomposability Decomposability does not necessarily make sub-components being complementary, especially without orthogonality- like constraints between them. cub-components may learn similar patterns, leading to some redundancy (Michel et al. 2019). For example, HHou et al. (2020) demonstrated that BERT could be pruned in both width and height without performance drop.
Moreover, BERT is pereettreeter in the stream tasks is much fewer (393k training examples for the biggest task in GLUE, i.c., MNLI) comparing to the enormous number of parameters (1 10M parameters for BERT-base) this was thought problematic according to the traditional machine learning theory (Vapnik 2013 We believe that the pre-trained language models downstream tasks could compressed.
that the number of training examples in down-
3.2 EXPLORATION OF AN PRE-TRAINED TRANSFORMER
Beyond the dccomposability of Transformer, we conduct exploration widely-used Transformer- based PLM, i... BERT (Devlin et al. 2018). Technically, we calculate the captured variance ratio by Principal Component Analysis (PCA), as indicator to measure the parameter redundancy.
The main weight matrices of Transformer layers are ( w V W W In Out + Sec. 2 and Appendix B show that FFN could be separately calculated in multi-head fashion. We could sequentially split both (W'") and Owt) into four groups like { (W/ hem and (W-Ouy h-4
"e exclude embedding layer for compression, (Ben Noach Goldberg 2020 did. Note that the lookup time-consuming since it matrix multiplication. Moreover, this paper focuses on the
operation in embedding layers is fast: therefore, decomposing embedding will involves additional computationally-expens. components in Transformer, namely SAN and FFN, which are the majority parameters that also increase lincarly with network depth; while the parameter scale embedding layer is constant w.r.t. network depth.

---

Published conference paper at ICLR 2022
1
P
I: raw weight matrices W! W.
II: matrix decomposition W U.
III: tensor-train decomposition
IV: tucker decomposition W UIRCIV
W = US.
Figure 2: The three methods for parameter compression. To compress the raw weights W', II decomposes each matrix in w' into small matrices, i.., two 'narrow' matrices and a small square matrix. III further shares the two 'narrow' matrices for all weights. IV introduces matrix bank for these small square matrices, making parameter scale nearly constant w.r.f. the number of layers.
respectively, By doing so, we could reconcile all weight matrices be same shape (i.e. D X D). Here, we could get 12 1) x 1) weight blocks for each Transformer layer, for SAN, and 8 for FFN.
In Fig. we could find the intra-matrix and inter-matrix redundancy: Figure la shows that half dimensions could capture more than 90% variance of all weight matrices, this confirms statement Sec. 3. Furthermore, we also study the redundancy between two matrices by conducting PCA on the concatenated matrix between two arbitrarily paired weight matrices. See Fig. [bb] half dimensions could capture nearly 80% variance, which suggests
possibility to compress inter- matrix redundancy. This inter-matrix redundancy may be twofold: (1) subFFNs are decomposable; (2) calculations (e.g.. attention maps) in different layers may be similar. Regarding the latter, some existing works like RNNs (Goyal & Bengio 2020), Neural Ordinary Differential Equations (ODE) (Chen et al.) (0188), and cross-layer sharing ALBERT (Lan et al. (2119) show that it could work even with the parameter equivalence hypothesis among layers.
GENERAL FRAMEWORK FOR PARAMETER COMPRESSION
The observation that the main weight blocks in BERT could be approximated in a low-rank manner (thanks to the intra-matrix and inter-matrix redundancy) inspires us to use decomposition. Here we introduce and compare some standard decomposition methods (see App. in compressing PLMs.
4.1 EXPLORING PARAMETER COMPRESSION
In principle. SANs and FFNs could be separately compressed; in this work, we additionally explore stacking SAN weights and FFN weights together a unified protocol since each weight block has an identical shape (i.e., D) x D). The main weight matrices in the j-th Transformer layer are
w [[w w W. w (W!")" @ ( (r]], B 12XDDD
Weights a L-layers Transformer are stacked as a 3-rd order tensor in R 12LxDxD The original non-decomposed weights is called I: w ( I2LD? Each weight matrix in W is W W; E R DXD Here, we explore standard decomposition methods including matrix decomposition. tensor train decomposition (Oseledets) 2011) and Tucker decomposition (De Lathauwer et al. 2000).
II: matrix decomposition Motivated by intra-matrix redundancy, one can adopt Matrix decom- position factoriize/approximate matrix into some smaller ones. A typical example is singular value decomposition (SVD), called 'Il-a,, for each D D matrix W. w
W, = W. = U,E.V, E R DxD
(4)
One can also drop the diagonal E; by decomposing it into two parts that are multiplied to U, and V; I namely W, U.V., denoted as 'Il-33. U E R Dxd and V, R ddDD and usually d < D. Since the compression ratio is 2nd with reducing the rank from 1) d, the preserved rank of the approximated matrix linearly decreases with the compressing rates.
By decomposing E, into two diagonal matrices, each of which has diagonal elements that are the square Σ. By multiplying these two diagonal matrices U, and V,. W. U.V. R DxD

---

Published conference paper at ICLR 2022
Table 2: Overview of methods. The most expensive term in space complexity is in bold.
Methods
Space complexity
Difference
Reference
12LD"
FAW
Devlin al.
2018
I--
Matrix decorapusition
24LDd + 12
w/ low rank approsimation
Matrix decomposition
24LDd
diagonal matriess
Goldborg 2020
III
Tensor-train decomposition
12Ld² 2Dd
parameser sharing
IV
Tucker decompusition ld + 1211 + 2Dd
w/ mattis bank
III: tensor train decomposition Inspired by the inter-matrix redundancy, one could expect to share weights among matrices. The biggest terms in Eq. Mare U, and V, while (E,) is relatively small since Eg E R dhd and d is relatively small compared to D. We could share (U,) and (V,) among matrices to save parameters. This results
W; W. = UE.V E DXD
(5)
Here, [E,} are not necessarily diagonal. This results in tensor-train (TT) decomposition (Oseledets) 20 One can also consider higher-order TT decomposition (i.e., a longer chain for tensor multiplications) which could be more parameter-effiientt this often needs to reshape the raw tensor into higher-order tensor with heuristics. However, it is more time-consuming and costs more GPU memory during training, which we leave future work.
IV: Tucker decomposition In Eq. the biggest term the {E,} E 12Lxd number of layers may be large in practice (e.g.. L 24 for BERT-lrgge). To this end, we propose fixed-sized matrix bank such that a weight matrix considered linear combination of these matrices inside the bank, making the parameter scale become nearly a constant with respect to the number of layers. Namely,
especially the
W. W! V(P.C)V& E R DxD
(6)
where C E R 1xd
is a matrix bank with size of I, each matrix is assigned with a weight vector P: E R txl ALBERT (Lan et al. 2019) could be considered as a special case of IV, see App.
4.2 COMPARISON BETWEEN I,II, III, AND
The comparison in parameter scale between these decomposition methods is in Tab. 2 Since D > d and I. >>> I, we could generally conclude that the parameter scales decrease from I, IT. III to IV. We can observe that marginal parameter cost to add a new layer in is nearly 127, which is negligible compared to the other parameters. During the inference phase, the terms that do not involve batch size or sequence length n could be calculated in an offline way only once before starting inference, which costs more storage but gets slightly acceleration since the main purpose of this work compress models, we ignore it in this work but encourage doing it in speed-sensitive scenarios.
5 EXTREMELY COMPRESSING BERT USING TENSOR DECOMPOSITION
5.1 DECOMPOSITION PROTOCOL
IV reduces space complexity from O(12DD)) Old + 12L1 2Dd) where d D and 1 12L. I determines to which degree we want share Transformer parameters among all modules, flexible factor to smoothly transform vanilla BERT to layer-shared BERT (or called 'ALBERT" (Lan et al. ). d' determines the expressive power (rank) of each linear transformation (originally 1) x D).
The decomposition protocol does not change the architecture of BERT, atternatively. it introduces a new reparameterizztion of the existing weights. However, the approximated weights W usually
'A traio decomposition is approximate high-order tensor with product of many smaller three- order tensors - except for the first and last ones being matrices. Here, for a three-order tensor W E R 12fxXDDD it approximated by W UGV and shape transpose, where U B Dxrs G e R 22212 and V 22 For specific slice W, W, UG.... V. r and ra are the 'TT ranks".

---

Published conference paper at ICLR 2022
Table 3: Computational complexity with different order of matrix multiplication
Multiplication order
Computing complesity
IV-1
XUV(
O + Dd D²d+ Id")
IV-2
(XV)
Dd2bnDd+ + bnd +1d")
IV-3
(XUX(P,CIV)
O(2bnDd + Dd Id")
are not exactly equal to the raw weights w'. Moreover, the tiny decomposition discrepancy of weight matrices in low-layer may lead to accumulated difference in the final output due to the multiple-layer neural network architecture In this work, we propose to use knowledge distillation simulate the final output of raw models.
fw( = fw (x)
Jwi the raw BERT model and fwe the compressed one. argue that approximation prediction (like knowledge distillation in Eq. 7) is more important than approximation in weights.
Such loss-aware strategy in compression could be found in quantization (Hou et al. 2018).
5.2 RECONSTRUCTION PROTOCOL
A slice of x D parameter block is represented matrix product, w U(P.C)V R DxD For input X E IR booDD where b is the batch size and is the sequence length, output of linear transformation between X and a D x D parameter block will be Y XW (P.C) Since matrix multiplication is associative different multiplication order will not affect the final result but their computational complexity may be different One can see the computational complexity for multiplication order in Tab. 3 In practice, the batch size & will be set as big as possible to increase data throughput and make training more stable, we could conclude that IV-3 is more efficient than 12- IV-3 is more efficient than IV-1 when D > 2d: in practice, D is typically much bigger than d and 1) > 2d. In conclusion, setting IV-3 is most efficient in this scenario.
EXPERIMENTS
6.1 SETTINGS
12, D 768, ). w' E I2 1400 is decomposed into core tensor and three factor matrices (see Fig. 2) its reconstruction could be seen Sec. 5.2
Decomposition For BERT-base (L
Knowledge distillation As Uiao ct al. 2020: Zhang et al., 2020 Bai ct al. 2020) did, we use (wo-stage knowledge distilaaiion for the compressed model. At General Distillation (GD) stage, we adopt Knowledge Distillation (KD) for the compressed model to simulate last-layer hidden states and last-layer attention maps of the general teacher model (BERT-hase). At the second stage, we adopt Task-specific Distillation (TD) simulate the logits of task-specific BERT model (e.g.. fine-tuned on MNLI task). GD, compressed models are trained with two epochs. In TD, we also augment training data by randomly replacing a random word with a similar word according to either word vector similarity using Glove (Pennington et al. (2014) or the predicted logistics of BERT when masking the target word, see more details (Jiao el al. 2020).
GLUE evaluation GLUE (Wang et al. 2018)) (see App. | for more details) includes datasets for single document classification and sentence pair classification. Fine-tuning and evaluation GLUE follows the settings from Huggingface (Wolf al. 2019). The hest-performed model is selected according to the dev set, where we select the learning rate in [1e-5, 2e-5] and batch size in [16, 32].
"Our experiments also show that a direct decomposition results in very low performance, Tab. 6 'For a sequence of matrices (e.g.. G.. [A, B, C)), matrix multiplication with different calculation orders results in a identical result, i.c., (AB)C = A(BC)
In this paper, we define the computational complexity of matrix multiplication between matrix and x p matrix O(mp)) corresponding the number of performed multiplication operations.

---

Published conference paper at ICLR 2022
Table 4: Experimental results on test set in GLUE 'Para'' counts the parameters in encoder layers, excluding the embedding layer and prediction layer; Note the compression ratios will become smaller when considering parameters in the embedding layer. Requests Per Second (RPS) is Throughput calculated by a single Nvidía V100 GPU (16G) using full GPU memory, see App. [ for actual inference time. The single numeric suffix in BERT-III is the dimension rank d; the two numeric suffixes in BERT-IV correspond the layer rank / and dimension rank in IV respectively. The evaluation metrics follow the official GLUE benchmark (Wang et al. 2018). The best performance each task is bold. See App. for the tailored comparison with (Ben Noach & Goldberg 2020) since (Ben Noach & Goldberg 2020) used nonstandard evaluation metrics in GLUE. (Lan et al. 2019) and (Mao et al. (020) did not use all tasks in GLUE, we use and to calculate the average for their selected tasks, means that these methods have a same architecture that has identical parameters, FLOPS, and RPS.
Model four models in bokd)
Para.
FLOPS
KES
SST- MNLI
D0C
F/1
MRDC.
30C
QNLI
F1
QQP
acc
RTR
Spear
STS-B
all
Aug
BERE Devian
2018 BERT-)
420.1
93.4
83.083,44
66.4
55.7
84.5
452.2
W1.0K37
53.3
BERT-III
4.38
1143.6
91.9
8001/796
87.2
70.7
63.3
80.7
80.0
86.0
82.0
BEKE-I 72-384
12.3M
22.50
452.3
93.1
83W9332
87.5
90.2
71.0
07.3
83.6
82.6
85.5
85.5
84.0
BERT-IV -66-56
39M
15.20
596.9
92.7
82. 88818
87.1
88.9
71.4
65.2
61.8
81
87.6
82.3
83.5
BEKF-IV -36-125
19M
8,08
36.30
92.4
81.1/902
$5.5
85.3
71.9
64.1
51.4
80.8
M6.8
81.7
82.
ALHAET al
7.6
22.58
4.4.0
90.6
B2-0-
Coliber 2021
(BFRT-IT)
41.8M
6568
90.8
67.8
mairis decompsstion
75.5
manix decompostion
17.3M
92.8
71.8//1.8
75.4
60.3
72.4
TenaryBERT Zhung et a 2013
86.0M
93.4
83, 1/82.5
86.9
90.2
71.0
68.9
K3.1
82.4
BinirRFERT
[Go i X7XI
91.9
84. 1/835
65.9
89.8
71.6
67.1
R3.3
82.0
BERT Glaye
43,5M 11.38
537.2
90.7
90.4 79.7
96.7
63.6
50.0
79.6
Vanilla KD [inoonetal
2015
41.5M
11.38
837
91.5
80.2/
79.9
86.2
88.3
70.1
64.7
00.3
801
DERT-PKD (Sm aal
2019
43.5M
11.30
837.2
92.0
81.5/810
85.0
89.0
70.7
65.5
61.6
80.8
BERT Theseus
Xu 2020
43.5M
537.2
92.2
82.4/821
06.2
82.0
Table 5: GLUE results on test set TinyBERT-IV and comparison with KD based methods.
Model four models in bold)
Para.
FLOES
RPS
SST2
800
MNLI
800
MRPCC
F1
QNIL
Q12"
F1
RTB
SES-U
Avg
TinyBliKI Wayer
43.5M
11.38
84,033.
87.
90.4
71.6
83.0
BERT-IV-EE Thy BERT--V-72--666
12.3M
11.38
152.2
87.7
71.7
6.2M
7.68
1188.4
8278119
86.7
87.9
70.0
81.1
6.2 RESULTS
As shown in Tab. 4 our decomposed BERT with layer rank 144 and dimension rank 384, called BERT-II-384", outperforms the BERT-base, with only 1/7 parameters in Transformer layers and slightly bigger throughout. BERT-IV-72-384 performs on-par with raw BERT, which is slightly worse than BERT-III-384 due to the smaller size. Observe that a bigger rank (both for layer mode and dimension mode) usually consistently leads to better performance. BERT-III-64 achieves 96.7% performance (82.7 vs. 80.0) with only 1 1/48 parameters of Transformer layers and 2.7x speedup.
Tab. 4 shows BERT-IVs smaller than existing parameter sharing method (Lan an et al. 2019) and decomposition method (Noach & Goldberg 2020; Mao al., 2020). BERT-III -384/ BERT-IV -72-384 achieve comparable or even slightly better results than (Noach & Goldberg 2020). BERT-IV outperforms (Mao et al. 2020) with a large margin. BERT-IV outperforms ALBERT the latter needs training from scratch (IM training steps) while the former does not (less than 0.2M steps).
Observe that BERT-III-388 outperforms BERT-IV- 172-3384 since the former has more parameters and therefore is more expressive. Note that BERT-III-384 and BERT-IV- have nearly identical example to compare with matrix
RPS and inference latency. Note, take BERT-IV- -6- 128 decomposition (a.k.a, II, which is implemented by Noach & Goldberg (2020) with rank 245, denoted as BERTTIV--45), BERT-IV--36- 28 is faster (see RPS in Table 4 and inference time in 8). smaller, and better perfoomed (see Table 10 for full performance comparison) than BERT-IV-244 evidencing the advantage of BERT-IV over matrix decomposition for compression.
To contextualize BERT- BERT-II/BERT-VV with other compression methods like knowledge distillation, achieves comparable performance with knowledge distillation Xu et al. 2020 Jiao et al. 2020) while with fewer parameters. Our results
Tab, shows that methods (Sun al. 2019

---

Published conference paper at ICLR 2022
Table 6: Ablation experiments of knowledge distillation (KD) (including GD and TD). The test set GLUE with the setting 72-384 is reported. The best performance of each task is bold.
Setting
SSE2
MILII a0c
MRPC F1
QNLI
QUF
F1
KTE
S1S
1
875
fio-tnining weo
49.9
35.8/6.55
790
49.7
Table 7: Experiment results test of GLUE with SAN and FFN.
Model
Para
FLOPS
RPS
SST2
MNLL
MIRPC' F1
QNLI
QUP
KIE
STS-B
BERFossee BERT-)
86,004
22.58
420.1
93.4
43.9995.
87.5
90.9
71.1
66.4
85.3
K2.7
BERIFTV 364
12,351
22.58
452.3
MA.1
83,9885 3
87.5
90.2
71.5
67.3
83.0
K2.0
$4.584,1
88.0
BERT-IV-SAN-24-1
61.9M
84.5883.7
86.0
also comparable to quantization methods Zhang et al. (2020); Bai al. (2020) that use 2-bit or b-tit weights, and pruning Lagunas et al. 22221) (see Tab. 1i Ap
To show the proposed method is orthogonal to existing compression methods like pure knowl- edge distillation, we further explore the proposed method in TinyBERT (Jiao et al., 2020), called "TiyyBER-IV" Tab. 5 shows performance loss to compress TinyBERT (degrading from 83.0 to 81.6 in B is bigger than compressing raw BERT (degrading from 82.7 to 82.6 in BERT-IV -72-384). This is probably due smaller redundancy in TinyBERT compared to BERT.
6.3 ANALYSIS
Ablation the necessity of knowledge distillation Tab. G shows that both GD and TD are essential for an effective decomposed BERT. In particular, the overall performance decreases from 82.6 to 80.0 by removing TD. Note that the model will collapse we directly take decomposed BERT for fine-tuning without knowledge distillation.
Decomposition on FFNS SANs For FFNs and SANs, we use the half size of matrix bank (i.c., 24 for SANs and 48 for FFNs) and half dimension rank (i.c., 384) respectively. The two settings are called SEERT-V--FN-443
and 'BERT-IV-SAA -SV---A-----88- SANs FFNs could nearly achieve on par with the model since smaller compression ratios are achieved. In detail, FFNs are slightly easier to be compressed even with a big compression ratio comparing to SANs. It is intriguing to notice that BERT-IV- -7--783------- outperforms BERT-IV-SAN-2 384, although the former additionally compresses FFNs and has much fewer parameters. The reason (i.c., 72), which is shared between FFNs
Tab. shows that solely compressing
may be that the size of the matrix bank in BERT-IV-72-38 and SANs, is bigger than its counterpart in BERT-IV SERT--V-SAN-24-- SAN-24-384 (i.e., 24). This can shed some light on the benefit stacking FFNs and SANs together, see more discussions in App
7 CONCLUSION AND FUTURE WORK
To largely compress PLMs, also comprehensively compare many matrix/tensor decomposition methods and conclude tensor decomposition has the potential for extreme parameter compression, We therefore efficiently implemented tensor decomposition inspired compression for BERT, thanks efficient reconstruction protocol. To compensate for the decomposition discrepancy. knowledge distillation used to simulate the output of raw BERT, which is demonstrated to be essential ablation study. Our method with 1/7 parameters could achieve comparable with the original model, with slight speedup. A tiny version achieves more than 96.7% performance of BERT-base with 1/48 parameters Transformer layers and 2.7x faster on inference. In the future, we expect compression of PLMs to shift purely encoder- based language models (c.g.. BERT) to decoder language models, since the latter has been designed big potential of the proposed methods for compressing larger model is discussed in App- N Furthermore, hybrid methods by mixing knowledge distillation, quantization, parameter sharing, weight pruning. and matrix/tensor decomposition together potential in practice.
could afford, e.g. GPT3 (Radford al. 2019). The

---

Published conference paper at ICLR 2022
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv: 3Xiv:1803..0375, 2018.
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization, 2020.
Matan Ben Noach and Yoav Goldberg. Compressing pre- rre-trained language models by matrix decom- position. Proceedings of the 1st Conference of the Asia-Pucific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Pro- cessing, pp. 884 889, Suzhou, China, December 2020. Association for Computational Linguisties. URL BtR.ttts::// /aap::://aallaaii http:///aacaaaaaahooooororr http:/orgg/00000 2020 aacl n. 88
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fuirness, Accountability, and Transparency, FAccT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 450383097. doi: URL /ttppbttss///d..oro org. /2tp////11.1111421111111
10.1145/34421888.445555
3445922
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165. 2020. URL httpt::tp///.om...tt. http://aavvv.v//. wwtp:/wwwwww...b//b.bbbs./// 14165
Qingqing Cao, Harsh Trivedi, Aruna Balasubramanian, and Niranjan Balasubramanian. Deformer: Decomposing pre- traitedi transformers for faster question answering. CoRR, abs/2005.0066 00000777 2020. URL
00697
J Douglas Carroll and Jih-Jie Chang. Analysis of individual differences in multidimensional scaling via n-way generalization of "eckart-young decomposition. Psychometrika, 35(3):283-31 319, 1970.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In NeurlPS, 2018.
Krzys/tof Choromanski, Valerii I. Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. Masked language model- ing for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.03555, 2020.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning, What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04333 2019.
Bean-Baptistt Cordonnier, Andreas Loukas, and Martin Jaggi. Multi-head attention: Col- laborate instead of concatenate, 2021. URL htt.:::://ppeeeeeeewweeeee. fww.mm/ifffooo
bK- KKKrJMKrOsm
Lieven De Lathauwer, Moor Bart De, and Vandewalle Joos. A multilinear singular value decomposi- tion. SIAM journal on Matrix Analysis and Applications. 2000.
Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. arXiv preprint 1rXiv:1404.07366 2014.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: (tt.:/88/00000.. 2018.
Yihe Dong, Jean-Banne Baptistt Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103. 103.033044 2021.
10

---

Published conference paper at ICLR 2022
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling trillion parameter models with simple and efficient sparsity. arXiv preprini arXiv:2101.03961 2021.
Ze-Feng Gao, Song Cheng, Ling-Qiang He, Z. Y. Xie, Hui-Hai Zhao, Zhong-Yi Yi Lu, and Tao Xiang. Compressing deep neural networks by matrix product operators. Phys. Rev. Research, 2:023300, Jun 2020. doi: 10. hew.:PhhyyReeeeeeeerh.h22.22222
WtRLhhtps://Iine aps.iikkk..ppppommmmm..mm... doi
1.0.1103/E 2t._1003.yyy.exeeeacch22222222. 023300
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer ford-forward layers are key-value memories. arXiv preprini arXiv:2012. 14913, 2020.
Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cognition. arXiv preprint arXiv:201 75091, 2020.
Richard A Harshman. Determination and proof of minimum uniqueness conditions for parafacl. UCLA working papers in phonetics, 22(111-117)3, 1972.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 08415, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015.
Lu Hou, Quanming Yao, and James T Kwok. Loss-aware weight quantization of deep networks. In ICLR, 02 2018.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth, 2020.
Oleksii Hrinchuk, Valentin Khrulkov, Leyla Mirvakhahova, Elena Orlova, and Ivan Oseledets.
Tensorized embedding layers. Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: Findings, pp. 4847 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding, 2020.
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets. Tensorized embed- ding layers for efficient model compression. arXiv preprint arXiv: 10787, 2019.
Nikita Kitaev, Eukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04455 2020.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky, Revealing the dark secrets of bert. arXiv preprint 1rXiv:1908.0859; 2019.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10619-10629, 2021.
Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Alhert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11 11942, 2019.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned ep-decomposition. arXiv preprint 141265:412.6553. 2014.
Peiyu Liu, Fe-Feng Gao, Wayne Xin Zhao, ZY Xie, Zhong-Yo Yi Lu, and Ji-Rong Wen. Enabling lightweight fine-tuning for pre-trained language model compression based matrix product operators. arXiv preprint arXiv:2106.02205, 2021.
Xindian Ma. Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawci Song. A tensorized transformer for language modeling. Advances in Neural Information Processing Systems, 32:2232-2242, 2019.

---

Published conference paper at ICLR 2022
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang. Quanlu Zhang, Yunhai Tong, and Jing Bai. Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv preprint urXiv:2004.04122 2020.
JS McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of bert-based question answering model. arXiv preprint arXiv: arXiv: 1910.06360, 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances Neural Information Processing Systems. 32:14014-14024 2019.
Deepak Narayanan, Mohammad Shocybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, et al. Efficient large-scale language model training on gpu clusters. arXiv preprint arXiv:2104.0443 2021.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix de- composition. In Proceedings of the 1st Conference of the Asia-Paccia Chapter of the Association for Computational Linguistics and the 10th International Joint Conference Natural Language Processing, pp. 884-889, 2020.
Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing neural networks, arXiv preprint arXiv: 2015.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5): 2295-2317, 2011.
Aliakhar Panahi, Seyran Saeedi, and Tom Arodz. word2ket: space-efficient word embeddings inspired by quantum entanglement. arXiv preprint arXiv: 1.04975 2019.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang. Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arXiv preprint 2 104. 10350, 2021.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Alee Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What we know about how bert works. arXiv preprini 20Xiv:2002.12327, 2020.
Victor Sanh. Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter, 2020.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression, arXiv preprint orXiv:79088 7935, 2019.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint arXiv:2009.088 06732, 2020.
Ledyard R Tucker. Implications of factor analysis of three-way matrices for measurement of change. Problems in measuring change, 15:122-13: 137, 1963.
Vladimir Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Jesse Vig. A multiscale visualization of attention in the transformer model. arXiv preprint 1906.019.6 2019.

---

Published conference paper at ICLR 2022
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprini arXiv:7905.0944 09478, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint 1804. 07441,, 2018.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint vrXiv:2006.047000 2020.
Genta Indra Winata, Andrea Madotto, Jamin Shin, Elham Barezi, and Pascale Fung. On the effectiveness of low-rank matrix factorization for Istm model compression. arXiv preprint arXiv:1908 09982, 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jemite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: Statc-of-the-art natural language processing, ArXiv, abs/1910.03371, 2019.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming 7hou. Bert-of-thesess: Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925, 2020.
Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie- Yan Liu, Nas-bert: Task- agnostic and adaptive-size bert compression with neural architecture search. arXiv preprint arXiv:2105. 14444, 2021.
Yinchong Yang. Denis Krompass, and Volker Tresp. Tensor-train recurrent neural networks for video classification. In International Conference on Machine Learning, pp. 3891-3900. PMLR, 2017.
Jinmian Ye, Linnan Wang, Guangxi Li, Di Chen, Shandian Zhe, Xinqi Chu, and Zenglin Xu. Learning compact recurrent neural networks with block term tensor decomposition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 9378-9387, 2018.
Yichun Yin, Cheng Chen, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Autotinybert: Automatic byper-parameter optimization for efficient ge-trrined language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 5146-5157, 2021.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint ar arXiv:2007. 1 44662, 2020.
Wei Zeng. Xiaozhe Ren, Teng Su, Hui Wang. Yi Liao, Zhiwei Wang, Xin Jiang. ZhenZhang Yang. Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong. Yifan Yao, Xinjing Huang, Jun Wang. Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi. Fang Peng, Fangqing Jiang, Han Zhang, Lingfeng Deng, Yehong Zhang, Zhe Lin, Chao Zhang, Shaojie Zhang, Mingyue Guo, Shanzhi Gu, Gaojun Fan, Yaowei Wang, Xuefeng Jin, Qun Liu, and Yonghong Tian. Pangu-o: Large-scale autoregressive pretrained chinese language models with auto auto-parallel computation, 2021.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. Ternaryhert: Distillation-wware ultra-low bit bert, 2020.
A BACKGROUND OF TRANSFORMER AND BERT
A.1 TRANSFORMER
A Transformer layer (see Fig. 3 consists of self-attention (SAN) module and a fordaro network (FFN) module. An input X for SAN will be linearly transformed into query, key, value, and

---

Published conference paper at ICLR 2022
0000
W
Matmul
barafaun
Scaled product
K
O
OOOO
O
O
W/
WA
WV
Wo
DXD
DXD
DXD
D Irardfam
Itansform
learafeem
franztorm
Iorafonn
Input
OOOO
Add & Norm
O
Figure 3: Transformer architecture.
output space {Q, K, K, V) below
Q KK
W
X
X
W
(8)
W
The self-attention mechanism (a.k.a Scaled Dot-Product Attention) is calculated
QK
(itention(Q.KV)) softmax
V
(9)
For a multi-head version the self-attention mechanism, it linearly projects Q, K, V with times using individual linear projections to smaller dimensions (e.g. d, Muntt ), instead of performing a single attention function with d -dimensionl keys, values and queries. Finally, the output of SAN is
SAN(X) = [head]; :..;; head, ] head, = Attention Q., K,, Vt).
(10)
where Q -=[QQi:-.Quu. [ [..KKKKK.. and [Vi:....V, The individual attention heads are independently calculated, Cordonnier al. 202 claims that there is some redundancy in multi-head attention.
Since the output of SAN a linear transformation (using W V. which is weighted sum V. A stack of many purely SAN layers is not expressive (Dong et al. 2021), since is equivalent single linear transformation. To this end, feed-forward network with non-lincar activation is alternately used with each SAN layer,
FFN(X) 5(XW" W
(11)
Since some neurons after the activation function (e.g., ō is ReLU or GELU (Hendrycks & Gimpel 2016)) become inactivated (zero), d, usually bigger than daooas avoid the low-rank bottleneck. typically, dm 4 X dmodel = dou- Other tricks, such as layer normalization, residual connection, dropout, and weight decay are also adopted to relieve the optimization and overfitting problems when it goes deeper.
A.2 BERT
BERT is Transformer architecture-basee pre-trained language model trained plain corpora by using masked language model pre-training objective, thanks to the capacity and scalability of the
For all linear transformation this paper, the bias term is default omitted

---

Published conference paper at ICLR 2022
Transformer (Devlin et al. 2018). BERT significantly improved the SOTA performance of many downstream tasks, including classification benchmark GLUE (Wang et al., 2018). Note that the parameters of SAN and FAN linearly increase with the number of layers while the embedding layers keeps constant with respect the layer number.
B 'MULT-HEAD" FEED FORWARD NEURAL NETWORK
O
0000
O W
O
O
W
40 fransform
RDLL
D Imanaform
DXD transfom
transfom
Imundarr
OOOO
O
O
O
O
O
tranffom
W m
Dx transform
D tncstoom
D tronatorm
OOOO
transfam
OOOO
(a) FFN
(b) 'multi-head FFN
Figure Fig. 4a and Fig. are equivalent. Note that FFN could sum of independent sub-FFNs; each them conducts full-rank x D transformation.
the multi-head attention mechanism, individual attention heads separately calculated, however the calculation is low-rank due to the redundancy among heads (Cordonnier et al.] [202]. In this paper, we argue that such redundancy may also appear in the feed-forward neural networks. In the feed-forward neural networks, element-wing wise activation functions usually adopted, e.g., GELU (Hendrycks Gimpel, 2016) and Relu (Agarap) 2018); that is, each activation can be independently calculated.
By partitioning the original w-in into H column groups and Wtst into row groups, one can revise a feed-forward neural layer (i.e. FFN(X) 8( 8(XW" woll) as a sum of H/ independent 'thin' sub-FFNs with dimension of DH H as below:
# Σ 5(XW=) h-1
FFN(X)
(12)
Where whth E RR DxD, and W E R D.DD 4 [w e W R Dx4D and V
[W @ ... @ W E E 1Dx In this paper, we set H 4. since W W E R DxD and each
transformation in sub-FFNs layer full-rank transformations. See Fig 4 for graphical illustration. One can refer to block partitioned matrix multiplication to understand the equivalence between Fig.
4a and 4b
C CONCATENATION OVER ROWS OR COLUMNS
PCA on columnly stacked and rowly-stacked matrices are shown in Fig. 5a and Fig. 5b Observe that there too much difference between them.

---

Published conference paper at ICLR 2022
0.9
sonsage
0.9
0.5
0.5
0.0
128
256
394
788
84 128
258
S84
768
PCA comconents
PCA components
(a) PCA for pair of matrices along columms
(b) PCA for pair of matrices along rows
Figure 5: PCA for existing weight block matrices in BERT-base
D MATRIX AND TENSOR DECOMPOSITION
Tensor generalized 'matrix' with typically dimension of 3 sometimes, also call a matrix as bi-dimensional 'tensor' and vector as a one-dimensional 'tensor'. Formally, an r-dimensional tensor is an array with n indexes as
X e R fumm.
(13)
An entry in X is accessed by selecting n ordered index (i,,...,,,)) and s is In,4 E N). this section, we mainly discuss order-3 tensors for simplicity while higher-order tensors also hold.
Typical tensor decomposition includes Canonical Polyadic (CP) decomposition (Carroll & Chang 1970 Harshman (972) and Tucker decomposition (Tucker 1963). CP decomposition approximates a high-dimension tensor with as a sum many rank-one tensors. case three-dimensional tensor decomposition,
X
Σ r=1
A. 2 B.
(14)
Where A, R B, R C. E R and A Rxl, B I Txl2 and C tensor product
I2 Rxly @ is the
Tucker decomposition decomposes tensor into of factor matrices and one small low-rank core tensor,
X = G A Bx;
(15)
where A R Rixh,
B E R Raxls C C R Raxla and G RR RxXRxRRs X1, X2. and X3 mode-k (R, R2, R3) is sometimes called Tucker ranks. An entry with index (4, is) is
products 10
calculated
Ri Ro R4 G.
ΣΣ>
Brisg Ceia
(16)
12.45
a-1 1m1 cml
"Here we give an example to caleulate the tensor product of three 2-dimensional vectors, at, y. resulting in a tensor of (g. 2x222 R the CP rank shared in all modes.
IR?
WF R1
3192
21
91
21
V1 21 22
V1 2/
3r R2
31
422
92
22
3/.22
3228 2223/
32
Given tensor G E B Ri RR, and matrix M e I Raxe mode-K between G and M results in X., M ig RRgRRaraaa Kix mm..

---

Published conference paper at ICLR 2022
Tucker decomposition will degrade to CP decomposition, where the core tensor G constrained to be super-diagonal III and R R₂ R3.
k-ssice the first mode, i.e., matrix with size 12 x 13, would
Ra
Gin.tadi d @ Cig), X, E ER
(17)
All slices a specific mode share factor matrices (i.c., B, C) in other modes. Therefore, there exists inter-correlation between these slices. These shared factor matrices not only make the compression ratio of tensor decomposition being much bigger than multiple independent matrix decomposi- tion (Noach & Goldberg 2020), but also can be beneficial especially when there redundancy among these parameter matrices and factor matrices can be utilized common features.
DIFFERENCE WITH EXISTING WORK
F. DIFFERENCE WITH TENSOR DECOMPOSITION USED FOR COMPRESSING CNNS, RNNS AND EMBEDDINGS
Compressing pre-traini trainini models is a new scenario. We believe that exploring tensor decomposition pre-trained language models is non-trivial. In the pre-trained language models, we could test tensor decomposition very deep and wide networks like GPT 3 (96 layers and hidden dimension of 12288), while this work is the first step.
Existing work Ma ct al. 20 Liu et al. (2021;; Gao et al. (2020); Khrulkov et al. 2019); g Hrinchuk al. (2020); Panahi et al. (2199) which tensor network for compressing neural networks do not have the potential for acceleration. The bottleneck in speed limits the application of tensor decomposition in real-world applications: compressing models but consuming longer inference time seems be useful in very rare scenarios. We argue that it is nontrivial compress neural networks using tensor decomposition with acceleration effects, as this work did.
Work mechanisms for compression are totally different, previous works compress each weight matrix (W²2 w WV W W W each layer) individually using matrix/tensor decomposition. They are making use of local redundancy inside each matrix. While in big models (PLMs), we believe that making use of cross-matrix/cnt matrix/ccross-ayer redundancy is also, sometimes more, beneficial. We believe that using tensor decomposition for cross matrix/cross layer redundancy is a significant difference.
E.2 DIFFERENCE WITH CORDONNIER BT AL.. 222
(Cordonnier et al. 2021 is quite impressive and inspiring. It does inspire this paper, however, we want to highlight the difference with (Cordonnier et al. 2021).
The motivation different, (Cordonnier et al. 202 found redundancy different heads. We make use of redundancy of both intra-matrix redundancy and inter-matrix redundancy (including cross-layer redundancy).
Motivation
Method The architecture in (Cordonnier et al., 202I) slightly different from Transformer (or BERT) since it redesigns the self-attention network with collaborated attention. While our architecture is nearly the same as the raw model, simulate each matrix multiplication of BERT with a product of many smaller matrices.
Goal Our work aims to extremely compress neural networks, which cannot achieved by (Cordon nier et al. 2021). (Cordonnier al. 202T) is to explore the possibility to share key/query projection SANs. Note that SAN only has 1/3 parameters in Transformer, this proportion even becomes smaller when considering the embedding layer. By compressing only SANs with 1/3 parameters, its overall compression ratio is limited.
Namely, G.,.i...g equals! if i2 is. and otherwise

---

Published conference paper at ICLR 2022
Potential for efficiency (Cordonnier et al., 2021 is faster than the raw model only De small. The smallest setting with D3 64 has 20% FLOPs reduction. This shows its potential for acceleration is limited. A similar setting with D3 64, we have nearly 80% reduction in terms of FLOPs.
Potential for compression ratios The smallest model for BERT-base (110M) in (Cordonnier et al., 202T) has 96.6M when Dr 128: its best compression ratio is 1.14. While in our model, the smallest model has a much bigger compression ratio, while being faster and performing better.
Potential for effectiveness The model in Cordonnier et al. 2021 (96.6M parameters) achieves 93.5% (77.6/83.0) performance with BERT base, while our smallest model with 25M parameters (plus parameters in the embedding layer) achieves 97.7% performance (80.8/82.7) of BERT base. A slight difference is that test our model on the test dataset through GLUE online benchmark while Cordonnier et al. (202T) test their model on offline dev dataset through average results for three runs, so we use the relative performance w.r.t. the original BERT-base.
F MORE DISCUSSIONS TO COMPRESS WEIGHTS MATRICES IN SANS AND FFNs TOGETHER
E1 SHARING LEADS TO A BIGGER MATRIX BANK
If we separately compress SANs and FFNs, we would have two matrix hanks: one for SANs and one FFNs: PFPN C FFN and PSAN c SAN, Eaac weight matrix in FFN(or SAN) is specific to matrix
bank C CFN for FFN (or C SAN for SAN) and its weight vector over the bank P. PFN (or SAN ). Note that the two matrix banks have the most parameters since it is three-order tensor while others (U, V. P) are matrices.
Note that matrices in two matrix banks have the same shape, one could merge (share) the two matrix banks (a m-size d d matrix bank and n-size m-size d x d matrix bank) to get single bigger (In + ni-size) matrix bank, this could boost the expressive power for both FFNs and SANs due the bigger matrix bank (denoted as [C PN SAN
The unmerged one is special case of the merged one. Let us define new p., each element in which defined as below:
[P: [0, 0, 01] for SANS
P/
[[0, 0, ...00:: P.] for FFNs
P/ is the new weight vector over the shared (m + ri)-size matrix bank.
Without losing any generality, we could relieve the zero-constrains in Pl to get a general form that each element in p' is not forced be zero. This could make FFNs and SANs share more matrix bases and therefore get hetter capacity.
The benefit could be empirically evidenced in Table 7: solely compressing SANs (without compress- ing FFNs) underperforms both compressing FFNs and SAÑs; although the former has much more parameters. Since in the shared setting, the shared matrix bank could compensate the counterpart SANs. Another naïve motivation is to design a unified protocol for both SANs and FFNs.
F.2 DISCUSSIONS ON THE SCALE DIFFERENCE BETWEEN WEIGHTS MATRICES IN SANS AND FFNS
We want to clarify that we do not intend to approximate the raw weights (and their norm/scale). Previously, we tried to add a regularizer in the training loss function to force the reconstructed weights using decomposition be close the weights.

---

Published conference paper at ICLR 2022
5
Figure 6: L2 norms of raw weights in BERT
[ fraining +
X X/
w
This does not improve the performance, but worsen the performance. The reason may as below. Since we cannot perfectly approximate the raw weights with decent compression ratio, there always exists some difference between the weights and the reconstructed weights using decomposition. However, even sliihtt difference might lead to big difference output. Plus, we also observe the finally learned W has big difference with W So we give up our efforts to approximate the raw weights.
For example, we found that even randomly initializing the compressed BERT nearly achieves identical performance, compared decomposition from given model. This also shows that approximating raw weights is not beneficial. Instead, we use knowledge distillation to simulate the input-output mapping from the original model and tolerate the difference between raw weights and reconstructed than the latter. Empirical results show the
weights. We believe that the former makes former performs well.
Norm difference in decomposition three factor tensors and a core tensor.
Decomposition to approximate a three-order tensor with
(CP.),x
Each matrix has its specific factor vector P.. which does not have any constraints. The flexibility of norms in P. could compensate the norm difference the original matrices.
case the readers may be curious, we show the norms in Figure 6 The norms in different matrices do have some differences. We also compare the norms between the raw weight matrices and the reconstructed weights matrices in Figure 7 It shows that the norms of reconstructed weights also have big difference compared to that of corresponding original weight matrices. We argue that the knowledge distillation does not aim to approximate the raw weight matrices but only approximate its input-output mapping functions.

---

Published conference paper at ICLR 2022
5
Figure 7: L.2 norms of raw weights in BERT and reconstructed BERT-IV.
LINKING IV TO ALBERT
For IV, we have W
U(PC)V.
Considering specific case when U = V = I(D), 4 1= 12 (i.e.,
[(12)
E
12xD2
and P
I(12)
E E
12LX12
Where I(a) is an i x identity matrix. Namely
[(12)
1 0
0
[(s)
0
1
E
R ixi
(18)
...
0 0
1
Then
W = U(P.C)V
= PC
C
(19)
E R 12LD2
Eq. 19 results layer-shared BERT like ALBERT (Ln al. 2019). Lan al. 2019) additionally
compresses the embedding layer using matrix factorization.
H How IV GENERALIZES III
III (lensor train decomposition) special case TV (lucker decomposition) in this paper. Note that, we are not stating the correspondence hetween tensor train decomposition and tucker decomposition in general case, but three-order tensor train decomposition is special case of tucker decomposition.
Let recall their definition:
W !! =UEV W. IV U(P.C)V
Or in another format:
W!'l = Ex w (PC) X2 Ux
X2 and X3 are the mode-2 and mode-3 multiplication. The only difference Y R 12Lxdxd In the latter, P RR 12fxi and C n txaxd In special of tucker
(PC) IR 12Lxxxxd
20

---

Published conference paper at ICLR 2022
Table 8: Task descriptions and statistics in GLUE (aag et al. 2018). NLI is for 'Natural Language Inference" and QA is for "Question Answering', SST-2, MNLI, QNLI, QQP considered relatively-big dataset according the scale of their train
MNLI
QNLI
STS-B
task
classification
classification
NLJ 395K
paraphrase
QA/NLI
paraptrase
NLA
simillrity
train
8.5
67k
3.7K
105K
364K
2.5k
7
1k
1.8k
20k
L.K
5.4k
391k
3k
1.4k
0.8
BERT
ALBERT
BERT-II-245
GPU GPU Inference Time (sec.)
BERT-III-394
BERT-III 64
BERT-IV 384
BERT-IV-88122
BERT-IV-38 256
0.2
64
128
Batch size
Figure Inference time. Inference time in Y-axis (in a Nvidia V100 16G GPU) increases when batchsizes (in X-axis) become bigger. BERT-HI/BER--V with d < 384 are faster than raw BERT.
decomposition when 1) 12L = I and 2) P = I12z- it is equivalent to the 3-order tensor train decomposition (III). 122 is (square) identity matrix with size of 12L x 12L, in which the diagonal element equals one O elsewhere.
In our experiments, there existed two settings which set 12L = / = 144 that satisfied 1). In such settings, the rigorous difference with BERT-III and BERT-IV is that we do not force /' to be equal an identity matrix, instead, P is flexible be trained. We argue that adding such flexible P could enable crossing layer sharing and enhance the capacity of III. Interestingly, once we finish before starting inference, one could merge P and
training BERT-IV- 144-384 and BERT-IV-144,
!2Lxd which does not affect the model but slightly improve inference speed, By doing it is rigorously equivalent to IlI. Thus, we rename DERT-IV-1444384/BERRTI.
and C as a single matrix: RR 12LX121. x 12
R
as BERT-III-884 and BERT-III-384.
I GLUE BENCHMARK
The data statistics in GLUE is shown in Tab. g
J INFERENCE TIME
As shown in Fig. the BERT base is the slowest one. BERT-II/BERT-IV with dimension rank = 384 is slight faster than raw BERT. The inference time consistently decreases when becomes smaller. Especially, BERT-1II-64 is 2.6 x faster than BERT, which is consistent to RPS shown in Tab.

---

Published conference paper at ICLR 2022
Note that BERT-III -384 and BERT-IV -72-384 are faster than BERT terms of inference time RPS (see RPS in Tab. 4 although they have the identical FLOPS. A similar observation could be found in ALBERT (Lan et al. 2019) layer-shared BERT has the same FLOPS with the raw BERT while the former faster than the latter. The better inference time and RPS with the same FLOPS may be due to that the relatively smaller model consumes less memory and bandwidth inside GPU, see https: eepenrevieen net fest/fooo depends on specific platforms and hardware.
Hee.HHeee77700 This speedup effect in this matter
BERT-II implemented by Noach Goldberg (2020) has rank hidden dimension of 245, there we called 'BERT-1I-245'. It inference time is close to our SERT-IV-36-2555 this is because the inference
time is mainly related the dimension rank d.
K ON THE COMPRESSION OF THE EMBEDDING LAYER
In this work, we did not compress the embedding layer. The reasons to not compress the embedding layer are manyfold. First, compressing the embedding layer will definitely increase training/inference time, since a single embedding lookup is fast enough.
Secondly, the embedding layer does not increase the total parameters when BERT has more trans- former layers in the encoder. Since the parameters in the embedding layer are constant with respect to network depth. Table shows the parameter numbers when compressing main weight matrices with a half dimension rank (when the hidden dimension 768, its half is 384 as the rank d, meanwhile keeping layer rank unchanged). Here, we also consider embedding layer for a fair comparison as suggested.
models
Paras.
# layers (L)
V
BERT-IV- 121.
compression ratios
2
DEnn-hase-uncased
110M
12
768
30522
35.7M
3.1
LERT-laage-uncasee
340M
24
1024
30522
75.8M
4.5
GPT-3 Small
125M
768
50257
50.7M
2.5
GPT-3 Medium
350M
24
1024
50257
85.8M
4.1
GPT-3 Large
760M
24
1536
50257
165.5M
4.6
GPT-3 XL
1.3B
2048
50257
243.0M
5.3
GPT-3 2.7B
2.7B
32
2560
50257
498.0M
5.4
GPT-3 6.7B
6.7B
32
4096
50257
I.IB
6.3
GPT-3 13B
13.0B
5140
50257
1.9B
6.8
GPT-3 175B
175.0B
96
12288
50257
22.8B
7.7
Table 9: Compression ratios with increasing models when consider parameters in the embedding layer. The bigger the model, the closer it is to the theoretical upper bound of compression ratios (i.e., 8).
Note that the parameters of embedding becomes negligible when PL.Ms have layers and bigger hidden dimension, which case the compression ratio will approximate an ideal upper bound = 2 2², which linear to the deduction times layer rank and quadratic to the deduction times of dimension rank; in practice, we could use bigger deduction in both ranks, as we did in the experiments).
Plus, the shape of embedding layer (V (D) related the size of vocabulary, which is heteroge- neous other weight matrices in Self-attention network and Feed-forward network (D² or 4D²), Therefore is incompatible with the current compression protocol. additionally compress the embedding layer, we might have to design a totally different compression protocol for the embedding layer, which makes this work more complicated.
Finally, the embedding layer is relatively easy compress, see tensorized embedding (Hrinchuk) ct al. (2020) and word2ket (Panahi et al., (2019) which compress embedding with maximum 307 and 93,675 times respectively with a slight performance drop. We believe that it is trivial to additionally compress the embedding layer. Thus, leave compressing the embedding layer future work.

---

Published conference paper at ICLR 2022
Table 10: Experimental comparisons between the proposed work and (Noach & Goldberg, 2020).
indicates the reported number is average between these metrics. Metrics Accuracy (MNLI (average of MNLI match and MNLI mis-match). QNLI, RTE, SST-2), Avg of Accuracy and FI (MRPC, QQP), Matthew's correlation (CoLA), Avg of Pearson and Spearman correlations (STS-B).
Model
Para
HLOPS
RPS
D02
BcC (mámmm)
acc8311
MRPC
QNIII acc
00c (mézum))
QQP
RTE:
spear.ppeers.
STS-B
Aug
BERT-use
66.4
BUKE-II--
1,00
4.38
1088.0 6
91.9
55.3
82.3
79.3
83.3
80.5
NFRRT-V-77--8 22-22-364
12.3M
22.55
452.1
93.1
83.6
36.0
90.2
80.4
67.3
85.6
63.7
BEKE-IV 256
3,903
15.28
396.9
92.7
82.2
84.4
88.9
80.2
00.2
82.4
82.5
BERT-JV-36-BB
1.9M
8.08
963.0
92.4
80.7
86.5
83.2
80.6
64.4
82.0
81.4
desempottion dollboy 10:50 BERT-112245)
413M
656
853
82.
10.5
Table 11: Experimental comparisons between the proposed work and (Lagunas et al. 2021
Model
Para.
FLOPS
SST-2
MNLI 60C (má.mm)
QOP
Avg
LCC
11
BiRRT-bas
RA.5194.9
452.2
92.6
HERT 64
1.3M
4.38
1088.6
91.4
90.7190.8
87.2
BERT-IV-72-34
12.3M
22:58
452.3
92.5
K3.69566
87.7
87.9
BERE 36 256
3.9M
15.28
596.9
92.2
K3.4/53.8
95.0
96.9
RERT-IV-36-12 128
18M
3.08
$63.0
91.6
82.7/92.6
87.5
86.1
Syaniited
Legunas
L COMPARISON TO METHODS THAT USES NON-STANDARD METRICS IN GLUE
Noach & Goldberg 2020) uses non-standard setting for GLUE tasks. Metries in (Nach & Goldberg
2020) are Accuracy (MNLI (average of MNLI match and MNLI mis-match). QNLI, RTE, SST-
2), Avg of Accuracy and FI (MRPC, QQP). Matthew's correlation (CoLA), Avg of Pearson and Spearman correlations (STS-B). Tab. [1] tailored our result in their evaluation protocol. In this work, we do not include CoLAA dataset since training samples in CoLA are too few to have stable evaluations.
M ANALYSIS OF THE LEARNED MODEL
Observing that in IV, each matrix is parameterized W U(P.C)V With shared V and V, each matrix has specific parameter P: (called 'factor vectors' later), we analyze (P,) to shed some light on understanding what BERT-IV learns.
In BERT-IV-72-384, we calculate the cosine distances between any two factor vectors. d(P., P2) matrix. We could observe that there a cross-layer pattern that
1 cos(P,, P,). lead a 144 {W""} (the weight matrices in the layer of FFNs) among layers are relatively close. This pattern becomes weaker in deep (top) layers, this might suggest that top layers may learn more diverse feature projections. The reason for this pattern needs further investigation.
also visualize in-matrix pattern in each layer (e.g. in the Ist, 4-th, 8-th, and 12-th layer) Figure 10 It shows that a pair between w and WDat relatively close, e.g., W and Wat W and Wout etc.
N POTENTIAL OF COMPRESSTON IN LAGER MODELS
this section, we compare various compression ratios of III, and IV, see the parameters and their compression ratios (this additionally considers parameters in the embedding layer) in Table It shows that III and IV could achieve much bigger compression ratios than II when using the same rank for hidden dimension (i.e., 2048). claim that proposed model has better potential compress bigger models.
23

---

Published conference paper at ICLR 2022
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Figure 9: Distances between the factor vectors among 144 matrices in trained BERT-IV-72-384 The order is listed as ( w w. W WT", W2 W.". WT" w W2 W Out W
from the first layer to last layer.
(a) The first layer
(b) The 4-th layer later
(c) The 8-th layer later.
(d) The 12-th layer later.
Figure
10:
Distances between the factor of trained BERTIV-72-384,
vectors among The order
matrices
in each layer
is
listed as
[ W W 0 w W. In W... W w Out
W
W W-Out Out
model
Paras
L D
GPT-II-2048
GPT-III-2048
GPT-IV-12-768
GPT-IV-12-2048
GPT-IV-144-2048
GPT-3 Small
125M
768
493.1M (0.3x 647.2M (0.2x
L)
48.3M (2.6x 1)
93.5M (1.3x J)
647.2M (0.2x J)
GPT-3 Medium
350M
24 1024
1.3B
(0.3x !)
1.3B (0.3x 1)
56.7M (6.2x
102.5M (3.4x
656.2M (0.5x L)
GPT-3 Large
760M
24 1536
1.98
(0.4x 1)
1.38 (0.6x 1)
90.0M (8.4x 1)
137.IM (5.5x
690.8M (L.IX L)
GPT-3 XL
1.3B
24 2048
2,5B
(0.5x 1)
1.3B 1)
102.3M (12.7x 1) 150.8M (8.6x 1) 704,5M (1.8x L)
GPT-3 2.7B
2.7B
32 2560
4.2B
(0.6x 1)
1.8B (1.5x 1)
194.4M (13.9x 1) 244.2M (11.1x L) 797.9M (3,4x L)
GPT-3 6.7B
6.7B
32 4096
6.7B
(1.0x 1)
1,9B (3.6x 1)
270.9M (24.7x 1) 324.7M (20.6x J) 878.4M (7.6x 1)
GPT-3 13B
13:0B
40 5140
10.4B
(1.2x L)
2.4B (5.5x 1)
333.6M (39.0x 1) 390.0M (33.3x 4) 943.7M (13.8x L)
GPT-3 175B
175.0B
96 12288 59.0B (3.0x 4)
5.9B (29.5x 4)
L.B (162.1x 1) 1.2B (151.6X J) 1.7B (102.4X L)
Table 12: Parameter compression ratios in various models


Reviews:
[{"review_id": "sXgGHGMq2UX", "reviewer": "Reviewer_rMZS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. The framework of decomposability and tensor decomposition allows this paper to encompass and explain the benefit of multiple previous work using a single viewpoint\n2. The results of compression are strong. Possibility of 50x compression, albeit at 1.5-2% accuracy loss opens a lot of possibilities for on device deployment\n\n\nWeakness:\n\n1. The use of tensor decomposition for compressing neural networks has been explored extensively for CNNs, RNNs and Embeddings. The use here to compress transformers is a natural extension of the idea\n2. Decomposability and low-rank nature of FFN and MHA layers has been discussed previously in the literature. The authors themselves refer to these prior works\n3. Cordonnier et al 2021, further discusses the use of tucker decomposition to express MHA layers, albeit in a slightly different context. \n\nIn order to improve the paper, I recommend providing more insights into the workings of the method and the bias that the fixed structure like tucker decomposition can lead to. I also recommend exploring the systems impact of running training and inference using tucker decomposed layers and why say 50x reduction in parameter count, does not lead to a 50x improvement in RPS (Table 4). Further, I would encourage the authors to explore and understand why finetuning with KD in Table 6 leads to such large accuracy drop. GD+TD should lead to better accuracy, but improving accuracy by 40% is an interesting data point.\n\nQuestions:\n\n1. In section 4.2, the authors say \"During the inference phase, the terns that do not involve batch size b or seq length n could be calculated in an offline way...\". Could you expand on what you were referring to?\n2. Table 4 should have comparisons to sparsified BERT, esp for data points with 2-3x parameter compression. Both structured sparsity and random sparsity could achieve said compression. Eg - https://arxiv.org/abs/2109.04838\n3. Tucker decomposition of matrices across layers forces common parameters between the matrices. A possible way this methodology could go wrong is if these matrices have different scales. Is the norm value across matrices similar for different layers? Have you looked at the problem from this point of view?", "summary_of_the_paper": "This paper proposes to use tensor decomposition to compress the multi-head attention (MHA) and FFN layers in transformer architecture. \n\nPremise:\n\nDrawing from previously published research and their own experiments using PCA, the paper shows that the MHA and FFN layers are over-parameterized and exist in a lower dimensional subspace. Further, the authors talk about where these redundancy might come from and point to the decomposability of FFN and MHA layers and how that can lead to different parts learning similar behavior. Using PCA the authors show that these layers exhibit both inter-matrix and intra-matrix redundancy. Inter-matrix redundancy calls for low dimensional representation of each matrix in the layer, while intra-matrix redundancy implies possibility of parameter sharing between matrices.\n\nProposed Solution:\n\nBased on this observation, the authors discuss the merit of various methods for decomposition of MHA and FFN matrices. Specifically, they discuss matrix decomposition, tensor train decomposition and tucker decomposition. The last technique will have the largest impact on compression and also has the most marginal cost of adding a new layer. \n\nResults:\n\n1. Impressive accuracy/parameter pareto frontier. Can compress BERT-base by ~50x with 1.5 to 2% loss in accuracy (Table 4)\n2. Ablations on the need for 2 stage knowledge distillation - stage 1 to ensure attention maps and output of last layer are similar and stage two for task distillation\n", "summary_of_the_review": "Overall, I think this paper is an incremental improvement to previous state-of-the-art. Tucker decomposition has been used extensively in NN to compress RNNs, CNNs and Embeddings. Thus use of tucker decomposition and its ability to compress BERT MHA and FFN layers is incremental improvement over the previous results, especially given the fact that prior work has also shown that MHA and FFN layers can be decomposed in a low rank structure and talked about the redundancy in the parameters in those layers.  However, the results of the paper are interesting from an engineering point of view. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fDs1N9uLH84", "reviewer": "Reviewer_MtV7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons for score:\n\nI think the idea proposed in the paper is novel, but some design choices can be further elaborated and there should be more experiments on larger models and more ablation studies. Detailed comments:\n\nStrengths:\n\n- Applying tensor decomposition across layers to utilize the similarity between layers is novel. This is a valuable contribution to the model compression community.\n- The paper analyzes the optimal way to perform matrix multiplication given the compression method proposed in the paper.\n\nWeaknesses:\n\n- The paper proposed multiple potential ways of compressing weight matrices (matrix decomposition and tensor train decomposition) as some alternatives to the proposed Tucker decomposition. However, the author didn't compare with tensor train decomposition due to time constraints. I believe the paper will be more solid by adding ablation studies on tensor train decomposition.\n- The paper only performs experiments on BERT-base and TinyBERT models, but I believe that the compression method proposed in the paper should be more demanded by larger models.\n- Some design choices in the paper seem arbitrary. For example, why do you jointly compress all the layers, including all the FFN weights and attention weights? I can understand the similarity of the attention query weight vectors across the layers, but I can’t understand why all the weights are merged. In addition, the reason for splitting each FFN weight matrix into 4 seems to make it possible to combine it with attention weights.\n\nOther comments:\n\n- How does this method compare with previous works in terms of training/inference latency?", "summary_of_the_paper": "This paper proposes to use tensor decomposition to jointly compress the model weights in all attention and FFN layers of a Transformer model, which reaches similar performance as the original BERT model while marking the model much smaller.", "summary_of_the_review": "The paper would be better with more experiments on larger models and ablation studies. Also, the presentation and the rationale behind the idea are not clear to me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Hfe4DcQ7Rzt", "reviewer": "Reviewer_7TPf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well written and the motivating analyses (e.g. Fig 1) are interesting. I also appreciated the thorough appendix.\n- The main technical contributions of the paper (i.e. using all the matrices to perform matrix decomposition, using a bank of matrices) is novel to my knowledge. However, I have some reservations about whether this is sound (see weaknesses).\n\nWeaknesses:\n- The paper introduces many variants of decomposing the matrices (Table 2). However the results only seem to be based on one of the methods (i.e. IV). I realize that some of variants have already been studied in the literature (e.g. Mao et al., Noach and Goldberg), but since the setup is not identical, it is crucial that the proposed approach is compared against both II and III. Therefore, it is not clear whether the improvement in performance is coming from the actual proposed method, or something else.\n- Since the matrices across layers often have different scaling, and since tensor decomposition is approximating some reconstruction error (i.e. L2 in the case of SVD), it's not clear that performing decomposition with all the matrices makes sense. \n- The method requires a pipelined approach where one must first perform generalized distillation against the BERT model before doing task-specific distillation.\n\nQuestions:\n- In GD, did you try distilling also the masked LM logits, in addition to (or instead of) the last layer hidden states and attention maps?\n- Have you checked the norms of the matrices? Are they of similar scales? If not, do you obtain better performance by normalizing the matrices such that they are in the same scale?\n", "summary_of_the_paper": "After author rebuttal: thank you very much for the detailed response to both my and the other reviewers' comments. I have updated my score. \n------------\nThis paper proposes an approach to compressing the Transformer family of pretrained language models via tensor decomposition. Compared to existing work, the main differences are: (1) performing global tensor decomposition which takes into account matrices across different layers, and (2) using a matrix bank to enable even greater model compression. The approach, when combined with distillation, outperforms existing methods for compressing BERT models on the popular GLUE benchmark.\n", "summary_of_the_review": "A new approach to model compression with strong-ish empirical results. Some reservations about the soundness of the method. And it is furthermore not clear that the improvements are coming from the proposed approach as opposed to something else, since an ablation study across the different decomposition methods is missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-67AtyFnFQl", "reviewer": "Reviewer_fa7a", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Large scale pre-trained language models have demonstrated their effectiveness. However the large model size makes it difficult to deploy and compressing such models have drawn a lot of interest. This paper aims to compress PLMs to extremely small size mainly from the perspective of decomposition. It introduces several decomposition methods and makes a comprehensive comparison among them from the perspective of compressing Transformer layers. The Tucker decomposition is chosen to be the final solution due to its compression ratio.\n\nThe motivation is clear and the methods are technically sound. Though the introduced decomposition methods are not new, the adaption to the Transformer layers and corresponding analysis are comprehensive. The experimental results demonstrate the effectiveness of the method. Especially, the compressed model size is really competitive.\n\nSome weaknesses:\n1. The authors do not include embedding layer and prediction layer size in experiments, while only report the Transformer encoder size. I know that this can make the size of compressed model really amazing (e.g., 1.8M) and the compression ratio amazing (e.g., 86M/12.3M=7) but is not fair as the whole model including the embedding layer are used when deploying. If the embedding layer is added, the model size will increase a lot, and the compression ratio will decrease, which make the experimental results less surprising. But this should be made clear.\n2. The authors name a lot of related works, but compare only very few of them in the experiments.\n3. Some other method(s) are missing in the related works. For example: [1]\n\nSome typos:\n1. Section 5.1, \"...are not exactly equal to the the raw weights...\", duplicate \"the\"?\n2. Section 6.2, \"...outperforms ALBERT - the latter needs...while the latter does not...\", two \"latter\"?\n\nreference:\n[1] Xu, Jin, et al. \"NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search.\"", "summary_of_the_paper": "This paper explores extreme parameter compression for pre-trained language model, especially BERT. It introduces and compares several tensor decomposition methods and proposes to leverage Tucker decomposition as the final solution. The compressed BERT model achieves much smaller size with promising performance.", "summary_of_the_review": "The paper presents extreme compression on pre-trained language models. Though the introduced methods are not new, the adaptation to the Transformer layers and the analysis are interesting, and the experiments are convincing. Though there exist some weaknesses, I think the paper is of good quality, if the authors could mitigate them.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper reviews a number of parameter decomposition methods for BERT style contextual embedding models. The authors argue for the application of Tucker decomposition to the attention and feedforward layers of such models. Evaluation is performed for a range of models on the GLUE benchmark. Further ablation studies indicate that the distillation procedure employed is crucial for obtaining competitive results and the raw decomposition approaches are ineffective at directly approximating the original pre-trained model.

Strengths: The reviewers generally agree that the methods explored and results presented in this paper are interesting and could be of use to those deploying large embedding models. The authors review a range of possible decomposition methods and use this to motivate their approach. The resulting levels compression are high while maintaining good performance, while the ablation study clearly shows the contribution of the various steps of the training pipeline.

Weaknesses: The main weakness identified by the reviewers is the incremental nature of this work in comparison to previous works applying various decomposition and compression techniques to neural networks. They also highlight that many of the techniques discussed early in the paper are not compared in the evaluation. The authors have effectively responded to this issue by providing further comparisons and justification for their modelling choices (e.g. not compressing the embedding layers). 

Overall, despite the incremental nature of this work, I believe that there are enough though provoking ideas and results presented to warrant publication. Interestingly, as the authors emphasise in their response, the ablation study highlights that this work is not really about approximating the original models weights, as all of the work appears to be being done by the distillation procedure in concert with the choice weight decomposition. In general I wonder whether this paper would be better presented as exploring a structured distillation procedure rather than weight compression.
Decision: Accept (Poster)