Language model compression with weighted low-rank factorization
Authors: Yen-Chang Hsu
Ting Hua
Sungen Chang
Qian Lou
Yilin Shen
Hongxia Jin
Venue: ICLR 2022 Poster (2022)
Keywords: model compression
low-rank approximation
transformer
language model

Abstract: Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

Published as a conference paper at ICLR 2022
LANGUAGE MODEL COMPRESSION WITH WEIGHTED LOW-RANK FACTORIZATION
Yen- Bh-nChan Hsu"! Ting Hua" Sung-En En Chang", Qian Lou', Yilin Shen', and Hongxia Jin'
'Samsung Research America, NNrtthestern University.
(yenchang.her hsu, ting. hua, lou, yiein.shen,
hongxia. jin Sannussms com, {chang wsnn/ennrtheasteen..e.
ABSTRACT
Factorizing a large matrix into small matrices is a popular strategy for model com- pression. Singular value decomposition (SVD) plays a vital role in this compres- sion strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix with- out gauging the importance of the parameters, potentially giving larger recon- struction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accu- racy. We analyze this previously unexplored problem, make observations, and ad- dress it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Wae Weighted SVD (FWSVD). Although the factorized matrices from our approach do not re- sult in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-basee language models, showing our weighted SVD largely allevi- ates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compress- ing already compact model shows our method further reduce 9% 30% parameters with an insignificant impact on task accuracy.
INTRODUCTION
Language models built with transformers (Devlin et al. 2018) have attained extensive success in natural language tasks such language modeling (Radford et al. 2018), text classification (Wang et al. 2018), question answering (Rajpurkar et al.) 2016), and summarization (Liu) 2019). The success is achieved by fine-tuning a big transformer model pre-trained with a large corpus. The target task for fine-tuning may only focus on restricted scenario such as sentiment analysis (Socher et al.) 2013) and multiple-choice question inference (Zellers al. 2018). Having a big transformer model often overkill for the target task and prohibits the model deployment to resource-constrained hardware. Therefore, language model compression raises immense interest.
The popular strategy creates compact model from scratch (Jiao et al. 2019) subset of the big model's layers (Sun et al. 2019; Sanh et al. 2019), then pre-trains with large corpus and distills knowledge from the big model. This process is called generic pre- pre-training (Wang et al.) 2020b Sun ct al. 2019; Sanh et al. 2019) and is necessary for a compact model to achieve good performance on the target tasks. However, the generic pre-training could still cost considerable computational resources. For example, it takes 384 NVIDIA V100 GPU hours to get the pre-trained BiRyBERT Jiao et al. 2019) on the Wiki corpus dataset. So it may not be affordable for everyone who wants to create a compact model. In contrast, another line of strategy, specifically low-rank factorization (Golub& Reinsch 1971 Noach Goldberg 2020), can potentially reduce big model's parameters
"These authors contributed equally this work.

---

Published as a conference paper at ICLR 2022
without the generic pre-training. Since the factorization aims to approximate the learned model parameters, the method has the nature of directly inheriting the knowledge of the big trained model.
However, approximating the learned weights with standard factorization often loses most of the task performance. This work investigates this issue with the most popular strategy, which uses singular value decomposition (SVD) to compress the learned model weights. With SVD, the learned matrix is factorized into three matrices (U, S, V). The portion associated with small singular values will be truncated to produce a smaller version of factorized matrices. The multiplication of these smaller matrices will approximate the original one with fewer total parameters to achieve the model compression. In other words, SVD minimizes the reconstruction error with fewer parameters as its objective. However, this objective does not necessarily correlate to the ultimate goal of keeping task performance. Specifically, the SVD algorithm is biased to reconstruct the parameters associated with large singular values. As result, the parameters mainly reconstructed by the ranks with small singular values will become the victim in the compression process. Are these victimized parameters less critical to achieving good task performance? We argue that this is not true, and the optimization objective of SVD is not properly aligned with the target task objective. This paper is the first work to provide an empirical analysis of this issue, proposing novel weighted SVD to mitigate it.
Our weighted SVD addresses the above issue by assigning importance scores to the parameters. This score has to correlate to how much the task performance is affected by the parameter change. The Fisher information nicely fits into this purpose (Pascanu & Bengio 2014). Besides, the calculation of Fisher information is usually simplified to accumulating parameter's s squared gradient over the training dataset based on its task objective (e.g. f...ccros-eetrrpy.. regression error, etc.), conveniently providing the importance of each parameter in model. Then we modify the optimization objective of factorization (i.e., reconstruction error) by multiplying it with Fisher information, providing a new objective that jointly considers matrix reconstruction error and the target task objective.
In summary, this work makes the following contributions: (I) we analyze the issue of mismatched objectives between factorization and the target task for model compression; (2) we propose a novel compression strategy with the SVD weighted by the Fisher information; (3) we perform extensive analysis on varied language tasks, showing our Fishhr-weighted SVD can compress an already com- pact model, and it can achieve comparable compression rate and performance with methods that require an expensive generic model pre-training.
BACKGROUND
2.1 MODEL COMPRESSION WITH LANK APPROXIMATION
Given a matrix W E NNM, the low-rank approximation is achieved via singular value decompo- sition (SVD):
W R USV",
where U E N V E Mxr, and & is the rank of matrix W. S is a diagonal matrix of non- where 0, 02 or 2 dr > O. The low-rank
zero singular values io.. approximation with targeted rank r is obtained by setting zeros to Ir Ok-
Given input data X RIXN, linear layer in neural networks is represented below with the weight matrix W E XM and bias E RIIM
Z = XW +b (XUS)VT +b.
Factorizing W with Equation ①I leads to Equation (2). which can be implemented with two smaller linear layers: 1) The first layer has Nr parameters without bias. Its weight matrix is U.S 2) The second layer has Mr parameters plus bias. Its weight matrix and bias are V and b, correspondingly. The total number of parameters for approximating W is Nr Mr. In the case of full rank matrix and A = N, the model size is reduced when r < 0.5NN For example, if we set r to reserve the largest 30% singular values, the method will reduce about 40% of the parameters from W. In general, the reduced size will be NM (Nr + Mr).
Low rank approximation in neural networks has been extensively studied (Jaderberg et al. 2014 Zhang et al., 2015) Denton et al.] 2014). In more recent works, SVD is often applied to compress the word embedding layer (Chen et al. 2018a Acharya al. 2019). Noach & Goldberg (2020)
2

---

Published as a conference paper at ICLR 2022
applies SVD to the transformer layers, but it does not investigate why SVD gives very poor result without tinettuning. Our work explores this issue and provides a weighted version to address it.
2.2 FISHER INFORMATION
The Fisher information measures the amount of information that an observable dataset carries about model parameter The computation of its exact form is generally intractable since it requires marginalizing over the space of D, which includes data and its labels. Therefore, most of the previous works estimate its empirical Fisher information:
Iw = E
{
log (ogp(D/u)) Ow
a
)
~
D Σ
O dw
C(d;;
=Iu- Iw
=
(3)
D
The estimated information I. accumulates the squared gradients over the training data de E DD where L is the target task objective (e.g., cross-entropy for classification task, mean squared еπтоr for regression task). This approximation provides straight intuition: the parameters that change the task objective with large absolute gradient are important the target task; therefore, those parameters should be reconstructed better than others in the compression process.
The above estimation of Fisher information computes only the first-order derivatives and has been shown to measure the importance of parameters effectively. Kirkpatrick et al. (2017) and Hua et al. 2022 use it to avoid the model catastrophic forgetting continual learning scenario. Liu et al. 2021 and Molchanov et al. (2019) use it similar variant to help the structured model pruning. However, no previous work has explored potential in assisting SVD for model compression.
3 MODEL COMPRESSION WITH SVD MAY LOSE PERFORMANCE QUICKLY
The singular values in S implicitly give an importance score for a group of parameters. Since the small singular values will be truncated first, those parameters affected by the truncation are expected to be not important for the task performance. We verify the above assumption with a brute force attack: truncate one singular value at time, then reconstruct the matrix, put it into a model, evaluate and get its performance. Ideally, we hope to see less performance drop when we truncate the smaller singular values. This process can written as having the reconstructed model weights W, with the i-th singular value be truncated:
W, air + 11-10,-1011 + u. 1 10/41 ... + usoNUk
(4)
where in and U; the i-th column in U and V. correspondingly.
Applying this brute force attack to test deep neural network not straightforward since a model can have hundreds of linear layers. Therefore, truncate group of singular values together instead of only one. Specifically, we split the singular values of a layer into 10 groups sorted by their values. The 1st group has the top 10% singular values, while the 10th group contains the smallest 10%
SVD
Ideal
Performance
0.1
0.0
4
7
10
Rank Group
Figure 1: The grouped truncation and its performance. The truncation of the 10th group, which has the smallest singular values resulting from SVD, is expected to have a minor performance impact (i.e., follow the ideal trend of red dashed line), but this may not be true in actual cases (blue bar).
3

---

Published as a conference paper at ICLR 2022
VT
Important
parameters
W
W
Pootl secorisi
parameters
Truncated
parameters
Figure 2: The dilemma of vanilla SVD. Some parameters (the overlap meshed orange and green) that significantly impact the task performance may not be reconstructed well by SVD because their associated singular values are small and truncated.
S
V-T
W
=
7-
U
W
Figure 3: The schematic effect of Fisher-Weighhed SVD (FWSVD). is diagonal matrix containing estimated Fisher information of parameters. By involving Fisher information to weigh the importance, our method reduces the overlap between meshed orange and green, making less performance drop after truncation.
values. When we truncate specific group, e.g., 5th group, the 5th group of all the layers in a model are truncated together. In other words, we observe the summed impact in a rank group. This results smoothed trend for the observation.
Figure [ plots the result of truncating the 10 groups separately in standard 12-layer BERT model Devlin et al., 2018)) trained for STS-B task (Cer et al. 2017). The red dashed line shows an ideal trend which has smaller performance drop with the tail groups. The blue bars show the actual performance drop. The 10th group surprisingly caused a performance drop as large as the 2nd group. This means the parameters associated with the 10th group However, the magnitude of singular value does not reflect this importance, causing a model to lose its performance quickly even when truncating only small portion.
important the 2nd group.
4 FISHER-WEIGHTED LOW-RANK APPROXIMATION
The issue in Section 3 has an intuitive cause: the optimization objective of SVD does not consider each parameter's impact the task performance. This issue illustrated in Figure and we address it by introducing the Fisher information into SVD's 's optimization objective, described as below.
In the generic low-rank approximation, its objective minimizes ||W AB|12. SVD can solve this problem efficiently by having A = US and B = V Since we can obtain the importance of each element Wiy in W. we weigh the individual reconstruction error by multiplying with the estimated Fisher information Iw.y:
min A.B
Σ Iww,, (W.; (AB).,)²
(5)
In general, weighted SVD does not have closed-form solution (Srebro Jaakkola 2003) when each element has its weight. To make our method easy to deploy and analyze, we propose a simpli- fication by making the same row of the W matrix to share the same importance. The importance for row is defined to be the summation
row, i.e., iw.=) Tww=Eww Iw..
=
Define the diagonal matrix i - diag( / iw.,.., v (ww)), ) then the optimization problem of Equation
can be written as:
min A.B
[[ - fAB
(6)
4

---

Published as a conference paper at ICLR 2022
Equation can be solved by the standard SVD iW. We use the notation sed(iW) (U", ,5, V*), then the solution of Equation (6) will be A f- and B V*T In other words, the solution is the result of removing the information î from the factorized matrices. Figure 3 iillustrates this process and its schematic effect of reducing the overlap between important param- eters and poorly reconstructed parameters. We will measure this overlap with the performance drop analysis of Section 3 Lastly, to compress W, we will have A = U- and B = V."" where r denotes the truncated U*, S and V with reserving only ranks.
We call the above method FWSVD in this paper. One thing to highlight is that since we share the same optimization process with the standard SVD, any advantage we observed will be the result of direct contribution from the / in Equation (6
EXPERIMENTS
5.1 THE PATHS TO A COMPRESSED LANGUAGE MODEL
This section describes how obtain a compressed model under the popular pre-training schemes of language models. Figure 4 illustrates three paths that we examined for creating compressed language models. All the paths start from retraining large transformer-base model pre-trained with large language corpus in self-supervised way, called the generic pre-training (T. (> 1.)).
The path-1 (S + 5a - S*) is a popular scheme that creates a small model first, then performs the generic distillation for the small model to learn the knowledge of the large model. The resulting small generic model 59 will be fine-tuned with the target task dataset to obtain the task-specific model St. The representative works of path-1 I include DistilBERT (Sanh et al., 2019). TinyBERT (Jiao et al., al,, 2019), MobileBERT (Sun et al. 2020), and MiniLM v1/v2 [Wang et al. 2020b a). Some previous methods may include task-specific distillation (L + S²) and data augmentation (Jiao al. 2019). but we exclude those from the scheme (and all the experiments in this paper) to make a fair and clean comparison across methods. The task-specific distillation and data augmentation orthogonal to all the methods and can be jointly applied with low-rank factorization to make further improvements.
The path-2 (L3 + L'' (4) avoids the costly generic pre-training, directly compresses the task- specific model with factorization and task-specific fine-tuning (optional). Our analysis for the mis- matched objectives phenomenon is based on this path. We also compare the models from path- I and path-2, showing that path-2 can generate model with comparable performance under the same compression rate. Although path-2 requires much less training than path-1 (no generic pre-training for the compressed model).
The path-3 (S" + 5) is challenging setting that aims to compress an already compact model. This setting examines whether FWSVD can further improve the compression rate on models ob- tained by path-1. Our experiments show the answer is yes.
With the three compression paths, we make four examinations as follows. Section 5.3 the compar- ison of path-1 1 versus path-2; Section 5.4 the compression of an already compact model (path-3); Section 5.3 the detailed comparison between FWSVD and vanilla SVD; Section 5.5. 5.5..
the empiri-
cal evidence for the schematic effects illustrated in Figures] 2 andd]
5.2 EXPERIMENT SETUP
5.2.1
LANGUAGE TASKS AND DATASETS
We evaluate the methods of all three paths in Figure on the General Language Understanding Eval- uation (GLUE) benchmark (Wang et al. 2019) and token classification task. We include 2 single sentence tasks: CoLA (Warstadt et al. 2018) measured in Matthew's 's correlation, SST2 (Socher et al. 2013) measured in classification accuracy; 3 sentence similarity tasks: MRPC (Dolan et al. 2005 measured in F-1 score, STS-B (Cer et al. (017)) measured Pearson- Spearrman correlation,
QQP Chen et al. 2018b) measured in F-1 score; and natural language inference tasks: MNLI Williams et 2018) measured in classification accuracy with the average of the matched and mis- matched subsets, QNLI (Rajpurkar al., 2016) measured in accuracy. The token classification task
5

---

Published as a conference paper at ICLR 2022
L
Large
L
L9
L'
Small model
Distillation
Distillation
Compression path (previous work)
Compression path (this work)
S
S
S
S
works
Generic
Task fine-tuning
Factorization & fine-tuning (optional)
pre-training
Added step of this work
Figure 4: The three paths to create compressed language models are examined in this paper. L/S denote the initial models, / are models after generic pre-training, L'/S" correspond to task- specific models, and L / are factorized tasc-specific models. Detailed elaborations Sec- tions 5.1 nad 5.2.2
Table 1: Results of CoNLL and GLUE benchmark. G-Avg means the average of GLUE tasks, A-Avg denotes the average of all tasks, including CoNLL. Our FWSVD+fine-tuning is the best performer terms both average scores, without the expensive generie pre-training required by path-1 models (c.g., DistillBERT costs 720 V100 GPU hours for training).
Model
#Param CoNLL
CoLA MNLI MRPC QNLI QQP SST-2 STS B G-Avg
A-Avg
Original
BERT base
109.5M
94.1
56.2
84.7
87.4
91.3
87.8
93.0
88.5
84.1
85.4
DistiBBRTT
67.0M
93.2
49.8
82,2
88.7
89.3
86.7
90.4
86.1
81.9
83.3
Path-1
MiniLMv2
67.0M
92.2
43.3
84.0
89.1
90.6
86.7
91.4
88.1
81.9
83.2
BERT-PKD
67.0M
45.5
85.7
88.4
88.4
91.3
86.2
81.0
BERT+SVD
66.5M
12.0
2.7
35.6
61.4
37.2
60.0
76.7
26.8
42.9
39.0
Path-2
+fine-tuning
66.5M
92.4
40.5
82.8
84.1
89.6
87.3
90.9
85.7
80.
81.6
BERT+FWSVD
66.5M
49.6
13.5
52.8
81.2
52.2
65.7
82.1
68.6
59.4
58.2
+finc-tuning
66.5M
93.2
49.4
83.0
88.0
89.5
87.6
91.2
87.0
82.2
83.6
used is the named entity recognition (NER) on the CoNLL-2003 dataset (Sang & De Meulder 2003]. In summary, our evaluation includes 8 different natural language tasks.
5.2.2 IMPLEMENTATION DETAILS AND THE BASELINE MODELS
First of all, we use the same training configuration for all the experiments in this paper and avoid any hyperparameter screening to ensure a fair comparison.
For the SOTA models on path-1 (MiniLMy2 and DistilBERT), pact models (S9) provided by the original authors as the starting point, then directly fine-tune them with 3 epochs the target task training data. The fine-tuning is optimized by Adam with learning rate x 10-5 and batch size of 32 on one GPU.
the pre-trained generic com-
For the methods path-2 (FWSVD and SVD), start from the pre-trained generic large model (L9), which is the standard 12-layer BERT model (Devlin et al., 2018). Then we fine-tune it with the training setting exactly the same models (L²). The last step is applying the low-rank factorization (SVD or FWSVD) followed by another fine-tuning with the same training setting described above. The performance with and with- out fine-tumie tuning will be both reported. We also note that we compress only the linear layers in the transformer blocks by reserving only 33% of the ranks in this work. The setup intentionally makes a fair comparison to the path-1 methods. In other words, we do not compress the non-transformer modules such as the token embedding. Previous works (Chen et al. 2018a) have shown significant success in using low-rank factorization to compress the embedding layer, which occupies 23.4M (21.3%) parameters in the standard BERT model. Therefore, the results we reported for the path- 2 methods still have room for improvement by applying Lastly, we add BERT-PKD (Sun et al. 2019) based on its reproduced results (Chen et al.) 20188)) for comparison. BERT-PKD uses knowledge distillation instead of factorization in the path-2 process.
used for the path-1 models get the large task-specific
method non-transformer modules.

---

Published as a conference paper at ICLR 2022
Table 2: Results of compressing an already compact model. The original task-specific models are directly downloaded from Huggingface pretrained models Our FWSVD successfully reduces more parameters from all the compact models, while achieving the same level of accuracy. (ft: fine-tuning)
Original Compact Model (S*)
Path-3 Compression (St + 541
Model-Task
#Param,
Perf.
#Param.
SVD
SVD+ft.
FWSVD
FWSVD+f..
Tiny BERBBERTTTTT B
14.4M (7.8x)
87.5
11.8M (-18%)
73.8
86.1
84.9
88.0
MiniLM M-CONLL
22.7M (4.8x)
88.5
18.4M (-19%)
12.5
88.0
70.1
88.6
MobileBRTT-MNLI
24.6M (4,4x)
83.6
22.5M (-9%)
36.4
81.9
51.1
82.5
DistllBERT-MRRC
66.9M (1.6x)
88.7
46.7M (-30%)
0.0
83.4
67.9
89.0
Table 3: Results of compressing an already compact model. This table compresses ALBERT (Lan al. 2019). which uses the parameter-sharing strategy to create the compact model. FWSVD pre- serves the performance significantly better than SVD in all 8 tasks, indicating its excellent compati- bility in combining the parameter-sharing strategy. This experiment examines the path-3 process.
Model
#Param
CoNLL
CoLA MNLI MRPC QNLI QQP SST-2 STS-B G-Avg
A-Avg
BERT
109.5M
94.1
56.2
84.7
87.4
91.3
87.8
93.0
88.5
84.1
85.4
ALBERTiarge
17.7M
93.5
50.9
84.3
89.9
91.7
86.7
90.7
90.1
83.5
84.7
SVD
15.2M
(-14%)
0.3
0.0
41.1
0.0
54.2
5.4
70.2
9.6
25,8
22.6
SVD+ft.
15.2M (-14%)
92.2
46.4
83.4
81.8
49.5
86.9
89.8
86.8
74.9
77.1
FWSVD
15.2M (-14%)
22.8
0.0
65.2
50.0
78.2
72.6
81.4
76.4
60.5
55.8
FWSVD+fi.
15.2M (-14%)
93.0
50.6
83.3
90.4
90.6
87.0
90.6
89.0
83.1
84.3
ALBERT hose
11.7M
92.1
43.0
82.3
88.6
90.6
86.6
89.7
89.1
81.4
82.7
SVD
9.6M (-18%)
3.5
0.0
32.0
0.0
55.1
53.4
52.4
9.6
28.9
25.7
W SVD+ft.
9.6M (-18%)
89.8
28.8
81.3
81.2
88.3
85.5
88.2
75.0
75.5
77.3
FWSVD
9.6M (-18%)
16.9
6.9
55.6
47.7
69.1
54.7
72.9
54.0
51.6
47.2
FWSVD+ft.
9.6M (-18%)
91.2
42.2
81.8
86.9
88.9
86.2
88.7
87.0
80.2
81.6
For the path-3 experiments, ing point. These pre-trained models have a much smaller size (TinyBERT-STSB. MobileBERT-MNLL)
the pre- pre-trained task-specific compact models (5*) the start-
better performance path-1 and path-2, indicating they may contain denser knowledge in their compact models. There- fore, compressing these models introduces significant challenge. In order have better coverage for all tasks, we additionally use ALBERT and ALBERT Dose (Lan et al.) 2019) as the already compact models generate all task-specific models (S9 S²). Then follow path-3 to compress the compact models. All the training involved here has the same setting as described in path-1.
MMRM than the models used in the
Lastly, our implementation and experiments are built on top of the popular HuggingFace Transform- ers library (Wolf et al., 2020). All other unspecified training settings use the default configuration of the library. Since no hyperparameter tuning involved our experiments, we directly report the results on the dev set of all the datasets, making the numbers convenient to compare and verify.
5.3 PATH-1 VERSUS PATH-2
Table reports reeprrss the results of the GLUE benchmark and a NER task. Our FWSVD with fine-tuning achieves an average score of 83.6, beating all other path-1 and path-2 methods. This is non-trivial accomplishment since FWSVD with fine-tuning does not need the expensive generic pre-training. Furthermore, FWSVD has consistent performance retention for all the tasks; it contrasts the path-1 methods, which may have more considerable variance. For example, DistilBERT is good CoLA but poor at STS-B B; oppositely, MiniLMy2 is a strong performer at STS-B but is weak with CoLA. In contrast, FWSVD+fine-tunig does not show obvious shortcoming.

---

Published as a conference paper at ICLR 2022
190 e nevv
8VD PWBWD
DVD + FW3VD
0.65
146
1.00
Maythee Cooweeton
0:40
0.5
1.75
6.98
8052
0:30
0.0
0.
0.2
044
0.00
(a) COLA
(b) NER (CoNLLL-2003)
(c) STS-B
Figure 5: FWSVD versus SVD by varying the ratio of reserved ranks. The model with rank ratio 1.0 indicates the full-rank reconstruction with the same accuracy as the original model (i.e., the L in Figure 4). Note that all the models here do not have fine-tuning after factorization.
5.4 COMPRESSING AN ALREADY COMPACT MODEL
The setting of path-3 targets to further compress the lightweight models. This is challenging as the compact models are already 1.6x ~ 7.8x smaller than the original BERT. The results in Table 2 demonstrate the effectiveness of FWSVD on further reducing the number of parameters. The original SVD is almost useless without fine-tuning, while our FWSVD can still retain a significant part of the performance. For example, SVD ends with a accuracy when compressing Dis- tillBERT, while our FWSVD keeps a score of 67.9 under the same setting. When combined with fine-tuning, FWSVD can cut off 30% redundancy for DistilIBERT, Even for the highly compact model TinyBERT (only 14.4M parameters), FWSVD+fine-tuning still successfully reduces 18% of the parameters without any performance loss. More interestingly, the TinyBERT, MiniLM, and DistilBERT-MRRCC compressed by FWSVD+fine-tunin exceed the original performance slightly. The result suggests WWSVD+fine-tuniggg might introduce small regularization effect improve the model's generalizability.
Lastly, Table 3 examinns the compatibility between SVD/FWSVD and the parameter-sharinn strat- egy of the ALBERT model. The average score of ALBERT-large is 84.7%. The performance of FWSVD (84.3%) is far better than that of SVD (77.1%) when both reducing 14% parameters, sug- gesting FWSVD is more robust than SVD in combining the parameter-sharing strategy.
5.5 FWSVD VERSUS SVD
In Table [ FWSVD consistently produces much better results than SVD all tasks. On average, FWSVD without fine-tuning obtains an absolute improvement of 17.5% over SVD. To highlight, FWSVD without fine-tuning can maintain significant portion of performance in challenging tasks such as CoNLL and STS-B, where SVD completely fails. With fine-tuning, FWSVD provides better initialization for fine-tuning and consistently achieves a better or comparable performance.
Figure plots the performance trend with respect to the change of targeted rank ratio, where the full-rank reconstruction corresponds to the results rank ratio 1.0. These results demonstrate the apparent advantage of FWSVD over standard SVD. First, at each rank ratio, FWSVD shows signifi- cant improvements over SVD. Second, the performance of FWSVD keeps growing with the increase of rank ratio, while SVD shows fluctuations in its trend. Specifically, two tasks (COLA and STS-B) in Figure 5 show that SVD has abrupt performance drops at some points. On the STS-B task, the performance of SVD rank ratio 0.3 is significantly lower than having smaller rank ratio of 0.2. In contrast, FWSVD shows a much stable trend of increasing performance along with the rank ratio.
5.5.1 REVISIT THE BRUTE FORCE ATTACK
This section applies the same analysis of Section 3 but adds FWSVD to see if it matches the task's objective better. In Figure [aa] the red bars are significantly lower than the blue bars, especially for the tail groups, which will be truncated first. We specifically highlight group-10 in 6a which has the smallest 10% singular values. The height of the blue bar is equivalent to the size of the overlapped (green and meshed orange) region in Figures 2 Similarly, its red bar (close zero) is equivalent to the overlapped region in Figure 3, In other words, the illustrations of Figures 2 and 3 are strongly supported by the results here. Although FWSVD shows smaller performance drop
8

---

Published as a conference paper at ICLR 2022
SVD FWSVD
SVD FWSVD
Performance Drop
Relative Reconstruution
0.0
Rank Group
Rank Group
(a) STS-B performance drop
(b) STS-B reconstruction error
Figure 6: Results of grouped rank truncation on STS-B task. (a), FWSVD shows a consis- tent trend of having less performance drop with the small singular value groups (group 10 has the smallest singular values), mitigating the issue of Figure (b), FWSVD results a larger recon- struction error with almost all truncated groups, although FWSVD retains the model accuracy better than SVD.
shown by Figure 6aa it has a more significant reconstruction error than SVD many cases (see Figure 6b especially with the rank groups that will be truncated first (e.g., groups 5 to 10). In other words, FWSVD's objective (Equation 6 aligns with the task objective better by sacrificing the reconstruction error.
LIMITATION AND FUTURE WORK
FWSVD has two limitations. First, FWSVD relies on a given task objective and target task training dataset to compute the importance matrix; thus, it is more proper to compress a task-specific model (e.g., L' or 5tt) than the pre-trained generic model (e.g., L ). contrast, vanilla SVD can apply to any case. In other words, FWSVD trades the method's applicability for the target task perfor- mance. Second, FWSVD only uses a simplified importance matrix that gives the same importance for the parameters on the same row of matrix W. Although this strategy is simple and effective, does not fully utilize the Fisher information. Therefore, a future improvement can be made by directly seeking element-wise factorization solution for Equation
CONCLUSION
In this work, we investigate why using standard low-rank factorization (SVD) to compress the model may quickly lose most of its performance, pointing out the issue of the mismatched optimization objectives between the low-rank approximation and the target task. provide empirical evidence and observations for the issue, and propose new strategy, FWSVD, to alleviate it. Our FWSVD uses the estimated Fisher information to weigh the importance of parameters for the factorization and achieve significant success in compressing an already compact model. Furthermore, FWSVD reuses the existing SVD solver and can still implement its factorized matrices with linear layers; therefore, it is simple implement and deploy. We believe FWSVD could be of the most easy-to-use methods with good performance for language model compression.
REFERENCES
Anish Acharya, Rahul Goel, Angeliki Metallinou, and Inderjit Dhillon. Online embedding com- pression for text classification using low rank matrix factorization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp.
2019.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpin and Lucia Specia. SemEval-201 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1thh International Workshop on Semantic Evaluation (SemEval-20n) -2077), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics.

---

Published as a conference paper at ICLR 2022
Patrick H. Chen, Si Si. Yang Li, Ciprian Chelba, and Cho-Jui Hsieh. Groupreduce: block-wise low-rank approximation for neural language model shrinking. In Proceedings of the 32nd Inter- national Conference Neural Information Processing Systems, pp. 11011-11021, 2018a.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs. 2018b.
Emily I. Denton, Wojciech 7aremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances neural informa- fion processing systems, pp. 1269-1277, 2014.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Bill Dolan, Chris Brockett, and Chris Quirk. Microsoft research paraphrase corpus. Retrieved March, 29(2008):63, 2005.
Gene H Golub and Christian Reinsch. Singular value decomposition and least squares solutions. In Linear algebra, pp. 134-151, Springer, 1971.
Ting Hua, Yilin Shen, Changsheng Zhao, Yen-Chane Hsu, and Hongxia Jin. Hyperparameter-FFe free continuous learning for domain classification in natural language understanding. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, pp. 2669-2678, 2021.
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint 1rXiv:1405.3866 2014.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 10351, 2019.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kicran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- -aroiska,, et al. Over- coming catastrophic forgetting in neural networks. volume 114, pp. 3521-3526. National Acad Sciences, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11944 2019.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pp. 7021-7032. PMLR, 2021.
Yang Liu. Fine-tune bert for extractive summarization. arXiv preprint arXiv: /903. 10318, 2019.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-1122. 2019.
Matan Ben Noach and Yoav Goldberg. Compressing pre-trained language models by matrix de- composition. In Proceedings of the 1st Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pp. 884-889, 2020.
Asia-Pacific
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In In Interna- tional Conference on Learning Representations (ICLR), 2014.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
10

---

Published as a conference paper at ICLR 2022
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. pp. 142-147, 2003.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1eXivv:910.01 108, 2019.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cexsing, pp. 1631-1642, 2013.
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the 20th International Conference Machine Learning (ICML-03), pp- 720-727, 2003.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model 2019.
compression. arXiv preprint
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact task-agnostic hert for resource-limited devices. arXiv preprint arXiv:2004.0200 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804.800444450 2018.
Alex Wang, Amanpreet Singh, Julian Michael. Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-
head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828 2020a.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957, 2020b.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. 2018.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL HLT 2018, pp. 1112-1122 122. Association for Computational Linguisties (ACL), 2018.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL http:::: http://wwwwwwww..pppppptptwtpppp awwweb. wttp://///gaaatotggy..o.o 2020. eww.eedenssede..
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprini
2018.
Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Accelerating very deep convolutional networks for classification and detection. JEEE transactions on pattern analysis and machine inteliigence,38110111443335 38(10) 2015.
11

---

Published as a conference paper at ICLR 2022
A
SUPPLEMENTARY
CoLA (ALBERT base)
MNLI (ALBERT base)
FW/VD
FWEND
40.00%
90.00
23.00%
10.00%
7
10
10
Rank Group
Pank Group
(b)
QNLI (ALBERT base)
STS-B (ALBERT base)
PWSSV
90.00%
100
100.00%
40.00%
75.00%
30.00%
10,00%
3.00%
0.00%
(c)
(d)
MRPC (ALBERT base
NER-CONLL2003 (ALBERT base)
BVD
FWSVD
BVD
PWAVO
100
100.00%
00%
02%
25 00%
25,00% 03.0%%
0.00%
0.00%
10
Rank Group
Rank Group
(e)
QQP (ALBERT base)
SST-2 (ALBERT base)
00%
Performnon DeP
50 50.00%
20.00%
25.00%
10.00%
10
Rank Group
Rank Group
(h)
Figure 7: The grouped rank truncation experiment. The experiments are the same with Figure 6a we use ALBERT base (11.2 (11.7M parameters) model for this figure.
12

---

Published as a conference paper at ICLR 2022
CoLA (BERT base)
MNLI (BERT base)
SVD
FW/SVD
SVD
FW/VD
80.00%
80.00%
40.0%%
40.0%
20.00%
20.00%
2.00%
2.00%
Group
(a)
(b)
QNLI (BERT base)
STS-B (BERT base)
BVD
FWEVD
SVD
WSVG
60.00%
30.00%
20.00%
Perborwoo
20.00%
10.00%
10.00%
3.00%
0.00%
10
10
Parnk Group
Pank Groupp
(c)
(d)
MRPC (BERT base)
NER-CONLL2003 (BERT base)
75 13.00%
75.00%
Parommn'o
50 02%
50.00%
25.00%
Rank Group
Rank
QQP (BERT base)
SST-2 (BERT base)
BVD
FWVVD
BVD
FWSVD
90.00%
90.00%
60.00%
40.00%
30.00%
40.00%
Peffmeeee
20.00%
10.00%
3.00%
3.00%
Group
(g)
(h)
Figure 8: The grouped rank truncation experiment. The experiments are the same with Figure 6a but this figure includes all 8 language tasks with BERT base (109.5M parameters) model. FWSVD has a smaller performance drop with those groups truncated first (e.g., group to 10) in all the cases. SVD usually shows significant drop with group 10, which has the smallest singular values and is truncated first. FWSVD has no such issue in all cases.
13

---

Published as a conference paper at ICLR 2022
CoLA (BERT base)
MNLI (BERT base)
BVD
FWVVD
BVD
PWEVO
20.00%
15.00%
10.00%
10.00%
5.00%
5.00%
3.00%
0.00%
10
Parnk Group
Pank Grouu
(a)
(b)
QNLI (BERT base)
STS-B (BERT base)
15.00%
Paformena
3.00%
10.00%
2.00%
Rank
Rank
(d)
MRPC (BERT base)
NER-CONLL2003 (BERT base)
BVD
FWVVD
BVD
FWEVO
40.00%
8.00%
90.00%
4.00%
20.00%
2.00%
10.00%
3.00%
0.00%
(e)
(
QQP (BERT base)
SST-2 (BERT base)
BVD
FWSVO
BVD
PW/ND
10.00%
4.00%
Pertmmann Dere
Perbmana
2.0%%
2.50%
1.0%
3.00%
0.0%%
6
10
5
10
Rank Group
Rank Group
(
(h)
Figure 9: This figure shows only groups 3 to 10 of Figure 8 to better visualize the groups of smaller performance drop.
14

---

Published as a conference paper at ICLR 2022
We additionally include the averaged singular values for each truncated group. The singular values from FWSVD are multiplied with Fisher information; thus their scales are different from SVD.
Table 4: The raw values for Figure 6a
Truncated group
SVD performance drop FWSVD performance drop
25.4%
2.8%
2.2%
1.1%
0.9%
0.7%
0.4%
0.1%
4.9%
24.1%
6.1%
2.7%
1.5%
0.8%
0.6%
0.2%
0.3%
0.2%
0.2%
SVD average singular value
2.674
1.381
0.994
0.828
0.671
0.519
0.353
FWSVD average singular value
1093
298
201
157
110
15


Reviews:
[{"review_id": "nwG4sbZhdv", "reviewer": "Reviewer_jnTC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. \n\nStrengths:\n+ the approach does not require re-training (doing pre-training again)\n+ experimental results on several tasks show only a small drop in model quality\n+ the approach can further reduce the size of already-reduced-size models (e.g. TinyBERT) \n\nWeaknesses:\n- the reduction in the number of parameters is modest compared to methods that re-train the network in a compact representation, like ALBERT\n- Results in Table 2 look incomplete: why is only one task reported for each of the models? Why different task is used for each different model? It would be better to pick e.g. three tasks and use them for all four models.\n- the work does not provide much insight into the problem that motivates the approach (e.g. why \"group 10\" would affect \"important parameters\" more than \"group 9\")\n\nQuestions:\n- how common is the behavior in Fig 1? It would be interesting to see evidence of it on at least one more dataset/task. It might also be useful to use log scale for the axis, currently it is not possible to see if e.g. removing group 9 leads to lower/similar/higher performance drop vs group 8 (i.e., is group 10 special, if yes, why?).\n- what is the average/median magnitude of singular values in group 9 vs in group 10? \n- have you tried minimizing eq. (5) using SGD? Do you expect substantial gap between (5) and the simplified version in eq. (6)?\n\n\n---\nPost-rebuttal:\nThank you for the clarifications and additional experiments, I've updated my score to 6.", "summary_of_the_paper": "SVD of matrices in a language model is used to reduce model size. Instead of standard SVD that minimizes the MSE of reconstructing the matrix elements, a modified version that weights elements is used. Specifically, importance of each parameter (matrix element) is estimated based on the magnitude of gradients w.r.t. to that element. To have closed-form solution for weighted SVD, average row weights are used instead of element weights.", "summary_of_the_review": "The paper provides a simple method to reduce (modestly) the size of an already trained language model, without the need for costly re-training. The evidence in favor of the method is empirical, with little in terms of theoretical or experimental exploration of the motivating phenomenon: the overlap between \"important parameters\" and \"parameters poorly captured by SVD\".", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NUvoltr_vqw", "reviewer": "Reviewer_Kuwu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n-) Written English needs improvement.\n\n-) The idea is interesting and the authors explain the main intuition adequately clear. \nThe authors explain the advantages and limitations properly.\n\n-) How easy/fast is to generate the fine-tuned model? In other words, what is the overall \ncomputational cost to create a fine-tuned model and apply FWSVD versus generic training \nand SVD?\n\n-) Unfortunately it is not clear to me whether the proposed method comes with any guarantees. \nI understand that minimizing the Euclidean norm of the approximation of W does not necessarily \nimply lower classification (or other tasks) error, however I am not sure the proposed method does \neither. Is that so? For example, will the new method always be better than SVD? One needs to \naccount for the cost of pre-processing costs too. \n\n-) In addition to my comment above, Figure 6 seems to complicate the discussion. The legend reads \nas \"Unlike SVD, the performance drop of our FWSVD perfectly imitates the ideal trending shown in Figure 1.\", \nhowever I can see that on the left subfigure \"Rank group 8\"  leads to a higher performance drop than \"Rank group 7\" which \nis exactly what the authors wanted to avoid in the first place. Furthermore, on the right subfigure, FWSVD \nactually introduces higher reconstruction error, which does not even reduce monotonically. I understand that \nthe goal is to achieve higher accuracy in the classification task and not the matrix reconstruction part, \nbut this seems to complicate the optimization part of the model, e.g., how do the authors make sure \nthey do not overfit or spend more time than necessary computing the low-rank approximation?\n\n-) The above experiment is explained better in Section 5.5.1 but the following sentence is confusing \n\"The results suggest that FWSVD’s objective (Equation 6) aligns with the task objective better by sacrificing the reconstruction error.\"\n\n-) Section 5.2.2 is rather unclear; can you please provide more details, perhaps in the appendix? Without \nproper explanation in this step, the experiments can not be judged fairly. \n\n-) Please make the y-axis in Figure 6 run in logarithmic scale. \n\n-) While generally providing better results than SVD for the experiments shown in this paper, the proposed \nmethod does not offer dramatic improvements in accuracy, nor it is guaranteed to do so. It would be very \nhelpful if the authors could provide additional experiments about other benefits of the method, if applicable, \nin the appendix (e.g., is the new method faster than previous approaches based on SVD etc.)\n\nMinor comments:\n\n-) Equation (1) and related text is confusing. The \"\\approx\" symbol should be \"=\" since matrix W is of rank 'k'. I suggest the authors to simplify discussion by using 'r' instead of 'k' and simply mention that matrix W is of rank k\\geq r.\n\n-) \"we propose a simplification by making the same row of the W matrix to share the same importance\". Did you mean to say that each weight located in row 'i' will use the average weight of its row?", "summary_of_the_paper": "The proposed paper focuses on a new technique to compress weight matrices in Machine Learning, e.g., weight matrices of layers in DNN. \nOne idea is to simply exploit truncated SVD. While this minimizes the Euclidean norm of the error, it doesn't necessarily lead to a lower task error since the truncated part might still have a noticeable effect in the resolution power of the model. The authors suggest to solve an alternative minimization problem which takes empirical Fisher information under consideration. Experiments on a few datasets imply that the proposed method can achieve better results than simply using truncated SVD of the weight matrix.", "summary_of_the_review": "The proposed idea is interesting and seems to lead to improvements over truncated SVD, however these improvements \nare not major. Moreover, the proposed algorithm is less general than truncated SVD.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-onpufoQbH", "reviewer": "Reviewer_d1T4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. Solid experiments demonstrating the proposed method outperforms other existing approaches. \n2. Novel weighting scheme for SVD for low-rank weight compression (though should double check this more thoroughly).\n\n\n\nSome comments: \n\n1. I would suggest using the phrase \"weighted SVD\" early on in the introduction (e.g., exactly when you introduce your new method).\n2. It might be good to address the existing literature on compressing trained neural networks which also goes beyond simply trying to minimize the Frobenius norm of the difference between the weights. For example, the paper https://arxiv.org/abs/1505.06798 also goes beyond simply considering the reconstruction objective on the weights, and includes the nonlinearity as well in the reconstruction objective. Your work obviously is different enough to stand on its own, but it might be good to make a mention of this work and others (e.g. do more of a lit search / related work on low rank compression).\n3. It might be good to find the best hyperparameters for each setting to truly do a fair comparison (avoiding hyperparameter tuning isn't really a fair comparison, IMO -- but this depends on how much compute you have available). \n4. You forgot to bold the best performer in line 3 of Table 2 (in this case, the original compact model).\n", "summary_of_the_paper": "This paper studies the compression of the weights of a neural network via a modification of the standard paradigm of using low rank approximations via SVD on the weight matrices. Instead, they propose a weighted SVD objective where the weights are computed based on the Fisher information of the (trained) parameter with respect to the dataset. They simplify the method so it is more computationally tractable by having the same row of each weight matrix share the same weight. This translates to solving SVD on a modified reconstruction objective (multiply by a diagonal). Here the Fisher information for parameter w is approximated by the average squared l2 norm of the gradient with respect to w over the empirical data distribution.\n\nThey then conduct several experiments with this approach in the problem domain of compressing language models. They evaluate their method on compressing task specific models as well as the problem of compressing an already compressed model, and demonstrate their approach is generally better.\n\n", "summary_of_the_review": "The paper introduces a new approach to weighting the low-rank compression of neural nets, and empirically demonstrates that it outperforms other methods. The approach isn't too surprising, but it does appear to be novel (though I would recommend doing a more thorough lit review and make the related work section larger, since I feel certain that there are many more related papers doing similar things (I mention one in the main review). I think the experiments are overall good and demonstrate the point nicely, which is essentially that the important thing in compressing networks is not the parameter compression -- but rather, the compression of the function specifically on the relevant distribution. Overall I vote to accept.\n\nUPDATE:\nI read the comments, my opinion is unchanged. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the problem of task-specific model compression obtained from fine-tuning large pre-trained language models. The work follows the line of research in which model size is reduced by decomposing the matrices in the model into smaller factors. Two-step approaches apply SVD and then fine-tuned the model on task specific data. The present work makes the observation that after the first step (the SVD compression) the model can dramatically lose its performance, due to the mismatched optimization objectives between the low-rank approximation and the target task. The work provides evidence backing this claim. The paper proposes to address this problem by weighting the importance of parameters for the factorization according to the Fisher information. Experimental evaluation shows that the proposed method can achieve better results than variants that use truncated SVD of the weight matrices.

The paper is well written and easy to read. The method is simple and effective and can be applied to in a wide range of settings. The authors provided a thorough response which clarified several points. This led Reviewer Kuwu to increase the score to 6.

All three reviewers agree that the main observation in the work is interesting and informative for researchers and practitioners working on the problem.

Reviewer jnTC points out that the paper would have been stronger if it included theoretical exploration of the reasons behind the "importance of low SVs" phenomenon.

Reviewer Kuwu and jnTC consider the results marginally novel. Reviewer Kuwu considered the significance of the reported results to be limited, and put the work marginally above the acceptance threshold. Reviewer jnTC disagrees with this view, considers and appreciates the generality of the method and the fact that it can work well even for compressed models, while improving in accuracy by a few percent over competing approaches which result in similar parameter counts. The AC agrees with Reviewer jnTC.

Overall all reviewers consider the paper borderline but recommend accepting the paper. The AC overall the topic important (reducing the footprint of language models), the method simple and well motivated. The empirical evaluation is very thorough and shows clear gains across a large number of settings.
Decision: Accept (Poster)