Dynamic Low-rank Estimation for Transformer-based Language Models
Authors: Ting Hua
Xiao Li
Shangqian Gao
Yen-Chang Hsu
Yilin Shen
Hongxia Jin
Venue: EMNLP 2023 Findings (2023)
Keywords: low-rank estimation; matrix factorization;

Abstract: Matrix decomposition methods, such as Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformer-based language models. While importance-weighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two fundamental assumptions:  1) unchanged importance distribution during further fine-tuning,  2) equal importance across weight matrices in different layers. 
Furthermore, these methods necessitate a well-trained task-specific model as the starting point and require additional fine-tuning after compression.
In this work, we proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process.
Starting from a general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuning.
The extensive evaluations demonstrate that RankDyna  can outperform  current SOTA methods under various parameter budget levels, and the advantage of RankDyna is further enhanced with higher compression rates.

Dynamic Low-rank Estimation for Transformer-based Language Models
Ting Hual Xiao Li Shangqian Gao', Yenchang Hsu', Yilin Shen', Hongxia Jin'
*Samsung Research America, University of Michigan, Ann Arbor [[ting. hua, s. gao1, yenchang. hsu, yilin. shen, hongxia. jesaamsung...... com 2 {[xlxia))eumich.. edu
Abstract
applying low-rank estimation methods (Golub and Reinsch, 1971; Noach and Goldberg, 2020), the large weight matrix in a layer can be decomposed into two smaller matrices, resulting in the creation of two reduced-size linear layers. Also, it is impor- tant to note that matrix factorization technologies are orthogonal to other compression approaches such as Knowledge distillation (Sun et al., 2019; Sanh et al., 2019; Jiao al., 2019) or Quantization (Shen et al., 2020; Zhao al., 2021).
Matrix decomposition methods, such Singular Value Decomposition (SVD) and its importance-weighted variants, have been widely used for compressing Transformer- based language models. While importance- weighted decomposition methods alleviate the strong assumption of equal importance for each parameter in SVD, they still rely on two funda- mental assumptions: 1) unchanged importance distribution during further fine-tuning 2) equal importance across weight matrices in different layers. Furthermore, these methods necessi- tate a well-trained task-specific model as the starting point and require additional fine-tur tinettunin after compression. In this work, proposed RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process. Starting from general pre-trained model, RankDyna accomplishes the dual goals of compression and adaptation to the downstream task, all within a single round of fine-tuming. The extensive evaluations demon- strate that RankDyna can outperform current SOTA methods under various parameter budget levels, and the advantage of RankDyna is fur- ther enhanced with higher compression rates.
The standard SVD aims to minimize the recon- struction error, which results in the removal of the portion associated with small singular values. How- ever, previous work have revealed that this objec- tive does not always align with the objective of preserving task performance (Hsu et al., 2022; Hua et al., 2022), This is primarily due to the fact that parameters have various influences on the overall performance, whereas the standard SVD treats all parameters equally significant.
To tackle this issue, previous work FWSVD (Hsu et al., 2022) and TFWSVD (Hua et al., 2022) at- tempt to conduct low-rank estimation the matrix weighted by Fisher information. These approaches need an already fine-tuned model as their start- ing point to calculate importance scores based on Fisher information, and often require an additional round of fine-tuning to attain better performance. Besides, they are based on the assumption that the Fisher information of parameters, computed from fine-tuned task-specific model, can accurately represent the importance distributions of those pa- rameters. Additionally, they also assumed that these importance distributions remain unchanged throughout the additional round of fine-tuning.
1 Introduction
Transformer-based language models (Devlin et al., 2018; Waswani et al., 2017) have obtained signifi- cant success a variety of Natural Language Pro- cessing tasks. However, these models are usually pre-trained by the huge corpus with millions or even billions of parameters, and hard to be de- ployed to resource-constraingg fore, the compression of the Transformer-based language model have attracted extensive attentions. The Transformer blocks are primarily built with linear layers, which will convert the input features into output features through a weight matrix. By
devices. There-
In this paper, we propose novel method called RankDyna, which conducts low-rank estimation throughout the fine-tuning process. Starting with full-rank matrix, RankDyna gradually reduces the parameter budget level using a scheduler. At each step, the least important singular groups are
Theee authors contributed equally to this work.

---

removed based on their importance scores, which are determined by the amount of impact they con- tribute to the loss. RankDyna collectively evaluate the importances of all singular groups within the model, and therefore capable of assigning different ranks to each matrix. The advantages of the pro- posed RankDyna can be summarized follows: Dynamic tracking of importance for singular groups. Previous importance-aware decomposi- tion approaches aimed to optimize an objective weighted by static importance assigned to each pa- rameter. However, assuming that the importance distribution of parameters remains unchanged dur- ing fine-tuning is unrealistic. Unlike these meth- ods, RankDyna can capture the dynamic changes in importance associated with singular groups, thus improving performance compared to the methods based on the static importance assumption.
zero singular values (ia oi), where 01 02 ス 0) > 0. Ur, E,, and V,, are the truncated matrices with rank r and serve as approximations to the original matrix. In practice, it is common to represent matrix W as the product of two smaller matrices P E I IMMx xr and Q E B*X,, by setting P = US and Q = VT, Given Equation (1), the forward pass of linear layer in the Transformer can be rewritten as below:
h = Wx + bb=
(2)
where x is the input feature, b is the bias. The above process can be implemented with two smaller liner layers, whose weight matrix is equal to P and Q correspondingly. The number of parameters in the original weight matrix W is MN, and the combined number of parameters in P and Q is Nr + Mr. Therefore, by utilizing matrix factoriza- tion techniques, we can reduce the model size by NM (Nr+ + Mr).
Better parameter allocation accross matrices. The varying sensitivities of different layers in Transformer models during compression are well- established. Given a targeted model size budget, previous work on low-rank estimation will assign uniform ranks to all layers. In contrast, RankDyna enables dynamical parameter allocation across lay- ers by monitoring their importance variations, lead- ing significantly better compression outcomes. Reduced computational and storage resources. Unlike previous methods that rely on already fine- tuned task-specific models, our approach begins with a general pre-trained model. As a result, Rank- Dyna only needs a single round of fine-tune fine-tuning. This significantly reduces the computational overhead compared to methods that require two rounds of fine-tuning. Previous approaches require factoriza- tion for each fine-tuned task-specific model that can be increasingly expensive when model size gets larger'. In contrast, RankDyna only requires a one-time factorization on the base model, which can be shared across all downstream tasks.
2.2 Related work
Transfomer-based language models (Waswani et al., 2017; Devlin al., 2018; Brown et al., 2020; Raffel et al., 2020) have attracted significant atten- tion and, as result, there is growing demand to reduce their model size. Although matrix decompo- sition approaches, such as LoRA-like models (Hu et al., 2021; Zhang et al., 2023), have demonstrated their effectiveness in adapting pre-trat pre-trained models to downstream tasks. The predominant focus of pre- vious techniques for compressing language models has been on knowledge distillation (Sanh et al., 2019; Jiao et al., 2019), quantization (Shen et al.. 2020; Kim et al., 2021), and pruning (Sanh et al.,
2020; Xia et al., 2022). Only a few attempts have been made in the direction of matrix decomposition towards the goal of compressing language model.
The closest prior research direction to this paper involves approaches that compress Transformer- based language models using matrix factorization: FWSVD (Hsu et al., 2022) and TFWSVD (Hua et al., 2022). However, there exist at least two inherent drawbacks in these methods that can po- tentially cause performance bottlenecks. Firstly, the importance of parameters is assumed to remain static throughout the training process. Secondly, uniform rank budget typically allocated all weight matrices. In contrast to these approaches, our proposed RankDyna is capable of addressing both challenges.
2 Background
2.1 Matrix factorization
The matrix factorization methods can decompose a large matrix W E MxN into smaller matrice:
W- - (U, P,Qr, (1)
where U C M VE Nx!, and I is the rank of matrix W. E diagonal matrix composed of non-
For weight matrix W E M*N, the computational complexity of SVD is Ominn (2 2M)

---

Another research direction related to our work is the importance criteria for parameters. Using the magnitudes of parameters as a measure their importance a straightforward and intuitive ap- proach (Zhu and Gupta, 2017; Renda et al., 2020). However, this basic metric fails to quantify the actual contributions of parameters toward the over- all performance. A more reliable measurement is sensitivity, which approximates the change in the loss function resulting from the removal of spe- cific parameter (Molchanov et al., 2019; Sanh et al.,
RMXN the sum of the singular groups:
Σ ORUATE ~ (onuu)oo k=1
(px)gk
1=1
A=1
(5)
where is the k-th singular value. and Uk are k-th columns in U and V. Pk and @k are k-th columns in P and Q. The approximation occurs when the target rank is less than the original rank Note that, setting the k-th singular value og zero will completely eliminate the effect of the singular group Φx = {uu, ok. U/ in the recon- struction process. Based on this observation, we adopt singular group d% as the fundamental unit for reconstructing the matrix W.
2020), which is also the measurement adopted by proposed RankDyna.
3 Methodology
this section, we introduce RankDyna, a solution that enables dynamic resource allocation by mon- itoring changes in parameter importance during fine-tuning. This approach effectively addresses the above mentioned issues in previous work. As overview, outline proposed RankDyna in Algorithm
Algorithm 1: RankDyna
Input: Data set D, total training steps T, targeted compress ratio c.
for 1,.... T do
Sample mini-batch from D;
3 Calculate current rank R(O) through Equation (12);
for each (W)(() in ((M)() for each Φx in & ((W) do
3.1 Problem statement
s
Dyna aims to compress a pre-trained language model M by decomposing the weight matrices set W in its linear layers* At the initial step (denoted step 0), the total number of ranks (0) in all
Update gradients through Equation (6) (8);
Compute importance score
I(eb)) in Equation (9);
matrices of model M is computed
Update moving average score
I(((() Equation (11);
R()) = Σ Iw.
WEW
Update global remained set ((M)(t) and all the local remained set $ (WW) (W)())
where Iw is the original rank number of matrix W. Given the desired compression ratio c, the targeted number of overall ranks R!T) after T steps of fine- tuning determined follows:
10 for each weight matrix W in Model M do Compute P and Q by singular groups in ((W)(T) through Equation (5);
Output: Compressed model M
RT = c- R(O)
(4)
where [-] is the rounding up operator. Given targeted rank budget R(T), our task is search for the optimal resource allocation for model M.
3.3 Iterative approximation
In this part, we discuss how to update singular groups during the training. To start with, we initial- ize U, Σ, and V by performing standard Singular Value Decomposition (SVD) on the weight matrix W. At the t-th step, a stochastic gradient descent step is taken to update elements each singular group. For Σg. the update of its only non-zero element at step L can be described as follows:
3.2 Singular group
Besides Equation (1), an alternative perspective on matrix decomposition is to approximate W E
'For Transformer-ee models, W includess various types of weight matrices, such as W, (query). We (key). W. (valuc). W. (attention Wu (first FFN), and W5, (sccond FFN).
ok + ox (1-1) V.
(6)

---

where 1) is the learning rate. For the i-th parame- ter in vector uy or uk. the update at step & can be denoted as:
3.5 Rank allocation scheme
The approach for tracking changes in parameter importance has been introduced in the previous sections. In this part, we will discuss how to inte- grate it with dynamic rank resource allocation. Scheduler for adjusting rank budget. Inspired by the sparsity scheduler widely used in pruning (Zhu and Gupta, 2017: Sanh et al., 2020), we grad- ually decrease the rank budget during training by a dynamic scheduler. Specifically for our task, the scheduler works as three stages. 1) Warm-up phase. The initial to steps are considered the warm-up phase, during which the current rank budget is kept as R(O) in Equation (3). 2) Cool-down phase. The last t1 steps treated the cool-down phase, where the targeted rank budget R(1)) computed in Equation (4) has already been attained and is main- tained till the end of training. 3) Adjusting phase. The steps between ln and t are adjusting phase, where the current R(() decreases as follows:
10) (1-1) 2 f Uk,
v/ C.
2 Uk
(t-1)
V uh. C.
(8)
3,4 Importance score of singular-group
Unlike previous work that evaluates the importance of individual parameters, our proposed RankDyna estimates the importance changes for each singular group as a whole. Specifically, the importance score of a singular group is defined as follows:
d1
Σ
(1) ((
(
+
(9) where S(.) is score function to measure the change in loss if we remove a certain parameter, d1 and d2 represent the dimension of uy and 2) respec- tively. Given any parameter 8, the change in loss of removing 0 can be formulated by the second-order Taylor series expansion (Le Cun et al., 1989):
R = =R + (122 R(
t- 4
(12)
- -
Global rank allocation. To evaluate the impor- tance differences among parameters across differ- ent weight matrices, we maintain two sets of sin- gular groups: 1) The local remaining set ((W). which comprises all singular groups of weight ma- trix W: and 2) The global remaining set, which is the union of all local remaining sets, denoted as ob (M). Initially, remaining set (M)) contains all singular groups in model M, thus (*(M)(() is equal to R!)), At step I, for any singular group o, if its importance score [(c(0) is not among the top-R(()) of ((M)(1, RankDyna will remove é from both global remained set (MM)() and local remained set & (W))).
S(0) = ACol = |C- - C-ol
(10)
= =10T - VoL + O HA 0(10)))))
where C -0 is the loss with parameter zeroed out, and H is the Hessian matrix. The larger the value of (0), the greater impact the parameter 0 will have on the loss L, indicating the higher importance. Note that, the importance comparisons based S(0) are not limited parameters within the same matrix or layer, but are applicable to any parameters in model M. More details about the calculation can be found in Section 4.5.2.
Experiment
4.1 Tasks and datasets
RankDyna and baselines are evaluated on GLUE benchmark (Wang et al., 2019) and the NER task the CoNLL-2003 dataset (Sang and De Meul- der, 2003). Also, we reported the results of the summarization task (ROUGE score) on SAMSUM dataset, and language modeling task (Perplexity) on the Wikitext2 and FTB datasets. More details can be found in Appendix A.I.
Previous work (Zhang al., 2022) has pointed out that: the importance scores calculated using Equation (10) may exhibit fluctuations, as they are computed on randomly sampled mini-batches. In practice, these fluctuations can be mitigated by ap- plying the "momentum" scheme (Rumelhart et al.. 1986), where the next update is determined as weighted average of the current update and the pre- vious score:
4.2 Implementation details and baselines
We utilize pre-trained BERT (Devlin et al., 2018), GPT-2 (Radford et al., 2019), and BART (Lewis
[()(()) = 3 -0(0 (1-1)) + (1- - - (11)

---

Table I: Results of CoNLL and GLUE benchmark. All model sizes reported here exclude the embedding layer. G-Avg represents the average of GLUE tasks, while A-Avg denotes the average of all tasks, including CoNLL..
Task
#Param
CoNLL
CoLA
MNLI
MRPC
QNLI
QQP
SST-2
STS-B
G-Avg
A-Avg
Bert_base
86.2M
94.1
56.2
84.7
87.4
91.3
87.8
93
88.5
84.1
85.4
SVD
43.1M
92.4
40.5
82.8
84.1
89.6
87.3
90.9
85.7
80.1
81.6
FWSVD
43.1M
93.2
49.4
83.0
88.0
89.5
87.6
91.2
87.0
82.2
83.6
TFWSVD
43.1M
94.2
52.2
83,4
89.0
90.3
86.9
91.1
88.5
83.1
84.4
RankDyna
43.1M
94.3
54.5
83.2
89.5
90.6
88.0
90.9
89.0
83.7
85.0
SVD
26.6M
92.8
19.3
81.0
82.0
86.6
86.9
89.2
80.6
75.1
77.3
FWSVD
26.6M
92.9
38.7
81.4
80.3
88.0
87.2
88.4
82.9
78.1
80.0
TFWSVD
26.6M
93.5
39.3
82.2
88.3
88.8
87.0
89.9
87.0
80.4
82.0
RankDyna
26.6M
93.3
49.1
82.5
88.0
89.2
87.5
90.7
88.3
82.1
83.6
SVD
13.9M
90.4
13.8
78.0
82.0
79.6
84.1
87.5
58.7
69.1
71.7
FWSVD
13.9M
3.5
18.7
78.2
78.6
82.3
84.5
88.9
67.9
71.3
62.8
TFWSVD
13.9M
91.9
21.4
79.1
85.0
84.3
85.9
89.0
86.0
75.8
77.8
RankDyna
13.9M
91.6
36.2
81.3
84.7
87.6
86.8
88.2
86.4
78.7
80.4
SVD
6.4M
87.4
0.0
71.6
74.7
65.8
79.8
84.3
22.7
57.0
60.8
FWSVD
6.4M
88.4
17.2
73.5
80.9
71.2
81.3
84.9
39.2
64.0
67.1
TFWSVD
6.4M
87.8
17.8
76.7
81.2
76.5
83.4
81.3
45.3
66.0
68.8
RankDyna
6.4M
88.4
21.3
79.2
82.1
85.5
86.1
87.5
83.3
75.0
76.7
al., 2020) as the starting points for all models, namely RankDyna, TFWSVD, FWSVD, and SVD. For TFWSVD, FWSVD, and SVD, we first fine- tune the base model using task-specific data for 8 epochs and then apply these low-rank factoriza- tion methods to the obtained task-specific mod- els. Finally, we conducted an additional round of fine-tuming with epochs. For RankDyna, di- rectly fine-tune the base model for epochs each downstream task. More details can be found in Appendix A.2.
Model
R1
R2
RL
BART-base
52.1
27.3
43.5
SVD
45.3
21.3
37.5
FWSVD
47.0
22.6
38.8
TFWSVD
47.8
23.1
39.3
RankDyna
48.4
23.4
39.7
Table 2: Compressing BART model for summarization task SAMSUM (Higher is better). By applying a desired rank ratio of 0.2 across all methods, we reduce these 94.6M parameters I..
all the tasks. Moreover, as the compression rate increases, the advantage of RankDyna becomes more apparent. Even under the limited budget of 6.4M linear parameters, our RankDyna is capa- ble of maintaining respectable G-Avg score of 75.0 and A-Avg score of 76.7. These scores are comparable to the G-Avg score and A-Avg score generated by other methods operating under a more generous 13.9M parameter budget.
4.3 Performance comparisons with SOTA
The results of GLUE tasks and one NER task (CoNLL) are presented in Table Our RankDyna model, with 43. linear parameters, achieves a G-Avg score of 83.7 and an A-Avg score of 85.0. In fact, RankDyna not only surpasses the scores of state-of-the-art (SOTA) compression methods (SVD, FWVSD, TFWSVD) that require additional fine-tuning, but also beats the full-size Bert base model in some tasks (e.g.. QQP, STSB, MRPC, and CoNLL) with half parameter budget. This suggests that our RankDyna functions as both a compression method and an adaptation approach. This not surprising, considering the proven effec- tiveness of low-rank adapters like LoRA in fine- tuming the general pre-trdined model for down- stream tasks (Hu et al., 2021).
In most settings, the standard SVD method shows the poorest performance. But it exhibits greater stability compared to the two importance- aware decomposition methods. TFWSVD is the second-best performer in most cases. However, this method heavily relies numerical optimiza- tion process for matrix factorization, which may be trapped in local optima (Hua et al., 2022). How- ever, TFWSVD is the approach that requires the most computation resources. More details about
RankDyna consistently yields good results on

---

Figure I: Importance changes of singular groups. To simulate the starting points of FWSVD and TFWSVD, we further fine-tur the Bert base model the STSB dataset for 1200 steps. In the initial step, groups within each weight matrix in descending order, based on their importance scores. We then select specific groups (e.g., Group I. 10, 40, this case) and monitor their relative importance positions throughout subsequent steps. At step 200, remove 80% of less important singular groups in W Wa. W. and W f and observe the changes in importance of the singular groups in Wo and Wz--
the singular
#1
# 10
# 40
# 70
(a) Layer 1. Wo
(b) Layer 5, W.
(c) Layer 12, W.
Order mporanne
200
302
150
175
403-
420
Stapk
100
1000
1200
200
400
Stepa
8000
500
1000
1200
200
400
600
800
1000
1200
(d) Layer 1, W/
(e) Layer 5, W
(l) Layer 12, W
Orter of Inperanne
20
50.
100
150
150
200
300
260
200
030
1083 1750
203
400
800
800
1000
7200
200
400
500
800
1000
1200
Staps
the training time can be found in Appendix A.3. We have additional results on generation tasks, such as summarization and language modeling. Table shows the performance of compressing BART model summarization dataset SAMSUM, and Table the comparison of compressing GPT-2 model through language modeling task on dataset wiki2 and PTB. Generally, these results show the similar patterns we observed in GLUE dataset: our RankDyna can achieve the best perfor- mance, TFWSVD is the second best performer, and FWSVD shows consistently better performance than traditional SVD.
Model
wiki-2
PTB
GPT-2
20.9
21.4
SVD
124.7
79.6
FWSVD
121.6
72.2
TFWSVD
79.3
62.9
RankDyna 77.7
39.4
Table 3: Compressing GPT-2 model for language mod-
eling tasks. (Lower is better) By applying rank ratio of 0.2 for all methods, we reduced this parameter count to 28.9M.
tant singular groups (such as Group and 10) main relatively stable throughout the entire pro- cess. However, other groups, such as Group 40 and Group 70, already exhibit fluctuation patterns be- fore the removal of singular groups. These observa- tions definitely contrast with the stable assumption held by FWSVD and TFWSVD.
4.4 Importance changes during fine-tuning
The two underlying assumptions held by pre- vious importance-aware decomposition methods, FWSVD and TFWSVD, are as follows: 1) Stable assumption: The distribution of importance over parameters is supposed to remain relatively stable during the subsequent fine-tuning process. 2) Inde- pendent assumption: The matrices assumed to be independent of each other.
Furthermore, observed significant fluctua- tions in all matrices immediately after removing unimportant singular groups in previous layers. This observation suggests that the removal of singu- lar groups in one weight matrix could trigger chain reactions in the importance scores subsequent
As can be seen from Figure 1, the most impor-

---

4.5.2 Approximation of importance score In this section, we examine different approxima- tions of importance scores shown in Equation (10). To conduct this investigation, we fist fine-tune the base BERT model into task-specific models and then calculate the different scores at the conver- gence stage. The results shown in Table 4 indicate that the second-order term in Equation (10) and the Fisher information can be considered negligible when compared to the first-order term. Specifically, the magnitude of the first-order term in Equation (10) (around E-05) significantly larger than that of the second-order term (around E-09) and the Fisher information (around E-08). Therefore, this work, we use the first-order term to approxi- mate the importance score shown in Equation (10). layers, demonstrating the invalidity of the indepen- dent assumption held by FWSVD and SVD.
by significant margin. We also observed a slight increase in the loss of the baseline (local) after re- ducing the parameter budget, indicating that the baseline may mistakenly delete important parame- ters, resulting in a drop in performance.
In contrast, the success of RankDyna does not rely on these two assumptions, as we actively mon- itor the dynamic changes in importance for all sin- gular groups across different layers.
4.5
Ablation Study
4.5.1
Global vs. local rank allocation
In order to examine the impact of global rank allo- cation, discussed in Section 3.5, we introduce variant baseline approach. This approach selects the top-R(() singular groups within each matrix, instead of utilizing the global allocation strategy utilized by RankDyna.
Figure 2: Comparison of RankDyna (green line named "global"") and its baseline variant (blue line named "lo- cal"") with STSB task. The red dashed line in Figure 2b represents the rank budget scheduler (Section 3.5).
87.5-
Note that the scenario we studied here is the trained task-specific model. FWSVD and TFWSVD assume that the first-order term is zero in such models, which leads to the utilization of Fisher information as their importance score. How- ever, our findings demonstrate that the first-order term remains non-negligible, thereby challenging the validity of solely relying on Fisher information as the importance score.
82.5
Peerson Cort.
80.0
73.0
Local Global
70.0
67.5
0.05
0.075
0.1
0.126
0.16
0.176
0.2
Rank RankRalio
(a) Different rank budget levels.
2.00
4.6 Rank distribution over layers
1.75
Global
Figure shows the rank distributions generated by RankDyna on MNLI with 13.9M and 6.4M linear layer parameter budgets. The rank distributions of the fixed assignment can be found in the Appendix. RankDyna can effectively assign different ranks to the weight matrices based on their importance toward performance. Taking the 13.9M model an example, each weight matrix is supposed be assigned 77 ranks. Figure illustrates that when using RankDyna, the majority of weight matrices are assigned ranks approximately around 77, while only small number of them attaining an exact rank of 77. As the parameter budget decreases from 13.9M 6.4M, more deep green blocks (rep- resenting matrices with low-rank assignments) ap- pear. However, the most important matrices are still assigned high ranks despite the tighter budget. We have also observed that RankDyna tends to allocate a greater portion of the budget to the front
1.50
0.8
1.25
Trainin
1.00
Rank Ratio
Ronk
0.75
0.4
0.50
0.25
0.00
1000
2000
3000
4000
5000
Stsps
(b) Training loss.
As shown in Figure 2a, the performance of Rank- Dyna (global) consistently surpasses that of its vari- ant baseline (local) across different parameter bud- get levels. To further analyze the underlying factors behind this phenomenon, we look into the trends of loss changes shown in Figure 2b. Initially, both methods exhibit similar behavior. However, as we dynamically adjust the rank budgets, RankDyna (global) starts to outperform the baseline (local)

---

Table 4: Comparison of first-order, aecond-ordes and empirical Fisher information importance score in trained specifc models. We fine-tune Bert model different tasks for 6 epechs and calculate the average of importance scores of the last 100 steps. The first-order and second-order term are the first and second term in Equation (10). The definition and calculation of Fisher information can be found in Appendix.
CoNLL
CoLA
MRPC
QNLI
SST-2
STSB
First-order term
2.53E-05
2.84E-05
2.59E-0S
2.74E-05 2.94E-05
2.50E-05
Second-order term
6.84E-09
7.30E-09
8.16E-09 5.16E-09
6.06E-09
8.07E-09
Fisher information
1.23B-08
1.17E-08
1,03E-08 9.83E-09 1.11E-08
1.03E-08
Figure 3: The resulting rank of each weight matrix when compressing Bert-base MNLI with RankDyna. The x-axis represents the layer index, while the y-axis represents different types of weight matrices.
(a) 13.9M linear parameters (equal to rank ratio 0.1)
(b) 6.4M linear parameters (equal to rank ratio 0.05)
222
224
W.
11
25
S7
333
-75
Wk
20
27
212
302
134
343
131
Ws
329
93
43
10
12
25
35
55
249
51
W
122
75
152
363
244
21
305
204
302
106
75
73
64
42
14
60
75
72
23
71
330
W.
326
114
300
307
202
125
125
110
W.
4.00
52
22.3
57
10
50
67
61
22
343
165
We
25
Wo
We
3
9 11
layers. This trend is evident in both Figure 3a and Figure 3b, where a higher number ranks is preserved in the front layers, particularly in the attention matrices Q K, and V. In contrast, more re- dundant parameters are eliminated in the tail layers. For instance, both Figure 3a and Figure 3b demon- strate that the last FFN layers receive zero rank allocations. This behavior aligns with the empiri- cal observation presented in Q-BERT (Shen et al., 2020) for the MNLI task.
Figure 4: Comparison of selection strategy of SVD and RankDyna. Each here represents a subset containing 55 singular groups, sorted by the average singular values. The colors denote the portion of singular groups that are selected the top important groups. The green color represents the selection made by SVD, while the red color indicates the selection made by RankDyna. The matrix here is the Value matrix of the last layer in 26.66 26.6M model fine-tuned on QNLI.
(a) SVD
(b) RankDyna
Singulaar Value
Rank based
Slnguide Value
Importance
4.7 Selection strategy
standard SVD, only the portion associated with large singular values will be kept in the compressed model. It is already established that this selection strategy may not necessarily align with the ultimate objective of maintaining task performance. Then, under the selection strategy of RankDyna, which parameters are chosen as the important ones? Fig- ure illustrates the different selection strategies conducted by SVD and RankDyna. As can be seen from Figure 4a, only the singular-groups with the largest singular values are treated as the important ones under SVD. In contrast, RankDyna will pick up more groups from those with larger singular values, and at the same time, it will also choose certain singular groups with smaller singular values as long as they are important for the final perfor- mance. The behavior of RankDyna demonstrates a well-balanced objective that considers both the
Rank Group
Rank Group
numerical features of the weight matrix itself and the model performance.
Conclusion
RankDyna provides a comprehensive solution for compressing Transformer-bant models. Through the integration of importance tracking and dynamic allocation of ranks across different layers, Rank- Dyna addresses the unsolved issues left by the pre- vious importance-awart decomposition methods. Extensive results demonstrate that RankDyna out- performs current by significant margin in terms of performance.
(SOTA) models

---

Limitations
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Language model compression with weighted low-rank factorization. In International Conference on Learning Representa- lions (ICLR). Edward Hu, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li. Shean Wang, Lu Wang. Weizhu Chen, et al. 2021. Lora: Low-rank adaptation of large lan- guage models. In International Conference on Learn- ing Representations(ILRR)
The primary limitation of RankDyna lies in its po- tential requirement for additional memory to keep tracking the importance scores during fine-tuning. For each weight matrix, an equivalent amount of memory needed to enable the computation of importance scores. The need for additional mem- ory is specific to the training phase only. During inference, the compact models produced by Rank- Dyna function similarly to models generated by other matrix decomposition methods.
Ting Hua, Yen-Chang Hsu, Felicity Wang, Qian Lou, Yilin Shen, and Hongxia Jin. 2022. Numerical opti- mizations for weighted low-rank estimation on lan- guage models. In Proceedings of the 2022 Confer- ence Empirical Methods Natural Language Processing (EMNLP), pages 1404-1416.
6 Ethical Statement
For our experiments, we used open datasets with- out sensitive information, which have been widely mentioned in previous work. No licenses are re- quired for the GLUE dataset and CoNLL dataset. In the implementation of our model, we do not think there is an obvious issue that may lead to risk to ethics.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language under- standing. arXiv preprint arXiv: 1e0..99999033 10351.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael Mahoney, and Kurt Keutzer. 2021. 1-bert: Integer- only bert quantization. In International conference on machine learning (ICML). pages 5506-5518.
References
Yann Le Cun, John S Denker, and Sara A Solla. 1989. Optimal brain damage. In Proceedings of the 2nd International Conference on Neural Information Pro- cessing Systems (NeurIPS), pages 598 605.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. volume 33, pages 1877-1901.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to to-e-uuenceee pre- pre-training for nat- ural language generation, translation, and comprehen- sion. In P Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Daniel Cer, Mona Diab, Eneko Agirre, Tdigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 1hth International Workshop on Semantic Evaluation HommEval-20177 pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. volume 19, pages 313-330.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. Quora question pairs.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els. In International Conference on Learning Repre- sentations(ICLR))
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv: 1810.0400..
Molchanov, Tyree, Karras, T Aila, and Kautz. 2019. Pruning convolutional neural networks for re- source efficient inference. In 5th International Con- ference on Learning Representations, ICLR 2017- Conference Track Proceedings.
Bill Dolan, Chris Brockett, and Chris Quirk. 2005. Mi- crosoft research paraphrase corpus. Retrieved March, 29(2008):63.
Bogdan Gliwa, wona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. Samsum corpus: A human- annotated dialogue dataset for abstractive summa- rization. page 70.
Matan Ben Noach and Yoav Goldberg. 2020. Compress- ing pre- trained language models by matrix decompo- sition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Compu- tational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884-889.
Gene H Golub and Christian Reinsch. 1971. Singular value decomposition and least squares solutions. In Linear algebra, pages 134-151. Springer.

---

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Lan- guage models are unsupervised multitask learners. volume page
for natural language understanding. In 7th Inter- national Conference on Learning Representations, ICLR 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2018, Neural network acceptability judgments.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. volume 21, pages 1-67.
Waswani, N Shazeer, Parmar, J Uszkoreit, L Jones, A Gomez, L Kaiser, and Polosukhin. 2017. Atten- tion is all you need. In NIPS.
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2018. A coveraee challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI. HIT 2018, pages 1112- 1122. Association for Computational Linguis- tics (ACL).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392.
Alex Renda, Jonathan Frankle, and Michael Carbin. 2020. Comparing rewinding and fine-tuning in neural network pruming. In International Conference on Learning Representaaiions/(CLR))
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: State-of-ther of-the--rt natural language processing, In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational Linguistics.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. 1986. Learning representations by back- propagating errors. volume 323, pages 533 533.536.70 Na- ture Publishing Group UK London.
Erik Tjong Kim Sang and Fien De Meulder. 2003. In- troduction to the conll-2000 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011000
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022. Structured pruning learns compact and accurate mod- els. In Proceedings of the Annual Meeting of the As- sociation for Computational Linguistics (ACL), pages 1513-1528.
Victor Sanh, Thomas Wolf, and Alexander Rush. 2020. Movement pruning: Adaptive spansity by fine-tuning. In Advances in Neural Information Processing Sys- tems (NeurIPS), volume 33, pages 20378-20383
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. 2023. Adaptive budget allocation for parameter-efficient fine-tuning. In International Con- ference on Learning Representaations((LLR))
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao. Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAA/ Conference on Artificial Intelligence, vol- ume 34, pages 8815-8821.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengeheng He, Weizhu Chen, and Tuo Zhao. 2022. Platon: Pruning large transformer mod- els with upper confidence bound of weight impor- tance. In International Conference on Machine Learning (ICML), pages 26809-2688 22802.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning. Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods in natural language processing, pages 1631 1661-
Changsheng Zhao, Ting Hua, Yilin Shen, Qian Lou, and Hongxia Jin. 2021. Automatic mixed-precision quan- tization search of hert. In the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI). pages 3427-3433.
Michael H Zhu and Suyog Gupta. 2017. To prune, or not to prume: exploring the efficacy of prun- ing for model compression. arXiv preprint arXiv:1710.01888.
Sigi Sun, Yu Cheng. Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model com- pression. arXiv preprint vrXiv:/908.093::
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Glue: A multi-task benchmark and analysis platform

---

A
Appendix
Table 5: Hlyper-paramenn setting for training Rank-
Dyna.
A.1
Datasets
Dataset
Batch Size
Initial
Final
Two single-sentence tasks are included in eval- uation: CoLA (Warstadt al., 2018) measured in Matthew's correlation, SST2 (Socher et al., 2013) measured classification accuracy. Three sentence similarity tasks are considered: MRPC (Dolan et al., 2005) measured in F-1 score, STS-B (Cer et al., 2017) measured in Pearson-Spearman cor- relation, QQP (Chen et al., 201 8) measured in F-1 score. Also, three natural language inference tasks are conducted: MNLI (Williams et al., 2018) mea- sured in classification accuracy with the average of the matched and mismatched subsets, QNLI (Ra- jpurkar et al., 2016) measured in accuracy.
CoNLL
16
500
2500
CoLA
32
500
1500
MNLI
32
5400
27000
MRPC
8
300
1500
QNLI
32
2000
12000
QQP
32
5400
27000
SST-2
16
500
1500
STSB
16
500
2500
resources by only requiring one base model that can be applied all tasks. Furthermore, unlike the other methods that require two rounds of model fine-tuning (one before compression and one after), RankDyna accomplishes compression with only one round of fine-tuning.
SAMSUM dataset contains staged chat conver- sations and corresponding summaries made by lin- guists (Gliwa et al., 2019), including 14,732 train- ing examples and 819 test examples. The per- formance is evaluated through Rounded ROUGE values (R1/R2/RL). We evaluate language model- ing task through metric perplexity on two popular model datasets: PTB (Marcus al., 1993) and WikiText-2 (Merity et al., 2016).
In comparison to SVD, FWSVD requires addi- tional time to compute Fisher information for each specific task. TFWSVD requires performs iterative calculations of decomposition.
time
A.4 Fisher information
Fisher information measures the amount of in- formation that a given dataset D provides about a model parameter w. However, obtaining pre- cise values for Fisher information is generally in- tractable due to the computational complexity. In practice, the empirical Fisher information is esti- mated as follows:
A.2 Implementation details
The BERT model in this paper has 110 M param- eters, out of which 94.6M correspond to the pa- rameters in the linear layers. The BART model utilize comprises 133M parameters,
of which 94.6 44.6M correspond to the parameters the linear layers. The GPT-2 model we use has 117M param- eters, and 81.0M parameters are associated with its linear (Conv ID) layers.
(
27
log p (D(w))
Iw E
(13)
2 =
Only the linear layers within the transformer blocks are compressed in this study, and non- Transformer modules, such the token embedding, remain uncompressed. The settings not explicitly mentioned in this work utilize the default config- urations of the HuggingFace Transformer library (Wolf et al., 2020).
Σ
Ow (di; w)
1
D
Given a target task objective (e.g., cross-entropy for a classification task), the estimated information i. accumulates the squared gradients over the train- ing data d; E D. The parameters that cause large absolute gradient of the task objective will have large value in i,.. and are considered important the target task.
A.3 Training Time
RankDyna begins with generic pre-trained model, which can be reused across all downstream tasks. In contrast, the other factorization methods men- tioned in the paper (SVD, FWSVD, TFWSVD) require a task-specific fine-tuned model for factor- ization. Consequently, as the model size increases and factorization becomes more computationally expensive, RankDyna saves more computational
Training Details
We use fixed 3 0.85 for importance score mo- mentum calculation in Equation (11), and learning rate 1E-04 throughout all experiments. We list addi- tional syper-parrmettr settings used by RankDyna in Table 5.

---

Figure 5: In Section 4.6, presented the rank distribution of RankDyna the MNLI dataset. Here we provide demonstration of the rank allocation when assigning uniform rank to cach weight matrix. The rank distribution on MNLI with fixed assignment. x-axis represents the layer index, while the y-axis represents different types of weight matrices.
13.9M linear parameters (equal rank ratio 0.1)
(b) 6.4M linear parameters (equal ratio 0.05)
W
52
W
Figure 6: The resulting rank of each weight matrix when compressing Bert-base on QNLI with RankDyna. The X- X-axis represents the layer index, while the y-axis represents different types weight matrices.
(a) 13.9M linear parameters (equal to rank ratio 0.1)
(b) 6.4M linear parameters (equal to rank ratio 0.05)
We
Ws
Wx
W W,
211
210
W
165
204
Wo
-5
291
W.
We
332
30
W/
15
11
9
11
Figure 7: The resulting rank of each weight matrix when compressing Bert-base QQP with RankDyna. The x-axis represents the layer index, while the y-axis represents different types of weight matrices.
(a) 13.9M linear parameters (equal to rank ratio 0.1)
(b) 6.4M linear parameters (equal to rank ratio 0.05)
>A2
253
:03
200
75
W5
203
I02
e5
A3
1L
T3
21
W
30
15c
305
232
273
82
236
76
152
63
52
11
20
T4
22
W,
374
176
226
209
362
W
100
134
124
76
50
10
20
1G
120
164
202
200
374
200
144
172
131
W
25
W4

---

Figure 8: The resulting rank of each weight matrix when compressing Bert-base STSB with RankDyna. The x-axis represents the layer index, while the y-axis represents different types of weight matrices.
(a) 13.9M lincar parameters (oqual to rank ratio 0.1)
(b) 6.4M lincar parameters (equal to rank ratio 0.05)
247
212
200
:05
92
IX
153
42
11
20
1G
W.
323
223
2F2
:00
66
27
26
26
26
W
124
20.4
59
22
11
14
332
249
234
160
XX
27
W6
259
238
167
Wic
WA
240
205
151.
Ws.
-25
Ws


Reviews:
[{"review_id": "dUhaOh5hPF", "reviewer": "Reviewer_NNuk", "summary": "", "questions": "1. In the paper you mention \"RankDyna only requires an one-time factorization on the base model, which can be shared across all downstream tasks.\" Don't you need to apply RankDyna for every downstream task since you would be working on a separate dataset each time?\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please refer to response to Reviewer **WKgh** for additional results with GPT-2 and BART. We answer all other questions as follows.\n\n> Concern 1. Missing comparison to tensor decomposition techniques [a,b,c], which perform well at high compression rations.\n\nThanks for sharing these papers. According to the background (section 2) of Reference A:  matrix factorization and tensor-train decomposition are two directions.   Reference A and C seem to align with the concept of tensor-train decomposition, whereas our approach RankDyna aligns more closely with the matrix factorization (SVD, WSVD, and TFWSVD). Reference B is a combination of knowledge distillation (KD) and  matrix-factorization.  Previous work WSVD and TFWSVD  also show that knowledge distillation and matrix-factorization are two orthogonal directions that can be combined.    Our evaluation focus is to examine the effectiveness of SOTA matrix-factorization  methods (SVD, WSVD, and TFWSVD), as they are the most closely related approaches to ours. We are happy to add Reference ABC  to the related work, and thanks again to refer them to us. \n\n> Question 1. In the paper you mention \"RankDyna only requires an one-time factorization on the base model, which can be shared across all downstream tasks.\" Don't you need to apply RankDyna for every downstream task since you would be working on a separate dataset each time?\n\nIt is indeed true that we will apply our RankDyna to each downstream task. However, it differs from SVD/WSVD/TFWSVD in that RankDyna achieves compression through a single round of fine-tuning instead of the two rounds typically required by the other methods.\n\nSpecifically, the training process for SVD/WSVD/TFWSVD comprises three main phases:\n\n1. Fine-tuning the general base model to adapt it to the specific downstream task.\n2. Applying matrix factorization methods such as SVD/WSVD/TFWSVD for compression.\n3. Fine-tuning the model obtained in step 2 once again.\n\nIn contrast, our RankDyna method  only needs one round of fine-tuning, which is essentially equivalent to step 1. This property can benefit storage efficiency. For instance, if we consider ten downstream tasks, the SVD/WSVD/TFWSVD approach will require the storage of ten distinct task-specific models prior to compression. On the contrary, our RankDyna approach only requires the storage of a single general base model. We achieve compact task-specific models by simultaneously conducting fine-tuning and compressing this base model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Simple algorithm which operates at the singular-group level and can achieve better performance on GLUE, specially at higher compression rates.\n\n2. Interesting analysis on the importance score discussing the suitability of using Fisher Information as an importance score.", "reasons_to_reject": "1. Missing comparison to tensor decomposition techniques [a,b,c], which perform well at high compression rations.\n\n2. No results on generation. There is a rich literature [d,e,f] of matrix decomposition methods for language models and machine translation.\n\n\n\n\na. https://aclanthology.org/2022.emnlp-main.475/\nb. https://aclanthology.org/2022.naacl-main.154.pdf\nc. https://proceedings.neurips.cc/paper_files/paper/2019/file/dc960c46c38bd16e953d97cdeefdbc68-Paper.pdf\nd. https://proceedings.neurips.cc/paper_files/paper/2018/hash/a2b8a85a29b2d64ad6f47275bf1360c6-Abstract.html\ne. https://aclanthology.org/2020.findings-emnlp.250.pdf\nf. https://www.isca-speech.org/archive/pdfs/interspeech_2018/shi18_interspeech.pdf", "questions_to_authors": "1. In the paper you mention \"RankDyna only requires an one-time factorization on the base model, which can be shared across all downstream tasks.\" Don't you need to apply RankDyna for every downstream task since you would be working on a separate dataset each time?\n", "paper_topic_and_main_contributions": "This work proposes a dynamic SVD technique for compressing transformer based language models for downstream tasks. They perform the compression at the granularity of a singular group instead of the level of a parameter. As with other recent methods, they compute an importance score to guide their compression. They only need to fine-tune once instead of twice like most methods. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Mentioned above in reasons to reject."}, "extra_text": {}}, {"review_id": "oLKMw3LUAF", "reviewer": "Reviewer_ncwh", "summary": "", "questions": "Question A. In Table 2, the results are the average of the last 100 steps and the model is fine-tuned for 6 epochs. While the other figures display the results of the early thousands of steps. Were the findings in Table 2 also consistent in the early days of fine-tuning?\n\nQuestion B. In line 313, what is the exact number of \"top-R\"? Like 50%?\n\nQuestion C. Line 205, why don't these linear layers contain the embedding matrix?\n\nQuestion D. Doesn't keep track of the importance of parameters during training take extra time? Why?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please refer to response to Reviewer **WKgh** for additional results with GPT-2 and BART. We answer all other questions as follows.\n\n>Question 1. In Table 2, the results are the average of the last 100 steps and the model is fine-tuned for 6 epochs. While the other figures display the results of the early thousands of steps. Were the findings in Table 2 also consistent in the early days of fine-tuning?\n\nFWSVD and TFWVSD hold an assumption that the first-order term of importance score is zero in trained models. Both Table 2 and Figure 1 demonstrate that this assumption is untrue. Note that, the scenarios in  Table 2 and Figure 1 are actually very close. Figure 1 is a converged task-specific model plus 1200 steps, while Table 2 shows 200 steps before obtaining the fine-tuned task-specific model. To maintain consistency and minimize confusion, we will adjust Table 2 to align with the scenario presented in Figure 1 (a task-specific model with an additional 1,200 steps). \n\nBesides clarifying the confusion above, we guess you might be interested in observing the scores during the early stages of fine-tuning.  We show the scores of first 100 steps during the initial fine-tuning. The overall pattern is similar to that of Table 2, with slightly higher values. This is due to the fact that the numbers in Table 2 are closer to the converged states.\n\n **Scores**         | **CoLA** | **MRPC** | **QNLI** | **SST-2** | **STSB** | \n--------------------|----------|----------|----------|-----------|----------|\n First-order term   | 2.72e-05 | 2.65e-05 | 3.01e-05 | 2.68e-05  | 2.77e-05 |     \n Second-order term  | 8.66e-09 | 9.02e-09 | 8.34e-09 | 6.05e-09  | 8.87e-09 |    \n Fisher information | 1.08e-08 | 1.05e-08 | 1.07e-08 | 1.03e-08  | 1.04e-08 | \n\n>Question 2. In line 313, what is the exact number of \"top-R\"? Like 50%?\n\ntop-\\\\( R^{(t)}\\\\) will change along with the training time \\\\( t\\\\) . The final top-\\\\( R^{(t)}\\\\) is related to the desired rank number. Taking a square matrix (\\\\( N \\times N\\\\)) for example, if the targeted compression rate is 50% of its original size, then the rank number will be  \\\\(  \\frac{1}{4} N\\\\). More details  can be found in Section 2.1.  Similarly, if we wanted to compress the linear layers in the model to around 50% of their original parameters, then the final number of top-\\\\( R^{(t)}\\\\) will be around 25%. \n\n>Question 3. Line 205, why don't these linear layers contain the embedding matrix?\n\nWe follow the configuration of prior studies such as WSVD/FWSVD, which retains the embedding layer uncompressed. Earlier research [Q-BERT] has indicated that the embedding layer is more sensitive than the regular linear layer. But technically, our method can be applied to the embedding layer as well. \n\n[Q-BERT] Shen, Sheng, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. \"Q-bert: Hessian based ultra low precision quantization of bert.\" In *Proceedings of the AAAI Conference on Artificial Intelligence*, vol. 34, no. 05, pp. 8815-8821. 2020.\n\n> Question 4. Doesn't keep track of the importance of parameters during training take extra time? Why?\n\nIndeed, such an operation does introduce some additional time. For detailed information about the training time, please refer to the response provided to Question 1 of reviewer **YbZE**.\n\n> Additional comment: Nowhere is there a definition of what FWSVD and TFWSVD are and their reference.\n\nWe mentioned the two previous work several times in the introduction (line 56, line 61-64) and related work (line 169-line172), but it is true that we didn’t denote  FWSVD and TFWSVD clearly. Thank you for bringing this to our attention, and we will make sure to include references to FWSVD and TFWSVD.\n\n We will add the above discussion to the final version. Thanks again for all the insightful questions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Reasonable idea, decoupled from the assumptions required by previous work.\n2. The main results and analytical experiments corroborate the method's effectiveness and the proposed claims.\n3. SOTA results.", "reasons_to_reject": "1. There is only a 12-layer BERT model as the base model and no experiments with generative models.", "questions_to_authors": "Question A. In Table 2, the results are the average of the last 100 steps and the model is fine-tuned for 6 epochs. While the other figures display the results of the early thousands of steps. Were the findings in Table 2 also consistent in the early days of fine-tuning?\n\nQuestion B. In line 313, what is the exact number of \"top-R\"? Like 50%?\n\nQuestion C. Line 205, why don't these linear layers contain the embedding matrix?\n\nQuestion D. Doesn't keep track of the importance of parameters during training take extra time? Why?", "paper_topic_and_main_contributions": "This article discusses the topic of parameter compression of Transformer-based language models via matrix decomposition. \n\nThey proposed RankDyna, an algorithm that dynamically tracks parameter importance and adjusts the rank budget to obtain a compressed model during fine-tuning.\n\nTheir proposed method addresses two dependent assumptions of previous work: \n1. The importance distribution of the parameters is invariant during the finetune process. \n2. The weights of the different layers are independent of each other.\n\nTheir approach achieves reasonable improvements in language model benchmark at different compression rates.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It seems the training loss line in Figure 2(b) goes from left to right and the person corr in Figure 2(a) goes from right to left, which looks weird to me.\n2. Nowhere is there a definition of what FWSVD and TFWSVD are and their reference."}}, {"review_id": "YaOz3Q4XVT", "reviewer": "Reviewer_WKgh", "summary": "", "questions": "Did the authors experiment with any of the other transformer based models except BERT?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We have additional results on seq2seq model BART (summarization task SAMSUM) and GPT-2 (language modeling task wiki2 and PTB). Generally, these results show the similar patterns we observed in GLUE dataset. \n **Summarization**       | **R1**   | **R2**   | **RL**   |  \n|--------------|------|------|------|\n| BART-base| 52.1 | 27.3 | 43.5 | \n| SVD          | 45.3 | 21.3 | 37.5 |  \n| WSVD         | 47.0 | 22.6 | 38.8 |  \n| TFWSVD       | 47.8 | 23.1 | 39.3  |\n| RankDyna | 48.4 | 23.4 | 39.7  |\n\nThe BART model we utilize comprises 133M parameters, out of which 94.6M correspond to the parameters in the linear layers. By applying a desired rank ratio of 0.2 across all methods, we reduce these 94.6M parameters to 37.1M.\n **LM**   | **wiki-2** | **PTB** | \n------------|------------|---------|\n GPT-2-base | 20.9       | 21.4    | \n SVD        | 124.7      | 79.6    |      \n WSVD       | 121.6      | 72.2    |       \n TFWSVD     | 79.3       | 62.9    |         \n RankDyna   | 77.7       | 39.4    |        \n\nSimilarly, the GPT-2 model we use has 117M parameters,  and  81.0M parameters are associated with its linear (Conv1D) layers. Through applying a rank ratio of 0.2 for all methods, we reduce this parameter count to 28.9M.\n\nIt's worth noting that applying a rank ratio of 0.2 yields a compact model with approximately 35~40% of its original model size. For more details, please refer to the reply to Question 2 for reviewer ****ncwh**** and Section 2.1 in our paper.\n\nThank you once more for suggesting the inclusion of additional experiments. This is indeed crucial for enhancing the quality of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n+ Proposes a matrix decomposition approach that captures dynamic changes the importance associated with singular groups.\n+ Extensive experiments demonstrating the importance of the method\n+ Reduced requirement of computational and storage resources.", "reasons_to_reject": "- Demonstrate the results only on the BERT model.", "questions_to_authors": "Did the authors experiment with any of the other transformer based models except BERT?", "paper_topic_and_main_contributions": "The authors propose RankDyna, a matrix decomposition method that enables dynamic rank resource allocation among matrices across different layers during the training process. Their proposed method achieves compression and adapts to different downstream task by fine-tuning the model just once.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wygi687D18", "reviewer": "Reviewer_YbZE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please refer to response to Reviewer **WKgh** for additional results with GPT-2 and BART.  We answer all other questions as follows. \n\n> Question 1. The proposed method will introduce extra computation and memory cost. However, there is no efficiency analysis, such as detailed training time (even though it is mentioned briefly in appendix) and memory usage comparison.\n\nTaking the PTB task (GPT-2) for example, the training time of different models are as follows. The time unit is minutes. As we mentioned in Question 1 to reviewer **NNuk**, SVD/WSVD/TFWSVD will require two-rounds of finetuning, resulting in a larger total number of training epochs compared to our RankDyna. For WSVD and TFWSVD, additional time is needed for calculating the Fisher information or numerical estimation required for matrix factorization. Our RankDyna indeed costs more time per epoch as we will do more calculations. Nonetheless, since we only require one round of fine-tuning, our total training time is the shortest among all methods.\n|          | Time per Epoch | Training Epoch | Basic SVD| Fisher Information | Numerical Estimation | Overall Training time |   |   |   |\n|----------|----------------------|----------------|---------------|-------------------------|---------------------------|-------------------------------|---|---|---|\n| SVD      | 1.9                  | 6              | 0.22          | -                       | -                         | 11.6                       |   |   |   |\n| WSVD     | 1.9                  | 6              | 0.22          | 6.3                     | -                         | 17.9                       |   |   |   |\n| TFWSVD   | 1.9                  | 6              | 0.22          | 6.3                     | 53.2                      | 71.1                       |   |   |   |\n| RankDyna | 2.9                  | 3              | 0.22          | -                       | -                         | 8.9                        |   |   |   |\n\n\nAs mentioned in the Limitations section, the primary downside of RankDyna is the need for additional running memory (almost equivalent in size to the model itself). This drawback doesn't cause notable issues for models like GPT-2, BART, or BERT. However, to be honest, when dealing with huge models like GPT-3 or LLAMA, it could potentially lead to memory errors. A possible strategy is to maintain the importance data structures on the CPU, which would help reduce GPU memory consumption.\n\n> Concern 1. The idea is not that novel since there are already works such as FWSVD and TFWSVD. The main contribution of this paper is to introduce dynamic estimation compared to preivous works.\n\nFWSVD/TFWSVD learn importance in a static manner, while our dynamic methodology is not only innovative but also indispensable. As mentioned in Section 4.4, both FWSVD and TFWSVD are built upon two assumptions: the stability assumption and the Independent assumption (line 406 - line 412). These unrealistic assumptions consequently limit their performance. In contrast, our RankDyna operates independently of these two assumptions, leading to better performance outcomes.\n\nThank you once again for the inspiring suggestion. We will add the above discussion to the final version if the paper is accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The motivation and the design of proposed method seem reasonable.\n\n(2)  Organized sections with clear statement in mathematical fomulation.\n\n(3)  The improvement is significant.", "reasons_to_reject": "(1) The proposed method will introduce extra computation and memory cost. However, there is no efficiency analysis, such as  detailed training time (even though it is mentioned briefly in appendix) and memory usage comparison.\n\n(2)  Using only bert_base (encoder-only architecture of transformer-base models) for experiment makes this paper a little bit overclaim.   Also, it is better to verify effiectiveness of  RankDyna on NLG tasks.\n\n(3)  The idea is not that novel since  there are already works such as FWSVD and TFWSVD. The main contribution of this paper is to introduce dynamic estimation compared to preivous works. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed RankDyna, a matrix decomposition method  that enables dynamic rank resource allocation among matrices across different layers during the training process. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a method for compressing Transformer model parameters by leveraging matrix decomposition. Basic idea is to dynamically tracks the importance of parameters and adjust the rank budget to derive a model in fine tuning.

Strengths
* The proposed method is simple yet well-designed by capturing the changes in the importance associated with singular groups.
* Experiments on GLUE show better performance under higher compression rates with detail analysis on the impact of importance score.

Weaknesses
* Experiments are limited to BERT and it has no experiment on other variations. It is not clear whether the proposed approach is also effective for the text generation tasks, e.g., MT.
* Novelty might be limited given the prior work on FWSVD/TFWSVD, i.e., a static variant.
Decision: Accept-Findings