Small Pre-trained Language Models Can be Fine-tuned as Large Models via Over-Parameterization
Year: 2023

Abstract:
By scaling the model size, large pre-trained language models (PLMs) have shown remarkable performance in various natural language processing tasks, mostly outperforming small PLMs by a large margin. However, due to the high computational cost, the huge number of parameters also restricts the applicability of large PLMs in real-world systems. In this paper, we focus on scaling up the parameters of PLMs only during fine-tuning, to benefit from the over-parameterization, while without increasing the inference latency. Given a relatively small PLM, we over-parameterize it by employing a matrix product operator, an efficient and almost lossless decomposition method to factorize its contained parameter matrices into a set of higher-dimensional tensors. Considering the efficiency, we further propose both static and dynamic strategies to select the most important parameter matrices for over-parameterization. Extensive experiments have demonstrated that our approach can significantly boost the finetuning performance of small PLMs and even help small PLMs outperform 3× parameterized larger ones. Our code is publicly available at https://github.com/zfgao66/OPF.

heading: 1 Introduction
text: Due to the remarkable performance, finetuning large-scale pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020; Zhao et al., 2023) has become the de facto method in the natural language process (NLP) field. With the help of large-scale pre-trained data and parameters, these large-scale PLMs are able to process an extensive range of world knowledge (Roberts et al., 2020; Jiang et al., 2020) and generalize well on a variety of tasks (Brown et al., 2020; Lester et al., 2021). Following this direction, more data and more parameters have been widely explored in improving the
∗Corresponding author.
model capacity of PLMs in recent years (Raffel et al., 2020; Chowdhery et al., 2022; Chen et al., 2022), leading to the expansion of PLM parameters from millions to billions.
Despite the effectiveness, the huge model size also limits the use of large PLMs in both research and industry communities. Specifically, the computational cost of pre-training and the efficiency of utilizing large PLMs are often unaffordable for researchers as well as resource-limited applications. Therefore, as a compromise, a number of studies (Gururangan et al., 2020; Chang et al., 2020; Zhang et al., 2020) focus on pre-training relatively smaller language models (e.g., BERT-baseuncased) on the domain-specific or task-specific corpus. However, small PLMs are not highly over-parameterized as larger models. As a result, their generalization capabilities are relatively weaker (Brown et al., 2020), leading to a suboptimal fine-tuning performance on downstream tasks.
In order to narrow the performance gap between small and large PLMs, this work seeks to overparameterize small PLMs as large models during fine-tuning, in pursuit of improving their generalization capability. In the essential form, PLMs are based on the Transformer architecture (Vaswani et al., 2017), and most of their parameters are stored as matrices. According to the matrix decomposition techniques (Tucker, 1966; Henry and Hofrichter, 1992; Oseledets, 2011) (e.g., Singular Value Decomposition), each matrix can be factorized as the multiplication of a set of matrices. In this way, the total number of parameters would be enlarged during fine-tuning. While, after convergence, the factorized matrices can be merged to re-organize the parameter matrix of the small PLMs. Such a paradigm effectively harnesses the advantages of over-parameterization, without incurring any additional inference latency for small PLMs during the fine-tuning process.
3819
Although it is promising to incorporate the matrix decomposition to over-parameterize small PLMs, there are two major concerns remaining to be investigated. First, the potential information loss caused by the matrix decomposition strategy should be greatly reduced, since small computation errors might be exponentially accumulated and propagated in the stacked multiple Transformer layer of PLMs. Second, small PLMs are comprised of multiple parameter matrices, although they may not always fulfill pivotal roles in fine-tuning various downstream tasks (Voita et al., 2019; Zhang et al., 2022). Thus, it is costly and unnecessary to over-parameterize all of them during fine-tuning. Therefore, there is a need to select the proper matrix decomposition method and leverage it to overparameterize these selected (more important) parameter matrices.
To address the above issues, we introduce the technique of matrix product operator (MPO) (Gao et al., 2020) as the matrix decomposition strategy. MPO has been widely used in the quantum manybody physics area, as it can efficiently factorize any matrix with arbitrary dimensions into a set of higher-dimensional tensors with arbitrary scales, and the factorized tensors can reconstruct the original matrix in almost lossless condition (Liu et al., 2021a; Gao et al., 2022b). Such merits make MPO a potentially good method for over-parameterizing small PLMs during fine-tuning. Based on MPO, we also devise static and dynamic strategies to adaptively select important parameter matrices for over-parameterization. Specially, the static strategy estimates the importance of each parameter matrix based on the variation of the loss values after removing it from a fine-tuned model (Voita et al., 2019) and then over-parameterizes the top-N important ones. The dynamic strategy computes the variation of gradients within several fine-tuning steps, which is the approximation of the above loss variation (Hou et al., 2020), which can dynamically guide the matrix over-parameterization process during fine-tuning.
To this end, in this paper, we propose a general Over-Parameterization Framework, namely OPF to improve the fine-tuning performance of small PLMs. Given the pre-trained parameter matrices of a small PLM, we first utilize the static or dynamic strategies to select the most important ones and then over-parameterize them by the MPO decomposition. Such a framework does not affect the
fine-tuning process, hence it is general to various small PLMs and NLP tasks. We conduct extensive experiments on the GLUE benchmark (Wang et al., 2019), a widely-used natural language understanding benchmark. Experimental results show that our OPF can boost the performance of small PLMs on GLUE significantly, e.g., improving BERT-base by +2.64 in average, improving T5-base by +2.41 in average. Besides, our approach also helps small PLMs outperform 3× parameterized ones, e.g., BERT-base+Ours (83.68) v.s. BERT-large (83.60) in average metrics on GLUE.

heading: 2 Related Work
text: Pre-trained Language Models. Pre-trained language models (PLM) (Devlin et al., 2019; Liu et al., 2019; Zhao et al., 2023) have yielded state-of-theart performance on a wide range of natural language processing tasks. Based on the Transformer architecture (Vaswani et al., 2017), BERT (Devlin et al., 2019) incorporated the “pre-training + finetuning” paradigm and has significantly improved the performance on a variety of NLP benchmarks, e.g., GLUE (Wang et al., 2019). Following this way, the T5 model (Raffel et al., 2020) and RoBERTa model (Liu et al., 2019) leveraged more data, more parameters and more pre-training steps, further improving the fine-tuning performance. Moreover, GPT-3 (Brown et al., 2020) showed that scaling up language models can greatly improve few-shot performance. In our approach, we improve the performance of PLMs by just scaling up the model during fine-tuning, which would not increase the inference latency.
Over-parameterization in Neural Network. Over-parameterization has shown the superiority on providing better model initialization (Arpit and Bengio, 2019), improving model convergence (Du et al., 2019; Allen-Zhu et al., 2019b; Gao et al., 2022a) and generalization (Allen-Zhu et al., 2019a). After the lottery theory hypothesis (Frankle and Carbin, 2019) was introduced, a surge of works pointed out that over-parameterization might be helpful to enhance the training efficiency (Malach et al., 2020; Pensia et al., 2020) and improve the model performance (Chen et al., 2020; Brix et al., 2020; Prasanna et al., 2020). Among them, Liu et al. (2021b) employed in-time overparameterization to narrow the performance gap between sparse and dense training. Our study aimed to use the over-parametrization strategy to better
inspire the potentiality of PLMs, enhancing their fine-tuning performance.
Tensor Decomposition in Neural Network. Tensor decomposition methods have been widely applied in a neural network for efficient training and inference, e.g., model compression (Gao et al., 2020; Sun et al., 2020; Gao et al., 2022c) and lightweight fine-tuning (Liu et al., 2021a). There are a surge of typical applications using the tensor decomposition methods on the parameter matrices of deep models to compress the linear layers (Novikov et al., 2015) and convolutional kernels (Garipov et al., 2016). Besides, existing works also apply the MPO method for the lightweight finetuning of ALBERT (Liu et al., 2021a) and the efficient expansion for the MoE framework (Gao et al., 2022b). Unlike existing methods, our approach focuses on the property that tensor decomposition can be used to map parameters from low-level spaces to high-dimensional spaces for over-parameterizing PLMs during fine-tuning, making PLMs benefit from more parameters.

heading: 3 Preliminary
text: Tensor. A tensor Ti1,i2,...,im can be viewed as an array with m indices, where {i1, i2, . . . , im} denotes the dimensions of the m indices, respectively. In this way, a vector (i.e., v) and a matrix (i.e., W) can be regarded as a 1-order tensor and 2-order tensor, respectively.
Tensor Product. Suppose {ψ1, . . . , ψp} and {ϕ1, . . . , ϕq} are the orthonormal basis of tensors T (1) and T (2), respectively. The tensor product can be derived by contraction of T (1) and T (2), denoted as ⊗. Formally, the tensor contraction of T (1) = ∑pi=1 aiψi1 and T (2) = ∑q j=1 bjϕi2 is defined as follow:
T (1) ⊗ T (2) = { p∑
i=1
aiψi1
} ⊗    q∑
j=1
bjϕi2   
=
p∑
i=1
q∑
j=1
aibjψi1 ⊗ ϕi2 . (1)
Tensor Decomposition Tensor decomposition can be seen as the inverse operation of the tensor product. A widely-used way is the singular value decomposition (SVD) algorithm. Given a Tensor T ∈ Ri1×···×im , the m times SVD operation can decompose this tensor into m local tensors
{T (k)}mk=1. Conversely, the decomposed tensors can also reconstruct the original tensor by sequentially performing the tensor product operator. The details of tensor decomposition are shown in Supplementary Materials A.1

heading: 4 Approach
text: In this part, we describe our proposed overparameterization framework for improving the finetuning performance of small PLMs. We first give an overview of our approach, and then introduce the details of matrix decomposition and over-parameterized matrices selection strategies.

heading: 4.1 Overview
text: Existing work mostly requires compressing a large PLM into a small one for benefiting from overparameterization (Sun et al., 2019; Shen et al., 2020). Different from them, our approach can scale up the number of parameters of arbitrary small PLMs during fine-tuning without using large ones. To achieve it, we leverage a matrix decomposition method, to factorize the important parameter matrices from the small PLM into a sequence of high-order tensors. These high-order tensors can be utilized to reconstruct the original matrix, while greatly increasing the number of trainable parameters during fine-tuning. After reconstruction, in the inference phase, the number of parameters in the fine-tuned PLM will remain the same as the original one, without increasing the inference latency and model size.
In our proposed OPF, we incorporate an MPO-
based matrix decomposition strategy to scale up the parameter matrices in PLMs and devise both static and dynamic selection strategies to determine important matrices for over-parameterization (Section 4.2). During fine-tuning, the static strategy first decides the important parameter matrices from the PLM based on the variation of training loss after removing each matrix and then relies on MPO to over-parameterize the selected top-N ones. The dynamic strategy computes the variation of gradients to estimate the importance of each matrix once a few steps and dynamically selects important matrices for over-parameterization (Section 4.3). The overview of our approach is presented in Figure 1. We also present a detailed algorithm for our OPF in Algorithm 1.

heading: 4.2 Over-parameterizing PLMs via Matrix Product Operator
text: To make small PLMs benefit from the overparameterization during fine-tuning, our approach employs Matrix Product Operator (MPO), a matrix decomposition technique to expand the number of model parameters. In this part, we first introduce the details of the MPO method and then describe how to adapt it for over-parameterizing PLMs.
Matrix Product Operator. MPO is an efficient algorithm that can factorize a parameter matrix W ∈ RI×J into a sequential product of multiple tensors (Gao et al., 2020), denoted as:
MPO(W) = T (1) ⊗ · · · ⊗ T (m), (2)
where {T (k)}mk=1 are the set of 4-order tensors with size [dk−1, ik, jk, dk], in which
∏m k=1 ik = I ,∏m
k=1 jk = J , and dk is calculated by:
dk = min( k∑
l=1
il × jl, m∑
l=k
il × jl). (3)
Given the parameter matrix W, the tensor sizes {dk}mk=1, {ik}mk=1 and {jk}mk=1, MPO can be regarded as a determined mapping process from W to multiple high-order tensors {T (k)}mk=1. Concretely, the MPO process consists of m-turn iterative matrix reshaping and SVD decomposition operations (Henry and Hofrichter, 1992), where the parameter matrix will gradually shrink and the decomposed tensor will be generated one by one. In the k-th turn, given the output parameter matrix Wk−1 from the last turn, we first reshape it into a new matrix W
′ k−1 whose first dimension
is dk−1 × ik × jk. Then, we perform the SVD decomposition on it as:
UλV⊤ = SVD (W ′ k−1) (4)
where U and V are complex unitary matrices, λ is a rectangular diagonal matrix with non-negative real numbers on the diagonal. Following truncated SVD methods (Henry and Hofrichter, 1992; Hansen et al., 1992), we extract the first dk columns of U corresponding to the dk largest singular values to compose the decomposed tensor T (k), and reshape it to the size [dk−1, ik, jk, dk]. Besides, we adopt λV⊤ as the output parameter matrix Wk for the decomposition in the following turns. After mturn iterations, we can obtain the decomposed multiple high-order tensors {T (k)}mk=1, and the contraction of these tensors in order would reconstruct the original parameter matrix W in almost lossless condition (Gao et al., 2020) (See Algorithm 2 in Appendix A.1).
Over-parameterizing PLMs. Based on the MPO method, we aim to expand the parameter scale of small PLMs during fine-tuning, for benefiting from over-parameterization. Generally, PLMs are based on the Transformer architecture (Vaswani et al., 2017), consisting of an embedding layer, stacked multi-head attention layers, and feed-forward networks. These modules contain necessary parameter matrices that have been pre-trained on large-scale corpus, e.g., the query projection matrices in the multi-head attention layers. Therefore, we can utilize the MPO method to decompose part of the parameter matrices into multiple tensors as Eq. (2). After the MPO decomposition, the parameter number of the matrix W would be increased according to the values of {dk}mk=1, {ik}mk=1 and {jk}mk=1. The detailed added parameter number Padd can be calculated as follows:
Padd =
m∑
k=1
ikjkdk−1dk − m∏
k=1
ikjk. (5)
According to Eq. (8), {dk}mk=1 are determined by {ik; jk}mk=1. Hence we can adjust the values of {ik; jk}mk=1 to control the number of added parameters by the MPO decomposition strategy. Therefore, during fine-tuning, we can adopt MPO on several selected parameter matrices from the PLM to generate their corresponding multiple tensors. In this way, we can scale up the total parameter
the number of the PLM. After fine-tuning the overparameterized PLM to convergence, we will perform tensor contraction on these decomposed tensors, to reconstruct the parameter matrices of the PLM. This new PLM owns the same parameter number and inference latency as the original one and has benefited from over-parameterization during fine-tuning.

heading: 4.3 Over-parameterized Matrices Selection
text: Despite the efficiency and flexibility of the MPO method, it is still costly to utilize it for overparameterizing all the parameter matrices in small PLMs. To concentrate the benefits of overparameterization on the most important parameters, we only select the most important parameter matrices from PLMs for decomposition. In particular, we propose a static selection strategy as well as a dynamic selection strategy, which pre-selects the important parameter matrices or dynamically chooses the ones during fine-tuning, respectively.
Static Selection Strategy. The proposed static selection strategy requires to pre-compute the importance scores of all parameter matrices before fine-tuning and then leverages MPO to overparameterize the top-N ones. After that, the architecture of the over-parameterized PLM would be static during fine-tuning. Inspired by network pruning methods (Molchanov et al., 2017; Voita et al., 2019), we utilize the change of the training loss LW after removing each parameter matrix W, to measure the importance scores since important parameters would play a key role to predict the correct label (Voita et al., 2019). Therefore, the importance score IW of a parameter matrix W can be computed as:
IW = |LW − LW=0|, (6)
where LW=0 denotes the value of loss after zeroing W and the symbol || represents the operation of taking the absolute value. To calculate the loss, we need to fine-tune a small PLM using the same pre-trained parameter as ours before. Generally, the parameter matrices from different modules of the PLM (e.g., multi-head attention layer and feed-forward network) may have different sizes and functions, making it inappropriate to directly compare them. Thus, we first categorize all parameter matrices by module, where each group contains one module for L layers. Then we pick the top-N ones from each group for over-parameterization.
Algorithm 1 Fine-tuning a PLM with our OPF. Input: Parameters matrices set of a PLM {W}. 1: Divide {W} into several groups by module. 2: if is Static Strategy then 3: Fine-tuning the PLM until converged. 4: Compute IW for {W} using Eq. (6). 5: Sort {W} in each group according to IW. 6: Perform MPO on the top-N matrices. 7: Train the other PLM until converged. 8: else 9: Define S = {} 10: while Len(S) < N do 11: Train the PLM for t steps. 12: Compute IW for {W} using Eq. (7). 13: Sort {W} in each group according to IW. 14: Add top-n matrices into S, and perform MPO. 15: end while 16: Continually train the PLM until converged. 17: end if
Dynamic Selection Strategy Our proposed dynamic selection strategy aims to dynamically calculate the importance scores and choose the immediate important parameter matrices for overparameterization during fine-tuning. Such a way can dynamically capture the importance of change w.r.t. the optimization of the whole PLM. Following Hou et al. (2020), we perform the first-order Taylor expansion on Eq. (6) to obtain the approximation of the importance score as:
IW = |LW−(LW− ∂L ∂W (W−0)+RW=0)| ≈ | ∂L ∂W
W|, (7)
where once the remaining part RW=0 is omitted, the important score can be estimated by the absolute values of the gradients of the parameter matrix. In practice, we accumulate the absolute values of the gradients for all the parameter matrices during fine-tuning. We dynamically calculate the importance score using Eq. (7) and over-parameterize the top-n parameter matrices from the categorized groups once t steps. The above process will be performed multiple times until N parameter matrices from each group have been selected.

heading: 5 Experiments
text: In this section, we first set up the experiments, then report the results and give a detailed analysis.

heading: 5.1 Experimental Setup
text: Datasets. To verify the effectiveness of our approach, we conduct experiments on the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019), which consists of 8 datasets (MNLI, QQP, QNLI, RTE, MRPC, CoLA,
SST-2, STS-B) to systematically evaluate the abilities of similarity matching, sentiment classification, linguistic acceptability estimation and natural language inference. Since the labels of their original test sets are not available, we randomly split their original validation sets in half, and use one half as the validation set and the other half as the test set. For the evaluation metrics, following existing works (Gao et al., 2022b), we use Matthews correlation for CoLA, Spearman correlation for SST-B, F1 for MRPC, and accuracy for other tasks. We also compute the average score across all tasks.
Baseline Methods. We implement our approach on the following PLMs, BERT (Devlin et al., 2019), T5 (Raffel et al., 2020) and BART (Lewis et al., 2020). BERT is a widely-used PLM based on the bidirectional Transformer architecture. We select the publicly released BERT-small, BERTmedium (Liu et al., 2021b), BERT-base and BERTlarge (Devlin et al., 2019) for comparison. T5 and BART adopt the sequence-to-sequence Transformer architecture, and we choose their base and large versions. Besides, we also compare our approach with SVD (Henry and Hofrichter, 1992), a classic matrix decomposition method that can also be used for over-parameterizing PLMs. Concretely, we leverage SVD to replace MPO in our framework and perform over-parameterization on all the parameter matrices of the PLM during fine-tuning.

heading: 5.2 Main Experimental Results
text: In this part, we report and analyze the experimental results on BERT, T5 and BART.
Evaluation on BERT. We present the results on BERT in Table 1. First, we can see that the BERT models with more parameters perform consistently better than smaller ones, i.e., BERT-large > BERTbase > BERT-medium > BERT-small. It demonstrates that more parameters are helpful for PLMs to achieve better performance, showing the effectiveness of over-parameterization. Second, after combining PLMs with the over-parameterization methods, their performances are most improved. Although these methods just increase the model parameters during fine-tuning, they can also benefit from over-parameterization to improve the generalization capacity. Between the two matrix decomposition methods, we observe that SVD mostly underperforms MPO. As SVD just performs the matrix decomposition once in the 2D space based on the singular value, it is hard to greatly increase
the number of the model parameters as our approach (e.g., 34M v.s. 81M in BERT-small). As a comparison, MPO can factorize the matrix into arbitrary scales by increasing the order, making it more proper for over-parameterization.
Finally, by comparing our approach with different matrix selection strategies, we can see that the dynamic strategy mostly outperforms the static one, under the setting of the same parameter scale. The reason may be that the dynamic strategy can estimate the importance of immediate parameter matrices w.r.t. the training steps. Such a way is able to adapt to the change of parameter importance during fine-tuning, and better guides the overparameterization. Surprisingly, by using our framework with the dynamic strategy, the BERT-base model can be fine-tuned to achieve comparable performance as the BERT-large model, where the number of its parameters is just increased into a similar scale during fine-tuning.
Evaluation on T5 and BART. We show the results on T5 and BART in Table 2. Similar to BERT, we can also see that the large models consistently outperform base models, and our proposed over-parameterization method narrows this performance gap. It indicates that our approach is general to different model architectures and pre-training tasks and can benefit from over-parameterization to improve the fine-tuning performance of different PLMs. Besides, the performance of T5 is improved more than BART under a similar parameterincreasing rate, and the over-parameterized T5base model also achieves comparable performance with T5-large. A possible reason is that T5 has been pre-trained using a much large corpus C4 (Raffel et al., 2020), and over-parameterization can better inspire its potentiality during fine-tuning.

heading: 5.3 Further Analysis
text: Next, we continue to investigate our proposed approach in a more detailed analysis.
Performance Comparison w.r.t. Parameter Increasing Rate. During fine-tuning, our approach can increase the number of model parameters for improving the over-parameterization of PLMs. As our approach is a general and flexible way to increase the model parameters into arbitrary scales, here we investigate how the performance changes w.r.t. a different number of increased model parameters. Based on BERT-base and BERTmedium, we expand their parameter scales after
over-parameterizing from 1× to 4×, reporting the performance on STS-B and CoLA tasks. As shown in Figure 2, we can see that the model performance is consistently improving w.r.t. the increasing of parameter scales. Comparing the improved performance between BERT-base and BERT-medium, BERT-medium has gained more boost. It indicates that a small PLM is much more hungry for more parameters. Besides, after reaching the 4× parameter scale, the improvement becomes relatively smaller. It shows that the 4× parameter scale seems to be the limit that can significantly improve the model performance via over-parameterization.
Hyper-parameters Tuning. For our OPF using the dynamic strategy, the numbers of total selected parameter matrices N and the selection number at one time n in each parameter matrix group are important hyper-parameters that require tuning. Larger N means that more parameter matrices are selected and over-parameterized and larger n denotes that more matrices are over-parameterized at one time. To investigate the effect of their values on the model performance, we conduct experiments on the CoLA task using BERT-base as the backbone. As shown in Figure 3, we can see that the
performance steadily improves as N increases and eventually reaches a plateau as a result. The reason may be that over-parameterizing too few matrices is not able to sufficiently over-parameterize the PLM. Besides, we can see that too large n would degrade the performance. A possible reason is that too large n will over-parameterize too many parameter matrices at one time, causing the dynamic strategy to degrade into the static one. Whereas, we can see that our approach consistently outperforms the baseline method. It shows that our approach is not very sensitive to the above hyper-parameters.
Sensitivity Analysis. As our approach is based on the matrix decomposition method to overparameterize the PLM, once a small error arises during performing decomposition, it would accumulate into an extremely large value that may ruin the PLM. To avoid it, our approach incorporates the MPO method, which can factorize the parameter matrix in almost lossless conditions. Such a way could stabilize the performance of our ap-
proach and make it less sensitive to perturbation on hyper-parameters. To validate it, we select a commonly-used hyper-parameter, the learning rate to evaluate the sensitivity of our approach on RTE, CoLA and STS-B tasks using BERT-base, and report the performance change w.r.t. tuning it in the set {5e-6, 1e-5, 3e-5, 5e-5, 1e-4} in Table 3. We can observe that the performance of our approach consistently stables around certain values, i.e., 72.0 for RTE, 60.0 for CoLA, and 88.5 for STS-B. It indicates that our approach is not sensitive to the learning rate during fine-tuning. Besides, setting the learning rate to a commonly-used value 3e5 is enough for our approach to achieving good performance, no longer requiring time-consuming parameter tuning.

heading: 6 Conclusion
text: In this paper, we proposed OPF, a novel overparameterization framework to scale up the number of parameters for PLMs just during fine-tuning, for benefiting from more parameters. In our OPF, we incorporated the matrix product operator method, which decomposes the parameter matrices in PLMs into high-order tensors for increasing the parameter number, and also devised the static and dynamic strategies to select the most important parameter matrices for over-parameterization. Extensive experiments have demonstrated that our OPF approach can boost the performance of small PLMs
significantly, and even help small PLMs outperform big ones.
In future work, we will investigate more efficient and effective tensor decomposition methods for PLM over-parameterization. In addition, we will also apply OPF to other important backbone models in computer vision and multimodal domains.
Limitations
Further research is needed to understand the robustness of our over-parameterization framework properly. The results given in this study are constrained by the natural language processing tasks and datasets used for evaluation. Even though we employ standard classifications from the literature, the choice of downstream tasks and datasets is still subjective. Furthermore, due to computing limitations, we could not investigate the scaling behavior of the Large PLMs. Additional study is needed in this area. In addition, as our approach is based on PLMs that may learn biased information from pretrained corpus, a potential risk is that our approach may also be affected by it and generates improper texts.

heading: Acknowledgments
text: This work was partially supported by National Natural Science Foundation of China under Grants No. 62206299 and 62222215, Beijing Outstanding Young Scientist Program under Grant No. BJJWZYJH012019100020098 and CCF-Zhipu AI Large Model Fund. Xin Zhao is the corresponding author.

heading: A More Details about Tensors
text: A.1 Tensor and Matrix Product Operator
As introduced in (Cichocki et al., 2009), a tensor can be defined as follows.
Tensor. Let D1, D2..., DM ∈ M denote the index upper bounds. A tensor T ∈ RD1,...,DM of order M is an M -way array where elements T [d1, d2, ..., dM ] are indexed by dm ∈ {1, 2, ..., DM} for 1 ≤ m ≤M .
Matrix Product Operator. The k-th order (Kolda and Bader, 2009) and k ∈ {1, . . . , D}. The bond dimension dk is defined by:
dk = min
( k∏
m=1
im × jm, n∏
m=k+1
im × jm ) . (8)
From Eq. (8), we can see that is going to be large in the middle and small on both sides. Algorithm 2 presents a thorough algorithm for MPO decomposition.
Algorithm 2 MPO decomposition procedure. Input: matrix W ∈ RI×J , the number of local tensor m Output : local tensor set {T (k)}mk=1 1: for k = 1, . . . ,m− 1 do 2: W[dk−1 × ik × jk,−1]← Reshape(W[I, J]) 3: UλV⊤ ← SVD (W) 4: T (k)[dk−1, ik, jk, dk]← Reshape(U) 5: Calculate W = λV⊤
6: end for 7: Let T (m) ←W 8: Normalization 9: return local tensor set {T (k)}mk=1
The MPO representation of W is obtained by factorizing it into a sequential product of local tensors. The algorithm has been depicted in Section 4.2 of the main text. With the MPO decomposition technique, we can get local tensor as follows:
Wi1···in,j1···jn = T (1)[i1, j1] · · · T (m)[im, jm] (9)
where T (k)[jk, ik] is a Dk−1 × Dk matrix with Dk the virtual basis dimension on the bond linking T (k) and T (k+1) with D0 = Dm = 1. With Eq. (9) we can decompose an original matrix W to a sequential product of the derived local tensors.
A.2 Theorem
Theorem 1. Suppose that the tensor W(k) of matrix W that is satisfy
W = W(k) +E(k), D(W(k)) = dk,
where ||E(k)||2F = ϵ2k, k = 1, ..., d− 1. (10)
Then MPO(W) with the k-th bond dimension dk upper bound of truncation error satisfy:
||W −MPO (W)||F ≤
√√√√ d−1∑
k=1
ϵ2k (11)
Proof. The proof is by induction. For n = 2 the statement follows from the properties of the SVD. Consider an arbitrary n > 2. Then the first unfolding W(1) is decomposed as
W(1) = U1λ1V1+E (1) = U1B (1)+E(1), (12)
where U1 is of size r1 × i1 × j1 and ||E(1)||2F = ϵ21. The matrix B1 is naturally associated with a (n − 1)-dimensional tensor B(1) with elements B(1)(α, i2, j2, ..., in, jn), which will be decomposed further. This means that B1 will be approximated by some other matrix B̂1. From the properties of the SVD it follows that UT1 E
(1) = 0, and thus
||W − B(1)||2F = ||W1 −U1B̂1||2F = ||W1 −U1(B̂1 +B1 −B1)||2F = ||W1 −U1B1||2F + ||U1(B̂1 −B1)||2F (13)
and since U1 has orthonormal columns,
||W − B(1)||2F ≤ ϵ21 + ||B1 − B̂1||2F . (14)
and thus it is not difficult to see from the orthonormality of columns of U1 that the distance of the k-th unfolding (k = 2, ..., dk − 1) of the (d − 1)- dimensional tensor B(1) to the dk-th rank matrix cannot be larger than ϵk. Proceeding by induction, we have
||B1 − B̂1||2F ≤ d−1∑
k=2
ϵ2k, (15)
combine with Eq. (14), this complets the proof.

heading: B Additional Experimental Details
text: Experimental Details in Pre-trained Language Modeling In this paper, the MPO decomposition is proposed for enlarging model parameters. In order to show the process of incorporating several MPO structures into BERT, T5, and BART respectively. Moreover, we denote an MPO, defined by Eq. (2), as:
T j1,j2,...,jmi1,i2,...,im (D) (16)
BERT-small and BERT-medium models were released by Google Research in (Turc et al., 2019), which have 4 and 8 Transformer layers respectively, and each of the layers has 512 hidden nodes. We perform the pre-compute stage, which would compute the importance scores of all parameter matrices before fine-tuning and then leverages MPO to over-parameterize the top-N ones. The important score can be computed by Eq. (6) and Eq. (7). In our main experiments, the detail hyper-parameter setting was shown in Table 4.
We have conducted sufficient trials for both the T5 and the BART models using the following particular experimental parameter configurations in Table 5.
Hardware We trained our model on one machine with 8 NVIDIA V100 GPUs. For our base mod-
els, we adopt all these models released by Huggingface 1. The hyperparameters are described in Table 6.
Optimizer We used the Adam optimizer and vary the learning rate over the course of training. The vary formula (Vaswani et al., 2017) follows in our work. We also used the warmup_steps = 1000.
Details of Fine-tuning Datasets GLUE benchmark covers multiple datasets (MNLI, QNLI, SST2, RTE, QQP, CoLA, STS-B, MRPC) 2.
Details of Evaluation Metrics Following Gao et al. (2022b), the metrics that we use for the GLUE benchmark are Matthew’s correlation for CoLA (Mcc.), Spearman for STS-B (Spear.), F1 for MRPC, and accuracy (Acc.) for the remaining tasks. We compute and report the average scores for all of the aforementioned metrics based on all of the test samples taken into consideration. Since the original test sets are not accessible, we divide the original validation set in half and use one half for validation and the other for the test for datasets with fewer than 10,000 samples (RTE, MRPC, STS-B, CoLA) (Zhang et al., 2021).
1https://huggingface.co/ 2In line with Raffel et al. (2020), we do not test WNLI due
to its adversarial character with respect to the training set.

heading: C Additional Discussion
text: Different Tensor Decompostion In the field of mathematics, the MPO-based approximation may be seen as an alternative form of the low-rank approximation approach. Now we will evaluate it in light of many other low-rank approximation techniques, such as SVD (Henry and Hofrichter, 1992), CPD (Hitchcock, 1927), and Tucker decomposition (Tucker, 1966).
We present the categorization of these methods in Table 7. Because the work of low-rank decomposition only needs to be done once, and it does not take a long time, thus we mainly focus on the forward propagation time in practical use. In point of fact, each of the techniques may either be based on a tensor-based decomposition (that is, a list of tensors for factorization) or a matrix-based decom-
position, and we quantify the amount of time each approach requires using standard parameters. Indeed, MPO and Tucker are examples of two different classes of low-rank approximation algorithms. In most cases, the capacity of the algorithm will rise in proportion to the value of m (more tensors). When m is more than three, the temporal complexity of MPO is lower than that of Tucker decomposition. It is clear that SVD may be thought of as a special example of MPO when the dimension of the tensor is equal to two, and that CPD is a particular case of Tucker when the super-diagonal matrix is the core tensor. Both of these relationships can be observed here.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section-Limitations
3 A2. Did you discuss any potential risks of your work? Section-Limitations
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section-Abstract; Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 5.
3 B1. Did you cite the creators of artifacts you used? Section 5.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Section-Experiments( Section 5)
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section-Experiments( Section 5)
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section-Experiments( Section 5.1); Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.2, we report the mean of 5 runs using different random seeds.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: In future work, we will investigate more efficient and effective tensor decomposition methods for PLM over-parameterization. In addition, we will also apply OPF to other important backbone models in computer vision and multimodal domains. Limitations
Further research is needed to understand the robustness of our over-parameterization framework properly. The results given in this study are constrained by the natural language processing tasks and datasets used for evaluation. Even though we employ standard classifications from the literature, the choice of downstream tasks and datasets is still subjective. Furthermore, due to computing limitations, we could not investigate the scaling behavior of the Large PLMs. Additional study is needed in this area. In addition, as our approach is based on PLMs that may learn biased information from pretrained corpus, a potential risk is that our approach may also be affected by it and generates improper texts