Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing
Authors: Peiyu Liu
Ze-Feng Gao
Yushuo Chen
Xin Zhao
Ji-Rong Wen
Venue: EMNLP 2023 Findings (2023)
Keywords: parameter-efficient
pre-trained language models
scalability
TL;DR: In this paper, we propose a highly parameter-efficient approach to scaling pre-trained language models to a deeper model depth.

Abstract: In this paper, we propose a highly parameter-efficient approach to scaling pre-trained language models (PLMs) to a deeper model depth. 
Unlike prior work that shares all parameters or uses extra blocks, we design a more capable parameter-sharing architecture based on matrix product operator (MPO), an efficient tensor decomposition method to factorize the parameter matrix into a set of local tensors. Based on such a decomposition,  we share the important local tensor across all layers for reducing the model size and meanwhile keep layer-specific tensors (also using Adapters) for enhancing the adaptation flexibility. To improve the model training, we further propose a stable initialization algorithm tailored for the MPO-based architecture. Extensive experiments have demonstrated the effectiveness of our proposed model in enhancing scalability and achieving higher performance (i.e., with fewer parameters than BERT-base, we successfully scale the model depth by a factor of 4x and even achieve 0.1 points higher than BERT-large for GLUE score). The code to reproduce the results of this paper can be found at https://github.com/RUCAIBox/MPOBERT-code.

Enhancing Scalability of Pre-trained Language Models via Efficient Parameter Sharing
Peiyu Liu 12; Ze-Feng Gaol Yushuo Chen Wayne Xin Z: and Ji-Rong Wen
Gaollnng School of Artificial Intelligence, Renmin University of China BBejjing Key Laboratory of Big Data Management and Analysis Methods
Department of Physics, Renmin University of China (liupeiyustu, zfgao, chenyushuo, j juwen eéruc. enu.ch, batmanf yeggmal com
Abstract
Small
Medium
Large
50
this paper, we propose a highly parameter- efficient approach to scaling pre-trained lan- guage models (PLMs) to a deeper model depth. Unlike prior work that shares all parameters or uses extra blocks, we design a more capable parameter-sharing architecture based on ma- trix product operator (MPO), efficient tensor decomposition method to factorize the param- eter matrix into set of local tensors. Based such decomposition, we share the impor- tant local tensor across all layers for reduc- ing the model size and meanwhile keep layer- specific tensors (also using Adapters) for en- hancing the adaptation flcxibility. To improve the model training. further propose stable initialization algorithm tailored for the MPO- based architecture, Extensive experiments have demonstrated the effectiveness of proposed model in enhancing scalability and achieving higher performance (i.e., with fewer parame- ters than BERTRASE.
Ours
Deep
40-
ROBERTa
Moderate Shallow
Depth
BART
T5
BERT
20
XLNet...
ALBERT
100
200
250
400
#Params
Figure 1: A comparison of our model and representative PLMs in the dimensions of model size and model depth.
et al., 2020). While, very deep architecture typ- ically corresponds to a significantly large model size, leading to high costs in both computation and storage (Gong et al., 2019). And, it is difficult to deploy deep networks in resource-limite settings, though it usually has stronger model capacity. Therefore, there is an urgent need for developing parameter-efficient way for scaling the model depth.
successfully scale the model depth by factor of 4x and even achieve 0.1 points higher than BERT ArR for GLUE score). The code to reproduce the results of this paper can found https: //gi thub. com
To reduce the parameters in deep networks, weight sharing has proven to be very useful to de- sign lightweight architectures (Zhang et al., 2022; Lan al., 2019). As representative one by across-layer parameter sharing. ALBERT (Lan et al., 2019) keeps only ten percent of the whole parameters of BERT while maintaining compara- ble performance. Although the idea of parameter sharing is simple yet (to some extent) effective, it has been found that identical weights across dif- ferent layers are the main cause of performance degradation (Zhang et al., 2022). To address this issue, extra blocks are designed elevate parame- diversity in each layer (Nouriborji et al., 2022). While they still use the rigid architecture of shared layer weights, having a limited model capacity. Be- sides, it is difficult to optimize very deep models,
1 Introduction
Recently, pre-trained language models (PLMs) have achieved huge success a variety of NLP tasks by exploring ever larger model architec- ture (Raffel et al., 2020; Radford et al., 2019). has been shown that there potentially exists a scal- ing law between the model size and model capacity for PLMs (Kaplan et al., 2020), attracting many ef- forts to enhance the performance by scaling model size (Chowdhery al., 2022; Wang al., 2022b). As straightforward approach, we can directly increase the layer number networks for improv- ing the model capacity (Wang et al., 2022b; Huang
Authors contributed equally. Conresponding author.

---

especially when shared components are involved. Although recent studies (Wang al., 2022b; Huang et al., 2020) propose improved initialization meth- ods, they do not consider the case with parameter sharing, thus likely leading to a suboptimal perfor- mance on parameter-shaiee sharing architecture.
2020b; Sun et al., 2020a). Based on MPO decom- position, recent studies designed lightweight fine- tuning and compression methods for PLMs (Liu et al., 2021). developed parameter-efficient MoE ar- chitecture (Gao et al., 2022); pverpparamttiizzttoon PLMs (Gao et al., 2023) and empirical study the emergency ability in quantized large language mod- els (Liu et al., 2023). Unlike these works, our work aims to develop very deep PLM with lightweight architecture and stable training.
To address these challenges, in this paper, we propose highly parameter-efficient approach to scaling PLMs to a deeper model architecture. As the core contribution, we propose a matrix prod- uct operator (MPO) based parameter-sharing ar- chitecture for deep Transformer networks. Via MPO decomposition, a parameter matrix can be decomposed into central tensors (containing the major information) and auxiliary tensors (contain- ing the supplementary information). Our approach shares the central tensors of the parameter matrices across all layers for reducing the model size, and meanwhile keeps layer-specifie auxiliary tensors for enhancing the adaptation flexibility. In order to train such a deep architecture, we propose an MPO- based initialization method by utilizing the MPO decomposition results of ALBERT. Further, for the auxiliary tensors of higher layers (more than 24 lay- ers in ALBERT), propose set the parameters with scaling coefficients derived from theoretical analysis. We theoretically show it can address the training instability regardless of the model depth.
Prrameter-Efficiens PLMs. Existing efforts to reduce the parameters of PLMs can be broadly cat- egorized into three major lines: knowledge distil- lation, model pruning, and parameter sharing. For knowledge distillation- based methods (Sanh al., 2019; Sun et al., 2020b,b; Liu et al., 2020; Wang et al., 2022a), PLMs were distilled into student net- works with much fewer parameters. For pruning- based methods, they tried to remove less impor- tant components (Michel et al., 2019; Wang et al., 2020) or very small weights (Chen et al., 2020). Moreover, the parameter-sharing method was fur- ther proposed by sharing all parameters (Lan al., 2019) or incorporating specific auxiliary compo- nents (Reid et al., 2021; Nouriborji et al., 2022). Different from these works, we design an MPO- based architecture that can reduce the model size and enable adaptation flexibility, by decomposing the original matrix.
Our work provides a novel parameter-sharing way for scaling model depth, which can be gen- erally applied to various Transformer-basel mod- els (Zhao et al., 2023). We conduct extensive exper- iments to evaluate the performance of the proposed model the GLUE benchmark in comparison PLMs with varied model sizes (tiny, small and large). Experiments results have demonstrated the effectiveness of the proposed model reducing the model size and achieving competitive performance. With fewer parameters than BERTBASE we scale the model depth by factor of 4x and achieve 0. points higher than BERT for GLUE score.
Optimization for Deep Models. Although it is simple to increase the number of layers for scal- ing model size, it is difficult to optimize very deep networks due to the training instability issue. Sev- eral studies have proposed different strategies to overcome this difficulty for training deep Trans- former networks, including Fixup (Zhang et al., 2019) 9) by properly rescaling standard initialization, T-Fixup (Huang et al., 2020) by proposing a weight initialization scheme, and DeepNorm (Wang et al., 2022b) by introducing new normalization function. As a comparison, we study how to optimize the deep MPO-based architecture with the parameter sharing strategy, and explore the use of well-trained PLMs for initialization, which has different focus from existing work.
2 Related Work
Matrix Product Operators. Matrix product op- erators (a.ka. tensor-train operators (Oseledets, 2011)) were proposed for more effeetive rep- resentation of the linear structure of neural net- works (Gao et al., 2020a), which was used to compress deep neural networks (Novikov et al., 2015), convolutional neural networks (Garipov et al., 2016; Yu et al., 2017), and LSTM (Gao et al.,
Method
In this section, we describe the proposed MPOBERT approach for building deep PLMs via a highly parfmeter-efficient architecture. Our ap-

---

proach follows the classic weight sharing paradigm while introducing principled mechanism for shar- ing informative parameters across layers and also enabling layer-specific weight adaptation.
Transformer Layer
MPO-buse bused Layer
SEN
W
C
Ar
200
NItA
MPO (T
W Adaper
Layerspeeifie parameters
Sharing parameter
3.1 Overview of Our Approach
Although weight sharing has been widely explored for building compact PLMs (Lan et al., 2019), ex- isting studies either share all the parameters across layers (Lan et al., 2019) incorporate additional blocks to facilitate the sharing (Zhang al., 2022; Nouriborji al., 2022). They either have limited model capacity with rigid architecture or require additional efforts for maintenance.
L/C
groups
L/G
МPОBERT
MPOBERT
Overview architecture MPOBERT and MPOBERT We blocks with dashed borderlines to represent shared central tensors. Central tensors are shared across all Layers in MPOBERT and within groups in MPOBERT
Figure 2:
Considering the above issues, we motivate our approach in two aspects. Firstly, only informa- tive parameters should be shared across layers, in- stead of all the parameters. Second, it should not affect the capacity to capture layer-specific varia- tions. To achieve this, we utilize the MPO decom- position (Liu al., 2021) develop a parameter- efficient architecture by sharing informative com- ponents across layers and keeping layer-specific supplementary components (Section 3.2). As an- other potential issue, it is difficult to optimize deep PLMs due to unstable training (Wang et al., 2022b), especially when weight sharing (Lan et al., 2019) is involved. We further propose simple yet ef- fective method to stabilize the training of PLMs (Section 3.3). Next, we introduce the technical details of our approach.
decrease with the increasing distance to the middle tensor. Following Gao et al. (2022), further simplify the decomposition results of matrix central tensor C. (the middle tensor) and auxiliary tensors {At)= (the rest tensor).
As a major merit, such a decomposition can ef- fectively reorganize and aggregate the information of the matrix (Liu et al., 2021): central tensor C can encode the essential information of the original matrix, while auxiliary tensors (Ai) serve its complement to exactly reconstruct the matrix.
3.2.2 MPO-based Scaling to Deep Models
Based on MPO decomposition, the essence of our scaling method is to share the central tensor across layers (capturing the essential information) and keep layer-specific auxiliary tensors (model- ing layer-specific variations). Fig. 2 shows the overview architecture the proposed MPOBERT.
3.2 MPO-based Transformer Layer
this section, we first introduce the MPO decom- position and introduce how to utilize it for building parameter-efficienl deep PLMs.
3.2.1 MPO Decomposition
Given weight matrix W I MPO decom- position can decompose matrix into product of n tensors by reshaping the two dimension sizes and J:
Cross-layer Parameter Sharing. To introduce our architecture, we consider a simplified struc- ture of L layers, each consisting of single ma- trix. With the five-order MPO decomposition (i.e..
5), we can obtain the decomposi- tion results for a weight matrix (W()), denoted as {C()) (49 A AA (AA 10)1- where C()) and (4 are the central tensor and auxiliary ten- of the 1-th layer. Our approach is to set shared central tensor across layers, which means that (()) (Vl 1... L). As shown in Ap- pendix A. , the central tensor contains the major proportion of parameters (more than 90%), and thus our method can largely reduce the parame-
Manduaiiion = (11, ... ((n) Ben. Jal,
(1)
where we have I - IT Mc. J - Ih and (4) (i,, j&] is a 4-dimensional tensor with size d is j& de in which de is bond dimen- sion linking (A) and (1++1) with dg d, 1. For simplicity,
omit the bond dimensions in Eq. (4). When n is odd, the middle tensor con- tains the most parameters (with the largest bond dimensions), while the parameter sizes the rest

---

ters when scaling PLM to very deep architecture. We implement our proposed efficient parameter- sharing strategy upon the BERT (Devlin et al., 2018), named MPOBERT, which shares the central tensor across layers. Note that this strategy can be easily applied to multiple matrices in Trans- former layer, and we omit the discussion for the multi-matrix extension. Another extension is to share the central tensor by different grouping layers. We implement a
3.3.1 MPO-based Network Initialization
Existing work has found that parameter initializa- tion important for training deep models (Huang et al., 2020; Zhang et al., 2019; Wang et al., 2022b), which can help alleviate the training instability. To better optimize the sealing PLMs, we propose a specially designed initialization method based on the above MPO- architecture.
Initialization with MPO Decomposition. Since MPO-based architecture shares global compo- nents (i.e.. the central tensor) across all layers, our idea is to employ existing well-trained PLMs based on weight sharing for improving parameter initial- ization. Here, we use the released 24-layer AL.- BERT with all the parameters shared across lay- ers. The key idea is to perform MPO decompo- sition on the parameter matrices of the ALBERT model and obtain the corresponding central and auxiliary tensors. We first divide the model into several groups by structure (embedding, attention, and feed-forward network). Then, for each group, We initialize central tensors by the derived central tensors from the MPO decomposition results of ALBERT. Since they are globally shared, one sin- gle copy only needed for initialization regardless of the layer depth. Next, for auxiliary tensors, we directly copy the auxiliary tensors from the MPO decomposition results of ALBERT.
strategy upon the BERT (Devlin et al., 2018), named MPOBERT, + which divides the layers into multiple parts and sets unique shared central tensors in each group.
Layer-specific Weight Adaptation. Unlike AL- BERT (Lan et al., 2019), our MPO-based architec- ture enables layer-specific adaptation by keeping layer-specific auxiliary tensors [[4,)]]] These auxiliary tensors are decomposed from the origi- nal matrix, instead of extra blocks (Zhang et al., 2022). They only contain a very small propor- tion of parameters, which does not significantly increase the model size. While, another merit of MPO decomposition is that these tensors highly correlated via bond dimensions, and small pertur- bation on an auxiliary tensor can reflect the whole matrix (Liu et al., 2021). If the downstream task requires more layer specificity, we can further in- corporate low-rank adapters (Hu et al., 2021) for layer-specific adaptation. Specifically, we denote (l(1)
Scaling the Initialization. A potential issue is that ALBERT only provides 24-layer architecture, and such a strategy no longer supports the initialization for an architecture of more than 24 layers (without corresponding auxiliary tensors). As our solution, Inspired by the idea in Wang et al. (2022b) that avoids the exploding update by incorporating an additional scaling coefficient and multiplying the randomly initialized values for the auxiliary tensors (those in higher than 24 layers) with a coefficient of (2L) 1, where L is the layer number. Then, we present a theoretical analysis of training stability.
Adopter as the low-rank adapter for W(). In this way, (()) can be formulated as set of ten- sors: {C(), (C
: AA)). ,A w(I) h. The parameter scale of adapters, I. x x diotall is deter- mined by the layer number L, the rank and the shape of the original matrix (dhotal dun + dout is the sum of the input and output dimensions of a Transformer Layer). Since we employ low-rank adapters, we can effectively control the number of additional parameters from adapters.
(1) AA(
3.3 Stable Training for MPOBERT
3.3.2 Theoretical Analysis
With the above MPO-based approach, we can scale a PLM to a deeper architecture in a highly parameter-efficient way. However, as shown in prior studies (Lan et al., 2019; Wang et al., 2022b), it is difficult optimize very deep PLMs, espe- cially when shared components are involved. In this section, we introduce simple yet stable train- ing algorithm for MPO-based PLM and then dis- cuss how it addresses the training instability issue.
To understand the issue of training instability from theoretical perspective, Trunsformer-based model F(x, W) with æ and W as input and parameters, and consider one-step update F!. According to Wang et al. (2022b), a large model update (A.F) at the beginning of train- ing is likely to cause the training instability of deep
consider
IA _ F(a, W
2 L(F(()) - v)) F(a; W).

---

Transformer models. To mitigate the exploding up- date problem, the update should be bounded by constant, i.e. || = O(1). Next, we study how the A is bounded with the MPOBERT.
3.3.3 Training and Acceleration
To instantiate our approach, we pre-train a 48- layer BERT model (i.e., MPOBERT 48). For a fair comparison with BERT and BERT we adopt the same pre-training corpus (BOOKCOR- PUS (Zhu et al., 2015) and English Wikipedia (De- vlin et al., 2018)) and pre-training tasks (masked language modeling. and sentence-order predic- tion). We first perform MPO decomposition on the weights of ALBERT and employ the initializa- tion algorithm in Section 3.3.1 to set the parame- ter weights. During the training, we need to keep an updated copy of central tensors and auxiliary tensors: optimize them according to the pre- training tasks in end-to-end way and combine them to derive the original parameter matrix for forward computation (taking relatively small cost of parallel matrix multiplication).
MPO-based Update Bound. Without loss of gen- erality, we consider a simple case of low-order 3 in Eq. (4). Follow- ing the derivation method in Wang et al. (2022b), we simplify the matrices W, A1, C and A2 to scalars W,U.C,U, which means the parameter wi wi
MPO decomposition:
the 1-th layer can be decomposed uy q - Ut. Based on these notations, we consider I-layer transformer-basee model F(x, (w =
{ (w, W/2 ..... wr.)), where each sub-layer is normal- ized with Post-LN: #1+1 = (N(x; + Gr(zz, wi)). Then we can prove [|] satisfies (see Theo- rem A.I in the Appendix):
Typically, the speed of the pre-train pre-training process is affected by three major factors: arithmetic band- width, memory bandwidth, or latency. We further utilize series of efficiency optimization ways accelerate the pre-training, such as mixed preci- sion training with FP16 (reducing memory and arithmetic bandwidth) and fused implementation of activation and normalization (reducing latency). Finally, we can train the 48-layer MPOBERT at time cost of 3.8 days (compared with a non- optimized cost of 12.5 days) on our server con- figuration (8 NVIDIA V100 GPU cards and 32GB memory). More training details are can be found in the experimental setup Section 4.1 and Ap- pendix A.3 (Table and Algorithm 2).
||A
cr |u wll tru loo vill
+ Uru | - l),,,
(2)
The above equation bounds the model update in terms of the central and auxiliary tensors. Since central tensors (cz) can be initialized using the pre- trained weights, we can further simplify the above bound by reducing them. With some derivations (See Corollary A.2 in the Appendix), we can ob- tain (u? + (1), = ) in order to guar- antee that || | = (1). For simplicity, we set u4 = U (2L) to bound the magnitude of each update independent of layer number L. In the im- plementation, we first adopt the Xavier method for initialization, and then scale the parameter values with the coefficient of (21)-44
Experiments
In this section, we first set up the experiments and then evaluate the efficiency of MPOBERT on a variety of tasks with different model settings.
Comparison. Previous research has shown that using designed values for random initialization can improve the training of deep models (Huang et al., 2020; Zhang et al., 2019; Wang et al., 2022b). These methods aim to improve the initialization of general transformer-based architectures for train- ing from scratch. As comparison, we explore the use of pre-trained weights and employ the MPO decomposition results for initialization. In partic- ular, Gong al. (2019) have demonstrated the ef- fectiveness of stacking pre-trained shallow layers for deep models in accelerating convergence, also showing performance superiority of pre-trained weights over random initialization.
4.1 Experimental Setup
Pre-training Setup. For the architecture, we denote the number of layers as L, the hidden size as H. and the number of self-attention heads as A. We report results on four model sizes: MPOBERT (L=12, H=768, A=12), MPOBERT (L=48, H=1024, A=16) and MPOBERT 48+ that implement cross-layer param- eter sharing in three distinct groups as discussed in subsection 3.2.2. We pre-train all of the models with a batch size of 4096 for 10% steps.

---

MRPC SST-2
CoLA Mcc.
RTE Acc.
STS-B
QQP
MNLI Acc.
QNLI SQuAD F1
Avg.
#To (M)
Experiments
Acc.
Spear.
F1/Ace.
Acc.
Development
Small Models (#To 100M)
12
89.0
90.6
53.4
71.1
88.2
-f89.1
84.5
89.4
89.3
82.7
ALBERT24
84.6
93.6
52.5
79.8
90.1
-/88.1
85.0
91.7
90.6
84.0
T512
89.2
94.7
53.5
71.7
-/91.1
87.8
93.8
90.0
84.8
1POBERT
55.2
71.8
-/90.1
84.7
91.2
90.1
MPOBERT
94.7
58.3
77.3
-/89.5
86.3
92.0
92.3
Base Models (#To>
100M)
BE
90.7
91.7
48.9
71.4
91.0
-990.8
83.7
89.3
88.5
82.9
110
XLNet12
85.3
94.4
49.3
63.9
85.6
-/90.7
90.9
91.8
90.2
82.5
117
RoBERTa12
91.9
92.2
59.4
72.2
89.4
-/91.2
88.0
92.7
91.2
85.4
125
BA
91.4
93.8
56.3
79.1
89.9
-/90.8
86.4
92.4
91.6
82.8
140
MPOBERT
89.7
94.4
57.4
79.8
91.1
-/89.3
87.1
92.4
91.4
86.0
102
Test set
Small Models (#To < 100M)
1LBERT
89.2
93.2
53.6
70.2
87.3
70.3/-
84.6
92.5
89.3
81.1
11
ALBERT24
88.7
94.0
51.7
73.7
86.9
69. 1/-
84.9
91.8
90.6
81.2
18
MobileBERT22
88.8
92.6
51.1
70.4
84.8
70.5/-
83.3
91.6
90.3
80.4
25
15
89.7
91.8
41.0
69.9
85.6
70.0/-
82.4
90.3
90.0
78.7
60
TinyBERT,
87.3
93.1
51.1
70.0
83.7
71.6/-
84.6
90.4
87.5
79.9
67
DistiBERT
86.9
92.5
49.0
58.4
81.3
70.
82.6
88.9
86.2
77.3
67
MPOBERT
89.2
91.9
52.7
70.6
87.1
69.6/-
85.0
91.0
90.1
80.8
20
MPOBERT
90.0
94.0
55.0
74.0
88.7
71.0/-
86.5
91.8
92.3
82.6
75
Base Models (#To
100M)
,F 88.9
93.5
52.1
66.4
85.8
71.2/-
84.6
90.5
88.5
79.1
110
XLNet12 89.2
94.3
47.3
66.5
85.4
71.9/-
87.1
91.4
90.2
80.4
117
RoBERTas2
89.9
93.2
57.9
69.9
88.3
72.5/-
87.7
92.5
91.2
82.6
125
BART
89.9
93.7
49.6
72.6
86.9
71.7/-
84.9
92.3
91.6
81.5
140
MPOBERT
89,9
94.5
56.0
74.5
88.4
70.5/-
86.5
92.6
91.4
82.7
102
Table I: Performance comparison of different models natural language understanding tasks (in percent). "# To (M)" denote the number (in millions) of total parameters. We compare MPOBERT with PLMs (i.e. BERT and ALBERT) and Parameter-efficient Transformers (i.e., MobileBERT, TinyBERT and DistilBERT), respectively. best and the second-best performance in each task are highlighted in bold and underlined. Experimental results by Sun et al. (2020b). Experimental results by Jiao et al. (2019). Experimental results by Devlin et al. (2018).
Fine-tuning Datasets. To evaluate the perfor- mance of our model, we conduct experiments on the GLUE (Wang et al., 2018) and SQuAD v1.1 (Rajpurkar et al., 2016) benchmarks. Since fine-tuning is typically fast, we run an exhaustive parameter search and choose the model that per- forms best on the development set to make predic- tions on the test set. We include the details in the Appendix(see Appendix 1 for the datasets and Appendix A.4.2 for evaluation metrics)
compressed models that have similar parameters, namely MobileBERT (Sun al., 2020b), Distil- BERT (Sanh al., 2019) and TinyBERT (Jiao al., 2019). We compare these compressed models to show the benefit of scaling to deeper models over compressing large models to small variants.
Base models (#To 100M). We compare with BERT12. XLNet12. RoBERTaj2 and BART for this category. Note that we only include the base variants that have similar model sizes in order to make a fair comparison.
Baseline Models. We compare our proposed MPOBERT' to the existing competitive deep PLMs and parameter-efficient models. In order to make fair comparisons, we divide the models into three major categories based on their model sizes:
More details about the experiments are described in Appendix A.4.
4.2 Main Results
Small models (#To <100M), JLBERT (Lan al., 2019) the most representative PLM that achieves competitive results with only 11M. In addition, we consider PLMs (T5 and three
Full Fully- -uply-su setting. We present the results of MPOBERT and other bascline models GLUE and Squad for fine-tuning in Table 1. Firstly, we evaluate MPOBERT's performance

---

in comparison other models with similar num- bers of parameters. In particular, for small mod- els, M outperforms the best baseline models and achieves substantial improvements on both the development set (85.8 v.s. 84.8 for 1512) and test sets (82.6 v.s. 81.2 for ALBERT24). This highlights the benefits of increased capacity from layer-specifie parameters (i.e., the auxiliary ten- sors and layer-specific adapters) in MPOBERT. Furthermore, for small and base models, 48-layer MPOBERT consistently achieves better results than all parameter-efficient models, while also achieving comparable results to other 12-layer PLMs with reduced number of parameters. This demonstrates the significant benefits of scaling along the model depth with layer-specific parame- ters in MPOBERT.
Datasets
B12
Mas
M12
A12
MNLI (Acc.)
83.9
85.4
82.8
82.7
QNLI (Acc.)
90.8
91.1
90.0
89.4
SST-2 (Acc.)
91.7
93.0
90.9
90.6
RTE (Ace.)
81.2
82.0
79.8
79.1
QQP (Acc.)
91.2
87.6
90.4
89.7
CoLA (Mcc.)
53.6
54,9
45.0
35.9
MRPC (FI)
84.2
91.8
89.9
STS-B (Spear.)
87.4
89.0
86.9
87.5
Avg. #To (M)
83.0
84.4
82.0
80.5
110
75
20
Table 2: Performance of multi-task learning on GLUE benchmark obtained by fine-tuning B (B12), MPOBERT (M4s). MPOBERT12 (M12) and ALBERT (A12) (in percent).
SST-2
MNLI
Shots (K)
10
20
30
10
20
30
Secondly, we assess MPOBERT's parameter ef- ficiency by comparing other PLMs within the same model depth. For instance, when consider- ing models with 1=12 layers, MPOBERT achieves comparable results or even outperforms (+1.7 for BERT 12 and +0.4 for x PLMs while hav- ing fewer parameters. This further highlights the advantages of MPOBERT's parameter-efficient ap- proach in constructing deep models.
BERT12
54.8
59.7
61.6
37.0
35.6
35.7
ALBERT12
56.7
59.3
60.0
36.3
35.6
36.5
MPOBERT12
58.9
65.4
64.6
36.7
36.7
37.1
Table 3: Comparison of few-shot performance.
that MPOBERT outperforms BERT, which suffers from over-fitting, and ALBERT, which does not benefit from its reduced number of parameters. These results further demonstrate the superiority of our proposed model in exploiting the potential of large model capacity under limited data scenarios.
Multitask Fine-tuning Setting. To demon- strate the effectiveness of our proposed parameter- sharing model in learning shared representations across multiple tasks, we fine-tune MPOBERT, BERT and ALBERT on the multitask GLUE bench- mark and report the results in Table 2. Specifi- cally, we design two groups of experiments. (1) Deep vs. shallow models. Comparing with BERT MPOBERT has much deeper Trans- former layers but still fewer total parameters (i.e., 75M vs. 110M). We find that MPOBERT achieves 1.4 points higher average GLUE score than BERT 12. (2) Central tensors sharing all weight sharing. Comparing with ALBERT12. 1POBERT12 only shares part of weights, i.e., cen- tral tensors, while ALBERT12 shares all of the weights. We find that sharing central tensors may effectively improve the average results than sharing all weights (82.0 v.s. 81.4 for MRPC).
4.3 Detailed Analysis
Analysis of Initialization Methods. This experi- ment aims to exclude the effect of initialized pre- trained weights on fine-tuning results. We plot the performance of the model on SST-2 w.z. training steps. In particular, of MPOBERT using different initialization meth- ods (Xavier in Fig. 3(a) and decomposed weights of ALBERT in Fig. 3(b)) for pre-training. The results demonstrate that pre-training MPOBERT from scratch requires around 50k steps to achieve performance comparable to BERTBASE, while ini- tializing with the decomposed weights of ALBERT significantly accelerates convergence and leads to obvious improvements within the first 10k training steps. In contrast, the gains from continual pre- training for ALBERT are negligible. These results provide assurance that the improvements observed in MPOBERT are not solely attributed to the use of initialized pre- pre-traine trained weights.
compare the performance
Few-shot Learning Setting. We evaluate the per- formance of our proposed model, MPOBERT, in few-shot learning setting (Huang et al., 2022) on two tasks, SST-2 and MNLI, using a limited num- ber of labeled examples. Results in Table show

---

93.0
93.0
Experiment
SST-2
RTE
MRPC
#To (M)
BERT MPOBERT
MPOBERT,2
92.8
72.9
91.8
20.0
92.0
92.0
w/o Adapter
92.3
71.8
90.3
19.4
91.0
91.0
w/o PS
91.4
67.9
85.8
11.9
MPOBERT
90.0
90.0
ALBERT
Table 4: Ablation study on the SST-2, RTE, and MRPC datasets (in percent).
0K 10k 20k 30k 40k 50k
0k 10k 20k 30k 40K
Pre-training Stops
Pre-training Stops
(a) Pre-trane Pre-training from scratch
(b) Continual Pre-training
Rank
SST-2
RTE
MRPC
#To (M)
Figure 3: Comparison of the SST-2 accuracy achieved through pre-training from scratch and pre-training with the initialization of decomposed ALBERT weights.
4
91.9
69.7
88.2
19.7
8
92.8
72.9
91.8
20.0
64
91.6
69.3
88.1
24.3
Surface
Syntactic
Semantic
Table 5: Comparison of different adapter ranks on three GLUE tasks (in percent). "Rank" denotes the adapter rank in MPOBERT.
BERT
also observe a decrease in the performance of the variant with adapter rank 64. This illustrates that further increasing the rank may increase the risk of over-fitting in fine-tuning process. Therefore, we set rank of for MPOBERT in the main results.
MPOBERR
Analysis of I Linguistic Patterns. To investigate the linguistic patterns captured by MPOBERT, BERT, and ALBERT, we conduct suite of probing tasks, following the methodology of Temney et al. (2019). These tasks are designed to evaluate the encoding of surface, syntactic, and semantic information in the models' representations. The results, shown in Fig. 4, reveal that BERT encodes more local syntax in lower layers and more complex semantics higher layers, while ALBERT does not exhibit such clear trend. However, MPOBERT exhibits similar layer-wise behavior to BERT in some tasks (i.e., task 0,2,4), and improved results in lower layers for others (i.e., task 3) which similar ALBERT. The result demonstrates that MPOBERT captures linguistic information differently than other models, and its layer-wise parameters play an important role in this difference.
Performance
High Performance
Figure 4: A visualization of layer-wise linguistic pat- terns. Each column represents a probing task, and each row represents a Transformer layer. The red dashed box indicates the layer that performs best.
Ablation Analysis. To assess the individual impact of the components in our MPOBERT model, we conduct an ablation study by removing either the layer-specific adapter or the cross-layer parameter- sharing strategy. The results, displayed in Table 4, indicate that the removal of either component re- sults in decrease in the model's performance, highlighting the importance of both components in our proposed strategy. While the results also indicate that cross-layer parameter sharing plays more crucial role in the model's performance.
Conclusion
We develop MPOBERT, a parameter-efficient pre- trained language model that allows for the efficient scaling of deep models without the need for addi- tional parameters computational resources. We achieve this by introducing an MPO-based Trans- former layer and sharing the central tensors across layers. During training, we propose initialization methods for the central and auxiliary tensors, which are based on theoretical analysis to address training
Performance Comparison w.r.f Adapter Rank. To compare the impact of the adapter rank in layer- specific adapters on MPOBERT's performance, trained MPOBERT with different ranks (4,8 and 64) and evaluate the model downstream tasks in Table 5. The results demonstrate that a rank of sufficient for MPOBERT, which further shows the necessity of layer-specific adapters. However, we

---

stability issues. The effectiveness of MPOBERT is demonstrated through various experiments, such as supervised, multitasking, and few-shot where consistently outperforms other competing models.
formation Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayand Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, Abs/2204.02311
Limitations
The results presented in our study are limited by some natural language processing tasks and datasets that are evaluated, and further research is needed to fully understand the interpretability and robustness of our MPOBERT models. Addi- tionally, there is subjectivity in the selection of downstream tasks and datasets, despite our use of widely recognized categorizations from the litera- ture. Furthermore, the computational constraints limited our ability to study the scaling performance of the MPOBERT model at deeper depths such 96 layers or more. This is an area for future research.
Ethics Statement
The use of a large corpus for training large language models may raise ethical concerns, particularly re- garding the potential for bias in the data. our study, we take precautions to minimize this issue by utilizing only standard training data sources, such as BOOKCORPUS and Wikipedia, which are widely used in language model training (Devlin al., 2018; Lan et al., 2019). However, it is im- portant to note that when applying our method to other datasets, the potential bias must be carefully considered and addressed. Further investigation and attention should be given to this issue in future studies.
Jacob Devlin, Ming-Wor Wei Chang. Kenton Lee, and Kristina Toutanova. 2018 8. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint 1xXiv:8800.04800
Ze- Feng Gao, Song Cheng, Rong- miin--qang He, Yuan Xie, Hui-Hai Zhao, Zhong- Yi Lu, and Tao Xiang. 2020a. Compressing deep neural networks by ma- trix product operators. Physical Review Research, 2(2):023300.
Ze-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong- Yi Lu, and Ji-Rong Wen. 2022. Parameter-efficient muitureeoff-expee architecture for pre-trained lan- guage models. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, Oc- tober 12-17, 2022, pages 3263-3273. International Committee Computational Linguistics.
Acknowledgments
This work was partially supported by National Nat- ural Science Foundation of China under Grants No. 62206299 and 62222215, Beijing Natural Science Foundation under grant No. L233008, Beijing Out- standing Young Scientist Program under Grant No. BJIWZYJHO1 00020098 and CCF-Zhipu AT Large Model Fund. Xin Zhao is the corresponding author.
Ze-Feng Gao, Xingwei Sun, Lan Gao, Junfeng Li, and Zhong-Y Yi Lu. 2020b. Compressing 1stm net- works by matrix product operators. arXiv preprint arXiv:2012.11943.
Ze-Feng Gao, Kun Zhou, Peiyu Liu, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Small pre-trained language models can be fine-tuned large models via over- parameterization. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 3819-3834.
References
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained BERT networks. In Advances in Neural In-
Timur Garipov, Dmitry Podoprikhin, Alexander Novikov, and Dmitry Vetrov. 2016. Ultimate ten- sorization: compressing convolutional and fo layers alike. arXiv preprint 161.044611..3214.

---

Linyuan Gong, Di He. Zhuohan Li. Tao Qin, Liwei Wang, and Tie- Yan Liu. 2019. Efficient training of BERT by progressively stacking. In Proceedings of the 36th International Conference on Machine Learn- ing. ICML 2019, 9-15 June 2019, Long Beach, Cali- fornia, USA, volume 97 of Proceedings of Machine Learning Research, pages 2337-2346. PMLR.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Haotang Deng, and Qi Ju. 2020. Fastbert: a self- distilling bert with adaptive inference time. arXiv preprint 270442200..02/78.
Paul Michel, Omer Levy, and Graham Neubig, 2019. Are sixteen heads really better than one? In Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.09683
Mohammadmahdi Nouriborji, Omid Rohanian. Samaneh Kouchaki, and David A. Clifton. 2022. Minialbert: Model distillation via parameter-efficient recursive transformers. CoRR, abs/2210.06425..
Xiao Shi Huang, Felipe Pérez, Jimmy Ba, and Maksims Volkovs. 2020. Improving transformer optimization through better initialization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research,
Alexander Novikov, Dmitry Podoprikhin, Anton Os- okin, and Dmitry Vetrov. 2015. Tensorizing neural networks. arXiv preprint arXiv:
pages 4475 4483, PMLR.
Ivan V Oseledets. 201 Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5):2295- 2317.
Zixian Huang. Ao Wu, Jiaying Zhou, Yu Gu, Yue Zhao, and Gong Cheng. 2022. Clues before answers: Generation Generation-enhanneed enhanced multiple-choice QA. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seatlle, WA, United States, July 10-15. 2022, pages 3272-3287. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(140):1-67. 17.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang. Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019, Tinybert: Distilling bert for natural language under- standing. arXiv preprint arXiv: 1909, 1909,1035. 10351.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages Texas, Association for Computational Linguisties.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. CoRR, abs/2001.08361.
Austin,
Machel Reid, Edison Marrese- Taylor, and Yutaka Mat- suo. 2021. Subformer: Exploring weight sharing for parameter efficiency in generative transformers. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-2 November, 2021, pages 4081-4090 Association for Computational Linguis- tics.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self- eef-suupee learn- ing of language representations. arXiv preprint arXiv:1909.1194
Peiyu Liu, Ze- Ze-Feng Gao, Wayne Xin Zhao, Zhi- Yuan Xie, Zhong-Yi Lu, and Ji-Rong Wen. 2021. En- abling lightweight fine-tuning for pre-trained lan- guage model compression based on matrix product operators. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 5388-5398.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01 108.
Xingwei Sun, Ze-Feng Gao, Zhong-Yi Yi Lu, Junfeng Li, and Yonghong Yan. 2020a. A model compression method with matrix product operators for speech enhancement. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2837-2847,
Peiyu Liu. Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji- Rong Wen. 2023. Do emergent abilities exist in quantized large language models: An empirical study. arXiv preprini arXiv:2307.08072.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020b. Mobilebert: compact task-agnostic bert for resource-limited de- vices. arXiv preprint arXiv:2004.02980

---

lan Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextu- alized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li. Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE in- ternational conference on computer vision, pages 19-27.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: Sadi:08804.0440
Benyou Wang, Yuxin Ren, Lifeng Shang, Xin Jiang, and Qun Liu. 2022a. Exploring extreme parameter compression for pre-trained language models. In The Tenth International Conference on Learning Repre- sentations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022b. Deep- net: Scaling transformers 1, 000 layers. CoRR, abs/2203.00555.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2020. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 6151 6162. Association for Computational Linguisties.
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. 2017. Long-term forecasting using tensor-train rnns. Arxiv.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. 2019. Fixup initialization: Residual learning without normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OevnRevie.
Jinnian Zhang, Houwen Peng. Kan Wu, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. 2022. Minivit: Compressing vision transformers with weight mul- tiplexing. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 12135- 12144, IEEE
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yinggian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du,

---

A Appendix
A.2 Proofs
A.1 Matrix Product Operators
Notations. We denote L-) as the loss function. (x(x) as the standard layer normalization with scale y = and bias 3 = Let O-)) denote stan- dard Big-O notation that suppresses multiplicative constants. +
Formally, given weight matrix W E RIxJ, we can factorize the two dimensions into a prod- uct of natural numbers, and reshape it into an n- dimension tensor W minde which satisfies:
stands for equal bound magnitude. We aim to study the magnitude of the model up- dates. We define the model update as ||F]].
II T
II ja J.
(3)
Definition F(z, 0) is updated by ( per SGD step after initialization as 17 + 0. That is, [ || - O(n) where (x) can be calcu- 8 (F(()) u)) (xr
This dccomposition can be written
W Wa = T [11. B..
(4)
where the (hp (it. j&l is 4-dimensional tensor with size dx-1 X 2 jik d2 in which dş is a bond dimension linking (kk) and T(++1) with do = dn 1. with size d x in x jk x dg in which II 4 = I, I[ Js = J and do = dn- = 1. This bond dimension indicates the associative strength between two adjacent tensors. For clarity, we can rewrite the decomposition results as cen- tral tensor C and auxiliary tensors {A, As an important merit, such a decomposition can effec- tively reorganize and aggregate the information of the matrix (Gao et al., 2020a): central tensor C can encode the essential information from the original matrix, while auxiliary tensors {= serve as its complement to precisely reconstruct the matrix. The k-th order and & E (1,.... D). The bond dimension dş is defined by:
lated through F(x, A - 1 3
Theorem A.1 Given an N-layer transformer- based model F(x, 0) (01. (01,02..... 0g.00..0000 0N)), where Oi denotes the parameters in 1-th layer and each sub- is normalized with Post-LN: 2111 LN (at + Gi( 0()). In MPOBERT, Ot decom- posed by MPO to local tensors: 0₃ = -u C Us and
we share {ci),1 across N layers: oi = c1, 1,2, .. N. Then ||AF] satisfies:
N |AFI
Σ (cr2 ||u uzll + +auullof mal
1=1
+ 4.4 || - cill)
-i
(6)
Proof. We follow (Zhang et al., 2019) and make the following assumptions simplify the deriva- tions:
dz -imn II
im x Im
II m=k+1
im x Jm (5)
From Eq. (5), we can see that is going to be large in the middle and small on both sides. Algorithm I presents thorough algorithm for MPO decompo- sition.
Hidden dimension equals to 1;
2. var(z + Gi(z)) @ var(z) + var(Gi(z));
Algorithm MPO decomposition procedure.
3. All relevant weights 0 are positive with mag- nitude less than 1.
Require: matrix W IXJ. the number local tensor m Output local tensor set (7 (k) y
1: for l: ,,
do
Given Assumption 1, if Gi is MLP with the weight 0, then Gz(z) Ozz. With assumption 2, we have:
2: Wide-1 ig x je, - I]] + Reshape(W[[)
3: UAVT SVD (W)
9
4: T (1K)dd11, in. de] + Reshape((U U)
5:
Calculate W AV
6: end for
Let (()) W
x Gi(z)
11 f((11 =
(7)
Normalization
VVrl( + G((z))
9: return local tensor set (7))) m
1 A
(8)
+ 07
The MPO representation of W is obtained by factorizing it into sequential product local ten- sors.
Then, with Taylor expansion, the model update

---

|AF | satisfies:
Proof. For an N-layer MPOBERT, we have:
||FI|
Fl,, )²) F(z, A11
11 - -N++Ill
|AFI
S
(us
||
uil
+
u4
|
U/
I)
(16)
i=1
Ilf(o' 0%) - (o0)-((xap) f(INN ON)l
N
0c
0F
00:
I.f(ar, Uj. VN.VV.V3 Vx)
17
U1s
OF
00,
au;
f(NN, UN. ON, VN)1 0f dz (IN -xN)
0F
00;
+
(17)
OF
00;
Oua
0f OB
00 OUN
(UN - UN)
By assumption
ac @F
=
-(11
and
09,
aF
0f 00
OP N
@
|6
we achieve:
(CN CN)T
8A NCN
N
0f 00
(Vλ VN)
ac
aF
00:
a0 OVN
(9)
7
(12
0F
00;
With Eq. (8), the magnitude of dr
0A
and
0f is 30
+
Us
0C
OF
00;
(18)
bounded by:
0F
00:
Oug
N
05
9
1+0
(10)
1)
(ef uNNN + uZuNON)
Fx
V + f9
1=1
0h 9
0
0
2
(uf + ((xxe)))) O(1),
(19)
(1 + 0?) Ft
(11)
00,
Since we apply MPO decomposition to 8, we get:
Finally, we achieve:
0 U-- CU- C.-Vi Vi
(12)
(0f + (UN()N)))) = 0( n
(20)
For simplicity, we reduce the matrices U.C.V to the scalars U,C,U. Thus with Assumption 3, Eq. (9) is reformulated as: Finally, with Assumption 3 we have:
Due to symmetry, we set in U. U3 = V. Thus, from A.2, we set v (2N) to achieve bound the magnitudes of each update to be inde-
pendent of model depth N, i.e. ||AF| - O().
|AF||
|x xN+1ll
(13)
A.2.1 Details of Training
1- 4,012 (c12 1|4 will
A.3 Training Details
1= (1 + =(cco??))
Here describe the details of the pre-training process in Algorithm 2. For pre-training, we tune the learning rate the range of [1.0 x 10 5 1.0 10-61 and use the LAMB optimizer (You et al., 2020). Since fine-tuning is typically fast, we run an exhaustive parameter search (i.e., learning rate in the range of [2.0 x 10 4, 2.0 x 10 61, batch size in (8, (,31)) and choose the model that performs best on the development set make predictions the test set.
crui || - w;ll) + uuu || cill)
(c12 114 - u + + ||o - wlll
1=1
rrl |ca lil
(14)
O
Corollary A.2 Given that we initialise C1 in MPOBERT with able to assume that updates of 01 are well-bounded. Then &F satisfies |AFI| - O(1) when for all 1, N:
weights, it is reason-
A.3.1 Details of Training Configurations
In this part, we list the training configurations of MPOBERT and other representative PLMs in Ta- ble 6.
(0) + u()(uN))= - )
(15)

---

Models
#To (M)
Depth
Samples
Training time
GLUR Dev.
GLUE Test
T511B
1 11000
24
89.0
T5BASE
220
24
128x
524k
TPU v3 Day (t5-hase)
84.1
82.5
LERGLARGE
330
24
256x
1000k
16 Cloud TPUs 4 Days
84.1
81.6
ALBERT XXLARGE
235
4096x 1.5M
TPU v3 16 Days
90.0
BARTLARGE
407
24
8000
500k
88.8
ROBERTALARGE
355
24
8000 500k
1024 V100 GPUs Day
88.9
XLNeILARGE
361
24
8192x 500k
512 TPU v3 5.5 Days
87.4
MPOBERT 48+
102
4096x 10k
8 V100 GPUs
85.6
81.7
3.8 Days
Table Comparison with the strongest variants of popular PLMs. Since T5mB has far more parameters than other candidates, it's reasonable to use T5 for fair comparison.
Algorithm 2 The MPOBERT training procedure.
SQuAD is collection of 100k crowd-sourced question/answer pairs. Given question and pas- sage, the task to predict the answer text span in the passage.
Require: (!)); Weight matrix (-th layer MPOBERT.
W Pre-trained weight matrix in ALBERT. U and
D Matrices in low-rank adapter. yy: learning rate. L:
Stochastic objection function. L: Model layers number.
(MPO decomposition)
A.4.2 Details of Evaluation Metrics
1: (A A c.. .A
(0)) +- MPO (W
2: (_4 - c(o) 50 1 MPO (W
Following Gao et al. (2022), we employ Matthew's ss correlation for CoLA. Spearman for STS-B, F1 for MRPC, and accuracy for the remaining tasks as the metrics for the GLUE benchmark. We compute and present the average scores across all test samples for each of the aforementioned metrics.
3: (Initialization Procedure)
24 4: c + (()) {AA 15= + 54
5: end for
6: for 24 < 1 < L do
7:
Ct9 +- ((0) (A4
1 (14
- {(2L) - 1
8: end
9: U
+ O. D OO)
A.5 Analysis of Inference Speed
10: W A A
(()) a A + (1()
Adauter
(Training procedure with mixed precision and fused im- plementation techniques.)
In this part, we provide FLOPs and inference costs for MPOBERT-48, MPOBERT-12 and BERT-12. Note that have primarily focused the FLOPS within the Transformer layer. By conducting mul- tiple experiments and averaging the results, we report the results in Table 7. (1) MPOBERT-12 exhibits higher FLOPs than BERT-12. Notably, owing to MPOBERT-I2's significantly reduced pa- rameter count in comparison to BERT-12 (20M vs. 10M), it becomes more feasible to increase the inference batch size within similar GPU memory (from 4x 5x). (2) In the case of MPOBERT-48 compared BERT-12, there higher increase in FLOPs. The added computational load originates from increased hyperparameters of layers "L" (48 vs. 12) and hidden dimension size "H" (1024 vs.
while converged do
12:
(c(
13:
Ge S(w
14:
@ + w.!
--1 7
15:
end while
16: return Converged model
A.4
Experimental Details
A.4.1 Details of Fine-tuning Datasets
GLUE benchmark covers multiple datasets (MNLI, QNLI, QQP, CoLA, RTE, MRPC, SST-2) 2 The
In line with Raffel et al. (2020), we do not test WNLI due to its adversarial character with respect to the training set.

---

Table Model Performance Comparison
Model
GLUE
GFLOPs
Inference
BERT-12
79.1
96.64
1.00x
MPOBERT-12
80.8
139.80
0.94x
MPOBERT-48
82.6
670.01
5.88x
768). The remarkable enhancement the perfor- mance of MPOBERT-48 is clearly demonstrated by the results of the GLUE Test (82.6 vs. 79.1), un- derscoring the value of the computational load as- sociated with this part. As result, recommend prioritizing MPOBERT-48 in scenarios where per- formance takes precedence.
is important highlight that MPOBERT holds the promise of substantially enhancing inference speed. This is mainly attributed to the fact that the speed of inference is typically constrained by memory bandwidth, which, in turn, is restricted by the available memory capacity. To elucidate this point, envision situation where the model weights utilized in matrix multiplications stored in smaller yet high-bandwidd memory. In such cases, there exists a notable potential for achieving a significant speed boost, particularly when com- pared with the situation where these weights must be fetched from larger memory characterized by lower bandwidth.


Reviews:
[{"review_id": "Gicv3uY3YP", "reviewer": "Reviewer_mWkV", "summary": "", "questions": "Figure 3 illustrates a continuous improvement in model metrics. Why was the training stopped prematurely?\n\n\n\n\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful suggestions and we have listed our response to your concerns as follows. If you also have any other questions, please feel free to let us know. We will continue to try our best to answer for you.\n\nR1:\nWe appreciate the reviewers' attention to the scope of our experiments. Our approach focuses on targeted modifications to specific matrices within the Transformer structure, namely the Q,K,V and MLP matrices. These components are foundational to various models based on the Transformer architecture. So, we only chose the most typical BERT for validation, and we will expand our development on more models in the future.\n\nR2:\nWhile our method has shown improvement over BERT, we acknowledge that surpassing models like RoBERTa-large and T5 have not been achieved. In this study, we used the same training data as BERT (wiki and bookcorpus) and trained for approximately 10k steps over around 3.8 days. The limitations in data availability and training steps could be factors contributing to why our model did not outperform the current best-performing models.\nIt's important to note that the core focus of this paper was to contrast against BERT and validate a parameter-efficient structure for building deeper models. Expanding resources and training time is quite time and resource consuming which we intend to explore in our future work, considering the current resource constraints. \n\nR3:\nThank you for your valuable suggestion. We have incorporated a table presenting the training time and GPU memory usage for the 48-layer MPOBERT in our submission. Utilizing the Deepspeed (DS) training framework with FP16 (fp16) training, our model was trained in just 3.8 days. We will ensure that this table is included in the subsequent sections of the main paper for better visibility and context.\nCopy code\n| Exp.       | BS | Mem (G) | Days  |\n|------------|----|---------|-------|\n| MPOBERT\\_48| 4  | 11.6    | 4.9   |\n|            | 8  | 16.3    | 3.8   |\n| w/o DS     | 4  | 11.4    | 5.4   |\n| w/o fp16   | 4  | 18.7    | 12.2  |\n| w/o DS,fp16| 4  | 19.2    | 12.5  |\n\nR4:\nWe sincerely appreciate your valuable suggestion. The parameter $L$ represents the length to which the MPO is decomposed.  It's worth noting that during the MPO decomposition process, Liu et al. [1] conducted an extensive study on the effects of varying the length parameter L for MPO decomposition. Their findings indicate that the choice of L has minimal impact on the error between the reconstructed weight matrix obtained through MPO decomposition and the original matrix. In light of your feedback, we will add to this discussion in the main text to offer readers a more comprehensive understanding of the parameter $L$. \n\nQ1:\nWe sincerely appreciate your insightful query. In our study, we established a consistent number of pretraining steps to ensure a fair basis for comparison. While Figure 3 showcases the potential for enhanced performance through more pretraining steps, it's crucial to acknowledge that pursuing further steps and utilizing larger datasets demand substantial computational resources. This intriguing avenue for research is undoubtedly a direction we plan to delve into more deeply in the future.\n\n[1] Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operators", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The author conducted experiments on multiple datasets, and the results indicate that the proposed method achieves competitive performance compared to the state of the art (SOTA).\n\n- The proposed method appears to have applicability across a wider range of Transformer-based models.\n\n- The author provides theoretical analysis of the proposed method.", "reasons_to_reject": "- In line 95, the author claims that the proposed method could be applied to various Transformer-based models, yet the experiments are exclusively conducted on the BERT model.\n- Some important baseline comparisons are missing, such as RoBERTa-large. Even if the proposed method does not outperform the SOTA baseline, the author should analyze the reasons behind this.\n- An analysis of training/inference time is not present.\n- The ablation analysis lacks examination of certain crucial hyper-parameters, such as the analysis of length L.\n\n\n\n ", "questions_to_authors": "Figure 3 illustrates a continuous improvement in model metrics. Why was the training stopped prematurely?\n\n\n\n\n", "paper_topic_and_main_contributions": "This work introduces a parameter-sharing approach for scaling pre-trained language models (PLMs) to a deeper model depth using the matrix product operator (MPO).\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VyirQHLLI8", "reviewer": "Reviewer_4FdP", "summary": "", "questions": "How does parameter-sharing affect pre-training? Does it slow down or speed up the convergence?\n\nHow does parameter-sharing bring efficiency improvements?\n\nHow does parameter-sharing affect performance?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your insights and suggestions are greatly appreciated. Below, we have provided detailed responses to address your concerns. If you have any additional questions or inquiries, please feel free to share them with us.\n\nR1:\nThank you for your valuable feedback. In order to highlight the advantages of our approach in terms of increased layer depth with parameter sharing, we have incorporated the loss curves during model training. As illustrated in the table below, we present the validation loss of two models in different steps: (1) MPOBERT_48, representing the deep model with parameter sharing, and (2) MPOBERT_12, representing the shallow model with parameter sharing. By comparing the results of (1) and (2), it becomes apparent that during the initial training stages, the shallow model exhibits lower loss due to better initialization from pre-trained weights across all layers. However, as training progresses, the deep model, owing to its larger model capacity, tends to have a lower loss, showcasing its greater potential during training.  In terms of perplexity, the language modeling perplexity for held-out training data is 6.068 for MPOBERT_12 and 4.855 for MPOBERT_48.\n\n| Model      | 0      | 2000   | 4000   | 6000   | 8000   | 10000  |\n|------------|--------|--------|--------|--------|--------|--------|\n| MPOBERT_48 | 4.368  | 2.160  | 1.844  | 1.698  | 1.583  | 1.580  |\n| MPOBERT_12 | 3.280  | 2.260  | 2.243  | 2.080  | 1.829  | 1.803  |\n\nR2&Q2&Q3:\nWe summarize the efficiency of our approach as follows: (1) Real-Time Inference Latency: the MPO-based Transformer's weights can be reconstructed into matrices, which requires a one-time reconstruction before inference. Subsequent inference processes share the same latency as the standard Transformer, without introducing extra computations. (2) Memory Consumption: the adoption of MPOBERT_48 allows for a 48-layer depth with only 16.3G GPU memory, enabling feasible pretraining on a single 24G GPU. (3) Training Time: utilizing 8 V100 GPUs, the training process is completed in just 3.8 days, a substantial reduction compared to the training cost of BERT large. Furthermore, it's noteworthy that MPOBERT_48 outperforms BERT large, demonstrating its superiority with a notable GLUE test score of 81.7, surpassing BERT large's 81.6.\n\n| Model        | #To (M) | Mem (G) | #samples   | #Train time                | GLUE test |\n|--------------|-----|---------|------------|----------------------------|-----------|\n| BERT large   | 330 | -       | 256*1000k  | 16 Cloud TPUs, 4 Days      | 81.6      |\n| MPOBERT_48   | 75  | 16.3    | 4096*10k   | 8 V100 GPUs, 3.8 Days      | 81.7      |\n\nR3:\nIt appears that you're referring to the comparison between applying parameter sharing on a MPO-based Transformer layer. Considering the inherent over-parametrization often seen in Transformer models, not implementing parameter sharing results in considerable parameter proliferation as model depth increases. To illustrate, we conducted a comparison involving 12-layer models, designated as \"share\" (with parameter sharing) and \"w/o share\" (without parameter sharing). The outcomes reveal that forgoing parameter sharing leads to a substantial 4.9-fold increase in parameters, without yielding proportionate benefits. For instance, in the case of MPOBERT_48, the model features fewer parameters (75M vs. 85M) while exhibiting a lower validation loss (1.583 vs. 1.778).  This outcome underscores that parameter sharing proves more efficient in resource utilization. We appreciate your observation and have discussed this aspect further to provide clarity on the advantages of our approach.\n| Model      | #To (M) | Pre-training steps | Validation Loss |\n|------------|---------|--------------------|-----------------|\n| Share      | 20      | 10k                | 1.803           |\n| w/o Share  | 98      | 10k                | 1.778           |\n\nQ1:\nPlease refer to our response ``R3'' to reviewer-mWkV, where we reported the training time and the effects of parameter sharing and initialization. These factors enabled us to train a 48-layer large-scale model in just 3.8 days, which is the shortest training time among all the large-scale models considered.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Large language models are overparameterized models. Sharing parameters across modules make sense.\n\nThis paper demonstrates the effectiveness of the model both experimentally and theoretically.", "reasons_to_reject": "Missing some important experimental results. (loss curve and ppl.). I think that the results on GLUE benchmark are not strong enough to support the claim that deeper models with parameter sharing are better.\n\nI doubt that parameter sharing can really bring efficiency improvement. The authors should provide a more explicit explanation of the \"efficiency\" of their model. For instance, presenting real-time inference latency, memory consumption, or training time would be more persuasive.\n\nIf I am not mistaken, parameter sharing involves reparameterizing the original module without altering the model architecture. However, the authors did not compare models with the same architecture.", "questions_to_authors": "How does parameter-sharing affect pre-training? Does it slow down or speed up the convergence?\n\nHow does parameter-sharing bring efficiency improvements?\n\nHow does parameter-sharing affect performance?", "paper_topic_and_main_contributions": "This paper proposes a parameter-sharing method that can scale pre-trained language models to a deeper depth while maintaining a relative parameter size.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AmlIyfUwZu", "reviewer": "Reviewer_v9WE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable insights and suggestions. In response to your concerns, we have outlined our responses below. Please do not hesitate to share any further inquiries you may have.\n\nR1:\nThe bond dimension is calculated within the MPO decomposition based on the matrix structure and cannot be manually adjusted. It is determined by the hyperparameter MPO shape. In contrast, the adapter rank is a tunable hyperparameter designed to control the additional parameter count introduced. These two parameters are not directly interrelated. We will make this clear in the main text of our paper.\n\nR2:\nWe appreciate your attention to this issue. The choice of [4, 8, 64] was primarily based on the observation that increasing the adapter rank beyond 8 did not yield significant performance gains in our experiments. The selection aimed to encompass a broad range of values, evaluating the impact across varying levels of adapter rank. Our findings indicated that adapter ranks from 8 to 64 did not substantially enhance performance. We recognize the need to clarify this rationale in our paper and will ensure to provide a more detailed explanation in the main text.\n\nR3:\nThank you for your insightful observation regarding stable training in Section 3.3. Your feedback is invaluable in highlighting the importance of this aspect. It's worth noting that, without the proposed initialization method, convergence for the MPOBERT_48 model would not be achievable. To address this concern, we will provide additional explanations and thorough validation of this aspect in the main text of our paper.\n\nTypos:\nThank you for your thorough review. We have diligently reviewed the labeling errors within the table and revised the grammatical errors present in the paragraphs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper offers a novel perspective on the conventional tensor-train decomposition, introducing a fresh parameter-sharing approach that can be applied to deep models. Ample theoretical derivation supports its stability in training deep architectures, thereby enhancing the method's interpretability.\n2. Multiple sets of experiments are devised to demonstrate the effectiveness of the model, encompassing full supervised settings, few-shot learning setting, multi-task learning setting, and more. The experimental setups are comprehensive, and detailed analyses are conducted for each structural aspect of the model.\n3. The writing is clear and facilitates easy comprehension for readers.", "reasons_to_reject": "1. The connection between bond dimension and adapter rank is unclear, further theoretical explanation is needed.\n2. . In the experiments of Adapter Rank in Section 4.3, it seems that it is not a comparison fair with numerical choices of [4, 8, 64]. why not [4, 8, 12] or [4, 16, 64].\n3. As per my understanding, stable training in Section 3.3 is a highlight of the paper, and the authors have provided ample theoretical derivation. However, this point has not been further validated in the experimental section.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a highly parameter-efficient approach.\n1. The core contribution involves a parameter-sharing architecture based on the Matrix Product Operator (MPO) for deep Transformer networks. To train such deep architectures, an MPO-based initialization method is proposed, leveraging the MPO decomposition results of the ALBERT model. This approach shares the central tensors of parameter matrices across all layers to reduce model size.\n2. To maintain layer adaptability, a low-rank adapter is introduced.\n3. Theoretical analysis is employed to demonstrate that the proposed MPOBERT can accommodate deeper networks and address training instability issues.\n4. The authors conducted thorough experiments.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. There seem to be some labeling errors in the results of Table 1. Please double-check them.\n2. Line 321-Line 327 contains grammar mistakes."}, "extra_text": {}}, {"review_id": "OIycBwtCSE", "reviewer": "Reviewer_M4qE", "summary": "", "questions": "- Given that the parameter matrices exhibit different shapes, it is imperative to elucidate how the rank for the shared and layer-specific tensors is determined.\n- The division of groups in MPOBERT+ remains unclear in the paper, also leaving ambiguity surrounding the differentiation between MPOBERT and MPOBERT+. Is the key distinction solely related to the sharing granularity (layer vs. group)?\n- How about the inference speed of MPOBERT compared with models of the same size? Considering that ALBERT's FLOPs remain nearly identical to the original BERT model, the inference speed for MPOBERT48 maybe much more slower than BERT12. \n- Does the training speed in table 6 for MPOBERT48+ benefit from initialization  from BERT?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your perceptive insights, and our detailed responses to your concerns are outlined below. If you have any additional inquiries, please feel free to let us know.\n\nR1Q1:\nWe appreciate your insightful feedback. In the MPO decomposition, we note that the central tensor (i.e., the 3rd tensor when we have 5 decomposed tensors) have the most parameters, leading to the choice of central tensors for shared tensors. Empirical validation of rank determination for layer-specific tensors showed consistent performance across various ranks (rank=4, 8, 64 in Table-5). Considering the balance between efficiency and performance, a rank of r=8 is generally a favorable choice. We acknowledge the importance of clarifying these details and intend to incorporate this discussion into the main text to provide a more comprehensive understanding.\n\nR1Q2：\nYour observation is accurate. The fundamental difference between MPOBERT and MPOBERT+ primarily lies in the sharing granularity. MPOBERT+ introduces parameter sharing at the more refined \"group\" level, thus intending to enhance both the parameter count and the corresponding expressive capacity of the model which potentially leads to enhanced feature extraction and improved overall model performance.\n\nR2Q3：\nIn addressing the concern regarding real-time inference latency, we would like to emphasize that the MPO-based Transformer's decomposed tensors undergo a one-time reconstruction into matrices before inference begins. This reconstruction step is incurred only once, and as a result, subsequent inference processes exhibit latency comparable to that of the standard Transformer. Importantly, this approach eliminates the need for additional computations during inference, thereby effectively maintaining efficiency without introducing any overhead.\n\nQ4：\nI guess you are inquiring whether the training speed of MPOBERT_48+ benefits from the initialization method mentioned in our Section 3.3 (ALBERT initialization). Indeed, it does. Referring to the response provided by Reviewer-4FdP (R1), both MPOBERT_48 and MPOBERT_48+ converge to lower loss values quickly. This rapid convergence can be attributed to the favorable initial values that facilitate more stable optimization during the early stages.\n\nMissing reference:\nThank you for your careful reading and we will cite this work and discuss it.\nThe mentioned paper introduces a tensor decomposition technique that relies on the assistance of pre-existing models for knowledge distillation. However, it's important to note that our work revolves around creating novel, deeper models without the reliance on existing larger models for distillation. In this context, relying solely on tensor decomposition is insufficient. Our approach is further enriched by the incorporation of a theoretically guided initialization method to address the training instability issue. These aspects, which distinguish our work, will be thoroughly elaborated upon in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors systematically conduct experiments across various settings, demonstrating a thorough and comprehensive evaluation.\n- Theoretical analysis is offered by the authors for initialization.\n- The analysis of linguistic pattern is interesting.", "reasons_to_reject": "- The clarity of the proposed methods is lacking, and essential details necessary for result reproduction are missing.\n\n- Regarding the training of such a model, it appears that the inference speed may be slow, raising potential concerns.", "questions_to_authors": "- Given that the parameter matrices exhibit different shapes, it is imperative to elucidate how the rank for the shared and layer-specific tensors is determined.\n- The division of groups in MPOBERT+ remains unclear in the paper, also leaving ambiguity surrounding the differentiation between MPOBERT and MPOBERT+. Is the key distinction solely related to the sharing granularity (layer vs. group)?\n- How about the inference speed of MPOBERT compared with models of the same size? Considering that ALBERT's FLOPs remain nearly identical to the original BERT model, the inference speed for MPOBERT48 maybe much more slower than BERT12. \n- Does the training speed in table 6 for MPOBERT48+ benefit from initialization  from BERT?", "paper_topic_and_main_contributions": "This paper presents a parameter-efficient approach, utilizing a matrix product operator (MPO) to scale pre-trained language models (PLMs) to a deep structure. The models are parameterized into shared local tensors and layer specific tensors. To train such models, the authors introduce an MPO-based initialization method, leveraging the MPO decomposition results of ALBERT. Extensive experiments under fully-supervised setting, multitask fine-tuning and few-shot learning settings show the effectiveness of the propose methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The authors should include the paper [1], which is also a MPO-based architecture with the parameter sharing strategy. And discuss the difference in related work. Since the authors argue that their paper has a difference focus from existing work, while [1] optimizes the deep MPO-based architecture with the parameter sharing strategy, and explore the use of well-trained PLMs for initialization.\n[1] Exploring extreme parameter compression for pre-trained language models, ICLR 2022"}}]

Author rebuttal:

Meta-review: The paper introduces a novel parameter-sharing method based on the Matrix Product Operator (MPO) for scaling pre-trained language models (PLMs) to deeper model depths. The work emphasizes sharing central tensors of parameter matrices across all layers to achieve efficient deep Transformer architectures. The paper also includes theoretical analysis for why the proposed approach may address observed training instability of deeper PLMs.

Generally, the reviewers liked this approach as it seems broadly applicable to many Transformer-based architectures and there is some consensus that these are often over-parameterized making parameter sharing seem justified. There was some skepticism that parameter sharing can actually lead to improved performance (or similar performance using fewer parameters). The authors provided additional experiments which appeared to satisfy the reviewers.
Decision: Accept-Findings