A Multi-Level Framework for Accelerating Training Transformer Models
Authors: Longwei Zou
Han Zhang
Yangdong Deng
Venue: ICLR 2024 poster (2024)
Keywords: Large Model
Transformer
Multi-Level
Training Acceleration
Primary area: optimization

Abstract: The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other domains. Training such models, however, poses an unprecedented demand for computing power, which incurs exponentially increasing energy cost and carbon dioxide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by a set of key observations of inter- and intra-layer similarities among feature maps and attentions that can be identified from typical training processes, we propose a multi-level framework for training acceleration. Specifically, the framework is based on three basic operators, Coalescing, De-coalescing and Interpolation, which can be orchestrated to build a multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters between adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model while preserving the performance.

Published as a conference paper at ICLR 2024
A MULTI-LEVEL FRAMEWORK FOR ACCELERATING TRAINING TRANSFORMER MODELS
Longwei Zou, Han Zhang, Yangdong Deng
Tsinghua University (zoulw22 w22, (han-zhan20) 2ma @mai 1s ..s inguuag edu. en dengydet edu. cn
ABSTRACT
The fast growing capabilities of large-scale deep learning models, such as Bert, GPT and ViT, are revolutionizing the landscape of NLP, CV and many other do- mains. Training such models, however, poses an unprecedented demand for com- puting power, which incurs exponentially increasing energy cost and carbon diox- ide emissions. It is thus critical to develop efficient training solutions to reduce the training costs. Motivated by set of key observations of inter- and intra- layer similarities among feature maps and attentions that can be identified from typical training processes, propose multi-level framework for training accel- eration. Specifically, the framework is based on three basic operators, Coalescing. De-coalescing and Interpolation, which can be orchestrated to build multi-level training framework. The framework consists of a V-cycle training process, which progressively down- and up-scales the model size and projects the parameters be- tween adjacent levels of models via coalescing and de-coalescing. The key idea is that a smaller model that can be trained for fast convergence and the trained parameters provides high-qualities intermediate solutions for the next level larger network. The interpolation operator is designed to break the symmetry of neurons incurred by de-coalescing for better convergence performance. Our experiments on transformer-based language models (e.g. Bert, GPT) as well as a vision model (e.g. DeiT) prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-BBas models and up to 51.6% on training the BERT-Large model while preserving the performance.
INTRODUCTION
Recent years have witnessed an unprecedented success large scale deep learning models in areas such natural language processing (NLP), computer vision (CV) and graph analysis. BERT (De-
Vlin et al. 2019). GPT (Radford et al. 2018 2019: Brown et al. 2020 OpenAl 2023) and other large language models have revolutionized the landscape of NLP, while the transtormer based CV models (e.g. ViT model (Dosovitskiy et al.) (02II) and more conventional convolutional neural net- works are demonstrating remarkable performance
vision processing applications. The excellent performance of these large models, however, demands an ever rising amount of computing power increasing exponentially with the model size and thus escalating energy cost and carbon emissions. al.
For example, LLaMA-65D was trained with 2048 A100 GPUs in period of 21 Mays/Touvar )023) and took an estimated cost of $4 million 2 Morcover, it has become a growing concern of the increasing monopoly of large models by few giant companies who can afford the excessive developing cost. is thus pressing to seek efficient solutions to accelerating the training process for large models so as to democratize the large Al models a wider audience.
Increasing the number of parameters in deep learning models proves to be the most effective way to improve the expressiveness of deep neural networks and ineur emergent capabilities. However, the training complexity is also largely determined by the number of parameters in the pervasively used gradient-based descent formulation in which the parameters are incrementally adjusted to minimize
"ode available at https://gthuh.coo//// wwt..oooooooo//ouuiiliicccc Traeee--rrinng----- Training-Framewort Framework
"hhe estimation is hased average cost of $3.93 per A100 GPU per hour on Google Cloud Platform:

---

Published as a conference paper at ICLR 2024
H10
H11
H7 L3H1 L3
L55 H2
15 H4
H8
Figure I: Visualization of attention patterns in BERT-Base of a randomly chosen sample sentence. The darker the color, the more attention a token pays to another one. The first row shows the attention patterns of various heads on layer 4 and represents the similarity within a layer. The second row shows the attention patterns for adjacent layers of layer 4, i..., layer 3 and layer 5, and demonstrates the similarity between layers. The similarities inter and intra-layers offer the potential for accelerating training with multi-level framework.
the difference between the model output and the ground truth result. Therefore, is appealing to develop training framework that works on a reduced set of parameters but at the same time re- serve expressiveness of the underlying model. We realized that the well-known multigrid algorithm (Trottenberg et al., 2000), which is an efficient numerical procedure to solve linear equations and partial differential equations, actually offers such a potential. The central idea is to re-structure the underlying problem into hierarchy of interrelated grids. Solving the problem on the coarse grid is equivalent to removing low frequency errors. The resultant solution process can be more efficient as only reduced number of variables have to be tackled. The solution is then mapped back to the fine grid as an initialization for the removal of high frequency errors. In comparison with traditional solvers, multigrid enables remarkable performance in terms of convergence speed and robustness (Stben 2001). Inspired by the central computing pattern of the multigrid algorithm, we perform systematic investigation on accelerating the training of large models with a multi-level framework.
The feasibility of multi-level training framework is justified by the following three observations. First, in given family of models, the smaller ones (e.g. BERT-Base and GPT-Base) always con- verge faster than the bigger ones (e.g. BERT-Large and GPT-Large) but deliver lower level of expressiveness. Second, our experimental results on BERT demonstrates that there are many similar patterns in each layer. Figure | visualizes the attention patterns of randomly selected sentence on BERT-Base. The attention patterns of various attention heads extracted from layer 4 are shown in the first row. It can be seen that many heads, e.g. the 7th head (L4 H7) and 9th head (L4 H9), have almost identical patterns. The similarity within a layer has also been discovered in Convolutional Neural Networks (Zeiler & Fergus 2014). Third, previous works show that given model tends to exhibit significant similarities among adjacent layers. In (Gong et al. 2019). the authors indicate that neighboring layers in transformer model have similar attention patterns. We illustrates some attention patterns from layer and layer 5 at the second row in Figure [ The results confirm the observation of (Gong et al. 2019). In addition to the above observations, few recent works (Gong et al. 2019 Yang et al.J 2020; Chen et al. 2016 2022) exploit the fast convergence of a smaller model to train large scale model by progressively increasing the model size. Such works can be regarded as special cases of our multi-level framework with a single de-coalescing operation that maps the parameters of a small network to a larger one.
In this work, we propose the first overall framework for multi-level training of deep learning models. The framework is built upon three key operators, Coalescing, De-coalescing, and Interpolation. Consisting V-cycle working flow, our multi-level framework progressively down- and up-scales the model size and project the parameters between a models in adjacent levels. The fast convergence of smaller models provides high-quality intermediate solutions for the next level larger network, which saves computational cost. Figure 2 demonstrates the V-cycle training process with 2 levels.
The major contributions of this work framework for transformer based deep learning models. (2) We develop a formal formulation for the three basic operators and introduce guidelines to design these operators for numerical robustness
follows. (1) We propose an efficient multi- level training
2

---

Published as a conference paper at ICLR 2024
(start)
Training Flow
(convergence)
(WI...Wi.l Toin
N, [WL.... Train(())
Coalescing
& Interpalation
Model
Pa wiri
W1
E1 : Rea
[1
[.
winh Cookecing
Dast Conlescing
w83 Decoeissog
Dept DocONNs
(srel))
(WE..... WEl
Tran(16, Rman)
Figure 2: A 2-level V-cycle training process. M, with parameters of [W] s WL, is the original model train. M2 with parameters of [W? W is a smaller model which coalesced from M1 by coalescing intra- and inter-layer neighboring nodes. F and F are used to decrease the input and output dimension of parameter W in layer The depth coalescing matrix is decomposed into an L x L2 matrix and an identity matrix via Kronecker product. We first train the Mj model for E. epochs to initialize model parameters. Then train the coalesced M2 model, which converges faster. After that, we de-coalesce the parameters of M2 to the original size and interpolate them with the parameters of M, before coalescing. Finally, we continue train the interpolated M1 model.
and training performance. (3) We conduct extensive experiments on two transformer based language model, BERT and GPT, well vision model, DeïT. The results show that the proposed frame- work enables considerable speed-up in training large scale models. Compared with the traditional training methods, our multi-level framework reduces the training cost by 20% on BERT-Bas and GPT-Base, 51 1,6% on Bert-Large, and 27.1% 1% on DeiT-B.
RELATED WORK
2.1 EFFICIENT PRE- TRAINING
Various efficient pre-training techniques have been developed. Most of these, c.g., ELECTRA CCarr al. 2020)) large batch optimization (You et al.) 2020). layer dropping (Zhang & He 2020), token dropping (Hou et al. 2022), and weight sharing (Yang et al.] 202T), are orthogonal to our work.
KI QQin al. 2022) distills knowledge from small models into larger models for efficient pre- training. A few recent works (Wen et al. 2020 Haber et al. 2018; Chang et al. 2018 Gaedke- Merzhäuser et al., 20200 Cyr et al. 2019) Gong et al. 2019: Yang et al. (020) pre-train a large scale model by progressively stacking layers. Meanwhile, Net2Net (Chen et al., 2016) increases
network complexity in the width direction by copying neurons and also expands depth by adding identity layers preserve the learned features. bert2BERT (Chen al.) 2022) extends Net2Net to the transformer architecture. In Network Expansion (Ding et al., 2023) the authors propose to impose the orthogonality across the expanded filters in CNN and utilize the exponential moving averaged model to expand the ViT along the depth direction. In CompoundGrow (Gu et al., 2021), the authors prove it beneficial to expand the network in width, depth, and tokens together. In the staged training approach (Shen et al., 2022). Shen et al. focus on growth operators scheduling to keep the training dynamics. LiGO (Wang et al. (023) learns linear mapping matrix between the small and the large model through stochastic gradient descent (SGD). Similar approaches (Wu et al.) 2020 Wang et al., 202T) are also proposed accelerate the training other architectures. Most of the above approaches start from a small model and progressively increase the model size by reusing parameters. Therefore, they should be regarded as special cases of the multi-level framework with only de-coalescing operation.
2.2 MULTIGRID
Our work is inspired by the multigrid method (Trottenberg et al. 2000), which is an efficient nu- merical procedure to solve linear equations and partial differential equations with multiple levels. The method has received successful applications in many application domains such as finite element
3

---

Published as a conference paper at ICLR 2024
method (Zhu & Cangellaris 2006), eigenvalue computation (Knyazev & Neymeyr 2003). and graph partitioning Sabou-Rjeill & Karypis, 2006). Of course, a direct porting of the multigrid method to neural network training, which is under different set of constraints, is not feasible. Borrowing the spirit of the multigrid solvers, we propose a systematic investigation on accelerating the training of large models with multi-level framework.
3 METHODOLOGY
We aim to accelerate the training large transformer models by leveraging the fast convergence be- haviors of smaller model derived from the larger ones through coarsening operation. We propose three operators, Coalescing, De-coalescing, and Interpolation to develop a complete working flow. These operators merge parameters, transform the parameters back to the original model, and correct the parameters the original model with de-coalescing parameters, respectively. The design of the operators enable the framework to reduce most emrors in small model at less total training cost. In the following sections, we first introduce the Coalescing, De-coalescing and Interpolation operators and then elaborate the V-cycle training process built upon them.
For simplicity, we assume that all layers are feed forward layers without bias and have the same input and output dimensions. We give the generalization for other components of transformer in Appendix A We use M2, k = 1, ...K to refer to the model at level k. Mj is the original model and the level number increases by one when we coarsen the model
smaller one. We W E "Ru to represent the parameter of layer / in model Mg. d and di refer to the input and output dimensions. sumleweel (A) denotes the summation of each resulting a sum vector. We adopt diag(a) I) to convert a vector a into a diagonal matrix.
column of matrix A,
3.1 COALESCING
As the first step, we coalesce the model in width direction, while keeping the layered structure untouched. Then we coalesce the model in the depth direction to get smaller model coarsened in both width and depth directions.
Width Coalescing
Wh E R. is the parameter of layer the model Mg. We transform
it into Uk++1 R at+1 with width coalescing matrices p Id E Rdt. att and out
+ 1
Id ouer oue for in and out dimensions as follows.
U kk++1 w pk+1d
(I)
out
To ensure the output layer is properly received by layer I + and stabilize the output of each layer, we define + in. as:
F k++11+1 Fovt
(angg1/ (F ( p
in
where diag(1/sumeol out The width coalescing matrix Pout matrix we used is detailed in Section
k+ F. }) is used to normalize the product of E and 1.1T Fk+1, is arbitrary as long the matrix has full column rank. The
Depth Coalescing After width coalescing, we have pose them into W 1,2, Ix+1- La and L&-1 layer numbers of the model Mr and ME+1. Lb+1 < La always holds. To reduce the depth, we use a depth coalescing matrix RR+1 E TR bets dftt) shtt to map those parameters follows.
1, Le, and need to trans-
Wk+1 Wh+1 Wh+1| [U]+] Uk+1
Uh+1 | R
4

---

Published as a conference paper at ICLR 2024
As see, the dimension of R can be large. Therefore, decompose it into L3 Lk+1 matrix with elements of t 1.... L 1..., LK-1 and an identity matrix via Kronecker
product as in Ling et al. 2023):
gh+t
gk+1
1.b111
R
K I
(4)
go+1 Lx.1
po+
L2.
where the @ is the Kronecker product that multiplies (g with the identity matrix respectively.
In general, we coalesce the parameters M₁ to those of a smaller model Ma-1 as follows.
F W Fktli
pk+1 1,b2+1
1,1
I
(5)
Wh+t
I Wh 1 Li
pk+1 1x.
pk+1 baelan
3.2 DE-COALESCING
De-coalescing is the inverse operator of Coalescing. To map the parameters back to the original model, first do depth de-coalescing model M++1 to increase the depth, and then perform width de-coalescing to increase the width based on the result of depth de-coalescing.
Depth De-coalescing Given model M which has parameters W 18 Bout ,
1,2, Lk+1. We could use the depth de- de-coalescing matrix Gk+1 E R df to transform them into U Rdd 1,kkl1
1,2, ..., Lk.
U Uk+1
U (W W
Wh+1 Gk+1 Le41
(6)
As in the previous section, we decompose the depth de-coalescing matrix G into an L841 x La matrix with elements of Cij,, .... Lx+1.J 1..., Lg and an identity matrix I.
ch
k+1
S1
Gk+l
3
Consider that we are doing the depth de-coalescing on the result of depth coalescing as follows.
U U
U (U++t U
Jutllll I Gk++1
(8)
To keep the value of parameters in each layer stable after coalescing and de-coalescing i.e., the column sum + Gk+1 equals to we have the G as:
G ck+1 Rk+11 1 "iag(1 R 1 R=+1?))
(9)
where the diag(1/ (1suumeal +1 R*+1T)) performs normalization of the parameters.
Width De-coalescing After depth de-coaleseing, we have U
gd k+1 gattt We use width
de-coalescing matrices T and T to transform them into W E Rdh ab for in and out dimensions, respectively.
Recalling the width coalescing, have the following equation after coalescing and de-coalescing.

---

Published as a conference paper at ICLR 2024
W Tktlll U - p out
(10)
T F wh W p out
out
Similar to the definition of depth de-coalescing, we define Th+Ld and Thttd as follows.
in
T Tk+ diag(1 Saaag(1/summagg //sumrow F F p++111 in
in
in
(11)
Tk+11 = F I diag(1/sumcoll ( p p*+1,"))
out
In summary, we de-coalesce the model M4+1 to a larger model My as follows.
W
T W
1 out
R+1
in
(12)
W
T
1.1 Wh++1 Li+1
T 1,
k+1
S
L
3.3 INTERPOLATION
Since we define the de-coalescing matrix as the normalized transposition of the coalescing matrix, it's easy find that the parameter W obtained after de-coalescing far from full rank. In the worst case, half of the elements of W will be identical the respective ones in the other part (Chen et al. 2016). Such value distribution significantly limits the capability of the model. We call the symmetry of neurons.
Instead of injecting noise (Chen et al. 2016) or parameters from higher layers (Chen et al. 2022) to alleviate this problem, we propose more general operator, interpolation, which (I) transfers knowl- edge back to the larger model M₁ from the ne-ccolesced smaller model, (2) avoids similar adjacent parameters incurred by de-coalescing, and (3) improves the convergence rate further. We find that the interpolation operation allows the acceleration scales with the times of model enlargement. On the contrary, a higher number of mapping the previous literature leads to a worsened convergence speed. This empirical observation is elaborated in Appendix B
Specifically, after training the smaller model M coalesced from Mg. we first de-coalesce M&+1 to the size of Ms and get model MR.de -coee-cooss- Then we merge the M2 and Mg.de coolesced the control a hyperparameter a E (0, 1).
M.g.e-cooleecce under
M4 = (1 - M) + a.Mgdd-
(13)
In addition to alleviate the problem of symmetric neurons, a also determines how much updated knowledge incorporated into larger model from smaller one. The effect of discussed in Appendix In most cases, a = 0.25 suffices to give satisfying results.
3.4 V-CYCLE TRAINING PROCESS
With the Coalescing, De-coalescing, and Interpolation operators, we can build V-cycle training process inpired by the V-cycle in multigrid method (Trottenberg et al.) 2000). We leave other training processes like W-cycle and FMG in multigrid method for future work.
Algorithm I expounds the V-cycle training process. At first, it progressively coalesces the model reduce the model complexity and then trains the smallest model for Esmall epochs. The hyper- parameter Esmall is used to allow early stop of training smaller models at the end of the fast con- vergence phase. Next, the smaller model is de-coalesced and interpolated to set the parameters of larger model. Finally, the model size reaches its original size and Mi will be further trained until convergence. addition, we train the models at each level for E, epochs before coalescing for parameters initialization. E, is set to the number of warm-up steps. We always stop the training of smaller models for one half of the number of steps for the large model. We further discuss the robustness of hyper-parameters in Appendix D

---

Published as a conference paper at ICLR 2024
Algorithm 1:
V-cycle Training Process
Input untrained model M1, dataset D, number of levels K, interpolation factor a, epochs Esmall for training smaller models, epochs Eg for initializing.
Output: the pre-trained model M
for / = + K do
2
update M2 on D for Eg epochs
M1+1 (oalescing(M)) // decrease model size, more details in Algorithm 2
end
for
1
K + do
update M1 for Femall epochs
7
= De-coalescing(MM) // recover size, more details in Algorithm MM-1 Interpolation( M -oe-1, a) // more details in Algorithm 4
end
10 update M on D until convergence
4 EXPERIMENT
4.1 EXPERIMENTAL SETUP
Architectures and Datasets We exercise our multi-level framework two language models, BERT (Devlin et al. 2019) and GPT (Radford et al. 2018), as well as a vision model, DeiTT (Touvron al. [) We use English Wikipedia and Books (Zhu et al. 2015) pre-training data for BERT and GPT, while DeiT is trained with ImageNet (Deng et al.) (2009). For evaluation, we test the pre-trained BERT on the GLUE benchmark (Wang et al. 2019). We evaluate the pre-trained GPT on LAMBADA (Paperno et al. 2016). PTB. WikiText-2 and WikiText103 under zero-shot setting without fine-tuning on the training set. CIFARIO (Krizhevsky et al. 2009). CIFAR100 (Krizhevsky
et al. 2009), Flowers 102 (Nilsback & Zisserman 2008), and Stanford- (Krause et al. 2013) are
adopted to test the downstream performance of DeiT.
Baselines We evaluate our framework by comparing it with five baselines. The first baseline is a the model trained from scratch. The second is StackBERT (Gong et al. 2019), in which the depth is increased by progressively stacking layers. 3 The third bascline is 2ert (Chen et al. 2022 that extends the width with advanced knowledge initialization. The fourth one is LiGO (Wang al. 2023) that learns to expand the network both in depth and width. The fifth method Network Ex- pansiorDing et al. (023) that expands the model with the exponential moving averaged parameters. We choose KI (Qin al. (002) that distills knowledge from a small model into a larger model as the fifth baseline. Specifically, bert2BERT, LiGO, and KI all assume that there are well-trained small models and do not consider the training cost of smaller models. To be fair, we take into account the training cost of smaller models for BerT2BERT, LiGO and KI when comparing with them.
Implementation Details During pre-training, we train the BERT-Base with the following settings, 40 training epochs, 10K warm-up steps, peak learning rate of Ic-4, and a batch size of 512. We remove the next sentence prediction task (Liu et al. 2019) and use fixed sequence length of 128. We use the same settings for BERT-Large. In the case of GPT-Base, we use 20 training epochs, 10K warm-up steps, a peak learning rate of le-4, and a batch size of 256. We train the DeiT-B with a peak learning rate of le-3, 300 training epochs and batch size of 1024.
For the evaluation of BERT-Base and BERT-Large, we take the tasks from the GLUE benchmark. We choose batch size of 32, learning rate from (5e-6, le-5, 2e-5, 3e-5, 5e-5), and train the model with 5 epochs on all GLUE fine-tuning tasks. We run each training process for three times with random seeds for GLUE. We evaluate the GPT-Basc on LAMBADA (Paperno et al. 2016), PTB, Wik and Wik under zero-shot setting, i.e., without fine-tuning. For DeiT-B, fine-tune the pre- pre-trained model for 1000 epochs with batch size of 768 and learning rate of 0.01 under SGD optimizer and the same data augmentation in the training.
We implemented MSLT Yang et al. 2020) based training procedure, which is similar to StackBERT. We find that it's hard for MSLT to reach sufficient, performance when trained from scratch as observed by bert2BER (Chen al. 2022). So we only include the results of StRcKBERT.
7

---

Published as a conference paper at ICLR 2024
BORT-3as LenZBENT
GFT 3300
5 Wed
2.8
2isel
4.2
2.8
101
4.0
LO35
1,600
3.8
2.08
2.4
2.2
9.5
9.5
10.5
3.6
145
24
19
(7%
MLM
2.2
5.4
6.5
7.7
2.0
1.8
3.0
0.0
2.5
5.0
7.5
10.0
3
FLOPs (1e18)
FLOPs (1e18)
FLOPs (1e19)
BERT-Base
(b) GPT-Base
BERT-Large
Figure 3: Results on BERT-Base, GPT-Base and BERT-Large. (a-c) show loss curves of BERT-Base, GPT-Base and BERT-Large pre-training. The dashed lines are the final results of models training from scratch. For BERT-Base and GPT-Base, our approach saving about 20% computational costs. For BERT-Large, we save 37.4% training cost with 2-1evel training process and 51.6% with 3-level.
In the multi-level training, we coalesce the model to reduce width and depth by half. We define the width coalescing matrix as Fk+1,1 w,out = [I/ I/2/T and elements of depth coalescing matrix as pk+1 2, 0.5, 1.., Ls++1 Intuitively, they coalesce two neighboring neurons and merge two adjacent layers. The de-coalescing matrices can be derived from Eq. and Eq. [1] We use a = 0.25 for GPT and DeiT, a = 0.5 for BERT. Unless otherwise noted, we train the model with two-level mapping. The length of initializing stages for BERT/GPT and DeiT are 10K steps and five epochs. We always stop the training of the smaller model halfway through the training, For example, the BERT training steps 300K and thus Esmall will be 150K steps.
Our experiments are conducted on NVIDIA A100 GPUs. As the mixed precision training (Mi- cikevicius al. 2018) and DeepSpeed framework (Rajbhandari et al. 2020) are orthogonal to method, we use both for the pre-training of BERT and GPT.
4.2 EXPERIMENTAL RESULTS
Table I: GLUE benchmark results between our approach and baselines. BERT-Base means the model trained from scratch. All results GLUE benchmark are run in three times with different seeds and numbers in parentheses are the standard deviation across the runs.
Method
Saving (FI.0Ps)
Saving (Walltime)
SST-2
MNLI
MRPC
CoLA
QNLI
QQP
STS-B
Avg
(Ace)
(Acc)
(Mee)
(Ace)
(Acc)
(Acc)
BERT-Base
0%
0%
89.5(0.3)
(7.8(22)
18.10.35
52.90.33
85.1(0.5)
19.(011)
83.5(0.4)
79.70.2)
StackBERT
15.2%
8.2%
89 69.4())4)
77.500.2)
71.7(0.4)
52.100.5)
85.8(0.3)
89.3(..1)
82.3(0.4)
79.3(0. 1)
ben2BERT
89.5(0.5)
77.7(0.1)
76.5(0:9)
49.510.4)
84.8(0.4)
88.30.1)
81.0(0.4)
78.2(0.3)
LiGO
17.4%
7.9%
89 (0.3))33)
79.1(0.2)
77.4(211
53.7(0.3)
85.9(0.1) 1)
89.0(0.1)
83.6(0.3)
79.710.3 ()
Network Expansion
14.8%
8.1%
89.6(0.3)
77.610.2)
70.1)
53.4(0.4)
85.5(0.3)
89.6(0.1)
82.7(0:4)
79.4(0.
-6.9%
-25.9%
99.9(0.2)
78.3(0.2)
76.700.4)
55.100.6)
85.610.3)
89.6(0.1)
84.7(0.3
79.9(0.1
Ourx
19.0%
10.8%
90.1(0.1)
78.1)0.2)
79.9(0.7)
52.2(0.3)
85.1
89.2(0.1)
84.2(0.4)
79.8(0.1)
Table 2: Zero-shot results of GPT-Base. "w/o FT" means that evaluate the pre-trained model
without fine-tuning, Our approach achieve similar and even better perplexities than the GPT-Base.
Saving (FLOPs)
Saving (Walltime)
LAMBADA (w/o FT)
PTB
WikiText-2 (w/o FT)
WikiText103 (w/o FT)
Method
(w/o FT)
GPT-Base
0%
0%
54.5
146.3
49.8
50.2
StackBERT
9.5%
8.4%
53.3
140.6
46.5
46.9
bert2BERT
11.5%
8.3%
53.9
147.1
48.8
49.4
LiGO
14.1%
6.9%
54.0
139.7
50.1
50.5
Network Expansion
15,2%
12.2%
54.7
143.7
50.7
51.2
Ours
24.1%
16.5%
53.2
142.5
47.2
47.5
BERT-Base We compare the results our method and baselines in Figure 3a and Table []] The comparison shows that multi-level framework save 19.0% FLOPS and 10.8% walltime for
8

---

Published as a conference paper at ICLR 2024
BERT-Base pre-training, while achieving similar down-stream tasks performance training from scratch. In addition, we find that bert2BERT and KI cannot deliver reduced walltime in our settings.
GPT-Base Figure 3b and Table show the results of GPT-Base model. The results demonstrate that our multi-level framework saves 24.1 % FLOPs and 16.5% walltime on GPT-Base pre-training. We observe that our zero-shot results are slightly better than training from seratch.
DeïT-B The results of pre-training DeïT-B are illustrated in Table 3 The multi-level framework saves 27.1% FLOPs and 24.3% walltime in pre-training DeïT-B on ImageNet. Results on downstream tasks prove that our multi-level framework does not impair the model's generaliza- tion capability.
Table 3: The transfer learning of DeiT-B. DeiT-B pre-trained with multi-level approach shows a similar level of perfor- mance as DeiT-B pre-trained from scratch.
Methed
Sorvng (FLOPS)
Saving (Walltime)
ImageNNtt (Top 1.3c0
CIPARI6
(hoc)
CIFAR100
(Acc)
Flowers
(Aec)
Cars (Acc)
Deil'n
0S
51.1%
19:15
062415
07.355
92.1%
SuckBERT
$1.2%
00.35
97.65
LGGO
25.49L
12.0%
51.7%
Netwaric Pepursiio
25.084
12.49
61.4%
40.00
00.75
07.40 454
95.19
27.88
34.98
K1.5%
99.19
90.3%
97.88
92.14
More Levels on BERT-Large In our current setting, the number of parameters reduced by eight-fold in one coalescing operation. Although framework supports an arbitrary number of levels, we observe that there little gain on three or more levels in pre- pre-training BERT- Base because the level 3 model has only 1.72M parameters, which is too small to learn from the large dataset. Therefore, we pre-train BERT-Large with two and three levels to evaluate the effectiveness of the number of levels. Figure 3c and Table show the results of the multi-level framework with two and three levels on LaRT-Large. When compared with BERT-Base, our approach achieves higher reduction in FLOPs and walltime on BERT-Large with levels. On the other hand, an increased number of levels results in greater computational savings, indicating our method's 's potential.
Table 4: Downstream tasks performence of BERT-Large with more levels. Results of BERT-Large indicate that more levels does not lead to performance deterioration while saving more training cost.
Saving (FLOPs)
Saving (Walltime)
SST-2
MNLI
MRPC
CoLA
QNLI
QQP
STS-B
Level
Avg
(Acc)
(Acc)
(Ace)
(Mee)
(Acc)
(Acc)
(Ace)
0%
80.6(0.2)
79.2 (0.2)
(1.0
86.7 (0.4)
(0.0 (0.1)
80. (0.2)
2
37.4%
32.0%
90.9 (0.3)
79.8 (0.3)
78.1 (0.4)
54.8 (0.1)
86.9 (0.2)
89.7 (0.2)
83.200.4
80.8 (0.1)
3
51.6%
41.9%
90.5 (0.3)
79.8 (0.4)
82.6 (0.7)
56.3 (0.4)
86.9 (0.1)
(0.1)
84.5 (0.3)
(0.1)
DISCUSSION
The proposed approach does not need to re-compile the setup of parallel computing LLMs when transiting from small model to a larger one as the model architecture and the underlying training environment do not change. Compared with the traditional single level training, the only overhead is incurred by resuming training of the larger model. Here we need to load parameters from storage. In our experiment on BERT-Large, resuming can be done within one minute. This overhead has been taken into account in our results of the walltime saving. For LLaMA-65B, the loading process can be finished in less than 5 minutes on SSD. The estimation is detailed in Appendix C In summary, the overhead of our proposed V-cycle training is negligible.
6 CONCLUSION
This paper proposes an efficient multi-level framework for reducing the training time of transformer- based models. Our framework built on three basic operators, Coalescing, De-coalescing, and Interpolation, which can be combined to build up a V-cycle training process. The multi-level training offers great potential to deliver a reduced computation time by balancing the fast convergence of smaller models and the high expressiveness of large models. Our experimental results justify the effectiveness of the multi-level training on BERT, GPT, and DeiT models. In addition, our extensive experiments show that our methodology has the potential for saving the computational cost of larger models with more levels. In the future work, we will apply the multi-level framework to the pre- training of large scale models with over 100B parameters.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to thank Wei Chen, Ke Lin, Bowen Zhang, Yuru Zhang and Dr. Jianguang Lou for valuable discussions on early drafts of this paper. We are grateful the NLP group of Didi Global Inc. We thank all anonymous reviewers for their insightful feedback.
REFERENCES
Amine Abou-Rjeil Rjeili and George Karypis. Multilevel algorithms for partitioning power-law graphs. In 20th International Parallel and Distributed Processing Symposium (IPDPS 2006), Proceedings, 25-29 April 2006, Rhodes Island, Greece. IEEE, 2006. doi: 10. 1.11100//PP.20000 1639360. URL
https://dot .ww.ii.oggiioo.... /1rgg/10.0 1109/ 2006. 1639360
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford. Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL haw.e///06607111100aaaa144466660 Abst html
Bo Chang. Lili Meng, Eldad Haber, Frederick Tung. and David Begert. Multi-level resid- ual networks from dynamical systems view. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018, BRLLhtpp::/// hhttss://openreveewwree net/ ffoum?io P-JS-OgRe
Cheng Chen, Yichun Yin, Lifeng Shang. Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang. Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 2134-2148, Association for Computational Linguis- 1/653//1/2022.allloong 2022. aalalcclooooo 151
tics, 2022. doi:
hatppa//ool.orr/1111866333
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016. URL http:
org/ abs 151...111....01 05641
Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. htthstt::: /ttp::::////eeeveeeeeee net http//oormmidddeee (e1xMMM1BBBB BBVB
Eric C. Cyr, Stefanie Günther, and Jacob B. Schroder. Multilevel initialization for layer-parallel deep neural network training. CoRR, abs/1912.08974, 2019. URL http:://axii..///
abs/1912.082 01917.08974
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CYPR 2009), 20-25 June 2009, Miami, Florida, USA, pp. 248-255. IEEE Computer Society, 2009. doi: 10. 1110///V..2209992266666
URI.ttps: wwt.hhttp///....... //o/i doi org/10.
11C9/CVPR, 2009. 5206848
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and
10

---

Published as a conference paper at ICLR 2024
Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT
2019, Minneapolis, MN, USA. June 2-7, 2019, Volume I (Long and Short Papers), pp. 4171- 4186. Association for Computational L Linguistics, 2019. doi: 10. 10.18663//1/1119-422. URL
https://doi: http///tooor//.////// wtt..oo......./..1/..... 1ww11//1//111--11111
Ning Ding, Yehui Tang, Kai Han, Chao Xu, and Yunhe Wang. Network expansion for practical training acceleration. In IEEE/CVF Conference on Computer Vision and Pattern Recognition,
CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 20269-20279 IEEE, 2023. doi: 10. URL https: /wi./iiiooo/ooo org/ 10. 1109/ (2w.1///VVR5229..222 2023.
1109/CVPR522992003..111
01941
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URUhttpp::/ppeenreiiieee formmo id-YicbF dNTTy
Lisa Gaedke-Merzhäuse Merzhäuser, Alena Kopanicáková, and Rolf Krause. Multilevel minimization for deep residual networks. CoRR, abs/2004.06196, 2020. URL
abxzg//4b0
061 96
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Yan Liu. Efficient training of BERT by progressively stacking. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, ICMI.2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp- 2337-2346. PMLR, 2019. URL html
mlr wwwppppp.//pp..ptettppppp http/////////g/////////
Ian J. Goodfellow and Oriol Vinyals. Qualitatively characterizing neural network optimization prob- lems. In Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Rep- resentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
abs/////////// 1412. 2..22.. 6544
Xiaotao Gu, Liyunn Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive BERT training. In Kristina Toutanova, Anna Rumshisky. Luke Zettle- moyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.). Proceedings of the 2021 Conference of the North American Chapter of the Association for Computationall Linguistics: Human Language Technologies, NAACL-HLLT 2021, Online, June 6-11, 2021, pp. 5174-5180. Association for Computational Linguistics, 2021. doi: 10.18653/v 2/22222222222222 waw.cl-mmminnne. URL https:://ddat org http://6...rrr100.00165/30/00220
naacl ma 406
Eldad Haber, Lars Ruthotto, Elliot Holtham, and Scong-Hwan Jun. Learning across scales multi- scale methods for convolution neural networks. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the -terty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3142-311112 314.. AAAI Press, 2018 URL https: /ttp://www... www. wwa..arraaaaaaaaaaaa..a..aa..... org/
ocs/indes ndex. php/ AAAAAAAAAAAAA /AAAAAAAAABA.. http:///AAAAAA//A///aAaaaaaa http:/pppp//iiii
6580
Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, and Denny Zhou. Token dropping for efficient BERT pretraining. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3774-3784, Association for Computational Linguisties, 2022. doi: 10.18653/v 1/2022. acl-long. 262. URL https::..... /ttp:///...oooo htt.://.rrr11..0 18 85336633///0 /0w.////2222/22 ac acl- long. 262
Andrew V Knyazev and Klaus Neymeyr. Efficient solution of symmetric eigenvalue problems using multigrid preconditioners in the locally optimal block conjugate gradient method. Electron. Trans. Numer. Anal, 15:38-55, 2003.
11

---

Published as a conference paper at ICLR 2024
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, ICCV Workshops 2013, Sydney, Australia. December 1-8. 2013, pp. 554 561. IEEE Computer Society, 2013. doi: 10. D0.100//CCV...213.7... URL https: /ttp::////oo..rg/////./ 1109/ ICCV// ACCCV.2213.. 777
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. http://t:/ http://aaxxvv..r/.aae/ abs////b///// 907 11 692
Paulius Micikevicius, Sharan Narang. Jonah Alben, Gregory F. Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Leurning Representations, ICLR 2018, Vancouver, BC. Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
net dow.://oooonnnde1110ol..-e
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large num- ber of classes. In Sixth Indian Conference on Computer Vision. Graphics & Image Processing, ICVGIP 2008, Bhubaneswar. India, 16-19 December 2008, pp. 722-729, IEEE Computer Soci- ety, 2008. doi: 10.1 N0.1109/CCVIPP2200..88 URL https: doi http:////0.....1000111111///1//0 2008. 47
OpenAl. GPT-4 technical report. CoRR, abs/2333.08774 2023. doi: 10445550/axxv.2200.877744 URL
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics, 2016. doi: 10. 10.18653/v1/ p16-1144. UtL /ooi org/10. 18653/v1/pppttt 1144
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Knowledge inheritance for pre-trained language models. In Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 3921-3937. Association for Computational Linguistics, 2022. doi: 10.18653/v1/202.. maicI-main.288. URL wwtp:httt///eet http//o....oogg// 100 16653/111111133/12210222 naacl - ma in. 288
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-trainnn pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations
toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T.
Kramer (eds.), Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020, pp. 20. IEEE/ACM, 2020. doi: 10.1 C,.1099/C4414445200000000 7/doi org/10.
URL https:
10. 11111110111111 20200.0024
2020.
Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy. Staged training for transformer language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning. ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Pro- ceedings of Machine Learning Research, pp. 19893-19908. PMLR, 2022. URL https: 7/proceed ngs. ml www.spresseeem / v1 62/ shen22 html
12

---

Published as a conference paper at ICLR 2024
K. Stuben. A review of algebraic multigrid. JOURNAL OF COMPUTATIONAL AND APPLIED MATHEMATICS, 128(1-2, SI):281-309 MAR 1 2001. ISSN 0377-0427. doi: 10.1016/
50577-0027(00)005666..
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine ngs. .llr.
Learning Research, pp. 10347-10357. PMIR, 2021. URL html
szess/v139/towvrorte
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient founda- tion language models. CoRR, abs/2302. 13971, 2023. doi: 1048855WaxXiv.230..1977.1 URL
btttps://doi.orgo
48550/ arwiv. 2302 13971
UIrich Trottenberg, Cornelius W Oosterlee, and Anton Schuller. Multigrid. Elsevier, 2000.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL httpps://ppereeeeeeee
net/ forum d=
rJ4 km5t.77
Jiachun Wang, Fajie Yuan, Jian Chen, Qingyao Wu, Min Yang, Yang Sun, and Guoxiao Zhang. Stackrec: Efficient training of very deep sequential recommender models by iterative stacking. In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai (eds.),
SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pp. 357-366. ACM, 2021. doi: 346.84448383333890
10.1145/3404835.346222
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogério Feris, David Daniel Cox, Zhangyang Wang. and Yoon Kim. Learning to grow pretrained models for efficient transformer training. In The Eleventh International Conference on Learn- ing Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL net /ttt/pddffa http://dEE....DDDSS5.....
httpss://cperees/
Wei Wen, Feng Yan, Yiran Chen, and Hai Li. Autogrow: Automatic layer growing in deep convolu- tional networks. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash (eds.), KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 841, ACM, 2020. doi: 10. 10.14553394886.400133 URL
https: //doi
orgg 10. 00111/3339486666 311./448..3443310 3403126
Chao-Yus Yuan Wu, Ross B. Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krähenbühl. A multigrid method for efficiently training video models. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp.
159. Computer Vision Foundation / IEEE, 2020. doi: (0.1109/CVPP422202200000000
URL https: https////opeaacccssooo www.fe..cccccccccco.coooc.. www.to/twwoooooooooooooooooeoooe Betpateet___________ee 2e1e32222000 Mul
111220///mm/// A
Met.odehhoooooo for hitaaaaaaaoooooooffffeeeeeeeeeee
Video Mode _s_CVPR______
2020 0 paper html
Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li. Ru He, and Jingqiao Zhang. Progres- sively stacking 2.0: A multi-stage layerwise training method for BERT training speedup. CORR, abs/2011.13635, 2020. Utt/htttss//aaxxiv.ooo
abs 2011 13635
Shuo Yang, Le Hou, Xiaodan Song, Qiang Liu, and Denny Zhou. Speeding up deep model training by sharing weights and then unsharing. CoRR, abs/21 10.03848, 2021. URL https: :tapxii.//axxv.....
org/ abc//b//220 / 03848
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiao- dan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep
13

---

Published as a conference paper at ICLR 2024
learning: Training BERT in 76 minutes. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https: /ttp://e///ppeeeveieee net Fot.//oommmmmiis4emaaa
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland. September 6-12, 2014, Proceedings, Part
volume 8689 of Lecture Notes in Computer Science, pp. 818 833, Springer, 2014. doi: 10.1007/ 978-3-319 10590-1 1) 53. URL. https: /doi wtg/10///00 300
53
Minjia Zhang and Yuxiong He. Accelerating training of transformer- language models with progressive layer dropping. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems
33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, De- cember 6-12, 2020, virtual, 2020. URL httpttts////pooceediiiii .ew.eeeiiiinnnnnnni............ 2020
newrips cc/paper/ Abstract .htm
a wtw.aa/0000dofcecceee/aaaaaaaaa.
Yu Zhu and Andreas C Cangellaris. Multigrid finite element methods for electromagnetic field modeling, volume 28. John Wiley Sons, 2006.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pp. 19-27. IEEE Computer Society, 2015. doi: 10.1 10.100/CCC..005..111. UtLhhtts:://daa /ttp://::///o..og///ooooo. ICVV 2015.11
14

---

Published as a conference paper at ICLR 2024
A COALESCING AND DE-COALESCING FOR TRANSFORMER
In the Transformer architecture, certain constraints are required to ensure alignment between the input and output across the residual connection, normalization layers. and attention layers. Specifi- cally, embeddings and feed-forward networks (FFNs) only need to adhere to Eq.
Residual Connection Consider layer j follows layer i and has residual connection, denoted as h2 f(ay) + h4, where aj and hy represent the outputs of layer j before and after applying the activation function f. respectively. It is requisite that F out F ensuring the output alignment of layer j with layer i.
Normalization Layer Given that the normalization layer following layer i performs affine transformation on the normalized result, essential that the width coalescing matrix Fnorm for parameter W norm is equivalent Fout-
Attention Layer The attention layer employs matrices WQ, w and WV to derive the queries Q. keys K and values V. respectively. The output of attention layer can be formalized follows:
QKT T IV de
Attention QQ,K, V) softmaat
(14)
where the d2 is the dimension of queries and keys. Given that matrices Q and K are subject to matrix multiplication, we need to limit F = F Furthermore, as required by Eq. 2 it follows that F FK out out FVV. In the case of multi-head attention layer, these requirements must be satisfied for each head.
B WHY NOT INCREASE MODEL SIZE MONOTONICALLY
In this work, we do not increase the model size monotonically as pro- posed in the previous literature, e.g. LiGGWang et al. (2023). The rea- is that the model enlarged from smaller one in the existing works will have low-rank weight matrices and thus limit the model's 's capability. The more times the model mapped, the more severe the limitation becomes. Figure show the training curves of two GPT-Large models. They are trained by following 1) GPT-Small GPT-Basc - GPT-Large and 2) GPT- Base - GPT-Large. The smaller model is mapped to the larger ing LiGO. For faimess, we enlarge GPT-Bases with close level of per- formance in 1) and 2) and use the same training settings for the train- ing of GaT-Large. As shown in the figure, the model mapped twice con- verges significantly slower than the model mapped only once. confirms that it is not beneficial to repeatedly enlarge the model in monotonic fashion as suggested in the literature. The V-cycle training proposed in this work proves to be advantageous. In Figure 3d we show that the V-cycle training process with three-level mapping converges faster than the model with two-level mapping. Therefore, we interpolate the de-coalesced model into the original one instead of directly training the enlarged model.
GPT-Large (by GPT-Base GPT-Large)
3.2
GPT-Large (by GPT-Small GPT-8 GPT-Large)
3.1
LosS
3.0
2.9
2.8
20
100
120
Step (k)
Figure 4: Pre-training GPT-Large mapped once and twice with LiGO. Results show that GPT-Large mapped twice converges significantly slower than GPT-Large mapped once. It confirms that it is not beneficial monotonically increase the model size as proposed previous literature.
15

---

Published as a conference paper at ICLR 2024
C DEPLOYMENT OVERHEAD
After interpolating the de-coalesced model into the original one, we need to resume training of the larger model with updated parameters. Since the model architecture and underlying training environment does not change, there is no need to recompile the model, i.., the setup of parallel computing do not need be configured again from scratch. The time spent on this step mainly comes from retrieving parameters from storage. Since we re-init the optimizer's parameters when training the interpolated model, we only need to load model parameters from the storage. When pre-training LLMs with fp16, 24 bytes are required to save model with & parameters. For the LLaMA-65B, the amount is around 130GB. A typical HDD will deliver a read speed of 80- 160MB/s and a standard SSD read data at rate of 500 to 550MB/s. Therefore, parameters within half an hour from HDD and five minutes from SSD. As the training LLaMA- 65B takes approximately 21 days, the resuming cost negligible.
can load the model
D EFFECT OF HYPER-PARAMETERS
Table 5: Effect of Hyper-parameters in algorithm. Unlisted values taken be identical to those the BERT-Base. E, is the training steps before coalescing. Esmail is the training steps for smaller models. The de-coalesced model would be interpolated into the original model with ratio a. A full training cycle of BERT-Base is 300K steps. L6-H6 indicates that the model has layers and attention heads, and the same applies the others.
E.
Esmall (Steps)
Coalesced Model Size
Saving (FLOPs)
Saving (Walltime)
(Steps)
BERT-BBae
10K
150K
0.5
L6-H6
19.0%
10.8%
50K
9.2%
1.0%
(A)
100K
0.8%
-7.4%
50K
14.5%
11.8%
100K
17.7% %
6.6%
(B)
200K
17.8%
4.4%
300K
12. %
0.1%
0.05
-2.3%
10.4%
0.25
18.1%
9.8%
(C)
0.75
13.0%
4.7%
1.0
-15.8%
-18.5%
L4-H4
12.1%
5.7%
(D)
L8-H8
13.4%
7.1%
L10-H10
7.4%
-2.6%
Effeet of E. We demonstrate the effect of hyper-parameter E, in Table (A), Results show that small E,, is enough. An excessively large E, would reduce the effect of acceleration because this time the large model has passed the stage of coarse coarse-grained learning, and the small model no longer plays role in fast convergence.
Effect of Esmall To study the robustness of the proposed framework, we change the value of the hyper-parameter Exmalt from 50K to 300K. The results in Table rows (B) demonstrate that Esmatt is robust over one half the full cycle. There is no need for a large Esmatt because the smaller models have already passed the fast convergence stage and do facilitate further acceleration of the training of larger models.
Effeet of a To investigate the effect of the interpolation operator, we tune the from 0.05 to 1.0 for BERT-Base pre-training, Results in Table 5 rows (C) indicate that there is no saving when interpolation operation is removed i.c., a = 1.0. The importance of the Interpolation operator for the framework is thus justified. In addition, too small an cannot efficiently transfer the knowledge from a small model to the large model and thus gives a negative saving ratio. It is also observed that higher saves less computational cost because the capability of network is limited.
a
16

---

Published as a conference paper at ICLR 2024
Effect of Coalesced Model Size We demonstrate the effect of the coalesced model size in Table rows (D). Generally, the performance of the pre-trained coalesced model improves as its parameter size increases, but this comes with corresponding rise in computational cost. The results in Table Si show that the BERT model with 6 layers and 6 heads offers an optimal balance between the performance and the computational cost, and thus result in the best FLOPS saving.
E DETAILS OF COALESCING MATRICES
In Section 4 under the assumption of d'it = di 2, doni d we define the width coalescing matrix F and the depth coalescing matrix R as follows:
0.5
Fk+L1 M,out
H
0.5
(15)
0.5
0.5
T0.5
0.5
0
k++1
(16)
0
0.5
0
0.5
To demonstrate how the width coalescing matrix coalesces parameters in a transformer model, we decompose the matrix F into matrix H and identity matrix I in Eq. 15 The dimension of the identity matrix
64, i.c. the size attention head in transformer. Thus the matrix H E 12,6 reveal the way merge the attention heads. For example, The H we used in Eq. 15 will merge the i and + 6 (4 = 1, 2, 6) attention heads pair by pair.
For simplicity, we denote the width coalescing matrix in Eq. 15 as k
sttcktack" and shown the depth
coalescing matrix Eq. 16 with Rk+1 There are different choices for coalescing matrices. For adj""
example, could define the width coalescing matrix p as Eq. 17 And could define
stack as the inverse operator of stacking layers (Gong et al. 2019).
0.55
0.5
out
(17)
0.5
0.5
TO.5
Rk+1
0.5
8
(18)
0.5
0
0.5
We conduct experiments to verify the effect of coalescing matrices. The comparison reveals minimal differences in FLOPs savings (<c 0.3%) among various matrices, which means that the choice of coalescing matrices may not be particularly crucial in framework.
17

---

Published as a conference paper at ICLR 2024
F EFFECT OF COALESCING OPERATION
The coalescing operation establishes a crucial link between the large and small models. To evaluate the effect of the coalescing operation, we remove the coalescing operation in our algorithm, i.e. randomly initialize the small model within the v-cycle. The comparative result, presented in Figure 5a suffers from an 8.3% drop FLOPs saving when the small model randomly initialized. The dramatic drop demonstrates the necessity of the coalescing operation.
Coolescing Coalescing
Conleccing Coalescing
3.4
9
3.3
Loss
3.2
3.0
2.9
3
20
0.0
Step (k)
Interpolation Ratio
(a) Training Curve
(b) Interpolation Loss Curve
Figure 5: Effect of the Coalescing Operation. In Figure (b), the model corresponds the GPT-Base before coalescing when the interpolation ratio is set to zero. Conversely, when this ratio is at one, the model becomes equivalent to the de-coalesced model, with without the application of the coalescing operation.
We examined the interpolation loss curve between the large model prior to coalescing and the de- coalesced model (with without coalescing). By interpolating models across various alpha values and assessing their validation loss, we could chart linear path in the optimization spac(Goodfellow Vinyals (2015). The results, displayed in Figure 55) reveal lower losses along the interpolation path for the de-coalesced model with coalescing operation. This finding underscores the tighter correlation between the de-coalesced and original models when coalescing applied.
G CONTINUE TRAINING THE DE-COALESCED MODEL.
GPT-Base De- Re-CColesced aaee
3.8
3.6
Loss
3.4
3.2
3.0
0
20
40
60
80
100
120
Step (k)
Figure 6: Training Curves of the GPT-Base training from scratch and the De-Coalesced GPT-Bas..
To delve deeper into the importance of the interpolation operation, let's consider a straightforward example highlighting the issue of symmetric neurons following width de-coalescing. Consider two
18

---

Published as a conference paper at ICLR 2024
layers feedforward neural network with an activation function f. represented as f(W,, 11, W2+ by. After de-coalescing the network with Tout I II 2 1/2|" in width dimension, the de-coalesced network can be formulated Eq. 19 For simplicity, let's refer the de-coalesced model as f(x W +b1 W/z In this model, the number of first layer's neurons/columns doubles, and the count of second layer's rows increases correspondingly from the first layer. However, the column count in the second layer remains constant to preserve the output quantity, for example, the number of classes.
accommodate the doubled output
W2/2 S(X [1, Wi] + [b,, b1])
+ b
(19)
W2/2
The output of the de-coalesced network is identical to the original. Assuming n neu- at
rons/columns in W1. it's straightforward to deduce that [t][1.mm
F. 1:2n] 0o, [Im]
11:22a1 awin JWinn+1:nn.j' With identical gradients, in the layer the first neurons will always mirror the subsequent n neurons, implying that the network's s learning capability doesn't proportionally increase with the number of parameters.
0L aL
AL
To illustrate this issue and underscore the interpolation operation's necessity, we continued training the de-coalesced GPT-Base model and have depicted the training curve in Figure 0 The empirical results reveal that due the limited learning ability caused by symmetric neurons, convergence performance of the de-coalesced model is significantly inferior to that of the model training from scratch.
H RESULTS ON DEIT-S
Table 6: Transfer learning performance of DeiT-S. DeiT-S pre-trained with multi-level approach shows similar performance DeiT-S pre-trained from scratch. The complexity and redundancy in DeiT-S less than DeiT-B. Therefore, the FLOPs and walltime saving is less on DeiT-S.
Saving (FLOPs)
Saving (Walltime)
ImageNet (Top 1 Acc)
CIFAR10
CIFAR100
Flowers (Acc)
Cars (Acc)
Method
(Ace)
(Acc)
DeiT-S
0%
0%
80.1% 1%
98.9%
89.8%
95.4%
92.2%
Ours 12.2%
6.5%
80.1% 1
98.9%
89.3%
95.4%
92.3%
IMPLEMENTATION DETAILS OF OPERATORS
19

---

Published as a conference paper at ICLR 2024
Algorithm 2: Coalescing Operation
Input
large transformer with L1 layers, hidden dimension of E1, vocabulary size T. Denote the embedding weight as ((emb) R parameters of attention layers as W W.K. W. W
R R bf bR by of E R parameters FFN layers (fel) I RIXEE, w E EIEEXEE EEE b(fe)) E ER .b ( ERE,
parameters of layernorm layer as W
wl(n2) E RRE, bf ((ml) b (in2) ERE.
Width coalescing matrix F(emb) F(OK)
F Fifel) F E Depth
coalescing matrix R E R411xa,
Output: small transformer with L2 layers, hidden dimension of Ez. Utilize upper right stroke to denote the parameters of the small transformer.
Preparation
plebb]
plemb) T
F (emb) plemb)" T out out (QK) F p(QK)
diag(1/sum.an
))
out p!QK)?' diag(1/ [aag(1/sumg col out
FQQK)
in F F(V)² out diag(1 /sum F (V) (V)²
out
out
in
out out
(fel)) F(fet) diag(1 (1/s1/mmmlll (Flfe)) F(fel)? ))
in
// Width Coalescing
(emb)) <- (emb) (emb))
for I + Lı do
w + F(emb)
w ((R))
in
out
W.K. Flemb)
W F QQK)
10
in
W. F(OK)
WV VV)
WO + F(V)
in W.
(emb)
out
b, < F(O)(O))
13
The bias vector consistently multiplies with the width coalescing matrix.
15
For the sake of brevity, we will omit the bias vector in following codes.
w(lmn)) < W((ml)) plemb)
16
w((ro)) (emb) w p(fel)
out
w + (fell) w (J22) plemb)
W (nn2) t w((n22) Fllmm)
19
out
20
end
// Depth Coaleseing
22
for I + L2 do
w/O + E W Ra
WiK EL
WK R.
24
WVV + E W. Ra
W + E WO R.
26
27
(((l)))
w
(1ml)
Ra
(JeD) Rad
W
Rid
29
W (ln)))
(((122) Ray
30
31
end
20

---

Published as a conference paper at ICLR 2024
Algorithm 3: De-Coalescing Operation
Input small transformer with Le layers, hidden dimension Ez, vocabulary size T. Denote the embedding weight as (emb)) RxxxE parameters of attention layers as W W.K. W/, WO
R Rax Fa, bf bR by of E R2, parameters FFN layers (fel)) I ExxEE, w((e2) E I
by (Fel) E RREs, ,b, (fc2) R
parameters of layernorm layer as W (Il))
W wl(n2)
E CRE, (inl) by (1n2) E ERE,
Width coalescing matrix F(emb) F(OK)
F
Fifel) E EEX Depth
coalescing matrix G E F 11112,
Output: large transformer with I layers, hidden dimension of Ej. Utilize an upper right stroke to denote the parameters of the large transformer.
Preparation
(lemb)
F (pmmb) diag(1/sumaal
F (emb) out (QK)T
pilemb) ))
out TQQK)= F(QK)" diag(1/ (1/uum
out
(QK)
out T(V) (VV) out out diag(1/ /sumcaal
out
out (V)
out
F
(V) out out
T(fel) F(fe1)" diag(1 (1/sumgnl
F(11)
F(
// Calculate F., as in Algorithm 2
(lemb) diag( (1/ /sw
(Flemb)
Flembl) in (QK)T
plemb)? T
in (QK)] diag(1
in F (V)
/ /sumpow
(QK)
(QK)
in
TVV) diag(1/ rou
(V)) F(V)
in
in (fe))
10 (fcl)) diag(1 (1/ag(1/ss /sum
(Sol) T
p(fel)
bn
G R diag(1/sumoo (RR'
12 Width De- Coalesssnn
13 (eemb) t (emb)) (((mb)
14 for 1 = + L2 do
wº Ty(emb)
we T (QK)
15
W.K Tlemml WKT (QK)) QQK)
out
W.V. + T(QKK) W (V)
17
out WO + -TV(V) Wo in
18
out
6, (b
20
The bias vector consistently multiplies with the width coalescing matrix.
21
For the sake of brevity. we will omit the bias vector in following codes.
w(lml)) +- w (1ml) (((mm))
w(fal) - (emb) w (fell
(ffel) out (emb)) out
23
W((02) - -T(fal) w (fe2)
24
W (In) w (in2) ( Tlemb)
25
end
27 // Depth De-Coalescing
28
for l + L1 do
w + E G
29
W + EL W.G.
w + E W.Y Gu
31
W + Σ L2 G
Li= L.2 (
33
W (lml)
+
G
w Whltanl
Σ wifah G
34
E w(ta G
35
W (ln2)
1 w (n2))
Gu
end
21

---

Published as a conference paper at ICLR 2024
Algorithm 4: Interpolation Operation
Input
large transformer with L1 layers, hidden dimension of E1, vocabulary size T. Denote the embedding weight as ((mm)) R parameters of attention layers as W.O W.K W/. W
R R by bR by of E R parameters of FFN layers (Fol) E R X4EE, w(fe2) E EREEEE, b(fe)), E IR b
(Fe2) ERE,
parameters of layernorm layer as W (iln)) wr(m2)) E RRE, by ((mm)) by (in2)
E ERE, A
de-coalesced transformer with the same size of the large transformer. Denote the
parameters of the de-coalesced transformer with a subscript of "d"". Interpolation ratio a.
Output: An interpolated transformer with the same size of the input transformers. The subscript "intp" is used designate the parameters of this interpolated transformer. f (1 a) W (emb) (embb) + aa adde (eeb))
(eemb) (emb)
W
inlp
de.
for l
= 1-11- L1 do
1 Lintp
+ (1
W + ta Ldd Ldde (1 a) W.K aWK
WK
ide WV t (1 a) W. + +aWy
Lintp
WO Litp
I,de
Lintp (1 a) W +oWo
i,de
b
t (1 a)b, + abl
Lintp
de
bk
(1 a b)b + aab de
Lintp
by
+ (1 - a)6 + ab
Lintp
bo
(1 of ab d,de
Lintp
((ln))
(1 a)) W (b11)
aa (b11)
Lintp
(bal)
(1 bbbb (lm1) (l11)
12
Lintp
i
W(fee)
(1 a) W (fl)
(Fel)
13
Lintp
ddee
(fel)
(1 a) ((/a)))
(b(ll)
(fel)
14
Lintp
+
i,de
((
(1 a) a)W(
(f2) aaw
15
Lintp
i,de
(1n1)
(1 - a)bl flmi)
+ ab (111)
16
Lintp
d,ee
W (ln2)
- (1 aoW
(1n2)
+ +aW
(bn2)
17
Lintp
1,de
(1n2)
+ (1 -a)b (1n2)
+ab (ln2)
Dinttp
I,e
end
22

---

Published as a conference paper at ICLR 2024
J
LEARNED TRANSFORMATION
4.0
GPT-Base w/o Learned Transformation
GPT Base Learned Transfermation
3.8
3.6
Loss
3,4
3.2
3.0
20 Step (k)
Figure 7: Further Training Curves of the GPT-Base training with or without learned transforma- tion. For learned transformation, we train the de-coalescing matrices in LiGO to perform better mapping function, resulting a lower initial loss after interpolated. However, we observe that the model with learned transformation finally converges to the same performance level the with- out learned transformation. The empirical evidence suggests that a direct combination does not lead advantages.
K THE CONNECTION WITH LORA
3.6
3.4
3.2
3.0
2.8
2.6
2.4
Coalesced BERT
LoRA r-48
2.2
LaRA 96
LaRA r=1
2.0
25
50
75
100
125
150
Step (k)
Figure 8: Training Curves of Coalesced BERT and BERT-Basc with LoRA: Our findings reveal that the rate of loss decrease in LoRA is substantially slower compared to the coalesced model. Additionally, the FLOPs required for the coalesced model are significantly lower than those for LoRA. Therefore, employing coalescing operation proves to be far more efficient than using LoRA.
Our framework bears a certain degree of resemblance with LoRA. Notably, when we apply width coalescing alone, the interpolation operator can be redefined as W Wntp Wtarge + Tin F,n FonrgeFont Our work focus on updating a segment the low-rank weights, specifi- cally F., W Warrge for both forward and backward propagation. This approach is different from LoRA. which involves using both the original and low-rank weights in the forward propagation. As result, approach significantly reduces FL.OPs and walltime in the pre-training phase of the
23

---

Published as a conference paper at ICLR 2024
smaller model. In contrast, LoRA does not alter the forward propagation process of the original model and even requires additional computations for the low-rank parameters. While LoRA do not need gradient calculations for weights in the feedforward, layernorm and embedding layers, as well as biases in all layers, the backpropagation process still needs to process gradients from the top layer down by calculating the product of weights in each layer and the gradients of the layer outputs, which is a major contributor to FLOPs in backpropagation. Therefore, the FLOPs sav- ing for LoRA is marginal compared with the coalesced model. Furthermore, our measurements of training time on models starting from scratch and those trained with LoRA revealed similar rates (around 4.0 iterations per second), which further confirms that the computation reduction of BERT with LoRA is marginal. Finally, in Figure g comparing the pre-training of BERT-Base with LoRA against Coalesced BERT, we observed that coalesced model converges much faster than BERT- Base with LoRA. This underscores the importance of focusing on intensive low-rank updates, i.e. the coalescing operation.
24


Reviews:
[{"review_id": "8lXtrd0URK", "reviewer": "Reviewer_2jMP", "summary": "The authors propose an efficient multi-level training framework, inspired by the observation of similarities within layers of these models during training. This framework employs a novel approach using three operators: Coalescing, De-coalescing, and Interpolation, to manage model scaling and parameter projection across different model sizes. It introduces a V-cycle training process that alternates between smaller, quickly trained models and larger networks, using the former to provide intermediate solutions for the latter. The interpolation operator is particularly crucial for enhancing convergence by adjusting neuron symmetries after de-coalescing. Experiments show that this framework can reduce computational costs by approximately 20% for BERT/GPT-Base models and up to 51.6% for BERT-Large, without compromising on model performance", "questions": "1. Can you explain how the intra- and inter-layer similarity is utilized in the coalescing and de-coalescing procedure? \n2. Can you explain why the performance is limited on the vision transformer?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea inspired by the multigrid algorithm to accelerate the large model training by coalescing, de-coalescing and interpolation is very clear and promising.\n2. Demonstrating the effectiveness of the proposed method is very solid and sound. The interpolation plays an important role in improving the learning ability.\n3. The reduction in FLOPs and training time is very significant in NLP transformer models.", "weaknesses": "1. Though significant speedup in the NLP transformer, the proposed method has limited improvement in FLOPs and time reduction in the large vision model. \n2. It's unclear how to initialize the matrix F. It seems the F can be arbitrary and the initialization of F is not discussed sufficiently.\n3. The evaluation result on GPT and DeiT-S lacks a comparison with other works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zUIS6A72sR", "reviewer": "Reviewer_subK", "summary": "This work proposes a multi-level framework for accelerating the training of large-scale deep learning models. This approach is inspired by the observation that training smaller models is more cost-effective and thus the authors propose a solution by generating high-quality intermediate solutions for subsequent larger networks.\n\nSpecifically, the authors propose a V-cycle learning process composed of three operations: Coalescing, De-coalescing and Interpolation. The Coalescing operator reduces the model size in terms of width, followed by depth. The De-coalescing operator is the inverse operation of Coalescing, with the de-coalescing matrices defined as the normalized transposition of the coalescing matrices. To address the low-rank issue present in the transformations, the authors also propose the Interpolation operation, which merges the de-coalesced model into the previous one. The authors suggest integrating the three operations into a V-cycle training framework, which learns to coalesce and train small models and then de-coalesce them into bigger models with Interpolation progressively.\n\nThe authors also provide experimental results on transformer-based models (BERT, GPT) and a vision model (DeiT), demonstrating significant speed-up (up to >50%) in training while maintaining performance.", "questions": "What is the significance of the coalescing operation within the overarching framework, and what benefits does it offer compared to initiating the training process with smaller models? In the coalescing step, the compression matrices F_in and F_out in equation 1 and 2, and R in equation 4 are heuristically defined. The recover matrices G in equation 7, and T_in and T_out are also manually defined without further explanation or theoretical basis. I feel the key point of this framework lies in modeling the correlation of parameters between large models and small models. I am not very convinced, from a methodological point of view, why the proposed framework can help to converge faster on the training set D.\n\nAside from conserving computational resources, what benefits do the interpolating model M_{k} and the de-coalesced model M_{k, de-coalesced} offer compared to continuing training the de-coalesced model M_{k, de-coalesced}?\n\nAt the end of the algorithm, the M_{1} model necessitates further training to achieve convergence. I'd like to see the computational overhead of this phase, as well as the comparative experimental results after removing this component.\n\nMost efforts of the experiments are on BERT models. Competitive compared methods, say LiGO, are not included in the results of GPT and DeiT in Table2 & 3. \n\nIn Table 4, as the number of Levels increases, the author's method not only saves more computational resources but also further improves the performance of the final model. What is the rationale for this observed enhancement in performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea is clearly presented, and the experimental results appear robust, providing strong support for the conclusions drawn.", "weaknesses": "I feel the overall novelty of this paper is a bit limited, as compared with LiGO. I find the major differences lie in two aspects:\n\n -LiGO learns linear mapping matrices via SGD, while this work intuitively defines the coalescing matrix as $[I, I]^T$, seeking to directly coalesce two neighboring neurons and adjacent layers;\n -As discussed in Appendix B, LiGO gradually learns to increase the model size, whereas this paper introduces V-cycle, a first-coalescing-then-decoalescing learning process equipped with interpolation.\n\nDespite the above, the improvements in FLOPs & Walltime and GLUE over LiGO are marginal (see Table 1). Additionally, more controlled experiments would be beneficial to substantiate the rationale for choosing heuristically defined mapping matrices over learnable parameters. The interpolation operation, which the authors claim mitigates the low-rank issue encountered in LiGO, is reminiscent of well-known PEFT methods like LoRA. More comparisons with this line of research would enhance the persuasiveness and credibility of the proposed method.\n\nLastly, the authors introduce the multigrid algorithm with a detailed description. However, it seems the proposed framework has little to do with this algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9B0ElMUm1F", "reviewer": "Reviewer_d5rc", "summary": "Transformer-based models perform well in many research areas such as NLP, CV, etc. However, they usually incur exponentially increasing energy costs in the training process. This paper proposes a multi-level framework for training acceleration. The whole working flow is composed of Coalescing, De-coalescing, and Interpolation. More specifically, first, the model is coalesced in both the width and depth direction. Then the large model can be coalesced into a smaller model. Next, to map the parameters back to the original model, the model is depth de-coalesced and then width de-coalesced. Next, after training the smaller model generated by coalescing, it conducts de-coalescing and then merges the coalesced model and de−coalesced model under the control of a hyperparameter. Finally, the merged larger model is trained. The proposed framework is evaluated on both accuracy and speed. The evaluation results show that the framework can keep or even slightly improve the accuracy and reduce the FLOPs and wall time.", "questions": "1. How different are the original model and the final model merged by the coalesced model and de−coalesced model? Do they have the same dimension? What are the differences between these two models?\n2. What does the number in the brackets represent in Tables 1 and 4?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The work proposes a novel method for improving the speed of Transformer-based models.\n+ It is carefully written.\n+ It offers enough analysis and explanations about the coalescing and de-coalescing details of the Transformer and the reason why this framework is designed in this way.", "weaknesses": "- The explanations in section 3 are helpful. However, it would be more helpful if it could include a flow chart or a figure of the structure of the whole framework.\n- Algorithm 1 in section 3.4 can help the readers understand the whole flow of the framework but is also kind of sketchy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vd1n9Kguo8", "reviewer": "Reviewer_Csxk", "summary": "The paper introduces a multi-level framework for training large-scale deep learning models like BERT, GPT, and ViT. The framework utilizes operators such as Coalescing, De-coalescing, and Interpolation to exploit inter- and intralayer similarities in feature maps and attentions. It follows a V-cycle training process that progressively adjusts the model size and transfers parameters between levels. Experimental results demonstrate that the proposed framework significantly reduces computational costs  while maintaining performance.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a novel multi-level framework for training large-scale deep learning models. By leveraging inter- and intralayer similarities, the framework addresses the challenge of high computational costs in training such models. The proposed operators and V-cycle training process provide a unique and effective solution. The V-cycle training process is different to the previous width/depth expansion methods like bert2BERT [1] and network expansion [2].\n\n[1] Chen, Cheng, et al. \"bert2BERT: Towards Reusable Pretrained Language Models.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n[2] Ding, Ning, et al. \"Network Expansion for Practical Training Acceleration.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n2. The paper supports its claims with extensive experiments conducted on transformer-based language models (BERT, GPT) and a vision model (DeiT). The experimental results demonstrate the effectiveness of the proposed framework, showcasing significant reductions in computational costs while preserving performance.\n\n3. Broad Applicability: The strengths of the paper lie not only in its application to specific models like BERT and GPT but also in its potential applicability to other large-scale deep learning models such as ViT. This suggests that the proposed framework has broader relevance and can contribute to addressing the training cost challenges across various domains and tasks.", "weaknesses": "1. The paper lacks in-depth technical explanations about the proposed operators (Coalescing, De-coalescing, and Interpolation) and their implementation. Why the V-cycle training process is better than the previous width/depth expansion methods like bert2BERT and Network Expansion?\n\n2. The paper does not provide a thorough comparison with existing methods or alternative approaches for training acceleration, e.g., Network Expansion [1].\n\n[1] Ding, Ning, et al. \"Network Expansion for Practical Training Acceleration.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n3. The paper does not extensively discuss the potential trade-offs or limitations introduced by the proposed framework. For example, are there any trade-offs in terms of model accuracy, generalization ability, or robustness? A thorough analysis of these aspects would provide a more comprehensive understanding of the framework's impact on model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)