Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Authors: Yu Pan
Ye Yuan
Yichun Yin
Zenglin Xu
Lifeng Shang
Xin Jiang
Qun Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Growth
Efficient Training
Pretrained Model
Multi-linearity
TL;DR: Using multi-linear transformation for growing pretrained model for accelerating training process.

Abstract: Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model (termed the ``target model''), leading to a considerable acceleration in training. Despite the successes of these previous studies, they  grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interactions among the weights of both the pretrained and the target models. As a result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, we propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that our method can save 76\% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12\% and LiGO by +21\%, respectively.

Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Yu P Ye Yuan Yichun Yin Zenglin Xu Lifeng Shang Xin Jiang Qun Liu Harbin Institute of Technology Shenzhen, Shenzhen, Guangdong, China Pengcheng Laboratory, Shenzhen, China Peking University, Beijing, China Huawei Noah's 's Ark Lab, Shenzhen, Guangdong, China
Abstract
Training large models from scratch usually costs a substamtial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize large model (termed the "target model"), leading considerable acceleration in training. Despite the successes of these previous studies, they grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interaction among the weights of both the pretrained and the target models. As result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, that linearly correlates each weight of the target model to all the weights the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that method can save 76% computational costs on DeiT-base transferred from Deil-small, which outperforms bert2BERT by +12.0% and LiGO by +20.7%, respectively.
propose method
Introduction
Transformers 47 have recently achieved great 20] Gener-
$
W Pay
successes in various scenarios ally, Transformers tend to be larger for more ex- pressive power and better performance (e.g.. ViT-
W
W
W
Wr
Wider
Expancing
Doph Expanding Operator
[ and GPT-3 B ). As the size of models con-
Figure 1: Expanding operators. W and We mean small pretrained and target weights, re- spectively. 49 and 4, denote width and depth expanding operators with parameters f., and 6g.
tinues to grow, training Transformers takes longer and can result higher CO2 emissions, which con- flicts with the principles of Green Al
4 Thus, training Transformers efficiently is crucial not only for financial gain but also for environmental sustainability wise option to grow pretrained small model into larger one, since the pretrained small model has already learned knowledge from the data, which allows for faster training compared starting from scratch. [3]. Moreover, there are numerous pretrained models that are easily accessible [33]. reducing the cost of utilizing a smaller pretrained model. Furthermore, empirical evidence shows This demonstrates the
[5]. To achieve efficient training, it is
that Transformers have inductive biases that facilitate scaling fitting 42 22 feasibility of learning from pretrained models [53]
"Tis work is done when Yu Pan is intern Huawei Noah's Ark Lab.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

S-(L10, A6)
S-(L10, A7I
A71,,
S-LL10, A7]
S-(L11, A2]
S-L11, A3)
5-LL9, A2)
S-1L10,A2)
SLL11, AS]
BL1L0,
B-[L11, A9)
B-(L11, A12)
SL300, B-[L11, B-IL12, A1)
B-(L12, A11)
Example 0
Example
Example 2
Example
Figure 2: Attention pattern maps between BERT-Small (shorted "S"" and BERT-Base (shorted as "B") in four examples. "I." represents the layer index, while "A" denotes the index of the attention head. For example, "S-(L10, A6)" denotes the attention map from layer 10, head 6 of the BERT-Sma model. The examples are derived from different sentences. In each example, we can observe the similarities of attention maps in both inter-layer and intra-layer connections, even across different scaled models.
The concept of reusing pretrained models essentially involves exploring the efficient mapping from the pretrained model to the target model. This mapping process can be represented as linear growth operators, as depicted in Figure [I] One viable approach to mapping is to leverage knowledge from other weights. For instance, StackBERT [1] utilizes low-layer information to construct new higher expands width through the expanding operator of Net2Net 77. Differently, bert2BERT utilizes weights of neighbor layers for enhancing the ability, which also shows the benefit of taking knowledge from other weights. Moreover, distinct from the aforementioned research directly employing fixed operator, LiGO [53] trains expanding operators which are tied with layers to achieve superior transfer accuracy by considering knowledge from other layers, which is advantageous for training efficiency. However, they neglect the potential connectivity between models.
layers, by duplicating layers to increase depth during training epochs. bert2BERT
Potential Connectivity between Models. The attention maps of BERT, as depicted in Figure 2 indicate that there are similarities not only the weights of the same or different layers, but also across different models. These similarities suggest that we can leverage full mapping transformation to capture this correlation scaled models. By doing so, we can reuse the parameters of pretrained small models and accelerate the training larger models. However, previous studies have overlooked the overall network connectivity and have instead expanded each weight through partial mapping transformation. For instance, Net2Net 7 only considers preserving functionality by transforming weights individually, while bert2BERT S increases model width head by head in Transformers. On the other hand, primarily focuses on extending the weights of the same type (e.g., query, key, and value in Transformers), which heavily affects the performance of the transferred models.
both inter-layer and intra-layer connections, even across different
LiGO 53
Multi-linear Operators. Based on the above observations, we propose mapping the entire pretrained model to each weight of the target model, instead of employing a partial transformation. This approach can ensure high expressive power, promoting the potential connectivity in models for better training efficiency. However, the completed mapping has huge parameter tensor, occupying enormous space (see Section
3.2) To address this issue, we propose Mango, a multi-lincar structure which decomposes the large mapping tensor into four smaller tensors that are bonded by ranks to construct multi-linear operators. Mango allows for efficient training by reducing the space required for the mapping tensor. Formally, Mango be considered generalization of LiGO and bert2BERT (see Section (.3). By using the full mapping approach, Mango save 76% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12.0% and LiGO 5% by +20.7%, respectively.
(i.e., tensor ring 3 2
Related Work
Efffcciett Training from Scratches. Model scratches can roughly regarded as models without knowledge priors. Some training strategies for scratches are universal and orthogonal to method.

---

For example, Adam 23 and large-batch size training [66] accelerate the training process from angles of optimizer and magnitude of input data, respectively. Wang et al. [50] enable the training of very deep neural networks with limited computational resources via technique called active memory. Shoeybi et al. 45 use mixed precision training to assist training. Low-rank methods benefit training for less memory and time [21]. Wu et al. [59] take notes of rare words for better data efficiency. Dropping layers [68], knowledge inheritance
38 and merging tokens [2] are also efficient 14 is termed progressive training, which gradually increases the model size within the training process for training efficiency. Li et al. 127 employ neural architecture search (NAS) method to search optimal sub-networks for progressive training on ViTs. Xia et al. [60] suggest a three-phase progressive training regime to achieve a good trade-off between training budget and performance. Wang et al. [55] introduces novel curriculum learning approach for training efficiency through firstly learning simpler patterns, then progressively introducing more complex patterns.
methods for training. Another line of work 03
63
Efficient Training from Pretrained Models. Pretrained models usually contain abundant data knowledge, which is helpful for training [2] By preserving the function of a pretrained model while expanding the model size, it is feasible to give the corresponding larger model an initial state with high performance. Net2Net [77 is the first work to propose the concept of function-preservinn transformations by expanding width by splitting neurons and growing depth with identity lavers. However, Net2Net splits neurons randomly. Towards this problem, series of studies [56 58 20 [57]
propose to select the optimal subset of neurons to be split by utilizing functional steepest descent. bert2BERT 5 expands small transformers by following the function preserving idea. Recently, LiGO 5 utilizes a trainable linear operator to learn a good expanding formula. Different from these prior studies, our method tries to implement full mapping that achieves comprehensive utilization of the whole smaller model.
Neural Network Initialization. Our method is related to neural network initialization techniques. Xavier 12 and Kaiming [T6] initialization aim to control input variance equal to that of output. Generalizing Xavier and Kaiming methods, universal weight initialization paradigm proposed by Pan et al. can be widely applicable to arbitrary Tensorial CNNs. In addition, Hardt and Ma 11 has shown theoretically that network training benefits from maintaining identity, particularly for improving the efficiency of residual networks. Fixup [67] and ZerO 77 both set residual stem to (not residual connections) ensure the identity of signals, thereby successfully initializing ResNets. Skiplnit [ replaces Batch Normalization with a multiplier whose value is O. ReZero [1.. on the maintain identity, resulting in faster convergence. DDnnt is an initialization approach to keep the identity matrix for stable training of networks (366. In comparison, our work explores fully reusing smaller pretrained models as efficient initialization.
other hand, adds extra parameters of value
Low-rank Techniques in Neural Networks. Low-rank methods are feasible for reducing spatial For example, Idelbayev and Carreira- Perpiñán 19 uses matrix decomposition to compress convolutional neural networks (CNNs) for faster inference. LoRA [1 applies low-rank matrices for fine-tuning large language models (LLMs) in affordable resources. As a parameter-efficient tuning (PETuning) method (69 for federated learning, FedPara [[8] utilizes the Hadamard product low-rank parameters for reducing communication time in federated learning. Pan et al. [35) and Li et al. (2 use tensor ring decomposition for reducing the size of neural networks. Tucker decomposition and block-term Tucker decomposition have been used in T-Net and B'T-layers [24] 64 for improving the model performance, respectively. Ma et al. 30 apply block-term Tucker to compress Transformers. Yin al. (55) propose to use ADMM to optimize tensor training to achieve better compression and performance. These techniques have built transformation for transferring knowledge
and temporal complexities in neural networks 552
35
52)
a solid foundation for our work implement from pretrained models to target models.
3 Mango Operator
This section presents the proposed Mango operator. First, we provide an overview of the tensor
diagram, the concept of the tensor ring matrix product operator (TR-MPO), and the formulation of Transformer architecture in Sec. 3.T Next, we delve into the details of the proposed multi-linear Finally, we compare Mango with recent advances in
mapping operator (i.e., Mango) in Sec. 3.2
terms of tensor diagrams in Sec.
3

---

3.1 Preliminary
10
Tensor Diagram. Tensor decomposition is a tech- nique that involves splitting tensor into multiple smaller tensors, typically more than the two matrices that result from matrix factorization. To better illus- trate the interactions among multiple tensors, tensor diagrams are often used. A tensor diagram is com- posed of two primary elements: a tensor vertex and a tensor contraction. A tensor vertex represents a tensor, and its order is determined by the number of edges connected to it. Each edge is assigned an integer that indicates the dimension of the corresponding mode. As shown in Figure tensor T 1gixii Xiz can be drawn as circle with three edges. The process of taking the inner product of two tensors on their matching modes is called tensor contraction. For example, Two tensors, M Roxi xiz and / E Iio XJ1 Kja, can be contracted in corresponding positions to form tensor of gi when they have equal dimensions: ig jo A operation can be formulated as
i0
Ja
i
i
i Tensor Contraction
12
jo
3rd-arder Tansor
Figure 3: Tensor diagram instances.
3rd-order
eg. The contraction
(M x2 io.ii..o
Mig Ni.iait midarss.
m=0
The tensor diagram is an elegant tool for succinetly visualizing and manipulating multi-dimensional arrays or tensors. By representing tensor operations networks of nodes and edges, these diagrams provide an intuitive way to understand the underlying mathematical operations and their interactions. As a result, we have chosen to use tensor diagrams to illustrate our method and to analyze the connections between our approach and prior studies.
Tensor Ring Matrix Product Operator (TR-MPO). TR-MPO means tensor ring (TR) of MPO 37 format. Given a 2NN-ordoee tensor X' /1x...xlxx.....xxx. its TR-MPO decomposition can be mathematically expressed as
R1, Ry
X in Ja.J
Σ
P I Prsaadrr Tritsibars dN
(N)
(2)
where {Ri, R2,..... RN} denote the ranks, ((") E xI, R2.12 denotes 4th-order core tensor and Ri = RN+1, which indicates ring- like structure.
Transformer Architecture. The Transformer 47 is deep learning architecture that has revolution- ized the artificial intelligence field including computer vision (CV) and natural language processing (NLP). As shown in Figure Transformer block consists of two main sub-layers: the multi-head self-attention (MHSA) layer and the feed-forward neural network (FFN) layer.
(1) MHSA Layer. The MHSA layer in the Transformer block computes the attention scores between each imput element and every other element, allowing the model to attend to different parts of the input sequence during processing, Inputs of MHSA query matrix Q E R', key matrix K R and value matrix V E R with parameters W E R XO W E R lx0 WV E R | WO e R Usually, it constrains I O. Moreover, MHSA separates these parameters into heads: (WQitg", [W..1). Wiin (WO.)). The MHSA mechanism can be formulated as follows:
Qw (KWK..)"
Au, K, V) softmax
wwV.
dd
MHSA((Q. K, V)) Σ (.
where de is the dimensionality of the key vectors. The self-attention mechanism is performed multiple times in parallel, each time using a different set of learned parameters WQ., K and WV: to compute multiple "heads" of attention. The resulting attention heads linearly transformed by learned weight matrix W to produce the output. At last, MHSA concatenates the output as the final result of the MHSA layer.
(2) FFN Layer. The FFN layer in the Transformer block responsible for applying non-linear transformation to the output of the self-attention layer. X R is the input. The weights

---

The [-th Layer of Transformer
The whole parameters of Transformer
Ria0
Riso
2
2.50
R
Ri
2
W
W
W
W
W
W
W
Add Nor
W
Concaterate Layers
W
Feed-Forward
W
W
W
WE
W
W
WF
Model Growth
Ado Norm
By]
Concatenate
B3
Head Sei-Aftention
Scaled Dot-Production
Ma
O2
h
Mi
O2
TR-MPO
I
I
B
Rs
R
R3
W
W!
Large Model
Small Model
Mapping Operator
Lg!
Figure 4: The full mapping of the Mango operator. The left sub-figure shows that the parameters of Transformer layer are Wº, W W Wo, W.i. and WOU', means an input dimension size. O denotes output dimension size. L is the layer number. We concatenate all the parameters into a tensor M and then consider a full mapping operator S to transform this tensor. However, S multi-linear method TR-MPO to decompose it into four smaller tensors (S B, S1 }.} form the Mango operator S S.....
is huge, thereby,
W E 7 and WOUT E R where usually I= and & is ratio that is often set to 4. The FFN layer can be formulated
FFN(X) = GeLU(XW WOUT
(4)
We neglect biases the formulation as it is usually set to 0 at initialization. The output the FFN layer is obtained by applying two linear transformations the output of the MHSA layer.
Finally, both the self-attention output and the FFN output are processed through a residual connection and layer normalization prevent the model from collapsing overfitting to the training data. Apparently, the parameters in a Transformer are mainly based on its linear transformation matrices, i.., W², WK WV, W WVN and WOCT
3.2 Multi-linear Operator
The essential of model growth is to transfer the knowledge from the small model to the bigger counterpart. Prior study (5] finds that taking weights from the neighbor layer as initial parameters can further improve the convergence speed, which gives the insights that knowledge from other layers can help training as there are many similar attention maps among layer. Nevertheless, the fact is that this similarity exists all over the pretrained model as shown in Figure which motivates to consider the possibility of whether we can utilize the knowledge from all weights. Based on this consideration, construct mapping of the whole model. As the correlation is hard formulate heuristically, we learn the mapping parameter to implicitly transfer the pretrained model.
Notation. Here, give the notations for clearly formulating method. A model with L layers and a D hidden size can denoted as M(L, D). Following Section 3.1 parameters of j-th layer are
set 03 (w2 E IxO WK E IIX W. E / w E Txo W E IxkO WOUT E RAIX0), [L]. usually satisfying / O D. The weight of M(L, is 0L.D (0, 10 A growth operator with parameter S is denoted as 4g.. A mapping from M(L, D2) - M(L2, D2) can be denoted as Do = ( Di In the case of L1 < L2, this mapping represents a growth mapping. After growing, 0 will be used as initial weights for the target model.
utilize weights for knowledge transferring, we first concate- nate weights across layers. As shown in Figure we concatenate j-th layer to form tensor of shape B x T O along with order / and O where = 24 + 4. Then the final weight tensor M BxXXXXLL can be derived by combining the concatenated tensors of L layers. Giving
Full Mapping Operator.
5

---

Bg
Bg
B1
B1
B1
R
R2
1
O2
O
1
1
01
Oz
1
01
02
81
60
50
L1
L1
R4
L
Ra
LjT
Si
L2
L2
Lal
ber12BERT
LIGO
Mango
Figure 5: Growth processes with the tensor diagram. The circle with an oblique stroke means super-diagonal tensor. The blue color means trainable operator, while the gray color denotes an untrainable operator. The red color means the smaller model. Each tensor diagram means the growth process that growing M1 to bigger model through growth the operator S..
small model M, E Baxxix0xx big model M2 E BRxx1xx2xx2 and parameter S E (axxxxxxxxxxxxxxlxxxxxxo of a full mapping operator 4s we can transfer Mi 1 to M2 as
(Ma)bais = Σ
(Mi)on Sn
b1,41.91.4
where b1, i1, 01, 11, b2, i2, 02 and l2 are entries of corresponding tensors.
Multi-linear Mapping Operator. Note that S is extremely big, which makes it infeasible display this mapping by applying whole transformation tensor. Therefore, we propose a multi- linear operator named Mango which decomposes S through tensor ring matrix product operator (TR-MPO) in four small tensors {S, E R x X So E RRxx0 xO2xR5 S, E Raxxxxx,, S E R RR.xhx12xRR., }. R {R, R2, Ra, Ra) is the rank of TR-MPO. Then, can update @ss to P So and Eq. can be reformulated with a multi-linear form as
(M2)22d20222
Σ dn
(6) (Mi) (SB) br. (80)),,)00))))0..... (SL)aarhsssrrrr.rrr (Str
The total size of S B. S, So. and S is exponentially less than S, which makes Mango viable for practical implementation while maintaining the full correlation between the small and big models.
Training Target. After designing the multi-linear operators, we train these operators to obtain the function preserving M2. The training target can be denoted as
min Sn So.Si,
(.. w.r.t M2 = S. (Mi1),
(7)
where L is loss function, and D is data distribution. By replacing the full space of S with four small spaces, the spatial requirements of the training process are reduced exponentially.
Procedures of Applying Mango. The procedures of Mango can be divided into four steps: (i) con- catenating weights @Z,.D, of a pretrained model M M(L,, ; Dj) to construct a tensor Mi; (ii) training the growth operator $ S. items of Eq. 7 in few steps (e.g.. 100) to make transferred models maintaining function; (iii) recovering weight tensor M2 through the multi-lincar operator @ Sy So S (iv) splitting M2 to the weight DD of the target model M(L2, Dz) initializa- tion to continue training.
Remark. Tensor decomposition is often used to compress neural networks. Actually, there are spaces for further reducing the operator size. However, we are not exploring the border compression ratio, but the possibility of the learning ability of multi-linear operators and the influence on model growth. Therefore, we decompose the huge S into four interpretable smaller tensors. Sn means interactions on the parameters in the same layer. S, and So denote the transformations of input and output dimensions in one parameter, respectively. Sr. indicates the relationship among layers. means the low-rank level of S. Smaller R means less correlation of these four tensors.

---

3.3 Comparison with Recent Advances
Basically, the goal of model growth is to widen and deepen the width and depth of the pretrained models, and operation on width and depth be easy to implement. Therefore, most of the previous studies grow width and depth separately, although there may exist some correlation that may influence the training efficiency. In this part, we analyze the difference among methods of model growth with the help of the tensor diagrams.
We illustrate comparison among bert2ERR,, LiGO, and Mango in Figure The red circle means pretrained model, while the blue and gray circles denote trainable and untrainable operator parameters, respectively. The circle with oblique stroke represents super-diagonal tensor, where the diagonal elements are all 1. This super-diagonal tensor means that growth operators expand other modes along the mode of the super-diagonal tensor.
Table 1: The comparison among ber12BERT, LiGO and Mango. In case of M( D1) M[2, D2), there usually are 1 = 01 = D1 and 1 = O2 = D2. R = max(R1 R2, R3, Ra).
Method
Reference
Operator Parameter Trainability
Spatial Complexity
bert2BERT
Chen et al. 5
S1, Sk
X
2I, D1 D2 + T1
LiGO
Wang et al.
11 SL
/
2B, D1 D2 + L2
Mango
S S..S, So. S
2RD, + R2 (B, L2
As shown in Figure 5] the parameters in the growth operator of bemtBERT are all frozen, since they are designed by heuristic inspiration to preserve function. bert2BERT expands modes of I and O₃ along with the mode of L1, indicating that one weight is expanded without knowledge from any other weight. To further increase the training ability, bert2BERT applies S k to construct Advanced Knowledge Initialization (AKT) to utilize knowledge of other layers to help accelerate training.
bert2BERT.
LiGO. Different from bert2BERT, the LiGO operator can be trained. In addition, LiGO expands I and O, along the mode of B₁. With S L. LiGO can combine knowledge among layers. Operators of Net2Net and StackBERT grow with or depth separately, which can be regarded as sub-solution to S,. So. and S2 of LiGO. However, one weight in LiGO will not leverage the knowledge of weights in the same layer. Therefore, LiGO only implements partial mapping like bert2BERT.
Mango. We employ Mango on each mode of the weight Mi, Mango can approach the full mapping tensor S as low-rank approximation with rank R. rather than partial mapping operators like LiGO and beTT2BERT. Therefore, Mango can obtain M2 to capture adequate knowledge from pretrained weights formulated in Eq. 6 Moreover, the diagonal tensor in LiGO and contraction between diagonal tensor and S k in beR are subsets of S and S in Mango. Therefore, Mango can be a generalization of bert2BERT and LiGO with more expressive power.
4 Experiment
In this section, we design a set of experiments to validate the proposed Mango. To begin with, we conduct an ablation study to analyze the influence of Mango on width and depth in Sec. 4. Then we conduct image classification with DeiT-B 46 on ImageNet to show the acceleration in computer vision task in Sec. 4.2 Later conduct pretraining experiment with BERT-Base [10 to
demonstrate the effectiveness in natural language processing tasks in Sec. 4.3 At last, we also employ pretraining experiment with GPT-Base 39 to show the wide adaption of Mango in Sec. 4.4
Ratio of saving FLOPs. We evaluate the training efficiency in terms of the ratio of saved floating-
point operations (FLOPs). Let us assume that training model from scratch to convergence on metric (e.g., MLM loss or accuracy) necessitates ESeratch FLOPs. Then, for a method that reaches the same metric V with ç. FLOPs, its FLOP saving ratio can be computed as
EScratch - E.
(8)
Eseratch
7

---

30
35
7358
693
436
70.72
32.6
73.4
46.5
73.96
41.7
Accuracy
73.89
73.82
05.59
26.7
73.85
41.0
23.28
37.8
68.00
23
73.81
38.3
72.93
35.9
73.77
34.0
67.79
17.9
73.72
32.8
72.25
22.0
67.1
Rank 10
15.0
73.68
30.0
71.91
Rank
Rank
(a) Width: DeDe-T-A--Dei-...
(b) Depth: DeiT-T-B--DeiT-S.
(c) Both: DeiT-T-C--DeTT.S.
Figure 6: Influence from ranks on expanding width, depth, and both of them. The green curve means the accuracy of training operators for 100 steps. The red curve means the final acceleration ratio training DeiT-S. Mango can achieve training acceleration in every case. Moreover, in each sub-graph, when the accuracy increases with higher rank, the acceleration ratio keeps almost fixed.
4.1 Ablation Study
In this experiment, we explore the influence of Mango growing width, depth and both of them in
addition rank setting. We use three tiny vision Transformers (ViTs) [.] i.e., DeiT-T-A, DeiT-T-B, and DeiT-T-C, for growing to DeiT-S [6 ImageNet [ Structures of these DeiT' can be found in Table 4 the Appendix. For ease of setting, we set
ranks the same in the range of {1, 4, 10). We train Mango operators for 100 steps, which only requires negligible time. We use Adam with learning rate 1e-3 and weight decay le-2 for 300 epoch optimization. The batch size is 1024.
Results are shown in Figure Mango achieves acceleration of at most 31.0%, 46.0%, and 41.3% on expanding width, depth, and both of them, respectively, which shows Mango can fit various growth scenarios. Interestingly, these acceleration ratios are higher when the operator accuracies are better along the types of pretrained models, e.g., as DeiT-T-B at most achieves 73.81 accuracy which is higher than 72.89% of DeiT-T-A, DeiT-T-B derives higher acceleration ratio. This phenomenon suggests that when there are multiple pretrained models can be selected, the better accuracy for one pretrained model through Mango, the faster training speed can be obtained for the target models.
In each expanding case, the accuracies of operator training tend to increase with higher ranks. However, all the cases on the three pretrained models show that better accuracy will not lead to faster training with the same pretrained model. For example, Figure 6(a) the operator with rank 10 has an accuracy that is 1.19% higher than the operator with rank 1. Nevertheless, the two ranks reach the same acceleration ratio. This result suggests that rank is enough to use, and also enjoys the advantages of spatial and temporal complexities compared to bigger ranks. And we choose rank 1 for Mango construct the later experiments.
4.2 Results on Large-Scale Vision Transformer
We conduct the experiment to show the training acceleration on large-scale vision Transformers. We train the Deit-B from Deit-S 46 on ImageNet. Structures the two models are shown Table of the Appendix. Ranks of Mango
all set to for complexity benefit without performance loss. The operators of Mango and Ligo are all trained within 100 steps. We use Adam as the optimizer with learning rate le-3 and weight decay 1e-2. The batch size is 1024. The training epoch is 300.
Results are shown Figure 7(a) Mango saves 76% FLOPs from Scratch which converges to an accuracy of 80.45% in the end. Compared with schemes of training from Scratch (i.e., Sreatch and StackBERT), methods of training from smaller model (i.c., Mango, bert2BERT, and LiGO) can attain 70% accuracy in short time, even bentBERT start in low accuracy. Compared with the recent SOTA models, Mango has surpassed bert2BERT for +12.0%, and LiGO for +20.7%. This experiment shows the ability of Mango to achieve significant improvement in training acceleration. To investigate influence of Mango on transferring ability, we also conduct an experiment on downstream tasks, including CIFAR10 10 26 CIFAR 100 26 Flowers 31] Cars (55 and ChestXRay8 [54]. Results are shown in Table 2 easy to see that Mango achieves similar results the Scratch, which indicates Mango has not influenced the transferring ability.

---

85
2.3
2.90
80
22
2 2.55
2.1
Margo
2.80
75
Accuracy
2.0
2.75
70
2.70
65
1.8
60
1.6
2.55
55
1.5
2.50
50
14
2.45
4
12
FLOPs (1e18)
FLOPs (1e19)
FLOPs (1e19)
(a) DeiT-S Deit-B.
(b) BERT-Small SwwlS-allaaaaaaee -BETT-ERT---E Basc.
(c) GPT-Small- -GPI- Basc.
Figure 7: Results of pretraining DeiT-B, BRET-Base and GPT-Base. Compared to baselines, Mango can achieve the highest savings in FLOPs with 76.4% for DeiT-B, 39.2% for BERT-Base, and 59.9% for GPT-Base from the Scratch model. We also illustrate the corresponding results on wall time in Figure 10 the appendix.
Table 2: Results of transfer learning performance of DeiT-B. Mango achieve similar performance to the Scratch model in downstream tasks while saving 76.4% FLOPs.
FLOPs (x le18)
Ratio (Saving)
Method
CIFARIO
CIFARI00 Flowers
Cars
ChestXRay8
Average
Training from Scratch
Scratch
12,9
99.03
90.22
97.27
91.89
55.66
86.82
StackBERT 11.3
12.6%
99.11
90.10
97.44
91.71
55.63
86.80
Training from the Pretrained Model: M(12,384)
M(12, 768)
bert2BERT
64.4%
98.99
97.51
91.88
55.34
86.84
LiGO
5.7
55.7%
99.11
90.52
97.18
91.82
55.45
86.82
Mango
76.4%
99.13
90.23
97.49
91.83
55.46
4.3 Pretraining on BERT
In this experiment, we conduct the validation to show the training acceleration on BERT 10 51 The dataset is the concatenation of English Wikipedia and Toronto Book Corpus 77 We train the BERT-Base from BERT-Small The training epoch 40. The batch size 768. We list the structures of the two models in Table of the Appendix. The ranks of Mango all Mango and LiGO are both warmly trained for 100 steps. The optimizer is set to W. The learning rate is le-4 and the weight decay is 1e-2.
illustrate the training curves in Figure 7(b The methods of training from BERT-Small (i.e., Mango, bert2BERT, and LiGO) can all surpass the progressive training StackBERT of acceleration ratio 29.5 2.. bert2BERT is over StackBERT by +6.1%. Mango achieves the highest acceleration of 39.2% FLOPs which is +3.6% more than bert2BERT. Loss of StackBERT steeply decreases within training for Stacking layers, which can also demonstrate the pretrained weights are helpful for training efficiency. We show the effectiveness of Mango SQuAD and GLUE benchmark in Table 3 Mango shows the same transferring ability with faster convergence, which indicates a promising use for practical training.
4.4 Pretraining on GPT
We also implement the experiment to show the training acceleration on GPT 39 The dataset is the concatenation of English Wikipedia and Toronto Book Corpus 71 We train the GPT-Base from GPT-Small. The structures of the two models are shown in Table the Appendix. The ranks of Mango are Mango and LiGO are trained for 100 steps. We use Adamw with learning rate le-4 and weight decay le-2. The batch size is 512. The training epoch is 35.

---

Table 3: Results of downstream tasks of BERT-Base on GLUE 48. SQuADvL.1 40 and SQuADv2.0 41 dataset. Mango can also achieve similar performance with the Scratch model while enjoying training efficiency.
FLOPs
Ratio
SQuADvI.I SQuADv2.0 SST-2 MNLI MRPC COLA QNLI STS-B QQP GLUE SQuAD (FI)
Model
(x 1c19) (Saving)
(FI)
(Ace) (Ace) (Acc) (Mec) (Acc) (Acc) (Ace) Avg. Avg.
Training from Scratch
Scratich
8.9
89.21
77.90
92.18
84.19
87.55
56.35
91.50
89.16
90.25
84.45
83.56
StuckBERT
29.5%
89.82
78.21
84.63
87.65
61.61
90.95
87.13
90.20
85.01
84.01
Training from the Pretrained Model: M( 384)
-> M(22, 768)
ben2BERT
5.7
35.6%
90.02
78.99
92.89
84,92
86.91
60.32
91.81
88.
90.72
85.10
84.50
LIGO
5.9
33.5%
90.09
78.34
92.75
84,99
87.44
61.10
91.33
87.94
90.42
85.14
84.22
Mango
5.4
39.2%
90.17
78.77
92.71
84.86
87.94
62.88
91.49
88.73
90.62
85.60
84.47
We compare the Scratch model, ben2BERT, LiGO, and Mango in Figure [c We observe that the proposed Mango achieves a 59.9% acceleration ratio. While GPT different from BERT different structures, including layer normalization, mask method, and training strategy, Mango can always keep the highest performance. Although LiGO is lower than bert2BERT at the beginning, converges slower and reaches 38.1% last. By contrast, the Mango is almost at the lowest loss in the whole training process and achieves +16.7% more than bert2BERT and +2%.8% more than LiGO, which shows a significant acceleration than the baselines.
5
Conclusion
Training Transformers can pose significant demand computational resources. Reusing pretrained models as an initialization strategy for the target model offers a promising approach to reducing resource costs and accelerating training. However, previous studies only mapped partial weights when growing models, which may fail to consider potential correlations the whole model and result in inadequate growth mapping. Inspired by this observation, we propose to consider the interaction among all weights in the model to further improve acceleration ability. Specifically, we utilize a full mapping comprehensively reuse the pretrained model, taking into account all correlations between model weights. As the full mapping tensor is huge and cannot be employed in practice, we propose to use Mango, multi-linear operator, to reduce computation and space complexity. Experimental results demonstrate that Mango consistently achieves significant acceleration on various large-scale models (e.g.. DeÎT-B, BERT-Base, and GPT-Base). In the future, we hope that this method can contribute to green AI and significantly reduce the cost of training Transformers.
Limitation. While Mango significantly reduces training costs for large models, it still requires weights of a small pretrained model as the prior knowledge. Additionally, the Mango operator necessitates extra training for initializing. However, obtaining small model weights is relatively simple within the community, and given the substantial cost savings, the resources required for training the operator minimal.
Societal Impact. The acceleration from Mango significantly impacts energy conservation and environmental protection, promoting the growth of Green AI, which boosts efficiency and reduces computational demand, minimizing energy usage and carbon emissions. Mango can also democratize AI, allowing broader access to technologies of pertaining models without the need for large resources, and promoting sustainable and accessible AI solutions that respect our environmental limits.
Acknowledgements
This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100004), key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ202001091 11199111400 13403826), the Major Key Project of PCI. (No. 2022ZD01 and an Open Research Project of Zhejiang Lab (NO.2022RCOAB04).
10

---

References
[1] Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Julian J. McAuley. Rezero is all you need: fast convergence large depth. In UAI, volume 161 of Proceedings of Machine Learning Research, pages 1352-1361. AUAI Press, 2021.
[2] Daniel Bolya, Cheng- Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. CoRR, abs/2210.09461, 2022.
[3] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurlPS, 2020.
[4] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Man- ning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. CoRR, abs/2105.055337 2021.
[5] Cheng Chen, Yichun Yin, Lifeng Shang. Xin Jiang, Yujia Qin, Fengyu Wang. Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. In ACL (I), pages 2134-2148. Association for Computational Linguistics, 2022.
[6] Richard Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishman, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In CVPR, pages 16123-16134. IEEE, 2022.
[7] Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. knowledge transfer. In ICLR, 2016.
Net2net: Accelerating learning via
[8] Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In NeurlPS, 2020.
[9] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pages 248-255. IEEE Computer Society, 2009.
[10] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (I), pages 4171-4186. Association for Computational Linguistics, 2019.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
[12] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, volume 9 of JMLR Proceedings, pages 249-256. JMLR.org, 2010.
[13] Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie- Liu. Efficient training of BERT by progressively stacking. ICML, volume 97 of Proceedings of Machine Learning Research, pages 2337-2346. PMLR, 2019.
[14] Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han, On the transformer growth for progressive BERT training. In NAACL-HILT, pages 5174-5180. Association for Computational Linguistics, 2021.
[15] Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In ICLR (Poster). OpenRe- view.net, 2017.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034 IEEE Computer Society, 2015.

---

[17] Edward J. Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenReview.net, 2022.
[18] Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficiie federated learning. In ICLR. OpenReview.net, 2022.
[19] Yerlan Idelbayev and Miguel Á. Carreira-Perpiñán Low-rank compression of neural nets: Learning the rank of each layer. In CVPR, pages 8046-8056 Computer Vision Foundation / IEEE, 2020.
[20] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In NewrIPS, pages 14745-14758, 2021.
[21] Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N. Gomez. Exploring low rank training of deep neural networks. CoRR, abs/2209. 2022.
[22] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.088 1.88611, 2020.
[23] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
[24] Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. T-net: Parametrizing fully convolutional nets with a single high-order tensor. In CVPR, pages 7822-7831. Computer Vision Foundation IEEE, 2019.
[25] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops, pages 554-561. IEEE Computer Society, 2013.
[26] Alex Krizhevsky, Geoffrry Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[27] Changlin Li, Bohan Zhuang, Guangrun Wang, Xiaodan Liang, Xiaojun Chang, and Yi Yang. Automated progressive learning for efficient training of vision transformers. In CVPR, pages 12476-12486. JEEE, 2022.
[28] Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems, pages 1-15, 2021.
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings the IEEE/CVF international conference on computer vision, pages 10012-1002) 2021.
[30] Xindian Ma, Peng Zhang. Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song.
A tensorized transformer for language modeling. In NeurIPS, pages 2229-2239, 2019.
[31] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP, pages 722-729. IEEE Computer Society, 2008
[32] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans. Knowl. Data Eng., 22() 10:0)))11553555 2010.
[33] Yu Pan, Jing Xu, Maolin Wang, Jinmian Ye, Fei Wang, Kun Bai, and Zenglin Xu. Compressing recurrent neural networks with tensor ring for action recognition. In AAAI, pages 4683-4690. AAAI Press, 2019.
[34] Yu Pan, Zeyong Su, Ao Liu, Jingquan Wang, Nannan Li, and Zenglin Xu. unified weight initialization paradigm for tensorial convolutional neural networks. In International Conference on Machine Learning, ICMI. 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 17238-1722 17257. PMLR, 2022.
12

---

[35] Yu Pan, Maolin Wang, and Zenglin Xu. Tednet: A pytorch toolkit for tensor decomposition networks. Neurocomputingg 469:234-255 238, 2022.
[36] Yu Pan, Zekai Wu, Chaozheng Wang. Qifan Wang, Min Zhang, and Zenglin Xu. Identical ini- tialization: A universal approach to fast and stable training of neural networks. Openreview.nel, 2022.
[37] Bogdan Pirvu, Valentin Murg. J Ignacio Cirac, and Frank Verstraete. Matrix product operator representations. New Journal of Physics, 12(2):02001,, 2010.
(38) Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang. Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Knowledge inheritance for pre-trained language models. In NAACL-HLI, pages 3921-3993 Association for Computational Linguistics, 2022.
[39] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog. 188)99,
[40] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, pages 2383-2392, The Association for Computational Linguistics, 2016.
[41] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for squad. In ACL (2), pages 784-789, Association for Computational Linguisties, 2018.
[42] Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. ICLR. OpenReview.net, 2020.
[43] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AL. CoRR, abs/1907.10599 2019.
[44] Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy. Staged training for transformer language models. In ICML, volume 162 of Proceedings of Machine Learning Research, pages 19893-19908. PMLR, 2022.
[45] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Imp Training multi-biliion parameter language models using model parallelism. CoRR, /ts//199008053 2019.
[46] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training
image transformers & distillation through attention. In ICML, volume 139 of Proceedings of Machine Learning Reseurch, pages 10347-10357. PMLR, 2021.
[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pages 5998-6008, 2017.
[48] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR (Poster). OpenReview.net, 2019.
[49] Dilin Wang, Meng Li, Lemeng Wu, Vikas Chandra, and Qiang Liu. Energy-aware neural architecture optimization with fast splitting steepest descent. CoRR, abs/1910.03 2019.
[50] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song. Zenglin Xu, and Tim Kraska. Superneuron:: dynamic GPU memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2018, Vienna, Austria, February 24-28, 2018 pages 41-53. ACM, 2018
[51] Maolin Wang, Chenbin Zhang, Yu Pan, Jing Xu, and Zenglin Xu. Tensor ring restricted boltzmann machines. In International Joint Conference on Neural Networks, IJCNN 2019 Budapest, Hungary, July 14-19, 2019, pages 1-8. IEEE, 2019.
13

---

[52] Maolin Wang, Yu Pan, Zenglin Xu, Xiangli Yang, Guangxi Li, and Andrzej Cichocki. Tensor networks meet neural networks: survey and future perspectives. CoRR, abs/2302.09019, 2023.
[53] Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning grow pretrained models for efficient transformer training. In ICLR. OpenReview.net, 2023.
[54] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: cuspital-scale chest x-ray database and benchmarks on weakly- supervised classification and localization of common thorax diseases. In CVPR, pages 3462-
3471. IEEE Computer Society, 2017.
[55] Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficienttrain: Exploring generalized curriculum learning for training visual backbones. CoRR, abs/2211 1.09703, 2022.
[56] Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures. In NeurlPS, pages 10655-10665, 2019.
[57] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: general approach for growing neural networks. In NeurlPS, 2020.
(58) Lemeng Wu, Mao Ye, Qi Lei, Jason D. Lee, and Qiang Liu. Steepest descent neural architecture optimization: Escaping local optimum with signed neural splitting. CoRR, abs/2003. 10392, 2020.
(59] Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, and Tie- Yan Liu. Taking notes on the fly helps language pre-training. In ICLR. OpenReview.net, 2021.
[60] Zhuofan Xia, Xuran Pan, Xuan Jin, Yuan He, Hui Xue, Shiji Song, and Gao Huang. Budgeted training for vision transformer. ICLR. OpenReview.net, 2023.
[61] Jing Xiong, Chengming Li, Min Yang, Xiping Hu, and Bin Hu. Expression syntax information bottleneck for math word problems. In SIGIR, pages 2166-2171. ACM, 2022.
[62] Jiong Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yi,, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, Jing Tang, Chengming Li, and Xiaodan Liang. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. CoRR, abs/2310.0200 2023.
[63] Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progres- sively stacking 2.0: A multi-stage layerwise training method for BERT training speedup. CORR, Abs/2011.13635, 2020.
[64] Jinmian Ye, Guangxi Li, Di Chen, Haiqin Yang, Shandian Zhe, and Zenglin Xu. Block-term tensor neural networks. Neural Networks, 130:11-21, 2020.
[65] Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decomposition based DNN model compression with optimization framework. In CVPR, pages 10674-10683, Com- puter Vision Foundation/ / IEEE, 2021.
[66] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT 76 minutes. In ICLR. OpenReview.net, 2020.
[67] Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR (Poster). OpenReview.net, 2019.
[68] Minjia Zhang and Yuxiong He. Accelerating training of transformer-baser language models with progressive layer dropping. In NeurlPS, 2020.
14

---

[69] Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficiet tuning methods of pre- trained language models. In Findings of the Association for Computational Linguisties: ACI. 2023, Toronto, Canada, July 9- 14, 2023, pages 9963 9977. Association for Computational Linguistics, 2023.
[70] Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing residual networks with only zeros and ones. CoRR, abs/21 10.1110..1..... 2021.
[71] Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov. Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, pages 19-27. IEEE Computer Society, 2015.
15


Reviews:
[{"review_id": "QPMdHk4Jl8", "reviewer": "Reviewer_u4ot", "summary": "The contribution of this paper lies in proposing a new method to accelerate the training of large models by leveraging the correlations among all weights in the model, instead of just mapping partial weights as in existing methods. The authors' approach utilizes multi-linear operators to reduce computational and spatial complexity, enabling acceptable resource requirements. Experimental results demonstrate that their method has excellent performance in accelerating training, outperforming existing methods.", "questions": "+ In line 196 of the paper, for the loss function in Equation (6), what is the specific design of this loss function for the operators? Is it task-specific or fixed?", "limitations": "It might be interesting if the results can be extended to large language models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The method proposed in the paper can leverage the potential correlations among all weights in the model, significantly reducing computational costs.\n+  the authors conducted experiments in both image and natural language processing domains and achieved results that surpass existing methods. This indicates that their proposed approach has a certain level of generality and applicability, and can significantly accelerate model training across different domains. ", "weaknesses": "+ In the experiments of the paper, only DeiT-B, BERT-Base, and GPT-Base were used, and no further exploration was done on larger models.\n+ The performance of Mango, Ligo, and bert2BERT was improved in natural language processing methods, while only marginal gains were achieved in image tasks, and the experiment lacked analysis and explanations.\n+ For Figure 6(b), there is continuous improvement in accuracy with increasing rank, but there is a lack of experimental comparison for larger rank values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are thankful for the insightful comments of the reviewer!\n\n> In the experiments of the paper, only DeiT-B, BERT-Base, and GPT-Base were used, and no further exploration was done on larger models.\n\nWe construct an experiment to train a large model ViT-G-1B. We plot the early result in the uploaded figure pdf due to the short rebuttal period. Although the scratch model has not reached a high position, the tendency shows our acceleration on it.\n\n> The performance of Mango, Ligo, and bert2BERT was improved in natural language processing methods, while only marginal gains were achieved in image tasks, and the experiment lacked analysis and explanations.\n\nThis is an interesting phenomenon. We construct downstream tasks with the checkpoint of the same metric value (accuracy in vision task and loss in language task). The similar result in vision downstream tasks suggests that reusing a small pretrained model for training efficiency will not hurt the transferring ability of the target model. By contrast, downstream tasks in language show resuing a pretrained model can make an improvement. We guess the trained target may cause this difference, vision: main task: classification accuracy -> downstream task: classification accuracy; language: main task: pretraining loss -> downstream task: classification accuracy. However, as deep networks are in a black box, we cannot precisely interpret this phenomenon. In all, this phenomenon suggests that the methods (e.g., Mango, bert2BERT and LiGO) reusing a small pretrained model can improve the training efficiency without hurting model transferring ability, and derive a performance profit in downstream tasks mainly in language models when the loss is comparable.\n\n> For Figure 6(b), there is continuous improvement in accuracy with increasing rank, but there is a lack of experimental comparison for larger rank values.\n\nWe add a rank 24 to the experiment of \"DeiT-T-B -> DeiT-S\". Results are shown in the following table. The accuracy of the larger rank of 24 is slightly less than the rank of 10, while still achieving the same acceleration ratio as ranks of 4, 7, and 10. Therefore, as we also suggest in the Line 253 of the paper, rank 1 is enough to use, and also enjoys the advantages of spatial and temporal complexities compared to bigger ranks.\n \n|                    |   1    |   4    |   7    |   10   |   24   |\n|:------------------ |:------:|:------:|:------:|:------:|:------:|\n| Accuracy           | 73.72% | 73.75% | 73.76% | 73.81% | 73.77% |\n| Acceleration Ratio | 46.0%  | 45.7%  | 45.7%  | 45.7%  | 45.7%  |\n\n\n> In line 196 of the paper, for the loss function in Equation (6), what is the specific design of this loss function for the operators? Is it task-specific or fixed?\n\nThe loss function is consistent to the training of the target model. For example, when we train the target model in CV tasks with a classification loss, we adopt the same classification loss for training the operators. Similarly, we use the MLM loss in BERT models and the GEN loss in GPT models for training corresponding Mango operators."}, {"review_id": "FhATnd04zA", "reviewer": "Reviewer_kqva", "summary": "This paper studied the problem of solving a proper initialization for large deep learning models on top of smaller pre-trained networks. The key idea is to concatenate the weights of pre-trained models to a large matrix, and map the matrix to the parameter space of the larger model to be trained with a multi-linear operator. In this procedure, the rank of the transformation matrix can be manually adjusted. Experimental results on NLP/CV-based Transformer models are provided.", "questions": "Please see the weaknesses.", "limitations": "Personally, I think this paper may not have potential negative societal impacts.\n\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of efficient training is practically important.\n2. The proposed method is well-motivated.\n3. The design of the multi-linear operator is interesting.\n4. Mango can be applied to both NLP and CV models.", "weaknesses": "1. The writing needs to be improved. It is difficult to understand Eqs. (4-5), which are the key contributions of this paper.\n2. Currently, the topology of the network is not considered in Mango (e.g., the depth of the layer corresponding to each weight matrix). I think it may be beneficial to leverage this characteristic in Mango.\n3. Can we introduce non-linear operation to Mango to improve training efficiency?\n4. In Figure 7, the comparison with training from scratch may be unfair. Take DeiT for example. Obtaining DeiT-S needs a notable amount of computation. Hence, the computational cost of Mango should not start from zero FLOPs.\n5. This basic baseline may be interesting: initializing the large model (e.g., DeiT-B) by pre-training it using the same amount of computation as training the smaller model (e.g., DeiT-S).\n6. Can Mango improve the final accuracy of the model?\n7. In the field of CV, it would be interesting to validate the effectiveness of Mango on top of more network architectures, e.g., Swin Transformer.\n8. Some related works should be discussed or compared with Mango:\n- Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., & Yang, Y. (2022). Automated progressive learning for efficient training of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12486-12496).\n- Budgeted Training for Vision Transformer, ICLR'23\n- Wang, Y., Yue, Y., Lu, R., Liu, T., Zhong, Z., Song, S., & Huang, G. (2022). Efficienttrain: Exploring generalized curriculum learning for training visual backbones. arXiv preprint arXiv:2211.09703.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments!\n\n> The writing needs to be improved. It is difficult to understand Eqs. (4-5), which are the key contributions of this paper.\n\nEqs. (4-5) are both sum of element-wise production among tensors. The subscripts (e.g., $b_{\\ast}, i_{\\ast}, o_{\\ast}, l_{\\ast}, o_{\\ast}$, and $r_{\\ast}$) mean the entries of the tensors. We will try our best to further polish the clarification, e.g., adding more description of the tensors and calculation in the equations .\n\n> Currently, the topology of the network is not considered in Mango (e.g., the depth of the layer corresponding to each weight matrix). I think it may be beneficial to leverage this characteristic in Mango.\n\nThis is a great comment and could be a future perspective to explore. If our understanding of the term \"topology\" is accurate, Mango may incorporate topology-related information into its considerations. Considering the example of layer topology to illustrate this concept, it is noteworthy that different layers typically have distinct functionality. Therefore, accounting for these variations can be beneficial. An interesting aspect is that Mango assigns trainable weights (denoted as $\\mathbf{S}_L$) to facilitate the learning of layer importance within smaller models. This approach implicitly captures the aforementioned topology in some sense. Further utilization of network topology is beyond the scope of this work and can be explored in the future.\n\n> Can we introduce non-linear operation to Mango to improve training efficiency?\n\nWe apply a non-linear activation function GeLU after multiplying the smaller model with Mango operators. We construct this experiment by training DeiT-S from DeiT-T-A on ImageNet. The result shows GeLU can achieve acceleration ratios at 31.0% which is the same as training without a non-linear activation function. This preliminary exploration of non-linear activation functions shows that it is possible to apply non-linear functions. However, an efficient way to apply non-linear functions is still needed to be explored.\n\n\n\n> In Figure 7, the comparison with training from scratch may be unfair. Take DeiT for example. Obtaining DeiT-S needs a notable amount of computation. Hence, the computational cost of Mango should not start from zero FLOPs.\n>\n> This basic baseline may be interesting: initializing the large model (e.g., DeiT-B) by pre-training it using the same amount of computation as training the smaller model (e.g., DeiT-S).\n\nThis is a good perspective. We understand that the cost of pretraining small models could be added into the total cost of the proposed method and prior research (e.g., bert2BERT and LiGO), for a fair comparison with training from scratch. However, it is important to restate the setting of this paper and prior research (e.g., bert2BERT and LiGO) is that there are free and publicly available small or medidum scaled pretrained models (e.g., those can be downloaded from HuggingFace or released by public giants), while the cost of training large models from scrach is prohibited. Given the existing smaller pretrained models, Mango effectively leverages the insights within smaller pretrained models, thereby can achieve significant improvement in training efficiency in comparison with  prior research (e.g., bert2BERT and LiGO) and training from scratch. Our experiments, based on standard model configurations, demonstrate the practicality of utilizing available pretrained models to expedite training. Furthermore, we believe that the total cost of training small models and the expanded model training should be much smaller than large models (especially for LLMs which are often computationally prohibitive). Consequently, **it is reasonable to make a comparison between methods of training for a smaller pretrained model (e.g., Mango, bert2BERT, and LiGO) and training from scratch.**\n\nNonetheless, it is important to acknowledge that the necessity for a pretrained model remains a potential limitation, which we clearly address in the \"Limitations\" section. We hold the view that while this limitation exists, its impact on real-world applicability is not overly stringent.\n\n\n> Can Mango improve the final accuracy of the model?\n\nYes. With follow-up training, Mango can achieve better performance. For example, in the experiment of \"DeiT-S -> DeiT-B\", Mango can improve the accuracy from 80.45% to 80.75% in the end. Similarly, Mango also achieves loss reduction as \"BERT-Small -> BERT-Base\": 1.47->1.41, \"GPT-Small -> GPT-Base\": 2.49->2.46.\n\n> In the field of CV, it would be interesting to validate the effectiveness of Mango on top of more network architectures, e.g., Swin Transformer.\n\nWe conduct an experiment on training Swin-S. Mango adopts a Swin-T model as the smaller model. We train Swin-S for 240 epoch. We show a acceleration in the uploaded figure pdf.\n\n\n> Some related works should be discussed or compared with Mango:\n\nThanks for the suggestions regarding the related works. We appreciate your insights and have carefully considered the references you provided. [1] employs a neural architecture search (NAS) method to search optimal sub-networks for progressive training on ViTs. [2]  suggests a progressive growing mechanism used across a three-phase training regime to achieve a good trade-off between training budget and performance. [3] introduces a novel curriculum learning approach for training efficiency through firstly learning simpler patterns, then progressively introducing more complex patterns. We will add these related studies to the \"Related Work\" section of the paper for comparison.\n"}, {"review_id": "EkOAvf4R55", "reviewer": "Reviewer_ShVL", "summary": "This paper proposes a new method for reusing pre-trained models to a larger target model. They propose mango which decomposes the parameters into 4 smaller matrices and then learns to grow these parameters. They show their method requires less FLOPS to achieve as good as performance other baselines across vision and NLP tasks. ", "questions": "Is the acceleration in terms of FLOPS or wall clock time? \nHow does the multiring operator allow for interactions between all weights in the model?\nHow does S_L capture the relationship between layers?  \n", "limitations": "See weaknesses above. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper was easy to follow and the experiments are strong across various domains. \nMango seems well motivated in allowing for interactions between all weights in the model. \n", "weaknesses": "Missing background on tensor ring matrix product - since this is the main method proposed\nAssuming the acceleration ratio is in terms of FLOPS, then there are no experiments showing the wall-clock time of Mango. Decreasing FLOPS is only useful if it decreases wall-clock time (assuming the same machine). \nThe models seem to be small, only growing from small to base variants, not base to large variants \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the meaningful comments!\n\n> Missing background on tensor ring matrix product - since this is the main method proposed\n\nThanks for the suggestion. We will add the background of TR-MPO in the \"Preliminary\" section.\n\n> Is the acceleration in terms of FLOPS or wall clock time? \n> \n> Assuming the acceleration ratio is in terms of FLOPS, then there are no experiments showing the wall-clock time of Mango. Decreasing FLOPS is only useful if it decreases wall-clock time (assuming the same machine).\n\nYes, we calculate the acceleration ratio in terms of FLOPS. We also uploaded the wall time pdf.\n\n> The models seem to be small, only growing from small to base variants, not base to large variants\n\nWe construct an experiment to train a large model ViT-G-1B. We plot the early result in the uploaded figure pdf due to the short rebuttal period. Although the scratch model has not reached a high position, the tendency shows our acceleration on it.\n\n> How does the multiring operator allow for interactions between all weights in the model? \n>\n> How does S_L capture the relationship between layers?\n\nAs shown in Equation (5), each element within the target model $M_2$ results from the summation of the product between trainable weights and each weight from the small model $M_1$, thereby achieveing interactions between all weights in the model.\n\nAdditionally, Mango assigns trainable weights (denoted as $\\mathbf{S}_L$) to facilitate the learning of layer importance within smaller models."}, {"review_id": "sS0BuDhtX8", "reviewer": "Reviewer_9N2L", "summary": "This paper addresses the issue of high resource costs in training large AI models from scratch. To this end, the authors propose a novel approach that linearly correlates each weight of the larger target model to all weights of the smaller pre-trained model, employing multi-linear operators to decrease complexity and manage resource demands. Experimental results on ImageNet reveal that the proposed method can significantly reduce training costs while maintaining performance.", "questions": "1.\tIn the proposed method, the authors first concatenate weights across layers. However, given that the row count of $\\mathbf{W}_j^{OUT} \\in kI \\times O$ differs from that of other weight matrices, how is this discrepancy addressed? Elaborating on this point would enhance the clarity of the method.\n\n2.\tIn Equation (5), the authors suggest decomposing $\\mathbf{S}$ using a Tensor Ring Matrix Product Operator (TP-MPO) into four smaller tensors. I'm curious to know if it would be possible to decompose $\\mathbf{S}$ into five or six smaller tensors.\n\n3.\tEssential details seem to be lacking in the experimental section. Could the authors clarify how the acceleration ratio was measured? It would be better to provide more descriptions, which would strengthen the paper.\n\n4.\tThere are a few statements in the experiments section that cause confusion. The authors mention that the Mango operators are trained for 100 steps, which is claimed to take negligible time. However, they also state that the model is trained for 300 epochs. Could the authors clarify these seemingly contradictory statements?\n\n5.\tIn the experiments, the authors measure the training cost using FLOPs, which is a theoretical metric. It might be more insightful if the authors could also offer measurements related to GPU training hours, providing a more practical perspective on computational cost.\n\n6.\tThe proposed method significantly reduces training costs compared to the traditional 'training from scratch' approach, primarily by leveraging the knowledge of the pre-trained model. I am intrigued to understand if this approach would sustain its comparable performance if we were to remove the training cost restriction and instead train until convergence.\n", "limitations": "The authors have provided a dedicated discussion on the potential limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The main idea is well-written and easy to follow. The introduced Mango Operator is both innovative and intuitive, exhibiting a remarkable ability to reduce training costs.", "weaknesses": "The experimental section lacks essential details, particularly around how the acceleration ratio is measured. Also, there are potentially contradictory statements about training steps and epochs, and the use of only theoretical metrics such as FLOPs for measuring training cost. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thorough feedback provided by the reviewer!\n\n> In the proposed method, the authors first concatenate weights across layers. However, given that the row count of $\\mathbf{W}_j^{OUT}\\in \\mathbb{R}^{kI\\times O}$ differs from that of other weight matrices, how is this discrepancy addressed? Elaborating on this point would enhance the clarity of the method.\n\nThanks for the suggestion. In the implement of Mango, we permute $\\mathbf{W}_j^{OUT}\\in \\mathbb{R}^{kI\\times O}$ to the shape of $I\\times kO$. We will revise Figure 4 to fix this confusion.\n\n> In Equation (5), the authors suggest decomposing $\\mathbf{S}$ using a Tensor Ring Matrix Product Operator (TP-MPO) into four smaller tensors. I'm curious to know if it would be possible to decompose $\\mathbf{S}$ into five or six smaller tensors.\n\nYes, it is possible to decompose $\\mathbf{S}$ into five, six even more smaller tensors. The reason for decomposing $\\mathbf{S}$ into four smaller tensors is to interpret it as the transformation of the four modes of the smaller models, as explained in Line 183. To be specific, $\\mathbf{S}_B$ can capture the interactions among weights in the same layer. $\\mathbf{S}_I$ and $\\mathbf{S}_O$ denote the transformations of input and output dimensions, respectively. $\\mathbf{S}_L$ indicates the relationship among layers. $R$ means the low-rank level of $\\mathbf{S}$. Smaller $R$ means less correlation of these four tensors. Therefore, Mango can correlate all the weights in the small model, thereby achieving the full-mapping transformation, which demonstrates the effectiveness in training efficiency.\n\n>  The experimental section lacks essential details, particularly around how the acceleration ratio is measured. \n>  \n> Essential details seem to be lacking in the experimental section. Could the authors clarify how the acceleration ratio was measured? It would be better to provide more descriptions, which would strengthen the paper.\n\nThanks for your suggestion. We calculate the acceleration ratio items of the FLOPs. For example, assuming training from scratch converge to metric (accuracy or loss) value $\\psi$ with FLOPs $a$, if Mango reach $\\psi$ with FLOPs $b$, then the acceleration ratio can be calculated as \n$$\n1-\\frac{b}{a}.\n$$\nWe will add a formulation to clarify the calculation of the acceleration ratio.\n\n> Also, there are potentially contradictory statements about training steps and epochs\n> \n> There are a few statements in the experiments section that cause confusion. The authors mention that the Mango operators are trained for 100 steps, which is claimed to take negligible time. However, they also state that the model is trained for 300 epochs. Could the authors clarify these seemingly contradictory statements?\n\nSorry for the confusion. There are not contradictory statements for the training steps. We write the completed procedures of Mango at Line 194 of the paper. Specifically, the implement of Mango is divided into two phases. (1) Training Mango operators. We train the Mango operators (i.e., $\\mathbf{S}_B$, $\\mathbf{S}_I$, $\\mathbf{S}_O$, and $\\mathbf{S}_L$) for 100 steps for transfer the knowledge from the smaller pretrained model to the target model. This training cost is relatively small (usually several minutes), which can be negligible in comparison with the overall training of large models; (2) Training the target model. We derive the weight $W_2$ of the target model through Eq. (5.) with the trained Mango operators. Then, we train the target model initialized by $W_2$ for 300 epochs.\n\n>  the use of only theoretical metrics such as FLOPs for measuring training cost.\n>\n> In the experiments, the authors measure the training cost using FLOPs, which is a theoretical metric. It might be more insightful if the authors could also offer measurements related to GPU training hours, providing a more practical perspective on computational cost.\n\nThanks for the suggestion. We upload the wall time pdf. \n    \n> The proposed method significantly reduces training costs compared to the traditional 'training from scratch' approach, primarily by leveraging the knowledge of the pre-trained model. I am intrigued to understand if this approach would sustain its comparable performance if we were to remove the training cost restriction and instead train until convergence.\n\nWhen removing the training cost restriction, Mango can achieve better performance at the end of training. For example, in the experiment of \"DeiT-S -> DeiT-B\", Mango can improve the accuracy from 80.45% to 80.75% in the end. Similarly, Mango also achieves loss reduction as \"BERT-Small -> BERT-Base\": 1.47->1.41, \"GPT-Small -> GPT-Base\": 2.49->2.46."}, {"review_id": "Z0bXeaQqxc", "reviewer": "Reviewer_cdCX", "summary": "The authors propose a new approach, Mango, to use a small model as initializer to larger models in order to speed up the training of the larger model. Mango proposes to learn a mapping of weights from the smaller model to the larger model, not layer by layer or other restricted form of mapping, but a full mapping of weights.\n\nBecause the matrix that would represent this linear mapping of weights being huge (in the order of M x N where M and N are respectively the number of parameters of the small and the large model), Mango proposes a particular form of low rank factorization, allowing to keep a full mapping but constraining each of its sub-mapping to rank 1. The authors show that this mapping can be learned in as few as 100 steps, making its computing cost negligible.\n\nBy doing so, Mango shows that full mappings are useful, and shows faster convergence than the other methods on 3 types of architectures : vision transformers, BERT models and GPT models.", "questions": "* The authors mention the CO2 consumption of training large models. Do you have an estimate of those emissions and the type of reduction achieved by Mango? Is the improvement proportional to the improvement in FLOPs?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Well written paper, with very good motivating examples\n* Good theoretical part, with the need of low rank factorization clearly introduced\n* Compelling results of 3 different models: Vision Transformers, BERT and GPT\n* The gaps in the results are significant: on vision transformers, Mango converge in 2/3 of the time of the next best method, and in 1/4 of the time of naive from scratch training. The savings on BERT are less important but still quite significant.\n", "weaknesses": "The results are constrained to the case of Small to Base size transformers mapping :\n* Deit-S to Deit-B\n* BERT-S to BERT-M\n* GPT-Base from GPT-Small\n\nThe most interesting regime for this class of method is with bigger transformers since they are the most power hungry. The paper would gain a lot of impact by showing that Mango works for larger models (for instance to initialize a ViT-G from a ViT-H). I would definitely increase my rating if such a proof was shown, especially with a low rank such as 1.\n\nAnother interesting result would be to know, as we deal with larger models, whether it is best to initialize them with the larger pre-trained available, or if we can initialize them with Mango from a small model. For example, to initialize a ViT-G, should we preferentially use a ViT-H or ViT-L ? And how much would the difference be ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable comments!\n\n> The most interesting regime for this class of method is with bigger transformers since they are the most power hungry. The paper would gain a lot of impact by showing that Mango works for larger models (for instance to initialize a ViT-G from a ViT-H). I would definitely increase my rating if such a proof was shown, especially with a low rank such as 1.\n>\n> Another interesting result would be to know, as we deal with larger models, whether it is best to initialize them with the larger pre-trained available, or if we can initialize them with Mango from a small model. For example, to initialize a ViT-G, should we preferentially use a ViT-H or ViT-L ? And how much would the difference be?\n\nWe construct an experiment to train a large model ViT-G-1B. We plot the early result in the uploaded figure pdf due to the short rebuttal period. Although the scratch model has not reached a high position, the tendency shows our acceleration on it. The results also suggest a larger model will have more benefits for Mango.\n\n> The authors mention the CO2 consumption of training large models. Do you have an estimate of those emissions and the type of reduction achieved by Mango? Is the improvement proportional to the improvement in FLOPs?\n\nYes. The improvement is proportional to the improvement in FLOPs accoring to [1].\n\n[1] David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, Jeff Dean: Carbon Emissions and Large Neural Network Training. CoRR abs/2104.10350 (2021)"}]

Author rebuttal:
We deeply appreciate the insightful and considerate feedback provided by all of the reviewers.  We are glad that the reviewer kqva notes that efficient training is practically important, and all the reviewers (u4ot, kqva, ShVL, 9N2L, and cdCX) consider the proposed Mango can achieve significant acceleration on the training process. We are also appreciative of the reviewers (kqva, ShVL, 9N2L, and cdCX) who consider that Mango is well-motivated, novel, and interesting. Moreover, we also thank reviewers (9N2L and cdCX) who commend the clarity and accessibility of our writing. In addition, we have made earnest endeavors to thoroughly address all the inquiries raised by the reviewers, encompassing additional experimental comparisons and clarifications. The extra empirical results have been plotted and are readily accessible within the uploaded PDF document.

Meta-review: This paper proposes an approach reducing the required training time for a model by reusing the weights of a smaller network.  Through the course of the reviewing process the authors and reviewers discussed the merits of the proposed approach, with all reviewers recommending acceptance.  The ACs find that there is not sufficient justification to overturn the unanimous recommendation of the reviewers, and remind the authors to use the comments and rebuttals to update their paper given the reviewer's feedback.
Decision: Accept (poster)