Title: Reusing Pretrained Models by Multi-linear Operators for Efficient Training
Venue: NeurIPS 2023
Accepted: yes
Keywords: Model Growth

Efficient Training

Pretrained Model

Multi-linearity
Abstract: Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model (termed the ``target model''), leading to a considerable acceleration in training. Despite the successes of these previous studies, they  grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interactions among the weights of both the pretrained and the target models. As a result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, we propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that our method can save 76\% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12\% and LiGO by +21\%, respectively.

Reusing Pretrained Models by Multi-linear Operators for Efficient Training

Abstract

Training large models from scratch usually costs a substantial amount of resources. Towards this problem, recent studies such as bert2BERT and LiGO have reused small pretrained models to initialize a large model (termed the 'target model'), leading to a considerable acceleration in training. Despite the successes of these previous studies, they grew pretrained models by mapping partial weights only, ignoring potential correlations across the entire model. As we show in this paper, there are inter- and intra-interactions among the weights of both the pretrained and the target models. As a result, the partial mapping may not capture the complete information and lead to inadequate growth. In this paper, we propose a method that linearly correlates each weight of the target model to all the weights of the pretrained model to further enhance acceleration ability. We utilize multi-linear operators to reduce computational and spacial complexity, enabling acceptable resource requirements. Experiments demonstrate that our method can save 76% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT by +12.0% and LiGO by +20.7%, respectively.

1 Introduction

Transformers [47] have recently achieved great successes in various scenarios [11, 4, 20]. Generally, Transformers tend to be larger for more expressive power and better performance (e.g., ViTG [6] and GPT-3 [3]). As the size of models continues to grow, training Transformers takes longer and can result in higher CO 2 emissions, which conflicts with the principles of Green AI [43]. Thus, training Transformers efficiently is crucial not only

Wider

Higher

for financial gain but also for environmental sustainability [5]. To achieve efficient training, it is a wise option to grow a pretrained small model into a larger one, since the pretrained small model has already learned knowledge from the data, which allows for faster training compared to starting from scratch. [13]. Moreover, there are numerous pretrained models that are easily accessible [53], reducing the cost of utilizing a smaller pretrained model. Furthermore, empirical evidence shows that Transformers have inductive biases that facilitate scaling fitting [42, 22]. This demonstrates the feasibility of learning from pretrained models [53].

S-(L10, A6)

S-(L11, A5)

B-(L10, A4)

S-(L10, A7)

Example 0

S-(L5, A7)

S-(L10, A7)

B-(L11, A9)

B-(L11, A12)

Example 1

S-(L9, A2)

S-(L10, A2)

B-(L12, A1)

B-(L12, A11)

Example 3

S-(L11, A2)

S-(L11, A3)

S-(L10, A5)

B-(L11, A4)

Example 2

1H

The concept of reusing pretrained models essentially involves exploring the efficient mapping from the pretrained model to the target model. This mapping process can be represented as linear growth operators, as depicted in Figure 1. One viable approach to mapping is to leverage knowledge from other weights. For instance, StackBERT [13] utilizes low-layer information to construct new higher layers, by duplicating layers to increase depth during training epochs. bert2BERT [5] expands width through the expanding operator of Net2Net [7]. Differently, bert2BERT utilizes weights of neighbor layers for enhancing the ability, which also shows the benefit of taking knowledge from other weights. Moreover, distinct from the aforementioned research directly employing a fixed operator, LiGO [53] trains expanding operators which are tied with layers to achieve superior transfer accuracy by considering knowledge from other layers, which is advantageous for training efficiency. However, they neglect the potential connectivity between models.

Potential Connectivity between Models. The attention maps of BERT, as depicted in Figure 2, indicate that there are similarities not only in the weights of the same or different layers, but also across different models. These similarities suggest that we can leverage a full mapping transformation to capture this correlation in both inter-layer and intra-layer connections, even across different scaled models. By doing so, we can reuse the parameters of pretrained small models and accelerate the training of larger models. However, previous studies have overlooked the overall network connectivity and have instead expanded each weight through partial mapping transformation. For instance, Net2Net [7] only considers preserving functionality by transforming weights individually, while bert2BERT [5] increases model width head by head in Transformers. On the other hand, LiGO [53] primarily focuses on extending the weights of the same type (e.g., query, key, and value in Transformers), which heavily affects the performance of the transferred models.

Multi-linear Operators. Based on the above observations, we propose mapping the entire pretrained model to each weight of the target model, instead of employing a partial transformation. This approach can ensure high expressive power, promoting the potential connectivity in models for better training efficiency. However, the completed mapping has a huge parameter tensor, occupying an enormous space (see Section 3.2). To address this issue, we propose Mango, a multi-linear structure (i.e., tensor ring [33, 52, 51]), which decomposes the large mapping tensor into four smaller tensors that are bonded by ranks to construct multi-linear operators. Mango allows for efficient training by reducing the space required for the mapping tensor. Formally, Mango can be considered as a generalization of LiGO and bert2BERT (see Section 3.3). By using the full mapping approach, Mango can save 76% computational costs on DeiT-base transferred from DeiT-small, which outperforms bert2BERT [5] by +12.0% and LiGO [53] by +20.7%, respectively.

2 Related Work

Efficient Training from Scratches. Model scratches can be roughly regarded as models without knowledge priors. Some training strategies for scratches are universal and orthogonal to our method.

For example, Adam [23] and large-batch size training [66] accelerate the training process from angles of optimizer and magnitude of input data, respectively. Wang et al. [50] enable the training of very deep neural networks with limited computational resources via a technique called active memory. Shoeybi et al. [45] use mixed precision training to assist training. Low-rank methods benefit training for less memory and time [21]. Wu et al. [59] take notes of rare words for better data efficiency. Dropping layers [68], knowledge inheritance [38], and merging tokens [2] are also efficient methods for training. Another line of work [13, 63, 14, 44] is termed progressive training, which gradually increases the model size within the training process for training efficiency. Li et al. [27] employ a neural architecture search (NAS) method to search optimal sub-networks for progressive training on ViTs. Xia et al. [60] suggest a three-phase progressive training regime to achieve a good trade-off between training budget and performance. Wang et al. [55] introduces a novel curriculum learning approach for training efficiency through firstly learning simpler patterns, then progressively introducing more complex patterns.

Efficient Training from Pretrained Models. Pretrained models usually contain abundant data knowledge, which is helpful for training [32]. By preserving the function of a pretrained model while expanding the model size, it is feasible to give the corresponding larger model an initial state with high performance. Net2Net [7] is the first work to propose the concept of function-preserving transformations by expanding width by splitting neurons and growing depth with identity layers. However, Net2Net splits neurons randomly. Towards this problem, a series of studies [56, 58, 49, 57] propose to select the optimal subset of neurons to be split by utilizing functional steepest descent. bert2BERT [5] expands small transformers by following the function preserving idea. Recently, LiGO [53] utilizes a trainable linear operator to learn a good expanding formula. Different from these prior studies, our method tries to implement a full mapping that achieves comprehensive utilization of the whole smaller model.

Neural Network Initialization. Our method is related to neural network initialization techniques. Xavier [12] and Kaiming [16] initialization aim to control input variance equal to that of output. Generalizing Xavier and Kaiming methods, a universal weight initialization paradigm proposed by Pan et al. [34] can be widely applicable to arbitrary Tensorial CNNs. In addition, Hardt and Ma [15] has shown theoretically that network training benefits from maintaining identity, particularly for improving the efficiency of residual networks. Fixup [67] and ZerO [70] both set residual stem to 0 (not residual connections) to ensure the identity of signals, thereby successfully initializing ResNets. SkipInit [8] replaces Batch Normalization with a multiplier whose value is 0. ReZero [1], on the other hand, adds extra parameters of value 0 to maintain identity, resulting in faster convergence. IDInit is an initialization approach to keep the identity matrix for stable training of networks [36]. In comparison, our work explores fully reusing smaller pretrained models as efficient initialization.

Low-rank Techniques in Neural Networks. Low-rank methods are feasible for reducing spatial and temporal complexities in neural networks [52, 35, 62]. For example, Idelbayev and CarreiraPerpiñán [19] uses matrix decomposition to compress convolutional neural networks (CNNs) for faster inference. LoRA [17] applies low-rank matrices for fine-tuning large language models (LLMs) in affordable resources. As a parameter-efficient tuning (PETuning) method [69] for federated learning, FedPara [18] utilizes the Hadamard product on low-rank parameters for reducing communication time in federated learning. Pan et al. [33] and Li et al. [28] use tensor ring decomposition for reducing the size of neural networks. Tucker decomposition and block-term Tucker decomposition have been used in T-Net and BT-layers [24, 64] for improving the model performance, respectively. Ma et al. [30] apply block-term Tucker to compress Transformers. Yin et al. [65] propose to use ADMM to optimize tensor training to achieve better compression and performance. These techniques have built a solid foundation for our work to implement multi-linear transformation for transferring knowledge from pretrained models to target models.

3 Mango Operator

This section presents the proposed Mango operator. First, we provide an overview of the tensor diagram, the concept of the tensor ring matrix product operator (TR-MPO), and the formulation of Transformer architecture in Sec. 3.1. Next, we delve into the details of the proposed multi-linear mapping operator (i.e., Mango) in Sec. 3.2. Finally, we compare Mango with recent advances in terms of tensor diagrams in Sec. 3.3.

3.1 Preliminary

Tensor Diagram. Tensor decomposition is a technique that involves splitting a tensor into multiple smaller tensors, typically more than the two matrices that result from matrix factorization. To better illustrate the interactions among multiple tensors, tensor diagrams are often used. A tensor diagram is com-

A 3rd-order Tensor

Tensor Contraction

posed of two primary elements: a tensor vertex and a tensor contraction. A tensor vertex represents a tensor, and its order is determined by the number of edges connected to it. Each edge is assigned an integer that indicates the dimension of the corresponding mode. As shown in Figure 3, a 3rd-order tensor T ∈ R i 0 × i 1 × i 2 can be drawn as a circle with three edges. The process of taking the inner product of two tensors on their matching modes is called tensor contraction. For example, Two tensors, M ∈ R i 0 × i 1 × i 2 and N ∈ R j 0 × j 1 × j 2 , can be contracted in corresponding positions to form a new tensor of R i 0 × i 1 × j 2 × j 3 , when they have equal dimensions: i 2 = j 0 /defines e 0 . The contraction operation can be formulated as

The tensor diagram is an elegant tool for succinctly visualizing and manipulating multi-dimensional arrays or tensors. By representing tensor operations as networks of nodes and edges, these diagrams provide an intuitive way to understand the underlying mathematical operations and their interactions. As a result, we have chosen to use tensor diagrams to illustrate our method and to analyze the connections between our approach and prior studies.

Tensor Ring Matrix Product Operator (TR-MPO). TR-MPO means tensor ring (TR) of an MPO[37] format. Given a 2 N -order tensor X ∈ R I 1 × J 1 × I 2 × J 2 ...I N × J N , its TR-MPO decomposition can be mathematically expressed as

where { R 1 , R 2 , . . . , R N } denote the ranks, G ( n ) ∈ R R n × I n × I n × R n +1 denotes a 4th-order core tensor and R 1 = R N +1 , which indicates ring-like structure.

Transformer Architecture. The Transformer [47] is a deep learning architecture that has revolutionized the artificial intelligence field including computer vision (CV) and natural language processing (NLP). As shown in Figure 4, a Transformer block consists of two main sub-layers: the multi-head self-attention (MHSA) layer and the feed-forward neural network (FFN) layer.

(1) MHSA Layer. The MHSA layer in the Transformer block computes the attention scores between each input element and every other element, allowing the model to attend to different parts of the input sequence during processing. Inputs of MHSA are query matrix Q ∈ R I , key matrix K ∈ R I and value matrix V ∈ R I with parameters W Q ∈ R I × O , W K ∈ R I × O , W V ∈ R I × O , W O ∈ R I × O . Usually, it constrains I = O . Moreover, MHSA separates these parameters into n heads: { W Q,i } n , { W K,i } n , { W V,i } n , { W O,i } n . The MHSA mechanism can be formulated as follows:

where d k is the dimensionality of the key vectors. The self-attention mechanism is performed multiple times in parallel, each time using a different set of learned parameters W Q,i , W K,i , and W V,i to compute multiple "heads" of attention. The resulting attention heads are linearly transformed by a learned weight matrix W O,i to produce the output. At last, MHSA concatenates the output as the final result of the MHSA layer.

(2) FFN Layer. The FFN layer in the Transformer block is responsible for applying a non-linear transformation to the output of the self-attention layer. X ∈ R I is the input. The weights are

Multi-Head

Self-Attention

Add & Norm

Feed-Forward

Network

Add & Norm

Scaled Dot-Production

Attention

Concatenate

Large Model

Small Model

Full Mapping Operator

TR-MPO

The  -th Layer of a Transformer

Concatenate

Layers

The whole parameters of a Transformer

Model Growth

Permute

W IN ∈ R I × kO and W OUT ∈ R kI × O , where usually I = O and k is a ratio that is often set to 4. The FFN layer can be formulated as

We neglect biases in the formulation as it is usually set to 0 at initialization. The output of the FFN layer is obtained by applying two linear transformations to the output of the MHSA layer.

Finally, both the self-attention output and the FFN output are processed through a residual connection and layer normalization to prevent the model from collapsing or overfitting to the training data. Apparently, the parameters in a Transformer are mainly based on its linear transformation matrices, i.e., W Q , W K , W V , W O , W IN and W OUT .

3.2 Multi-linear Operator

The essential of model growth is to transfer the knowledge from the small model to the bigger counterpart. Prior study [5] finds that taking weights from the neighbor layer as initial parameters can further improve the convergence speed, which gives the insights that knowledge from other layers can help training as there are many similar attention maps among layer. Nevertheless, the fact is that this similarity exists all over the pretrained model as shown in Figure 2, which motivates us to consider the possibility of whether we can utilize the knowledge from all weights. Based on this consideration, we construct a mapping of the whole model. As the correlation is hard to formulate heuristically, we learn the mapping parameter to implicitly transfer the pretrained model.

Notation. Here, we give the notations for clearly formulating our method. A model with L layers and a D hidden size can be denoted as M ( L, D ) . Following Section 3.1, parameters of j -th layer are a set θ j = { W Q j ∈ R I × O , W K j ∈ R I × O , W V j ∈ R I × O , W O j ∈ R I × O , W IN j ∈ R I × kO , W OUT j ∈ R kI × O } , j ∈ [ L ] , usually satisfying I = O = D . The weight of M ( L, D ) is θ L,D = { θ j } L j =1 . A growth operator with parameter S is denoted as Φ S . A mapping from M ( L 1 , D 1 ) → M ( L 2 , D 2 ) can be denoted as θ L 2 ,D 2 = Φ S ( θ L 1 ,D 1 ) . In the case of L 1 < L 2 , this mapping represents a growth mapping. After growing, θ L 2 ,D 2 will be used as initial weights for the target model.

Full Mapping Operator. To utilize all weights for knowledge transferring, we first concatenate weights across layers. As shown in Figure 4, we concatenate j -th layer to form a tensor of shape B × I × O along with order I and O where B = 2 k + 4 . Then the final weight tensor M ∈ R B × I × O × L can be derived by combining the concatenated tensors of L layers. Giving

bert2BERT

LiGO

Mango

a small model M 1 ∈ R B 2 × I 1 × O 1 × L 1 , a big model M 2 ∈ R B 1 × I 2 × O 2 × L 2 , and a parameter S ∈ R B 1 × I 1 × O 1 × L 1 × B 2 × I 2 × O 2 × L 2 of a full mapping operator Φ S , we can transfer M 1 to M 2 as

where b 1 , i 1 , o 1 , l 1 , b 2 , i 2 , o 2 , and l 2 are entries of corresponding tensors.

Multi-linear Mapping Operator. Note that S is extremely big, which makes it infeasible to display this mapping by applying a whole transformation tensor. Therefore, we propose a multi-linear operator named Mango which decomposes S through a tensor ring matrix product operator (TR-MPO) in four small tensors { S B ∈ R R 1 × B 1 × B 2 × R 2 , S O ∈ R R 2 × O 1 × O 2 × R 3 , S L ∈ R R 3 × L 1 × L 2 × R 4 , S I ∈ R R 4 × I 1 × I 2 × R 1 } . R = { R 1 , R 2 , R 3 , R 4 } is the rank of TR-MPO. Then, we can update Φ S to Φ S B , S I , S O , S L , and Eq. (5) can be reformulated with a multi-linear form as

The total size of S B , S I , S O , and S L is exponentially less than S , which makes Mango viable for practical implementation while maintaining the full correlation between the small and big models.

Training Target. After designing the multi-linear operators, we train these operators to obtain the function preserving M 2 . The training target can be denoted as

where L is a loss function, and D is a data distribution. By replacing the full space of S with four small spaces, the spatial requirements of the training process are reduced exponentially.

Procedures of Applying Mango. The procedures of Mango can be divided into four steps: (i) concatenating weights θ L 1 ,D 1 of a pretrained model M ( L 1 , D 1 ) to construct a tensor M 1 ; (ii) training the growth operator Φ S B , S I , S O , S L items of Eq. (7) in a few steps (e.g., 100) to make transferred models maintaining function; (iii) recovering weight tensor M 2 through the multi-linear operator Φ S B , S I , S O , S L ; (iv) splitting M 2 to the weight θ L 2 ,D 2 of the target model M ( L 2 , D 2 ) as initialization to continue training.

Remark. Tensor decomposition is often used to compress neural networks. Actually, there are spaces for further reducing the operator size. However, we are not exploring the border of compression ratio, but the possibility of the learning ability of multi-linear operators and the influence on model growth. Therefore, we decompose the huge S into four interpretable smaller tensors. S B means the interactions on the parameters in the same layer. S I and S O denote the transformations of input and output dimensions in one parameter, respectively. S L indicates the relationship among layers. R means the low-rank level of S . Smaller R means less correlation of these four tensors.

3.3 Comparison with Recent Advances

Basically, the goal of model growth is to widen and deepen the width and depth of the pretrained models, and operation on width and depth can be easy to implement. Therefore, most of the previous studies grow width and depth separately, although there may exist some correlation that may influence the training efficiency. In this part, we analyze the difference among methods of model growth with the help of the tensor diagrams.

We illustrate a comparison among bert2BERT, LiGO, and Mango in Figure 5. The red circle means a pretrained model, while the blue and gray circles denote trainable and untrainable operator parameters, respectively. The circle with an oblique stroke represents a super-diagonal tensor, where the diagonal elements are all 1. This super-diagonal tensor means that growth operators expand other modes along the mode of the super-diagonal tensor.

S

S

S

S

bert2BERT. As shown in Figure 5, the parameters in the growth operator of bert2BERT are all frozen, since they are designed by heuristic inspiration to preserve function. bert2BERT expands modes of I 1 and O 1 along with the mode of L 1 , indicating that one weight is expanded without knowledge from any other weight. To further increase the training ability, bert2BERT applies S K to construct Advanced Knowledge Initialization (AKI) to utilize knowledge of other layers to help accelerate training.

LiGO. Different from bert2BERT, the LiGO operator can be trained. In addition, LiGO expands I 1 and O 1 along the mode of B 1 . With S L , LiGO can combine knowledge among layers. Operators of Net2Net and StackBERT grow with or depth separately, which can be regarded as a sub-solution to S I , S O , and S L of LiGO. However, one weight in LiGO will not leverage the knowledge of weights in the same layer. Therefore, LiGO only implements partial mapping like bert2BERT.

Mango. We employ Mango on each mode of the weight M 1 . Mango can approach the full mapping tensor S as a low-rank approximation with rank R , rather than partial mapping operators like LiGO and bert2BERT. Therefore, Mango can obtain M 2 to capture adequate knowledge from pretrained weights as formulated in Eq. 6. Moreover, the diagonal tensor in LiGO and contraction between a diagonal tensor and S K in bert2BERT are subsets of S L and S B in Mango. Therefore, Mango can be a generalization of bert2BERT and LiGO with more expressive power.

4 Experiment

In this section, we design a set of experiments to validate the proposed Mango. To begin with, we conduct an ablation study to analyze the influence of Mango on width and depth in Sec. 4.1. Then we conduct image classification with DeiT-B [46] on ImageNet to show the acceleration in computer vision task in Sec. 4.2. Later we conduct a pretraining experiment with BERT-Base [10] to demonstrate the effectiveness in natural language processing tasks in Sec. 4.3. At last, we also employ a pretraining experiment with GPT-Base [39] to show the wide adaption of Mango in Sec. 4.4.

Ratio of saving FLOPs. We evaluate the training efficiency in terms of the ratio of saved floatingpoint operations (FLOPs). Let us assume that training a model from scratch to convergence on metric Ψ (e.g., MLM loss or accuracy) necessitates ξ Scratch FLOPs. Then, for a method that reaches the same metric Ψ with ξ ∗ FLOPs, its FLOP saving ratio r can be computed as

(a) Width: DeiT-T-A

DeiT-S.

(b) Depth: DeiT-T-B

DeiT-S.

(c) Both: DeiT-T-C

DeiT-S.

→

→

→

4.1 Ablation Study

In this experiment, we explore the influence of Mango on growing width, depth and both of them in addition to rank setting. We use three tiny vision Transformers (ViTs) [11], i.e., DeiT-T-A, DeiT-T-B, and DeiT-T-C, for growing to DeiT-S [46] on ImageNet [9]. Structures of these DeiTs can be found in Table 4 of the Appendix. For ease of setting, we set all the ranks the same in the range of { 1 , 4 , 7 , 10 } . We train Mango operators for 100 steps, which only requires negligible time. We use Adam with learning rate 1e-3 and weight decay 1e-2 for 300 epoch optimization. The batch size is 1024.

Results are shown in Figure 6. Mango achieves acceleration of at most 31.0%, 46.0%, and 41.3% on expanding width, depth, and both of them, respectively, which shows Mango can fit various growth scenarios. Interestingly, these acceleration ratios are higher when the operator accuracies are better along the types of pretrained models, e.g., as DeiT-T-B at most achieves 73.81% accuracy which is higher than 72.89% of DeiT-T-A, DeiT-T-B derives higher acceleration ratio. This phenomenon suggests that when there are multiple pretrained models can be selected, the better accuracy for one pretrained model through Mango, the faster training speed can be obtained for the target models.

In each expanding case, the accuracies of operator training tend to increase with higher ranks. However, all the cases on the three pretrained models show that better accuracy will not lead to faster training with the same pretrained model. For example, in Figure 6(a), the operator with rank 10 has an accuracy that is 1.19% higher than the operator with rank 1. Nevertheless, the two ranks reach the same acceleration ratio. This result suggests that rank 1 is enough to use, and also enjoys the advantages of spatial and temporal complexities compared to bigger ranks. And we choose rank 1 for Mango to construct the later experiments.

4.2 Results on Large-Scale Vision Transformer

We conduct the experiment to show the training acceleration on large-scale vision Transformers. We train the Deit-B from Deit-S [46] on ImageNet. Structures of the two models are shown in Table 4 of the Appendix. Ranks of Mango are all set to 1 for complexity benefit without performance loss. The operators of Mango and Ligo are all trained within 100 steps. We use Adam as the optimizer with learning rate 1e-3 and weight decay 1e-2. The batch size is 1024. The training epoch is 300.

Results are shown in Figure 7(a). Mango saves 76% FLOPs from Scratch which converges to an accuracy of 80.45% in the end. Compared with schemes of training from Scratch (i.e., Srcatch and StackBERT), methods of training from a smaller model (i.e., Mango, bert2BERT, and LiGO) can attain 70% accuracy in a short time, even bert2BERT start in low accuracy. Compared with the recent SOTA models, Mango has surpassed bert2BERT for +12.0%, and LiGO for +20.7%. This experiment shows the ability of Mango to achieve significant improvement in training acceleration. To investigate the influence of Mango on transferring ability, we also conduct an experiment on downstream tasks, including CIFAR10 [26], CIFAR100 [26], Flowers [31], Cars [25], and ChestXRay8 [54]. Results are shown in Table 2. It is easy to see that Mango achieves similar results to the Scratch, which indicates Mango has not influenced the transferring ability.

(a) DeiT-S

DeiT-B.

(b) BERT-Small

BERT-Base.

(c) GPT-Small

GPT-Base.

→

→

→

4.3 Pretraining on BERT

In this experiment, we conduct the validation to show the training acceleration on BERT [10, 61]. The dataset is the concatenation of English Wikipedia and Toronto Book Corpus [71]. We train the BERT-Base from BERT-Small. The training epoch is 40. The batch size is 768. We list the structures of the two models in Table 5 of the Appendix. The ranks of Mango are all 1. Mango and LiGO are both warmly trained for 100 steps. The optimizer is set to AdamW. The learning rate is 1e-4 and the weight decay is 1e-2.

We illustrate the training curves in Figure 7(b). The methods of training from BERT-Small (i.e., Mango, bert2BERT, and LiGO) can all surpass the progressive training StackBERT of acceleration ratio 29.5%. bert2BERT is over StackBERT by +6.1%. Mango achieves the highest acceleration of 39.2% FLOPs which is +3.6% more than bert2BERT. Loss of StackBERT steeply decreases within training for Stacking layers, which can also demonstrate the pretrained weights are helpful for training efficiency. We show the effectiveness of Mango on SQuAD and GLUE benchmark as in Table 3. Mango shows the same transferring ability with faster convergence, which indicates a promising use for practical training.

4.4 Pretraining on GPT

We also implement the experiment to show the training acceleration on GPT [39]. The dataset is the concatenation of English Wikipedia and Toronto Book Corpus [71]. We train the GPT-Base from GPT-Small. The structures of the two models are shown in Table 5 of the Appendix. The ranks of Mango are all 1. Mango and LiGO are trained for 100 steps. We use Adamw with learning rate 1e-4 and weight decay 1e-2. The batch size is 512. The training epoch is 35.

We compare the Scratch model, bert2BERT, LiGO, and Mango in Figure 7(c). We observe that the proposed Mango achieves a 59.9% acceleration ratio. While GPT is different from BERT in different structures, including layer normalization, mask method, and training strategy, Mango can always keep the highest performance. Although LiGO is lower than bert2BERT at the beginning, it converges slower and reaches 38.1% at last. By contrast, the Mango is almost at the lowest loss in the whole training process and achieves +16.7% more than bert2BERT and +21.8% more than LiGO, which shows a significant acceleration than the baselines.

5 Conclusion

Training Transformers can pose a significant demand on computational resources. Reusing pretrained models as an initialization strategy for the target model offers a promising approach to reducing resource costs and accelerating training. However, previous studies only mapped partial weights when growing models, which may fail to consider potential correlations in the whole model and result in inadequate growth mapping. Inspired by this observation, we propose to consider the interaction among all weights in the model to further improve acceleration ability. Specifically, we utilize a full mapping to comprehensively reuse the pretrained model, taking into account all correlations between model weights. As the full mapping tensor is huge and cannot be employed in practice, we propose to use Mango, a multi-linear operator, to reduce computation and space complexity. Experimental results demonstrate that Mango consistently achieves significant acceleration on various large-scale models (e.g., DeiT-B, BERT-Base, and GPT-Base). In the future, we hope that this method can contribute to green AI and significantly reduce the cost of training Transformers.

Limitation. While Mango significantly reduces training costs for large models, it still requires weights of a small pretrained model as the prior knowledge. Additionally, the Mango operator necessitates extra training for initializing. However, obtaining small model weights is relatively simple within the community, and given the substantial cost savings, the resources required for training the operator are minimal.

Societal Impact. The acceleration from Mango significantly impacts energy conservation and environmental protection, promoting the growth of Green AI, which boosts efficiency and reduces computational demand, minimizing energy usage and carbon emissions. Mango can also democratize AI, allowing broader access to technologies of pertaining models without the need for large resources, and promoting sustainable and accessible AI solutions that respect our environmental limits.

References

Thomas Bachlechner, Bodhisattwa Prasad Majumder, Huanru Henry Mao, Gary Cottrell, and Julian J. McAuley. Rezero is all you need: fast convergence at large depth. In UAI , volume 161 of Proceedings of Machine Learning Research , pages 1352-1361. AUAI Press, 2021.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. CoRR , abs/2210.09461, 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS , 2020.

Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. CoRR , abs/2105.05537, 2021.

Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. In ACL (1) , pages 2134-2148. Association for Computational Linguistics, 2022.

Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In CVPR , pages 16123-16134. IEEE, 2022.

Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. In ICLR , 2016.

Soham De and Samuel L. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In NeurIPS , 2020.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , pages 248-255. IEEE Computer Society, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1) , pages 4171-4186. Association for Computational Linguistics, 2019.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR . OpenReview.net, 2021.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS , volume 9 of JMLR Proceedings , pages 249-256. JMLR.org, 2010.

Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. Efficient training of BERT by progressively stacking. In ICML , volume 97 of Proceedings of Machine Learning Research , pages 2337-2346. PMLR, 2019.

Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. On the transformer growth for progressive BERT training. In NAACL-HLT , pages 5174-5180. Association for Computational Linguistics, 2021.

Moritz Hardt and Tengyu Ma. Identity matters in deep learning. In ICLR (Poster) . OpenReview.net, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV , pages 1026-1034. IEEE Computer Society, 2015.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR . OpenReview.net, 2022.

Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. In ICLR . OpenReview.net, 2022.

Yerlan Idelbayev and Miguel Á. Carreira-Perpiñán. Low-rank compression of neural nets: Learning the rank of each layer. In CVPR , pages 8046-8056. Computer Vision Foundation / IEEE, 2020.

Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In NeurIPS , pages 14745-14758, 2021.

Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N. Gomez. Exploring low rank training of deep neural networks. CoRR , abs/2209.13569, 2022.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster) , 2015.

Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic. T-net: Parametrizing fully convolutional nets with a single high-order tensor. In CVPR , pages 7822-7831. Computer Vision Foundation / IEEE, 2019.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV Workshops , pages 554-561. IEEE Computer Society, 2013.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Changlin Li, Bohan Zhuang, Guangrun Wang, Xiaodan Liang, Xiaojun Chang, and Yi Yang. Automated progressive learning for efficient training of vision transformers. In CVPR , pages 12476-12486. IEEE, 2022.

Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu. Heuristic rank selection with progressively searching tensor ring network. Complex & Intelligent Systems , pages 1-15, 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Ming Zhou, and Dawei Song. A tensorized transformer for language modeling. In NeurIPS , pages 2229-2239, 2019.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In ICVGIP , pages 722-729. IEEE Computer Society, 2008.

Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Trans. Knowl. Data Eng. , 22(10):1345-1359, 2010.

Yu Pan, Jing Xu, Maolin Wang, Jinmian Ye, Fei Wang, Kun Bai, and Zenglin Xu. Compressing recurrent neural networks with tensor ring for action recognition. In AAAI , pages 4683-4690. AAAI Press, 2019.

Yu Pan, Zeyong Su, Ao Liu, Jingquan Wang, Nannan Li, and Zenglin Xu. A unified weight initialization paradigm for tensorial convolutional neural networks. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA , volume 162 of Proceedings of Machine Learning Research , pages 17238-17257. PMLR, 2022.

Yu Pan, Maolin Wang, and Zenglin Xu. Tednet: A pytorch toolkit for tensor decomposition networks. Neurocomputing , 469:234-238, 2022.

Yu Pan, Zekai Wu, Chaozheng Wang, Qifan Wang, Min Zhang, and Zenglin Xu. Identical initialization: A universal approach to fast and stable training of neural networks. Openreview.net , 2022.

Bogdan Pirvu, Valentin Murg, J Ignacio Cirac, and Frank Verstraete. Matrix product operator representations. New Journal of Physics , 12(2):025012, 2010.

Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Knowledge inheritance for pre-trained language models. In NAACL-HLT , pages 3921-3937. Association for Computational Linguistics, 2022.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP , pages 2383-2392. The Association for Computational Linguistics, 2016.

Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In ACL (2) , pages 784-789. Association for Computational Linguistics, 2018.

Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. In ICLR . OpenReview.net, 2020.

Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. CoRR , abs/1907.10597, 2019.

Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew E. Peters, and Iz Beltagy. Staged training for transformer language models. In ICML , volume 162 of Proceedings of Machine Learning Research , pages 19893-19908. PMLR, 2022.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR , abs/1909.08053, 2019.

Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML , volume 139 of Proceedings of Machine Learning Research , pages 10347-10357. PMLR, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS , pages 5998-6008, 2017.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR (Poster) . OpenReview.net, 2019.

Dilin Wang, Meng Li, Lemeng Wu, Vikas Chandra, and Qiang Liu. Energy-aware neural architecture optimization with fast splitting steepest descent. CoRR , abs/1910.03103, 2019.

Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Superneurons: dynamic GPU memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2018, Vienna, Austria, February 24-28, 2018 , pages 41-53. ACM, 2018.

Maolin Wang, Chenbin Zhang, Yu Pan, Jing Xu, and Zenglin Xu. Tensor ring restricted boltzmann machines. In International Joint Conference on Neural Networks, IJCNN 2019 Budapest, Hungary, July 14-19, 2019 , pages 1-8. IEEE, 2019.

Maolin Wang, Yu Pan, Zenglin Xu, Xiangli Yang, Guangxi Li, and Andrzej Cichocki. Tensor networks meet neural networks: A survey and future perspectives. CoRR , abs/2302.09019, 2023.

Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. In ICLR . OpenReview.net, 2023.

Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M. Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weaklysupervised classification and localization of common thorax diseases. In CVPR , pages 34623471. IEEE Computer Society, 2017.

Yulin Wang, Yang Yue, Rui Lu, Tianjiao Liu, Zhao Zhong, Shiji Song, and Gao Huang. Efficienttrain: Exploring generalized curriculum learning for training visual backbones. CoRR , abs/2211.09703, 2022.

Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural architectures. In NeurIPS , pages 10655-10665, 2019.

Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: a general approach for growing neural networks. In NeurIPS , 2020.

Lemeng Wu, Mao Ye, Qi Lei, Jason D. Lee, and Qiang Liu. Steepest descent neural architecture optimization: Escaping local optimum with signed neural splitting. CoRR , abs/2003.10392, 2020.

Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, and Tie-Yan Liu. Taking notes on the fly helps language pre-training. In ICLR . OpenReview.net, 2021.

Zhuofan Xia, Xuran Pan, Xuan Jin, Yuan He, Hui Xue, Shiji Song, and Gao Huang. Budgeted training for vision transformer. In ICLR . OpenReview.net, 2023.

Jing Xiong, Chengming Li, Min Yang, Xiping Hu, and Bin Hu. Expression syntax information bottleneck for math word problems. In SIGIR , pages 2166-2171. ACM, 2022.

Jiong Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han, Jing Tang, Chengming Li, and Xiaodan Liang. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning. CoRR , abs/2310.02954, 2023.

Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progressively stacking 2.0: A multi-stage layerwise training method for BERT training speedup. CoRR , abs/2011.13635, 2020.

Jinmian Ye, Guangxi Li, Di Chen, Haiqin Yang, Shandian Zhe, and Zenglin Xu. Block-term tensor neural networks. Neural Networks , 130:11-21, 2020.

Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan. Towards efficient tensor decomposition-based DNN model compression with optimization framework. In CVPR , pages 10674-10683. Computer Vision Foundation / IEEE, 2021.

Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. In ICLR . OpenReview.net, 2020.

Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR (Poster) . OpenReview.net, 2019.

Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. In NeurIPS , 2020.

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 9963-9977. Association for Computational Linguistics, 2023.

Jiawei Zhao, Florian Schäfer, and Anima Anandkumar. Zero initialization: Initializing residual networks with only zeros and ones. CoRR , abs/2110.12661, 2021.

Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV , pages 19-27. IEEE Computer Society, 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new approach, Mango, to use a small model as initializer to larger models in order to speed up the training of the larger model. Mango proposes to learn a mapping of weights from the smaller model to the larger model, not layer by layer or other restricted form of mapping, but a full mapping of weights.\n\nBecause the matrix that would represent this linear mapping of weights being huge (in the order of M x N where M and N are respectively the number of parameters of the small and the large model), Mango proposes a particular form of low rank factorization, allowing to keep a full mapping but constraining each of its sub-mapping to rank 1. The authors show that this mapping can be learned in as few as 100 steps, making its computing cost negligible.\n\nBy doing so, Mango shows that full mappings are useful, and shows faster convergence than the other methods on 3 types of architectures : vision transformers, BERT models and GPT models.\n\nSTRENGTHS:\n* Well written paper, with very good motivating examples\n* Good theoretical part, with the need of low rank factorization clearly introduced\n* Compelling results of 3 different models: Vision Transformers, BERT and GPT\n* The gaps in the results are significant: on vision transformers, Mango converge in 2/3 of the time of the next best method, and in 1/4 of the time of naive from scratch training. The savings on BERT are less important but still quite significant.\n\nWEAKNESSES:\nThe results are constrained to the case of Small to Base size transformers mapping :\n* Deit-S to Deit-B\n* BERT-S to BERT-M\n* GPT-Base from GPT-Small\n\nThe most interesting regime for this class of method is with bigger transformers since they are the most power hungry. The paper would gain a lot of impact by showing that Mango works for larger models (for instance to initialize a ViT-G from a ViT-H). I would definitely increase my rating if such a proof was shown, especially with a low rank such as 1.\n\nAnother interesting result would be to know, as we deal with larger models, whether it is best to initialize them with the larger pre-trained available, or if we can initialize them with Mango from a small model. For example, to initialize a ViT-G, should we preferentially use a ViT-H or ViT-L ? And how much would the difference be ?\n\nQUESTIONS:\n* The authors mention the CO2 consumption of training large models. Do you have an estimate of those emissions and the type of reduction achieved by Mango? Is the improvement proportional to the improvement in FLOPs?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of high resource costs in training large AI models from scratch. To this end, the authors propose a novel approach that linearly correlates each weight of the larger target model to all weights of the smaller pre-trained model, employing multi-linear operators to decrease complexity and manage resource demands. Experimental results on ImageNet reveal that the proposed method can significantly reduce training costs while maintaining performance.\n\nSTRENGTHS:\nThe main idea is well-written and easy to follow. The introduced Mango Operator is both innovative and intuitive, exhibiting a remarkable ability to reduce training costs.\n\nWEAKNESSES:\nThe experimental section lacks essential details, particularly around how the acceleration ratio is measured. Also, there are potentially contradictory statements about training steps and epochs, and the use of only theoretical metrics such as FLOPs for measuring training cost.\n\nQUESTIONS:\n1.\tIn the proposed method, the authors first concatenate weights across layers. However, given that the row count of $\\mathbf{W}_j^{OUT} \\in kI \\times O$ differs from that of other weight matrices, how is this discrepancy addressed? Elaborating on this point would enhance the clarity of the method.\n\n2.\tIn Equation (5), the authors suggest decomposing $\\mathbf{S}$ using a Tensor Ring Matrix Product Operator (TP-MPO) into four smaller tensors. I'm curious to know if it would be possible to decompose $\\mathbf{S}$ into five or six smaller tensors.\n\n3.\tEssential details seem to be lacking in the experimental section. Could the authors clarify how the acceleration ratio was measured? It would be better to provide more descriptions, which would strengthen the paper.\n\n4.\tThere are a few statements in the experiments section that cause confusion. The authors mention that the Mango operators are trained for 100 steps, which is claimed to take negligible time. However, they also state that the model is trained for 300 epochs. Could the authors clarify these seemingly contradictory statements?\n\n5.\tIn the experiments, the authors measure the training cost using FLOPs, which is a theoretical metric. It might be more insightful if the authors could also offer measurements related to GPU training hours, providing a more practical perspective on computational cost.\n\n6.\tThe proposed method significantly reduces training costs compared to the traditional 'training from scratch' approach, primarily by leveraging the knowledge of the pre-trained model. I am intrigued to understand if this approach would sustain its comparable performance if we were to remove the training cost restriction and instead train until convergence.\n\nLIMITATIONS:\nThe authors have provided a dedicated discussion on the potential limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

comments: SUMMARY:
This paper proposes a new method for reusing pre-trained models to a larger target model. They propose mango which decomposes the parameters into 4 smaller matrices and then learns to grow these parameters. They show their method requires less FLOPS to achieve as good as performance other baselines across vision and NLP tasks.

STRENGTHS:
The paper was easy to follow and the experiments are strong across various domains. 
Mango seems well motivated in allowing for interactions between all weights in the model.

WEAKNESSES:
Missing background on tensor ring matrix product - since this is the main method proposed
Assuming the acceleration ratio is in terms of FLOPS, then there are no experiments showing the wall-clock time of Mango. Decreasing FLOPS is only useful if it decreases wall-clock time (assuming the same machine). 
The models seem to be small, only growing from small to base variants, not base to large variants

QUESTIONS:
Is the acceleration in terms of FLOPS or wall clock time? 
How does the multiring operator allow for interactions between all weights in the model?
How does S_L capture the relationship between layers?

LIMITATIONS:
See weaknesses above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studied the problem of solving a proper initialization for large deep learning models on top of smaller pre-trained networks. The key idea is to concatenate the weights of pre-trained models to a large matrix, and map the matrix to the parameter space of the larger model to be trained with a multi-linear operator. In this procedure, the rank of the transformation matrix can be manually adjusted. Experimental results on NLP/CV-based Transformer models are provided.\n\nSTRENGTHS:\n1. The problem of efficient training is practically important.\n2. The proposed method is well-motivated.\n3. The design of the multi-linear operator is interesting.\n4. Mango can be applied to both NLP and CV models.\n\nWEAKNESSES:\n1. The writing needs to be improved. It is difficult to understand Eqs. (4-5), which are the key contributions of this paper.\n2. Currently, the topology of the network is not considered in Mango (e.g., the depth of the layer corresponding to each weight matrix). I think it may be beneficial to leverage this characteristic in Mango.\n3. Can we introduce non-linear operation to Mango to improve training efficiency?\n4. In Figure 7, the comparison with training from scratch may be unfair. Take DeiT for example. Obtaining DeiT-S needs a notable amount of computation. Hence, the computational cost of Mango should not start from zero FLOPs.\n5. This basic baseline may be interesting: initializing the large model (e.g., DeiT-B) by pre-training it using the same amount of computation as training the smaller model (e.g., DeiT-S).\n6. Can Mango improve the final accuracy of the model?\n7. In the field of CV, it would be interesting to validate the effectiveness of Mango on top of more network architectures, e.g., Swin Transformer.\n8. Some related works should be discussed or compared with Mango:\n- Li, C., Zhuang, B., Wang, G., Liang, X., Chang, X., & Yang, Y. (2022). Automated progressive learning for efficient training of vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12486-12496).\n- Budgeted Training for Vision Transformer, ICLR'23\n- Wang, Y., Yue, Y., Lu, R., Liu, T., Zhong, Z., Song, S., & Huang, G. (2022). Efficienttrain: Exploring generalized curriculum learning for training visual backbones. arXiv preprint arXiv:2211.09703.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nPersonally, I think this paper may not have potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

comments: SUMMARY:
The contribution of this paper lies in proposing a new method to accelerate the training of large models by leveraging the correlations among all weights in the model, instead of just mapping partial weights as in existing methods. The authors' approach utilizes multi-linear operators to reduce computational and spatial complexity, enabling acceptable resource requirements. Experimental results demonstrate that their method has excellent performance in accelerating training, outperforming existing methods.

STRENGTHS:
+ The method proposed in the paper can leverage the potential correlations among all weights in the model, significantly reducing computational costs.
+  the authors conducted experiments in both image and natural language processing domains and achieved results that surpass existing methods. This indicates that their proposed approach has a certain level of generality and applicability, and can significantly accelerate model training across different domains.

WEAKNESSES:
+ In the experiments of the paper, only DeiT-B, BERT-Base, and GPT-Base were used, and no further exploration was done on larger models.
+ The performance of Mango, Ligo, and bert2BERT was improved in natural language processing methods, while only marginal gains were achieved in image tasks, and the experiment lacked analysis and explanations.
+ For Figure 6(b), there is continuous improvement in accuracy with increasing rank, but there is a lack of experimental comparison for larger rank values.

QUESTIONS:
+ In line 196 of the paper, for the loss function in Equation (6), what is the specific design of this loss function for the operators? Is it task-specific or fixed?

LIMITATIONS:
It might be interesting if the results can be extended to large language models.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26