Incremental pre-training from smaller language models (2024)

Abstract: Large language models have recently become a new learning paradigm and led to state-ofthe-art performance across a range of tasks. As explosive open-source pre-trained models are available, it is worth investigating how to better utilize existing models. We propose a simple yet effective method, Incr-Pretrain, for incrementally pre-training language models from smaller well-trained source models. Different layer-wise transfer strategies were introduced for model augmentation including parameter copying, initial value padding, and model distillation. Experiments on multiple zero-shot learning tasks demonstrate satisfying inference performance upon transferring and promising training efficiency during continuing pre-training. Compared to training from scratch, Incr-Pretrain can save up to half the training time to get a similar testing loss.

heading: 1 Introduction
text: Large language models have led to state-of-theart accuracies across a range of tasks and have demonstrated strong performances with few-shot in-context learning (Zhang et al., 2020b; Zeng et al., 2021; Brown et al., 2020). From GPT (Radford et al., 2018) to Switch-Transformer (Fedus et al., 2021), the number of parameters grows from 125 million to 1.6 trillion at an exponential rate. The study of GPT3 (Brown et al., 2020) shows that a large language model (up to 175 billion) can have strong context learning ability, and obtains comparable performances with state-of-the-art finetune style methods even without any parameter updating. An empirical scaling law (Kaplan et al., 2020) shows that the larger models with wider and deeper architecture are significantly more sampleefficient on a relatively modest amount of data. Furthermore, as the model size increases, there is still room for performance improvement.
* R. Xu and H. Wang are corresponding authors.
However, training large language models from scratch always costs huge computing resources and time. For instance, NVIDIA leveraged their Selene supercomputer to perform scaling studies and used up to 3,072 A100 GPUs for training the largest Megatron (Shoeybi et al., 2019) model (1 trillion parameters). OpenAI spent 355 GPU-years for training GPT-3 (Brown et al., 2020), and the total costs are more than ten million dollars. Most existing model transfer methods aim at improving the performance of downstream tasks, e.g. transfer learning (Zhuang et al., 2019) or speeding up the inference process, e.g. knowledge distillation (Gou et al., 2020), but studies for accelerating model pre-training from scratch remain limited. To our knowledge, no research on how to transfer a small pre-trained model to a large model has been done.
We introduce Incr-Pretrain to augment a smaller source Transformer model to a larger target model and make them have comparable performances, both upon transferring and after continuing pretraining. Different layer-wise transfer strategies are introduced for model augmentation including parameter copying, padding and model distillation. Specifically, we propose a KL-divergence-based approximation method to distill the LayerNorm layer to address a mathematically intractable issue during transferring. We tested our method’s performance on zero-shot tasks of BERT-base and GPT-2, and the results show that the augmented models obtain satisfying performances. When incrementally training a dialogue-GPT model on different scales, the training and testing losses can continue declining from the values before transferring. The total training time can be saved up to half compared with that training from scratch.
To the best of our knowledge, this is the first parameter-based method for incrementally pretraining language models. Our method can help reduce the heavy resource cost of training large language models from scratch and can be applied
36
to almost any open-source pre-trained model in the Transformers library (Wolf et al., 2020). The proposed method is also compatible with mainstream parallel training techniques. We summarize our contributions as follows: 1) We prove that it is feasible to train a larger language model from smaller Transformer models without training from scratch; 2) We propose a distillation-based method to transfer the LayerNorm parameters.

heading: 2 Method
text: We present the implementation of Incr-Pretrain in the scenarios of both widening and deepening a Transformer model. For widening the model, we use parameter copying and padding to transfer the embedding, attention and MLP layers and a distillation-based method to adjust the LayerNorm’s parameters to the new input distribution due to the changed input dimension problem. For deepening the model, we initialize the deeper layers with small parameter values, the noise of which could be overwritten by the residual connection setting and have less adverse impact on the entire model. The overall framework is shown in Figure 1.

heading: 2.1 Widen the model
text: Linear transformation is the basic operator that exists in both the multi-head attention and feedforward layer. Incr-Pretrain transfers a smaller-size matrix of the linear transformation from the source model to a bigger matrix in the target model. As shown in Figure 2, by padding small random values or zeros at the tail of the source matrix, both vertically and horizontally, the result of the matrix multiplication is approximate to that by directly doing matrix multiplication on the source matrix. This is ensured by the block matrix multiplication
rule. We also prove that if we pad random values θ ∼ N(0, σ2) to the dense layer, the changes on the output can be controlled in O(σ2) (Appendix). Especially, if σ reduces to zero, the nonzero values in the matrix multiplication result will be unchanged.
In the multi-head attention layer, parameters are the weights and biases in linear transformation for queries, keys, and values. So we can also apply the above method to the attention layer. To ensure the attention score of each head is valid, we can keep each attention head dimension fixed and only increase the head number, or keep the head number and pad values to each attention head, either combination is feasible. In the embedding layer, we directly pad small random values or zeros to the source embeddings. According to the block matrix multiplication rule, the inner product similarity of any two-word vectors will not change much, which is critical to the attention layer.

heading: 2.2 Transfer LayerNorm
text: LayerNorm is a technique to normalize the distributions of intermediate layers. It enables smoother gradients and faster training by re-centering and re-scaling both inputs and weight matrix. However, both re-centering and re-scaling operations are related to the hidden size, which would change after transferring. Mathematical inequivalence will affect the performance of the target model, but the scaling weight and bias can be updated fast after training several steps to adjust new model parameters.
We introduce a distillation-based method for transferring the LayerNorm layer. Let input x ∈ RH , LayerNorm re-centers and re-scales x as hi = gi ·N(xi) + bi, where N(xi) = (xi − µ)/σ, µ = ( ∑H i=1 xi)/H , σ = ( ∑H i=1(xi − µ)2/H)1/2. h is the output of the LayerNorm layer, (·)i is the scalar value of the i-th dimension, and µ and σ are the mean and standard deviation of the input. The bias b and gain g are parameters with the same
dimension H . Let x̂ = (x1, x2, . . . , xH , θ1, . . . , θD−H) be the input e.g. padded word embeddings, x̂ ∈ RD. Since we padded D-H values to the input, the mean and variance were changed. We define ĥi = ĝi · ˆN(xi) + b̂i, ˆN(xi) = x̂i−µ̂ σ̂ , µ̂ = ( ∑H i=1 xi + ∑D−H i=1 θi)/D, σ̂ =
(( ∑H i=1(xi − µ̂)2 + ∑D−H
i=1 (θi − µ̂)2)/D)1/2. To make the outputs of source and target LayerNorm equal, for every integer i in section [1, H], we need to let
gi ·N(xi) + bi = ĝi · ˆN(xi) + b̂i, ∀i ∈ [1, H] ∩ N
(1)
So we established a equation where the variables are ĝi and b̂i, i ∈ [1, H] ∩ N. We need to find a set of solutions to Eq. 1 which are viable for ∀(x1, x2, . . . , xH). In particular, for any word index k ∈ [1, |V ocab|] ∩ N, the equation k is
( N̂k, E )( ĝ b̂ ) = hk (2)
where E is the unit matrix, N̂k = Diag( ˆN(xk1), ˆN(xk2), . . . , ˆN(xkH)). Unfortunately, we found that Eq. 2 is intractable, the proof is presented in Appendix.
The gain and bias are parameters that can be updated based on gradient, so we construct a loss function to train the LayerNorm parameters in the target model by calculating the KL-divergence between the outputs from the target and source. The loss function L is defined as
L = DKL(P (x|θsource), P (x|θtarget)) (3)
By minimizing the loss, the target and source LayerNorm outputs are converging.

heading: 2.3 Deepen the model
text: Deepening the neural network is the most common way to increase the model size. When we transfer a source model with few layers to a deeper target model, the parameters in deeper layers need to be initialized with small values. In both selfattention and MLP layers, the small parameters θ will result in small layer output layer(x|θ), so the output through residual connection layer(x|θ)+x approximates to x. It enables the deeper layers will not change the output distribution of shallow layers much, so a deeper target model can have similar output distributions to the source model.

heading: 3 Experiments
text: We conducted extensive experiments on inference upon transferring and continuing pre-training. We tested BERT on the cloze tasks and GPT on the next word prediction tasks, which are corresponding objectives at their stages of pre-training. To validate the time efficiency of using Incr-Pretrain, we continued to pre-train the target model and compared the loss curve with that of training from scratch.

heading: 3.1 Inference upon transferring
text: We tested BERT on the LAMA (Petroni et al., 2020) dataset and GPT-2 on the Lambada (Paperno et al., 2016), ClozeStory (Bugert et al., 2017), and HellaSwag (Zellers et al., 2019) datasets with a zero-shot method without any continuing pretraining.
Datasets BERT is a masked language model whose primary pre-training task is mask filling (cloze), so the performance on the cloze task is the most effective indicator. The language model tested on LAMA needs to understand the whole sentence and predict the masked keyword. Considering that some samples are too difficult to BERT in zero-shot tasks, to reduce the impact of randomness, we let BERT predict 5 times for each sample, and if any time the correct answer is predicted, we consider it correct.
GPT is a causal language model (CLM) that is pre-trained by predicting the next word with only one side of the text visible. LAMBADA, storyCloze, and HellaSwag are all datasets that aims to predict the ending text piece(s), so they are consistent with the pre-training process of the causal language model. In this task, we let GPT predict only one time on the test part of LAMBADA, the test part of StoryCloze, and the dev part of HellaSwag. On StoryCloze and HellaSwag datasets, the inference method is the same as the perplexitybased method (Zeng et al., 2021).
Configuration We compared three types of models on both datasets. The Source models exist as open-resource models, i.e. BERT and GPT-2. The Target models are the basic enlarged versions in which parameters of each layer are directly copied from the Source models with padded values. For attention, we let the number of heads increase but the dimension of each head is unchanged. We set σ as zero to make the calculation as equivalent as possible and also reduced the impact of randomness on the experiment. Compared to the target
models, the Target-LN models further transfer the LayerNorm parameters using the distillation-based method training on only 2,425 short dialogues (Eric and Manning, 2017).
The results of the inference tasks are shown in Table 1. We observe that after transferring, the performances of the Target models drop dramatically compared with the Source models. This is likely due to the LayerNorm part, which is not mathematically equivalent when transferring. In comparison, the Target-LN (Both GPT and BERT) models are comparable with the Source models, which shows that the distillation-based approximation method is effective.

heading: 3.2 Continuing pre-training
text: We pre-trained three sizes of Dialog-GPT models on a benchmark dialog corpus (Zhang et al., 2020a) to validate our incremental pre-training method, including small, medium, and large versions. We conducted transfers between models of different sizes and the model configuration details are shown in Appendix. We pre-trained the three GPT models for 14.5k steps with the batch size 32, and performed the model transfer from the checkpoints of the 5,000th steps. The testing losses of different models are shown in Figure 3. In general, the loss values can continue declining from a smaller value after transferring. We did not use the distillationbased method to transfer LayerNorm parameters since it is no longer needed to make the model converge to the source model during continuing pre-training. As shown in Table 2, training GPTm using Incr-Pretrain only costs 7.05k steps to reach a testing loss value comparable with that of 14.5k steps training from scratch. This shows that our method saves about 51% of the training time. However, as more parameters are padded, e.g.
from small to large, the amount of training time that can be saved declines, which is likely due to pre-training larger models needing more computation. (Kaplan et al., 2020).
Further experiments showed that using our transfer method, the amount of pre-training time saved depends on not only the enlargement of the number of parameters but also the padding values. When padding zeros, the testing loss can decline starting from near to the loss value of the source model but converging slowly. Considering previous work (Glorot and Bengio, 2010; He et al., 2015) on parameter initialization, we padded smaller random values instead of zeros, and the convergence can accelerate much. More experimental details are presented in the Appendix.

heading: 4 Conclusion
text: We propose a transfer strategy that can incrementally pre-train language models with acceptable performance decreases. The inference performances show that the target models are comparable with the source models. The continuing pre-training experiment demonstrates that the transfer method is computationally efficient compared to pre-training a large model from scratch. Our future directions will be transferring with different parallel styles and exploring the influence of padding values.
Limitations
During the process of incremental pre-training, although this method effectively reduces training time and achieves test losses close to those obtained by training from scratch, the percentage of pretraining time saved gradually decreases as the number of model parameters increases. For instance, transitioning from a small to a medium-sized model can save approximately 51% of training time, but extending from a small directly to a large model reduces the saving to 34%. This indicates a diminishing marginal return in pre-training efficiency as the model scale expands.

heading: Acknowledgements
text: This research was supported in part by the Major Key Proiect of PCL (NO.PCL2023A09 and NO.PCL2023AS7-1), the National Key Research and Development Program of China (2021ZD0112905), the National Natural Science Foundation of China (62176076), the Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies(2022B1212010005), Natural Science Foundation of Guangdong (2023A1515012922), and Shenzhen Foundational Research Funding (JCYJ20220818102415032).

heading: A Experimental Details
text: In the distillation-based method, we used the Adam optimizer, the learning rate was set as 1E-4, batch size as 8, and the sequence length as 512. Other model configurations are shown in Table 3. We used the KVRET corpus to train the LayerNorm parameters in GPT-2 for 20k steps (about 3 hours on NVIDIAV100) and discovered that using pseudo inputs constructed by randomly choosing word indices produced better results than those from the real corpus when transferring BERT’s LayerNorm parameters.
In the continuing pre-training experiments, we used the Adam optimizer and linear warm-up at the first 100 steps, the learning rate was 1.5E-4, and the batch size was 32. As shown in Figure 4, we compared two padding strategies, padding zeros and padding random values θ ∼ N(µ, σ2), µ = 0, σ = 0.02. When padding zeros, the testing loss starts from that of before-transferring, and it proved that our transferring method is feasible. However, the test loss converged slowly since padding zeros disturbed the initialization distribution. To speed up the incremental training process, we padded random values that follow a normal distribution with small variance instead of zeros. Although padding random values breaks the mathematical equivalence of transferring a bit and the loss value is higher at the beginning, the acceleration for the convergence is remarkable.
Table 3: Model configurations for inference
Model Heads Layers Dim Total Source BERT 12 12 768 119.5M Source GPT 12 12 768 124.4M Target(LN) BERT 16 12 1024 184.0M Target(LN) GPT 16 12 1024 203.7M
Table 4: Model configurations for pre-training
Model Heads Layers Dim Total Model-Small 6 6 384 15.8M Model-Medium 8 8 512 32.2M Model-Large 12 12 768 95.5M

heading: B Proof-1
text: We prove that if we pad random values θ ∼ N(0, σ2) to the parameters of the dense layer, the changes of output can be controlled in O(σ2).
For simplicity of proof, we consider a fully connected layer with a mapping function F : RH → RH , F (x) = Ax + b, where A ∈ RH×H , b ∈ RH , and x is a input column vector such as word embeddings. The padded layer can be expressed alike, i.e. F̂ : RD → RD, F̂ (x̂) = Âx̂ + b̂, Â ∈ RD×D, b̂ ∈ RD, where
Â = ( A, Λ1 Λ2, Λ3 ) , b̂ = ( b β ) , x̂ = ( x χ ) ,
All of the Λi(i = 1, 2, 3), β, and χ are independent and identically distributed to the Gaussian distribution N(0, σ2), while A and x are constants.
F̂ (x̂) = ( Ax+ Λ1χ+ b Λ2x+ Λ3χ+ β )
So the mathematical expectation is
E(F̂ (x̂)) = (
E(Ax) + E(Λ1χ) + E(b) E(Λ2x) + E(Λ3χ) + E(β)
)
= ( E(Ax) + E(Λ1)E(χ) + E(b) E(Λ2)x + E(Λ3)E(χ) + E(β) )
=
( Ax + b
0
)
The variance is
D(F̂ (x̂)) = (
D(Ax) + D(Λ1χ) + D(b) D(Λ2x) + D(Λ3χ) + D(β)
)
= σ2 (
(D −H)σ2E1 (1 + |x|2 + (D −H)σ2)E2
)
where 1(∗) is all-one tensor with the same shape of tensor (*) ,E1 ∈ RHandE2 ∈ RD−H are all-one column vectors, x2e is elements squared of x, and |x|2 is the square of the norm of x.
Above all, we proved that the padded dense layer maintains the same mathematical expectation with that of before-padding, and the variance is O(σ2). Therefore, if we pad random values with small variances, the output of the model will not change much. Especially, if the variance reduces to zero, the output will be unchanged.
When applying the method to the multi-head attention layer, we can pad random values θ ∼ N(0, σ2) to the parameters of queries, keys, and values. The output of the attention layer is SoftMax((QT + O(σ2))× (K+O(σ2)))× (V +O(σ2)), so if the variance σ2 is zero, the output of attention layer remain unchanged after padding.

heading: C Proof-2
text: We prove that Eq. 2 has no solution. According to the laws of linear equations, we only need to prove that the rank of the coefficient matrix is not equal to that of the augmented matrix. The augmented matrix Â0 of Eq. 2 is
Â0 =  
N̂1, E, h1 N̂2, E, h2 ... ... ...
N̂ |V ocab|, E, h|V ocab|
 
Let the last |V ocab| − 1 row blocks subtract the first row block, the augmented matrix Â0 changes to
Â1 =  
N̂1, E, h1
N̂2 − N̂1, 0, h2 − h1 ... ... ...
N̂ |V ocab| − N̂1, 0, h|V ocab| − h1
 
Let the first column block subtract the product of N̂1 and the second column block, and the third column block subtract the product of h1 and the second column block, the augmented matrix Â1 changes to
Â2 =  
0, E, 0 N̂2 − N̂1, 0, h2 − h1
... ... ... N̂ |V ocab| − N̂1, 0, h|V ocab| − h1
 
After going through a similar matrix transformation, the coefficient matrix A0 changes to
A2 =  
0, E N̂2 − N̂1, 0
... ...
N̂ |V ocab| − N̂1, 0
 
We define N̂∗ and h∗ as follows:
N̂∗ =   N̂2 − N̂1 N̂3 − N̂1
... N̂ |V | − N̂1
  , h ∗ =   h2 − h1 h3 − h1
... h|V | − h1
 
According to the calculation of N̂∗ and h∗, they are not linearly dependent. Obviously, it is equivalent to that the matrices Â2 and A2 have different ranks. So Eq. 2 has no solution.
D Figures

heading: Future Work
text: Our future directions will be transferring with different parallel styles and exploring the influence of padding values. Limitations
During the process of incremental pre-training, although this method effectively reduces training time and achieves test losses close to those obtained by training from scratch, the percentage of pretraining time saved gradually decreases as the number of model parameters increases. For instance, transitioning from a small to a medium-sized model can save approximately 51% of training time, but extending from a small directly to a large model reduces the saving to 34%. This indicates a diminishing marginal return in pre-training efficiency as the model scale expands