Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Authors: Wenyu Du
Tongxu Luo
Zihan Qiu
Zeyu Huang
Yikang Shen
Reynold Cheng
Yike Guo
Jie Fu
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Efficient LLM pre-training
Model growth
Primary area: natural_language_processing
TL;DR: We systematically study model growth techniques for efficient LLM pre-training, finding a depth-wise stacking operator with great acceleration and scalability, and propose clear guidelines for its use.

Abstract: LLMs are computationally expensive to pre-train due to their large scale.
Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. 
However, the viability of these model growth methods in efficient LLM pre-training remains underexplored.
This work identifies three critical $\underline{\textit{O}}$bstacles: ($\textit{O}$1) lack of comprehensive evaluation, ($\textit{O}$2) untested viability for scaling, and ($\textit{O}$3) lack of empirical guidelines.
To tackle $\textit{O}$1, we summarize existing approaches into four atomic growth operators and systematically evaluate them in a standardized LLM pre-training setting.
Our findings reveal that a depthwise stacking operator, called $G_{\text{stack}}$, exhibits remarkable acceleration in training, leading to decreased loss and improved overall performance on eight standard NLP benchmarks compared to strong baselines. 
Motivated by these promising results, we conduct extensive experiments to delve deeper into $G_{\text{stack}}$ to address $\textit{O}$2 and $\textit{O}$3.
For $\textit{O}$2 (untested scalability), our study shows that $G_{\text{stack}}$ is scalable and consistently performs well, with experiments up to 7B LLMs after growth and pre-training LLMs with 750B tokens.
For example, compared to a conventionally trained 7B model using 300B tokens, our $G_{\text{stack}}$ model converges to the same loss with 194B tokens, resulting in a 54.6\% speedup. 
We further address $\textit{O}$3 (lack of empirical guidelines) by formalizing guidelines to determine growth timing and growth factor for $G_{\text{stack}}$, making it practical in general LLM pre-training.
We also provide in-depth discussions and comprehensive ablation studies of $G_{\text{stack}}$. 
Our code and pre-trained model are available at https://llm-stacking.github.io/.

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Wenyu Du
Tongxu Lu Zihan Qiu
Zeyu Huang
Yikang Shen
Reynold Cheng' Yike Guo? Jie Fu
SScooo of Computing and Data Science The University of Hong Kong *KKST
USTB "Tsinghua University Unnveesstty Edinburgh wydu@cs. hku. hk tongxuluo@gmai1 com
MMIT-IBM Watson Al Lab jiefuOust .hk
Abstract
LLMs are computationally expensive to pre-train due to their large scale. Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. However, the viability of these model growth methods in efficient LLM pre-training remains underexplored. This work identifies three critical Obstacles: (01) lack of comprehensive evaluation, (O2) untested viability for scaling, and (03) lack of empirical guidelines. To tackle O1, we summarize existing approaches into four atomic growth operators and systematically evaluate them in a standardized LLM pre-training setting. Our findings reveal that depth- wise stacking operator, called Slack exhibits remarkable acceleration in training, leading to decreased loss and improved overall performance on eight standard NLP benchmarks compared strong baselines. Motivated by these promising results, we conduct extensive experiments to delve deeper into Guack to address 02 and 03. For (untested scalability), our study shows that Guaek scalable and consistently performs well, with experiments up to 7B LLMs after growth and pre-training LLMS with 750B tokens. For example, compared trained 7B model using 300B tokens, our Gstack model converges to the same loss with 194B tokens, resulting in a 54.6% speedup. We further address 03 (lack empirical guidelines) by formalizing guidelines determine growth timing and growth factor for Gruck- making it practical in general LLM pre- pre-training We also provide in in-depth discussions and comprehensive ablation studies of Gsuakk Our code and pre-trained model are available at http:://lmm-taakinn github.io/
conventionally
Introduction
Emergent abilities of Large Language Models (LLMs) rely on scaling-up 2 Empirical evidence from scaling laws (355] fuels the development of increasingly larger models, pushing the boundaries of LLMS capabilities. However, pre-training these gigantic models comes at a significant cost in terms of energy consumption and environmental impact 6 (e.g., pre-training Llama-3 (7) consumes a total of 7. 7.7M GPU hours and generates 2290 tons of carbon dioxide equivalent of carbon emissions). The efficient pre-training of LLMs is thus crucial, both from scientific and a socictal perspective, to ensure the continual growth and adoption of AI [].
One promising research direction accelerate model training involves leveraging trained smaller (base) models to expedite the training of larger (target) models, a technique known as model growth.
Equal Contributions.
Work done during interning at HKUST. Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Concretely, model growth studies how to leverage the trained smaller model's parameters (+)) to initialize the larger model's parameters ( Current popular methods generally focus on expanding
the parameters of the base model through techniques like splitting [10H12), copying [3 14 or matrix
mapping [15]. There are also some approaches that initialize new parameters from scratch [16 12 [7].
The primary objective is to accelerate the training of large models, and existing methods demonstrate
promising speedup results on models such as BERT [1]
Despite such empirical evidence and its alignment with the goal of efficient LLM pre pre-training model growth methods are not widely adopted in the context of LLM pre- pre-training 7 19] To our best knowledge, the only LLM that utilizes model growth for accelerating is FLMM-101 20 but lacks baseline LLM trained from scratch to compare. We observe three key Obstacles that hinder LLM pre-training from using existing model growth techniques, specifically:
14
18 [5]
12 B
01: Lack of comprehensive assessment. Some existing model growth methods report results on LLM pre-training, but either lack baseline comparison (20) or are still in exploratory stages [5
I3
In contrast, most growth approaches are evaluated in encoder-based BERT models 14] 8 12
13 16 17
which have different architecture and training configurations compared to prominent decoder-basee LLMs such as Llama (21]
02: The untested scalability. This scalability has two aspects: the model size and the amount of pre- training data. Regarding the model size, the existing approaches are only evaluated on smaller-scale BERT models in preliminary experiments with L.MMs. It is unclear whether these growth methods will continue accelerating training when applied to large-scale LLMs with more extensive evaluation. As for the amount of pre-training data, there are debates [22] over whether certain efficient training strategies may initially converge faster but ultimately perform similarly or worse than vanilla training methods when given ample computational resources (i.e., more training data).
03: Lack of empirical guidelines. Scaling laws give clear empirical guidelines on pre-training computaaiionallopimiize LLMs, greatly stimulating and advancing the field. Yet, there is a lack of empirical guidelines on growth techniques, discouraging LLM practitioners from adopting these approaches, especially considering the high costs of L.M pre-training.
These three obstacles are consequential in nature. Hence, this work, we empirically revisit the concept of model growth as solution to efficient LLM pre-training by tackling them one by one.
To tackle 01, we systematically evaluate model growth techniques on practical LLM pre-training, We first categorize existing growth methods and summarize them into four atomic growth op- erators, each of which can grow along two di- rections: widthwise (intra-layer) and depthwise (layer-wise). We illustrate them in Figure 2 These operators serve as representative choices for evaluating the performance of model growth techniques. We use these operators to expand 400M base models to I.IB Llama-like LLMs and continually pre-train them. Next, we evaluate these growth techniques on the training loss and eight standard NLP benchmarks from the Har- ness toolkit 23). We found the direct operator that stacks depthwisely Grnnc consistently out- performs others across overall evaluation metrics, demonstrating its potential in accelerating LLM pre-training. This motivates us to investigate ex- tensively by addressing 02 and 03 on G
Tokens (Billions)
50
100
150
200
250
300
scratch
Gordd
2.8
2.6
250
Training Loss
2.0
2.4
59.6%
0.0%
1.9
2.2
4p
60
80
100
2.0
1.8
0
20
40
60
80
100
FLOPs (1e+20)
Figure 1: The training loss for two 7B LLMs, trained from scratch and with G1 direct (Gstack). At 300B tokens, Gstack accelerates by 54.6% com- pared to scratch.
To address 02, we investigate the Grack opera- tor's scalability to larger model sizes and to more training data. We conduct extensive experiments by scaling model size up 7B parameters trained with 300B tokens, and pre-training 410M model with over 750B training tokens. This is in contrast to the previous largest L.M pre-training experiment that uses model growth methods and has baselines for comparison, which is reported in Ligo [5], where a GPT2-1 1155B model is trained for 15k steps (approximately 15B tokens). The
2

---

results are encouraging, as we consistently observe significant improvements Gract offers in both scenarios. For example, we achieve a remarkable 54.6% speedup in pre-training for a 7B model with 300B tokens (Figure[ D. Interestingly, the loss improvement in our 750B-toke experiment aligns with a logarithmic function. We further extend this logarithmic curve and determine that the improvement continues to be substantial even for the LLM trained with over 8T tokens. Moreover, we summarize all our experiments by estimating the LLM scaling law for LLMs pre-trained with Gstack Given the same target loss value, our analysis reveals significantly reduced computational cost compared to the common scaling law
For O3, we explore the practical guidelines for using Gruck in LLM pre-training. Given a compu- tational budget, we determine the optimal strategy for two key factors of stack growth timing d and growth factor g. Growth timing d relates to the training tokens used for small models before growing, and growth factor g refers to the factor between the non-embedding parameter number of the large models and the small models. We formalize our findings into equations that offer concrete suggestions for utilizing Srack We believe this work could significantly pique the interest and bolster confidence in future LLM pre-training with model growth techniques, both in academia and industry.
To summarize, our contributions are four-fold: 1) We first systematically investigate model growth techniques and identify four atomic model growth operators, establishing a better understanding of the field in Section 3.1 2) We then design a standard LLM pre- pre-training testbed and perform comprehensive evaluations on these operators, finding that a simple depthwise stacking Gstack exhibits significant superiority in Section 3 3) We further demonstrate the scalability of Gannk with experi- ments on LLMs ranging from 410M to 7B parameters and up to 750B training tokens in Section 4.1 4) We also provide guidelines of equations on determining growth timing and growth factors for optimal use of Gsinck in Section
2 Related Work - Model Growth for Efficient Pre-training
The idea of growing neural networks dates back to the 1990s (4 The pioneering work of Net2Net [TO] marks milestone, for the first attempt study model growth in deep learning Net2Net expands width and depth while keeping original functions (namely function preserving) via randomly splitting old neurons and injecting new identity layers. The widthwise splitting method of Net2Net represents a series of works that aim to "expand" the existing neurons to the desired larger size. Bert2Bert [T]] serves as a BERT-based extension of the widthwise Net2Net. StagedGrow [13 doubles the width by concatenating two identical layers and halves final loss to keep function- preserving. Lemon [12] suggests integrating a parameter into the splitting of neurons in Bert2Bert, aiming to break weight symmetry. Depthwisely, StackedBert (14] simply stacks duplicated layers to form a deeper model. In contrast to the above direct copy/split approaches, LiGO [5]senett a learning-basee based method that initializes the larger model's 's parameters via learning a linear mapping from the smaller model's 's parameters.
Alongside the approaches that expand existing parameters, there are works that initialize new ones without relying on existing ones. For instance, MSG (17) proposes multi-staged growing strategy that progressively expands transformer components, where the newly grown neurons are randomly initialized using a masking mechanism to ensure function preservation. Besides, some works have assigned specific values, like zero, to the newly initialized neurons to negate their influence [16] [2]
All the above methods are primarily explored in BERT or earlier stages of LLM pre-training. On the other hand, our objective is to present the first systematic review of model growth techniques in the LLMs era. To our knowledge, FLM-101B 20 is the only existing LLM that uses the growth method [17] for accelerating billion-scale LLM pre-training. Nonetheless, this work lacks a baseline model trained from scratch, making it difficult to assess the effectiveness of the model growth technique. In contrast, we aim to provide a comprehensive study by establishing a standardized testbed to compare LLMS trained from scratch and with various growth methods in LLM pre-training.
3 Systematically Assessing Model Growth for LLM Pre-Training
Existing model growth methods 14 with limited focus on decoder-only large-scale language models such as Ilama (2T]. Moreover, these growth methods are often not comparable due to different training settings 14 12
18
15
16
are mainly evaluated on BERT 27
3

---

Even some growth LLMs experiments are evaluated, their results are often incomplete 20
15 To
overcome these limitations, we first summarize existing works [14 TT [8 13
T
16 17 into
four atomic growth operators to represent these growth techniques. Then we build standardized
LLMMs training testbed to pre-train LLMS with four growth operators on depthwise and widthwise directions and evaluate the results with both training loss and eight evaluation metrics in Harness (23
3.1 Growing L.LMs with Growth Operators
Recent years, researchers have focused on enhancing the efficiency of training large models by making use of smaller pre-existing models [TO [4 18 15
16 [7]. These state-of-the-ar methods can be categorized into two distinct groups The first group focuses on deriving new neurons 151, while the second group focuses on initializing new Drawing from these two lines of research, we summarize four atomic growth operators. These operators include: (A) directly duplicating and stacking old layers in depthwise manner or splitting neurons in the same layer widthwisely, denoted as Gdirect- (B) generating expanded parameters using a learnable mapping matrix to the existing parameters, denoted as Cleam- (C) setting the new parameters to zero, denoted as Gzer. and (D) randomly initializing the new parameters, denoted as Grandom- The illustration of four operators is shown in Figure 2 The Gdireer and Gream growth operators produce neurons from the current ones, in contrast to Gzero and Gradom which initialize new parameters independently. For the formal definitions of the operators and the differences to the existing growth methods in design, please refer Appendix Complex growth methods, such as those involving auxiliary loss or exploring training dynamics like learning rates (28 29 [6] are interesting. But considering the high computational cost of LLM pre-traine pre-training, we focus on simple, universally applicable growth operators for different LLM pre- pre-training settings.
12
I3
from the existing ones [T0 II
12
parameters separately 18
13
16
17
Old parameter
R
New parameter
from random
Split
+ B
D
New parameter from the old
New parameter
assigned to zero
Layer
Training needed
Hyper network
aW
W
Do
fw.
Wo
W
xW
W
D2
Copy
Copy
W
W1
2
W
W
D,
D
D2
Layer
W
W3
aW:
W
pW
X
(a) Gdirect and Gltirett (tsakk
Base
H
Mask
Do
D
D2
Wo
W 1
R
W
W
Ro
o
W 2
W
R
W
W
R1
D3
D
D5
3
Layer
D6
D7
D8
0
R₂
R
R3
R2
(b)
Geaar
(c) Geero
(d) Grandom
Figure 2: The simplified illustration of four growth operators Gdireets Greams Gzero and Grandoms each of which grow along widthwise (intra-layer) G depthwise (layer-wise) GT, W, n is the parameters before growth, while D, R, and are the growth parameters derived from the old, randomly initialized, and aero-initialized respectively. Except Gaivas other three operators only illustrates the widthwise growth.
To make fair comparison the four growth operators LLM pre-training, we define a standardized "one-hop" growth process that involves two training phases, small model training before growth and large model training after growth. We first train the small L.MS with d tokens before growing. Then, use operator G to grow them to the target LLMs by factor of g for non-embedding parameters

---

and then continual pre- training the large LLMs for D tokens. Two key factors in the procedure are worth noting: the growth factor 9 and the data for base model training which can be interpreted as "growth timing". We further evaluate each growth operator by separately examining in depthwise (intra-layer) growth Gt and widthwise (layer-wise) growth G Concretely, we start with base models (400M LLMs) trained on d = 10B tokens, apply the four operators in both directions to scale them up to the target size of I.IB (approximately a growth factor g = 4), and then continue training for additional D 97.5B tokens. Appendix B contains the LLM's architecture configuration and training details.
3.2 Pre-Training 1.1B LLMS
We report results on training loss, eight standard Harness NLP benchmarks along with the average accuracy and the speedup ratio in Figure 3 Our key discovery reveals that depthwise growth G1 exhibits a significant acceleration over both widthwise growth G and training models from scratch, while surprisingly, G does offer any notable advantages. Among the depthwise growth operators, G Giirect G Gleam and G all outperform the baseline and G Tpmmkomm The underperformance of Gf in our study may be attributed to its design for gradual "mini-step" growth [17], whereas our unified approach uses single step. Most notably, depthwise stacking G emerges as the clear winner among growth operators, surpassing its competitors in speedup, training loss and nearly every Harness evaluation metric. For example, compared to training models from scratch for 100B tokens, G direct achieves significant efficiency gain, increasing training speed by 49.15 19.1%. The calculation of speedup please refer to Appendix B.2 The Appendix presents more experiments on these operators, including their loss training and evaluation figures.
ramdom
Depth
Width
Baseline scratch
Glirect
Geero
Grndom
Glearn
direct
Gzero
Grandom
Gearn
Lambada (f)-
48.20
48.67
44.14
48.
46.16
44.67
44.24
45.66
47.87
ARC-c (f)-
29.18
28. 32
28. 4.1
27.38
28.58
26.70
27.64
26.70
27.
ARC-e (f)-
54.25
51.76
52.69
51.17
51.55
49.70
53.82
50.37
48.86
Logiqa (f)
28.87
27.95
25.96
28
27.34
25.03
26.11
26.57
25.96
PIQA (f)-
71.98
71.81
70.78
71.16
69.47
69.74
70.13
69.91
69.64
Sciq (1)
81.1
81.9
77.7
80.0
81.4
76.0
79.5
79.5
76.8
Winogrande (f)-
56.03
56.98
53.35
54.45
54.22
54.93
52.95
53.51
54.53
Avg. (t)-
52.80
52.48
50.43
51.52
51.25
49.54
50.63
50.32
50.12
Wikitext ({)
17.35
18.76
18.29
Loss (L)-
2.151
2.161
2.258
2.156
2.209
2.249
2.227
2.233
2.204
Speed-up (1)-
49.1%
46.6%
-25.7%
48.6%
-0.7%
-17.9%
-13.8%
-15.4%
0.0%
Figure 3: We evaluate operators using training loss and Lambada [30], ARC-c (31), ARC-e [3]]
Logiga [32), PIQA [33], Sciq [34] Winogrande (35) and Wikitext PPL (36] totaling eight standard
NLP benchmarks. After 8 x 1020 FLOPs of training, G Girra demonstrates significant speedup.
Delving Deeper Into Depthwise Stacking (Gstack)
The empirical evidence suggests that certain growth operators, most notably GJ Ghneer exhibit impressive acceleration in LLM pre-training compared to the baseline approach of training models from scratch. We now turn our attention to a more in-depth examination of the direet For case of reference, we will henceforth denote this depthwise stacking approach as operator Gstuck
'Given growth factor g = 4, the sum of FLOPs for training d = 10B and D = 97. 5B approximately equals consumption for training large LLMs D 100B, which is the FLOPs of our baseline trained from scratch.
5

---

M = MoM.... M. where M is a small base model trained with d tokens, M is the target
gXM
model and g is the growth factor.
This section addresses the two main challenges (02 and 03) outlined the introduction: 1) evaluating the performance of Gruck in scaling scenarios, i.e. larger model sizes and more training tokens; and 2) determining the hyperparameters when using Guuck i.e., the growth timing and growth factor
4.1 Scaling Gstack
Scaling Model Sizes for Gstack+ Our scaled-up experiments in- volve two larger model sizes: 3B and 7B. We initially train smaller models with a layer count that is one-quarter of our target layers (growth factor g 4), utilizing 10B tokens (d = 10B). Then, we train the stacked models using over 300B tokens (D = 3)0B) for both sizes. Figures 4 and show the loss, and the NLP bench- marks average accuracy eval- uated using the Harness eval- uator for training 3B and 7B LLMs with 300B tokens, re- spectively. The acceleration 48.6% and 54.5% compared to scratch.
Tokens (Billig (Billions)
Tokens (Billions)
200
acrden
s2zich
54
Average Accuracy
Training Loss
233
2.4
266
50
35
2.2
44
10
30
FLOPs (1e+20)
FLOPS (1e+201
(a) Training Loss
(b) Average Accuracy
Figure 4: Training 3B LLMs with 300B tokens. Ganck significantly outperforms scratch (a) loss and (b) average accuracy across NLP benchmarks. At 180B and 240B tokens, Gstack accelerates by
Gauck is consistent across two models and all evaluation metrics. For instance, considering the 3B model, F demonstratte that Stack achieves 54.5% speedup in pre-training, improvement 2.1 in NLP benchmarks average accuracy compared to the baseline 3B model trained with 240B tokens.
When comparing the IB, 3B, and 7B models, it is evident that the benefits of Gstack not reduced as the model size increases, implying that its acceleration effect can be leveraged even with larger models. Details of the evaluation results, including evaluation with instruction tuning. can be found in Appendix D Appendix compares our baselines with the open-source LLMs Pythia and tiny
Tokens (Billions)
Tokens (Billions)
Scaling Training Tokens for Gstuck- We next evaluate the scalability the stacking opera- tor another dimension train- ing with more tokens. This especially important in light of recent discussions about the va- lidity of efficient training algo- rithms, which have sparked de- bate [22] over whether certain strategies may initially learn faster but ultimately perform similarly or worse than vanilla training methods when given more training data. Hence, we aim to pre-train LLM using Gstnck a substantial amount of training tokens.
200
250
300
100
200
s.20chh
54
Average Accuracy
52
Training Loss
50
1739
2.2
2.0
1.8
40
100
20
100
FLOPs (1e+20)
FLOPs (1e+201
(a) Training Loss
(b) Average Accuracy
Figure 5: Training 7B LLMs with 300B tokens. Gstack significantly outperforms scratch in (a) loss and (b) average accuracy across NLP benchmarks. At 160B, 220B and 280B tokens, Gstack accelerates by 40.8%, 55.3% and 53.8% compared to scratch.
'In this study, we always calculate the consumption by combining the FLOPs required for both training small models and large models. So given 4, the consumption for training small model d = 10B equals to the cost for training D = 2.5B, the plotted curves for Grack actually starts at 2.5B.

---

Concretely, we conduct an ex- periment on 410M LLM us-
Takens (Billions)
Takens (Billions)
400
8000
2.9
0.10
soude
ing 750B tokens. Following the
2.8
Training Difference
experimental setup in the previ- ous section, we set growth ra- tio g = 4 and growth timing d = 10B and conduct contin- uous pre-training on the target 410M LLMs for 750B tokens. Compared to the chinchilla- recommended 8B tokens [4] for the 410M model, our experi- mental setting also surpasses this value by nearly 100 times, reaching 750B tokens.
0.08
8
2:5
9 2.0
0.06
225
0.04
0.02
2.3
0.00
10 FLOPs (1e+20)
Flops (1e+20)
150
(a) Training Loss
(b) Estimated Loss Difference
Figure 6: Training 410M LLMs with 750B tokens. Gruck signifi-
The training dynamics on Fig- ure [6a] indicate that Gstack re- mains effective in such cases.
cantly outperforms scratch (a) loss. At 400B tokens, we observe 53.1% acceleration, and even at 700B tokens, there is still 31.0% acceleration. (b) We fit the difference between the losses of the scratch and Gtack and find that the acceleration with Stack remain sustainable for longer training.
Details of the evaluation results with the similar findings can be found in Appendix D.3 Building upon the exceptional stability of LLM pre-training 37 38
estimate loss improvements and plot them in Figure 6b The fitting curve indicates Gstakk will con- tinue to exhibit acceleration effects even after 8T tokens, which is over 1000 times longer than the recommended token number trivial for LLM pre-training, as previous studies (39 suggest that even minor improvements in the later phase can have a relatively substantial impact on downstream performance.
is also notable that this loss improvement after 8T training is not
From LLM practitioner's perspective, this is also crucial considering "overtraining". which involves training LLMs with significantly larger amounts of data than recommended by scaling laws (355), a common practice that has become prevalent. A notable example is the training of LLama 3-8B with 15T tokens, which is nearly 100 times greater than the token count recommended by the chinchilla scaling laws [4] Hence, this finding provides confidence in the consistent excellent acceleration of Grunc throughout the entire practical LLM pre-training process.
Estimating Scaling Laws. To further explore our find- ings, graph four models (410M, 1.IB, 3B, and 7B)
scrutchi41
Gauel78)
Grau((10MM
on the same figure and attempt to uncover our "scaling
scrstchh1181
Grae
*
acratch138)
law" using the Grtck operator. Following 3 4. de- fine the scaling power law using the equation Le = aC
serstch 38)
Gavel1381
Grau)38)
Training Loss
scratch/78)
Gaval 7001
where and are constants need to fit, C represents the FLOPs, and Le denotes the model's final loss under this FLOP. We use the curve_ filt function in SciPy [40] fit both the scratch model and the Ganek model and present the estimation scaling law in Figure 7 The figure shows that our Gstack scaling law exhibits improved efficiency com- pared to the scaling law estimated from baseline LLMs, achieving the same target loss while requiring much less computational resources. However, in light of the signifi- cant computational resources devoted to other scaling law studies [ 4. we acknowledge that our G scaling law is an initial estimate subject to computation constraints, and a comprehensive study is left for future research.
0.1 1 10 100 1000 FLOPs (1e+20)
Figure 7: We plot scaling law lines based on 410M, I.IB, 3B, 7B LLMs and make two predictions at the same losses of original computational- optimized 13B and 70B LLMs.
4.2 Determining Growth Timing and Growth Factor for Using Gstack
We comprehensively validate the effectiveness of the Ganck compared to training from scratch in Section 4.1 However, to incorporate Gsuck into a LLM's pre-training process, we need to determine two crucial hyperparameters: the growing time (d) and the growing factor (g). In our previous experiments, we rely ad-hoc choices for these parameters, thereby lacking systematic approach
7

---

to determining them when use Grack There exists research on investigating the growth timing 41 but the settings are quite different from the LLM pre-training. Therefore, this section offers clear guide for practitioners looking to optimize using the ssuck operator in L.M pre-training processes.
We begin by offering formal definition. When given computational budget C, established scaling power laws [3 [ exist to guide the non-embedding parameters N and the number of training tokens D achieve the lowest model loss in the case of training from scratch. However, tuning hyperparameters becomes more complex when the fixed budget C' is allocated to find the optimal model training strategy using the Grtack operator, which involves two training phases. Consequently, the overall computational budget C' can be expressed as the sum of the two components: C C1 + C2. Here, C1 and C22 represent the flops required to train the initial small models C'1 FFOPP((nd)) and the large model C2 FLOPs((N D) respectively, where n and denote the parameters and training tokens of the small model, and N and D represent the parameters and training tokens of the large model. Since the large model is grown by factor of g such that N gn, we have C = C1 + C2 FL. (gg, N, d) FLOPs(M D) FLOPs(g, N, D).
2.7
98
2.6
27
2.6
Training Loss
1.2
2.5
FLOPS (1e+ 20)
Training 1855
1.4
F [1e
1.6
1.8
2.3
2.0
2.2
2.2
Growth Timing (Billions of tokens)
Growth Timing (Billions of tokens)
Growth Timing (Billions of tokens)
(a) IsoFLOP on 410M
(b) IsoFLOP on 1.1B
(c) IsoFLOP on 3B
Figure 8: In 410M, I.IB, and 3B LLMs, plot smoothed loss curves for different growth timing d given a set of FLOPS to form IsoFLOP figures. We find a clear valley in loss, indicating that for a given FLOP budget, there exists optimal growth timing d for the Gstuck operation.
So when given budget C, our objective is to identify the optimized values D. N, d, that minimize the loss C(D, N,,,N, g). However, simultaneously optimizing the above four variables can be computationally expensive. Therefore, instead of searching for global optimals, separately determine two factors closely related to the Gsnak: the training tokens for the small model (growth timing) d and the growth factor g:
min C(D,N.d,0 where d= ( fhh
(1)
Determining Growth Timing: d. We first explore the effect of growth timing, i.e. the training token d for the small model. Particularly, we apply the Gauck operator to a series of small models trained with d OB, 1B, 5B, 10B, 20B, 50B tokens. Subsequently, we stack them to the target layers with growth factor g 4 and train for fixed set of computational FLOPs. We replicate the above experiments using three target model sizes N 410M. M. 1. LB, 3B and plot each set of IsoFLOP points Figure 8al 8b and Sc Surprisingly, even a small model trained with just 1B tokens exhibits a significant speedup compared the directly stacking small random initialized models (represented as "OB"). While OB's performance is similar models trained from scratch, implying stacking itself does not serve as an effec- tive initialization method. Furthermore, by applying smoothing techniques to model IsoFLOP curves parabolas, we identify the optimized value of d that minimizes loss for each FLOP count, leading to hypothesize the existence of logarithmic equation
( Sonltnn of tokens
Parameters (8 Illlns)
%
Growth liming
in
FLOPs (1e+20)
Figure 9: We fit contour figure for predicting d given and N. These optimal growth timing d fit the figure well. involving N, C, and d:

---

log1o(d) logio
+
log io
After fitting, we obtain a 0.88, 163.27 and - -5.74 and we plot the contour figure in
Figure It can be observed that our estimated curves align well with the actual optimal points.
Determining Growth Factor:
g. Another factor we deter- mine is the growth factor 9. As models with 3B and 7B pa- rameters have identical depths, we run experiments using two model sizes: 1.1B (24 layers)
2.9
2
Training Loss
FLOPs (1e+20)
Training Loss
FLOPs (1c+20)
2.6
2.8
and 3B (32 layers). Specifi- cally, vary the stack fac- tors to = 2, 4, 8, 24 for the 1.IB model and g 4, 8, 16,32 for the 3B model while keep- ing the base models trained with d = 10B tokens. The smoothed IsoFLOP curves are plotted in Figure 10 Interestingly, even with relatively shallow 2-layer base model and a growth factor of g 16, we observe a remarkable improvement compared to the baseline 3B model (g 1). However, when using 1-layer base model, Gstakk underperforms compared to the baseline. Our curves indicate that the optimal growth factor lies between 2 and 4.
2.4
2.6
2.2
(a) IsoFLOP on 1.1B
(b) IsoFLOP on
Figure 10: In 1.1B, and 3B LLMs, plot smoothed loss curves for different growth factor g given a set of FLOPs as IsoFLOP figures. The optimal g falls between 2 and 4.
However, unlike determining training token d, we cannot generate sufficient data to estimate the relationship between N, C, and g. due computational constraints. Thus, this work suggests constant growth factor of = 4. We also include our preliminary estimated equation and contour figure for g in the Appendix All evaluation results of Section 4.2 listed in Appendix G
Ablation and Discussion
To further give insights into adopting model growth techniques in LLM pre-training, ablate variances for Grack and discuss function preserving in general model growth techniques.
5.1 Ablation: How to Stack?
is worth noting that Gamck differs from the algorithm proposed in StackedBERT 14 which utilizes gradually stacking strategy. Hence, compare our "one-hop" Gruck and their gradual stacking approach. Following the methodology introduced in StackBERT, we employ two-step start with 6-layer model.
stack strategy. Given our target model size of 1.1B with 24 layers, Subsequently, we train it on 1OB tokens and double the model's depth through stacking, repeating this step twice stacin-tacc-traiin-- stack) to achieve the desired scale. Our experiments demonstrate that Guuck outperforms gradual stacking approaches on loss and downstream evaluations. For example, the evaluation results show that Gstakk achieves 2.4 higher average accuracy and 0.6 better Wikitext PPL than gradual stacking when pre- tra-training large models for 100B tokens. The results can be found in Appendix H..I. We further compare other stacking variations, such as stacking via interpolation and partial stacking of certain layers which are also adopted in LlamaPro 42 and Solar 43 and
leave our detailed findings the Appendix H.2 and H.3
5.2 Discussion: Why Does Function Preserving Fail?
Function preservation (FP) is key concept that underlies most model growth approaches 10 The idea is intuitive that larger model should initialize parameters that can represent the same

---

function the ones in the smaller model, i.e. Vx, (z; ( We give a mathematical definition of FP in the Appendix L.I
[az; (nt)): where is the input.
We find it intriguing that our Grack approach, which violates FP, emerges as the most effective in our study. To further investigate, we conduct a simple ablation study to break FP by introducing noise on the strict-FP operator direes- We initialize the new neurons by a weighted combination of two sets of parameters: those from Giieet and those from random initialization. The weighting factor is controlled by noise ratio. Our findings are intriguing. After 40B tokens training. adding 20% noise outperforms original Gireet by 0.27 on the Wikitext PPL and 0.41 on the average accuracy score.
We also add noise for Gatnck When we add 20% noise, our LLM performs slightly better than the no-noise model. However, when the noise level exceeds 20%, the performance significantly deteriorates. These results indicate that function preservation may not be the sole determining factor for model growth. In other words, exploring ways to accelerate the training of larger models and striet preserving f'unction during growth might represent two overlapping yet distinct research directions. The experimental details are provided in the Appendix 1.2
Conclusion
This work empirically explores model growth approaches for efficient LLM pre-training. We address three key challenges of current model growth research for efficient LLM pre-training. We first comprchensively evaluate model growth techniques into four atomic operators and explore depthwise growth Srack beats all other methods and baselines in various evaluations. We next address concerns about the scalability of Stack by extending the model and training data scales. Furthermore, systematically analyze the usage of the Guuuk operator, focusing on growth timing and growth factor. Based this analysis, we formalize set of guidelines for effectively utilizing the Gauck operator. In addition, we provide in-depth discussions and comprehensive ablation studies of Ganak, shedding light on the broader implications of our work.
Limitations
While our work has demonstrated remarkable potential, four limitations deserve further attention. One limitation is the constraint of computation resources. For example, we only compare two sets of growth factor d configurations, which limits the capacity to derive a formula for determining the optimal growth factor d. Another limitation of our work is the focus on relatively simple operator choices, where we prioritize simplicity over exploring more sophisticated strategies. For instance, we do not extensively investigate the multi-step growth or dynamic modifications to the training process, such as adjusting the learning rate during continual pre-training. The third limitation involves the incomplete cosine learning rate schedule during training. This also arises from the resource-intensive nature of pre-training LLMs and the constraints adopt a strategy where we initially set a large number of training tokens and then we pre-train LLMs until the training runs interrupted by tasks with higher priority. Lastly, although this study's scope is an empirical cxploration and the content self-contained, there is a lack of theoretical insights into the success of Gstuck in LLM pre-training Nonetheless, will release all LLM checkpoints to facilitate the community's 's investigation into the theoretical principles behind our observations.
available computational resources. Therefore, we
8 Acknowledgments
We thank all constructive comments from anonymous reviewers. Reynold Cheng and Wenyu Du were supported by the Hong Kong Jockey Club Charities Trust (Project 260920140), the University of Hong Kong (Project 109000579), the HKU Outstanding Research Student Supervisor Award 2022-23. and the HKU Faculty Exchange Award 2024 (Faculty of Engineering).
AA very recent paper indicates training LLMs via stacking may improve in reasoning [4]
10

---

References
[1] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
Cited on page I
[2] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models, 2022. Cited on [
[3] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. Cited on pages
and R
[4] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. Cited on pages 2 and g
[5] Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision, 2022. Cited on pages [and] 7
[6] Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, and Xuanzhe Liu. A survey of resource-efficient Ilm and multimodal foundation models, 2024. Cited on page [
[7] Al@Meta. Llama 3 model card, 2024. Cited on pages and 30
[8]
Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang. Charles Bai. Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. Sustainable ai: Environmental implications, challenges and opportunities, 2022. Cited on page [
[9] Alex de Vries. The growing energy footprint of artificial intelligence. Joule, 711))2191-2194, 2023. Cited on pagel]
[10] Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowl- edge transfer. arXiv preprint / 2015. Cited on pages
9 and 16
[11] Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang. Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. arXiv preprint arXiv:2 10.07143, 2021. Cited on pages
and [6]
[12] Yite Wang, Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin. Ruoyu Sun, and Hongxia Yang. Lemon: Lossless model expansion, 2023. Cited on pages 2
9 and 16
[13] Sheng Shen, Pete Walsh, Kurt Keutzer, Jesse Dodge, Matthew Peters, and Iz Beltagy. Staged training for transformer language models. In International Conference Machine Learning, pages 19893-19908. PMLR, 2022. Cited on pages
and 16
[14] Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In International conference on machine learning, pages 2337-2346. PMLR, 2019. Cited on pages 2 and 16

---

[15] Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning grow pretrained models for efficient transformer training. arXiv preprint arXiv:2303.009800, 2023. Cited on
pages
and 16
[16] Utku Evci, Bart van Merrienboer, Thomas Unterthiner, Max Vladymyrov, and Fabian Pe- dregosa. Gradmax: Growing neural networks using gradient information. arXiv preprint arXiv:2201.05/25, 2022. Cited on pages
and 16
[17] Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster language model pre-training, 2024. Cited on pages 2
and 16
[18) Cheng Yang, Shengnan Wang, Chao Yang, Yuechuan Li, Ru He, and Jingqiao Zhang. Progres- sively stacking 2.0: A multi-stage layerwise training method for bert training speedup. arXiv
preprint arXiv:201 13365, 2020. Cited on pages 3 and T6
[19] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. Cited on
page 2
[20] Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, and Yequan Wang. Flm-101b: An open Ilm and how to train it with $100k budget, 2023. Cited on pages
3 and
[21] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. Cited on pages 2 and 30
[22] Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt J. Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models, 2023.
Cited on pages and (
[23] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. Cited on pages 2 and
[24] Scott Fahlman and Christian Lebiere. The cascade-correlation learning architecture. In D. Touret- zky, editor, Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann 1989. Cited on page 3
[25] Scott E. Fahlman. The recurrent cascade-correlation architecture. In Neural Information Processing Systems, 1990. No citations.
[26] Steven Gutstein, Olac Fuentes, and Eric A. Freudenthal. Knowledge transfer in deep convolu- tional neural nets. In Int. J. Artif. Intell. Tools, 2007. Cited on page
[27] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidircctional transformers for language understanding, 2019. Cited on page 3
12

---

[28] Lemeng Wu, Bo Liu, Peter Stone, and Qiang Liu. Firefly neural architecture descent: general approach for growing neural networks, 2021. Cited on page 4
[29] Xin Yuan, Pedro Savarese, and Michael Maire. Accelerated training via incrementally growing neural networks using variance transfer and learning rate adaptation, 2023. Cited on page 4
[30] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoe Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. Cited on page 5
[31] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. Cited on page 5
[32] Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. Cited on
page 5
[33] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. Cited on page 5
[34] Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. Cited on page 5
[35] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. Cited on page 5
[36] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016. Cited on page 5
[37] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large language models, 2023. Cited on page 7
[38] Ziheng Jiang, Haibin Lin, Yinmin Zhong, Qi Huang, Yangrui Chen, Zhi Zhang, Yanghua Peng, Xiang Li, Cong Xie, Shibiao Nong, Yulu Jia, Sun He, Hongmin Chen, Zhihao Bai, Qi Hou, Shipeng Yan, Ding Zhou, Yiyao Sheng, Zhuo Jiang, Haohan Xu, Haoran Wei, Zhang Zhang, Pengfei Nie, Leqi Zou, Sida Zhao, Liang Xiang, Zherui Liu, Zhe Li, Xiaoying Jia, Jianxi Ye, Xin Jin, and Xin Liu. Megascale: Scaling large language model training to more than 10,000 gpus, 2024. Cited on page 7
[39] Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective, 2024. Cited on page
[40] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cour- napeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Bric Larson, CJ Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas. Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, Aditya Vijaykumar, Alessandro Pietro Bardelli, Alex Rothberg, Andreas Hilboll, Andreas Kloeckner, Anthony Scopatz, Antony Lee, Ariel Rokem, C. Nathan Woods, Chad Fulton, Charles Masson, Christian Häggström, Clark Fitzgerald, David A. Nicholson, David R. Hagen, Dmitri V. Pasechnik, Emanuele Olivetti, Eric Martin, Eric Wieser, Fabrice Silva, Felix Lenders, Florian Wilhelm, G. Young, Gavin A. Price, Gert-Ludwig Ingold, Gregory E. Allen, Gregory R. Lee, Hervé Audren, Irvin Probst, Jörg P. Dietrich, Jacob Silterra, James T Webber, Janko Slavič, Joel Nothman, Johannes Buchner, Johannes Kulick, Johannes L. Schönberger, José Vinícius de Miranda Cardoso, Joscha Reimer, Joseph Harrington, Juan Luis Cano Rodríguez, Juan Nunez- Justin Kuczynski, Kevin Tritz, Martin Thoma, Matthew Newville, Matthias Kümmerer, Maximilian Bolingbroke, Michael Tartre, Mikhail Pak, Nathaniel J. Smith, Nikolai Nowaczyk, Nikolay Shebanov, Oleksandr Pavlyk, Per A. Brodtkorb. Perry Lee, Robert T.
13

---

McGibbon, Roman Feldbauer, Sam Lewis, Sam Tygier, Scott Sievert, Sebastiano Vigna, Ste- fan Peterson, Surhud More, Tadeusz Pudlik, Takuya Oshima, Thomas Pingel, Thomas P. Robitaille, Thomas Spura, Thouis R. Jones, Tim Cera, Tim Leslie, Tiziano Zito, Tom Krauss, Utkarsh Ypadhyay, Yaroslav O. Halchenko, and Yoshiki Vázquez-Baeee Bazza. Scipy 1.0: fundamen- tal algorithms for scientific computing in python. Nature Methods, 17(3):261-273 February 2020. Cited on 7
[41] Haihang Wu, Wei Wang, Tamasha Malepathirana, Damith Senanayake, Denny Octomo, and Saman Halgamuge. When to grow? fitting risk-aware policy for layer growing in deep neural networks, 2024. Cited on page
[42] Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ping Luo, and Ying Shan. Llama pro: Progressive Ilama with block expansion. arXiv preprint arXiv:2401.0244 2024. Cited on pages and 37
[43] Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeon- woo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, al. Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling. arXiv preprint arXiv:2312. 15166, 2023.
Cited on pages andd 37
[44] Nikunj Saunshi, Stefani Karp, Shankar Krishnan, Sobhan Miryoosefi, Sashank J. Reddi, and Sanjiv Kumar. On the inductive bias of stacking towards improving reasoning. 2024. Cited on pagel
10
[45] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024. Cited on page 22
[46] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- exact attention with ao-awareness, 2022. Cited on page 22
[47] Daria Soboleva, Faisal Al-Khateeh, Robert Myers, Jacob R Steeves, Joel
Hestness, and Nolan Dey. SlimPajama: http::///wwwwccebrrss...o. www.brareeeeeeeeee.. net./llog plicated version of RedPajama.
627B token cleaned and dedu-
https: s1 slimpa jama a 627b token-cleance and-dedduee wwwpdddddccddddeeeeeeeeee versioo of redpa jama
2023. Cited on page 22
[48] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. Cited on page 27
[49] Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2022. Cited on page 27
[50] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. Cited on page
27
[51] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference Machine Learning, pages 2930- PMLR, 2023. Cited on page
29
[52] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling, arXiv preprint arXiv: 00.00002, 2020. Cited on pages 29 and 39
[53] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A.
14

---

Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024. Cited on page 39
[54] Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto. Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Roberto Iriondo, Cun Mu, Zhiting Hu, Mark Schulze, Preslav Nakov, Tim Baldwin, and Eric P. Xing. Llm360: Towards fully transparent open-source Ilms, 2023. Cited on page 39
[55] Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Rus- sell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen. Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy. Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint, 2024. Cited on page 39
15

---

A Details of Growth Operators
A.1 Four Growth Operators
A.1.1 Operator :direct: Direct Derivation of Grown Parameters From Old Parameters
One intuitive strategy for expanding neural networks involves directly duplicating splitting existing neurons. [14] [1] [2]. Unlike other growth operators, we distinguish between growth in terms of depth and width.
For width-wise expansion, the Net2Net technique and its transformer implementations involve splitting old neurons into two or more parts, with each splitting step achieving a=b+c. Depending on the specific splitting mechanism, there are two variations: even splitting and uneven splitting. The latter is proposed to address symmetry issues that arise when neurons are evenly split. In this paper, we adopt the approach of uneven splitting.
In the context of desth-wise expansion. a common practice is to duplicate layers, often referred to as "stacking" [14). Therefore, we use the term Gtack to represent this operator. While this approach may appear to deviate from function preservation, it surprisingly yields a strong baseline.
A.1.2 Operator Glearn: Generation of New Parameters through Matrix Transformation
Gream is an operator that learns a matrix transformation function to map small models to larger one [[S]. This operator is applicable both width and depth expansion. Considering the original model with parameters 0, the target model F with parameters O and Gteam as the hypernetwork for meta-learning, the training corpus denoted as D, and the language model loss denoted as Then, we optimize the following objective:
arg min EnnD L(x; Fe), where θ = Greamn (0) Gream
(3)
A.1.3 Operator Gzero: Setting New Parameters to 0
Setting new parameters zero is often considered simple method achieve function preservation. However, optimizing networks with a significant number of zeros can present challenges. To tackle this issue, we adopt current practices that selectively zero out either the fan-in or fan-out parameters [13 [6 [2]. Specifically, for operator Gzen. during width growing, we zero out only the set of fan-out parameters for new neurons and randomly initialize the remaining ones. the case of depthwise expansion, zero out the final output layer of the newly- newly-duplicated transformer blocks' MultiHead Attention and MLP.
A.1.4 Operator Grandom: Random Initialization of New Parameters
This group follows the common practice of randomly initializing new parameters. In earlier attempts, old neurons were frozen after the growth process [18 17.. However, to ensure function preservation, recent study introduces a mask for new neurons after expansion [17]. This mask is gradually removed during ongoing training. We refer to this new approach the growth operator Grandom-
A.2 Difference of Our Operators and Base Methods
The operators Gairee shares similar setting Lemon with minor variances due Llama achitectures. Gleam is consistent with the methods LiGO, but with our own implementation. For Gyem: our approach aligns with Lemon in terms of depth, but differs from staged'Training in width. Unlike stagedTTraining, we do not double the width and assign zeros to the off-diagonal entries. Instead, our approach is more flexible; by zeroing out the submatrix the bottom-left corner, can extend it to any dimension. Our Gramdom does not exhibit the "multi-hop" growth like MSG, instead, it grows "one-hop" directly to the target size. Our implementation of Gf (Gstuck) differs from the algorithm employed in stackedBert. stackedBert, a gradual growing technique is utilized, whereas our operator follows more direct approach.
16

---

A.3 Details of Gdirect
Embedding Consider E TR Vxd and our goal is to expand it E' E TRVVD Gdirea just copy
some columns:
E*
Gairect (E)
(4)
ER
(5)
I
E
(6)
1
d
where R E RaxD is used to copy the embedding matrix E.
Linear Consider W E RRdoue target parameter W E R Doutx where dout Dout, din S
D Din G direct is defined as:
W
Girrect (W)
LWR R
(8)
dout
I
W
Q
din
where R R is used for expanding the fan-in and L E R Poue Xdout is used for expanding the fan-out, To satisfy function preserving, ensure that a + 3 = T.
RMSNorm For RMSNorm, similar approach is adopted, consider parameter μ R expanded La parameter μ² vD [M, Po. e RD:
RMSNorm' (x')
(10)
Eln x2
Fd
i= x x RMSNorm(2)) <)
(11)
x
Therefore, using the Gairect not possible to achieve function preservation for RMSNorm
Depth (Gstack) Consider transformer with layers represented as F= fo fi o ... o fi. Our objective is expand it layers, where L multiple of We have various stacking forms for this purpose, such as (a) direct stacking: F² = Fo F Fo...o F.
Algorithm 1 Operator Grack
Input: Base model Mt with / layers trained using dataset dy where is iteration steps. Growth factor
Output: Target Model Mgi with gl layers Mb=M}
for 2 to g do
Model Stacking
M = 0 (1-1) M end
17

---

A.4 Details of Geero
Embedding Consider embedding matrix E Vxd The Gzer operator expands it to F' RVxD with O, where d S D. Formally:
E O]
(12)
Therefore, give token x, the expanded embedding can be expressed as:
Embedding (a) 1g (Embeeding(()) 0D-d]
(13)
Linear Consider parameter W E Ridouu xden Gzero expand it to W E PRDexxxDss where dout S Dou and din I Din- Formally:
W
A C
W
(14)
0
where A. C are randomly initialized parameters. Considering the input token æ E Rdh before expansion, and the input after expansion x' E R Pin:
x' [z, Opin
(15)
Linear' (x')
WT
(16)
WT O AT CT
= [x,, -D
(17)
LxWT, 0 O douel [Linear(x), Opone-doull
(18)
(19)
RMSNorm Considering the parameter fa IRd, Gzem expand it to pr' [ap,, { like Grandom in Appendix A.S because the input must be = [xx OD-al R.D
Depth In depth, by retaining only the residual part and initializing the MHA and SwiGLU final linear projections to zero, the MHA and SwiGLU layers can achieve function preservation.
A.5 Details of Grandom
Embedding Consider an embedding matrix E E RVx, The goal of Grandom is to expand it to E' E'ERV VVD where d D. Formally:
E' E' E, E3
(20)
where & E (DD-d)) represents randomly initialized new parameters. We use mask E R to mask out the randomly initialized parts:
1d, 0p-a] - [1a, D
(21)
Therefore, for a token x, the masked embedding can be expressed as:
Embedding (x) E, o [Embedding(z), Op-al
(22)
18

---

Linear Consider parameter W Xdin Our goal is to expand it to W E P where dout I D Dut and din I Din Formally:
W
A C
W
(23)
B
where A, B, C. are randomly initialized new parameters. Considering the input token E Rd before expansion, and the input after expansion x' E R Pin:
s' (2c, OD... -dnl
(24)
WT BT AT CT
z'W'
[z, OD... ddin I=: xB²1
(25)
(26)
To ensure that the expanded part of x starts with zeros, we still utilize a mask:
c [Iduue> ODaut-doutl + [Adouer Doue-dlourl
(27)
Linear (x') x I' c [Linear(x), O
(28)
RMSNorm Considering the parameter f E RRd, our objective is to expand it μ = (ap E E RRD, where is an undetermined coefficient and & is a randomly initialized new parameter. Let the input be x' = [x, Op-a] RP, then we have:
Σ
(29)
1=0
1=0
x'
RM NNrr'
(30)
1 Ei 0
x2
xx, Op-d)
lap, E
(31)
D EE=0, x;2
VD x
au, Op-d
(32)
Vd
E5
By observing equation conclude:
32 we can conclude that, to achieve function preservation, a
Vd Finally,
RMSNorm' [RMSNorm(z)) Op-a]
(33)
Depth
depth, preserving only the residual part and masking the MHA and SwiGLU layers can achieve function preservation:
Y = X + MHA(RMSNorm((X) O
(34)
X SwiGUU (RMSNorm O
(35)
0p + lp
(36)
19

---

A.6 Details of Geearn
Using Gleam for width expansion, for the embedding layer B E RRVd the parameter Bemb R Dxd defined as follows:
E'
E emb
(37)
For Attention layer, where WQ, WK, Wv, and Wo Idddd and RMSNorm 141 Rd the parameters Bq, BK, and By E Rxd, have:
W = BoWaB
emb W = B BT
W. By B B
gmb emcy
(38)
WO Wo
P1 =
Bemb/11
For MLP, where Wup, W Waate Rdmdp xd W Wdown E Rdxxmip RMSNorM (42 E Rd, the parameter
Bmlp E Dmip Xdmlp, we have:
W
Bmip W up B Bemb
W
Bemb W B mlp
down
(39)
W
Bmip W gate Bdd emb
gote
1/2
Bembl
For the output head W E R xd
we have:
W W Wemb
(40)
Using Gteam for depth expansion, consider transformer model with L1 layers, we use Gleam to expand it to Le layers. For / E {1,2, ... :o):
wor D
W = Σ DK Wi
wy
wo E DV Dy
(41)
E/ DI Wi
(wa)) E/ ((ln)) my
(mm)
where DV..VV L2 represents learnable parameters. These parameters are used to expand the MHA vertically in depth. Similarly, for SwiGLU, we also perform expansion using similar method. Formally, this can be written as:
Whp
Ih D WHP
Wdoun'
= 1= Doown Wiown
Whote!
Σ Doate Wiaat
(42)
=
(l22)'
ΣTH D(h22) (bn2) (ln2)
Mi
where duw.down.ggate.ng E RLaxL represents learnable parameters used for expanding SwiGL. in the depth.
20

---

B LLMs Framework and Training Details
Embedding Consider vocabulary size V and embedding size d. Then, the embedding matrix E E gVxe and the one-hot vector for input tokens X is denoted 1x E R where ' is the sequence length. Formally, it can be written as:
Embedding((X X)= 11x
(43)
for i, [V], where * j. is guaranteed that E; F E.j.
Multi-Head Attention Multi-Head Attention (MHA) consists of multiple attention heads, each of which computes its own self-attention. The results of these attention heads then concatenated and projected obtain the following output:
Q₁, K,, V XW XWK XWV
H. = so Imarr Q.K. MHA(X) = Concat
vah VV,
(44)
H
here, the input X E Rxx, parameters W.O E Rdxda, w.K E Rdxdi wY E Rdxd, and wo pdxd, where n x d₃ = d.
Feed Forward Network The Feed Forward Network (FFN) consists of two linear layers and the activation function GeLU. Typically, the two linear layers first perform an up-projection to dpFN and then down-project back to the dimension d. Therefore, FFN is defined as:
FFN(X) GeLU(XW up) W down
(45)
where the input X E RRxx parameter Wap E &RdxdFr and Wdman E BddENNd
SwiGLU LLaMA replaces the original FFN in the Transformer Decoder with SwiGLU, resulting in improved performance. SwiGLU consists of three linear layers and the swiglu activation function. can be defined as:
SwiGLU(X) = (XW Wgate O swiglu(XW W down
(46)
where 2 means the element-wise multiplication, the input X Rxxd parameter Wap RdxdFFN W gate pdxdErN and W down E RdeeNxe xd
RMSNorm Before MHA, FFN, or SwiGLU, there is a layer of RMSNorm enhance the stability of the model. Compared to LayerNorm, RMSNorm is simpler in form. Formally, it can be written as:
RMSNorm(X)) =
O
(47)
V V
x?
where X E Txxd parameter μ Rdd
21

---

B.1 LLMs Training with Growth Operator
Algorithm 2 LLMs Training with Growth Operator
Input:
Growth operator G. Loss function C. Iterative optimizer A. Dataset {d1, d2, ... dp) for base model. Dataset {D1, D2, Dxx } for target model.
Output: Target Model Mr
Initial Phase: Initialize base model Mo from scratch.
for t to 1: do
Base Model Training
loss L(M d+)
M, + A(M2-1,loss)
end
Mo = G(Ma))
for to K do
Target Model Training
loss [(Me-1, Dz)
M, A(M
end
B.2 Details of Speedup Calculation
We calculate speedup between operator G and scratch model pre-training by:
PT..Psscratte
sp =
(48)
FLOPsG
where FLOPss scratch and FLOPss represent the FLOPs required by the scratch model and the G model, respectively, to achieve the same loss.
B.3 Details of Training Settings
We use TinyLlama 45 as our pre-training codebase. We employ FSDP (Fully Sharded DataParallel) along with FlashAttention 46) 2.0, and other acceleration techniques. We use the open-source dataset Slimpajama-627B 47 for pre-training. The hyperparameters used for each model size are listed in Table I
Our 7B model is trained over around 100B tokens per day on NVIDIA Hopper cluster.
Table 1: Hyperparameters
Size
Context Length
Batch Size
max-IR
min-LR
Warmup Steps
Scheduler
410M
2048
2M tokens
6e-4
6e-5
3000
cosine
1.IB
2048
2M tokens
3e-4
3e-5
3000
cosine
3B
2048
2M tokens
1.6e-4
166-5
3000
cosine
7B
2048
2M tokens
lc-4
le-5
3000
cosine
C Training Loss and Evaluation Results of Four Operators in both Depth and Width growth
We have two small (base) models, one trained with token count d = 10BB and another trained with token count 50B.
Apache-2 2.0 license
'The license of Slimpajama-622 627B includes: Common Crawl Foundation Terms of Use; C4 license; GitHub was limited to MIT. BSD. or Apache licenses only: Books: the the_pile____ books3 license and pg19 19 license; ArXiv Terms of Use: Wikipedia License; StackExchange license the Internet Archive
22

---

Tokens (Billions) 40
Tokens (Billions) 40
60
80
20
60
80
3.0
3.0
Grinton
Guinon
2.9
Gane
Gaéra
2.9
Gane
Gian
Glam
2.8
2.8
Training Loss
2.7
Training Loss
2.7
2,25
2.25
2.6
2.6
2.20
2.20
2.5
2.5
2,15
2.15
2.4
2.4
2.3
2.3
2.2
2.2
2.1
2.1
4
8
2
4
8
FLOPs (le +2)+
FLOPs (1e+20)
(a) Growing in depth from small model (10B)
(b) Growing depth from small model (50B)
Tokens (Billions)
Tokens (Billions)
20
40
60
80
0
10
20
30
40
50
60
3.0
Frndow
Grivon
2.9
Gaiua
Gjir
3.2
Gaiua
Giuz
Giam
Giam
2.8
70
80
3.0
50
Training Loss
2.7
2,30
Training Loss
2.40
2.6
2.25
2.8
2.30
2.5
2.6
4.0
4.5
2.4
2.3
2.4
2.2
2.2
0
FLOPs (le+ +20)
FLOPs (1e+20)
(c) Growing in width from small model (10B)
(d) Growing in width from small model (50B)
Figure 11: Training Loss on Slimpajama.
23

---

23
AccurSc
Accumcy
Accursay
FLOPs (1)
FLOPS (1cr20
FLOPS (le
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Acc 1)
Tojons (BBf
Takons (Bi)
Trlond Mons)
2e
Accuart
FLOPs (1e+200
FLOPs (1e+200
TLOPs (1e+20)
FLOPs Cle+20)
(c) PIQA (Ace f)
(f) Sciq (Ace 1)
Winogrande (Acc 1)
(h) Wikitext (ppl 4)
Figure 12: Evaluation results on growth depth from small model (10B) by four operators.
Trbens 18 (BHloo))
Tokana (hillons)
Tohans (B1) crr)
Tibers 13 Hoona)
62.
2
Ascurso)
Socursor
25
42
FLOPS (1)
FLOPS (1e:221
(a) ARC-c (Acc
(b) ARC-- (Acc 1)
(c) Lambada (Acc
(d) Logiqa (Acc bagan 19 Hons)
Tolonn (811ion))
Tokgna (311ions)
Tokans (8i))
21
2
Word Popieaty
Arcurry
02
(le+2)
(e) PIQA (Ace +)
(1) Sciq (Ace 1)
(g) Winogrande (Ace ?)
(h) Wikitext (ppl !)
Figure 13: Evaluation results on growth in depth from small model (50B) by four operators.
24

---

(Bi)
Accursc
Accumcyy
36
23
FLOPS (10
FLOPS 20)
FLOPS (1cr20)
FLOPS (Le
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace 1)
Tolonn (88f
(Bllions)
Takong (Bi)
Mons)
50
Word Paploday
53
20
FLOPs (1e+200
FLOPs (1e+20)
TLOPs (1e+20)
FLOPs cle+20
(c) PIQA (Ace f)
(f) Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 14: Evaluation results on growth width from small model (10B) by four operators.
Trbens 18 (BHloos)
Tokens (hillons)
Tobaos (Bil)om))
Tobens 13 Hona))
Ancurncy 12
Accurser
FLOPS (10120)
FLOPS
FLOPS 220
ARC-c (Acc
(b) ARC- (Acc 1)
(c) Lambada (Acc
(d) Logiqa (Ace
Tokonn (811ion))
Tokena (8111ons)
Tobans (Bi))
bignn 19 Hons)
Word Popioaty
Arcuncy
20
54
15
(le+20)
(e) PIQA (Ace 1)
Sciq (Ace 1)
Winogrande (Ace ?)
(h) Wikitext (ppl !)
Figure 15: Evaluation results on growth in width from small model (50B) by four operators.
25

---

Agd Aggrage Aqourayy
Aweryy Accuncyy
Avcrge Acquracy 5
FLOPs (1)
RLOPS 20)
(1c220
FLOPS (ee
(a) Growing depth from (b) Growing in depth from (c) Growing in width from (d) Growing width from small model (10B)
small model (50B)
small model (1OB)
small model (50B)
Figure 16: Average accuracy seven standard NL.P benchmarks.
D Evaluation Results of Scaling G
D.1 3B
Tokens (Billions) 150
100
200
250
300
56
scratch
G
54
Average Accuracy
52
50
48
46
44
10
20
30
40
FLOPs (1e+20)
Figure 17: Average accuracy of standard NLP benchmarks at 3B size.
26

---

120
230
200
Acurncy
Accuray
FLOFS (1)
FLOPS 20)
FLOPS (1c120)
FLOPS (10
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace 1)
(Blllios)
Tokong (Bil
100
150
250
1P0
130 270
291 mn
100 150 de
200 100
Word Paploday
FLOPs 21e+200
FLOPs 11e+200
T1O0gs (1e+20)
FLOPs cle
(e) PIQA (Ace 1)
(f) Sciq (Ace f)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 18: Evaluation results on scratch model and Ssackk model 3B size.
D.2 7B
Takons (looos 0x2
(iillons) Do
Tikons (Bil 102
100
100
35
Acourry
Acoino
5
22
26
PLDP%
1L.0% (1.+220)
FLDON (1++20)
FLOPs
(a) ARC-c (Ace 1)
(b) ARC-e (Ace 1)
(c) Lambada (Acc 1)
(d) Logiga (Acc 1)
Titens (Bilios)
Tekenn (Dilloon) 120
(Bil aro)
Tobens 13 Hooa)
209
2
12
20
PLOPS
FLOPS
(c) PIQA (Acc
Sciq (Ace 1)
Winogrande (Ace ?)
(h) Hellaswag 48 (Ace 1)
Takons (9lllos)
Tokang (Siligns)
Iakons (Bil
Totans Hooes)
62
Word Pepiorty
Averge Aocureyy
Accurac
60
80
116
400
20
63
60 30
(1e+20)
(i) TruthfulQA[99 49
(Acc f)
()) MMLU
50 (Ace t
(k) Wikitext (ppl L)
(1) Avg. (Ace 1)
Figure 19: Evaluation results on scratch model and Getnck model at 7B size.
27

---

D.3 410M
To kens (Billions)
200
400
600
scratch
49
48
Average Accuracy
47
46
45
44
43
42
2.5
5.0
7.5
10.0
12.5
15.0
FLOPs (1e+20)
Figure 20: Average accuracy of standard NLP benchmarks at 410M size.
Diens (allnd))
[tillons)
[i and)
Hona))
200
600
230
200
500
Accurpa
Rccurny
26
FLOPS (1.22)
FLOPs 21
FLOPs (16+20)
FLOPs cle+0
(a) ARC-c (Acc 1)
(b) e (Ace 1)
(c) Lambada (Ace 1)
(d) Logiga (Ace
Jolama 18Hlons)
Tokana (taliona)
Johans (Bil
plaa 15 Hons)
230
2.
Vorr Repoott
Accomcy
77
21
10.0
125
FLOR
1L.0%
FLDes (1e+20)
FLOS
(c) PIQA (Ace 1)
(f) Sciq (Ace 1)
Winogrande (Ace ?)
(h) Wikitext (ppl 4)
Figure 21: Evaluation results on scratch model and Gruck model at 410M size.
D.4
Instruction Tuning Results on 3B
Table 2: Evaluation Results after Instruction-Tuning (Higher better)
Method
Tokens
Tuning
lambada
are-e
logiqa
piqa
scig
winogrande
avg
54.07
28.84
55.35
26.88
73.94
82.0
59.43
54.36
scratch
400B
60.35
31.48
27.04
74.32
81.2
55.8
x
55,04
32.34
558.08
28.88
73.88
79.6
61.8
35.66
Guac
2908
61.34
34.98
59.97
29.65
75.14
80.1
60.22
57.34
28

---

E Compare with Other Opensource LLMs
In Table wC compare the harness evaluation results after training the Stuck model and the scratch model (Baseline) for 100B tokens with Pythia- [5T] and TinyLLama-I.IB. which trained on the same number of tokens. The comparative results indicate that our baseline performs normally, comparable to pythia- IB. Meanwhile, the Ganck model significantly outperforms both the baseline and pythia- IB, demonstrating the acceleration effect of Gstack on the pre-training process.
Table 3: Compare with opensource LLMs IB
Pythia-IB
TinyLlama-I.1B
Baseline-
Datasets
Pile-300B 522
Slimpajama-62778 Starcoder
Slimpajama-6277B
Slimpajama-6e
Tokens
100B
103B
100B
100B
lambada
53.52
48.20
47.87
ARC-c
25.59
24.32
29.18
27.21
ARC-e
47.26
44.91
54.25
48.86
piqa
69.31
67.30
71.98
69.64
logiqa
29.49
28.87
25.96
sciq
77.3
81.1
76.8
winogrande
51.22
53,28
56.03
54.53
Avg.
50.53
52.80
50.09
Fitting Results for the Growth Factor g
Although due to computational resource limitations, we only explore predicting g given N and C on the 1.IB and 3B models, we still attempted to fit using equation:
1og (g)
log (N) + log1o(C)
+
(49)
In the equation 49 N represents the number of target parameters, g represents the growth factor. The fitting result is as follows:
29.88
(g) 1.01 (N)
7.36
(50)
(C)
We also visualize the fitted curves in Figure but the results were mediocre due to the lack of data.
12
5
10
-4
Parameters (Billions)
Growth Factor
0000000000005000
10
15
FLOPs (1e+20)
Figure 22: Visualization of the Equation 50
29

---

F1
Stacking Law Guidelines For Llama Families
We give example of empirical usage of stack by using the configurations of Llama2 and Llama3 families [21]7 to show the estimated optimal base model training tokens d and growth factor g in Table 4
Table 4: "Stacking Law" Guidelines
Model
N
D
g
Llama3-8B
8B
15T
6,58B
4
Llama2-7B
7B
2T
11.1IB
Llama2-13B 13B
13B
2T
15.84B
4
Llama2-700 70B
70B
2T
42.48B
4
G Training Loss and Evaluation Results of "growth timing" and "growth factor
G.1 "Growth Timing" d
Tokens (Billions)
Tokens 40
(Billions) 60
60
3.6
scratch
Gus/61: 108)
47
scratch
16 108)
Ga 18)
(60
Gaax
G (00 208)
3.4
Gouee 58)
Gus16L:508)
Gueee66;;
(BL (6L:500)
46
Average Accuracy
Training Loss
3.2
45
3.0
2.4
44
2.8
1.6
1.
43
2.6
2.4
0.0
0.5
1.0
1.5
2.0
0.5
1.0
1.5
2.0
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 23: Training loss and standard NLP benchmarks average accuracy of 410M.
30

---

24
20
20
12
20
Soatlh 2002
Toaath
2b.
FLORS (1)
FLOPS 11c
FLOPS (Acr20)
FLOPS (1e+20)
(a) ARC-c (Acc f)
(b) ARC-c (Ace t)
(c) Lambada (Ace 1)
(d) Logiqa (Ace f)
Toloens (Bltons)
(Bllion))
Monc)
40
Word Poploday
FLOPs 21e
FLOPs 11e+200
TLOPs (1e+20)
FLOPs Cle
(c) PIQA (Ace 1)
( Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 24: Evaluation results on 4 410M.
Tokens (Billions)
Tokens (Billions)
60
80
3.0
scratch
G6s 108)
2.9
Ga 18)
(60)2)
Goued (61:
(us16L:508)
52
2.8
80
Average Accuracy
Training Loss
2.7
50
2.25
2.6
2.20
48
2.5
2.15
2.4
2.3
Scratch
G6vs/61, 108)
2.2
26)))
Gueel6i; 58)
G (61: 508)
2.1
2
4
8
2
6
8
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 25: Training loss and standard NLP benchmarks average accuracy of 1.1B.
31

---

23
ACCUrSC)
Accursay
24
FLOPs (1c120)
FLOPS 20)
FLOPS (Acr20)
FLOPS (20
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace f)
Tojons (B
Tokeri (Bilions)
Takonn (Bi)
Trions Mons)
2e
Word Poplodty
10:
Tsathh::
FLOPs 21e
FLOPs 11e+200
TLOPs (1e+20)
FLOPs Cle
(e) PIQA (Ace 1)
(f) Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 26: Evaluation results I.IB.
Tokens (Billions)
Tokens (Billions) 40
60
80
60
scratch
Gunec (BL. 2081
3.0
Guas/81118 18)
Gaax: (BL: 508)
Gus//LL:BB
Gluce 1008)
Guss/LL:108
Average Accuracy
2.8
Training Loss
2.25
50
2.20
2.6
2.15
2.4
10
46
scratch
(8L, 2081
2.2
Gras(BL;
Gaac (BL: 508)
Gas
Gaac. 1008)
Guu//L; 108)
42
0.0
2.5
5.0
7.5
10.0
12.5
2
6
10
12
14
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 27: Training loss and standard NLP benchmarks average accuracy of 3B.
32

---

22
Accunry
Stutthc 2M1
Traditi 100
FLOPs (1)
FLOPS 21)
FLOPS (1cr22)
FLOPS (1o+20)
(a) ARC-c (Ace f)
(b) ARC-c (Ace 1)
(c) Lambada (Ace 1)
(d) Logiqa (Ace f)
Toloes (B1l
Tokeno 03
Takong (Bi)
Trions Monc)
2/2
2
Goallt Riit
2
Word Paplodty
66
13
2011
15
FLOPs (1e+20)
FLOPs 11e+200
TLOPs (1e+20)
FLOPs cle
(c) PIQA (Ace 1)
( Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4
Figure 28: Evaluation results on 3B.
G.2 "Growth Factor" g
Tokens (Billions)
Tokens (Billions)
60
80
scratch
Gus/BL 8)
3.0
(1 x2)
GuaflL 24)
52
(6
80
Average Accuracy
50
Training Loss
2.8
2.30
2.25
48
2.6
2.30
46
7315
2.4
44
42
Scratch
Gua
2.2
Gu 24)
40
Go x
2
4
8
2
6
8
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 29: Training loss and standard NLP benchmarks average accuracy of 1.1B.
33

---

AccurSc
Accumnyy
Accunao
Smath PM
FLOPS (10220)
FLOPS (1))20)
FLOPS (1cr200
FLOPS (1e
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace ()
(d) Logiqa (Ace 1)
Toloon (B8)ons)
Tikons (Bi)
Mons)
80
20
4)
21
Star
FLOPs
RL0% (1e+20)
TLOPs (le+20)
FLOPs cle
(c) PIQA (Ace 1)
(f) Sciq (Ace f)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 30: Evaluation results on 1.IB.
Tokens (Billions)
Tokens 40
(Billions) 60
60
80
scratch
Gus 16)
3.0
Ga
GuaflL 32)
52
G
Average Accuracy
50
2.8
Training Loss
2.25
2,20
2.6
46
2.15
44
2.4
10
2.2
scratch
Guaa 16)
GovelB
Gua
38
0.0
2.5
5.0
7.5
10.0
12.5
2
10
12
14
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 31: Training loss and standard NLP benchmarks average accuracy of 3B.
34

---

22
57
Accumcyy -5
25
Starhh
FLOPs (10220)
FLOPS 110+
FLOPS (1cr20)
FLOPS (10+20)
(a) ARC-c (Acc f)
(b) ARC-e (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace 1)
Toloon (B1l
Tokeno
Takono (Bi))
Trinn Monc)
2/2
50
AGDUOCC 3
Word Poploday
63
FLOPs
FLOPs 11e+20)
TLOPs (1e+20)
FLOPs cle
(c) PIQA (Ace 1)
(f) Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 32: Evaluation results on 3B.
H Discussion on "How to stack?" and Evaluation Results
H.1
Training Loss and Evaluation Results of Gradual Stack
Tokens (Billions) 40
Tokens (Billions)
20
60
80
20
40
60
80
3.0
Gyosua
Gymese
2.9
Gaax 108)
Gyssua [121,
Gauex
2.8
70
80
Average Accuracy
Training Loss
2.7
2.6
2,20
48
2.5
2.15
2.4
46
2.3
2.2
44
2.1
2
FLOPs (1e+20
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 33: Training loss and standard NLP benchmarks average accuracy of scratch, Guack and
G gradual
35

---

23
ACCurSc)
Accumcy
Accurso
24
FLOPs (1)
FLOPS (1c120)
FLOPS (1e
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace f)
Tolons (B8fons)
Biong (Bi)
Mons)
89
2e
Word Papledoy
FLOPs (1e+20)
FLOPS (1e+20)
TLOPs (1++20)
FLOPs Cle+20)
(e) PIQA (Ace f)
(f) Sciq (Ace f)
Winogrande (Ace 1)
(h) Wikitext (ppl !)
Figure 34: Evaluation results on scratch, Stack and gradual stacking StackBert.
H.2
Ablation: /2 o fi o fo f2 o or 12 S So fo (interpolation)
To investigate whether the connections between layers affect the performance of stacking, we conduct comparison of two approaches for stacking small models into larger ones. We explore two approaches for stacking small models into larger ones. The first approach involves taking the entire small model unit and directly stacking which can retain the connections between most layers. The second approach involves replicating and interleaving each layer in the small model, which almost break the connections. To measure the degree of retention of inter-layer connections after stacking, we define the connection rate Ra:
Con
R.
(51)
Con
where the Con, is number of retained connections, the Conau is number of connections.
For example, if we had a small model with three layers, denoted as f2 a fi o fo. and desired a model depth of the first approach would result in fa o fi fo fa fi fo. where its R. 80%. The second approach would result in f2 f2 fi f1 fo fo. where its R. = 40%.
In experiments, we stack small model with 8 layers timing is 10B tokens and growing factor s is 3. The R. of Gana 91.3% and the R. of Ginterpoiate 30.4%. We report the training loss and standard NLP benchmarks average accuracy in Figure 35 At the beginning of training. interpolated stacking perform as well as stacking entire small model. However, the training continues, the performance of interpolated stacking deteriorates.
24 layers target model. The growth
Therefore, we can conclude that the higher the connection rate of stacking, the better the effect of stacking. In Appendix H.3 continue to validate this conclusion.
36

---

Tokens (Billions)
Tokens (Billions)
20
40
60
80
20
40
60
80
Scratch
Guack
scratch
3.0
G vireppion
Gyterpatie
52
Average Accuracy
2.8
Training Loss
2,25
50
2.20
2.66
2.15
2.4
10
12
46
2.2
42
0.0
2.5
5.0
7.5
10.0
12.5
2
6
10
12
14
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 35: Training loss and standard NLP benchmarks average accuracy of scratch, Gtack and interpolation.
We also report the details of evaluation results about standard NLP benchmarks.
Jokena 18
Tohens (BI (Bil
22
69
00
40
62
23
30
20
61
80
14
A4C0/00
Accurtty
Accurscy
23
FLOPS FLOPs (1o+20)
FLOPs (1c+200
FLOPS (1c+20)
FLOPs (lo+20)
(a) ARC-c (Acc 1)
(b) ARC-e (Ace 1)
(c) Lambada (Ace +)
(d) Logiga (Ace 1)
Tolons (811ons)
Tokens (Billons)
Talono (BIl
Diions 18 Hond)
Word Portloity
Arrurcy
66
FLOPS (1++200
F1.0%
TLOPs (1++20)
FLOP
(c) PIQA (Acc
(f) Sciq (Ace 1)
Winogrande (Ace ?)
(h) Wikitext (ppl 4)
Figure 36: Evaluation results on scratch, Gruck and interpolation.
H.3 Ablation: Partial Stacking
Partial stacking has been explored in LLMs like LlamaPro 42 Solar 433. But their goal is to stack an off-the-shelf LLMs such as Llama2, while our aim is to accelerate LLM pre-trame training process.
To explore stacking which layers of the small model can achieve the best performance, we con- duct experiments on partial stacking. In our experiments, we stack a small model with lay- ers ([L,, L2, La }) to 24 layers target model. We set growth timing d 10B tokens and growth factor g = For simplicity, we use format such as 1-234#7-56 to denote stacking 234 layers 7 times.
37

---

Tokens (Billions)
Tokens (Billions)
40
60
80
20
40
3.0
scrach
Gaxed323 436
Saadd32
3456 56)
2.9
G 22-21
Gaad -4-10-55)
56)
10)
2.8
Gesel12
Average Accuracy
Training Loss
2.7
2,25
2.6
2.20
2.5
2,15
2.4
2.3
Gaad125 446 +71
2.2
Geuil123-7
Gase112
Gruail 234 56
Gaso61234 56 110)
Gessltz -345
2.1
2
4
8
2
8
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 37: Training loss and standard NLP benchmarks average accuracy of scratch, Gstack and other partial stacking.
report the training loss and standard NLP benchmarks average accuracy in Figure 37 By observing the loss curves in Figure 377l divided into three groups based on their loss. The first group. 1124456*4, 12-35565--56, 12-345#7-6, 123-456*7), achieves the best performance. The second group consisting of (1234-56* 10, 12-34* 10- 56, 1-234*7-56), performs just so-so. The third group, (123*7 7-456), performs poorly, even worse than the baseline.
find that the eight partial stacking methods are clearly
In Table 5 we summarize the eight partial stacking and calculate the Rg of each partial stacking methods based on Equation 51
For partial stacking, we conclude that: all >> middle x back front. Meanwhile, when the stacked parts are the same, the larger the Re, the better the performance.
Table 5: R2 and stacked parts of each partial stacking method
Group
Method
Stacked parts
R₂
123456*4
all 87.0%
12-3456#5-56
middle-back
78.3%
First
12-345#7-6
middle-back
74.0%
123-456*7
back 74.0%
1234-56#10
back 60.7%
Second
12-34*10-56
middle 60.7%
1-234*7-56
front-middle
74.0%
Third
123#7-456
Tront 74.0%
Then, report the evaluation results here.
38

---

(bi)
23
ACCUSSC
Accumry
Accurso 3
24
FLOPs (1c
FLOPS
FLOPS (1cr20)
FLOPS (1)
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace f)
(d) Logiqa (Ace 1)
Tajons (B
Tokeno 09
Takons (Bi)
Mons)
20
darry
Word Spppai
13
ELOPS (1e
FLOPS (1e+20)
TLOPs (1e+20)
FLOPs 2le+
(e) PIQA (Ace f)
( Sciq (Ace f)
(g)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 38: Evaluation results on scratch, Gunck and other partial stacking.
H.4
Compare with Pythia, OLMo and Amber on 7B Size
Table 6: Compare with opensource 7B LLMs on 130B tokens.
Pythia-6.9B Pile-300B [2
OLMo-7B 53
Amber-7B
54
Gstack 7B
Datasets
Dolma [55
Amber
Slimpajama-6221
Tokens
130B
133B
132B
130B
ARC-
33.28
28.58
29.01
35.24
ARC-
59.81
51.60
55.05
63.64
boolg
63.39
55.05
60.18
66.45
hellaswag
60.03
54.52
61.21
65.85
lambada
65.11
49.91
57.13
57.93
logiqa
28,88
28.42
26.73
26.88
obqa
37.20
33.60
37.40
36.40
piqa
75.03
74.43
76.01
76.82
sciq
82.7
74.4
82.0
85.9
winogrande
60.14
53.75
56.83
62.75
Avg.
56.56
50.43
54.16
57.79
Wikitext
13.3340
18.4690
15.6202
12.5635
I Details of Function Preserving
I.1 Function Preserving
Function preservation is a key concept that underlies diverse model growth approaches. It entails ensuring consistent output from model, regardless of its expansion. Mathematically, let us define function as and growth operator as G. The ultimate aim is to apply the operator G to the function F. thereby obtaining the target function denoted F.. The core objective here is to maintain the model's 's function to generate the same output for given input. Formally,
F(x) F(x), where F = G(F)
(52)
39

---

1.2 Breaking Function Preserving by Adding Noise
For the down projection in SwiGL.U and the output projection MultiHeadAAtention, we apply noise:
W noise + (1 a)W + tcc c N(0, d x
(53)
For the Embedding Layer and other Linear Layers, we apply noise:
2
Wnoise (1-a)1 a
where N/(0
(54)
5d
Adding Noise Gdtrect Break FP
Tokens (Billions)
Tokens (Billions)
0
10
20
30
40
10
20
30
40
G (20%nois)
3.4
Gä (D Sinoise)
3.2
25
Average Accuracy
48
Training Loss
2.45
3.0
2.40
2.35
2.8
2.30
2.6
3.00
45
2.4
scratch
Ga (20%solss)
Gúne (0%noise)
2.2
0.5
1.0
1.5
2.0 2.5
3.0
3.5
FLOPs (1e+20)
FLOPS (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 39: Training loss and standard NLP benchmarks average accuracy of scratch, G Griset and direct with 20% noise.
40

---

2700
Accumyy 12
Acursy
03.
242
FLOPS (10120)
FLOPS 20)
FLOPS (1c120)
FLOPS (1)
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace f)
Tilons (8
Tokeno rattions)
Bkons 181 ons)
Dians Mons)
30
20
Accurry
24
20
Squ
FLOPs.Ce+3
PLOPs 200
FLOPs (1e+2))
FLOPs cle
(e) PIQA (Ace f)
( Sciq (Ace 1)
(g) Winogrande (Acc 1)
(h) Wikitext (ppl 4)
Figure 40: Evaluation results on scratch, G direct and diroct with 20% noise.
Training Loss And Evaluation Results on Adding Noise Gdirect
Adding Noise Gstack noise on Grack-
Since adding noise actually improve the Garea performance, also add
We stack an 8 layers small model to 24 layers, and then add noise with 0.2. We report training loss and standard NLP benchmarks average accuracy in Figure 41 Adding noise demonstrates an advantage in Training loss.
Tokens (Billions)
Tokens (Billions)
20
40
60
80
20
40
60
80
3.0
(P%onoo)))
Training Loss
2.8
2.30
60
70
80
Average Accuracy
2.25
48
2.6
2.20
46
4.5
5.0
5.5
2.4
44
2.2
scretch Gaua(U%noise)
Geas(208nnaks))
1
4
6
3
6
FLOPs (1e+20)
FLOPs (1e+20)
(a) Training Loss
(b) Average Accuracy
Figure 41: Training loss and standard NLP benchmarks average accuracy of scratch, Gstack and Stack with 20% noise.
Details of the evaluation results are as follows:
41

---

(bil
80
2.
Accumcy
Accurse
-2
24
24
G
Readl Sueeedd
FLORS (10120)
FLOPS 20)
FLOPS
FLOPS (L0+20)
(a) ARC-c (Acc f)
(b) ARC-c (Ace f)
(c) Lambada (Ace 1)
(d) Logiqa (Ace 1)
(Blloos) 64
Tokeno (Billgns)
(Bil ono)
Tolons Honc)
20
6o
72
2
Word Paplodty
53
FLOPS (te
FLOPS (1e=20)
FLOPs (1e+20)
FLOPs cle
(e) PIQA (Ace f)
(f) Sciq (Ace 1)
Winogrande (Ace 1)
(h) Wikitext (ppl 4)
Figure 42: Evaluation results on scratch, G stuck and Gsaakk with 20% noise.
J Results on Samba
We utilize the codebase from Samba which implements hybrid State Space Model using the Slimpajama dataset for LM. In this experiment, we follow the guidelines outlined in the main paper guide our stacking process. With parameter size of 410M and training the growth timing to 8B and the growth factor to 3. We opted for 3 instead of 4 because Samba is an interleaving of Mamba and self-attention layers. Since the target model has 12 layers, we can only stack even layers, leading us to select a 4-layer base model IMamba-SA, SA-Mamba-SAA
100B tokens, we
Our experiments results loss curves 4 and downstream tasks indicate stacking also works beyond Transformer- based LLMs. Please note that in Table 7 we select stack with 47B rather than 50B to count the additional consumption required train the base model on 8B tokens.
Tokens (Billions)
0
20
40
60
80
100
scratch
Geark
3.2
60
100
Training Loss
3.0
2.5
63.7%
0.8%
61.5%
0.00
2.4
11.2%
1.5
2.6
2.4
0.0
0.5
1.5
2.0
2.5
3.0
FLOPs (1e+20)
Figure 43: The training loss for two Samba LLMs, trained from scratch and with Gyact At loss=2.48, 2.45, 2.42, Grtnk accelerates by 61.7%, 61.5% and 58.2% compared scratch.
"aunss/githhbbcommmicroootttsut
42

---

Table Evaluation Results on Samba LLMs
Method
Tokens
lambada
logiqa
piqa
scig
avg
scratch
50B
36.41
25.34
43.77
27.50
67.36
70.00
45.06
Gand
47B
38.44
26.19
44.95
26.88
67.95
72.80
46.20
K
Loss Spikes
Figure illustrates the loss spikes that occur right after stacking.
Tokens (Billions) 30
3.8
Scratch Gytc
Galnoo base model
3.6
Training Loss
2.8
2.6
2.4
2.2
1
2
FLOPs (1e+20)
Figure 44: Loss Spikes in Gstack (Non-FP) and G1
random (FP)
Societal Impacts
As a successful exploration for efficient LLM pre-training, our work has great potential to give positive societal impact towards sustainable Al. Nevertheless, common drawback for LLMs, there are also chances that our LLMs might be misused intentionally or uniintentionally.
43

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the Abstract, clearly elucidate contributions, and at the end of Section Cection[[Introduutt Introducctio, we further detail our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section 7 we discuss the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
44

---

Answer: [NA]
Justification: Our study is empirical exploration.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We report our detailed training settings in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
45

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We open-source dataset Slimpajama-627B for pre-training, report this in Appendix [B.3] [B.3] We have submitted our code on OpenReview and will open-source on GitHub in the final version.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public http://uddddeeeeeeeeeeeeeeeeeeee
ssiiisssionn www.oooossssoooooiooooooooocc olicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ cC/ ptc//pblic//uue guicc//guiessseeeee CCoe:/oodeSubei Submie wwsssnnooliise
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We report the detailed settings in Appendix B.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: LLMs pre-training consumes a significant amount of computational resources, making impractical conduct multiple experiments to obtain error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
46

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the needed Compute Resources in Appendix B.3 and required FLOPs of each experiments in each Figures.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pwwp../aa......./////a/aa
cttp:/ccccc/ppcicccccccccccaaaaa ines?
Answer: [Yes]
Justification:
We have read this code.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have section in the Appendix D to discuss societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
47

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Our study is an empirical exploration. The dataset we use is a open-source high-quality corpus, and the models we release are intended solely for further research and are not meant for direct industrial application.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer Appendix B.3
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
48

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, patelpappssss wwwthhocee com/ / /ataaaaasssssee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All codes and models are will be full released under the license CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
49

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
50


Reviews:
[{"review_id": "LluwVPxJmk", "reviewer": "Reviewer_8DCi", "summary": "The presented work systematically investigated the major obstacles of applying model growth methods to large language models and the corresponding solution. The empirical results reveals that depthwise stacking methods works the best to LLMs. The paper then studied how to practically use the depthwise stacking methods in detail.", "questions": "See Weaknesses.", "limitations": "Authors discussed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The experiments are comprehensive and convincing.\n- This paper provides detailed and clear empirical guidelines.\n- The experiment results shows that the proposed model growth practice does accelerate the training of LLM by a lot.", "weaknesses": "- This paper provides practical guidelines on the usage of model growth methods on LLM training, but doesn't provide any theoretical analysis or intuition on why certain methods work or not work.\n- Current study focuses on to making use of an existing model growth method under the scenario of LLM training, instead of improve it or modify it. It is possible that there exists a better model growth method for LLM that doesn't fit into the 4 catagories summerized by this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your overall positive feedback and recommendation on our work!\n\nRegarding the two weaknesses you mentioned, we acknowledge that the current draft lacks a thorough theoretical analysis of the depthwise stacking and a comparison to more sophisticated growth methods. This is because our primary goal was to highlight three obstacles in model growth methods for LLM pretraining.\n\nWe are also very intrigued in conducting a deeper analysis, especially regarding the reasons behind the success of the depthwise stack, and we are actively working on this.\n\nThank you again for your thorough review! We will continue to build upon this work and address the points you have highlighted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mHXqIccgQA", "reviewer": "Reviewer_FXhk", "summary": "This paper studies the model growth technique for large language models, which expand a smaller pretrained large language model into a bigger one. The authors consider four natural way of expanding the parameters of the smaller pretrained large language models, and find that duplicating layers is the most effective technique. Therefore, this work delve deep into the dupicating/stacking layers technique, including the following aspects:\n\n- Scaling model sizes\n- Scaling training tokens\n- estimating laws\n- determining the growth timing\n- determining the growth factor\n\nand a bunch of other ablation studies.", "questions": "See weakness.", "limitations": "The paper is mostly about empirical observation. However, the reason behind this phenomenon is still unclear and not much discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Except in the introduction, the “model growth” concept is not introduced early enough, the paper is in general well-written and easy to follow.\n- The conducted experiments are very comprehensive and well-designed.\n- The problem is well-motivated and useful.\n- The proposed solution is very natural, simple yet effective.\n- In general, this paper provides very insightful observations for future usages.", "weaknesses": "- [Minor] The experiments are conducted on Transformer architecture. Trying different model architecture such as SSM can be more interesting.\n- [Minor] The concept “model growth” is not introduced clearly until the related works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thnak you for your appreciation and positive feedback on our work! Here are our point-by-point responses:\n\n1. _[Minor] The experiments are conducted on Transformer architecture. Trying different model architecture such as SSM can be more interesting._\n\nYes, we incorporated an SSM-based LLM experiment during the rebuttal period. Please see our general response for more details.\n\n2. _[Minor] The concept “model growth” is not introduced clearly until the related works._\n\nYes, in order to provide a clear introduction to the three key obstacles, we had to significantly reduce the content about model growth in the introduction to stay within the page limits.\n\nThank you for the suggestion; we will expand the current paragraph from L30 to L39 to give a more detailed introduction to model growth, particularly highlighting the relationship between the existing literature and our design of the four atomic growth operators.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYluC2uw86", "reviewer": "Reviewer_uHvr", "summary": "The paper introduces a novel method for pre-training large language models (LLMs) efficiently using model growth techniques. The authors tackle three main obstacles: lack of comprehensive evaluation, untested scalability, and absence of empirical guidelines. They propose a depth-wise stacking operator, \"Gstack,\" showing its effectiveness in reducing training time and computational resources across various LLM sizes, verified through extensive experiments and evaluations on multiple NLP benchmarks.", "questions": "Will this stacking method influence the robustness or generalizability of the LLMs on downstream tasks?", "limitations": "The paper lacks theoretical insights that could explain why the Gstack operator works well in practice. Including a theoretical analysis or rationale could strengthen the paper and provide a deeper understanding of the method.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The concept of model growth isn't new and stacking the weight depthwisely is not novel, but the paper innovatively applies it to LLMs through a systematic method, introducing the \"Gstack\" operator for depth-wise expansion. The paper is grounded in rigorous experimentation, presenting reproducible results with publicly available code and detailed experimental setups, ensuring high quality and reliability.", "weaknesses": "Although extensive, the experiments mainly focus on model performance from a computational and speed perspective. The impact on final model accuracy or downstream task performance is less explored, which could be crucial for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your appreciation of our work! Here are our pointwise responses:\n\n1. _Although extensive, the experiments mainly focus on model performance from a computational and speed perspective. The impact on final model accuracy or downstream task performance is less explored, which could be crucial for practical applications._\n\nThank you for your question. We acknowledge that evaluating only the loss is insufficient for fully assessing the work. But due to page limit, we have moved most of the evaluation details to the appendix, such as Appendix C (O1), D (O2), G (O3) and H (How to stack).\n\nIn terms of the final model accuracy and downstream task performance, we have evaluated using both 0-shot and SFT across different benchmarks, as shown in the Appendix D.4.\n\nWe will consider adding additional application settings. Thank you for highlighting this!\n\n2. _Will this stacking method influence the robustness or generalizability of the LLMs on downstream tasks?_\n\nYes, we agree that investigating the generalizability of our LLM stacking approach is also crucial for its wide-scale adoption. Based on our current downstream experiment results, it seems the generalizability may not be significantly different compared to vanilla LLMs. However, we acknowledge that using out-of-domain datasets is necessary to properly benchmark the robustness and generalizability of the LLMs on downstream tasks. We plan to expand our evaluation in this direction in future work.\n\n3. _The paper lacks theoretical insights that could explain why the Gstack operator works well in practice. Including a theoretical analysis or rationale could strengthen the paper and provide a deeper understanding of the method._\n\nYes! We're also intrigued by this and plan to pursue follow-up work to explore the reasons behind this. Thank you for your suggestions!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FkhqDH2LVn", "reviewer": "Reviewer_ox3X", "summary": "The paper tackles the problem of model growth during training. The authors focus on the problem of efficient LLM pretraining and analyze a plethora of proposed growing techniques. The paper focuses on and established from the very beginning three clear and important objectives: 1. Comprehensive evaluation for decoder transformer language model training, 2. Generalization of small-scale results to bigger scales and 3. The establishment of clear guidelines for practitioners.", "questions": "Some additional comments:\n- Can you comment with more detail on why you think stacking multiple-times does/should not work?\n- How specific are the results to Transformers and language models?\n- Since your growing operator is not function-preserving, should there be spikes in the loss curves? If yes why are these not visible in the current figures?", "limitations": "See above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is excellently written, with clear goals and intentions from the very beginning. It successfully conveys the message that stacking layers once with a specific growth factor, is the best possible solution.\n\nThe evaluation framework is comprehensive and sufficient. The authors analyze 4 dominant growing (although ignoring many others) techniques and present convincing results for \"stacking\" as the most promising technique. They then present experiments scaling up pretraining tokens and FLOPs and show that results generalize out of the box. \n\nFinally, the paper discusses the other two important choices when growing a model, namely when to perform these operations and how much to grow the model. Inspired by scaling laws, the paper fits curves on the \"optimum\" iso-FLOPs points derived by growing at different points and by different amounts.\n\nOverall the paper is very comprehensive in its experiments both in the main text and the appendix.", "weaknesses": "The motivation behind the three objectives is clear, yet at times insufficient. \n\nO1. Reported results in the literature are indeed focusing mostly on BERT pretraining. Although the evaluation framework proposed here is complete and comprehensive, it is not clear why decoder-transformer language modeling is fundamentally different and model growing results will in this case be different compared to BERT pretraining (apart from post-LN in the architecture). The authors should compare their findings with established results on BERT pretraining in the literature and discuss how their best configurations compare to them [1]. Note that other papers ([2, 19 from your pdf]) are dealing with growing operations during language model pretraining.\n\nO2. Viability for scaling is important.  One of the issues of model growth not discussed sufficiently in this paper is the problem of \"diminishing returns\", meaning that one can save compute, but the ratio of compute saved becomes smaller as training progresses (this is hinted in Figure 6a), especially since you are performing a single growing step. \n\nSection 4.2 is very interesting and the experiments enlightening, but the analysis is superficial. [2] has a more detailed analysis to determine growing timings. In general, assuming that models are training following a power law loss, growing timings can be determined based on the gradient of the training losses. This was analyzed in detail in [3], where they describe in detail how to determine these timings. Your case is admittedly more complicated since you are growing in a non-functional-preserving manner, but fundamental take-away messages should be the same.\n\nSection 5, is interesting, but both of the results seem to disagree with multiple findings in the literature. Although additional insights are presented in the appendix, these results seem preliminary and should be taken with a grain of salt.\n\n[1] Wang, Peihao, et al. \"Learning to grow pretrained models for efficient transformer training.\" arXiv preprint arXiv:2303.00980 (2023).\n\n[2] Shen, Sheng, et al. \"Staged training for transformer language models.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Anagnostidis, Sotiris, et al. \"Navigating Scaling Laws: Compute Optimality in Adaptive Model Training.\" Forty-first International Conference on Machine Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and valuable comments! Due to the word limit, we have omitted some of your partial questions and here are our pointwise responses:\n\n1. _O1. Reported results in ..._\n\nThank you for your comments regarding the connections to existing work. The aim of this study is to systematically explore scaling model growth for LLM pretraining. We began by reimplementing various methods, such as bert2bert, stackedBert, stageTraining, and MSG, and attempted to scale them for LLM training. However, after several unsuccessful attempts, we recognized that pre-training billion-scale LLMs presents challenges that differ significantly from those settings.\n\nOne clear indication of this is that most studies focus on growing neural models in both directions, while our findings in Figure 3 suggest that widthwise growth is much more challenging than depthwise growth, which aligns with our difficulties in replicating their success in LLM settings.  Another crucial insight from our work is that growth timing should be relatively early (for example, starting with a 10B base model before training the larger model for another 300B), whereas existing studies typically grow at a relatively later stage. (e.g. stackedBert train 3L 50k steps, stack and train 6L 70k steps and then train 12L 280k.) Additionally, the significant differences in scaling present challenges when directly applying their configurations to LLM pretraining, as they typically train on a smaller scale using models like GPT-2 and BERT, with only a small fraction of data of ours.\n\nHence, given the configuration challenges and the high costs of LLM pretraining, we decided to summarize existing approaches and design four fundamental growth operators to test under our configurations. Please note that our operators are closely related to existing methods, and we provide a detailed discussion in Appendix A.2, addressing stageTraining, LiGO, MSG, and others. Thank you for highlighting this; we will consider including our preliminary results for existing approaches under their own configurations.\n\n2. _O2. Viability for scaling ..._\n\nWe agree that diminishing returns are a crucial factor for any efficient pretraining algorithm. Therefore, we conducted a thorough study to examine scaling in O2. In Figure 6a, we observe a 31% speedup for a 410M LLM after processing 700B tokens, which is approximately 90 times the training tokens suggested by Chinchilla for a 410M LLM. This suggests that diminishing returns are not a significant issue in stacking. Additionally, in our pretraining experiment with the 410M LLM, we did not select the optimal growth timing for the model size because we have not reached the optimal growth timing at that time. So there is a chance that finding the optimal growth timing for the 410M LLM could lead to even greater speedup performance.\n\n3. _ Section 4.2 is very ..._\n\nThank you for highlighting the importance of using gradients as an indicator for finding optimal growth timing! We agree that it’s a promising direction. In fact, we are currently working on follow-up research to understand the reasons behind the success of stacking. The gradient indicator will also be valuable. We appreciate your suggestion and will consider it for determining growth timings!\n\n4. _Section 5, is interesting,..._\n\nFor multiple-times stacking, we will elaborate on point 5. Regarding function preserving (FP), as mentioned in L401 of the paper, we believe it is an important factor, but perhaps not the sole key to the success of model growth methods. For example, LiGO is not fully function preserving, yet it has become one of the popular methods in model growth. We recognize the importance of FP; even our stacking method is not entirely function preserving, as noise exceeding 20% results in a notable performance drop.\n\nThe goal of function preserving is to maintain the knowledge learned by the base model to accelerate training. However, the parameter space explored by the base model may not be optimal for a larger model. Thus, focusing solely on function preserving might not align with the goal of speeding up training. This could also explain why we need to grow earlier; a well-trained base model might confine the larger model to a suboptimal state. Nonetheless, this is just our intuitive explanation, and we acknowledge that this could also relate to the gradient indicator you mentioned. We are currently working on a followup work to give more formal proof of the success of stacking. Thank you for bringing this up and allowing me to discuss it!\n\n\n5. _Can you comment with ... _\n\nWe agree that the ablation study on multi-growth is preliminary. However, we must acknowledge that multiple-time growth suggests a more complex training dynamic overall. Considering the high costs of LLM pretraining and the focus of our work on \"atomic\" growth operators, we have only reported our preliminary results on multiple stacking.\n\n6. _How specific are the results..._\n\nThe primary motivation of this work is to \"scale\" model growth techniques, which is why we chose Transformer-based LLMs. However, our recent experiments with SSM-based LLMs, as mentioned in the general rebuttal response, suggest that this method may also be effective across different Transformer architectures. We will consider investigating other models, such as ViT, for further research.\n\n\n7. _Since your growing  ..._\n\nThank you for your thorough review! Yes, we observed a spike in the loss curves for stacking, and we included the figure in the PDF of general response. Specifically, we compared the baseline, stack, and random methods. It’s evident that while the function preserving operator, random, initially achieves a lower loss right after growth, it is soon surpassed by the stack operator. \n\nIn the script, we excluded the base model curve for simplicy. We will include add the base curve figure to Appendix in the revised version. Thanks for your detailed review!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all the reviewers for their time and effort in reviewing our paper. We are thrilled to receive positive feedback from all four reviewers and are honored that the reviewers generally acknowledge our strengths, including: 1. the paper is well-motivated and easy to follow [FXhk,ox3X], 2. the experiments are well-designed, systematic, and rigorous  [uHvr],  3. the findings are comprehensive [8DC,FXhki], convincing [8DCi,ox3X], and clear [8DCi].

We thank the useful suggestions from the reviewers, which help a lot in further improvement of this paper. In addition to the pointwise responses below, main revisions are summarized as follows:

1.__Ablation on state space modeling (Mamba)__
In response to the requests from reviewers FXhk and ox3X for additional ablations using other methods rather than transformer architecture, we have carried out the following ablation study, as detailed in Figure 1 and Table 1 of the attached PDF.

We utilize the codebase from [GitHub - microsoft/Samba](https://github.com/microsoft/Samba), which implements a hybrid State Space Model using the Slimpajama dataset for LM. In this experiment, we follow the guidelines outlined in the paper to guide our stacking process. With a parameter size of 410M and training on 100B tokens, we set the growth timing to 8B and the growth factor to 3. We opted for 3 instead of 4 because Samba is an interleaving of Mamba and self-attention layers. Since the target model has 12 layers, we can only stack even layers, leading us to select a 4-layer base model (Mamba-SA-Mamba-SA).

Our experiments results on loss curves (Figure 1) and downstream tasks (Table 1) indicate stacking also works beyond Transformer-based LLMs. Please note that in Table 1, we select stack with 47B rather than 50B to count the additional consumption required to train the base model on 8B tokens.

2.__Loss curves with base model__ Figure 2 in the PDF addresses reviewer ox3X's request to illustrate the loss spikes that occur right after stacking.

Please contact us if we can do something else to help you better understand and recommend our paper.

Meta-review: This paper systematically investigates the existing model growth techniques for efficient LLM pretraining. The authors classify the existing approaches into four atomic growth operators; and then by conducting comprehensive experiments, they find that the depth-wise stacking operator is the most effective one and can indeed accelerate the training of LLM significantly. Besides, the authors also provide clear empirical guidelines for the use of the depth-wise stacking operator, which is valuable for LLM practitioners aiming to accelerate LLM pretraining. 

All the reviewers agree that the experiments are comprehensive and convincing, and the empirical findings are informative and strong. All the reviewers appreciate this paper, though it lacks a theoretical exploration. I recommend acceptance.
Decision: Accept (spotlight)