Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

Authors: wenyu  du
tongxu  luo
zihan  qiu
zeyu  huang
yikang  shen
reynold  cheng
yike  guo
jie  fu
hong  kong
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 12:28:36

Abstract: LLMs are computationally expensive to pre-train due to their large scale.
Model growth emerges as a promising approach by leveraging smaller models to accelerate the training of larger ones. 
However, the viability of these model growth methods in efficient LLM pre-training remains underexplored.
This work identifies three critical $\underline{\textit{O}}$bstacles: ($\textit{O}$1) lack of comprehensive evaluation, ($\textit{O}$2) untested viability for scaling, and ($\textit{O}$3) lack of empirical guidelines.
To tackle $\textit{O}$1, we summarize existing approaches into four atomic growth operators and systematically evaluate them in a standardized LLM pre-training setting.
Our findings reveal that a depthwise stacking operator, called $G_{\text{stack}}$, exhibits remarkable acceleration in training, leading to decreased loss and improved overall performance on eight standard NLP benchmarks compared to strong baselines. 
Motivated by these promising results, we conduct extensive experiments to delve deeper into $G_{\text{stack}}$ to address $\textit{O}$2 and $\textit{O}$3.
For $\textit{O}$2 (untested scalability), our study shows that $G_{\text{stack}}$ is scalable and consistently performs well, with experiments up to 7B LLMs after growth and pre-training LLMs with 750B tokens.
For example, compared to a conventionally trained 7B model using 300B tokens, our $G_{\text{stack}}$ model converges to the same loss with 194B tokens, resulting in a 54.6\% speedup. 
We further address $\textit{O}$3 (lack of empirical guidelines) by formalizing guidelines to determine growth timing and growth factor for $G_{\text{stack}}$, making it practical in general LLM pre-training.
We also provide in-depth discussions and comprehensive ablation studies of $G_{\text{stack}}$. 
Our code and pre-trained model are available at https://llm-stacking.github.io/.

Main paper: 
 introduction: 
 Emergent abilities of Large Language Models (LLMs) rely on scaling-up [1, 2] . Empirical evidence from scaling laws [3] [4] [5] fuels the development of increasingly larger models, pushing the boundaries of LLMs capabilities. However, pre-training these gigantic models comes at a significant cost in terms of energy consumption and environmental impact [6] (e.g., pre-training Llama-3 [7] consumes a total of 7.7M GPU hours and generates 2290 tons of carbon dioxide equivalent of carbon emissions). The efficient pre-training of LLMs is thus crucial, both from a scientific and a societal perspective, to ensure the continual growth and adoption of AI [8, 9] .One promising research direction to accelerate model training involves leveraging trained smaller (base) models to expedite the training of larger (target) models, a technique known as model growth.Concretely, model growth studies how to leverage the trained smaller model's parameters Θ (s) to initialize the larger model's parameters Θ (l) . Current popular methods generally focus on expanding the parameters of the base model through techniques like splitting [10] [11] [12] , copying [13, 14] , or matrix mapping [15] . There are also some approaches that initialize new parameters from scratch [16, 12, 17] . The primary objective is to accelerate the training of large models, and existing methods demonstrate promising speedup results on models such as BERT [11, 14, 18, 15, 12, 13] . Despite such empirical evidence and its alignment with the goal of efficient LLM pre-training, model growth methods are not widely adopted in the context of LLM pre-training [7, 19] . To our best knowledge, the only LLM that utilizes model growth for accelerating is FLM-101B [20] , but it lacks a baseline LLM trained from scratch to compare. We observe three key Obstacles that hinder LLM pre-training from using existing model growth techniques, specifically:• O1: Lack of comprehensive assessment. Some existing model growth methods report results on LLM pre-training, but either lack a baseline comparison [20] or are still in exploratory stages [15, 13] . In contrast, most growth approaches are evaluated in encoder-based BERT models [14, 11, 18, 12, 13, 16, 17] , which have different architecture and training configurations compared to prominent decoder-based LLMs such as Llama [21] .• O2: The untested scalability. This scalability has two aspects: the model size and the amount of pretraining data. Regarding the model size, the existing approaches are only evaluated on smaller-scale BERT models or in preliminary experiments with LLMs. It is unclear whether these growth methods will continue accelerating training when applied to large-scale LLMs with more extensive evaluation. As for the amount of pre-training data, there are debates [22] over whether certain efficient training strategies may initially converge faster but ultimately perform similarly or worse than vanilla training methods when given ample computational resources (i.e., more training data).• O3: Lack of empirical guidelines. Scaling laws [3, 4] give clear empirical guidelines on pre-training computational-optimized LLMs, greatly stimulating and advancing the field. Yet, there is a lack of empirical guidelines on growth techniques, discouraging LLM practitioners from adopting these approaches, especially considering the high costs of LLM pre-training.These three obstacles are consequential in nature. Hence, in this work, we empirically revisit the concept of model growth as a solution to efficient LLM pre-training by tackling them one by one. The training loss for two 7B LLMs, trained from scratch and with G ↑ direct (G stack ). At 300B tokens, G stack accelerates by 54.6% compared to scratch.To tackle O1, we systematically evaluate model growth techniques on practical LLM pre-training. We first categorize existing growth methods and summarize them into four atomic growth operators, each of which can grow along two directions: widthwise (intra-layer) and depthwise (layer-wise). We illustrate them in Figure 2 . These operators serve as representative choices for evaluating the performance of model growth techniques. We use these operators to expand 400M base models to 1.1B Llama-like LLMs and continually pre-train them. Next, we evaluate these growth techniques on the training loss and eight standard NLP benchmarks from the Harness toolkit [23] . We found the direct operator that stacks depthwisely G stack consistently outperforms others across overall evaluation metrics, demonstrating its potential in accelerating LLM pre-training. This motivates us to investigate extensively by addressing O2 and O3 on G stack .To address O2, we investigate the G stack operator's scalability to larger model sizes and to more training data. We conduct extensive experiments by scaling model size up to 7B parameters trained with 300B tokens, and pre-training a 410M model with over 750B training tokens. This is in contrast to the previous largest LLM pre-training experiment that uses model growth methods and has baselines for comparison, which is reported in Ligo [15] , where a GPT2-1.5B model is trained for 15k steps (approximately 15B tokens). The results are encouraging, as we consistently observe significant improvements G stack offers in both scenarios. For example, we achieve a remarkable 54.6% speedup in pre-training for a 7B model with 300B tokens (Figure 1 ). Interestingly, the loss improvement in our 750B-token experiment aligns with a logarithmic function. We further extend this logarithmic curve and determine that the improvement continues to be substantial even for the LLM trained with over 8T tokens. Moreover, we summarize all our experiments by estimating the LLM scaling law for LLMs pre-trained with G stack . Given the same target loss value, our analysis reveals a significantly reduced computational cost compared to the common scaling law [4] .For O3, we explore the practical guidelines for using G stack in LLM pre-training. Given a computational budget, we determine the optimal strategy for two key factors of G stack , growth timing d and growth factor g. Growth timing d relates to the training tokens used for small models before growing, and growth factor g refers to the factor between the non-embedding parameter number of the large models and the small models. We formalize our findings into equations that offer concrete suggestions for utilizing G stack . We believe this work could significantly pique the interest and bolster confidence in future LLM pre-training with model growth techniques, both in academia and industry.To summarize, our contributions are four-fold: 1) We first systematically investigate model growth techniques and identify four atomic model growth operators, establishing a better understanding of the field in Section 3.1. 2) We then design a standard LLM pre-training testbed and perform comprehensive evaluations on these operators, finding that a simple depthwise stacking G stack exhibits significant superiority in Section 3. 3) We further demonstrate the scalability of G stack with experiments on LLMs ranging from 410M to 7B parameters and up to 750B training tokens in Section 4. 1. 4) We also provide guidelines of equations on determining growth timing and growth factors for optimal use of G stack in Section 4.2.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: In the Abstract, we clearly elucidate our contributions, and at the end of Section 1 Introduction, we further detail our contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work -model growth for efficient pre-training: 
 The idea of growing neural networks dates back to the 1990s [24] [25] [26] . The pioneering work of Net2Net [10] marks a milestone, for the first attempt to study model growth in deep learning era. Net2Net expands width and depth while keeping original functions (namely function preserving) via randomly splitting old neurons and injecting new identity layers. The widthwise splitting method of Net2Net represents a series of works that aim to "expand" the existing neurons to the desired larger size. Bert2Bert [11] serves as a BERT-based extension of the widthwise Net2Net. StagedGrow [13] doubles the width by concatenating two identical layers and halves final loss to keep functionpreserving. Lemon [12] suggests integrating a parameter into the splitting of neurons in Bert2Bert, aiming to break weight symmetry. Depthwisely, StackedBert [14] simply stacks duplicated layers to form a deeper model. In contrast to the above direct copy/split approaches, LiGO [15] presents a learning-based method that initializes the larger model's parameters via learning a linear mapping from the smaller model's parameters.Alongside the approaches that expand existing parameters, there are works that initialize new ones without relying on existing ones. For instance, MSG [17] proposes a multi-staged growing strategy that progressively expands transformer components, where the newly grown neurons are randomly initialized using a masking mechanism to ensure function preservation. Besides, some works have assigned specific values, like zero, to the newly initialized neurons to negate their influence [16, 12] .All the above methods are primarily explored in BERT or earlier stages of LLM pre-training. On the other hand, our objective is to present the first systematic review of model growth techniques in the LLMs era. To our knowledge, FLM-101B [20] is the only existing LLM that uses the growth method [17] for accelerating billion-scale LLM pre-training. Nonetheless, this work lacks a baseline model trained from scratch, making it difficult to assess the effectiveness of the model growth technique. In contrast, we aim to provide a comprehensive study by establishing a standardized testbed to compare LLMs trained from scratch and with various growth methods in LLM pre-training.We calculate speedup sp between operator G and scratch model pre-training by:EQUATIONwhere F LOP s scratch and F LOP s G represent the FLOPs required by the scratch model and the G model, respectively, to achieve the same loss.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In Section 7, we discuss the limitations of our work.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
systematically assessing model growth for llm pre-training: 
 Existing model growth methods [14, 11, 18, 15, 12, 13, 16, 17] are mainly evaluated on BERT [27] , with limited focus on decoder-only large-scale language models such as Llama [21] . Moreover, these growth methods are often not comparable due to different training settings [14, 11, 17, 12] .Even some growth LLMs experiments are evaluated, their results are often incomplete [20, 15] . To overcome these limitations, we first summarize existing works [14, 11, 18, 15, 12, 13, 16, 17] into four atomic growth operators to represent these growth techniques. Then we build a standardized LLMs training testbed to pre-train LLMs with four growth operators on depthwise and widthwise directions and evaluate the results with both training loss and eight evaluation metrics in Harness [23] .Recent years, researchers have focused on enhancing the efficiency of training large models by making use of smaller pre-existing models [10, 11, 14, 18, 15, 12, 13, 16, 17] . These state-of-the-art methods can be categorized into two distinct groups. The first group focuses on deriving new neurons from the existing ones [10, 11, 14, 12, 15] , while the second group focuses on initializing new parameters separately [18, 13, 16, 17] . Drawing from these two lines of research, we summarize four atomic growth operators. These operators include: (A) directly duplicating and stacking old layers in a depthwise manner or splitting neurons in the same layer widthwisely, denoted as G direct , (B) generating expanded parameters using a learnable mapping matrix to the existing parameters, denoted as G learn , (C) setting the new parameters to zero, denoted as G zero , and (D) randomly initializing the new parameters, denoted as G random . The illustration of four operators is shown in Figure 2 . The G direct and G learn growth operators produce new neurons from the current ones, in contrast to G zero and G random which initialize new parameters independently. For the formal definitions of the operators and the differences to the existing growth methods in design, please refer to Appendix A. Complex growth methods, such as those involving auxiliary loss or exploring training dynamics like learning rates [28, 29, 16] To make a fair comparison of the four growth operators for LLM pre-training, we define a standardized "one-hop" growth process that involves two training phases, small model training before growth and large model training after growth. We first train the small LLMs with d tokens before growing. Then, we use operator G to grow them to the target LLMs by a factor of g for non-embedding parameters and then continual pre-training the large LLMs for D tokens. Two key factors in the procedure are worth noting: the growth factor g and the data for base model training d, which can be interpreted as "growth timing". We further evaluate each growth operator by separately examining in depthwise (intra-layer) growth G ↑ and widthwise (layer-wise) growth G → . Concretely, we start with base models (400M LLMs) trained on d = 10B tokens, apply the four operators in both directions to scale them up to the target size of 1.1B (approximately a growth factor of g = 4), and then continue training for an additional D = 97.5B tokens. 4 Appendix B contains the LLM's architecture configuration and training details.We report results on training loss, eight standard Harness NLP benchmarks along with the average accuracy and the speedup ratio in Figure 3 . Our key discovery reveals that depthwise growth G ↑ exhibits a significant acceleration over both widthwise growth G → and training models from scratch, while surprisingly, G → does not offer any notable advantages. Among the depthwise growth operators, G ↑ direct , G ↑ learn , and G ↑ zero , all outperform the baseline and G ↑ random . The underperformance of G ↑ random in our study may be attributed to its design for gradual "mini-step" growth [17] , whereas our unified approach uses a single step. [30] , ARC-c [31] , ARC-e [31] , Logiqa [32] , PIQA [33] , Sciq [34] , Winogrande [35] and Wikitext PPL [36] totaling eight standard NLP benchmarks. After 8 × 10 20 FLOPs of training, G ↑ direct demonstrates a significant speedup.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
delving deeper into depthwise stacking (g stack ): 
 The empirical evidence suggests that certain growth operators, most notably G ↑ direct , exhibit an impressive acceleration in LLM pre-training compared to the baseline approach of training models from scratch. We now turn our attention to a more in-depth examination of the G ↑ direct . For ease of reference, we will henceforth denote this depthwise stacking approach as operator G stack :M = M • M • • • • • M g×M, where M is a small base model trained with d tokens, M is the target model and g is the growth factor. This section addresses the two main challenges (O2 and O3) outlined in the introduction: 1) evaluating the performance of G stack in scaling scenarios, i.e. larger model sizes and more training tokens; and 2) determining the hyperparameters when using G stack , i.e., the growth timing d and growth factor g. FLOPs (1e+20) ). Then, we train the stacked models using over 300B tokens (D = 300B) for both sizes. Figures 4 and 5 show the loss, and the NLP benchmarks average accuracy evaluated using the Harness evaluator for training 3B and 7B LLMs with 300B tokens, respectively. 5 The acceleration of G stack is consistent across two models and all evaluation metrics. For instance, considering the 3B model, Figure 4 demonstrates that G stack achieves a 54.5% speedup in pre-training, improvement of 2.1 in NLP benchmarks average accuracy compared to the baseline 3B model trained with 240B tokens.When comparing the 1B, 3B, and 7B models, it is evident that the benefits of G stack are not reduced as the model size increases, implying that its acceleration effect can be leveraged even with larger models. Details of the evaluation results, including evaluation with instruction tuning, can be found in Appendix D. Appendix E compares our baselines with the open-source LLMs Pythia and tinyLlama. 6b . The fitting curve indicates G stack will continue to exhibit acceleration effects even after 8T tokens, which is over 1000 times longer than the recommended token number [4] . It is also notable that this loss improvement after 8T training is not trivial for LLM pre-training, as previous studies [39] suggest that even minor improvements in the later phase can have a relatively substantial impact on downstream performance.From a LLM practitioner's perspective, this is also crucial considering "overtraining", which involves training LLMs with significantly larger amounts of data than recommended by scaling laws [3] [4] [5] , a common practice that has become prevalent. A notable example is the training of LLama 3-8B with 15T tokens, which is nearly 100 times greater than the token count recommended by the chinchilla scaling laws [4] . Hence, this finding provides confidence in the consistent excellent acceleration of G stack throughout the entire practical LLM pre-training process.0.1 1 10 100 1000 FLOPs (1e+20) Estimating Scaling Laws. To further explore our findings, we graph our four models (410M, 1.1B, 3B, and 7B) on the same figure and attempt to uncover our "scaling law" using the G stack operator. Following [3, 4] , we define the scaling power law using the equation L C = aC b , where a and b are constants we need to fit, C represents the FLOPs, and L C denotes the model's final loss under this FLOP. We use the curve_filt function in SciPy [40] to fit both the scratch model and the G stack model and present the estimation scaling law in Figure 7 . The figure shows that our G stack scaling law exhibits improved efficiency compared to the scaling law estimated from baseline LLMs, achieving the same target loss while requiring much less computational resources. However, in light of the significant computational resources devoted to other scaling law studies [3, 4] , we acknowledge that our G stack scaling law is an initial estimate subject to computation constraints, and a comprehensive study is left for future research.We comprehensively validate the effectiveness of the G stack compared to training from scratch in Section 4.1. However, to incorporate G stack into a LLM's pre-training process, we need to determine two crucial hyperparameters: the growing time (d) and the growing factor (g). In our previous experiments, we rely on ad-hoc choices for these parameters, thereby lacking a systematic approach to determining them when use G stack . There exists research on investigating the growth timing [41] , but the settings are quite different from the LLM pre-training. Therefore, this section offers a clear guide for practitioners looking to optimize using the G stack operator in LLM pre-training processes.We begin by offering a formal definition. When given a computational budget C, established scaling power laws [3, 4] exist to guide the non-embedding parameters N and the number of training tokens D to achieve the lowest model loss in the case of training from scratch. However, tuning hyperparameters becomes more complex when the fixed budget C is allocated to find the optimal model training strategy using the G stack operator, which involves two training phases. Consequently, the overall computational budget C can be expressed as the sum of the two components:C = C1 + C2.Here Determining Growth Timing: d. We first explore the effect of growth timing, i.e. the training token d for the small model. Particularly, we apply the G stack operator to a series of small models trained with d = 0B, 1B, 5B, 10B, 20B, 50B tokens. Subsequently, we stack them to the target layers with growth factor g = 4 and train for a fixed set of computational FLOPs. We replicate the above experiments using three target model sizes N = 410M, 1.1B, 3B and plot each set of IsoFLOP points in Figure 8a , 8b and 8c. Surprisingly, even a small model trained with just 1B tokens exhibits a significant speedup compared to the directly stacking small random initialized models (represented as "0B"). While 0B's performance is similar to models trained from scratch, implying stacking itself does not serve as an effective initialization method. Furthermore, by applying smoothing techniques to model IsoFLOP curves as parabolas, we identify the optimized value of d that minimizes loss for each FLOP count, leading us to hypothesize the existence of a logarithmic equation involving N , C, and d: Training Loss Determining Growth Factor: g. Another factor we determine is the growth factor g.As models with 3B and 7B parameters have identical depths, we run experiments using two model sizes: 1.1B (24 layers) and 3B (32 layers). Specifically, we vary the stack factors to g = 2, 4, 8, 24 for the 1.1B model and g = 4, 8, 16, 32 for the 3B model while keeping the base models trained with d = 10B tokens. The smoothed IsoFLOP curves are plotted in Figure 10 . Interestingly, even with a relatively shallow 2-layer base model and a growth factor of g = 16, we observe a remarkable improvement compared to the baseline 3B model (g = 1). However, when using a 1-layer base model, G stack underperforms compared to the baseline. Our curves indicate that the optimal growth factor g lies between 2 and 4.However, unlike determining training token d, we cannot generate sufficient data to estimate the relationship between N , C, and g, due to computational constraints. Thus, this work suggests a constant growth factor of g = 4. We also include our preliminary estimated equation and contour figure for g in the Appendix F. All evaluation results of Section 4.2 are listed in Appendix G. 
ablation and discussion: 
 To further give insights into adopting model growth techniques in LLM pre-training, we ablate variances for G stack and discuss function preserving in general model growth techniques.It is worth noting that G stack differs from the algorithm proposed in StackedBERT [14] , which utilizes a gradually stacking strategy. Hence, we compare our "one-hop" G stack and their gradual stacking approach. Following the methodology introduced in StackBERT, we employ a two-step stack strategy. Given our target model size of 1.1B with 24 layers, we start with a 6-layer model. Subsequently, we train it on 10B tokens and double the model's depth through stacking, repeating this step twice (train-stack-train-stack) to achieve the desired scale. Our experiments demonstrate that G stack outperforms gradual stacking approaches on loss and downstream evaluations. For example, the evaluation results show that G stack achieves a 2.4 higher average accuracy and 0.6 better Wikitext PPL than gradual stacking when pre-training large models for 100B tokens. The results can be found in Appendix H.1. We further compare other stacking variations, such as stacking via interpolation and partial stacking of certain layers which are also adopted in LlamaPro [42] and Solar [43] , and leave our detailed findings in the Appendix H.2 and H.3.Function preservation (FP) is a key concept that underlies most model growth approaches [10] [11] [12] 17] .The idea is intuitive that a larger model should initialize parameters that can represent the same function as the ones in the smaller model, i.e. ∀x, f (x; Θ (s) ) = f (x; Θ (l) init ), where x is the input. We give a mathematical definition of FP in the Appendix I.1.We find it intriguing that our G stack approach, which violates FP, emerges as the most effective in our study. To further investigate, we conduct a simple ablation study to break FP by introducing noise on the strict-FP operator G → direct . We initialize the new neurons by a weighted combination of two sets of parameters: those from G → direct and those from random initialization. The weighting factor is controlled by a noise ratio. Our findings are intriguing. After 40B tokens training, adding 20% noise outperforms original G → direct by 0.27 on the Wikitext PPL and 0.41 on the average accuracy score. We also add noise for G stack . When we add 20% noise, our LLM performs slightly better than the no-noise model. However, when the noise level exceeds 20%, the performance significantly deteriorates. These results indicate that function preservation may not be the sole determining factor for model growth. In other words, exploring ways to accelerate the training of larger models and strict preserving function during growth might represent two overlapping yet distinct research directions. The experimental details are provided in the Appendix I.2. 
conclusion: 
 This work empirically explores model growth approaches for efficient LLM pre-training. We address three key challenges of current model growth research for efficient LLM pre-training. We first comprehensively evaluate model growth techniques into four atomic operators and explore depthwise growth G stack beats all other methods and baselines in various evaluations. We next address concerns about the scalability of G stack by extending the model and training data scales. Furthermore, we systematically analyze the usage of the G stack operator, focusing on growth timing and growth factor. Based on this analysis, we formalize a set of guidelines for effectively utilizing the G stack operator. In addition, we provide in-depth discussions and comprehensive ablation studies of G stack , shedding light on the broader implications of our work. 
limitations: 
 While our work has demonstrated remarkable potential, four limitations deserve further attention. One limitation is the constraint of computation resources. For example, we only compare two sets of growth factor d configurations, which limits the capacity to derive a formula for determining the optimal growth factor d. Another limitation of our work is the focus on relatively simple operator choices, where we prioritize simplicity over exploring more sophisticated strategies. For instance, we do not extensively investigate the multi-step growth or dynamic modifications to the training process, such as adjusting the learning rate during continual pre-training. The third limitation involves the incomplete cosine learning rate schedule during training. This also arises from the resource-intensive nature of pre-training LLMs and the constraints on available computational resources. Therefore, we adopt a strategy where we initially set a large number of training tokens and then we pre-train LLMs until the training runs are interrupted by tasks with higher priority. Lastly, although this study's scope is an empirical exploration and the content is self-contained, there is a lack of theoretical insights into the success of G stack in LLM pre-training. 6 Nonetheless, we will release all LLM checkpoints to facilitate the community's investigation into the theoretical principles behind our observations. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: Our study is an empirical exploration. The dataset we use is a open-source high-quality corpus, and the models we release are intended solely for further research and are not meant for direct industrial application.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Please refer to Appendix B.3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Four Growth Operators A.1.1 Operator G direct : Direct Derivation of Grown Parameters From Old Parameters One intuitive strategy for expanding neural networks involves directly duplicating or splitting existing neurons. [14, 11, 12] . Unlike other growth operators, we distinguish between growth in terms of depth and width.For width-wise expansion, the Net2Net technique and its transformer implementations [10, 11] involve splitting old neurons into two or more parts, with each splitting step achieving a=b+c. Depending on the specific splitting mechanism, there are two variations: even splitting and uneven splitting. The latter is proposed to address symmetry issues that arise when neurons are evenly split. In this paper, we adopt the approach of uneven splitting.In the context of depth-wise expansion, a common practice is to duplicate layers, often referred to as "stacking" [14] . Therefore, we use the term G stack to represent this operator. While this approach may appear to deviate from function preservation, it surprisingly yields a strong baseline.G learn is an operator that learns a matrix transformation function to map small models to a larger one [15] . This operator is applicable to both width and depth expansion. Considering the original model f with parameters θ, the target model F with parameters Θ, and G learn as the hypernetwork for meta-learning, the training corpus is denoted as D, and the language model loss is denoted as L. Then, we optimize the following objective:EQUATIONA.1.3 Operator G zero : Setting New Parameters to 0Setting new parameters to zero is often considered a simple method to achieve function preservation. However, optimizing networks with a significant number of zeros can present challenges. To tackle this issue, we adopt current practices that selectively zero out either the fan-in or fan-out parameters [13, 16, 12] . Specifically, for operator G zero , during width growing, we zero out only the set of fan-out parameters for new neurons and randomly initialize the remaining ones. In the case of depthwise expansion, we zero out the final output layer of the newly-duplicated transformer blocks' MultiHead Attention and MLP.This group follows the common practice of randomly initializing new parameters. In earlier attempts, old neurons were frozen after the growth process [18, 17] . However, to ensure function preservation, a recent study introduces a mask for new neurons after expansion [17] . This mask is gradually removed during ongoing training. We refer to this new approach as the growth operator G random .The operators G → direct shares a similar setting to Lemon with minor variances due to Llama achitectures. G learn is consistent with the methods LiGO, but with our own implementation. For G zero , our approach aligns with Lemon in terms of depth, but differs from stagedTraining in width. Unlike stagedTraining, we do not double the width and assign zeros to the off-diagonal entries. Instead, our approach is more flexible; by zeroing out the submatrix in the bottom-left corner, we can extend it to any dimension. Our G random does not exhibit the "multi-hop" growth like MSG, instead, it grows "one-hop" directly to the target size. Our implementation of G ↑ direct (G stack ) differs from the algorithm employed in stackedBert. In stackedBert, a gradual growing technique is utilized, whereas our operator follows a more direct approach.Embedding Consider E ∈ R V ×d , and our goal is to expand it to E ′ ∈ R V ×D , G direct just copy some columns:EQUATIONwhere R ∈ R d×D is used to copy the embedding matrix E.EQUATIONwhere R ∈ R din×Din is used for expanding the fan-in and L ∈ R Dout×dout is used for expanding the fan-out. To satisfy function preserving, we ensure that α + β = I.RMSNorm For RMSNorm, a similar approach is adopted, consider parameterEQUATIONTherefore, using the G direct , it is not possible to achieve function preservation for RMSNorm Depth (G stack ) Consider a transformer with l layers represented asF = f 0 • f 1 • • • • • f l .Our objective is to expand it to L layers, where L is a multiple of l. We have various stacking forms for this purpose, such as (a) direct stacking:F ′ = F • F • • • • • F . Algorithm 1 Operator G stackInput: Base model M l k with l layers trained using dataset d k where k is iteration steps. Growth factor g. Output: Target Model M gl 0 with gl layersM l 0 =M l k for t = 2 to g do ▷ Model Stacking M tl 0 = M (t-1)l 0 • M l k end A.4 Details of G zeroEmbedding Consider an embedding matrix E ∈ R V ×d . The G zero operator expands it to E ′ ∈ R V ×D with O, where d ≤ D. Formally:EQUATIONTherefore, give a token x, the expanded embedding can be expressed as:EQUATIONLinear Consider parameter W ∈ R dout×din . G zero expand it to W ′ ∈ R Dout×Din , where d out ≤ D out and d in ≤ D in . Formally:EQUATIONwhere A, C are randomly initialized new parameters. Considering the input token x ∈ R din before expansion, and the input after expansion x ′ ∈ R Din :EQUATIONEQUATIONRMSNorm Considering the parameterµ ∈ R d , G zero expand it to µ ′ = [αµ, ξ] like G random in Appendix A.5, because the input must be x ′ = [x, 0 D-d ] ∈ R D .Depth In depth, by retaining only the residual part and initializing the MHA and SwiGLU final linear projections to zero, the MHA and SwiGLU layers can achieve function preservation.Embedding Consider an embedding matrixE ∈ R V ×d . The goal of G random is to expand it to E ′ ∈ R V ×D, where d ≤ D. Formally:EQUATIONwhere E ∈ R V ×(D-d) represents randomly initialized new parameters. We use a mask c ∈ R D to mask out the randomly initialized parts:EQUATIONTherefore, for a token x, the masked embedding can be expressed as:EQUATIONLinear Consider parameter W ∈ R dout×din . Our goal is to expand it to W ′ ∈ R Dout×Din , where d out ≤ D out and d in ≤ D in . Formally:EQUATIONwhere A, B, C are randomly initialized new parameters. Considering the input token x ∈ R din before expansion, and the input after expansion x ′ ∈ R Din :EQUATIONxEQUATIONTo ensure that the expanded part of x ′ starts with zeros, we still utilize a mask:EQUATIONEQUATIONRMSNorm Considering the parameter µ ∈ R d , our objective is to expand it toµ ′ = [αµ, ξ] ∈ R D ,where α is an undetermined coefficient and ξ is a randomly initialized new parameter. Let the input be x ′ = [x, 0 D-d ] ∈ R D , then we have:EQUATIONEQUATIONEQUATIONBy observing equation 32, we can conclude that, to achieve function preservation, α = √ d √ D . Finally, we can conclude:EQUATIONDepth In depth, preserving only the residual part and masking the MHA and SwiGLU layers can achieve function preservation:EQUATIONUsing G learn for width expansion, for the embedding layer E ∈ R V ×d , the parameter B emb ∈ R D×d is defined as follows:EQUATIONFor Attention layer, where W Q , W K , W V , and W O ∈ R d×d , and RMSNorm µ 1 ∈ R d , the parameters B Q , B K , and B V ∈ R D×d , we have:EQUATIONFor MLP, whereEQUATIONFor the output head W head ∈ R V ×d , we have:EQUATIONUsing G learn for depth expansion, consider a transformer model with L 1 layers, we use G learn to expand it to L 2 layers. For l ∈ {1, 2, • • • , L 2 }:EQUATIONwhere D Q,K,V,O,ln1 ∈ R L2×L1 represents learnable parameters. These parameters are used to expand the MHA vertically in depth. Similarly, for SwiGLU, we also perform expansion using a similar method. Formally, this can be written as:EQUATIONwhere D up,down,gate,ln2 ∈ R L2×L1 represents learnable parameters used for expanding SwiGLU in the depth.Embedding Consider a vocabulary size V and embedding size d. Then, the embedding matrix E ∈ R V ×d , and the one-hot vector for input tokens X is denoted as 1 X ∈ R T ×V , where T is the sequence length. Formally, it can be written as:EQUATIONfor i, v ∈ [V ], where i ̸ = j, it is guaranteed thatE i ̸ = E j .Multi-Head Attention Multi-Head Attention (MHA) consists of multiple attention heads, each of which computes its own self-attention. The results of these attention heads are then concatenated and projected to obtain the following output:Q i , K i , V i = XW Q i , XW K i , XW V i H i = sof tmax( QiK T i √ d h )V i M HA(X) = Concat(H 1 , • • • , H n )W O (44) here, the input X ∈ R T ×d , parameters W Q i ∈ R d×d h , W K i ∈ R d×d h , W V i ∈ R d×d h , and W O ∈ R d×d , where n × d h = d.Feed Forward Network The Feed Forward Network (FFN) consists of two linear layers and the activation function GeLU. Typically, the two linear layers first perform an up-projection to d F F N and then down-project back to the dimension d. Therefore, FFN is defined as:EQUATION)where the input X ∈ R T ×d , parameterW up ∈ R d×d F F N and W down ∈ R d F F N ×d .SwiGLU LLaMA replaces the original FFN in the Transformer Decoder with SwiGLU, resulting in improved performance. SwiGLU consists of three linear layers and the swiglu activation function.It can be defined as:EQUATION)where ⊙ means the element-wise multiplication, the inputX ∈ R T ×d , parameter W up ∈ R d×d F F N , W gate ∈ R d×d F F N and W down ∈ R d F F N ×d .RMSNorm Before MHA, FFN, or SwiGLU, there is a layer of RMSNorm to enhance the stability of the model. Compared to LayerNorm, RMSNorm is simpler in form. Formally, it can be written as:EQUATIONwhere X ∈ R T ×d , parameter µ ∈ R d . B.= L(M t-1 , d t ) M t ← A(M t-1 , loss) end M 0 = G(M k ) for t = 1 to K do ▷ Target Model Training loss = L(M t-1 , D t ) M t ← A(M t-1 , loss) end B.We use TinyLlama7 [45] as our pre-training codebase. We employ FSDP (Fully Sharded DataParallel) along with FlashAttention [46] 2.0, and other acceleration techniques. We use the open-source dataset Slimpajama-627B8 [47] for pre-training. The hyperparameters used for each model size are listed in Table 1 . Our 7B model is trained over around 100B tokens per day on an NVIDIA Hopper cluster. In Table 3 , we compare the harness evaluation results after training the G stack model and the scratch model (Baseline) for 100B tokens with Pythia-1B [51] and TinyLlama-1.1B, which are trained on the same number of tokens. The comparative results indicate that our baseline performs normally, comparable to pythia-1B. Meanwhile, the G stack model significantly outperforms both the baseline and pythia-1B, demonstrating the acceleration effect of G stack on the pre-training process. Although due to computational resource limitations, we only explore predicting g given N and C on the 1.1B and 3B models, we still attempted to fit using equation:EQUATIONIn the equation 49, N represents the number of target parameters, g represents the growth factor. The fitting result is as follows: We also visualize the fitted curves in Figure 22 , but the results were mediocre due to the lack of data. 50.We give an example of empirical usage of G stack by using the configurations of Llama2 and Llama3 families [21, 7] to show the estimated optimal base model training tokens d and growth factor g in Table 4 . FLOPs (1e+20) FLOPs (1e+20) FLOPs (1e+20) FLOPs (1e+20) FLOPs (1e+20) FLOPs (1e+20) H.2 Ablation:f 2 • f 1 • f 0 • f 2 • f 1 • f 0 or f 2 • f 2 • f 1 • f 1 • f 0 • f 0 (interpolation)To investigate whether the connections between layers affect the performance of stacking, we conduct a comparison of two approaches for stacking small models into larger ones. We explore two approaches for stacking small models into larger ones. The first approach involves taking the entire small model as a unit and directly stacking it, which can retain the connections between most layers. The second approach involves replicating and interleaving each layer in the small model, which almost break the connections. To measure the degree of retention of inter-layer connections after stacking, we define the connection rate R c :EQUATIONwhere the Con r is number of retained connections, the Con all is number of all connections.For example, if we had a small model with three layers, denoted as f 2 • f 1 • f 0 , and desired a model depth of 6, the first approach would result inf 2 • f 1 • f 0 • f 2 • f 1 • f 0 , where its R c = 80%. The second approach would result in f 2 • f 2 • f 1 • f 1 • f 0 • f 0 , where its R c = 40%.In our experiments, we stack a small model with 8 layers to a 24 layers target model. The growth timing d is 10B tokens and growing factor s is 3. The R c of G stack is 91.3% and the R c of G interpolate is 30.4%. We report the training loss and standard NLP benchmarks average accuracy in Figure 35 .At the beginning of training, interpolated stacking perform as well as stacking entire small model. However, as the training continues, the performance of interpolated stacking deteriorates.Therefore, we can conclude that the higher the connection rate of stacking, the better the effect of stacking. In Appendix H.3, we continue to validate this conclusion.0.0 2.5 5.0 7.5 10.0 12.5FLOPs (1e+20) We also report the details of evaluation results about 8 standard NLP benchmarks. FLOPs (1e+20) Partial stacking has been explored in LLMs like LlamaPro [42] , Solar [43] . But their goal is to stack an off-the-shelf LLMs such as Llama2, while our aim is to accelerate LLM pre-training process.To explore stacking which layers of the small model can achieve the best performance, we conduct experiments on partial stacking. In our experiments, we stack a small model with 6 layers ({L 1 , L We report the training loss and standard NLP benchmarks average accuracy in Figure 37 . By observing the loss curves in Figure 37a , we can find that the eight partial stacking methods are clearly divided into three groups based on their loss. The first group, {123456*4, 12-3456*5-56, 12-345*7-6, 123-456*7}, achieves the best performance. The second group consisting of {1234-56*10, 12-34*10-56, 1-234*7-56}, performs just so-so. The third group, {123*7-456}, performs poorly, even worse than the baseline.In Table 5 , we summarize the eight partial stacking and calculate the R c of each partial stacking methods based on Equation 51.For partial stacking, we conclude that: all > middle ≈ back ≫ front. Meanwhile, when the stacked parts are the same, the larger the R c , the better the performance. For the down projection in SwiGLU and the output projection in MultiHeadAttention, we apply noise:EQUATIONFor the Embedding Layer and other Linear Layers, we apply noise:EQUATIONAdding Noise on G direct to Break FP 0 1 2 3FLOPs (1e+20) FLOPs (1e+20) We utilize the codebase from Samba9 , which implements a hybrid State Space Model using the Slimpajama dataset for LM. In this experiment, we follow the guidelines outlined in the main paper to guide our stacking process. With a parameter size of 410M and training on 100B tokens, we set the growth timing to 8B and the growth factor to 3. We opted for 3 instead of 4 because Samba is an interleaving of Mamba and self-attention layers. Since the target model has 12 layers, we can only stack even layers, leading us to select a 4-layer base model (Mamba-SA-Mamba-SA).Our experiments results on loss curves 43 and downstream tasks 7 indicate stacking also works beyond Transformer-based LLMs. Please note that in Table 7 , we select stack with 47B rather than 50B to count the additional consumption required to train the base model on 8B tokens. FLOPs (1e+20) Tokens (Billions) As a successful exploration for efficient LLM pre-training, our work has great potential to give positive societal impact towards sustainable AI. Nevertheless, as a common drawback for LLMs, there are also chances that our LLMs might be misused intentionally or uniintentionally.NeurIPS Paper ChecklistGiven growth factor g = 4, the sum of FLOPs for training d = 10B and D = 97.5B approximately equals to consumption for training large LLMs D = 100B, which is the FLOPs of our baseline trained from scratch.In this study, we always calculate the consumption by combining the FLOPs required for both training small models and large models. So given g = 4, the consumption for training small model d = 10B equals to the cost for training D = 2.5B, so the plotted curves for Gstack actually starts at 2.5B.A very recent paper indicates training LLMs via stacking may improve in reasoning[44].Apache-2.0 licenseThe license of Slimpajama-627B includes: Common Crawl Foundation Terms of Use; C4 license; GitHub was limited to MIT, BSD, or Apache licenses only; Books: the_pile_books3 license and pg19 license; ArXiv Terms of Use; Wikipedia License; StackExchange license on the Internet Archivehttps://github.com/microsoft/Samba 


Decision: This paper systematically investigates the existing model growth techniques for efficient LLM pretraining. The authors classify the existing approaches into four atomic growth operators; and then by conducting comprehensive experiments, they find that the depth-wise stacking operator is the most effective one and can indeed accelerate the training of LLM significantly. Besides, the authors also provide clear empirical guidelines for the use of the depth-wise stacking operator, which is valuable for LLM practitioners aiming to accelerate LLM pretraining. 

All the reviewers agree that the experiments are comprehensive and convincing, and the empirical findings are informative and strong. All the reviewers appreciate this paper, though it lacks a theoretical exploration. I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The presented work systematically investigated the major obstacles of applying model growth methods to large language models and the corresponding solution. The empirical results reveals that depthwise stacking methods works the best to LLMs. The paper then studied how to practically use the depthwise stacking methods in detail.
strength_weakness: strength_weakness: - This paper provides practical guidelines on the usage of model growth methods on LLM training, but doesn't provide any theoretical analysis or intuition on why certain methods work or not work.
- Current study focuses on to making use of an existing model growth method under the scenario of LLM training, instead of improve it or modify it. It is possible that there exists a better model growth method for LLM that doesn't fit into the 4 catagories summerized by this paper.
questions: questions: See Weaknesses.
limitations: limitations: Authors discussed the limitations.
paper_summary: paper_summary: This paper studies the model growth technique for large language models, which expand a smaller pretrained large language model into a bigger one. The authors consider four natural way of expanding the parameters of the smaller pretrained large language models, and find that duplicating layers is the most effective technique. Therefore, this work delve deep into the dupicating/stacking layers technique, including the following aspects:

- Scaling model sizes
- Scaling training tokens
- estimating laws
- determining the growth timing
- determining the growth factor

and a bunch of other ablation studies.
strength_weakness: strength_weakness: - [Minor] The experiments are conducted on Transformer architecture. Trying different model architecture such as SSM can be more interesting.
- [Minor] The concept “model growth” is not introduced clearly until the related works.
questions: questions: See weakness.
limitations: limitations: The paper is mostly about empirical observation. However, the reason behind this phenomenon is still unclear and not much discussed.
paper_summary: paper_summary: The paper introduces a novel method for pre-training large language models (LLMs) efficiently using model growth techniques. The authors tackle three main obstacles: lack of comprehensive evaluation, untested scalability, and absence of empirical guidelines. They propose a depth-wise stacking operator, "Gstack," showing its effectiveness in reducing training time and computational resources across various LLM sizes, verified through extensive experiments and evaluations on multiple NLP benchmarks.
strength_weakness: strength_weakness: Although extensive, the experiments mainly focus on model performance from a computational and speed perspective. The impact on final model accuracy or downstream task performance is less explored, which could be crucial for practical applications.
questions: questions: Will this stacking method influence the robustness or generalizability of the LLMs on downstream tasks?
limitations: limitations: The paper lacks theoretical insights that could explain why the Gstack operator works well in practice. Including a theoretical analysis or rationale could strengthen the paper and provide a deeper understanding of the method.
paper_summary: paper_summary: The paper tackles the problem of model growth during training. The authors focus on the problem of efficient LLM pretraining and analyze a plethora of proposed growing techniques. The paper focuses on and established from the very beginning three clear and important objectives: 1. Comprehensive evaluation for decoder transformer language model training, 2. Generalization of small-scale results to bigger scales and 3. The establishment of clear guidelines for practitioners.
strength_weakness: strength_weakness: The motivation behind the three objectives is clear, yet at times insufficient. 

O1. Reported results in the literature are indeed focusing mostly on BERT pretraining. Although the evaluation framework proposed here is complete and comprehensive, it is not clear why decoder-transformer language modeling is fundamentally different and model growing results will in this case be different compared to BERT pretraining (apart from post-LN in the architecture). The authors should compare their findings with established results on BERT pretraining in the literature and discuss how their best configurations compare to them [1]. Note that other papers ([2, 19 from your pdf]) are dealing with growing operations during language model pretraining.

O2. Viability for scaling is important.  One of the issues of model growth not discussed sufficiently in this paper is the problem of "diminishing returns", meaning that one can save compute, but the ratio of compute saved becomes smaller as training progresses (this is hinted in Figure 6a), especially since you are performing a single growing step. 

Section 4.2 is very interesting and the experiments enlightening, but the analysis is superficial. [2] has a more detailed analysis to determine growing timings. In general, assuming that models are training following a power law loss, growing timings can be determined based on the gradient of the training losses. This was analyzed in detail in [3], where they describe in detail how to determine these timings. Your case is admittedly more complicated since you are growing in a non-functional-preserving manner, but fundamental take-away messages should be the same.

Section 5, is interesting, but both of the results seem to disagree with multiple findings in the literature. Although additional insights are presented in the appendix, these results seem preliminary and should be taken with a grain of salt.

[1] Wang, Peihao, et al. "Learning to grow pretrained models for efficient transformer training." arXiv preprint arXiv:2303.00980 (2023).

[2] Shen, Sheng, et al. "Staged training for transformer language models." International Conference on Machine Learning. PMLR, 2022.

[3] Anagnostidis, Sotiris, et al. "Navigating Scaling Laws: Compute Optimality in Adaptive Model Training." Forty-first International Conference on Machine Learning.
questions: questions: Some additional comments:
- Can you comment with more detail on why you think stacking multiple-times does/should not work?
- How specific are the results to Transformers and language models?
- Since your growing operator is not function-preserving, should there be spikes in the loss curves? If yes why are these not visible in the current figures?
limitations: limitations: See above.

Comments: comment: I'm satisfied with the authors' response and have decided to keep my rating as accept.
Thank you for the additional details, these are insightful. I would just be careful about over-claiming things regarding diminishing returns, especially since people nowadays train a lot past the Chincilla optimum.

Overall, the paper is a very good reflection and presentation of stacking approaches in the context of language modelling. I have increased my score.
I appreciate the authors response, which addressed all my questions. I am keeping my score.