Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
Authors: Mengzhou Xia
Tianyu Gao
Zhiyuan Zeng
Danqi Chen
Venue: ICLR 2024 poster (2024)
Keywords: pruning
efficiency
large language models
pre-training
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We prune existing large language models to produce strong smaller language models, with significant less compute compared to train such models from scratch.

Abstract: The popularity of LLaMA (Touvron et al., 2023a;b) and other recently emerged moderate-sized large language models (LLMs) highlights the potential of building smaller yet powerful LLMs. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains. We demonstrate the efficacy of our approach by presenting the Sheared-LLaMA series, pruning the LLaMA2-7B model down to 1.3B and 2.7B parameters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, OpenLLaMA and the concurrent TinyLlama models, on a wide range of downstream and instruction tuning evaluations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMs with structured pruning is a far more cost-effective approach for building competitive small-scale LLMs

Published as a conference paper at ICLR 2024
SHEARED LLAMA: ACCELERATING LANGUAGE MODEL
VIA STRUCTURED PRUNING
Mengzhou Xia', Tianyu , Zhiyuan Zeng², Danqi Chen'
!rinceton Language and Intelligence, Princeton University DDpaarmeett of Computer Science and Technology. Tsinghua University {mengzhou, t tianyug, danqic wwtngiiiees..prnntttt... princeton. edu sengzy200man .tt.sssiiii tsinghua. edu. on
ABSTRACT
The popularity of LLaMA (Touvron al., 2023a;b) and other recently emerged moderate-sized large language models (LL.s) highlights the potential of build- ing smaller yet powerful L.Mss. Regardless, the cost of training such models from scratch on trillions of tokens remains high. In this work, we study struc- tured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning. which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based varying losses across different do- mains. We demonstrate the efficacy of our approach by presenting the Sheared- LLaMA series, pruning the LLaMA2-7B model down 1.3B and 2.7B param- eters. Sheared-LLaMA models outperform state-of-the-art open-source models of equivalent sizes, such as Pythia, INCITE, LaMALLaMA and the concurrent TinyLlama models, on wide range of downstream and instruction tuning eval- uations, while requiring only 3% of compute compared to training such models from scratch. This work provides compelling evidence that leveraging existing LLMS with structured pruning is a far more cost-effective approach for building competitive small-scale LLMs.
INTRODUCTION
Large language models (LLMs) are extremely performant wide range of natural language tasks, but they require enormous amounts of compute to train (OpenAI, 2023; Anthropic, 2023). As such, there is growing interest in building strong moderate-sized models, such as LLaMA (Touvron al., 2023a;b), MPT MMosaicML, 2023), and Falcon (Almazrouei et al., 2023), that allow for efficient inference and fine-tuning. These LLMs are available in varied sizes suited for different use cases, but training each individual model from scratch- even the smallest billion parameter models requires substantial computational resources that are cost-prohibitive for most organizations. In this work, we seek to address the following question:
Can we produce smaller, general-pune purpose, and competitive LLM by leveraging existing pre-trained LLMs, while using much less compute than training one from scratch?
We explore structured pruning a means achieve this goal. Pruning is commonly viewed a so- lution for compressing task-specific models (Han et al., 2016; Li et al., 2016; Lagunas et al., 2021; Xia al., 2022; Kurtic et al., 2023), removing redundant parameters and accelerating inference without sacrificing task performance. However, for general-purpose L.MSs, pruning inevitably re- sults in performance degradation compared to original models (Frantar Alistarh, 2023; Sun al., 2023; Ma et al., 2023), especially when without significant compute invested post-pruning. In this work, we use pruning as an effective approach for developing smaller yet competitive LLMs that require only fraction of the training compute compared to training them from scratch.
Pleasee find our code and models at https: thub. com/pri nceton-ni n1 p/ L-M-- Shearing. We present frequently asked questions and answers in Appendix G.

---

Published as a conference paper at ICLR 2024
We identify two key technical challenges in this problem. First, how nal pruned architectures that are strong in performance and efficient for inference? Exist- ing structured pruning techniques for LLMs (Xia et al., 2022; Ma et al., 2023) do not spec- ify targeted structures and lead to suboptimal pruned models in terms of performance and in- ference speed (Table 4 and Appendix F.2). Second, how pruned model to reach desired performance? We observe that training using the original pre- training data leads imbalanced rates of loss reduction across different domains, compared to when training such models from scratch. This indicates that the pruned model retains vary- ing levels of knowledge for different domains (e.g.. GitHub vs. C4) and simply using the pre- training domain proportion results in inefficient use of data (Figure 4). To address these is- sues, we propose "LLLMshearrng", an algorithm consisting of the following two components:
decide fi-
continue pre-training the
We propose a novel pruning algorithm, dubbed targeted structured pruning, which prunes a source model to a specified tar- get architecture. The target architecture is determined by leveraging the configura- tions of existing pre-trained models. Our pruning approach searches for substruc-
Average Downstream (%6
Sheared-LLaMee (ours)
OpenLLaMA v2
32x faster
OpenLLaMA v1
55
INCITE
54
53
Pythia
tures within the source model that maxi- mally preserve performance while adher- ing the given constraints.
OPT +
We devise dynamic batch loading algo- rithm that loads training data from each domain in proportion to its rate of loss duction, thereby making an efficient use of the data and accelerating the overall per- formance improvement.
108
5DB
500B 1T
#Tokenss for Training
surpasses a se- ries open-source models at a similar scale and only requires 1/32 (3%) of budget to achieve on- par performance with OpenLLaMA-3B--V7
Figure 1: BLcared-LLaMA-2-.
We demonstrate the efficacy of our proposed method by pruning LLaMA2-7B model (Touvron et al., 2023b) into two smaller LLMs: Sheared-LLaMA-LL. 1.3B and using only 50 billion addtional tokens (i.e., 5% of OpenLLaMA's pre-training budget) for prun- and Sheared-LLLMM--2
Despite
ing and continued pre-training, Sheared-LLaMA-I1 popular LLMs at similar scales, including Pythia (Biderman et al., 2023), INCITE (TogetherAI, 2023b), and OpenLLaMA (Geng & Liu, 2023), on 11 representative downstream tasks (Figure I; commonsense, reading comprehension, and world knowledge) and instruction tuning for open- ended generation. Additionally, the downstream performance trajectory suggests that further train- ing the pruned model with more tokens would result even greater gains. While we only conduct experiments with up to 7B parameter models, our LLM- L..M-shcaring algorithm is highly generalizable and can be extended to large language models of any size in future work.
outperform other
LLM-SHEARING
Given existing large model My (the source model), study how to efficiently produce smaller, strong model Mr (the targer model). We consider this as a two-stage process: (1) Pruning Ms into My. This reduces the number of parameters but inevitably incurs a performance drop. (2) Continue pre-training My with a standard language modeling objective to reach a target performance, While most recent efforts (Xia al., 2022; Ma al., 2023) focus the former stage, we find the latter stage crucial for producing competitive general-purpose LLMs from structured pruning.
2.1 TARGETED STRUCTURED PRUNING
Structured pruning removes groups of model parameters to compress models and accelerate infer- ence. However, existing structured pruning approaches often result in unconventional model config- urations that deviate from popular architectures. For example. CoFiPruning (Xia et al., 2022) pro- duces models with non-unifee con-unnformma layer configurations (c.g., different numbers of heads across layers), which incurs inference overhead compared to standard uniform layer configurations (Section 4.2).
2

---

Published as a conference paper at ICLR 2024
2
head 4
inter
laver
Structured
Pruning
hidden-
EMB
MHA 1
FFN
MHA
FPN
EMB
MHA
FFN
MHA-2
FRN
MHA
FFN
Source Model
Target Model
Ls 3. 6, Hg m
LT dr H, 2, m,y
Figure 2: Targeted structured pruning produces compact and dense model of pre-specified shape. Light colors indicate pruned substructures. Masking variables are learned to control whether substructure is pruned (= 0) or retained (= = 1).
In this work, we aim to prune the source model into any target configuration that we specify. This goal is challenging because it requires surgically scaling down all dimensions in transformer ar- chitecture, an endeavor that, to our knowledge, has not been accomplished before for large language models. We leverage the configurations of existing pre-trained models the target architectures, based on the intuition that these configurations have already been well-optimized to balance model expressivity and inference efficiency. For example, we use the INCITE-Base-e architecture (To- getherAI, 2023a) the target structure when producing a 2.7B model.
Our method learns a set of pruning masks on model parameters at different granularities from global ones like layers and hidden dimensions (persist across all layers), to local ones like attention heads and intermediate dimensions. Assume that the source model Mg has Ls layers, with each layer consisting of one multi-head attention module (MHA) and one feed-forward network (FFN). My has a hidden state dimension of ds. Hs heads in each MHA, and an intermediate dimension of ms in each FFN. We introduce the following mask variables:
Granularity
Layer
Hidden dimension
Head
Intermediate dimension
Pruning masks
2 Jyer
R
hidden
E Rs
hend
E Rt Hs (xLs)
ii
R (xLs)
Each mask variable controls whether the associated substructure is pruned retained. For example, we remove a layer if its corresponding = _Inyer = O. Figure 2 illustrates an example of how the pruning masks control the pruned structures.
We formulate pruning as a constrained optimization problem (Platt & Barr, 1987) where we learn pruning masks to search for subnetwork matching pre-specified target architecture while maximizing performance." Following the 4o regularization approach (Louizos et al., 2018), we parametrize the pruning masks model hard concrete distributions. These distributions have sup- port on J0. 1] but concentrate their probability mass at 0 or 1, enabling discrete prune or retain decisions. While prior work usually control for target sparsity (Wang et al., 2020; Xia et al., 2022), we use pair of Lagrange multipliers to impose constraints on the pruned model shape di- rectly. For example, for target number of heads H (and we use dr, and m to represent target number of layers, hidden dimension, and intermediate dimension respectively), we have the imposed constraint on a single layer as:
Chead (), 0) Abead__ cE headd HT) + heoad
Σ
head H1T) 2
2
Similar constraints are applied to pruning other substructures. Overall, we jointly optimize the model weights and pruning masks by a min-max objective ming,: maxx.6 L0 X, Φ))
Ls
Ls
Cprume
(,, 50,
C(0, 2) + heaad + - Jint + Elayer + Chidden
where L(O, 2) the language modeling loss computed with the masked model weights. This objec- tive will produce a pruned model with the target shape. Ideally, running this pruning algorithm on a large amount of data will directly produce strong compact model. In practice, the pruning stage is expensive (roughly 5x slower compared to standard LM training), and we find that the learned
Pleasse find more detailed exposition of the algorithm in Appendix A.
3

---

Published as a conference paper at ICLR 2024
Algorithm 1: Dynamic Batch Loading
Require:
Training data of l domains D1, D2.... Da. validation data Diat D2 initial data loading weights E IR*, reference loss lret E R LM loss L or pruning loss Cpraee training steps T, evaluation per m steps, model parameters (9, 2, Φ, for pruning)
Dxal
for t = 1, 1,... T do
if mod m then
e < L(O, Dral) if pruning else L(0, Dial)
A - max [e, (ret(0)00)
b Calculate loss difference Update data loading proportion
We < Updat
(Wh-m- Ax)
end
Sample batch of data B from D1, D2, Dj with proportion wy; if pruning then
Update 8, o, A with Sprune (0, o, A) on B
else end end
Update 0 with (),
Subroutine Upt (w, A)
< w exexp (A) w
Calculate the unnormalized weights >> Renormalize the data loading proportion
+ return w
return 0
masks often converge fast. Therefore, we only allocate a limited budget for pruning (see Table 5). Following pruning, we finalize the pruned architecture by preserving the bighest-scoring compo- nents associated with the mask variables in each substructure, and continue pre-training the pruned model with the language modeling objective. We refer to this second stage as continued pre-training.
2.2 DYNAMIC BATCH LOADING
Continued pre-training on large amount of data crucial for recovering the pruned model perfor- mance. We observe a surprising finding in our preliminary experiments: continuing pre-training our pruned models on an existing pre-training dataset RedPajama (TogetherAI, 2023b; LLaMA's repli- cated pre-training dataset) reduces loss at different rates across domains compared to pre-training model from scratch, which signifies an inefficient use of data.
To be more specific, we begin by fitting a scaling function (Hoffmann et al., 2022; details in Ap- pendix B) on the series of LLaMA2 models for each domain. Using this function, we predict the loss of a hypothetical 1.3B LLaMA2 model if it were trained from scratch on the same data. We then compare these estimated reference losses to the losses of our pruned model after continued pre-training. Figure (left) shows that model's loss GitHub is better than the reference loss, while it is significantly worse than the reference loss on C4. This observation indicates that pruning preserves greater amount of knowledge in low-entropy and smaller domains (e.g., GitHuu) com- pared to high-entropy and larger domains (e.g., C4). Simply reusing the original pre-training data distribution results in an inefficient use of data and worse downstream performance, even the overall loss is seemingly low, as demonstrated later in Section 4.1.
Inspired by recent work (Xie et al., 2023), we propose dynamic batch loading, an efficient algorithm to adjust domain proportions on the fly based on losses. The goal is to ensure the model achieves the reference loss at roughly the same time across domains. We introduce the algorithm below.
Problem setup. The pre-training data comprises of & domains D1, Dy, ,Dx and we have held- out validation dataset for each domain, denoted as Dyal, At each training step t, a proportion wy [ of the data comes from domain D, We set a reference validation loss f D2)) for each domain and train the pruned model to reach the reference loss.
Thhe I.a.MA2 pre-training data is not public. We conducted the same analysis I.aMA1 models and observed similar phenomenon, indicating that this is universal issue unrelated specific pre-training data.
4

---

Published as a conference paper at ICLR 2024
Dynamic batch loading. We present the full algorithm in Algorithm 1. In a sketch, for every m steps, we evaluate the model to get the validation loss l (step t) on D and update wy based on the difference A, between Get [i] and E,[i] on each domain. The update rule is exponential ascent following Xie et al. (2023),
W/-m (x):
We apply dynamic batch loading to both the pruning stage and the continued pre-training stage. For pruning, we use the original pre-training data's 's domain weights wg. For continued pre-training, we use the final weights from the pruning stage as w/p Dynamic batch loading is an on-the-ffy solution that adjusts data proportions during training without the need for training auxiliary models. leverages reference losses on validation sets and adjusts the weights dynamically. adding minimal overhead to standard training. This approach differs from Xie et al. (2023), which requires a complex multi-stage process to train reference and proxy models.
More broadly, dynamic batch loading can train an LLM to match any reference model's performance by using open-source pre-training datasets like RedPajama, even without knowing the reference model's exact training data.
Choices of reference losses. By default, we use the loss predicted by the fitted scaling function as the reference (denoted as scaling reference). We also experiment with an alternative where we directly use the source model's 's domain validation loss as the reference (denoted as source reference). We show in F.4 that while both variants perform well, using scaling reference leads to slightly better downstream results, especially on math and coding tasks. However, source reference is viable alternative when a series of source models at different scales is not available.
EXPERIMENTS
3.1 SETUP
Model configurations. We use the LLaMA2-7B model (Touvron et al., 2023b) as the source model throughout all of our main experiments." We then conduct structured pruning experiments to compress this model down two smaller target sizes-2.7B and 1.3B parameters. We compare to strong pre-trained language models of similar sizes, including OPT-1.3B (Zhang et al., 2022), Pythia-I.4B (Biderman al., 2023), mmyLllma-1.1B (Zhang et al., 2024), OPT-2.7B. Pythia-2.8B, and OpenLLaMA-3B-
(TogetherAI, 2023b), OpenLLaMA-333 Liu, 2023). We use Pythia-1.4B and INCITE-Base-3BB the target architecture for the 1.3B and the 2.7B model respectively. Table summarizes model architecture details of all these models.
(Geng &
Data. As the training data for LLaMA2 is pub- licly accessible, we use RedPajama ATogetherAI. 2023b), which is a replicated pre-training dataset of the LLaMAI models (Touvron et al., 2023a), for pruning and continued-peerrainingg This dataset encompasses training data from seven domains: CommonCrawl, C4, Github, Wikipedia, Books, ArXiv, and StackExchange. We con- struct a held-out validation set with 2 million tokens (equivalent to 500 sequences of 4,096 tokens) for each do- main. We allocate 0.4 billion tokens for the pruning phase and 50 billion tokens for the continued pre-training pro- Following the conventions of LLaMA2,
Table 1: datasets used by Sheared-LLaMA and other models.
A summary of pre-training
Model
Pre-training Data
#Tokens
LLaMAI LLaMA2
LLaMA data
IT
Unknown
OPT
OPT data"
300B
Pythia
The Pile
3008
INCITE-Base
RedPujama
OpenLLaMA v1
RedPujama
1T
OpenLLaMA v2
OaeALLaMA data"
IT
main-
TanyLlama Sheured-LLaMA
Tinyl lama data'
3T
tain a sequence length of 4,096 tokens. Table provides a summary of the pre-training data used by our models and the baseline models.
RedPujama
50B
* Pleasee find results on LLaMAI models in Appendix F.6 and Pythia models in Appendix F.5. *OPT data contains BookCorpus (7hu et al., 2015). Stories (Trinh & Le, 2018), CCNews (Hamborg et al., 2017), the Pile (Gao et al., 2020). and PushShift.i Reddit (Baumgartner et al., 2020). 'Oeen v2 is pre-trained with a mixture RefinedWeh (Penedo et al., 2023), StarCoder (Li et al.,
2023), and part of RedPajama.
"TinLLJame data mixture of SlimPajama (Shen al., 2023) and StarCoder data.
5

---

Published as conference paper at ICLR 2024
Table Sheared-LLaMee outperforms publicly available models comparable size on downstream tasks. The shot number used is noted in parentheses, with 0-shot if not specified. Models with use different training data from RedPajama. Please refer to Table for details.
Commonsense Reading Comprehension
Model
(tiokens for training)
SciQ
PIQA
WinoGrande
ARC-E ARC-C (25) HellaSwag (10)
LLaMA2-7B an)
93.7
78.1
69.3
76.4
53.0
78.6
OPT-1.3B (300B)
84.3
71.7
59.6
57.0
29.7
54.5
Pythia-1.4B (300B)
86.4
70.9
57.4
60.7
31.2
53.0
TinyL (ST)
88.9
73.3
58.8
60.3
(508)
87.3
73.4
57.9
61.5
33.5
60.7
OPT-2.7B (800B)
85.8
73.7
60.8
34.0
61.5
Pythia-2.8B (300B)
88.3
74.0
59.7
64.4
36.4
60.8
INCITE-Base-BB (8008)
90.7
74.6
63.5
67.7
40.2
64.8
Open-LLaMA--3B-V (TT)
91.3
73.7
61.5
67.6
39.6
62.6
Open-LLaMA-3B--1 (IT)
91.8
76.2
63.5
66.5
39.0
67.6
Sheared-L.laMA-2.77 (508)
90.8
75.8
64.2
67.0
41.2
70.8
Continued
LM
World Knowledge
Average
Model
(tiokens for triiing))
LogiQA BoolQ (32)
LAMBADA
NQ (32)
MMLU (5)
LLaMA2-7B an)
30.7
82.1
28.8
73.9
46.6
64.6
OPT-1.3B (300B)
26.9
57.5
58.0
6.9
24.7
48.2
Pythia-1.4D (300B)
27.3
57.4
61.6
6.2
25.7
48.9
TinyLlama-I.I (3T)
26.3
60.9
58.8
50.0
(508)
26.9
64.0
61.0
9.6
25.7
51.0
OPT-2.7B (300B))
26.0
63.4
63.6
51.4
Pythia-2.8B (300B)
28.0
66.0
64.7
9.0
26.9
52.5
INCITE-Base-3B
27.7
65.9
65.3
14.9
27.0
54.7
Open-LLaMMA-3B-1 (T))
28.4
70.0
65.4
18.6
27.0
55.1
Open-LLaMA-3B-- (T)*
28.1
69.6
66.5
17.1
26.9
55.7
Sheared-IllaMA-22777 (50R)
28.9
73.7
68.4
16.5
26.4
56.7
Evaluation. We use the 1m- in-evaluation wewallation-hareess harness package (Gao et al., 2021) to evaluate on an extensive suite of downstream tasks: (1) We follow Pythia and LLaMA2 to report the 0-shot accuracy of ARC easy (ARC-E; Clark et al., 2018), LAMBADA (Paperno et al., 2016). LogiQA (Liu et al., 2020), PIQA (Bisk al., 2020), SciQ (Welbl et al., 2017), and WinoGrande (Sakaguchi al., 2021). (2) We report accuracy of the tasks used by Open LLM Leaderboard (Beeching et al., 2023), including 10-shot HellaSwag (Zellers et al., 2019), 25-shot ARC Challenge (ARC-C: Clark et al., 2018), and 5-shot MMLU (Hendrycks et al., 2021). (3) We also report exact match of 32-shot Natural Questions (NQ; Kwiatkowski et al., 2019) to measure the factual knowledge in the model.
As training models to follow instructions has become crucial application of LLMs (Ouyang al., 2022: Taori et al., 2023), we evaluate our models on instruction tuning and fine-tune both baseline models and Cheared-LLaMe
instruction-res response pairs sampled from the ShareGPT dataset." Please refer to Appendix E for more details.
3.2 SHEARED-LLAMA OUTPERFORMS LMS OF EQUIVALENT SIZES
We demonstrate that Sheared-LLate outperforms existing LLMs of similar sizes on both standard LLM benchmarks and instruction tuning, while using only fraction of the compute budget required to train those models from scratch.
Downstream tasks. Table 2 presents the zero-shot and few-shot downstream task performance of Sheared-LLaMA and similarly-sized pre-trained models. Even with limited budget of ap- proximately 50B tokens for pruming and continued pre-training. form existing models pre-trained significantly larger compute. Sheared-LLaMA-115. outper- forms OPT-1,3B, Pythia-1.4P (pre-trained with 300B tokens), and TinyLJama-1. 1. (pre-trained
Seeared-LLaMA models outper-
httpps:/ htttpss///shareptt.oo

---

Published as a conference paper at ICLR 2024
Sheared-LLaMA-L3B
Pythla-1.4B
Sheared-LLaMA-2778
INCITE- BaLTE-BBaee-3
57.4%
42.7%
63.5%
36.6%
Sheared- Sheared-LLaMA-2 2.7B
Open-LLaMA-v1-3B
Sheared-LLaMes 2.78
Open-LLaMA-v2-21
56.6%
43.5%
54.3%
45.8%
Figure 3: Sheared-LLaM.A outperform Pythia-1.49 1.4B, INCITE-Base-33 3B, OpenLLaMA-33-vv and OpenLLaMA-3BB La--MM--A-B-2222 in instruction tuning.
on 3T tokens). Sheared-LLaMA- 2.7BB outperforms INCITE-Base-3i (pre-trained on 800B Red- Pajama tokens), OpenLLaMA-3BB (pre-trained on IT RedPajama tokens), and OpenLLaMA-3BB. v2 (trained on IT tokens from RedPajama, RefinedWeb, and StarCoder). The most noteworthy outperforms TinyL.ama-I.
result is that Sheared-LLaMA-I133 pre-trained on 3T tokens-mon more than the total data used for pre-training LLAMA2-7B and our pruning process combined. This demonstrates that structured pruning is a more sample-efficies approach for training smaller-scale LLMs.
despite TinyLLama-1. being
Instruction tuning. As shown Figure 3, instruction-ture Sheared-LLaMA achieves higher win rates compared to all the other pre-trained models at comparable scale. This demonstrates that our 2.7B model can serve as strong foundation for instruction tuning and has the capacity to generate long, coherent and informative responses (See examples in Appendix E).
ANALYSIS
4.1 EFFECTIVENESS OF DYNAMIC BATCH LOADING
We amalyze the effectiveness of dynamic batch loading by examining its impact on three aspects: (I) the final LM loss across domains, (2) the data usage of each domain throughout training, (3) the downstream task performance. All results in this section are based on Cheared-LLaMA-115
Loss differences across domains. Dynamic batch loading aims balance the rate of loss reduc- tion across domains, ensuring that the losses reach the reference value at roughly the same time. Figure 4 shows the difference between model's loss (with both original and dynamic batch load- ing) and the reference loss, estimated by fitting a scaling function to a hypothetical 1.3B parameter ILaMA2 model. The original batch loading results in widely varying loss differences across do- mains; for example, the GitHub loss decreases below the reference value, while the C4 loss lags behind. Dynamic batch loading, however, reduces losses evenly and leads to very similar loss dif- ferences across domains, suggesting more efficient data use.
Data usage. Table 3 compares the data proportion of RedPajama and the data usage of our dy- namic loading approach (Figure illustrates how the domain weights change during training). shows that dynamic batch loading loads more data from the Book and C4 subsets, indicating that these domains are more challenging for a pruned model to recover.
Table 3: Domain data usage with dynamic batch loading compared to the original proportions.
CC
GitHub
Book StackExchange Wiki
ArXiv
C4
RedPajama (Original)
67.0%
4.5%
4.5% 9.1%
2.0%
4.5%
2.5%
15.0%
Dynamic Batch Loading 36.1%
0.8%
1.0%
3.1%
0.7%
49.2%
Downstream performance. As shown in Figure 5, pruned models trained with dynamic batch loading achieve better downstream performance than when trained on the original RedPajama dis- tribution. This suggests that the more balanced loss reduction from dynamic batch loading transfers to improved downstream capabilities.
We also experiment with a heuristic approach to exclude the easy domains from pruning, but find that the loss disparaty issue persists after continued pre-training. Please refer to Appendix F.8 for mode details.

---

Published as a conference paper at ICLR 2024
Original
Dynamic Batch Loading
Average Dounstream Acc (16)
0.1
Loos Diferenoe
48
Original
47
Dynamic Batchh Loading
Cc
Gitraa
Book
Way
Ardoy
Gilia Book
Wild
20
30
40
50
4TTokons Training
Figure 4: Loss difference between the pruned model (1.3B) and estimated reference loss, with original vs. dynamic batch loading.
Figure Downstream task performance of Sheare--LLaMA-1.333 with original data pro- portion and dynamic batch loading.
4.2 COMPARISON TO OTHER PRUNING APPROACHES
We compare our LLM-shearing other pruning approaches on validation perplexity, a strong indi- cator of overall model capabilities (Xia et al., 2023).
Targeted pruned models have higher inference speed. Previous works like CoFiPruning (Xia et al., 2022) produce structued pruned models, but these models often have non-uniform layer con- figurations (e.g., different numbers of heads across layers). Such non-uniformity across layers intro- duces training and inference overhead due to irregularities in model architectures. We experiment with both CoFiPruning and targeted structured pruning, with a 0.4B pruning budget with the Red- Pajama data proportion for fair comparison. Table 4 shows that our targeted pruned models have a higher inference speed compared to the non-uniformly pruned CoFiPruning model at the same sparsity, despite having slightly higher perplexity. Targeted structured pruning needs about 0.5B more tokens continued pre-tras pre-training to match CoFiPruning' s perplexity. However, this one-time it results in more efficient model architecture that is
extra compute during training is justified, essential for real-world applications and effective practical use. Please find more details on inference speed of different pruning methods in Appendix F.9.
Table 4: Validation perplexity and generation speed during inference (tokens/second) of targeted structured pruning with uniform layer configuration, and CoFiPruning, with a non-uniform layer configuration. Inference speed is measured on Nvidia A100 (80G) GPU, on a singal instance generating up to 512 tokens.
Layer Config
JPL.
Speed
Layer Config
PPI.
Speed f
CoPiPruning
9.1
51
CoFiPruning
7.0
37
1.3B Targeted pruning
2.7B
10.3
Targeted pruning
7.7
Comparison to LLM-Pruner (Ma et al., 2023). We compare targeted structured pruning to LLM-Pruner, recent work in structured pruning, in Appendix F.2. We demonstrate that, given the same compute budget. sparsity level, and training data distribution, our pruned models achieve lower perplexity, have more optimized architecture, and faster inference speed.
4.3 ADDITIONAL ANALYSIS
Budget allocation for pruning and continued pre- training. Intuitively, allocating more compute the pruning stage helps identify better subnetwork structures. We explore distributing data across pruning and contin- ued pre-training stages differently, within fixed budget of 5B tokens. Table 5 shows that when controlling the to- tal amount of tokens, increasing the pruning budget con- sistently improves perplexity. However, since pruning is more expensive than continued pre-training, to allocate 0.4B tokens to pruning. Please refer to Ap- pendix C for details training throughputs.
Table 5: Data budget allocation to prun- ing and continued pre-trat pre-training (CT) and corresponding perplexity.
# Tokens
PPL
Pruning
CT
Pruning
CT
0.2B
4.6B
12.99
7.46
decide
0.4B
4.4B
10.29
7.32
0.8B
4.0B
9.01
7.23
1.6B
3.2B
8.04
7.08
8

---

Published as a conference paper at ICLR 2024
More analysis. We provide further analysis in the appendix: (1) Sheared-LLaMe evaluation math and coding (Appendix F.3), (2) Pythia model pruming (Appendix F.5), and (3) impact of ex- cluding easy domains during pruning (Appendix E.8).
RELATED WORK
Pruning. Structured pruning has been extensively studied as model compression technique in computer vision and natural language processing, where task-specific models like classification often overparameterized and can be pruned significantly with minimal impact on perfor- mance (Han et al., 2016; Wen et al., 2016; Liu et al., 2017; Luo et al., 2017; Cai et al., 2019; Deng al., 2020; Hou et al., 2020; Wang et al., 2020; Lagunas et al., 2021; Xia al., 2022; Kurtic al., 2023). Unstructured pruning (Frankle & Carbin, 2018: Li et al., 2020; Chen et al., 2020; Sanh et al., 2020) prunes individual neurons instead of structured blocks. Though unstructured pruning usually achieve higher compression rates, they are not practical for model speedup.
In the era of LLMs, the prevalent NLP pipeline has shifted from task-specific models general- purpose LMs, which leaves little room for redundancy. Both unstructured pruning, semi-structured pruning (Frantar Alistarh, 2023: Sun al., 2023), and structured pruning (Ma et al., 2023) lead to significant performance drops on LLM even at modest sparsity. Noticeably, all previous works fix the original models or tune them minimally. We see pruning as an initialization and consider it nec- essary expend substantial compute to continually pre-training the model recover performance.
Efficient pre-training approaches. As orthogonal to our pruning approach, There is an extensive body of work on improving efficiency of training LLMs. For example, quantization reduces the numeric precision of model weights and activations and speeds up training and inference (Dettmers al., 2022; 2023; Xiao et al., 2023). Knowledge distillation (Hinton et al., 2015; Sanh al., 2019; Jiao et al., 2020; Sun et al., 2020), which trains a smaller model on a larger model's prediction. is shown to be effective for task-specific models (Xia et al., 2022). For pre-training I.LMs, though distilling from a teacher model is shown to improve the quality of student models given the same number of training steps (Rae et al., 2021; : Blakeney et al., 2022), is less cost-effective than pruning and continued training due to the exceeding inference cost incured by the teacher model (Jha al., 2023). More methods have been introduced to enhance the efficiency of training LMs, such as dynamic architectures (Gong et al., 2019; Zhang & He, 2020) and efficient optimizers (Chen et al., 2023; Liu et al., 2023). However, as indicated by (Kaddour et al., 2023; Bartoldson et al., 2023), the promised gains in training efficiency may not be consistently realized.
There are also data-based approaches to enhance training efficiency. Eliminating duplicated data is found to be effective (Lee al., 2021). Various batch selection techniques propose prioritize data based on criteria such as higher losses (Jiang et al., 2019) or greater reducible loss (Mindermann al., 2022). Xie al. (2023) propose optimize data mixtures by training a proxy model to estimate the optimal data weight of each domain.
6 DISCUSSION
Limitation and future work. First, The method heavily depends on the availability of open- source pre-training datasets and models. data, the method may not recover performance well on that domain. Second, Due to computational constraints, we only experimented with 7B parameter model as the source model. However, our method is highly generalizable and can be scaled up to larger models in future research.
specific domain not covered the pre-training
Conclusion. This work proposes structured pruning as an efficient method for creating competi- tive smaller-scale L.MMs. Our two-stage approach combines targeted structured pruning and contin- ued pre-training (continued pre-training), and we introduce dynamic batch loading to improve pre- training data efficiency. We train series of competitive Sheared-LLLaMA models using fraction of the compute required for standard pre-training, Our results show a promising path to producing low-cost, small LLMs when strong large-scale models are available. As more capable LLMs and larger pre-training datasets emerge, our method can easily extend to these advances to create even better small models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We express our gratitude to Sadhika Malladi, Tanya Goyal, Ofrr Press, Adithya Bhaskar, and the Princeton NLP group for reviewing the paper and providing helpful feedback. We also thank the en- gineering team MosaicML for their invaluable assistance with implementation specifics using the Composer package. Mengzhou Xia is supported by a Bloomberg Data Science Ph.D. Fellowship, and Tianyu Gao is supported by an IBM PhD Fellowship. This research is also supported by Mi- crosoft Azure credits through the "Accelerate Foundation Models Academic Research" " Initiative.
REFERENCES
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Co- jocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. Falcon-40B: an open large lan- guage model with state-of-the-art performance. 2023.
Anthropic. Introducing claude, 2023.
Brian R Bartoldson, Bhavya Kailkhura, and Davis Blalock. Compute-efficiet deep learning: Algo- rithmic trends and opportunities. Journal of Machine Learning Research, 24:1-77, 2023.
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackbum. The pushshift reddit dataset. ArXiv, abs/2001.08435, 2020.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https: /open le____eaderbbrrrr 2023.
//buggingface. ca cttccc/aaaaaeeeeeeeeeeccaeeeeaee spaces /
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien. Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings the AAAI conference artificial intelligence, volume 34, pp. 7432-7439, 2020.
Cody Blakeney, Jessica Zosa Forde, Jonathan Frankle, Ziliang Zong, and Matthew Leav- itt. Reduce, reuse, recycle: Improving training efficiency with distillation. arXiv preprint arXiv:221 1.00033, 2022.
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2019.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained bert networks. In Advances in Neural Information Processing Systems, 2020.
Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang. Yao Liu, Hieu Pham, Xu- anyi Dong, Thang Luong, Cho-Jui Hsieh, et al. Symbolic discovery of optimization algorithms. arXiv preprint arXiv:2302.06675 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 0118803.0555., 2018.
Tri Dao, Dan Fu, Stefano Ermon. Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:1 35:16344-1635, 2022.
10

---

Published as a conference paper at ICLR 2024
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer Llm.int8 O: 8-bit matrix multiplication for transformers at scale, arXiv preprint M.XXv:2208.073350 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14311 2023.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori В Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305. 2023.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference Learning Representations, 2018.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot, 2023. arXiv preprint arXiv:2301.00771 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprini arXiv:2101.0002. 2020.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. International conference on machine learning, pp. 2337-2346. PMLR, 2019.
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to (re) warm your model? arXiv preprint arXiv:2308. 230.::3300..4014,, 2023.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedinggs of the 15th International Symposium of Information Science, pp. 218 218-223, 2017.
Song Han, Huizi Mao, Dally, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:150..02531, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
Ananya Harsh Jha, Dirk Groeneveld, Emma Strubell, and Iz Beltagy. Large language model distil- lation doesn't 't need teacher. arXiv preprint arXiv:2305. 14864, 2023.

---

Published as a conference paper at ICLR 2024
Angela H Jiang, Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger, Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, et al. Accelerating deep learning by focusing the biggest losers. arXiv preprint ar.iv:191..00700 2019.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163-4174, 2020.
Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale Minervini, and Matt Kusner. No train no gain: Revisiting efficient training algorithms for transformer-based language models. arXiv preprint arXiv:2307.06440, 2023.
Eldar Kurtic, Elias Frantar, and Dan Alistarh. Ziplm: Hardware-aware structured praning of lan- guage models. arXiv preprini arXiv:2302. 04089, 2023.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04836 2021.
Katherine Lee. Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2
2021.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2016.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.061666 2023.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transform- ers. In International Conference on machine learning, pp. 5958-5965 PMLR, 2020.
Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-orde -oder optimizer for language model pre-training arXiv preprint arXiv:2305. 14342, 2023.
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3622- 3628, 2020.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision, pp. 2736-2744, 2017.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through 1.0 regularization. In International Conference on Learning Representations, 2018
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066, 2017.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305. 11627, 2023.
Sören Mindermann, Jan M Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Win- nie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Confer- ence on Machine Learning, pp. 15630-15649, PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
MosaicML. composer, 2021.
MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable Ilms, 2023.
Accessed: 2023-05-05.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35, 2022.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-1534, 2016.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 166.011166110 2023.
John Platt and Alan Barr. Constrained differential optimization. In Neural Information Processing Systems, 1987.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11444 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications the ACM, 64(9):99-106, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv::910.011000 2019.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning:
Adaptive sparsity by fine- tuning. Advances in Neural Information Processing Systems, 133:20378-20389 2020.
Noam M. Shazeer. Glu variants improve transformer. ArXiv, abs/2002.05202, 2020.
Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing. Slimpajama-dc: Understanding data combinations for 11m training. arXiv preprint arXiv:2309.10815 2023.
Mingjic Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.1 1695, 2023.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task -agostii bert for resource-limits devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2158-2170, 2020.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: catructton-following llama model, 2023.
TogetherAI.
2023a.
TogetherAI. Redpajama: An open source recipe reproduce llama training dataset, 2023b.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXi::2302.1397 2023a,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint w.Xiv:2307009288. 2023b.
13

---

Published as a conference paper at ICLR 2024
Trieu H. Trinh and Quoe V. Le. A simple method for commonsense reasoning. ArXiv, abs/1806.02847, 2018.
Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint 1rXii:3305.17922 2023a.
Yunhe Wang, Hanting Chen, Yehui Tang, Tianyu Guo, Kai Han, Ying Nie, Xutao Wang, Hailin Hu, Zheyuan Bai, Yun Wang, et al. Pangu-pi: Enhancing language model architectures via nonlinear- ity compensation. arXiv preprint arXiv:2312. 17276, 2023b.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6151-6162, 2020.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy bserr-eenrratd Text, pp. 94-106, 2017.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29, 2016.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 1513-1528, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.
Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. Training trajectories of language models across scales. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pp. 13711-13738, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 18.18/53/v www......l-loo.7770.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma. and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. arXiv preprint arXiv:2305. 10429, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33:14011- 14023, 2020.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.0224 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang. Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. 11277, 2023.
Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19-27, 2015.
14

---

Published as a conference paper at ICLR 2024
CONTENTS
Introduction
2
LLM-Shearing
2.1 Targeted Structured Pruning
2.2
Dynamic Batch Loading
3
Experiments
3.1 Setup
3.2
Sheared-LLaMA Outperforms LMs of Equivalent Sizes
Analysis
4.1
Effectiveness of Dynamic Batch Loading
4.2
Comparison Other Pruning Approaches
4.3
Additional Analysis
8
5
Related Work
6 Discussion
A A Detailed Exposition of Paramaterizing Pruning Masks
16
Reference Loss Predicted by Scaling Laws
16
C Training Details
17
D Model Configurations
17
Instruction Tuning
17
F
Additional Experiment Results
18
F.1 Data Usage in Continued Pre-training
18
F.2
Comparison LLM-Pruner
19
F3
Coding and Math Reasoning
20
F4
Scaling Reference vs. Source Reference
20
F.5
Pruning Pythia Models
21
F6
Pruning from LLaMAl vs LLaMA2
22
F77
Comparison Further Continual Pre-training INCITE-Base-3B
22
F88
Exeluding Easy Domains During Pruning
23
F99
Inference Speed Analysis
24
G Frequently Asked Questions
24
15

---

Published as a conference paper at ICLR 2024
A A DETAILED EXPOSITION OF PARAMATERIZING PRUNING MASKS
The key idea behind the pruning algorithm is to apply masks to the model parameters. After learning a binary mask, is equivalent to removing the corresponding parameters. The mask is parameterized using a hard concrete distribution introduced in Louizos et al. (2018). Given a masking variable parameterized by a, the hard concrete distribution is defined as follows:
u(0,1) 1)
log 1- + log ))
Sigmoid
B
s(C - 7) + min(1, max (0, 5))
where Le is the uniform distribution, 3 is temperature parameter, s is relaxed binary mask that conforms the hard concrete distribution, and Ç and y are the bounds of the hard concrete distribu- tion. The hard concrete distribution serves as continuous relaxation of the binary mask, allowing the model to learn the binary mask in a continuous manner during training. The effectiveness of this trick in learning sparse structures in neural networks has been demonstrated in previous studies (Wang et al., 2020; Xia et al., 2022). In our experiments, we set 3 0.83, Ç = 1.1, and y = -0.1.
To enforce the sparsity constraint, the masks are trained alongside with Lagrange multipliers X, as defined in Equation (I). After pruming, the parameters corresponding to the learned masks are removed ensure that the resulting model shape matches the target model. In practical implementa- tions, we set a threshold to binarize the masks. Due to the adoption of the hard concrete distribution, the masks typically converge binary values that match the target model shape in most cases, thereby avoiding any inconsistencies. However, in rare instances where the masks do not converge exactly 0 1, the masking variables need to be absorbed into the resulting model parameters.
As discussed in Section 2, we apply masks to heads, intermediate dimensions, layers and hidden di- mensions. For heads, we simply multiply the head output by the mask. For intermediate dimensions, we apply the mask to the intermediate output. For layers, apply the mask to the layer output. For hidden dimensions, we apply the mask to both the head and mlp output. Applying the mask to outputs is equivalent to removing the corresponding parameters. Please refer to composer llama.py for more details.
B REFERENCE LOss PREDICTED BY SCALING LAWS
The scaling law of language modeling is a function of model size N and dataset size D:
A
B
L(N,D)= E
No
D
where E captures the loss for the true language distribution in an ideal generation process, and A, a, 3 3 are scaling factors related to model scale or data size. Models in the same model family are usually trained with the same amount of tokens on the same data distribution. In this case, we need a minimum of three models to estimate the constant E. DE with different amount of tokens, we can estimate F, A, a, B, with a minimal of 5 models. Note that we will estimate the scaling factors for each domain seperately.
B A and a. If the models are trained
LLAMA2 models have been trained on the same 2T tokens (Touvron et al., 2023b). We take the LLAMA2-7B. 1LAMA2-13B. and LLAMA2-70B checkpoints, evaluate them each domain's 's validation set, and fit the scaling factors with the corresponding loss. Given the limited data points for estimating the scaling law constant, the projected loss of hypothetical LLLMA-2.78 model may be biased compared to the true value. Table 6 presents the predicted loss. The evaluation process takes less than 4 A 1100 GPU hours complete.
16

---

Published as a conference paper at ICLR 2024
Table 6: Estimated reference loss of hypothetical LLaMA2-1.3B and LLaMA2-2.7B models.
CC
GitHub
Book
StackExchange
Wiki
ArXiv
C4
LLaMA2-1.3B
1.964
0.746
2.139
1.612
1.759
1.445
2.125
LLaMA2-2.7B
1.871
0.688
2.033
1.535
1.630
1.356
2.033
C
TRAINING DETAILS
We present the hyperparameters used in our experiments in Appendix C. We use fully sharded data parallel (Zhao et al., 2023) to train our models in parallel. We use FlashAttention VI (Dao et al., 2022) to speed up training. We cosine learning rate scheduler and decay the learning rate to minimum of 10% of the peak value. We conduct some preliminary experiment to determine the peak learning rate for learning the masking variables and Lagrange multiplers, and we find that learning rate of 1.0 works well for pruning. We do not tune any other hyper-parameters. The throughput is dependent the implementations and believe that our throughput can be further improved by adopting more advanced recent optimizations such as FlashAttention V2 (Dao et al., 2022) and more recent version of Composer (MosaicML, 2021).
Table 7: Training tlyeer-prameeee and throughput.
Pruning
Contined Pre- Pre-training
Training budget
0.4B
50B
Learning rate of 2, 0, Learning Rate of 0
1.0
0.0001
0.0001
LR warmup ratio
10%
3%
Batch size (tokens)
131K
1M
Evaluation interval m (steps)
400
Steps
3, 200
200
# GPUs
16
Throughput (tokens/s)
15K
145K (L.3B) 77K (2.7B)
D MODEL CONFIGURATIONS
In this section, we provide the model configurations for both our Sheared-LLaMA models and the baseline models, as illustrated in Table 8. Our design closely adheres to the architecture of Pythia- 1.4B and INCITE-Base-31 albeit with some nuanced distinctions. noteworthy difference is found in the intermediate size of Sheared-LLaMA, which is consequence of its lineage from LLaMA2- 7B. Notably, LLaMA2-7B employs a GLU variant (Shazeer, 2020) within its feed-forward layer, comprising a gate matrix, contrast, other models employ the conventional double-matrix feed-forward layer structure. Fur- thermore, acknowledge that the shearing algorithm will have inherit the head dimension of the source model. Instead of explicitly specifying the number of heads based on existing language models, set the target number of heads be the target hidden dimension divided by the head dimension of the source model.
apward-projection matrix, and downward-projeee
matrix. In
E INSTRUCTION TUNING
We evaluate our models on instruction tuning and fine-tune both Sheared-LLaMA and baseline mod- els on 10,000 instruction-res perruction-responee sample another 1,000 instructions from ShareGPT, generate responses from our fine-tuned models and other baseline models, and use GPT-4 as an evaluator to compare the two responses (Dubois et al., 2023). We report the win rate of our model compared to the baseline model.
response pairs sampled from the ShareGPT " For evaluation, we
During instruction tuning training, the instruction is prepended with "You are helpful assistant. Write response that appropriately completes the request.". For evaluating the instruction tuning
hhttps::/ http::////aaaapt..eeo com. We only the first round in the multi-turn chat history.
17

---

Published as a conference paper at ICLR 2024
Table Model configurations of our Sheared-LLaMA and baseline models.
Model
#Param
#ayeers
Hidden
Intermediate
#Heads
Head Dim
OPT-1.3B
1.3B
24
2048
8192
32
64
Pythia-1.48
1.4B
2048
8192
16
128
TinyLlama-I.1B
1.IB
22
2048
5632
32
64
Sheared-LLaMA-I.3B
1.3B
24
2048
5504
16
128
OPT-2.7B
2.7B
32
2560
10240
32
80
Pythia-2.8B
2.8B
32
2560
10240
32
80
INCITE-Base-3B
2.8B
32
2550
10240
32
80
OpenI.LaMA-3B
2.7B
3200
8640
32
100
Sheare--LLaMA-2.7B
2.7B
32
2560
6912
20
128
L.aMA2-7B
6.7B
4096
11008
32
128
generations, Wang et al. (2023a) observes using GPT models as a judge could change its preference when swapping the presentation order of the two outputs. Therefore, we compare each output pair twice by swapping the presentation order of the two outputs and finally report the average win-rate of the two rounds to eliminate the position bias.
We randomly select an output generated by Sheared-LLaMA-1... and Sheared-LLaMM-22777 in response to given instruction, and present the generations in Table 10. Our findings demonstrate that, after instruction tuning. Shcared-LLaMe formative outputs in response the instruction.
consistently produces long, coherent, and in-
Table 9: Training syper-parameters for instruction tuning.
Instruction Tuning
Learning Rate of LR warmup ratio Batch size (tokens)
5e
5
3%
128
# GPUs
F ADDITIONAL EXPERIMENT RESULTS
F.1 DATA USAGE IN CONTINUED PRE-TRAINING
Figure 6 illustrates the evolution of domain weights throughout the training process and the final cumulative data usage for each domain. The trajectory reveals that the domain weights stabilize after approximately 30% of the training. In contrast to other domains, Wikipedia exhibits an anomalous spike early in the training process. The remaining domains show steady, monotonic change in data loading over time, as expected.
67.0%
0.7
36,
Domain Weight
0.6
GiHbub
0.8%
1.5%
0.5
Book
8.
4.5%
Original
Dynamic Batch Loading
0.3
Wki
3.1%
4.5%
0.
Aniv
0.7%
2.55 5%
0.1
49, 19.2%
50
#Tokens for Training (B)
Data Proportion in Training (%)
Figure 6: Left: Data weight of each batch during the continued pre-training stage. Right: Cumulative data usage for each domain.
18

---

Published as a conference paper at ICLR 2024
Table 10: An example output from Sheared-LLMMA-LBBB and Sheared-LLaMA-2.755 It shows that Sheared-LLaMA can generate long, coherent, and informative responses after instruction tuning.
Prompt: Act semi conducter industry analyst
Sheared-LLaMA-1. As a semicenductor industry analyst, would be responsible for conducting market research and analysis to heip companies in the industry understand and respond to market trends, customer preferences, and competitive dynamics. This would include analyzing market data such as sales, market share, and industry trends, as well as conducting interviews with industry leaders, executives, and customers to gain insights into customer needs and preferences. I would also be responsible for developing and implementing marketing strategies to promete products and services, and for identifying and pursuing business opportunities.
As semicenductor industry analyst, my primary caspensibility is to provide insightful and accurate analysis on the global semi conductor market. This includes evaluating the performance of individual companies, industries, and trends within the market.
Sheared-LLaMA-2. 7B:
Here are some key areas where would focus analysis:
1. Market size and growth: would track the size
global
semiconductor market and analyze the growth prospects for each segment, such as memory, logic, and sensors.
2. Competitive landscape: I would assess the competitive landscape, including the top players, their market share, and strategic noves.
3. Technology trends: I would monitor the development of new technologies, such as 5G, AI, and IoI, and their impact on the semiconductor industry.
Industry trends: would track industry trends, such as the shift towards sustainable technologies, and evaluate their impact on the market.
4.
5. Market forecasts: I would provide forecasts for the semi conductuutoo market, including revenue and unit shipment growth, based on range of scenarics and assumptions.
In addition to these areas, I would also keep track of industry news and developments, and provide insights on their impact on the market.
F.2 COMPARISON LLM-PRUNER
To ensure fair comparison with the LLM-Pruner approach, we match the parameters (excluding embeddings) to be roughly the same as our final model (1.23B), as embedding sizes do not affect inference speed. We continue pre-training the pruned models derived from both LLM-Pruner and our proposed targeted structured pruning. The total number of tokens for pruning and continued pre- training controlled be the same, and data from the RedPajama dataset is used directly without applying dynamic batch loading. We demonstrate that our proposed targeted structured pruning better approach compared to L.LM-Pruner from three aspects: the loss trajectory, the model architecture, and the inference speed.
terms loss trajectory, Figure 7 shows that our proposed targeted structured pruning achieves lower loss than LLM-Pruner when consuming the same amount of data.
Table compares the model configurations for an L.M--runer pruned model and our pruned model. The LLM-Pruner model has an unconventional architecture where the intermediate size is smaller than the hidden size, largely due to the algorithm's inability to prune the hidden dimension and layers, revealing limitation of LLM--Punne..
In terms training throughput and inference speed, we find Cheared-LLaMe structures run more efficiently than LLM-Pruner models. We performed an inference speed analysis comparing LLM- pruner and Sheared-LLaMAS LLaMA's model architectures using a single A100 GPU to generate up to 2048 tokens. As shown in Table 12, pruned model architecture is significantly more efficient than
19

---

Published as a conference paper at ICLR 2024
LLM-Pruner at inference time. Additionally, LLM-Pruner's model architecture introduces substan- tial overhead during continued pretraining (Measured with 16 A 80GB GPUs.). with a training throughput of around 60% of Bheerrd-LLLMAA Overall, our Sheared-LLaMA, LLaMA architecture enables higher throughput for both inference and continued training compared to L.MM-Pruner.
In summary, have demonstrated that at the same parameter scale, pruning method produces a model that has lower perplexity (loss), a more reasonable final model architecture, and a faster inference speed. We have effectively shown our targeted structured pruning algorithm be more effective for large-scale LLM pruning compared to LLM-Pruner.
LLM-Pruner (Ma al., 2023)
Ours
2.5
2.3
2.1
1.9
#Tokens for Training (B)
Figure 7: The loss of LLM-Pruner and Sheared-LLaMA during the continued pre-training stage. Note that we exclude dynamic batch loading and use the same data distribution for training both models for a fair comparison.
Table 11: Model structure of Pythia- LLM ----rrrrer (1.6B), and Ours (I.3B).
Layers
Heads
Head size Intermediate size Hidden size Params
Pythia-I.4B
128
8192
2048
1.4B
I.LM-pruner (1.6B)
32
128
2201
4096
1.6B
Sheared-LLaMA (L.3B)
5504
2048
1.3B
Table 12: Training throughput and generation speed of LLM- (1.6B) and Sheared-LLaMe (1.3B). With a similar parameter count, our pruned model structure has a lower perplexity when fine-tuned with the same amount of tokens (around 6B tokens). Yet our pruned model architectures are way more efficient for both training and inference.
Generation Speed
Training Throughput
PPL
L-
43 tokens/s
83K tokens/s
7.09
LLaAA
58 tokens/s
139K tokens/s
6.85
F.3 CODING AND MATI REASONING
We examine the math and coding abilities of our pruned models compared to other language models. We find that the math ability of existing 3B parameter models, including Mheared-LLaMA, still far below that of larger models. We also find that Sheared-LLaMAA coding ability lags behind models known to be trained on more code data, like Pythia- 1. and Open-LLaMA-3BBvV Sheared- LLaMA's coding ability likely comes from the original LLaMA2 model, speculated to have used more code data, and the minimal code data used our pruning experiments.
F.4 SCALING REFERENCE VS. SOURCE REFERENCE
Figure This section compares the performance of Sheared-LLaaee when trained with the scaling reference and the source reference in dynamic batch loading. The scaling reference uses the pre- dicted loss from the scaling law the reference loss, while the source reference uses the loss of the
20

---

Published as a conference paper at ICLR 2024
Table 13: Evaluation results on GSM8K and HumanEval and training percentage and tokens in ArXiv and GitHub.
GSM8K (8)
EvmanEval
ArXiv Pass@5 Percentage
Github Percentage
ArXiv Tokens
GitHub Tokens
Models
EM
Pass@1
LLaMA2-7B
13.7
12.8
23.8
OPT-2.7B
0.1
0.0
0.0
Pythia-2.8B
1.7
5.1
14.6
9.0%
7.6%
26,9
22.8
INCITE-Base-3B
1.8
4.3
4.9
2%
4.5%
16.0
36.0
Open-LI.aMA-3B-v1
2.5
0.0
1.2
2%
4.5%
20.0
45.0
Open-LLaMA-3B-v22
2.7
10.4
20.1
Sheared-LLaMA-227BB (Source)
2.7
3.7
5.5
0.7%
0.4%
0.3
0.2
Sheared-LLaMA-2.7B (Scaling)
2.4
4.9
9.2
1.0%
0.8%
0.5
0.4
source model as the reference loss. Although both methods efficiently train the model, the scaling reference consistently achieves slightly better downstream performance.
Source reference
Average Downstream Apo (%)
56
Scaling reference
55
54
30 iTokeess Training
Figure 8: Average downstream peformance of Sheared-LLaMM--13B with the scaling refer- ence and the source reference.
F.5 PRUNING PYTHIA MODELS
During the initial development of the approach, we experimented with a smaller-scale model on Pythia (Biderman et al., 2023), series of open-source models with open-source training data across scales from 70M to 13B. We took the Pythia-440M model, pruned it down to 160M parameters, and continued pre-training it using Pythia models' training data Gao et al. (2020). Specifically, we used 0.4B tokens for pruning and 33B tokens (32,000 steps) for continued pre-training of the pruned model. Table 14 shows that the pruned model achieves lower perplexity than the original model, and continued pre-training further improves performance. Notably, with minimal compute consumption (10B tokens), pruning Pythia-410M model reaches roughly the same performance as pretraining Pythia- 160M from scratch. Adding more tokens further enhances the performance.
Table 14: Zero-shot performance of Pythia Mithi-1600M 160M and Sheared- Sheared-Pythia.
Training Tokens
Performance
Pythia-160M
300B
43.56
Sheared-Pythe
(300B) 10B
43.51
Sheared- -yythia
(300B) 33B
45.78
Additionally, we compared Sheared-Pythia-1600 against keeping pre-training the Pythia-160M model with the same amount tokens. From Figure 9, we can see that continuing pre-training Pythia-160M starts off performing better, however, the Cheared-Pythia-16600 160M learns faster and even- tually exceeds the performance of continuing pretraining on Pythia- 160M. These are some very preliminary results see in this particular setting.
21

---

Published as a conference paper at ICLR 2024
46.0
Average (%)
45.5
44.0
43.5
43.0
Continue pre-training Pythia-160M
Sheared-Pytheee Sheared-Pythha-1600
425
10
20
30
#Tokens Training (B)
Figure 9: The downstream performance of continued pre-training Pythia-160M and Sheared-Pythin 160M. Seeaared-yythia: 160M eventually outperforms the performance of continued pre-training Pythia-160M.
We think that the benefit of pruning a larger model will be even more significant, based on the conclusions from previous work (Li et al., 2020) showing that pruning larger than compress leads to better performance as the larger models are easier to optimize. However, we'd like to defer more detailed analysis to future work.
F.6 PRUNING FROM LLAMA1 LLAMA2
This section compares the performance of pruning from LLaMA1 and LLaMA2. Both models demonstrate strong downstream task performance, although pruning from LLaMA2 unsurprisingly yields consistent advantage. However, it is worth noting that the performance difference between the two is not very large.
Prined from LLaMA
Average Dewnstream Ape (%)
Pruned from LLaMA 2
10
20 #Tokens for Training (B)
30
Figure 10: A comparison between pruning from LLaMA1 and LLaMA2 with dynamic loading for 1.3B.
F.7 COMPARISON FURTHER CONTINUAL PRE-TRANE TRE-IIIIN INCITE- BASE-3B INCITE-BASE-3B
We examine if pruning produces better initialization for continued pre-training than an existing LLM of equivalent size by comparing the performance of continually pre-trained INCITE-Base 3B model and Sheared-LL.aMA-22. performance in Figure 12. INCITE-Base-33 model starts with higher task accuracy but plateaus after training, while Sheared-LLaMe rapidly improves and surpasses the INCITE-Base-BB model, suggesting that pruned models from strong base model serve better initialization."
We present the loss curves in Figure 11 and the downstream
In cases where the existing small model is competitive compared to the pruning source model, the small model may offer better starting point than pruned model. Intuitively, the larger the discrepancy in perfor- between the model and the small model, the advantages the pruned model has.
22

---

Published as a conference paper at ICLR 2024
56
Cont' pre-training INCITE
2.2
Cont' pro-training pruned model
Average Downstream Ace (%)
55
2.1
54
2.0
53
1.9
52
1.8
Cont' pre-training INCITE
51
pre wre-training our pruned model
1.7
10
Tokens for Training (B)
#Tokens for Training (B)
Figure 11: The loss of continued pre-training Figure 12: Average downstream performance INCITE-3B and pruned LLaMA model. of continuing pre-training Sheared-LLaMA vs Both models have around 2.7B parameters.
SCITE-Base-22
We used a learning rate le 5 for continued pre-training INCITE-Base-31 along with scheduler to warm up the learning rate to 5 in the first 3% of the training steps, and follows a cosine decay schedule. In hindsight, how continued pre-training the INCITE-Base model may not be optimal according to recent research (Gupta et al., 2023).
F.8 EXCLUDING EASY DOMAINS DURING PRUNING
During the development of this project, we explored an easy and intuitive idea to address the im- balanced loss decreasing rate during pruning and continued pre- pre-training. Specifically, we excluded GitHub, StackExchange, and ArXiv data during pruning since these three domains' losses decrease the fastest. We pruned LLaMA1-131 down to 7B using a composite dataset of C4, CC, Wiki, and Books, with a heuristically constructed proportion of 40%, 40%, 10,, 10%, respectively. We then continued pre- pre-training the pruned model the RedPajama dataset, which includes the excluded domains during pruning.
The results showed that the perplexity difference was more even across domains when pruning without using data from these three domains. However, after continued pre-training with all data from the seven domains in the RedPajama dataset, the loss disparity grew, with the GitHub difference being much smaller than domains like C4. These results demonstrate that simply excluding the domains that are easy recover during the pruning stage does not inherently resolve the imbalance of loss difference across domains.
This set of experiments motivated to develop dynamic batch loading more effective and principled approach to address the domain-specific loss disparities that arise during pruning and continued pre-training.
Table 15: Pruning LLaMAI-13B with a composite of 40% of CC, 40% of C4, 10% of Books and 10% of Wikipedia to 7B model. We present the domain loss of the source model (LLaMA1-13)) the loss of the pruned model and the loss after continued pre-training of the pruned model. The loss differentce from the target model (LLaMA1-7B) is more balanced after pruning, but more disparate after continued pre-traiee pre-training with all the domains.
GitHub
Book
StackExchange
Wikipedia
ArXiv
C4
LLaMAI-13B
1.7585
0.6673
1.9499
1.4207
1.4331
1.3855
1.8619
LLaMA1-7B
1.8366
0.7108
2.0322
1,5112
1.5291
1.4440
1.9331
Pruned model (w/o three domains)
2,1949
1.0971
2.3726
1,9080
2.1151
1.7542
2.3187
diff from LLaMA1-7B
0.3483
0,3863
0,3404
0.3968
0.5860
0.3202
0.3857
Continued Pretraining (w RP)
1.8344
0.6325
2,0984
1.4542
1.4549
1.4460
2.0395
diff from I. 1-7B
-0.0022
-0.0783
0.0661
-0.0570
-0.0743
0.0120
0.64
23

---

Published as a conference paper at ICLR 2024
F.9 INFERENCE SPEED ANALYSIS
In this section, we analyze the inference speed of different pruning approaches, including the fol- lowing models:
The source model, i.e., LLaMA2-7B.
Sheard--LLMM--L33 and
Wanda pruning (Sun et al., 2023) to prune L.LMs into semi-structured 2:4 and 4:8 sparsity pattern in one-shot.
LLM-Pruner (Ma et al., 2023), which produces model with the same number of non- embedding parameters as Sheared-LLaMA.
We use an A100 GPU test the generation speed (tokens/second) of all these pruned models. We generate up to 2048 tokens with a batch size of 1. We present the results in Table 16. Sheared- LLaMA's speed is better than that of LLM-Pruner, largely due to the more optimized resulting architecture. As shown in Table 11, LLM-LLumeee produces model structure with smaller interme- diate size than the hidden size, which goes against the transformer designs where the intermediate size is at least 3-4 times the hidden size.
Wanda-type semi- structurdd pruning also achieves inference speedup compared to the source model. However, is not fast small dense models and is less flexible because inference speedup is only feasible when the sparsity is at 50%.
Table 16: Inference speed (tokens/s) of different pruning approaches.
Model
Throughput
7B
LLaMA-7B
37
1.3B
2.7B
LLM Pruner
41
40
62
47
50% sparsity
Wanda (2:4)
42
Wanda (4:8)
42
G FREQUENTLY ASKED QUESTIONS
In this section, we provide answers to frequently asked questions about our work. Is fair say that Sheared-LLaMA models can be produced using only 50B tokens, even though the source model (LLaMAA2 was trained on 2T tokens?
At the time of our paper submission, there were no models sufficiently trained for 2T tokens the 1.3B and 2.7B scale to allow for fair comparison. However, the recently released Tiny Llaa--m--1111 models, trained on tokens, provide a suitable point of reference. We observe that the performance of Tlmy--ama-1.111 is comparable to Sheared-LLaMA-11 as base models. and a similar observation can be found in Wang et al. (2023b). Considering that finyLlama-111B is trained with 3T tokens, which exceeds the total amount of pre-training and prun- ing used by Sheared-LLaMe 1.3B (2T for pre- pre-training the source model, and 50.4B for pruning and continued training), regard this strong evidence suggesting that pruning might be intrinsi- cally more efficient and effective approach to training moderate-sizer LMs.
on downstream benchmarks when used
How is dynamic batch loading different from Doremi (Xie et al., 2023)?
Dynamic batch loading and Doremi share the same principle, which adjusts the data distribution of each domain based the model's loss using exponential ascent algorithm. However, dy-
24

---

Published as a conference paper at ICLR 2024
namic batch loading offers more flexible and less complex approach that can be applied to various scenarios.
Doremi follows multi-step process: (1) Train reference model. (2) Train proxy model to estimate the proportion of data from each domain by adjusting the proportion based on the proxy model's loss. (3) Train the final model using the estimated data distribution. In contrast, dynamic batch loading can be directly applied to any model without the need for a reference or a proxy model. Dynamic batch loading begins by deriving reference loss based on fixed evaluation set. This reference loss can be estimated using scaling laws or simply by using the source model's evaluation loss. During training, the data proportion is adjusted in real-time based on the periodically measured evaluation loss. The dynamic batch loading process can be seamlessly integrated into the standard pre-training pipeline, as evaluating the loss is computationally efficient and does not introduce significant overhead. Although dynamic batch loading relies on fixed evaluation set, which may not fully represent the model's performance on the entire dataset, this issue can be mitigated by periodically updating the evaluation set during training.
D. When multiple source model sizes are available, how do you choose the source model size for pruning?
Determining the optimal source model size for pruning is challenging. However, we can perform thought experiment by considering each parameter a uniform "unit of information." For in- stance, a source model with 7B parameters is trained using 2T tokens, we can assume that each parameter carries approximately 285 tokens of information, assuming uniform distribution of in- formation across the parameters. When randomly pruning this model down to 1.3B parameters, the total amount of information is reduced to 1.3B x 285 0.37T tokens. In contrast, if prune 13B model (also trained with 2T tokens) down to 1.3B parameters, the total amount of information reduced to 1.3B (2T 13B) = 0.2T tokens. Although this estimation rough, it suggests that pruning from larger model may be less effective, especially when the source models are trained with the same number of tokens. It is important to note that this simplified estimate, and the assumption of uniform information distribution across parameters may not hold in practice. More- over, the structured pruning process itself clearly breaks this assumption. Nonetheless, this thought experiment provides general sense of how the source model size can impact the effectiveness of pruning.
25


Reviews:
[{"review_id": "zKtytRb7s1", "reviewer": "Reviewer_wgpN", "summary": "This paper introduces to employ the structural pruning to reduce the pre-training cost of LLMs.  It focuses on the integration of two techniques: targeted structured pruning and dynamic batch loading. This paper proposes two small models: Sheared-LLaMA-1.3B and Sheared-LLaMA-2.7B. Comparative testing reveals that these lightweight models are capable of outperforming other counterparts with 1.3B and 2.7B parameters.", "questions": "1. Why the dynamic batch loading is a more efficient one than Doremi?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces two scaled-down versions of LLaMA-2, achieved through structured pruning.\n2. Experiments demonstrate that these two small LLMs exhibit superior performance when compared to other models such as OPT, Pythia, INCITE, and Open-LLaMA. \n3. The performance and capabilities of these two pruned models are extensively assessed using the Open LLM Leaderboard.", "weaknesses": "1. My main concern centers around the novelty of this paper. The first method, named targeted structured pruning, has been previously employed in several papers, with some offering further advanced variations [1, 2, 3, 4, 5]. As for the second method, dynamic batch loading, it closely resembles the application of the technique from [6]. The main observation by the authors that structured pruning can reduce training costs is a well-known advantage of all the structured pruning methods.\n\n2. The paper lacks experiments to show the effectiveness of targeted structured pruning. No experiments can be found to verify the effectiveness of the 'targeted structured pruning` compared with other pruning algorithms. Given that the proposed targeted structured pruning falls within the realm of structural pruning, a comparative analysis with LLM-Pruner[7] is essential to establish whether the newly proposed method improves upon the existing techniques for structured pruning of large language models. Utilizing the same experimental settings as LLM-Pruner for this comparison would provide a more direct and clear demonstration of the proposed method's effectiveness.\n\n3. I cannot tell whether the enhanced performance is attributed to a stronger foundational model (LLaMA2) from which pruning occurs. It's conceivable that starting with a stronger base model could lead to better results post-pruning. Have any experiments been conducted where a model, for instance, OPT-2.7B, is pruned down to 1.3B and then compared against the officially pre-trained OPT-1.3B? (Given the limited time available for rebuttal, it might not be feasible to pre-train a 1.3B model. Conducting a similar experiment with a considerably smaller model, such as a 350M version, could also provide valuable insights.)\n\n[1] Learning Sparse Neural Networks Though L0 Regularization. 2017.  \n[2] Learning Structured Sparsity in Deep Neural Networks. 2016.  \n[3] Neural Pruning via Growing Regularization. 2020.  \n[4] Fast Post-Training Pruning Framework for Transformers. 2022.  \n[5] Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers. 2020.  \n[6] DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. 2023.   \n[7] LLM-Pruner: On the Structural Pruning of Large Language Models. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XSnb9ygIEK", "reviewer": "Reviewer_qt4Y", "summary": "The manuscript introduces a technique that prunes pretrained models towards a target (smaller) architecture rather than towards a target sparsity level. After a small amount of retraining, the pruned models outperform similar-size models trained from scratch on many more tokens than the retraining budget, suggesting that the pruning approach is a more efficient path to producing a small model than training from scratch. The efficiency of retraining is enhanced by a simple and effective data selection scheme. Comparisons with other pruning methods show that \"targeted structured pruning\" (i.e., \"shearing\") can lead to models that are faster than those created by other approaches.", "questions": "Score-affecting:\n\n1. Expanding Table 4 with inference speeds for the following models could help readers better understand the importance of the Sheared-LLaMA targeted pruning approach.\n   - Before shearing\n   - Before shearing with 2:4 sparsity and with 4:8 sparsity. This would allow us to compare Sheared-LLaMA to both older and newer pruning approaches (like Wanda).\n   - LLM-Pruner model\n\nInteresting:\n\n1. To better understand the effect of the discovered pruning mask on performance during the \"continued training\" period, perhaps you could use only C4 data to find the pruning mask. If you did this, would \"continued training\" on the mixture of domains (without dynamic batch loading) still show GitHub performance reaching its reference loss and C4 performance not reaching its reference loss? Some related questions follow.\n   - Are losses being reduced at different rates across domains because the pruning mask was learned on too little data to account for the complexity of some domains (like C4)?\n   - 2 million tokens (500 sequences) are used for each held-out set: should the held-out set size be larger for more complex datasets (e.g., C4) to ensure that the pruning mask is found on a representative set of data?\n\n2. A version of Figure 4 with perplexity or loss. \n\nMinor:\n\n1. There are numerous typos. Please proofread the paper carefully. Some examples follow:\n   - \"Sheared-LLaMA-3B\" is mentioned but 2.7B is probably intended.\n   - $z^{inter}$ and $z^{int}$ are used interchangeably, so are $H^T$ and $H_T$.\n   - Table 2 seems to have NQ and LAMBADA switched in the 7B row. \n   - Section 4.1: \"hypothetical 2.7B parameter LLaMA2 model\" is stated. Do you mean 1.3B?\n   - Figure 8's caption is wrong.\n\n2. How are the Lagrange multipliers initialized? Relatedly, a reference for the specific Lagrange multiplier approach used might be nice to include -- Platt and Barr (1987) looks related but not exactly the same. \n\n3. In section 4.2, the sentence \"non-uniformity also introduces training and inference overhead due to irregularities in model architectures\" could be followed by intuition/clarification that explains why irregularities add overhead. \n\n4. In Table 4, consider adding perplexity of Sheared-LLaMA *with* continued pretraining to complement the *without* continued pretraining numbers.\n\n5. Like Kaddour et al. (2023), \"Compute-Efficient Deep Learning\" (Bartoldson et al., 2023) shows why promised gains \"may not be consistently realized\" -- it also discusses (in its survey) all of the various efficient training approaches mentioned in the submission's Related Work section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The manuscript focuses on the problem of creating a compact LLM -- existing approaches either train from scratch (which is expensive and forgoes inheriting knowledge from stronger/larger models) or prune an existing model (which can lead to a suboptimal structure for inference speed). By pruning pretrained models to a target architecture with targeted structured pruning and using a short retraining period (\"continued training\"), the submission's proposed approach is able to address the prior problems in this significant area. The resulting models are trained faster and perform better than models trained from scratch, and they perform inference faster than models pruned by other approaches.\n\nThe shearing algorithm (\"targeted structured pruning\") is very clearly explained (with nice illustrations) and is simple despite its flexibility and power. The idea of pruning towards a target architecture is (as far as I know) novel.\n\nTo enhance learning in the pruned model during \"continued training\", an original scheme to dynamically change the data mixture is developed (\"Dynamic Batch Loading\"). This scheme is not only interesting for its usefulness to Sheared-LLaMA -- as the authors suggest, it could be used to help make any model's training more efficient by avoiding usage of training data that makes relatively little progress towards the desired model performances.\n\nThe analyses (e.g., of Dynamic Batch Loading, of other pruning approaches, etc.) thoroughly support the manuscript's arguments.", "weaknesses": "While the paper is very thorough, additional inference timings for models produced by other competitive methods could help readers better understand the merits of the proposed approach (see \"Questions\" below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5WshVfX9zu", "reviewer": "Reviewer_bT2X", "summary": "This paper introduces Sheared LLaMA, a method focused on developing streamlined LLaMA models through the pruning of pre-trained ones. Central to this approach is a sparse training technique that employs L0 regularization to systematically zero out specific substructures. Additionally, the paper presents a dynamic batch-loading strategy, effectively recalibrating the significance of different data domain. In the fine-tuning phase, Sheared LLaMA undergoes training on 50 billion tokens, ultimately attaining a performance level comparable to models produced by scratch training.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This work studies a practical approach to craft lightweight LLaMA by pruning, which requires less training cost. \n* The proposed model achieves superior performance compared to publicly available models.\n* The dynamic batch loading is interesting.", "weaknesses": "My main concern lies in the technical novelty and effectiveness of the proposed pruning and sampling method. And it's unclear which part plays the most important role in pruning & fine-tuning.\n\n* Sheared LLaMA employs regularized training as a preliminary step before pruning. However, **the effectiveness of the proposed constrained optimization remains unclear**. As illustrated in Table 10, the performance gap between Sheared LLaMA and LLM-Pruner (a simple Taylor-based method) is marginal (ΔPPL=0.24). Questions arise regarding whether this improvement is attributable to the proposed pruning method or the dynamic batch loading. Additionally, it's unclear if the LLM-Pruner baseline was also trained with dynamic batch loading. \n* Based on the previous questions, this work mentioned that \"This observation indicates that pruning preserves a greater amount of knowledge in low-entropy and smaller domains (e.g., GitHub) compared to high-entropy and larger domains (e.g., C4)\". However, this phenomenon might be also caused by a biased regularization before pruning, since the regularized training step only saw 0.4B tokens as mentioned in Sec. (4.3). Maybe this problem can be avoided by sampling balanced data from different domains for regularization, or deploying a simple Taylor expansion [1, 2] with balanced data. \n* Table 4 presents some confusing results: CoFiPruning [1] shows superior Perplexity (PPL) but inferior Throughput compared to Sheared LLaMA. This raises the question: are CoFiPruning and Sheared LLaMA comparable in performance? Given these mixed results, it's difficult to conclusively state that Sheared LLaMA has outperformed CoFiPruning. \n* Might be a typo: There's an inconsistency in reporting the size of the LLM-Pruner baseline -- labeled as 1.3B in the figure caption but noted as 1.6B in the table. Clarifying these comparisons would be helpful.\n\n\n[1] Xia, M., Zhong, Z., & Chen, D. (2022). Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408.  \n[2] Molchanov, P., Mallya, A., Tyree, S., Frosio, I., & Kautz, J. (2019). Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11264-11272).   \n[3] LeCun, Y., Denker, J., & Solla, S. (1989). Optimal brain damage. Advances in neural information processing systems, 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Z9zLQctKo", "reviewer": "Reviewer_urdF", "summary": "This paper introduces a pruning technique and a dynamic batching technique to continue training pre-trained LLMs. The proposed pruning technique offers control over the final network shape by solving a constrained optimization problem, where the shape constraint must be optimized while maximizing language modeling performance. The dynamic batching technique works by first estimating reference losses for different data domains via computing scaling curves and then adjusting weighting terms for different domains based on the difference between the reference loss and validation loss at different validation intervals. The dynamic batch loading method sounds a bit heuristic, and it requires computing scaling curves which can be often expensive (otherwise the algorithm cannot work), which may need a lot of computing and training time to obtain reference losses if there are many domains. Therefore, the efficacy of this method is questionable.", "questions": "How much compute wasused to get the reference loss for each domain? Can the authors provide a breakdown and total costs?\n\nWhat happens if authors continue training the model (sheared LLaMA) beyond 50B tokens? \nThe training cost used for sheared LLaMA is as follows: 2T (7B parameter LLM) + 50B (2.7B LLM).\nIf one continues training one of the baselines, e.g., Open-LLaMA-3B-v2 for an additional 1T token, would sheared LLaMA still outperform?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and provides enough context to understand the proposed contents. Given the simplicity of the proposed pruning technique, reproducing the results seems fairly easy. Additionally, the pre-trained checkpoints of the source LLM and the training text dataset are open-sourced, which is another plus. The proposed methods could be practically useful, as there seem to be some use cases where they can help. The pruning technique with target shape constraint shows a potential that this can work on different types of neural architecture, however, there is no empirical evidence in the paper.", "weaknesses": "The proposed pruning method is sound, but it is very specific for the Transformer architecture.\n\nThe title \"ACCELERATING LANGUAGE MODEL PRE-TRAINING VIA STRUCTURED PRUNING\" is somewhat misleading, as it suggests that the paper is proposing a generic pre-training method. My understanding is that these methods only work in a limited setting. First, a competitive pre-trained checkpoint is required. The proposed methods cannot be used when a model needs to be pre-trained from scratch. However, they may be effective in certain cases, such as when an LLM needs to be compressed and trained on different data domains (not necessarily the same ones used in the original pre-training).\n\nAdditionally, the authors' claim that this method can significantly reduce training costs compared to other LLMs trained from scratch is an overstatement, e.g. saying things like the proposed variant is outperforming baselines (which are usually trained on 300B~1T tokens) by only trained on 50B tokens. The authors should count the cost of training the source LLM as the worst-case scenario. Comparing the performance with other baselines without counting the significant training computes used for the source LLMs (7B parameters and 2T tokens) is unfair. To my knowledge, the dynamic batch loading technique is required to recover the performance on the data domains after pruning. In that case, the cost to compute the reference losses should also be reported somewhere in the paper, and it should be mentioned in the main table. It is misleading to only count the tokens used for a single training run of the pruned model when there are significant prerequisites to make it work in the first place (pre-trained and competitive LLMs and performing scaling studies on different sub-datasets).\n\nIf the same methods can also be applied to another source LLM and show similar improvements over baselines, this would reinforce the reported findings. It is difficult to distinguish whether the improvement is due to the superiority of the source LLM or the proposed methods.\n\nMinor comments:\nIn Section 2.2, you wrote \" pre-training dataset RedPajama (TogetherAI, 2023b; LLaMA’s pre-training dataset)\". RedPajama is not the exact pre-training dataset used to train LLAMA2, but it's an open-source version trying to replicate the original training corpus, am I correct?\n\nIf Figure 8 is mentioned in the main paper, it should be included in the main paper, not the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)