Sparsity-Accelerated Training for Large Language Models (2024)

Abstract: Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks but often require additional training, such as continual pretraining and supervised fine-tuning. However, the costs associated with this, primarily due to their large parameter count, remain high. This paper proposes leveraging sparsity in pretrained LLMs to expedite this training process. By observing sparsity in activated neurons during forward iterations, we identify the potential for computational speed-ups by excluding inactive neurons. We address associated challenges by extending existing neuron importance evaluation metrics and introducing a ladder omission rate scheduler. Our experiments on Llama-2 demonstrate that SparsityAccelerated Training (SAT) achieves comparable or superior performance to standard training while significantly accelerating the process. Specifically, SAT achieves a 45% throughput improvement in continual pre-training and saves 38% training time in supervised finetuning in practice. It offers a simple, hardwareagnostic, and easily deployable framework for additional LLM training. Our code is available at https://github.com/OpenDFM/SAT.

heading: 1 Introduction
text: Large language models (LLMs), such as GPT4 (Achiam et al., 2023), Mistral (Jiang et al., 2023), and Llama-2 (Touvron et al., 2023), have demonstrated remarkable capabilities across numerous NLP tasks (Mao et al., 2023; Dillmann et al., 2024). In general, all these models are initially pre-trained on massive data by unsupervised learning. Furthermore, such models regularly necessitate additional training for two primary scenarios: 1) continual pre-training on new data as the pool of available pre-training data continuously expands (Gupta
*The corresponding authors are Lu Chen and Kai Yu.
et al., 2023; Cui et al., 2023; Ke et al., 2023). 2) supervised fine-tuing (SFT) on labeled data to enhance the capacity of LLMs to follow human instructions (Ouyang et al., 2022; Dong et al., 2023; Ansell et al., 2024).
In contrast to initial pre-training, the additional training demands relatively less time and computational resources. Regrettably, it remains somewhat challenging to afford, primarily attributable to the substantial parameter count (Zhao et al., 2024b; Wu et al., 2023). Fortunately, numerous practitioners (Zhao et al., 2024a) are dedicating efforts to saving such additional training costs. However, they mainly focus on the Parameter-Efficient FineTuning (PEFT) methods (Ansell et al., 2024; Lialin et al., 2023), which is not commonly employed in the aforementioned continual pre-training (Cui et al., 2023) scenario. Hence, it is imperative to explore novel acceleration methods suitable for both additional training scenes simultaneously.
In this paper, our objective is to leverage the structural sparsity (Li et al., 2023b) inherent in pre-trained LLMs to expedite their additional training scenarios. In detail, researchers (Zhang et al., 2022b; Liu et al., 2023) have observed that the
14696
activated (important) neurons1 exhibit sparsity in each forward iteration. Intuitively, we can speed up by omitting the calculation of inactive (unimportant) neurons. As depicted in Figure 1, suppose a neuron corresponds to a particular column of the weight matrix in a linear layer. In each training iteration, a compact and efficient neuron importance assessment module (right part) would evaluate the activation status of neurons in advance. By discarding those inactive neurons (columns in the weight matrix), training speed is enhanced.
Aside from the enhancement in computational speed, disregarding the computation of inactive neurons also poses the following three additional challenges: 1) determining the method of neurons to discard, 2) assessing whether omitting neurons harms the model performance, and 3) evaluating the effectiveness of the approach in real additional training scenarios of LLMs. For the first challenge, we borrowed and extended several famous neuron importance evaluation metrics (Liu et al., 2023; Han et al., 2015; Sun et al., 2023) within the context of transformer pruning, and unimportant neurons are dropped. Extensive experiments (§ 3.1) on TinyLlama-1.1B (Zhang et al., 2024) help us determine (§ 2.2) 1) using maxip (a neuron importance metric) to assess neuron importance, 2) adopting sampling (versus top-k) neuron selection strategy, and 3) introducing a ladder omission rate scheduler.
For the latter two, we separately conduct experiments over Llama-2 (7B and 13B) in both standard and sparsity-accelerated manners for both continual pre-training and supervised fine-tuning scenarios (§ 3.2). Evaluation results on several benchmarks show that SAT achieves comparable sometimes even better performance compared to normal training. Moreover, it can obtain about 45% throughput improvement for continual pre-training and 38% speedup in elapsed time for supervised fine-tuning (§ 3.3). In summary, this work contributes in the following three aspects:
1. We propose the Sparsity-Accelerated Training (SAT), a novel, hardware-agnostic, and easily deployable framework for additional training of LLMs.
2. We investigate several neuron importance computation and selection methods and summarize an optimal configuration for SAT. In
1In this paper, a neuron is associated with a specific row/column in a weight matrix.
addition, a novel ladder omission rate scheduler is designed to alleviate the overfitting problem of SAT.
3. Extensive experiments in both continual pretraining and supervised fine-tuning demonstrate that SAT achieves comparable performance to standard training of LLMs with speedup.

heading: 2 Methodology
text: In this section, we first give an overview of SAT. Next, more details are provided within the context of transformers, which is the predominant architecture of recent LLMs. Finally, we discuss our implementation and conduct a theoretical efficiency analysis for the SAT.

heading: 2.1 Overview of SAT
text: Suppose a neural network model is expressed by f(·;Nθ), where Nθ denotes the set of neurons. Recall that our objective is to disregard non-essential neurons to accelerate. Consequently, Sparse-Accelerated Training (SAT) can be delineated into the following two overarching steps for the t-th training iteration:
• Neuron Importance Computation: compute the importance scores for each neuron,
stθ = ComputeImportance ( ·,N tθ ) , (1)
where stθ ∈ R|N t θ |.
• Neuron Selction and Optimization: select important neurons upon stθ and update them,
Ñ tθ =SelectNeuron ( stθ, r,N tθ ) ,
N t+1θ = { nθ ∣∣∣nθ ∈ ( N tθ − Ñ tθ )
∨nθ ∈ Optimize ( Ñ tθ )} ,
(2)
where r ∈ [0, 1) denotes the neuron omission rate and Ñ tθ ⊆ N tθ . Actually, we optimize the subnetwork expressed by f(·; Ñ tθ) at training step t.

heading: 2.2 SAT for transformers
text: As widely recognized, contemporary LLMs are predominantly based on the Transformer (Vaswani et al., 2017) architecture. Therefore, our study primarily explores the SAT for transformers.
Review of Transformers The transformer is composed of numerous stacked layers, with each layer consisting of two main modules: MultiHead Attention (MHA) and an immediately subsequent Multi-Layer Perceptron (MLP). Formally, let X = [x1,x2, . . . ,xn] ∈ Rn×d denote the input activations of the ℓ-th layer2. The MHA can be formulated as
MHA (X) = Concat (head1, . . . , headh)WO,
headi = Ai · ( XWiV ) ,
Ai = Softmax
  ( XWiQ ) · ( XWiK )T √ dk   ,
(3) where h ∈ N+ represents the number of heads and d ∈ N+ is the dimension of hidden states. dk = d/h, WiQ,W i K and W i V ∈ Rd×dk (1 ≤ i ≤ h), and WO ∈ Rd×d are trainable parameters. Subsequently, the MLP takes the output of MHA with a residual connection as input. Mathematically,
X = X+ MHA (X) ,
MLP (X) = σ (XWup) ·Wdown, (4)
where σ (·) is the activation function, Wup ∈ Rd×4d and Wdown ∈ R4d×d are trainable parameters. The output of MLP with a residual connection is fed into the next layer3. Next, we shall proceed to a detailed discussion of the two steps of SAT within the Transformer architecture.
Neuron Importance Computation In this work, we leverage the structural sparsity of neurons across all linear layers, i.e., each row or column of the parameter matrix is conceptualized as a neuron. As our approach solely depends on the significance of columnar neurons, we exclusively focus on devising the methodology for assessing their importance. To clarify these methods, we first introduce some notations. Assume a column neuron v ∈ Rd×1 and it takes Z ∈ Rm×d as input. Zi and Zj are the i-th row and j-th column of Z, respectively. sv denotes the neuron importance score of v. The activation value of v is y = Z · v. Next, we first mathematically define four evaluation metrics (Liu et al., 2023; Han et al., 2015; Sun et al.,
2To maintain manuscript tidiness, we omit the subscript of ℓ in this paper.
3We omit the writing of layer norm layer and attention mask here.
2023) for neuron importance, which are
uniform : sv = 1,
magnitude : sv = ∥v∥2 , wanda : sv = [ ∥∥Z1 ∥∥ 2 , . . . , ∥∥∥Zd ∥∥∥ 2 ] · v,
maxip : sv =
( m∑
i=1
Zi / m ) · v.
(5) The uniform method is straightforward, as it entails randomly preserving neurons, akin to the wellknown vanilla dropout technique (Hinton et al., 2012). The other three methods are predicated on the assumption that neurons with larger activation values are relatively more important (Liu et al., 2023). To elucidate this assumption, we can consider an example. In the multi-head self-attention mechanism of the Transformer, the representation of each token is weighted by the representations of all input tokens. At this juncture, the weights (attention scores) between tokens become pivotal. These attention scores stem from the dot product of the representations (activation values) of each token. The higher the activation value, the greater the probability of a larger dot product, yielding a higher attention score, thus indicating greater importance to the token representation.
Based on this assumption, the remaining three methods posit that neurons more likely to generate higher activation values (y) are more important. In detail, the magnitude (Han et al., 2015) method solely focuses on the magnitude of the neuron parameter, i.e., the L2 norm of v. The remaining two methods consider the influence of both input and parameters on the activation values. Concretely, wanda (Sun et al., 2023) considers the magnitude of the input and parameter product, while maxip (Liu et al., 2023; Song et al., 2023) focuses on the input mean and parameter product.
Subsequently, we will delve into separate discussions regarding neuron selection for MHA and MLP according to the neuron importance score.
Neuron Selection for MHA For MHA, we operate at the granularity of individual heads, where we either omit or retain all neurons within a head. For the i-th head, each of its three linear parameter matrices WiQ, W i K, W i V ∈ Rd×dk includes dk neurons, and the neuron importance score of the j-th neuron in each matrix is denoted as si,jQ , s i,j K , si,jV respectively. We first calculate the importance
score of the i-th head in each parameter matrix via
sheadiQ =
dk∑
j=1
si,jQ dk , sheadiK = dk∑
j=1
si,jK dk , sheadiV =
dk∑
j=1
si,jV dk .
(6) Then, the importance score of i-th head is aggregated, as depicted in Figure 2 (①) by
sheadi = ( sheadiQ + s headi K + s headi V )/ 3. (7)
Subsequently, h̃ = ⌊h× (1− r)⌋ heads will be selected to optimize based on the head importance score in the following two ways:
• top-k: retain the top h̃ heads with the highest importance scores.
• sampling: sample h̃ heads according to the distribution:
headi ∼ p (headi) ,
p (headi) = exp ( sheadi/τ ) ∑
j exp ( sheadj/τ
) , (8)
where 1 ≤ i ≤ h and τ ∈ R+ is the temperature for sampling (Dupont et al., 2022).
As is shown in Figure 2 (②), only selected h̃ heads participate in the computation. Moreover, only the rows corresponding to the selected heads in
WO (referred to as W̃O) will be involved in the computation. Assuming heads (π1, π2, . . . , πh̃) are selected, the formula (3) becomes
MHA (X) = Concat ( headπ1 , . . . , headπh̃ ) W̃O.
(9)
Neuron Selection for MLP For MLP, we operate at the granularity of individual channels, where each channel (column) in Wup ∈ Rd×4d corresponds to a neuron and we select c̃ = ⌊4d · (1− r)⌋ neurons (denoted as W̃up ∈ Rd×c̃) to optimize similarly to MHA (see Figure 2 (③)). Likewise, rows corresponding to the selected channels in Wdown ∈ R4d×d (denoted as W̃down ∈ Rc̃×d) will participate in the computation as illustrated in Figure 2 (④). Then, the formula (4) turns into
MLP (X) = σ ( XW̃up ) · W̃down. (10)
Ladder Omission Rate Scheduler (LORS) Empirically, pruning some neurons may pose a risk of overfitting to the model. To mitigate this potential issue, we plan to divide the training process into two stages: 1) Train sparsely with a constant omission rate and 2) Gradually decrease the omission rate, making the model denser until it fully recovers as a dense model. Through the second stage, the overfitting problem of certain parameters in the model is alleviated to some extent.
For the second stage, we experimented with linear and cosine omission rate schedulers. In terms of model performance, it is common to employ these schedulers during training. However, due to the frequent fluctuations in omission rate, there is considerable variability in training speed, occasionally resulting in slower progress compared to training without neuron omission. Therefore, we considered reducing the frequency of changes in the omission rate by maintaining a constant rate for a short period before decreasing to the next constant rate segment. Additionally, similar to curriculum learning (Bengio et al., 2009), we gradually lengthen the period to allow the model to adapt to the changes in the omission rate.
Based on these insights, we propose a Ladder Omission Rate Scheduler. Figure 3 (c) visualizes an example of LORS. Mathematically, considering a maximum neuron omission rate r and total training steps T , supposing the neuron omission rate initiates its decrease at step η with a ladder number denoted as L, the neuron omission rate rt at step t can be formulated as follows:
rt =    r, t ≤ η max { 0, r − r
L · ℓt } , t > η , where
2ℓt−1 − 1 2L − 1 ≤ t− η T − η < 2ℓt − 1 2L − 1 and ℓt ∈ N +.
(11)

heading: 2.3 Implementation and Theoretical Efficiency Analysis
text: Implementation Taking into account that we are accelerating the training of LLMs at the algorithmic level, it is imperative for SAT to be compatible with lower-level acceleration techniques, such as operator-level acceleration (e.g., flash attention (Dao et al., 2022; Dao, 2023)) and training frameworks like DeepSpeed (Rasley et al., 2020). To accomplish this, we employ an exceedingly simple and convenient implementation approach: for a linear layer, during each training iteration, the optimizer remains consistent with standard training, preserving a complete parameter matrix. Nevertheless, during computation, we exclusively engage the sparse portions (several columns or rows) of the parameter matrix for calculation.
Efficiency Analysis In theory, we compare SAT and standard training in terms of both memory consumption and FLOPs (floating point of operations).
• Memory: Given that the primary memoryintensive components on the GPU, namely the model parameters and optimizer states, remain consistent with those of standard training, the actual memory demand of our SAT during training theoretically should align closely with that of standard training.
• FLOPs: During both the forward and backward processes, we omit the computation of r proportion of neurons. Consequently, by disregarding the calculation overhead of the second stage of LORS (mentioned in subsection 2.2), SAT can save approximately ηrT FLOPs.

heading: 3 Experiments
text: We conduct comprehensive experiments to address the following three questions:
Q1: What is the optimal configuration, including assessment of neurons importance (across various metrics) and neurons selection (top-k vs. sampling), with LORS for SAT?
Q2: Does SAT harm the performance of LLMs in both continual pre-training and supervised fine-tuning scenarios?
Q3: What is the efficiency achieved by SAT over popular LLMs in practice?

heading: 3.1 Neuron importance assessment and omission
text: We first conduct experiments over relatively small models to answer Q1 due to limitations in computing resources and time.
Setup We initially continue pre-training TinyLlama-1.1B (Zhang et al., 2024) over about 12B tokens of Chinese data to augment its capabilities for understanding and generating Chinese text. Inspired by Cui et al., we extend its vocabulary with an additional 33000 Chinese tokens. Following Wei et al., we validate the improvement of the Chinese language processing capability of models by evaluating perplexity over Skywork (Wei et al., 2023), which is a Chinese language modeling evaluation benchmark across 6 domains. For neuron importance assessment, we try the four aforementioned metrics: uniform, magnitude (Han et al., 2015), wanda (Sun et al., 2023) and maxip (Liu et al., 2023). For neuron selection methods, we explore both top-k and
sampling. All results are obtained by the popular LM-Evaluation-Harness (Gao et al., 2023) tool.
Hyperparameters We continue pre-training TinyLlama-1.1B by Megatron-DeepSpeed (Smith et al., 2022) framework on 32 A800-80G GPUs with Zero-3 (Rajbhandari et al., 2020) and FlashAttention (Dao et al., 2022; Dao, 2023) techniques. The batch size is 1024 and the maximum length is 4096. We adopt a cosine learning rate scheduler with a maximum learning rate of 5e-5. For the ladder omission rate scheduler, T = 3000, L = 5, and η = 2000. For the sampling neuron selection method, we explore three different temperature settings: [0.1, 0.05, 0.01]. The results are shown in Table 5 of Appendix A. Finally, we set τ = 0.05 in all our experiments.
Results First, from Table 1 (line 4-6 vs. 7-9), we discover the neuron selection method of sam-
pling beats top-k under all neuron importance metrics (Q1). We suspect the top-k method tends to concentrate on selecting relatively fixed neurons which causes overfitting. Then, Figure 3 demonstrates the continual pre-training loss of TinyLlama1.1B. From Figure 3(a), we can find the lowest losses achieved by all neuron importance metrics are higher than those of full pre-training due to the lack of half-trainable parameters in each iteration. Among all neuron importance metrics, wanda obtains the lowest training loss, while not as smooth as uniform and maxip during early stages (about 300-500 iterations). Considering the language modeling evaluation results of Table 1 (line 3, 7-9), the uniform and maxip equipped with sampling neuron selection method may be reasonable SAT configuration candidates.
Subsequently, we introduce the ladder omission rate scheduler (see Figure 3(c)). We can observe
the training loss decreases rapidly to a level comparable to full pre-training for both configuration candidates. However, the maxip metric obtains a faster rate of loss reduction. Combining with the results in Table 1 (line 10-11), we conclude maxip neuron importance metric with sampling neuron selection method, plus the LORS are optimal configurations for SAT (Q1). Furthermore, SAT achieves comparable perplexity to full pre-training with such optimal configuration.

heading: 3.2 Performance effects of SAT in both scenarios
text: Next, we validate whether SAT with the optimal configuration works in both continual pre-training and supervised fine-tuning scenarios (Q2).
Setup To better simulate real-world application scenarios, we perform experiments on larger LLMs (7B and 13B). For the continual pretraining scenario, we adopt the identical pretraining setup to TinyLlama-1.1B. Besides the language modeling evaluation on Skywork, we evaluate LLMs on two additional popular Chinese benchmarks: CMMLU (Li et al., 2023a) and AGI-Eval (Zhong et al., 2023)4. For the supervised fine-tuning scenario, we instruction-tune LLMs on 50K subsample of Flan v2 (Longpre et al., 2023; Wang et al., 2023). Following Ivison et al., we assess the instruction-tuned LLMs on GPT4ALL (Anand et al., 2023) for reasoning ability and MMLU (Hendrycks et al., 2020) for factuality ability.
Hyperparameters For the continual pre-training, all pre-training hyperparameters are the same as those of TinyLlama-1.1B. We evaluate models on CMMLU and AGI-Eval in a 3-shot manner.
4We only evaluate on single-choice problems.
For the supervised fine-tuning, we adopt the Huggingface+DeepSpeed (Wolf et al., 2019; Rasley et al., 2020) framework with Zero-3 technique. We instruction-tune models 3 epochs with a cosine learning rate scheduler whose maximum learning rate is 1e-5 on 64 A800-80G GPUs. The batch size is 128. Similar to PEFT methods (Hu et al., 2022; Liu et al., 2022), we only tune a small proportion of parameters at each training step. In particular, we set r = 96%, T = 1200, η = 600, and L = 1. The models are evaluated in a 5-shot manner for both GPT4ALL and MMLU. More details of baselines are in Appendix B.
Results Table 2 shows the results of continual pre-training. First, the continual pre-training enhances the Chinese text-processing ability of LLMs (line 1 vs. 2; line 5 vs. 6). The models perform comparably with SAT to full pre-training (line 2 vs. 4; line 6 vs. 8). Furthermore, we adjusted to adapt SPDF (Thangarasa et al., 2023) for continual pre-training. Specifically, we conducted sparse pre-training first, followed by dense continual pretraining with the model. We can find our SAT consistently outperforms SPDF (line 3 vs. 4; line 7 vs. 8). We can state the SAT harms little to the performance of LLMs in the continual pre-training scenario (Q2).
Table 3 illustrates the results of supervised finetuning. The SFT boosts the performance of Llama2 7B a lot (line 1 vs. 2) while contributing minor improvements to Llama-2 13B (line 7 vs. 8). We attribute this to the fact that Llama-2 13B, with its greater number of parameters, has a relatively strong inherent capacity before SFT (line 1, 2, and 7). Maybe it requires more SFT data to achieve further performance improvements. Compared to famous PEFT methods, (IA)3, SpIEL, and LoRA, our SAT obtains better performance on GPT4ALL
and comparable results on MMLU. Furthermore, our SAT achieves comparable results on MMLU and even better accuracy on GPT4ALL compared to FullFT. We suspect that this may be due to the removal of some unnecessary neurons, which alleviates the risk of overfitting. Overall, the SAT has almost no bad effects on the model performance (Q2).
After confirming that SAT has little to no detrimental effect on model performance, we finally examine the actual acceleration it can achieve (Q3).

heading: 3.3 Efficiency of SAT
text: Setup We compare the speedup of SAT with standard training in both continual pre-training and supervised fine-tuning scenarios. All experiments are performed on identical 4 machines, each equipped with 8 A800 GPUs under the setup described in § 3.1 and § 3.2. For the continual pre-training, we compare the number of processed tokens per second since all training samples are truncated or padded to the same length. For the supervised finetuning, we compare the total elapsed training time due to the variable lengths of training samples.
Results Figure 4 provides the speedup results for both continual pre-training and supervised finetuning. For the continual pre-training, we discover SAT improves throughput increases with the growth of model sizes (the dashed line in Figure 4(a)) and the maximum throughput improvement approaches 45%. For the supervised fine-tuning, the speedup of LoRA decreases significantly when the model size is relatively large. However, our SAT is more stable. In general, SAT can obtain
about 38% speedup in elapsed time. Table 4 presents the results in terms of theoretical FLOPs, actual FLOPS, training time, and memory usage. The FLOPS decrease is attributed to the reduced computation required for each iteration. As we can see, SAT achieved the optimal acceleration effect; however, there was no improvement in memory usage, which is consistent with the theoretical analysis and experimental results in the previous discussion. Finally, we can determine SAT indeed accelerates both continual pre-training and supervised fine-tuning (Q3).

heading: 4 Related Works
text: Transformer Pruning Pruning has demonstrated considerable potential in augmenting the inference speed and reducing the memory footprint of transformers, as highlighted in prior research (Kwon et al., 2022). Broadly speaking, transformer pruning techniques can be delineated into two principal categories (Chitty-Venkata et al., 2023): unstructured pruning and structured pruning. Unstructured pruning methods (Gordon et al., 2020; Campos et al., 2022; Zhang et al., 2022a) possess the capability to substantially diminish the number of parameters due to their fine-grained pruning granularity. Nonetheless, they frequently encounter challenges in achieving significant improvements in inference speed across diverse hardware platforms (Sanh et al., 2020).
Hence, numerous researchers are embracing structured pruning methods at the granularity level of entire layers, filters, channels, or heads (Li et al., 2021). For instance, Liu et al. prunes the unimportant heads within multi-head attention layers and insignificant channels in feed-forward networks on BERT (Kenton and Toutanova, 2019). Liu et al. and Song et al. extend such a method and scale it for large language models. In this study, distinct from their focus on the inference of large language models (LLMs), we draw upon the principles of structured pruning and apply them to the training of LLMs.
Sparse Fine-Tuning Sparse fine-tuning is a technique aimed at expediting model fine-tuning and diminishing model memory demands by selectively updating only a small subset of parameters (Ansell et al., 2022; Li et al., 2024). For relatively small language models, such as BERT, Guo et al. devise a binary mask controlled by a regularization term to manage the parameters for updating. Sung et al.
induce the mask by retaining parameters exhibiting higher Fisher information over numerous iterations and subsequently maintain the mask to sparsely fine-tune models. Additionally, Ansell et al. retain parameters that undergo the most significant changes during an initial round of full fine-tuning.
With the widespread adoption of LLMs, Ansell et al. and Zhao et al. commence scaling sparse fine-tuning to LLMs. However, to mitigate the memory overhead of LLMs, they both incorporate the parameter-efficient fine-tuning (PEFT) methods (Houlsby et al., 2019; Li and Liang, 2021; Lialin et al., 2023), which may not always be entirely suitable (Cui et al., 2023; Zhao et al., 2024b). In this work, instead of employing the PEFT methods, we place greater emphasis on conventional sparse training techniques, making it applicable to common scenarios such as continual pre-training and supervised fine-tuning for LLMs.

heading: 5 Conclusion
text: In this paper, we aim to accelerate two regular additional training scenarios of LLMs, i.e., continual pre-training and supervised fine-tuning. Our main intuition is to leverage the sparsity of neurons in large models, as discovered by previous researchers, to accelerate training by disregarding computations for unimportant neurons and propose a sparsity-accelerated training (SAT) framework. Extensive experiments on several benchmarks demonstrate SAT can accelerate the training of LLMs and maintain performance simultaneously.
Limitations
On one hand, this work primarily focuses on large language models and can explore even larger and more diverse model structures in the future. Furthermore, several other neuron importance metrics are necessary to explore. On the other hand, training numerous large-scale models requires substantial computational resources, incurring high costs and resulting in significant carbon emissions.

heading: Acknowledgements
text: We thank all the reviewers for their valuable suggestions on this work. This work is funded by the China NSFC Projects (92370206, U23B2057,62106142 and 62120106006) and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).

heading: B Baselines Hyperparameters in Supervised Fine-tuning

heading: A Temperature Exploration

heading: Future Work
text: Limitations
On one hand, this work primarily focuses on large language models and can explore even larger and more diverse model structures in the future. Furthermore, several other neuron importance metrics are necessary to explore. On the other hand, training numerous large-scale models requires substantial computational resources, incurring high costs and resulting in significant carbon emissions