Title: DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Training Acceleration

Memory Efficient Fine-Tuning

Large Language Models

Backpropagation Optimization.
Abstract: Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5$\times$, and enable training with a sequence length 6.2$\times$ larger on a single NVIDIA-A100 GPU. Furthermore, our DropBP enabled a throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at [https://github.com/WooSunghyeon/dropbp](https://github.com/WooSunghyeon/dropbp).

DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation

Abstract

Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5 × , and enable training with a sequence length 6.2 × larger on a single NVIDIA-A100 GPU. Furthermore, our DropBP enabled a throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at https://github.com/WooSunghyeon/dropbp.

1 Introduction

Since the advent of the transformer architecture [1], the field of language modelling has experienced dramatic advancements. Especially, following the scaling laws [2, 3], the development of Large Language Models (LLMs) [4, 5, 6, 7, 8, 9] continues with the aim of achieving or outperforming human capabilities. However, tremendously increasing the size of the model results in significant costs for training from scratch. An alternative approach for developing high-capability LLMs without the costly pretraining on extensive datasets is instruction tuning [10, 11, 12, 13]. This method fine-tunes well-trained foundation models on relatively small instruction-following datasets, enabling the models to better understand and follow prompts.

While fine-tuning Large Language Models (LLMs) on instruction-following datasets is more costeffective than training from scratch, it still requires substantial memory for parameters and activations, along with significant floating-point operations (FLOPs). In this context, Parameter-Efficient FineTuning (PEFT) techniques [14, 15, 16] effectively reduce the memory required for parameter gradients and optimizer states by freezing pretrained weights and selectively training newly added modules. Moreover, when combined with quantization methods [17, 18, 19, 20], these techniques can further significantly decrease the memory requirements for parameters.

ForwardPropagation

BackwardPropagation

300

Available max sequence length

(sw)

per sample(

250

200

57%

48%

6.2x

191.9

153.1

2K

time

150

3.7K

23.2

Training t

100

29.6

1K

2.0K

50

90.8

91.1

117.7

117.5

1.2K

0.6K

0

Full-FT Full-FT+DropBP

LoRA

LoRA+DropBP

QLoRA

QLoRA

QLoRA

QLoRA

+

+

+

(p=0.875)

(p=0.875)

DropBP

DropBP

DropBP

(p=0.5)

(p=0.75)

(p=0.875)

(a) Training time per sample of LLaMA2-7B.

(b) Available max sequence length of LLaMA2-70B.

While Parameter-Efficient Fine-Tuning (PEFT) has successfully reduced memory associated with parameters, two significant challenges remain for efficient fine-tuning: computational cost and activation memory, both of which are linked to the backpropagation [21]. First, fine-tuning Large Language Models (LLMs) using a backpropagation requires substantial floating-point operations (FLOPs). Specifically, the backpropagation algorithm necessitates forward propagation to calculate outputs and backward propagation to compute gradients for inputs and parameters. Notably, backward propagation demands twice the computational operations compared to forward propagation, thus becoming the primary bottleneck. Second, all intermediate outputs (i.e., activations) generated during forward propagation must be stored for compute in backward propagation. This activations consume considerable memory, which becomes especially critical when training LLMs on long sequence contexts [22, 23].

In this paper, we introduce Dropping Backward Propagation (DropBP), an efficient fine-tuning algorithm for LLMs that significantly reduces computational costs and activation memory. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. As a result, these undropped layers no longer require FLOPs and activation memory during backward propagation. Additionally, DropBP calculates the sensitivity of each layer, an indicator of its impact on the total training process, to adjust drop rate. This careful calibration of drop rate according to layer sensitivity ensures more stable training. This DropBP algorithm can be seamlessly integrated with any PEFT algorithm, operating orthogonally by simply dropping layers during backward propagation.

We implemented DropBP as an easy-to-integrate PyTorch library [24], requiring only minimal changes to the existing training codes. In experiments, DropBP successfully reduces training time as shown in Fig. 1a, maintaining comparable accuracy on the MMLU [25] and commonsense reasoning tasks [26, 27, 28, 29, 30]. The DropBP also accelerated the convergence of the same perplexity by 1.5 × in LLaMA2-70B [8]. Moreover, DropBP substantially decreases activation memory, increasing an available maximum sequence length by up to 6.2 × in LLaMA2-70B on a single NVIDIA A100 GPU [31], as shown in Fig. 1b. Finally, our DropBP increases training throughput by up to 79% and 117% on a single NVIDIA A100 GPU and Intel Gaudi2 HPU [32], respectively, when fully fine-tuning LLaMA3-8B [9]. In summary, the main contributions of our paper are:

We propose DropBP, an efficient fine-tuning algorithm that randomly drops backward propagation based on layer sensitivity.

We implemented DropBP as a user-friendly PyTorch extension with a straightforward API for ease of use, making it easily applicable to existing training codes.

DropBP reduces training time by 44% with comparable accuracy, increases convergence speed by 1.5 × , increases the available maximum sequence length up to 6.2 × , and enhances training throughput up to 117%.

2 Background & Motivation

Backpropagation Backpropagation [33], a core algorithm for training deep neural networks, involves both forward and backward propagation. Specifically, the training process in the linear layer is represented as follows:

where H and W represent the activations and parameters, respectively, with ' × ' indicating matrix multiplication operation. The gradients of H and W are denoted by ∇ H and ∇ W . The computational costs during forward propagation primarily arises from matrix multiplication for computing output activations by Eq. 1. In backward propagation, the computational burden is primarily due to matrix multiplication for calculating input gradients by Eq. 2 and parameter gradients by Eq. 3. Note that the computational costs of these equations are almost equal. Consequently, the FLOPs required for backward propagation including Eqs. 2 and 3 are approximately 2 × as large as the FLOPs needed for forward propagation by Eq. 1. Furthermore, the activations of all layers ( H T in ) must be stored in memory for use in backward propagation computations in Eq. 3. Therefore, focusing on reducing the computations during backward propagation is crucial for decreasing both the overall computational costs and the activation memory.

Interpretation the model with residual connections Residual connections are one of the widely used methods to address the issue of vanishing gradients [34]. Transformer [1] also incorporates residual connections that bypass multi-head attention and feedforward networks. Networks utilizing these residual connections can be interpreted as ensembles of several submodules [35]. For example, if we expand the model with three residual connections as shown in Fig. 2a, it can be represented as a combination of eight submodules, as depicted in Fig. 2b From this perspective, a network with n layers can be interpreted as an ensemble of 2 n submodules [35].

F1

F2

F3

(a) 3-layers model with residual connections.

F1

F1

F2

F1

F1

F2

F3

(b) Combination of multiple submodules.

3 Methodology

3.1 Dropping Backward propagation

In Section 2, we observed that the backpropagation algorithm consumes a significant amount of FLOPs and activation memory, particularly for the backward propagation. To reduce this overhead,

(a) The concept of DropBP.

(b) DropBP as combiation of shallow submodules.

ForwardPropagation

BackwardPropagation

Xout

Xout

No drop

on/off

p=0.8

F

F2

F3

Position-wise

Position-wise

MLP

MLP

Layer Norm

Layer Norm

F1

No drop

on/off

p=0.3

F1

F2

Multi-Head

Attention

Multi-Head

Attention

F1

Layer Norm

Layer Norm

F1

F2

F3

u!x

we propose a straightforward approach: Dropping Backward Propagation (DropBP). DropBP is designed to drop layers exclusively during backward propagation, effectively reducing both FLOPs and activation memory for the dropped layers, as demonstrated in following equations:

Here, X in , X out , and X imm represent the input, output, and immediate activation between the attention layer and feedforward network in a transformer block, respectively. f ATTN , f FFN , and f LN denote the attention layer, feedforward network, and layer normalization of the transformer block. The DropBP layer, defined as D p ( X ) , skips backward propagation in the input X with a probability of p , while not dropping forward propagation. Following to Eqs. 4 and 5, backward propagation in the attention layer and feedforward network is dropped with probabilities p i and p i +1 , respectively, as shown in Fig. 3a.

We can view the transformer as a collection of a lot of submodules composed of various modules (i.e., f ATTN ◦ f LN and f FFN ◦ f LN ) with residual connections, as described in Section 2. When the transformer block contains n units, each including multi-head attention and a feedforward network, the model can be interpreted as an ensemble of 2 2 n submodules. From this perspective, DropBP can be interpreted as training only certain shallow submodules. For example, as shown in Fig. 3b, if the F 2 layer is dropped, only the shallow submodule composed of the remaining layers is trained during backward propagation. Therefore, if the overall drop rate is p , DropBP can be interpreted as training shallow submodules with the depth of 2 n (1 -p ) or less, since 2 np layers are dropped. We anticipate that training these smaller modules alone can effectively fine-tune well-pretrained LLMs, based on the analysis that shallow submodules have a significant impact on the overall training process as detailed in Appendix A.

3.2 Sensitivity-based Drop Rate Allocation

In Section 3.1, we introduce DropBP, which selectively drops layers during backward propagation and trains only certain shallow submodules. In addition, we hypothesized that the significance of individual layers and the submodules encompassing these layers varies in their impact on the overall training process. Therefore, we assign different drop rate to each layer based on sensitivity, which is defined by defined by how much each layer and its encompassing submodules affect the overall training process in terms of parameter gradient. Specifically, we calculate the sensitivity of a layer by the variance in L2-norm of parameter gradients between when the backward propagation of that layer

is skipped or not, inspired by GradNormVar [36], a memory efficient gradient variance approximation, as below:

where S l denotes of the l -th layer. Here, ∇ W i represnets the parameter gradient of the i -th layer when no layers are dropped, while ∇ W ( l ) i denotes the parameter gradient of the i -th layer when the l -th layer is dropped during backward propagation. After calculating the sensitivity for each layer, we aim to minimize the expected sensitivities across all layers-essentially the expected gradient variance caused by DropBP-under a given FLOPs as follow:

where p i denotes the drop rate, and F i denotes the FLOPs of the i -th layer during backward propagation. F t represents the target FLOPs, derived from the target average drop rate p avg (i.e. F t = (1 -p avg ) ∑ i F i ). In other words, we determine the drop rates across all layers that minimize the expected sensitivities of the model, while satisfying a given FLOPs budget, by solving Problem 7. In practice, DropBP addresses the Prob. 7 using a simple greedy algorithm, as detailed in Section 4.1.

4 Evaluation

4.1 Implementation and Settings

(a) Code for preparing a model with DropBP.

DropBP aims to decrease the training costs in fine-tuning based on backpropagation, consequently enabling the acceleration of both full fine-tuning and parameter-efficient fine-tuning using backpropagation. To facilitate practical implementation, we developed a user-friendly DropBP library in PyTorch [24], as demonstrated in Fig. 4. In detail, we implemented a DropBP layer that internally drops backward propagation in the input direction according to a given drop rate as shown in Fig. 4a. The DropBP layer designed to initially receive the FLOPs of the layers that would be skipped ( F i ) as initial value to solve Prob. 7.

Additionally, we developed a DropBPHandler that automatically solve Prob. 7 by assigning varying drop rate to each layer using a simple greedy algorithm, as demonstrated in Fig. 4b. Specifically, the process begins by setting the drop rate ( p i ) of all layers to 0 and then gradually increases them to

align with the target average drop rate ( p avg ) set by the user. In each step, the drop rate for each layer is incremented by 0.1, ensuring that the increase in total expected sensitivity is kept to a minimum. We train with uniform drop rate for the initial 10% of total iterations and then adjust the drop rate for each layer based on sensitivity when training is stable. Since sensitivity is calculated only once at the 10% of the entire iteration, the overhead from this calculation is negligible.

We integrated our DropBP code into LitGPT [37] and HuggingFace [38], repositories for training and evaluating LLMs. We first fine-tuned LLaMA2-7B, 13B, and 70B [8] on Alpaca [11] and Dolly [13] datasets. The fine-tuned models were evaluated on 5-shot Massive Multitask Language Understanding (MMLU) tasks [25] and 0-shot commonsense reasoning tasks [26, 27, 28, 29, 30] using lm-evaluation-harness [39] library. We also fine-tunes LLaMA3-8B [9] on the Oasst1 dataset [40] and evaluating the model on MT-Bench task [41] using GPT4o-mini [5] as a judge. These experiments were conducted on a single NVIDIA A100 GPU, and we measured the training time, memory usage, and convergence speed. More details about setup can be found in Appendix F.

4.2 Main Results: Accuracy and Efficiency

Accuracy on MMLU and Commonsense Reasoning We employ DropBP to accelerate baseline fine-tuning processes, including full fine-tuning (Full-FT), LoRA [14], and QLoRA [18], on the

Alpaca and Dolly datasets. As demonstrated in Table 1, DropBP achieves accuracy comparable to the baseline, with deviations less than 1% in all scenarios, and it even outperforms the baseline in several instances. Specifically, when DropBP is applied to fine-tune LLaMA2-7B, there is a 1% or less decrease in 5-shot MMLU accuracy compared to the baseline, while maintaining comparable 0-shot commonsense reasoning accuracy. In contrast, for LLaMA2-13B and LLaMA2-70B, fine-tuning with DropBP results in almost no decrease in accuracy on the MMLU and commonsense reasoning tasks, even at the high drop rate of 0.875.

Accuracy on MT-Bench Similar trends are observed in the MT-Bench tasks, with negligible decreases in accuracy as shown in Table 2. Specifically, when fine-tuning LLaMA3-8B on the Oasst1 dataset, DropBP generates responses of comparable quality to the baseline across various generation tasks. Although scores slightly decrease as the DropBP rate increases, the model fine-tuned with a high DropBP rate of 0.875 still achieves significantly higher scores compared to non-tuned models.

Training Speed and Memory Usage Wemeasured the fine-tuning time required to obtain the results in Table 1, as presented in Table 3. When using DropBP to LoRA or QLoRA, training time is reduced by 25%, 38%, and 44% at drop rates of 0.5, 0.75, and 0.875, respectively. In contrast, using DropBP to Full-FT resulted in even higher training time reductions of 33%, 50%, and 57% at the same drop rates. These findings align with the theoretical reduction in FLOPs due to DropBP, as detailed in Appendix B. We also confirmed that DropBP can significantly reduce memory usage during fine-tuning, as shown in Table 2. Specifically, while not using DropBP results in a memory consumption of 57GB, applying DropBP with a drop rate of 0.875 reduces memory usage to 32GB by eliminating the storage of activation memory for dropped layers. Additionally, we evaluated the convergence speed to reach the same validation perplexity (PPL) on downstream tasks, as illustrated in Fig. 5 and Fig. 10-11 in Appendix C. The results indicate that our DropBP increases training speed by up to 1.5 × compared to the baseline in LLaMA2-70B.

4.3 Usability of DropBP

In this section, we evaluate the usability of DropBP, including its ability to train on long sequence data and its training throughput in constrained environments, such as a single NVIDIA A100 GPU or Intel Gaudi2 HPU.

Baseline

7

DropBP (p=0.5)

DropBP(p=0.75)

6

DropBP(p=0.875)

5

PPL

4

3

2

个

1.50 x speed up

0

1

2

3

4

5

Training time (Hours)

We first measured the maximum sequence length that could be trained without an Out Of Memory (OOM) on a single NVIDIA A100 GPU. The results in Table 4 indicate that our DropBP considerably increases the maximum sequence length, by up to 6.2 × the baseline when the drop rate was 0.875. This is because DropBP allows skipping certain layers during backward propagation, eliminating the need to store activations required for calculating parameter gradients of those skipped layers. We believe that this property of DropBP will be particularly useful for fine-tuning LLMs with long-context data [22, 23].

(a) Throughput in a single A100 GPU.

(b) Throughput in a single Gaudi2 HPU.

30

16

OOM

OOM

Throughput(sentence/s)

14

Throughput(sentence/s)

25

12

20

10

15

8

=

Baseline

10

Baseline

DropBP (p=0.5)

DropBP (p=0.5)

DropBP (p=0.75)

DropBP (p=0.75)

6

DropBP(p=0.875)

5

DropBP (p=0.875)

0

10

20

30

40

50

0

10

20

30

40

50

60

BatchSizeperIteration

BatchSizeperIteration

We also evaluated training throughput when full fine-tuning LLaMA3-8B using BF16 precision on a single NVIDIA A100 GPU and an Intel Gaudi2 HPU, increasing the batch size up to the point of OOMerrors. As shown in Fig. 6, applying DropBP allows for an increase in batch size per iteration by up to 3.3 × on the NVIDIA A100 GPU and 5.2 × on the Intel Gaudi2 HPU, ensuring high hardware utilization and scalability. Furthermore, DropBP demonstrates a sustained increase in throughput over the baseline at an identical batch size. Ultimately, with a drop rate of 0.875, DropBP achieves a throughput of 16.4 sentences/s on the NVIDIA A100 GPU and 28.4 sentences/s on the Intel Gaudi2 HPU, increasing by 79% and 117% over the baseline, respectively.

4.4 Ablation Study

Impact of the Number of Submodules We conducted an ablation study to investigate the impact of the number of trainable submodules on the finetuning of LLMs. This study compared DropBP, which trains varying submodules randomly at each iteration, with layer freezing, which trains submodules composed of only upper layers. Here, the skip rate p denotes the drop rate in DropBP and the proportion of layers that are frozen in the layer freezing.

First, we analyzed the number of submodules trained by layer freezing and DropBP. In the case of layer freezing, the lower 2 np layers are frozen and only the remaining 2 n (1 -p ) layers are trained. In this case, the number of trainable upper submodules is 2 2 n (1 -p ) . In contrast, DropBP randomly drops 2 np layers at each iteration, allowing it to train all submodules with a depth of 2 n (1 -p ) or less without the restriction of training only the submodules composed of the upper layers. In this scenario, since the

-o-

Baseline

8

Freeze (p=0.5)

Freeze (p=0.75)

7

Freeze (p=0.875)

DropBP (p=0.5)

6

DropBP (p=0.75)

5

DropBP(p=0.875)

4

3

2

0.4×1017

0.8×1017

1.2 × 1017

FLOPS

number of different submodules at depth i in the entire network is 2 n C i , DropBP can train ∑ 2 n (1 -p ) i =0 2 n C i submodules.

As shown in Table 5, when fine-tuning LLaMA2-7B using layer freezing or DropBP with a high skip rate of 0.875, we observed a significant 1.8% decrease in accuracy with layer freezing compared to

the baseline, while DropBP exhibited a relatively smaller accuracy decrease of 1.0%. Furthermore, as illustrated in Fig. 7, the convergence speed to the same validation PPL on the downstream task is much slower for layer freezing compared to DropBP, especially at high skip rates, where it converges even more slowly than the baseline. We believe this is due to the ability of DropBP to train a relatively larger number of submodules ( ∑ 8 i =0 64 C i ), compared to the fewer submodules trained by layer freezing ( 2 8 ). Moreover, when fine-tuning LLaMA2-70B, DropBP resulted in a 0.5% increase in MMLU5-shot accuracy compared to the baseline, despite a high skip rate of 0.875. This improvement is due to the large number of layers in LLaMA2-70B, enabling DropBP to train deeper and more numerous submodules ( ∑ 20 i =0 160 C i ) even with a high skip rate of 0.875.

Impact of Sensitivity-based Drop Rate We

also conducted an ablation study to analyze the effectiveness of sensitivity-based drop rate allocations. First, we identified the sensitivity of different layers by calculating Eq. 6 during the training of LLMs in various scenarios, as illustrated in Fig. 8a and Fig. 7 in Appendix D. While the distribution varies slightly depending on the number of parameters, fine-tuning approach, and target average drop rate, there is a consistent tendency to assign importance to both the initial and final layers. Consequently, drop rates for these layers are allocated to be lower

by a simple greedy algorithm, as explained in Section 4.1

Additionally, we fine-tuned the LLaMA2-7B and 13B using DropBP on Alpaca datasets, comparing sensitivity-based allocated drop rates with uniform drop rate. In detail, we compared the average accuracy of commonsense reasoning tasks when fine-tuning the models with a learning rate of 1e-4 and 3e-4, as shown in Table 6. Note that the PPL for fine-tuning LLaMA2-7B in Fig. 8b corresponds to a learning rate of 3e-4. The results indicate that sensitivity-based drop rates achieved a 1.6% higher

1.0

0.6x10-3

0.8

Sensitivity

0.4x10-3

0.5

0.2×10-3

0.2

0

10

20

30

40

50

60

Layer Index

(a) Distribution of drop rates determined by sensitivity when the average drop rate is set to 0.875.

uniform (p=0.875)

sensitivity-based（p=0.875)

1.2 × 1017

2.4 × 1017

3.6 × 1017

FLOPS

(b) Validation PPL with uniform and sensitivity-based allocated drop rates.

8

7

6

5

4

3

PPL

accuracy compared to uniform drop rates with a relatively high learning rate of 3e-4, while there was no significant difference in accuracy when the learning rate was set to 1e-4 in LLaMA2-7B. Fig. 8b also shows that sensitivity-based drop rates consistently stabilized the convergence of validation loss, whereas uniform drop rates occasionally diverged when the learning rate was set to 3e-4 in LLaMA2-7B. This phenomenon is even more pronounced with LLaMA2-13B, resulting in a 1.1% increase in accuracy through sensitivity-based drop rate allocation, even with a low learning rate of 1e-4. In other words, sensitivity-based drop rate allocation helps stabilize the training process, especially in the case of large learning rates or larger models.

5 Related Works

Parameter-efficient fine-tuning When fine-tuning LLM, substantial amount of memory is required to store parameters, gradients, and optimizer states. LoRA [14] successfully reduces the memory allocated to gradients and optimizer states by inserting trainable rank decomposition matrices into the linear layers of the model while keeping the original LLM parameters frozen. LLaMA-Adapter [15] and LLaMA-Adapter V2 [16] significantly reduce training memory using trainable adoption prompts and zero-initialized attention mechanisms. Some studies attempt to reduce not only the memory footprint of gradients and optimizer states but also that of parameters by considering quantization. PEQA [20], for instance, quantizes the original LLM parameters into a low-bit format and fine-tunes only the scale factor, thus saving memory for parameters during training. QLoRA [18] and QALoRA [19], built upon LoRA, also employ quantization on the original LM parameters, significantly reducing parameter memory during training. Our DropBP is orthogonal and easily combinable with these PEFT techniques, enabling memory and computationally efficient fine-tuning.

Parallelism Parallelism techniques are widely used to accelerate training LLM using multiple GPU efficiently. Data parallelism [42] is a technique that involves dividing data along the batch dimension for training across multiple GPUs, which still requires sufficient memory to load the entire model on each GPU. Conversely, tensor parallelism [43, 44, 45] partitions the model across GPUs, dividing matrix multiplication operations column-wise and row-wise. Pipeline parallelism [46, 47, 48] involves partitioning the model depth-wise across GPUs, which enables efficient pipeline scheduling. The Zero Redundancy Optimizer (ZeRO) [49] and Fully Sharded Data Parallelism (FSDP) [50] shard parameters, gradients, and optimizer states across multiple GPUs, retrieving parameters when needed to restore their non-partitioned form, enabling the overlapping of computation and communication during training. While these parallelism techniques are designed to efficiently manage the massive computational costs across multiple GPUs, our DropBP specifically aims to reduce the inherent computational costs required for training process.

Layer dropping Stochastic Depth [51], the first approach to randomly drop layers during neural network training, reduces overfitting and costs in image recognition. Layerdrop [52] randomly drops layers during training and selectively uses some layers during inference, accelerating both processes for transformers. Progressive Layer Dropping (PLD) [53] progressively increases the drop rate across depth and iterations, improving training speed without accuracy degradation in transformers. These techniques speed up pretraining of small transformer models like BERT [54] by dropping layers during the entire training process, whereas DropBP, specific to fine-tuning LLMs, drops layers only during backward propagation. Consequently, as detailed in Appendix E, our DropBP achieves higher performance compared to these layer dropping when fine-tuning LLMs.

6 Conclusion

We propose DropBP, an effective algorithm that accelerates the fine-tuning of LLMs by randomly dropping layers during backward propagation, which can be orthogonally integrated into both fullfine tuning and parameter-efficient fine-tuning. We developed the DropBP library as a user-friendly PyTorch extension to facilitate easy integration into existing training codes. Experimental results demonstrate that DropBP significantly accelerates training speed during the fine-tuning of LLMs, achieving comparable accuracy to baseline fine-tuning. Furthermore, DropBP reduces activation memory, enabling long-context training and increasing batch size on limited resources. Consequently, applying DropBP enables a 79% higher throughput on an NVIDIA A100 GPU and a 117% higher throughput on an Intel Gaudi2 HPU.

Acknowledgment

This research was supported in part by the NAVER-Intel Co-Lab. The work was conducted by Seoul National University and reviewed by both NAVER and Intel.

References

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Neurips, Long Beach, CA, USA, December 4-9, 2017, , pages 5998-6008, 2017.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurIPS, New Orleans, LA, USA November 28 - December 9, 2022 , 2022.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, NeurIPS, virtual, December 6-12, 2020 , 2020.

OpenAI. GPT-4 technical report. CoRR , abs/2303.08774, 2023.

Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR , abs/2312.11805, 2023.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien

Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR , abs/2307.09288, 2023.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR , abs/2407.21783, 2024.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In ICLR, virtual, April 25-29, 2022 . OpenReview.net, 2022.

Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca , 2023.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. CoRR , abs/2305.11206, 2023.

Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned llm. Technical report, Databricks, 2023.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, April 25-29, 2022, virtual . OpenReview.net, 2022.

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. CoRR , abs/2303.16199, 2023.

Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parameter-efficient visual instruction model. CoRR , abs/2304.15010, 2023.

Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantizationaware parameter-efficient adaptation of large-scale pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, EMNLP, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 3288-3305. Association for Computational Linguistics, 2022.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR , abs/2305.14314, 2023.

Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-aware low-rank adaptation of large language models. CoRR , abs/2309.14717, 2023.

Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. CoRR , abs/2305.14152, 2023.

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-propagating errors. nature , 323(6088):533-536, 1986.

Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. CoRR , abs/2309.12307, 2023.

Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. CoRR , abs/2309.16039, 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, NeurIPS 2019, Vancouver, BC, Canada, December 8-14, 2019 , pages 8024-8035, 2019.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, virtual, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAI, New York, NY, USA, February 7-12, 2020 , pages 7432-7439. AAAI Press, 2020.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, ACL, Florence, Italy, July 28- August 2, 2019 , pages 4791-4800. Association for Computational Linguistics, 2019.

Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer AI2 reasoning challenge. CoRR , abs/2102.03315, 2021.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, EMNLP, Brussels, Belgium, October 31 November 4, 2018 , pages 2381-2391. Association for Computational Linguistics, 2018.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, New York, NY, USA, February 7-12, 2020 , pages 8732-8740. AAAI Press, 2020.

Martin Svedin, Steven W. D. Chien, Gibson Chikafa, Niclas Jansson, and Artur Podobas. Benchmarking the nvidia gpu lineage: From early k80 to modern a100 with asynchronous memory transfers. arXiv preprint arXiv:2106.04979 , 2021.

Intel Corporation. Intel gaudi2 ai accelerators white paper. Technical report, Intel Corporation, 2023.

Henry J Kelley. Gradient theory of optimal flight paths. Ars Journal , 30(10):947-954, 1960.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain , pages 550-558, 2016.

Sunghyeon Woo, Sunwoo Lee, and Dongsuk Jeon. ALAM: Averaged low-precision activation for memory-efficient training of transformer models. In The Twelfth International Conference on Learning Representations , 2024.

Lightning-AI. Lit-gpt. https://github.com/Lightning-AI/lit-gpt , 2023.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the-art natural language processing. CoRR , abs/1910.03771, 2019.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.

Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow. , 13(12):3005-3018, 2020.

Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR , abs/1909.08053, 2019.

Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turing NLG 530b, A large-scale generative language model. CoRR , abs/2201.11990, 2022.

Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. CoRR , abs/2205.05198, 2022.

Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, NeurIPS 2019, Vancouver, BC, Canada, VDecember 8-14, 2019 , pages 103-112, 2019.

Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, and Phillip B. Gibbons. Pipedream: Fast and efficient pipeline parallel DNN training. CoRR , abs/1806.03377, 2018.

Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun. Bpipe: Memory-balanced pipeline parallelism for training large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, ICML, Honolulu, Hawaii, USA, 23-29 July 2023 , volume 202 of Proceedings of Machine Learning Research , pages 16639-16653. PMLR, 2023.

Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T. Kramer, editors, SC 2020, Virtual Event / Atlanta, Georgia, USA, November 9-19, 2020 , page 20. IEEE/ACM, 2020.

Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow. , 16(12):3848-3860, 2023.

Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, Amsterdam, The Netherlands, October 11-14, 2016 , volume 9908 of Lecture Notes in Computer Science , pages 646-661. Springer, 2016.

Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, NeurIPS, virtual, December 6-12, 2020 , 2020.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society, 2016.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR , abs/1710.03740, 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019.

Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In ICL, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017.

Appendices

A The importance of short paths in residual networks

In Section 3.1, we interpret transformer models as a collection of numerous blocks, each composed of various modules with residual connections. Our hypothesis is that we can fine-tune LLMs well by training only certain shallow submodules. To theoretically analyze this hypothesis, we measured the impact of submodules based on their path lengths in LLaMA2-7B, as shown in Fig. 9. Specifically, we followed these steps:

We first perform a forward pass through the entire network.

During the backward pass, we randomly sample k residual blocks, which are backpropagated without passing through skip connections, while the remaining n -k blocks are bypassed through the skip connections.

We then measure the norm of the gradient at the input.

We take 100 measurements for each path length k . Subsequently, we multiply by the distribution of all possible path lengths, which follows a Binomial distribution, to quantify the gradient contribution from paths of a specific length.

In Fig. 9b, we observed that the gradient per path length decreases as the path length increases. Consequently, Fig. 9c demonstrates that shorter path lengths have a greater impact on the gradient in LLaMA2-7B. These observations are consistent with the existing findings [35] in ResNet [55], which attributed this phenomenon to vanishing gradients. Therefore, our DropBP enables effective training LLMs by focusing on training important short submodules.

(a) Distribution of path length.

(b) Gradient per path length.

(c) Total gradient per path length.

2 ×1018

DistributionofPathLength

ndu

10-4

GradientMagnitudeperPathLength

TotalGradientMagnitude

5× 106

TotalGradientMagnitudeperPathLength

107

Number of Paths

at

101

GradientMagnitude

10-8

105

3×106

10-11

1×1018

10-17

10-12

0

102030405060

10-16

1 ×106

0

10

20

30

40

50

60

0

10

20

30

40

50

60

0

10

20

30

40

50

60

Path Length

Path Length

PathLength

B Theoretical FLOPs and Actual Training Time Using DropBP

In this section, we calculate the theoretical FLOPs reduction afforded by DropBP and compare this reduction to the actual training time reduction as shown in Table 7. As outlined in Section 2, the computational costs arise from output activation calculations by Eq. 1 during forward propagation, and input and parameter gradient calculations by Eqs. 2 and 3 during backward propagation. We denote the FLOPs for these operations as F out , F grad , and F param , respectively. Therefore, the total FLOPs for the backpropagation algorithm can be calculated by the following equation:

where F T represents the FLOPs during the entire training process, F fw for forward propagation (i.e. F fw = F out ), and F bw for backward propagation (i.e. F bw = F grad + F param ). DropBP reduces FLOPs for backward propagation by a target average drop rate ( p avg ). Therefore, total FLOPs in DropBP can be formulated as below:

Consequently, the theoretical FLOPs reduction ratio by DropBP can be represented as follow:

Note that in full fine-tuning (Full-FT), the computational costs for output calculations, input gradient calculations, and parameter gradient calculations are nearly identical (i.e., F out = F grad = F param ). Conversely, in parameter-efficient fine-tuning techniques (PEFT) such as LoRA and QLoRA, the costs of calculating parameter gradients are negligible ( F out = F grad , F param = 0 ) due to a very small number of trainable parameters and the freezing of original LLM parameters. By substituting this into Eq. 10, the theoretical FLOPs reduction ratio by DropBP can be expressed as:

Therefore, with target average drop rates of 0.5, 0.75, and 0.875, DropBP achieves theoretical FLOPs reductions in Full-FT of 33%, 50%, and 58%, respectively, according to Eq. 11. This aligns with the actual training time reduction when utilizing DropBP in Full-FT as shown in Table 3 and Table 7. This trend is also evident when utilizing DropBP in LoRA and QLoRA. According to Eq. 12, the reductions in FLOPs for various target average drop rates of 0.5, 0.75, 0.875 are derived as 25%, 38%, and 44%, respectively. This closely aligns with the actual training time reductions observed when DropBP is applied to LoRA and QLoRA as demonstrated in Table 7.

C Convergence Speed Up Using DropBP

(a) LLaMA2-7B w/ LoRA (Alpaca)

(b) LLaMA2-7B w/ LoRA (Dolly)

(c) LLaMA2-7B w/ Full-FT (Alpaca)

(d) LLaMA2-7B w/ Full-FT (Dolly)

(e) LLaMA2-13B w/ LoRA (Alpaca)

(f) LLaMA2-13B w/ LoRA (Dolly)

(g) LLaMA2-70B w/ QLoRA (Alpaca)

(h) LLaMA2-70B w/ QLoRA (Dolly)

8

Baseline

8

Baseline

DropBP (p=0.5)

DropBP (p=0.5)

7

DropBP (p=0.75)

7

DropBP (p=0.75)

DropBP(p=0.875)

DropBP (p=0.875)

6

6

PPL

5

PPL

5

4

4

3

2

1.47 x speed up

个

3

1.44 x speed up

↑

0.4 ×1017

0.8×1017

1.2 × 1017

0.4 ×1017

0.8 × 1017

1.2 × 1017

FLOPS

FLOPs

8

Baseline

8

Baseline

DropBP (p=0.5)

DropBP (p=0.5)

7

DropBP (p=0.75)

7

DropBP (p=0.75)

DropBP (p=0.875)

DropBP (p=0.875)

6

6

PPL

5

PPL

5

4

4

3

2

1.71 x speed up

个

3

1.56x speed up

不

0.6 ×1017

1.2 × 1017

1.8 × 1017

0.6 ×1017

1.2 × 1017

1.8 × 1017

FLOPs

FLOPS

8

Baseline

Baseline

DropBP (p=0.5)

DropBP (p=0.5)

7

DropBP (p=0.75)

DropBP (p=0.75)

DropBP (p=0.875)

DropBP (p=0.875)

6

6

L

P

5

PPL

5

4

4

3

2

3

1.48 x speed up

↑

1.44x speed up

0.8×1017

1.6 × 1017

2.4 × 1017

0.8 ×1017

1.6 × 1017

2.4 × 1017

FLOPs

FLOPS

Baseline

Baseline

7

DropBP (p=0.5)

7

DropBP (p=0.5)

DropBP (p=0.75)

DropBP (p=0.75)

6

DropBP (p=0.875)

6

DropBP (p=0.875)

PPL

5

PPL

5

4

4

3

3

2

1.50x speedup

2

1.41 x speed up

不

0.4 ×1018

0.8×1018

1.2 × 1018

0.4 ×1018

0.8×1018

1.2 × 1018

FLOPs

FLOPs

(a) Perplexity curve across training steps.

(b) Perplexity curve across training time.

8

Baseline

8

Baseline

DropBP (p=0.5)

DropBP (p=0.5)

7

DropBP (p=0.75)

7

DropBP (p=0.75)

DropBP (p=0.875)

DropBP (p=0.875)

6

6

PPL

5

PPL

5

4

4

3

3

2

2

2000

4000

6000

8000

0

1

2

3

4

5

Training Steps

Training time (Hours)

When analyzing training curves across training steps in Fig. 11a, the convergence of loss per step at a drop rate of 0.5 is almost identical to the baseline. However, with drop rates of 0.75 and 0.875, the convergence speed per step is slower compared to baseline. Nonetheless, DropBP significantly reduces the time consumed per training step, because it skips the backward propagation computations for the dropped layers. Consequently, the convergence speed per training time is actually faster for DropBP compared to the baseline as shown in Fig. 11b.

D Distribution of Drop Rates Determined by Sensitivity

(a) LLaMA2-7B w/ LoRA + DropBP (p=0.5)

(b) LLaMA2-7B w/ LoRA + DropBP (p=0.875)

1.2 × 10-3

1.0

1.0

0.6x10-3

0.8

0.8

Sensitivity

0.8x10-3

Sensitivity

0.4×10-3

0.5

0.5

0.4×10-3

0.2

0.2×10-3

0.2

D

10

20

LayerIndex

30

40

50

60

10

20

LayerIndex

30

40

50

60

1.0

1.0

2.4×10-2

1.8x10-1

0.8

0.8

Sensitivity

Sensitivity

1.6×10-2

0.5

1.2 ×10-1

0.5

0.8×10-2

0.2

0.6×10-1

0.2

D

10

20

30

40

50

60

10

20

30

40

50

60

LayerIndex

LayerIndex

(e) LLaMA2-70B w/ QLoRA + DropBP (p=0.5)

(f) LLaMA2-70B w/ QLoRA + DropBP (p=0.875)

1.0

2.4×10-5

1.0

2.4×10-5

0.8

0.8

Sensitivity

Sensitivity

1.6x10-5

1.6x10-5

0.5

0.5

0.8x10-5

0.2

0.8×10-5

0.2

20

40

60

80

100

120

140

160

20

40

60

80

100

120

140

160

LayerIndex

Layer Index

E Comparisons between Layer Dropping and DropBP on fine-tuning LLMs

In this section, we compare Layerdrop (LD) [52] and Progressive Layer Dropping (PLD) [53] with DropBP under the same LLMs fine-tuning scenario. We set the relative FLOPs of LD and PLD to 0.75 of the baseline (LoRA), which corresponds to the same relative FLOPs when the drop rate of DropBP is set to 0.5.

As shown in Fig. 8, our DropBP converges faster to the same validation PPL compared to LD and PLD. Moreover, as seen in Table 8, DropBP achieves comparable accuracy to the baseline even with a relative FLOPs of 0.56, whereas LD and PLD experience a significant accuracy drop of over 5% with a relative FLOPs of 0.75. We believe this difference arises from the high sensitivity of forward propagation throughout the fine-tuning process. Specifically, layer dropping techniques omit certain layers of well-pretrained LLMs during forward propagation, resulting in significant output deviations that adversely impact the loss and the overall training process. Conversely, DropBP maintains all layers during forward propagation, thereby ensuring precise outputs and loss calculations, which facilitate stable training. Please note that, as explained in Section 5, LD and PLD are designed to accelerate the pretraining of small transformer models (SLMs) like BERT by dropping layers throughout the entire training process while DropBP only focuses on fine-tuning LLMs. In future studies, we will explore whether DropBP can similarly

Baseline

8

LD (p=0.25)

PLD (p=0.25)

7

DropBP (p=0.5)

DropBP (p=0.75)

6

DropBP (p=0.875)

PPL

5

4

3

2

0.4 ×1017

0.8 ×1017

1.2 × 1017

FLOPs

accelerate the pretraining of transformer models and investigate ways to improve its effectiveness.

F Experimental Details

In our experimental setup, the AdamW [57] optimizer and a cosine annealing learning rate scheduler [58] were utilized as common settings. LoRA [14] and QLoRA [18] were integrated to every linear

layer of our model, with the LoRA parameters r and α set to 8 and 16, respectively. We experimented with all the learning rates presented in Table 9 and reported the best accuracy achieved in Table 1-2.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We appropriately present the contributions of the paper in the Abstract and Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In Section 5 and Appendix E, we clearly state that DropBP is developed specifically for fine-tuning, unlike other layer dropping techniques, and we indicate plans for further improvements in future research.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Appendix B, we mathematically calculate the theoretical reduction in FLOPs achieved by DropBP. Through experiments presented in Section 4.2 and Appendix B, we confirm that this theoretical reduction closely matches the actual training time reduction.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 4.1 and Appendix F, we provide detailed information to ensure reproducibility, and in the abstract, we present the anonymous code in Abstract to implement this.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We present the anonymous code for reproducibility in the Abstract.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide the experimental details in Section 4.1, Appendix F, and the anonymous code presented in the Abstract.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Although we conducted experiments with multiple seeds and report the average values, we did not include error bars or statistical information because the deviations were minimal, and including them would detract from the clarity of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mention the types of devices used for the experiments in Section 4.1, and provide the time required to reproduce the experimental results in Section 4.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Since we discuss a efficient fine-tuning algorithm for LLMs, we believe that our work does not have any direct negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is related to the efficient fine-tuning of LLMs, and therefore, there are no specific safeguards described for the responsible release of data or models, as the nature of our research does not involve high-risk misuse scenarios.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We appropriately cite the original paper and existing codes in Section 4.1 and our code in Abstract.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide the new code with a proper license as an anonymized URL in the Appendix.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DropBP, an innovative approach to accelerate the fine-tuning of Large Language Models (LLMs) by selectively dropping layers during backward propagation. This method is presented as a means to reduce computational costs and activation memory, significant challenges in the efficient fine-tuning of LLMs. The authors have provided a clear implementation of DropBP as a PyTorch extension and demonstrated its effectiveness through experiments on various LLMs and datasets.\n\nSTRENGTHS:\n- The concept of dropping backward propagation layers to reduce computational overhead is differential from previous work and addresses an important issue in training large models.\n\n- The paper includes extensive experiments that validate the effectiveness of DropBP in reducing training time and memory usage while maintaining accuracy.\n\n- The development of a PyTorch extension for DropBP facilitates easy integration with existing training codes, enhancing the practical applicability of the method.\n\nWEAKNESSES:\n- The motivation is not well illustrated. I agree with that dropping sublayers could lead to training efficiency as the model turns to a shallower counterpart. However, I mean, pervious work like LayerDrop and others omit the layer computation in the forward pass. Then the computation could be removed in the subsequent backward computation with essential engineering efforts. Thus it lacks a clear distinction in terms of technical innovation compared to these previous works.\n\n- While the paper proposes omitting sublayer computation in the backward pass, it's unclear why the forward pass computation remains unchanged. Justifying this choice or exploring alternatives would strengthen the contribution.\n\n- The faster convergence observed in Figure 5 with DropBP compared to the vanilla model is counterintuitive. The observation here quite confuses me since the backward pass optimizes a partial computation graph, concerns regarding overfitting arise. The paper would benefit from a discussion on potential regularization techniques employed to address this, and a comparison with related work (e.g., [1]) that utilizes sublayer dropping for regularization in training a deep Transformer model. \n\n\n  \n  [1] Li et al., 2021 (AAAI) Learning Light-Weight Translation Models from Deep Transformer\n\nQUESTIONS:\nsome typos\n\n- Line 49: As a results -> As a result\n- Line 62: a effective -> an effective\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel method to reduce the computational and memory costs associated with fine-tuning large language models (LLMs). The authors introduce DropBP, a technique that randomly drops layers during backward propagation, effectively reducing the computational operations (FLOPs) and activation memory needed. This method assigns drop rates based on the sensitivity of each layer to ensure stable training. The approach is applicable to both full fine-tuning and parameter-efficient fine-tuning (PEFT) methods. The paper reports significant improvements in training time, convergence speed, and maximum sequence length when fine-tuning LLaMA2 models with DropBP.\n\nSTRENGTHS:\n- DropBP introduces a novel method for reducing the computational and memory costs associated with fine-tuning LLMs. This is an important contribution to the field, given the increasing size and complexity of these models.\n\n- The paper provides empirical evidence that DropBP significantly reduces training time (by 44%), accelerates convergence (1.5× faster), and increases the maximum sequence length (up to 6.2×) on a single NVIDIA A100 GPU. These results demonstrate the effectiveness of the approach. The authors conduct thorough experiments on multiple datasets and models, providing a robust evaluation of DropBP's performance across different scenarios.\n\nWEAKNESSES:\n- The paper mentions that the sensitivity calculation is done only once and has negligible overhead. However, more details on this process and its potential impact on training time would provide a clearer understanding of any trade-offs involved.\n\n- The paper could benefit from a more detailed theoretical analysis of why DropBP works as effectively as it does. This would strengthen the paper by providing a deeper understanding of the underlying principles.\n\nQUESTIONS:\n- Can you provide more details on the sensitivity calculation process? Specifically, how is the sensitivity of each layer computed, and what is the computational overhead associated with this step?\n\n- What are the best practices for tuning the drop rates in DropBP? Are there guidelines or heuristics that practitioners can follow to optimize performance for their specific use cases?\n\n- How well does DropBP integrate with other recent advancements in efficient training techniques, such as mixed precision training or distributed training frameworks? Have you explored these combinations in your experiments?\n\nLIMITATIONS:\nLimitations have been discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed to drop layers during backward prop (BP) based on layer sensitivity. The method aims to reduce the cost for gradient computation and storage for intermediate activation in full BP.\n\nSTRENGTHS:\n1. Reducing the cost of full BP in PEFT has been an important challenge. \n2. The method is simple and is easy to integrate to either full fine-tuning or PEFT. \n3. Experiments demonstrate that DropBP can speed up the training while retaining the accuracy. The resulting memory reduction makes longer sequence modeling accessible.\n\nWEAKNESSES:\n1. The idea of optimizing NNs with sparse gradient is not new. This paper needs to add more discussion and comparison with related works in sparse learning e.g., [1-3]\n2. Table 1 only shows results on two datasets and limited benchmark. It is unclear if the method works well for generation tasks and domain-specific transfer learning.\n3. It is unclear which algorithm is used to solve the constraint minimization problem, i.e., to determine the layer-specific rates based on sensitivity, and its extra computational cost.\n4. (Minor) In fine-tuning, DropBP drops a set of layers. However, the sensitivity of a set of layers may not be accurately represented by the direct summation of the sensitivities of individual layers in the set.\n\n[1] Sun, Xu, et al. \"meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting.\"\n[2] Sung, Yi-Lin, Varun Nair, and Colin A. Raffel. \"Training neural networks with fixed sparse masks.\" \n[3] Brock, Andrew, et al. \"Freezeout: Accelerate training by progressively freezing layers.\"\n\nQUESTIONS:\n1. What is the long context modeling performance after applying DropBP?\n2. Could the authors present Figure 5 with # of steps as the x-axis to demonstrate faster convergence?\n3. I wonder if the sensitivities would evolve, and the drop rate needs to be re-allocated through training.\n\nLIMITATIONS:\nThe authors have addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}