DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
Authors: Sunghyeon Woo
Baeseong park
Byeongwook Kim
Minjung Jo
Se Jung Kwon
Dongsuk Jeon
Dongsoo Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: Training Acceleration
Memory Efficient Fine-Tuning
Large Language Models
Backpropagation Optimization.
Primary area: natural_language_processing
TL;DR: DropBP, randomly dropping backward propagation based on layer sensitivity,  significantly accelerates fine-tuning in Large Language Models (LLMs) with considerable memory reduction.

Abstract: Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5$\times$, and enable training with a sequence length 6.2$\times$ larger on a single NVIDIA-A100 GPU. Furthermore, our DropBP enabled a throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at [https://github.com/WooSunghyeon/dropbp](https://github.com/WooSunghyeon/dropbp).

DropBP: Accelerating Fine-Tuning of Large Language Models by Dropping Backward Propagation
Sunghyeon Woo't Se Jung Kwon
Baesung Park2*
Byeongwook Kim?
Minjung Jo
Dongsuk Jeon'
Dongsoo Lee²
Scoul National University!
NAVER Cloud?
Abstract
Large language models (LLMs) have achieved significant success across various domains. However, training these LLMs typically involves substantial memory and computational costs during both forward and backward propagation. While parameter-efficient fine-tuning (PEFT) considerably reduces the training memory associated with parameters, it does not address the significant computational costs and activation memory. In this paper, we propose Dropping Backward Propagation (DropBP), a novel approach designed to reduce computational costs and activation memory while maintaining accuracy. DropBP randomly drops layers during back- ward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. Additionally, DropBP calculates the sensitivity of each layer to assign an appropriate drop rate, thereby stabilizing the training process. DropBP is not only applicable to full fine-tuning but can also be orthogonally integrated with all types of PEFT by dropping layers during backward propagation. Specifically, DropBP can reduce training time by 44% with comparable accuracy to the baseline, accelerate convergence to the same perplexity by 1.5 x, and enable training with sequence length 6.2x larger a single NVIDIA-A 100 GPU. Furthermore, our DropBP enabled throughput increase of 79% on a NVIDIA A100 GPU and 117% on an Intel Gaudi2 HPU. The code is available at
1 Introduction
Since the advent of the transformer architecture [ the field of language modelling has experienced dramatic advancements. Especially, following the scaling laws [2 3,, the development of Large Language Models (LLMs) 4 6 9 continues with the aim of achieving or outperforming human capabilities. However, tremendously increasing the size of the model results in significant
costs for training from scratch. An alternative approach for developing high-capability L.LMs without
the costly pretraining on extensive datasets is instruction tuning [10 11 12] [3]. This method
fine-tunes well-trained foundation models on relatively small instruction followinn datasets, enabling the models to better understand and follow prompts.
While fine-tuning Large Language Models (LLMs) on instruction-followin datasets is more cost- effective than training from scratch, it still requires substantial memory for parameters and activations, Fine-
along with significant floating point operations (FLOPs). In this context, Garameter-Efficien Tuning (PEFT) techniques [14 [6] effectively reduce the memory required for parameter gradients and optimizer states by freezing pretrained weights and selectively training newly added modules.
Moreover, when combined with quantization methods [17 18 19 20 these techniques can further significantly decrease the memory requirements for parameters.
"quall contribution
Intern at NAVER Cloud
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Forward Propagation
Backward Propagation
300
Available max sequence length
Training time per sample (ms)
250
3K
200
57%
48%
6.2x
153.1
24
150
3.7K
23.2
100
29.6
1K
117.7
117.5
1.2K
91.1
Full-FT Full-FT-DroBBP
LoRA
LoRA+DropBP (p=0.875)
QLoRA QLoRA
QLoRA
QLoRA
(p=0.875)
DropBP
DropBP
DropBP
(pm0.5)
(p=0.75)
(p=0.875)
(a) Training time per sample LLaMA2-78
(b) Available max sequence length of LLaMA2-70B.
Figure I: Performance enhancements in fine-tuning large language models using DropBP when the represents the target average drop rate for backward propagation: (a) Training time per sample for fine-tuning LLaMA2-78 7B with DropBP, at sequence length of 512 and micro batch size of 2. (b) Available max sequence length for fine-tuning LLaMA2-70B with DropBP, at micro batch size of on an NVIDIA-A100 GPU,
While Parameter-Efficien Fine- Fine-Tuning (PEFT) has successfully reduced memory associated with parameters, two significant challenges remain for efficient fine-tuning: computational cost and activation memory, both which are linked to the backpropagation [21]. First, fine-tuning Large Language Models (LLMs) using backpropagation requires substantial moating-point operations (FLOPs). Specifically, the backpropagation algorithm necessitates forward propagation to calculate outputs and backward propagation compute gradients for inputs and parameters. Notably, backward propagation demands twice the computational operations compared to forward propagation, thus becoming the primary bottleneck. Second, all intermediate outputs (i.e., activations) generated during forward propagation must be stored for compute in backward propagation. This activations consume considerable memory, which becomes especially critical when training LLMs long sequence contexts [22 23).
In this paper, introduce Dropping Backward Propagation (DropBP), efficient fine-tuning algorithm for LLMs that significantly reduces computational costs and activation memory. DropBP randomly drops layers during backward propagation, which is essentially equivalent to training shallow submodules generated by undropped layers and residual connections. As a result, these undropped layers longer require FLOPs and activation memory during backward propagation. Additionally, DropBP calculates the sensitivity of each layer, an indicator of its impact on the total training process, adjust drop rate. This careful calibration of drop rate according layer sensitivity ensures more stable training. This DropBP algorithm can be seamlessly integrated with any PEFT algorithm, operating orthogonally by simply dropping layers during backward propagation.
We implemented DropBP as an easy-to-integrate PyTorch library 24), requiring only minimal changes to the existing training codes. In experiments, DropBP successfully reduces training time as shown in Fig. la maintaining comparable accuracy on the MMLU 25 and commonsense reasoning
tasks 26, 27 28 29
30 The DropBP also accelerated the convergence of the same perplexity by 1.5x in LLa 70B [ Moreover, DropBP substantially decreases activation memory. increasing available maximum sequence length by up 6.2 in L.aMAA-70B.
a single NVIDIA A 1000 GPU (31) as shown in Fig. [b] Finally, our DropBP increases training throughput by up to 79%
and 117% single NVIDIA A100 GPU and Intel Gaudi2 HPU 332. respectively, when fully
fine-tuning LLaMA3-8B 9 In summary, the main contributions of our paper are:
We propose DropBP, an efficient fine-tuning algorithm that randomly drops backward propagation based on layer sensitivity.

---

We implemented DropBP as a user-friendly PyTTrcc extension with a straightforward API for ease of use, making it easily applicable to existing training codes.
DropBP reduces training time by 44% with comparable accuracy, increases convergence speed by 1.3x increases the available maximum sequence length up 6.2x, and enhances training throughput up to 117%.
2 Background & Motivation
Backpropagation
Backpropagation 33 core algorithm for training deep neural networks,
involves both forward and backward propagation. Specifically, the training process in the linear layer represented follows:
Forward Prop:
H out W x Hin
(1)
Backward Prop:
WH = W x VHout
(2)
VW = VHou H
(3)
where H. and W represent the activations and parameters, respectively, with 'x' indicating matrix multiplication operation. The gradients of H and W are denoted by VH and VW. The computational costs during forward propagation primarily arises from matrix multiplication for computing output activations by Eq. In backward propagation, the computational burden is primarily due to matrix multiplication for calculating input gradients by Eq. and parameter gradients by Eq. Note that the computational costs of these equations are almost equal. Consequently, the FLOPs required for backward propagation including Eqs. 2 and 3 are approximately 2x as large as the FLOPs needed for forward propagation by Eq. Furthermore, the activations all layers (H must be stored in memory for use in backward propagation computations in Eq. Therefore, focusing on reducing the computations during backward propagation crucial for decreasing both the overall computational costs and the activation memory.
Interpretation the model with residual connections Residual connections used methods to address the issue of vanishing gradients (34. Transformer [1] also incorporates residual connections that bypass multi-head attention and feedforward networks. Networks utilizing
of the widely
these residual connections can be interpreted as ensembles of several submodules [35]. For example, we expand the model with three residual connections as shown in Fig. 2a it can be represented as combination of eight submodules, as depicted in Fig. 2b From this perspective, network with layers can be interpreted as an ensemble of 2" submodules
F
F
F
F2
P
(a) l-yyesss model with residual connections.
(b) Combination of multiple submodules.
Figure 2: Interpreting the model with residual connections as combination of multiple submodules.
Methodology
3.1 Dropping Backward propagation
In Section 2 we observed that the backpropagation algorithm consumes significant amount of FLOPs and activation memory, particularly for the backward propagation. To reduce this overhead,

---

Propagation
Backward Propagation
Xpnt
V Xunt
No drop
p= 8
F
F2
F3
Posttion-wtse MLP
Poatton-wise MLP
Layer Norm
Layer Norm
F
No drop
on/off
p=0.3
Pg
F
Mullt-Head
Allention
Mult-Haad Allention
F
①
Layer Norm
Layer Norm
F₂
F2
F3
X
V Xim
(a) The concept of DropBP.
(b) DropBP as combiation shallow submodules.
Figure 3: The overveiw of DropBP.
we propose straightforward approach: Dropping Backward Propagation (DropBP). DropBP is designed to drop layers exclusively during backward propagation, effectively reducing both FLOPs and activation memory for the dropped layers, as demonstrated in following equations:
X Ximm = Xin + Dp. (fATTN(FFNN (X.)()) Xout Ximm + Pri. (fFFN(fFNNN (fLN(X(mm)) (Ximm)))
(4)
(5)
Here. Xin. Xout, and Ximm represent the input, output, and immediate activation between the attention layer and feedforward network in transformer block, respectively. JATTN, JFEN, and JLN denote the attention layer, feedforward network, and layer normalization of the transformer block. The DropBP layer, defined as (X) skips backward propagation in the input X with probability of p. while not dropping forward propagation. Following to Eqs. and S backward propagation in the attention layer and feedforward network is dropped with probabilities and p++1+ respectively, as shown in Fig. 3a
We can view the transformer as a collection of a lot of submodules composed of various modules (i.e., fATTN o JIN and fFFN fLN) with residual connections, described in Section 2. When the transformer block contains n. units, each including multi-head attention and a feedforward network, the model can be interpreted as ensemble of 2 submodules. From this perspective, DropBP can be interpreted as training only certain shallow submodules. For example, as shown in Fig. 3b if the F2 layer is dropped, only the shallow submodule composed of the remaining layers trained during backward propagation. Therefore, if the overall drop rate is p. DropBP can be interpreted as training shallow submodules with the depth of 2n(1 p) less, since 2np layers are dropped. We anticipate that training these smaller modules alone can effectively fine-tune well- cell-pretrained LLMs, based on the analysis that shallow submodules have significant impact on the overall training process as detailed in Appendix A
3.2 Sensitivity- Drop Rate Allocation
we introduce DropBP, which selectively drops layers during backward propagation and trains only certain shallow submodules. In addition, we hypothesized that the significance of individual layers and the submodules encompassing these layers varies in their impact on the overall training process. Therefore, we assign different drop rate to each layer based on sensitivity, which is defined by defined by how much each layer and its encompassing submodules affect the overall training process terms of parameter gradient. Specifically, we calculate the sensitivity of layer by the variance in L2-nom norm of parameter gradients between when the backward propagation of that layer
In Section 3.1

---

is skipped or not, inspired by GradNorm Var [3), memory efficient gradient variance approximation, as below:
St Σ (llV ( 112)
(6)
where S₁ denotes of the 1-th layer. Here, VW, represnets the parameter gradient of the i-th layer when layers dropped, while Vw denotes the parameter gradient of the i-th layer when the I-th layer is dropped during backward propagation. After calculating the sensitivity for each layer, we aim minimize the expected sensitivities across all layers-essentially the expected gradient variance caused by DropBP-under given FLOPs as follow:
min E( (,, s.t. (1--A)) S P.
where p: denotes the drop rate, and F; denotes the FLOPs of the i-th layer during backward propagation. Fg represents the target FLOPs, derived from the target average drop rate Paeg (i.e. Fi = P Σ: F,). In other words, determine the drop rates across all layers that minimize the expected sensitivities of the model, while satisfying a given FLOPs budget, by solving Problem 7 In practice, DropBP addresses the Prob. using simple greedy algorithm, as detailed in Section
4 Evaluation
4.1 Implementation and Settings
Ieport
Arport lanch
frow 2epor Jipurt, Srips?
from dropap, handler seport Gropol
Blocktorch.rr..o tranaformor
(,ea),__________________________ +.):
optinizer (eropsy handler tre Dropeitandien (model)
Initial drop
se2f self.rom,_
self-attn
jeroppy Nandion Jestaal nutolores rets)
Step 1, Insert a DrogBP layer Into Transformer Block
se3f, Aucoo
training
32
bele cwoptpppppe..oooo jate arophi (Sapera:))))))) bel s, 3, wil attnl, #loper.. salf floper..)
for 1ter in
Step dropped layers
bear
Bropho handler Sandier, cropped layers()
Adjusi drop ratts lyyers sensitiitiiss
eain www.aaeennee..c..o aore 10K), ...)
(ode((dara)
31
X-w self cropte attuck)es Instead of 1x bac"
loss loss, fuec(sutpur, tarpet)
oxttalten zero gras() (occ.hrrbord)))
17
reture
Innjeex
droppp handler sensitiuiyy based dres bafbaciprop, drop, rate).
model
layers
aanaardd dropto, handier dww.acttoor.rrraeitt sotinen grad):
Loan bec.eerd))))
Code for preparing model with DropBP.
(b) Code for training with DropBP.
Figure 4: Code implementation for integrating DropBP.
DropBP aims to decrease the training costs fine-tuning based on backpropagation, consequently enabling the acceleration of both full fine-tuning and parameter-efficient fine-tuning using back- propagation. To facilitate practical implementation, we developed user-friendly DropBP library in PyTorch 224] as demonstrated in Fig. In detail, we implemented DropBP layer that internally drops backward propagation in the imput direction according to a given drop rate as shown in Fig. 4a The DropBP layer designed to initially receive the FLOPs of the layers that would be skipped (F;) as initial value to solve Prob. 7
Additionally, we developed DropBPHandler er that automatically solve Prob. 7 by assigning varying drop rate to each layer using a simple greedy algorithm, as demonstrated in Fig. 4b) Specifically, the process begins by setting the drop rate (p,) of all layers to 0 and then gradually increases them to
5

---

align with the target average drop rate (Pavy) set by the user. In each step, the drop rate for each layer incremented by 0.1, ensuring that the increase total expected sensitivity kept to a minimum. We train with uniform drop rate for the initial 10% of total iterations and then adjust the drop rate for each layer based on sensitivity when training is stable. Since sensitivity is calculated only once at the 10% of the entire iteration, the overhead from this calculation is negligible.
We integrated our DropBP code into LiGPP (37] and HuggingFace [38], repositories for training and evaluating LLMs. We first fine-tuned LLaMA2-7B, 13B, and 70B g on Alpaca [ and
Dolly [3] datasets. The fine-ture tne-tuned models were evaluated on 5-shot Massive Multitask Language
Understanding (MMLU) tasks [25] and O-shot commonsense reasoning tasks [26] 27 28
29 30
using mnn-vvalation--haa harness [3] library. We also fine-tunes LLaMA3-8B [9] on the Oasstl dataset
40 and evaluating the model on MT-Bench task [4I] using GPT4o-mini S as a judge. These
experiments were conducted on single NVIDIA A100 GPU, and we measured the training time,
memory usage, and convergence speed. More details about setup can be found in Appendix
Table 1:
Test accuracy MMLU and commonsense reasoning tasks. In DropBP, drop rate is the target average drop rate across all layers in backward propagation. Note that Full-FT stands for full fine-tuning.
Method
Drup Rate
Dataset
MMLU (5-shot)
Commonsense Reasoning (Ilsholl)
Human.
STEM
Social.
Other
Avg.
PIQA
H8
Arc-C
Are-E
OBQA
WG
Avg.
[LLMA2-7B
43.5
37.0
51.6
52.2
45.7
79.1
76.0
46.2
74.5
44.0
69.3
64.9
LoRA
Alpaca
14.7
75.9
48.5
75.0
45.2
70.5
LoRA+DropBP
0.5
Alpaca
42.8
35.7
50.3
$1.0
44.7
79.6
76.0
48.7
75.5
46.2
69.6
65.9
LoRA+DropBP
0.75
Alpuca
76.9
18.0
75.6
47.1
69.0
LoRA+DrophP
0
Alpuca
41.5
34.5
49.6
50.4
43.7
79.6
77.2
48.2
76.3
47.8
69.1
66.4
Fuli-FT
Alpuca
12.7
35.6
51.1
44.7
79.2
76.
18.0
75.8
45,2
69.8
65.7
Full-FT-DooBB
0.5
Alpuca
42.6
35.4
49.8
$1.0
44.4
79.5
76.2
47.8
75 4
45.4
68.5
65.5
Full-FT-DrohPP
0.75
Alpaca
42.6
36.7
45,0
78.8
77.0
6
45.6
69.8
65.9
Full-FT-DropDP
0,875
Alpuca
42.7
35.3
50.7
51.2
44.7
79.2
76.9
46.8
75.5
46.2
69.3
65.6
LORA
Bolly
"33.9"
"SS.1"
"3.0"
46.7
79930
'1.22
79730
'9.7
65.8"
LoRA+DroBBP
0.5
Dolly
44.0
36.8
53.1
53.2
46.4
79.3
76.3
47.3
76.5
44.8
68.8
65.5
I LoRA
0,75
Dolly
43.9
37.0
$2.4
53.1
46.3
79.4
76.2
46.2
75.4
44.8
68.8
65.1
DropP
0,875
Dolly
36.7
46.1
79.1
76.1
45.8
75.3
44.6
68.4
64.9
FIFTT
www.............................
Dally
"3.3""
35.1
33.8
""332"""""""""""""""""""""""""""
46.6
7993
76.2"
1663
76622
"4:2""
68.9
653""
Tull-FT+DroBB
Dolly
79.2
76.2
16.4
68.8
Full FT-DooBP
0.75
Dolly
43.1
36.7
51.8
52.6
45.77
79.2
76.4
45.8
75.4
44.6
69.1
65.1
Full-FT-DroBPP
0.875
Dolly
76.3
46.2
75.0
T 1MMA2-138
52.2
44.1
62.9
61.5
54.8
80.6
799.4
49.5
77.4
45.6
72.5
67.5
LoRA
Alpuca
51.7
43.8
63.3
61.7
54.7
50.6
79.5
51.6
78.5
45.8
72.1
68.0
L LoRpDD
0.5
Alpuca
52.4
44.2
63.1
62.0
55,0
80.7
79.6
50.9
78 4
44.8
71.7
67.7
LoRA-DrophPP
0.75
Alpaca
79.7
51.5
79. 1
45.6
71.7
68.1
LoRA+DropDP
0,875
Alpuca
$1.1
44.2
63.3
61.6
$4.6
80.8
79.8
51.0
78.3 22
45.0
71.4
67.7
LoRA
Dolly
31.9
43.
63.7"
50.4
79.9
"51.1
7S4
45.6
71..7"
67.9"
1 LoRA+DropHP
0.5
Dolly
52.4
44.1
63 4
62.1
55.1
80.7
79.9
50.9
7%.5
45.6
72,3
68.0
LoRA:DropBP
0.75
Dolly
52.1
44.3
63.3
61.7
54.9
80.6
79.8
51.4
77.5
45.4
72.1
67.8
LoRA+DropBP
0,875
Dolly
52.8
43.9
63.4
61.9
55.1
30.5
79.7
51.3
77.9
45.2
72.0
67.8
LLaMA2-708
64.7
$3.0
57.3
48.6
77.4
QLORA
Alpuca
64.9
57.0
79.6
74.0
68.3
$2.8
83.3
59.6
$2.2
48.4
78.5
72.5
QLoRA-DropBP
Alpuca
$2,9
60.2
72.4
OLoRA-DroBB
0.75
Alpuca
65.0
55.2
78.8
73.5
67.7
$3.5
83.1
58.9
$1.3
48.2
77.3
72.0
QLoRA+DropBP
0,875
Alpuca
83.7
72.6
QLaRA
Dolly
65.5
58.0
79.7
74.5
68.9
82.8
83.3
58.3
81.2
48.0
77.4
71.8
OLoRA-DropRP
0.5
Dolly
65.1
57.2
79.3
74.1
68.4
82.8
83.4
57.8
81.7
47.6
78.1
OLoRA+DropBP
0.75
Dolly
65.1
57.4
79.7
74.5
68.7
$2.4
83.5
58.4
82.0
48.2
77.8
72.0
QLoRA-DropBP
0.875
Dolly
65.1
56.6
79.6
71.1
68.5
83.1
83.1
57.6
81.6
48.0
78.5
72.0
Table 2: Training time, memory usage, and test score on MT-Bench task when fine-tuning LLaMA3- 8B with DropBP Oasstl datasets.
Method
Mem
Time
Human.
STEM
Role.
Extract.
Writing
Reuson.
Cuding
Math
Avg.
No-tunes
6.25
5.70
5:45
4.85
5.20
4.40
3.20
1.95
4.62
LoRA
57G
7.00
5.70
3.25
5.12
+DropBP (p=0.5)
42G
21m
6.55
6.25
6.05
5.50
5.05
4.45
3.75
3.25
5.11
+DropBP (p=0.75)
36G
17m
6.75
5.90
5.80
5.70
5.35
4,30
3.60
3.30
5.09
+PropBP (p=0.875)
32G
16m
6.60
6.55
5.90
5.50
5.70
3.95
3.40
2,80
5.08
4.2 Main Results: Accuracy and Efficiency
Accuracy on MMLU and Commonsense Reasoning We employ DropBP to accelerate baseline fine-tuning processes, including full fine-tuning (Full-FT), LoRA [1], and QLoRA (18), on the

---

Alpaca and Dolly datasets. As demonstrated in T DropBP achieves accuracy comparable to the baseline, with deviations less than 1 % in all scenarios, and it even outperforms the baseline several instances. Specifically, when DropBP is applied to fine-tune ILaMA2-7B, there is 1 1% or less decrease 5-shot MMLJU accuracy compared to the baseline, while maintaining comparable O-shot commonsense reasoning accuracy. In contrast, for LLaMA2-132 and LLaMA2-70B, fine-tuning with DropBP results in almost no decrease in accuracy on the MMLU and commonsense reasoning lasks, even at the high drop rate of 0.875.
Accuracy on MTT-Bench Similar trends are observed in the MT-Bench tasks, with negligible decreases accuracy as shown in Table Specifically, when fine-tuning LLaMA3-8B on the Oasst dataset, DropBP generates responses of comparable quality to the baseline across various generation tasks. Although scores slightly decrease as the DropBP rate increases, the model fine-tuned with a high DropBP rate of 0.875 still achieves significantly higher scores compared to non-tuned models.
Table 3:
Time required for fine-tuning LLaMA2 models with DropBP on the Alpaca datasets when denotes the target average drop rate. The number of fine-tuning samples is 50K.
DropBr p-0.5
Model
Precision
PEFT
p=O (Baseline)
p-0.75
p-0.875
BF16 mixed
LaRA
2.2h
1.7h
1,4b
1.3h
LLaMA2-7B
BF16
Fall-FT
2.0h
1.3b
1.06
0.8h
L1aMA2-138
BF16
LoRA
2.9h
2.1h
1.7h
1.5h
LLaMA2-70B
BF16
QLoRA
29.6h
22.2h
18.4h
16.5h
Training Speed and Memory Usage We measured the fine-tuning time required to obtain the results in
T as presented in Table When using DropBP to LoRA or QLoRA, training time is reduced by 25%, 38%, and 44% at drop rates of 0.5, 0.75, and 0.875, respectively. In contrast, using DropBP to Full-FT re- sulted in even higher training time reductions of 33%, 50%, and 57% at the same drop rates. These findings align with the theoretical reduction in FLOPs due to DropBP, as detailed in Appendix [ We also con- firmed that DropBP can significantly reduce memory usage during fine-tuning, as shown in Table Specif- ically, while not using DropBP results in memory consumption of 57GB, applying DropBP with a drop rate of 0.875 reduces memory usage 32GB by clim- inating the storage of activation memory for dropped layers. Additionally, we evaluated the convergence speed to reach the same validation perplexity (PPL) on downstream tasks, as illustrated in Fig. 5 and Fig. 10 I in Appendix C The results indicate that our DPo increases training speed by up to 1.5x compared to the baseline in LLaMAA270B..
Baseline
7
DropBP (p=0.5)
DropBP (p=0,75)
DropBP (p=0.875)
2
1.50 speed
0
2
3
4
5
Training time (Hours)
Figure 5: Validation perplexity (PPL) for fine- tuning LLaMA2-70B through QLoRA (base- line) with DropBP on the Alpaca dataset. The p represents the target average drop rate for backward propagation.
4.3 Usability of DropBP
In this section, we evaluate the usability of DropBP, including its ability to train on long sequence data and training throughput in constrained environments, such as a single NVIDIA A100 GPU or Intel Gaudi2 HPU.
Table 4: Available maximum sequence length for fine-tuning LLaMA2-70B using QLoRA with DropBP on NVIDIA A100 GPU, a micro batch size of 1.
Method
QLoRA
w/ DropBP
Drop Rate
0.5
0.75
0.875
Max Seq Len
0.6K
1.2K (2.0x)
2.0K (3.3x) 3.7K (6.2x)
7

---

We first measured the maximum sequence length that could be trained without an Out Of Memory (OOM) on a single NVIDIA A100 GPU. The results in TableeIindicate that our DropBP considerably increases the maximum sequence length, by up to 6.2x the baseline when the drop rate was 0.875. This is because DropBP allows skipping certain layers during backward propagation, eliminating the need to store activations required for calculating parameter gradients of those skipped layers. We believe that this property of DropBP will be particularly useful for fine-tuning LLMs with long-context data [2 23
16
Throughout (eetenncee))
Throughpu: (sentennces))
10
15-
x
Basebe
Drople (y-d 9
Depal (p-d
Dopete (pad 75]
Dopar (pa0 751
Dogdf
Dogett 2705
Batch
Iteratio
(a) Throughput in single A100 GPU.
(b) Throughput in single Gaudi2 HPU.
Figure 6: Throguhput (sentences/s) on single NVIDIA A100 GPU on single NVIDIA A100 GPU
and Intel Gaudi2 HPU when fine-tuning LLaMA3-8B with a sequence length of 512.
We also evaluated training throughput when full fine-tuning LLaMA3-SB using BF16 precision on a single NVIDIA A 100 GPU and an Intel Gaudi2 HPU, increasing the batch size up to the point of OOM errors. As shown in Fig. 6 applying DropBP allows for an increase in batch size per iteration by up 3.3x on the NVIDIA A10O GPU and 5.2 on the Intel Gaudi2 HPU, ensuring high hardware utilization and scalability. Furthermore. DropBP demonstrates a sustained increase in throughput over the baseline at an identical batch size. Ultimately, with a drop rate of 0.875, DropBP achieves throughput of 16.4 sentences/s on the NVIDIA A100 GPU and 28.4 sentences/s on the Intel Gaudi2 HPU, increasing by 79% and 117% over the baseline, respectively.
4.4 Ablation Study
Impact of the Number of Submodules We con- ducted an ablation study to investigate the impact of the number of trainable submodules on the fine- tuning of LLMs. This study compared DropBP, which trains varying submodules randomly at each iteration, with layer freezing, which trains submod- ules composed of only upper layers. Here, the skip rate p denotes the drop rate DropBP and the pro- portion of layers that are frozen in the layer freezing.
Freeze (p-0.5)
(p-0.75)
7-
Freaze (p-0.875)
DropBP (p=0.5)
Drop8P (p=0.75)
Drop8P (p=0.875)
PPL
First, analyzed the number of submodules trained by layer freezing and DropBP. In the case of layer freezing, the lower 2np layers are frozen and only the remaining 2n(1 p) layers are trained. In this case, the number of trainable upper submodules is 22m(1-p), In contrast, DropBP randomly drops 2np layers at each iteration, allowing it to train all sub- modules with a depth of 2n(1 p) or less without the restriction of training only the submodules com- posed of the upper layers. In this scenario, since the number of different submodules at depth in the entire network 2n C₁ DropBP can train Σ 22((1-)) 2n 2. submodules.
0.4 10
10"
x
1.2 1027
FLOPs
Figure 7: Validation perplexity (PPL) for fine- tuning LLaMA2-7B through LoRA with layer freezing or DropBP on the Alpaca dataset.
As shown in Table 5 when fine-tuning ILaMA2-7B using layer freezing or DropBP with a high skip rate of 0.875, observed significant 1.8% decrease in accuracy with layer freezing compared to

---

the baseline, while DropBP exhibited relatively smaller accuracy decrease of 1.0%. Furthermore, as illustrated in Fig. 7 the convergence speed to the same validation PPL on the downstream task is much slower for layer freezing compared to DropBP, especially at high skip rates, where it converges even more slowly than the baseline. We believe this is due to the ability of DropBP to train a relatively larger number of submodules 2 c4C;), compared to the fewer submodules trained by layer freezing (2*). Moreover, when fine-tuning LLLAMA-70BB, DropBP resulted in a 0.5% increase in MMLU 5-shot accuracy compared to the baseline, despite high skip rate of 0.875. This improvement due to the large number of layers in LLaMA2-70N enabling DropBP to train deeper and more
numerous submodules (Ee 160 C2) even with a high skip rate of 0.875.
Table 5: The number of submodules being trained and test accuracy on the 5-shot MMLU tasks with layer freezing DropBP the Alpaca datasets.
LLaMA2-7B
LLaMA2-70B
Method
LoRA
Foeeee
LoRA+DropBP
QLoRA
QLoRA+DropBP
Drop Rate
0.875
0.875
0.875
of Submodules
200
2
Σ C.
2180
Li=o C60
Accuracy (%)
44.7
42.9
(-1.())00)
68.3
68.8 (+0.5)
Impact of Sensitivity- Drop Rate We also conducted an ablation study analyze the effectiveness of sensitivity-based drop rate al- locations. First, we identified the sensitivity of different layers by calculating Eq. 6 during the training of LLMs in various scenarios, as illus-
Table 6: Test accuracy on the 0-shot commonsense reasoning tasks when fine-tuming LLaMA2-7B and 13B through LoRA with DropBP at uniform or sensitivity- bbssed drop rate on the Alpaca datasets. The target average drop rate is 0.875.
trated in Fig. Sa and Fig. 7 in Appendix D While the distribution varies slightly depending on the number of parameters, fine-tuning ap- proach, and target average drop rate, there is consistent tendency to assign importance to both the initial and final layers. Consequently, drop rates for these layers are allocated to be lower by a simple greedy algorithm, as explained in Section
LLaMA2 LR
7B
13B
le-4
3e-4
le-4
3e-4
LoRA
65.7
66.0
68.2
68.0
DropBBP (uniform)
66.4
63.1
65.8
+DropBP (sens)
66.6
64.7
67.7
67.3
Additionally, we fine-tuned the LLaMA2-78 7B and 13B using DropBP on Alpaca datasets, comparing sensitivity-biat allocated drop rates with uniform drop rate. detail, we compared the average accuracy of commonsense reasoning tasks when fine-tuning the models with learning rate of le-4 and 3e-4, as shown in Table Note that the PPI. for fine-tuning LLaMA2-7B in Fig. 8b corresponds to learning rate of 3e-4. The results indicate that sensitivity- based drop rates achieved 1.6% higher
uniform (p-0.3 0.855)
sensitivity (p=0.875)
7-
Sensitivity
Drop Rate
6
0.4
0.5
5-
4-
00x100-1
0.2
3-
2
0
10
20
30
40
50
60
1.2 10
2.4 1
3.6 1
Layer Index
FLOPs
(a) Distribution of drop rates determined by sensitivity when (b) Validation PPL with uniform and the average drop rate is set to 0.875.
sensitivity-based allocated drop rates.
Figure 8: Distribution of drop rates and the validation PPL when fine-tuning LLaMA2-7B through LoRA with DropBP at uniform or sensitivity-basee drop rate on Alpaca datasets.

---

accuracy compared to uniform drop rates with relatively high learning rate of 3e-4, while there was no significant difference in accuracy when the learning rate was set to le-4 in LLaMA2-7B. Fig. also shows that sensitivity-based drop rates consistently stabilized the convergence of validation loss, whereas uniform drop rates occasionally diverged when the learning rate was set to 3e-4 in LLaMA2-7B. This phenomenon is even more pronounced with LLaMA2-13B, resulting in a 1. 1% increase in accuracy through sensitivity-baat
drop rate allocation, even with a low learning rate of le-4. In other words, sensitivity-basee based drop rate allocation helps stabilize the training process, especially in the case of large learning rates or larger models.
Related Works
Parameter-efficient fine-tuning When fine-tuning LLM, substantial amount of memory is required to store parameters, gradients, and optimizer states, LoRA (I4] successfully reduces the memory allocated to gradients and optimizer states by inserting trainable rank decomposition matrices into the linear layers of the model while keeping the original LLM parameters frozen. LLaMA-Adee Addater [15] and L.aMA-AAaapte V2 [16] significantly reduce training memory using trainable adoption prompts and zero- prrroinitialized attention mechanisms. Some studies attempt to reduce not only the memory footprint of gradients and optimizer states but also that of parameters by considering quantization. PEQA 20), for instance, quantizes the original LLM parameters into a low-bit format and fine-tunes only the scale factor, thus saving memory for parameters during training. QLoRA [88 and QA- LoRA [1), built upon LoRA, also employ quantization on the original LM parameters, significantly reducing parameter memory during training. Our DropBP is orthogonal and easily combinable with these PEFT techniques, enabling memory and computationally efficient fine-tuning.
Parallelism Parallelism techniques are widely used to accelerate training LLM using multiple GPU efficiently. Data parallelism [42 is a technique that involves dividing data along the batch dimension for training across multiple GPUs, which still requires sufficient memory load the entire model on each GPU. Conversely, tensor parallelism
[4] 45) partitions the model across GPUs, dividing
matrix multiplication operations column wise and row-wise. Pipeline parallelism [46 47 48 involves partitioning the model depth-wise across GPUs, which enables efficient pipeline scheduling. The
Zero Redundancy Optimizer (ZeRO) [4 and Fully Sharded Data Parallelism (FSDP) (50) shard parameters, gradients, and optimizer states across multiple GPUs, retrieving parameters when needed to restore their non-partitioned form, enabling the overlapping of computation and communication during training. While these parallelism techniques are designed to efficiently manage the massive computational costs across multiple GPUs, our DropBP specifically aims reduce the inherent computational costs required for training process.
Layer dropping Stochastic Depth [5]], the first approach to randomly drop layers during neural network training, reduces overfitting and costs in image recognition. Layerdrop (52) randomly drops layers during training and selectively uses some layers during inference, accelerating both processes for transformers. Progressive Layer Dropping (PLD) (53) progressively increases the drop rate across depth and iterations, improving training speed without accuracy degradation transformers. These techniques speed up pretraining of small transformer models like BERT (54) by dropping layers during the entire training process, whereas DropBP, specific to fine-tuning LLMs, drops layers only during backward propagation. Consequently, as detailed in Appendix EE our DropBP achieves higher performance compared to these layer dropping when fine-tuning LLMs.
Conclusion
We propose DropBP, an effective algorithm that accelerates the fine-tuning of LLMs by randomly dropping layers during backward propagation, which can be orthogonally integrated into both full- fine tuning and parameter-efficient fine-tuning. We developed the DropBP library user-friendly Py PyTorch extension to facilitate easy integration into existing training codes. Experimental results demonstrate that DropBP significantly accelerates training speed during the fine-tuning of LLMs, achieving comparable aceuracy to baseline fine-tuning. Furthermore, DropBP reduces activation memory, enabling long-context training and increasing batch size on limited resources. Consequently, applying DropBP enables 79% higher throughput on an NVIDIA A A00 GPU and a 11 117% higher throughput on an Intel Gaudi2 HPU.
10

---

Acknowledgment
This research was supported part by the NAVER-Inte Co-Lab. The work was conducted by Seoul National University and reviewed by both NAVER and Intel.
References
[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Neurips, Long Beach, CA, USA, December 4-9, 2017., pages 5998-6008, 2017.
[2] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08
2020.
[3] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, NeurIPS, New Orleans, LA, USA November 28- December 9, 2022, 2022.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners, In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Seean-Tienn! Lin, editors, NeurIPS, virtual, December 6-12, 2020, 2020.
[5]
OpenAL. GPT-4 technical report. CoRR, abs/2303.087744 2023.
[6] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anails White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2312. 18805, 2023.
[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 13971, 2023.
[8] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien
11

---

Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. CoRR, 0tbb/2307.09288.. 2023.
[9] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Sarcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy. Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Iharra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lec, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadcokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang. Jiccao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2407.2177 21783, 2024.
[10] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In ICLR, virtual, April 25-29, 2022. OpenReview.net, 2022.
[11] Rohan Taori, Ishaan Guirajani. Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. ord aaca 2023.
https://gt thub. http:/tttsuubbbbttaaeeeeaa
[12]
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. CoRR, 1e/22305.11206. 2023.
[13] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 1lm. Technical report, Databricks, 2023.
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen AZu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, April 25-29, 2022, virtual. OpenReview.net, 2022.
[15] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Lllam--dapter: Efficient fine-tuning of language models with zero-init attention. CoRR, abs/2303.16199 2023.
[16] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter V2: parameter-effiiinn visual instruction model. CORR, abs/2304.1 15010, 2023.
[17] Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park. Byeongwook Kim, Lung-Woo Ha, Nako Sung, and Dongsoo Lee. Alphatuning: Quantization- aware parameter-efficient adaptation of large-scale pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, EMNLP, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 3288-3305. Association for Computational Linguisties. 2022.
[18] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305.1431 2023.
[19] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantization-awaz low-rank adaptation of large language models. CoRR, abs/2309.14717. 2023.
[20] Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficien fine-tuning of compressed large language models via sub-4-bit integer quantization. CoRR, 14152, 2023.
12

---

[21] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back-proper propagating errors. nature. 323(608) 1986.
[22] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. CoRR, abs/2309. 12307, 2023.
[23]
Wenhan Xiong, Jingyu Liu, Igor Molybog, Hejia Zhang, Prajjwal Bhargava, Rui Hou, Louis Martin, Rashi Rungta, Karthik Abinav Sankararaman, Barlas Oguz, Madian Khabsa, Han Fang, Yashar Mehdad, Sharan Narang, Kshitiz Malik, Angela Fan, Shruti Bhosale, Sergey Edunov, Mike Lewis, Sinong Wang, and Hao Ma. Effective long-context scaling of foundation models. CoRR, abs/2309.16033 2023.
[24] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, NeurlPS 2019, Vancouver, BC, Canada, December 8-14, 2019, pages 8024-8035, 2019.
[25]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, virtual, Austria, May 3-7, 2021. OpenReview.nel, 2021.
[26] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAI, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020.
[27] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Marquez, editors, ACL, Florence, Italy, July 28- August 2, 2019, pages 4791-4800. Association for Computational Linguistics, 2019.
[28] Sumithra Bhakthavatsalam, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, and Peter Clark. Think you have solved direct-answer question answering? try arc-da, the direct-answer A12 reasoning challenge. CoRR, abs/2102.03311. 2021.
[29] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii, editors, EMNLP, Brussels, Belgium, October 3/ November 4, 2018, pages 2381-2391. Association for Computational Linguistics, 2018.
[30] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In AAAI, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020.
[31] Martin Svedin, Steven W. D. Chien, Gibson Chikafa, Niclas Jansson, and Artur Podobas. Benchmarking the nvidia gpu lineage: From early k80 to modern a100 with asynchronous memory transfers. arXiv preprint arXiv:2106. 04979, 2021.
[32] Intel Corporation. Intel gaudi2 ai accelerators white paper. Technical report, Intel Corporation, 2023.
[33] Henry J Kelley. Gradient theory of optimal flight paths. Ars Journal, 30(10):947-954, 1960.
[34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[35] Andreas Veit, Michael J. Wilber, and Serge J. Belongie. Residual networks behave like ensembles of relatively shallow networks. In Daniel D. Lee. Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 550-558, 2016.
[36] Sunghyeon Woo, Sunwoo Lee, and Dongsuk Jeon. ALAM: Averaged low low-precision activation for memory-effiiient training of transformer models. In The Twelfth International Conference on Learning Representations, 2024.
13

---

[37]
Lightning-AL httggh....ttp///..b..o.o.ggggggn AI/Lit- gpt 2023.
https:// wwwp://oo//nnnnggg////A.A
[38]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony
Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers:
State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019.
[39] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023.
[40] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arna Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[41] Lianmin Zheng, Wei-Linn Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph F. Gonzalez, and lon Stoica. Judging Ilm-as-a-judee with mt- mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[42] Shen Li. Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow., 13(12):3005-33 3018, 2020.
[43] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.0805. 2019.
[44] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, and Bryan Catanzaro. Using deepspeed and megatron to train megatron-turre turrin NLG 530b, A large-scale generative language model. CoRR, abs/2201.11990, 2022.
[45] Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Moham- mad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large transformer models. CoRR, abs/2205.00 005000550519... 2022.
[46] Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Xu Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V. Le, Yonghui Wu, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alcéé-Buc Buc, Emily B. Fox, and Roman Garnett, editors, NeurlPS 2019, Vancouver, BC, Canada, VDecember 8-14, 2019, pages 103-112, 2019.
[47] Aaron Harlap, Deepak Narayanan, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger. and Phillip B. Gibbons. Pipedream: Fast and efficient pipeline parallel 2018.
DNN training. CORR,
[48] Taebum Kim, Hyoungjoo Kim, Gyeong-In Yu, and Byung-Gon Chun. Bpipe: Memory-balanced pipeline parallelism for training large language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, ICML, Honolulu, Hawaii, USA, 23-29 July 2023, volume 202 of Proceedings of Machine Learning Research, pages 16639-16653, PMLR, 2023.
[49] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory opti- mizations toward training trillion parameter models. In Christine Cuicchi, Irene Qualters, and William T. Kramer, editors, SC 2020, Virtual Event Atlanta, Georgia, USA, November 9-19, 2020, page 20. IEEE/ACM, 2020.
14

---

[50] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania. Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16(12:334883386) 2023.
[51] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV Amsterdam, The Netherlands, October 11 11-14, 2016, volume 9908 of Lecture Notes in Computer Science, pages 646-661. Springer, 2016.
[52] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. In ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[53] Minjia Zhang and Yuxiong He. Accelerating training of transformer-basee language models with progressive layer dropping. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, NeurlPS, virtual, December 6-12, 2020, 2020.
[54] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Pupers), pages 4171-416 Association for Computational Linguistics, 2019.
[55] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 770-778. IEEE Computer Society, 2016.
[56] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garefa, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR, Ahs/1710.037445 2017.
[57] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[58] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts, In ICL, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.ned 2017.
15

---

Appendices
A The importance of short paths in residual networks
we interpret transformer models as collection of numerous blocks, each composed of various modules with residual connections. Our hypothesis is that we can fine-tune LLMs well by training only certain shallow submodules. To theoretically analyze this hypothesis, we measured the impact of submodules based their path lengths in LLaMA2-7B, shown in Fig. 9 Specifically, we followed these steps:
In Section 3.1
We first perform forward pass through the entire network.
During the backward pass, randomly sample residual blocks, which are back- propagated without passing through skip connections, while the remaining n - k blocks are bypassed through the skip connections.
We then measure the norm the gradient at the input.
We take 100 measurements for each path length k. Subsequently, we multiply by the distribution of all possible path lengths, which follows a Binomial distribution, to quantify the gradient contribution from paths of specific length.
In Fig- 9b observed that the gradient per path length decreases the path length increases. Consequently, Fig. 9c demonstrates that shorter path lengths have greater impact on the gradient in LLaMA2-7B. These observations are consistent with the existing findings [35] ResNet (55), which attributed this phenomenon to vanishing gradients. Therefore, our DropBP enables effective training LLMs by focusing on training important short submodules.
24
Distribution Path Length
Gradient Megnitude at lrrput
Gradient Magnitude per Path Length
Mognttude
Total Gradient Magnitude per Path Length
Number of Pashs
32-4
30 40
Path Length
Path Length
Path Length
(a) Distribution of path length.
(b) Gradient per path length.
(c) Total gradient per path length.
Figure 9; The impact of path length for fine-tuning LLaMA2-7B.
B Theoretical FLOPs and Actual Training Time Using DropBP
In this section, we calculate the theoretical FLOPs reduction afforded by DropBP and compare this reduction to the actual training time reduction shown in Table As outlined in Section 2 the computational costs arise from output activation calculations by Eq. [ during forward propagation, and input and parameter gradient caleulations by Eqs. 2 during backward propagation. We denote the FLOPs for these operations as Fouts F Fgrad. and F Epoaras respectively. Therefore, the total FLOPs for the backpropagation algorithm be calculated by the following equation:
Fr = Ffw Fow
Fout + F Fyrud Fparam
(8)
where Fr represents the FLOPs during the entire training process, Fyw for forward propagation (i.e. F Fout), and Fow for backward propagation (i.e. Fro. = F Fgrad + Fparam). DropBP reduces FL.OPs for backward propagation by target average drop rate (Pavg). Therefore, total FLOPPs in
DropBP can be formulated as below:
16

---

Fr. = Fiw + (1 Pavg) Fbw
Fott Pavg) ( grad + F param)
(9)
Consequently, the theoretical FLOPs reduction ratio by DropBP can be represented as follow:
Pang (F + parram)
Reduction Ratio by DropBP:
(10)
Fout + Fgred pararm
Note that in full fine-tuning (Full-FT), the computational costs for output calculations, input gradient calculations, and parameter gradient calculations are nearly identical (i.e., Fout grad peram) Conversely, in purameter-efficient fine-tuning techniques (PEFT) such as LoRA and QLoRA, the costs of calculating parameter gradients are negligible (Fout grad: Fporam 0) due very small number of trainable parameters and the freezing of original LLM parameters. By substituting this into Eq. [O] the theoretical FLOPs reduction ratio by DropBP can be expressed as:
Reduction ratio in Full-FT:
3 Pavg
(11)
1
Reduction Ratio in PEFT:
Pang
(12)
Therefore, with target average drop rates of 0.5, 0.75, and 0.875, DropBP achieves theoretical FLOPs reductions in Full-FT of 33%, 50%, and 58%, respectively, according to Eq. [T This aligns with the actual training time reduction when utilizing DropBP in Full-FT shown in Table 3 and Table 7 This trend is also evident when utilizing DropBP in LoRA and QLoRA. According to Eq. 12 the reductions in FLOPs for various target average drop rates of 0.5, 0.75, 0.875 derived 25%, 38%, and 44%, respectively. This closely aligns with the actual training time reductions observed when DropBP is applied to LoRA and QLoRA A demonstrated in Table 7
Table 7: Training time (ms) per iteration for sequence length of 512 through Full-FT, LoRA or
QLoRA using DropBP. Mixed refers to mixed precision training (56) using BF (BF16) and
32-bit. MBS is denoted as the micro batch size. FW, BW, and Total respectively denote the time consumed for forward propagation, backward propagation, and the entire training process.
Model
Method
Precision
MBS
Drop Rate
FW
Total
LoRA
Mixed
159
320
Drop+
Mixed
0.5
159
81 (-50%)
239 (-25%)
LroA+
Mixed
0.75
159
43 (-74%)
201 (-37%)
LoRA+DropBP
Mixed
0.875
158
(-86%)
181 (-43%)
Full-FT
BF16
2
0
91
192
283
1LaMA2-7B
Full-FT+DropBP
BF16
0.5
91
98 (-49%)
189 (-33%)
Full-FT+DropBP
BF16
0.75
(-73%)
(-50%)
Full-FT-DropBP
BF16
0.875
91
30 (-85%)
121 (-57%)
LoRA
BF16
0
186
236
423
LrRp+
BF16
0.5
186
(-50%)
306 (-28%)
1LaMM2-138
LoRA+
BF16
0.75
187
64 (-73%)
251 (-41%)
LoRA+DropBP
BF16
0.875
187
33 (-86%)
219 (-48 %
QLoRA
BF16
1033
1100
2133
QLoRA+DropBP
BF16
0.5
1034
566 (-49%)
1599 (-25%)
LLaMA2-70B
QLoRA+DropBP
BF16
0.75
1033
290 (-74%)
1323 (-38%)
QLoRA+ DropBB
BF16
0.875
1032
158 (-86%)
1191 (-44%)
17

---

C Convergence Speed Up Using DropBP
Baseline
8
Drop@P (p=0.33
DropBP (p-0.30 5)
DroptP (p=0.75)
7-
DropUP (p-0.75)
Dropti (p=0.875)
Dropti (p-0.075)
6-
5
4-
3-
1.47 speed up
1,44 X speed up
10
0.8 102
1
0.4 x 10TH
0.8 102?
12x
FLOPs
FLOPs
(a) LLaMA2-7B w/ LoRA (Alpaca)
(b) LLaMA2-7B w/ LoRA (Dolly)
Baseline
Baseline
Drop@P (p-0.5)
DropBP (p-0.75)
Dropt? (p-0.75)
DropBP (p-0.8775)
DroptiP (p=s0.075)
5-
3-
1.58 speed up
0.6 1
10
108
x 10
12x10
10
FLOPs
FLOPs
(c) LLaMA2-7B w/ Full-FT (Alpaca)
(d) LLaMA2-7B w/ Full-FT (Dolly)
Baseline
DropBP (p-0.5)
7
(p-0.5)
DropOP (p=0.75)
DropdP (p-0.75)
DropSP (p=0.575)
Drop@P (p=0.875)
5-
4-
1.40 X speed up
1.44 upeod
0.8 10
1.6 1017
20
0.8 102
1.6 x5x100 10Ar
10
FLOPs
FLOPs
(e) LLaMA2-13B w/ LoRA (Alpaca)
(1) LLaMA2-13B w/ LoRA (Dolly)
Baseline
Baseline Drapti (pe0.5)
DroptP (pe0.5)
7.
DrapaP (pm0.75)
DropSP (p-0.875)
DropBP (p-0.875)
5-
PPL
4-
3-
speedd up
2-
1.41 up
10
10
12
1018
10
122
FLOPS
FLOPS
(g) LLaMA2-70B w/ QLORA (Alpaca)
(h) LLaMA2-70B w/ QLORA (Dolly)
Figure 10: Validation perplexity (PPL) when fine-tuning LLaMA2 models through Full-FT, LoRA, or QLoRA using DropBP on the Alpaca and Dolly datasets.
18

---

Develne
Deael
Dropti 1p
Dropap (p.0..)
Dropti (pen 731
Gropdh (pi0.77)
Dropti (p....875)
Groppti (p:0.
2000
4000
6000
Training time (Hours)
(a) Perplexity curve training steps.
(b) Perplexity curve training time.
Figure 11: Training curves across training steps and time for fine-tuning LLaMA2-70B through QLoRA with DropBP on the Alpaca datasets.
When analyzing training curves across training steps in Fig. Ila the convergence of loss per step drop rate of 0.5 is almost identical the baseline. However, with drop rates of 0.75 and 0.875, the convergence speed per step is slower compared to baseline. Nonetheless, DropBP significantly reduces the time consumed per training step, because it skips the backward propagation computations for the dropped layers. Consequently, the convergence speed per training time is actually faster for DropBP compared the baseline as shown Fig. [lbb
D Distribution of Drop Rates Determined by Sensitivity
12
16
05x
Sensttvity
Drop Rate
Sensitiiity
Drop Raite
8Ax
b4x10-1
20
Layer
Layer
LLaMA2-78 LoRA DropBP (p=0.5)
(b) LLLMM2--7 7B LoRA + DropBP (p=0.875)
14x
Sensibuity
Drap Rate
Sensibiity
Drop Rate
LEx
30
08'x10"
Index
Index
(c) LLaMA2-7B w/ Full-FT DropBP (p=0.5)
(d) LLaMA2-7B w/ Full-FT DropBP (p=0.875)
19

---

24x
Sensitivity
Drop Rate
Drop Rate
Layer Index
Index
(e) LLaMA2-70B w/ QLoRA + DropBP (p=0.5)
(f) 1LaMA2-70B QLoRA DropBP (p=0.875)
Figure 7: The distribution of drop rates determined by sensitivity when fine-tuning LLaMA2 through Full-FT, LoRA, or QLoRA using DropBP on Alpaca datasets.
E
Comparisons between Layer Dropping and DropBP fine-tuning LLMs
In this section, we compare Layerdrop (LD) [52] and Progressive Layer Dropping (PLD) [33 with DropBP under the same LLMs fine-tuning scenario. We set the relative FLOPs of LD and PLD to 0.75 of the baseline (LoRA), which corresponds to the same relative FLOPss when the drop rate of 0.5.
DropBP is
As shown in Fig. 8 our DropBP converges faster to the same validation PPL compared to LD and PLD. Moreover, as seen Table DropBP achieves com- parable accuracy to the baseline even with relative FLOPs of 0.56, whereas LD and PLD experience a significant accuracy drop of over 5% with relative FLOPs 0.75. We believe this difference arises from the high sensitivity of forward propagation through- out the fine-tuning process. Specifically, layer drop- ping techniques omit certain layers of well-pretrained LLMs during forward propagation, resulting in sig- nificant output deviations that adversely impact the loss and the overall training process. Conversely, DropBP maintains all layers during forward prop- agation, thereby ensuring precise outputs and loss calculations, which facilitate stable training. Please note that, as explained in Section S LD and PLD are designed to accelerate the pretraining of small trans- former models (SLMs) like BERT by dropping layers throughout the entire training process while DropBP only focuses on fine-tuning LLMs. In future stud- ies, we will explore whether DropBP can similarly accelerate the pretraining of transformer models and investigate ways to improve its effectiveness.
Baseline
LD (p:0.25)
PLD (ped 25)
Drsptit (0m0.8755
PPL
2
0.4 10
0.8 10
x12
FLOPs
Figure 8: Validation perplexity (PPL) for fine- tuning LLaMA2-7B through LoRA (baseline) with LayerDrop (LD). Progress Layer Drop- ping (PL.), or DropBP on the Alpaca dataset. The represents the target average drop rate for backward propagation in DropBP.
Table 8: Test accuracy the O-shot commonsense reasoning tasks when fine-tuning LLaMA2-7B through LoRA with layerdrop (LD), progressive layer dropping (PLD), and DropBP.
LoRA+ DropBB
Method
LoRA (baseline)
LoRA+LD
LoRA+PLD
p=0.5
p=7.75
p=0.875
Relative FLOPs
1.00
0.75
0.75
0.75
0.63
0.56
Accuracy (%)
66.0
58.7
61.0
65.9
66.1
66.4
F
Experimental Details
In our experimental setup, the AdamW (57] optimizer and a cosine annealing learning rate scheduler were utilized common settings. LoRA 14 and QLoRA [88] were integrated to every linear
58
20

---

layer of our model, with the LoRA parameters r and a set to 8 and 16, respectively. We experimented with all the learning rates presented in Table I and reported the best accuracy achieved in Table
Table 9:
Detailed Setup for Table BS and MBS are denoted as the batch size and micro batch
size, respectively. Mixed refers mixed precision training [56] using 16 (BF16) and 32-bit.
Fine-tuning
Dataset
# Iterations
BS
MBS
Precision
Learning rate
Alpaca
25K
128
2
Mixed
1c-4, 3e-4
LoRA
Dolly
7K
128
Mixed
1c-4, 3e-4
LLaMA2-7B
Alpaca
25K
128
2
BF16
3e-4"
Full-FT
Dolly
7K
128
2
BF16
le-4, 3e-4
Alpaca
25K
BFI6
le-4, 3e-4
LLaMA2-13B
LoRA
Dolly
7K
128
2
BF16
le-4, 3e-4
Alpaca
25K
128
BF16
le-4, 3e-4
LLaMA-30B
QLoRA
Dolly
7K
128
BF16
lc-4, 3e-4
Alpaca
50K
128
BF16
5e-5, le-4
LLaMA2-70B
QLoRA
Dolly
14K
128
BF16
5o-5, le-4
LLaMA3-8B
LoRA
Oasstl
2.5K
16
4
BF16
3c-4, 5c-4
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We appropriately present the contributions of the paper in the Abstract and
Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In Section 5 and Appendix B we clearly state that DropBP is developed
specifically for fine-tuning, unlike other layer dropping techniques, and we indicate plans for further improvements future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [Yes]
Justification: In Appendix we mathematically calculate the theoretical reduction FLOPs achieved by DropBP. Through experiments presented in Section 4.2 and Appendix [ confirm that this theoretical reduction closely matches the actual training time reduction.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
we provide detailed information to ensure reprodueibility, and in the abstract, we present the anonymous code in Abstract to implement this.
Justification: In Section 4.1 and Appendix F
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We present the anonymous code for reproducibility the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. Siiaps.cccpubliee wwtppbbiccgguueeeeeoooooocpppped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental details in Section 4.1 Appendix and the anonymous code presented in the Abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Although we conducted experiments with multiple seeds and report the average values, we did not include error bars or statistical information because the deviations were minimal, and including them would detract from the clarity of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the types of devices used for the experiments in Section 4.1 and provide the time required reproduce the experimental results in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpp: http://wwwppppccoooooo...cccctaa
Answer: [Yes]
Justification: Our paper adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since we discuss a efficient fine-tuning algorithm for LLMSs we believe that our work does not have any direct negative societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is related to the efficient fine-tuning of LLMs, and therefore, there are specific safeguards described for the responsible release of data models, as the nature of our research does not involve high-risk misuse scenarios.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately cite the original paper and existing codes Section 4.1 and code in Abstract.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
26

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We provide the new code with a proper license as an anonymized URL in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
27

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "DwLTvyVPUV", "reviewer": "Reviewer_nmtJ", "summary": "The paper proposed to drop layers during backward prop (BP) based on layer sensitivity. The method aims to reduce the cost for gradient computation and storage for intermediate activation in full BP.", "questions": "1. What is the long context modeling performance after applying DropBP?\n2. Could the authors present Figure 5 with # of steps as the x-axis to demonstrate faster convergence?\n3. I wonder if the sensitivities would evolve, and the drop rate needs to be re-allocated through training.", "limitations": "The authors have addressed the limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Reducing the cost of full BP in PEFT has been an important challenge. \n2. The method is simple and is easy to integrate to either full fine-tuning or PEFT. \n3. Experiments demonstrate that DropBP can speed up the training while retaining the accuracy. The resulting memory reduction makes longer sequence modeling accessible.", "weaknesses": "1. The idea of optimizing NNs with sparse gradient is not new. This paper needs to add more discussion and comparison with related works in sparse learning e.g., [1-3]\n2. Table 1 only shows results on two datasets and limited benchmark. It is unclear if the method works well for generation tasks and domain-specific transfer learning.\n3. It is unclear which algorithm is used to solve the constraint minimization problem, i.e., to determine the layer-specific rates based on sensitivity, and its extra computational cost.\n4. (Minor) In fine-tuning, DropBP drops a set of layers. However, the sensitivity of a set of layers may not be accurately represented by the direct summation of the sensitivities of individual layers in the set.\n\n[1] Sun, Xu, et al. \"meprop: Sparsified back propagation for accelerated deep learning with reduced overfitting.\"\n[2] Sung, Yi-Lin, Varun Nair, and Colin A. Raffel. \"Training neural networks with fixed sparse masks.\" \n[3] Brock, Andrew, et al. \"Freezeout: Accelerate training by progressively freezing layers.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reviewing our submission and providing valuable feedback. Please see below for our response to the questions and comments.\n\n**Q1.1.**  The idea of optimizing NNs with sparse gradient is not new. \n\n**A1.1.**  We acknowledge that the idea of optimizing neural networks with sparse gradients is not novel. However, our method differs significantly from sparse gradient methods like meProp, FISH Mask, and Freezeout in terms of purpose, methodology, and effect. Specifically:\n\n- ***meProp***: While meProp accelerates training by masking the output gradient of individual layers, **our DropBP skips entire transformer blocks by leveraging skip connections.** As shown in ```Table G.1.2``` in global response, **our DropBP achieves high accuracy compared to meProp.** Furthermore, meProp requires computing top-K gradient masks at each iteration, **whereas DropBP only needs to calculate the sensitivities once for the entire training process,** reducing the overhead on training time. Additionally, meProp must store activations for all layers during training, while **DropBP saves activation memory by not storing activations for dropped layers**.\n- ***FISH Mask***: FISH Mask, which is a parameter efficient fine-tuning such as LoRA, reduces communication costs by updating sparse parameters without decreasing FLOPs. In contrast, **our DropBP reduces computational costs directly by dropping backward operations.**  Furthermore, while FISH Mask have to store all activations for backward propagation, **our DropBP eliminates the need to store activations for dropped layers, reducing activation memory.** DropBP and FISH Mask employ complementary methods, which makes it possible to **apply them simultaneously,  just as DropBP and LoRA can be.**  We are working on this implementation but facing delays due to the complexity of FISH Mask's code. We will include related experiments in the paper after completion.\n- ***Freezeout***: Freezeout accelerates training by gradually freezing earlier layers, while **DropBP randomly drops layers from the start, regardless of their order.** Consequently, Freezeout requires storing all activations initially, which complicates increasing sequence length and parallel processing. In contrast, **DropBP maintains low and consistent memory allocation** as shown in ```Table G.1.2```, facilitating easier management, longer sequences, and better parallel processing.\n\nWe will include these distinctions and a more comprehensive comparison to suggested related works in our revised paper.\n\n**Q1.2.**   It is unclear if the method works well for generation tasks and domain-specific transfer learning.\n\n**A1.2.** Please refer to the global response **GA1** above.\n\n\n**Q1.3.**   It is unclear which algorithm is used to solve the constraint minimization problem, i.e., to determine the layer-specific rates based on sensitivity, and its extra computational cost.\n\n**A1.3.**  Please refer to the global response **GA2** above.\n\n**Q1.4.**  (Minor) The sensitivity of a set of layers may not be accurately represented by the direct summation of the sensitivities of individual layers in the set.\n\n**A1.4.** We agree with the reviewer's opinion that the total sensitivities of the dropped network do not strictly equal the sum of each individual layer's sensitivity. However, **given the practical constraints of calculating the sensitivities for the vast number of possible combinations of dropped networks in a deep neural network,** we have made the assumption that the total sensitivities of a dropped network can be approximated by the sum of the sensitivities of its individual layers. In our future research, we will explore more accurate methods to calculate the network's sensitivity and determine optimal drop rates.\n\n**Q1.5**. What is the long context modeling performance after applying DropBP?\n\n**A1.5**. \n+ ```Table R.1.1.``` *Processing Time Analysis for Calculating Sensitivities When Fine-Tuning LLMs on the Alpaca Dataset.*\n\n| No-tune | DropBP (p=0.875) | | \n|---------|---|---| \n|  | 16K|32K|\n | NaN | **6.81**|8.32|\n\n+ ```Figure A``` *in the attached PDF*.\n\nIn response to the reviewer's request, we trained the LLaMA2-7B-chat model on the LongAlpaca dataset using sequence lengths of 16K and 32K. The settings followed those of LongLoRA [1]. As shown in ```Fig. A```, our experiments demonstrated that **the model successfully converged on loss with long sequence data of 16K or more**. Additionally, when evaluating the fine-tuned model on a subset of the PG16 test set with a sequence length of 16K, we **achieved lower perplexity (PPL) compared to the non-fine-tuned model** as shown in ```Table R.1.1```, confirming that our **DropBP method enables effective long sequence modeling.**\n\nDue to limited GPU resources and review time, we conducted experiments on smaller training and test sets. Once we secure sufficient resources and time, we plan to obtain more robust experimental results for the revised version and integrate DropBP into LongLoRA for future work..\n\n**Q1.6** Could the authors present Figure 5 with # of steps as the x-axis to demonstrate faster convergence?\n\n**A1.6** Please refer to the global response **GA3** above.\n\n**Q1.7** I wonder if the sensitivities would evolve, and the drop rate needs to be re-allocated through training.\n\n**A1.7** \n+ ```Figure D``` *in the attached PDF*.\n\nAs shown in ```Fig. D```, our experiments demonstrated that **the sensitivity of each layer converges as training progresses.**  Accordingly, we calculates sensitivity just once at the 10% of the training process, minimizing overhead from sensitivity calculations. This approach proved effective in most experiments. However, we believe the reviewer's concern is also valid and  we **plan to add a feature that periodically recalculates sensitivity to adjust the drop rate to the DropBP library.**\n\n[1] Chen at al., \"LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZcNgB8amun", "reviewer": "Reviewer_Ep4o", "summary": "The paper proposes a novel method to reduce the computational and memory costs associated with fine-tuning large language models (LLMs). The authors introduce DropBP, a technique that randomly drops layers during backward propagation, effectively reducing the computational operations (FLOPs) and activation memory needed. This method assigns drop rates based on the sensitivity of each layer to ensure stable training. The approach is applicable to both full fine-tuning and parameter-efficient fine-tuning (PEFT) methods. The paper reports significant improvements in training time, convergence speed, and maximum sequence length when fine-tuning LLaMA2 models with DropBP.", "questions": "- Can you provide more details on the sensitivity calculation process? Specifically, how is the sensitivity of each layer computed, and what is the computational overhead associated with this step?\n\n- What are the best practices for tuning the drop rates in DropBP? Are there guidelines or heuristics that practitioners can follow to optimize performance for their specific use cases?\n\n- How well does DropBP integrate with other recent advancements in efficient training techniques, such as mixed precision training or distributed training frameworks? Have you explored these combinations in your experiments?", "limitations": "Limitations have been discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- DropBP introduces a novel method for reducing the computational and memory costs associated with fine-tuning LLMs. This is an important contribution to the field, given the increasing size and complexity of these models.\n\n- The paper provides empirical evidence that DropBP significantly reduces training time (by 44%), accelerates convergence (1.5× faster), and increases the maximum sequence length (up to 6.2×) on a single NVIDIA A100 GPU. These results demonstrate the effectiveness of the approach. The authors conduct thorough experiments on multiple datasets and models, providing a robust evaluation of DropBP's performance across different scenarios.", "weaknesses": "- The paper mentions that the sensitivity calculation is done only once and has negligible overhead. However, more details on this process and its potential impact on training time would provide a clearer understanding of any trade-offs involved.\n\n- The paper could benefit from a more detailed theoretical analysis of why DropBP works as effectively as it does. This would strengthen the paper by providing a deeper understanding of the underlying principles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reviewing our submission and providing valuable feedback. Please see below for our response to the questions and comments.\n\n**Q2.1.**  Can you provide more details on the sensitivity calculation process? Specifically, how is the sensitivity of each layer computed, and what is the computational overhead associated with this step?\n\n**A2.1.** Please referto the global response **GA2** above.\n\n**Q2.2.**  The paper could benefit from a more detailed theoretical analysis of why DropBP works as effectively as it does. This would strengthen the paper by providing a deeper understanding of the underlying principles.\n\n**A2.2.** \n\n+ ```Figure C``` *in the attached PDF*.\n\nThat's a good point. We interpret transformer models as a collection of numerous blocks, each composed of various modules with residual connections. Our hypothesis is that we can fine-tune LLMs well by training only certain shallow submodules. **To theoretically analyze this hypothesis, we measured the impact of submodules based on their path lengths in LLaMA2-7B, as shown in ```Fig. C```**. Specifically, we followed these steps as suggested in [1]:\n\n1.  We first perform a forward pass through the entire network.\n2.  During the backward pass, we randomly sample $k$ residual blocks, which are back-propagated without passing through skip connections, while the remaining $n-k$ blocks are bypassed through the skip connections.  \n3.  We then measure the norm of the gradient at the input.\n\nWe take 100 measurements for each path length $k$. Subsequently, we multiply by the distribution of all possible path lengths, which follows a Binomial distribution, to quantify the gradient contribution from paths of a specific length.\n\nIn ```Fig. C(b)```, we observed that **the gradient per path length decreases as the path length increases.** Consequently, ```Fig. C(c)``` demonstrates that shorter path lengths have a greater impact on the gradient. These observations are consistent with the findings in [1], **which attributed this phenomenon to vanishing gradients.** We confirmed that this also occurs in transformers, where **the paths that significantly influence training in LLMs are relatively short.** Therefore, DropBP enables effective training by focusing on these short submodules.\n\nThanks to the reviewer's advice, this analysis will be included in the final version, and we plan to conduct a more theoretical analysis of this phenomenon.\n\n**Q2.3.** Are there guidelines or heuristics that practitioners can follow to optimize performance for their specific use cases?\n\n**A2.3.** We are very pleased that the reviewer has shown interest in the use case of DropBP. Empirically, **using the identical settings as the baseline is sufficient to achieve good convergence of loss and high accuracy when applying DropBP.** At higher drop rates such as p=0.75 and p=0.875, however, increasing the learning rate by about 1.5 times can slightly improve accuracy. Thanks to the reviewer, we will incorporate these guidelines into the code we release.\n\n**Q2.4.** How well does DropBP integrate with other recent advancements in efficient training techniques, such as mixed precision training or distributed training frameworks?\n\n**A2.4.** We have developed a DropBP library that can be easily integrated into PyTorch, **allowing it to be readily combined with most efficient training techniques** that can be applied on a single GPU, such as parameter-efficient fine-tuning and mixed precision training. As shown in ```Fig. 4``` and ```Table 2``` of the manuscript, we incorporated these combinations into our experiments.\n\n**Additionally, we have confirmed that our library works well with recent distributed training frameworks based on PyTorch, such as FSDP [2].** However, we have encountered some errors when integrating with the DeepSpeed [3] framework. We are currently debugging these issues and plan to resolve them. In the future, we intend to analyze the experimental results once these issues are addressed.\n\n[1] Veit and al., \"Residual Networks Behave Like Ensembles of Relatively Shallow Networks.\"\n[2] Zho and al., \"PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel\".\n[3] Rajbhandari and al., \"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models\".", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pSderOOBkZ", "reviewer": "Reviewer_NAjC", "summary": "The paper introduces DropBP, an innovative approach to accelerate the fine-tuning of Large Language Models (LLMs) by selectively dropping layers during backward propagation. This method is presented as a means to reduce computational costs and activation memory, significant challenges in the efficient fine-tuning of LLMs. The authors have provided a clear implementation of DropBP as a PyTorch extension and demonstrated its effectiveness through experiments on various LLMs and datasets.", "questions": "some typos\n\n- Line 49: As a results -> As a result\n- Line 62: a effective -> an effective", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The concept of dropping backward propagation layers to reduce computational overhead is differential from previous work and addresses an important issue in training large models.\n\n- The paper includes extensive experiments that validate the effectiveness of DropBP in reducing training time and memory usage while maintaining accuracy.\n\n- The development of a PyTorch extension for DropBP facilitates easy integration with existing training codes, enhancing the practical applicability of the method.", "weaknesses": "- The motivation is not well illustrated. I agree with that dropping sublayers could lead to training efficiency as the model turns to a shallower counterpart. However, I mean, pervious work like LayerDrop and others omit the layer computation in the forward pass. Then the computation could be removed in the subsequent backward computation with essential engineering efforts. Thus it lacks a clear distinction in terms of technical innovation compared to these previous works.\n\n- While the paper proposes omitting sublayer computation in the backward pass, it's unclear why the forward pass computation remains unchanged. Justifying this choice or exploring alternatives would strengthen the contribution.\n\n- The faster convergence observed in Figure 5 with DropBP compared to the vanilla model is counterintuitive. The observation here quite confuses me since the backward pass optimizes a partial computation graph, concerns regarding overfitting arise. The paper would benefit from a discussion on potential regularization techniques employed to address this, and a comparison with related work (e.g., [1]) that utilizes sublayer dropping for regularization in training a deep Transformer model. \n\n\n  \n  [1] Li et al., 2021 (AAAI) Learning Light-Weight Translation Models from Deep Transformer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reviewing our submission and providing valuable feedback. Please see below for our response to the questions and comments.\n\n**Q3.1.**   Pervious work like LayerDrop and others omit the layer computation in the forward pass. Then the computation could be removed in the subsequent backward computation with essential engineering efforts. Thus it lacks a clear distinction in terms of technical innovation compared to these previous works.\n\n**Q3.2.** It's unclear why the forward pass computation remains unchanged. \n\n**A3.1-2.** We appreciate the reviewer's insightful comments. We understand the reviewer's questions as follows:\n\n1.  ***Motivation:*** What is the reason for focusing on reducing computations in the backward propagation rather than the entire computation process?\n2.  ***Methodology:*** Why does the approach omit only backward propagation computations while keeping forward propagation unchanged?\n3.  ***Related Works:*** How does this differ from algorithms like LayerDrop [2], which drop computations during forward propagation as well?\n\nFirst, the goal of LayerDrop is to ultimately perform layer-wise pruning to reduce inference time by dropping layers during forward propagation. In contrast, the aim of our DropBP is to **enhance training efficiency directly by reducing FLOPs and training memory usage.** Therefore, our approach does not require strictly dropping the forward path. Of course, while it is possible to drop the forward propagation, we chose to avoid this because **the training process is highly sensitive to dropping the forward propagation**, as shown in ```Fig. E```.\n\n-   ```Figure E``` *in the attached PDF*.\n\nIn ```Fig. E```, we compared the loss curves of DropBP and Progressive Layer Dropping (PLD) [3], a representative layer dropping algorithm, when the same drop rate was applied to training computations. **The results demonstrate that DropBP achieves much more stable training compared to PLD.** This is because dropping the forward path can cause output deviations, which can negatively impact loss and all gradients.\n\nFurthermore, in ```Appendix D``` of the manuscript, we compared our DropBP with Layer Dropping algorithms for fine-tuning LLMs. As shown in ```Table 7``` of the manuscript, our **DropBP achieved higher accuracy compared to traditional Layer Dropping algorithms.**\n\nIn summary, while Layer Dropping algorithms aim to improve inference efficiency by randomly dropping the forward path and pruning it during inference, our DropBP focuses on selectively dropping backward paths, that are relatively less sensitive than the forward path, to accelerate training and reduce memory usage while maintaining high accuracy.\n\nFollowing the reviewer's advice, we will incorporate these clarifications into the paper to reduce confusion and strengthen the contribution.\n\n\n**Q3.3.** The faster convergence observed in Figure 5 with DropBP compared to the vanilla model is counterintuitive. The observation here quite confuses me since the backward pass optimizes a partial computation graph, concerns regarding overfitting arise. The paper would benefit from a discussion on potential regularization techniques employed to address this, and a comparison with related work (e.g., [1]) that utilizes sublayer dropping for regularization in training a deep Transformer model.\n\n**A3.3.** We agree with the reviewer's observation that ```Fig. 5``` of the manuscript appears to show rapid convergence, which may suggest overfitting. However, this is because the x-axis is plotted against training time. When plotted against training steps, a completely different pattern emerges, as shown below:\n\n+ ```Figure B``` *in the attached PDF*.\n\nAs shown in ```Fig. B```, when the drop rate is 0.5, the convergence of loss per step is almost identical to the baseline. However, with drop rates of 0.75 and 0.875, **the convergence speed per step is slower**. Nonetheless, **DropBP significantly reduces the time consumed per training step** because it skips the backward propagation computations for the dropped layers. Consequently, **the convergence speed per training time is actually faster for DropBP compared to the baseline**.\n\nMoreover, our DropBP does not optimize a partial computation graph but instead randomly drops layers according to the drop rate, meaning the layers being trained change with each step. **This can be interpreted as structured dropout, leading to an ensemble effect where multiple models are effectively trained simultaneously.** As a result, it can serve as a regularization technique to mitigate overfitting issues. In fact, when applying DropBP, we did not observe overfitting, where training loss decreases but validation loss worsens. \n\nAdditionally, through [1], **our DropBP can also be interpreted that DropBP reduces co-adaptation among layers.**  Specifically, [1] claims that Layer Dropping (LD) [2] successfully achieves regularization by reducing co-adaptation among layers through dropping layers during training. This effect is also applicable to DropBP, where some backward paths are randomly dropped instead of strictly training all layers.\n\nThanks to the reviewer's insight, we have confirmed that DropBP can also be interpreted as a regularization technique. In the revised manuscript, we will incorporate this analysis, and in future research, we plan to specifically analyze the regularization effects of applying DropBP and work on improving it.\n\n**Q3.4.** some typos exist.\n\n**A3.4.** hank you for catching the typo. We will make sure to correct it in the revised version.\n\n[1] Li et al., \"Learning Light-Weight Translation Models from Deep Transformer.\"\n[2] Fan et al., \"Reducing Transformer Depth on Demand with Structured Dropout.\"\n[3] Zheng and He, \"Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the all reviewer for carefully reviewing our submission and providing valuable feedback. We would like to address several common and important questions in the following global response.

**GQ1.** It is unclear if the method works well for generation tasks and domain-specific transfer learning.

**GA1.** We have conducted additional experiments to evaluate our method on generation tasks and domain-specific transfer learning following the Reviewer's advice as below:

+ ```Table G.1.1.``` *Results of Generation Tasks (MT-Bench) with LLaMA3-8B fine-tuned on the OASST1 datasets.*

| Method         | Drop Rate | Memory | Time | Humanities | STEM | Roleplay | Extraction | Writing | Reasoning | Coding | Math | Avg. |
|----------------|-----------|-----|------|------------|------|----------|------------|---------|-----------|--------|------|------|
| No-tune        | .         | .   | .    | 6.25       | 5.70  | 5.45     | 5.20        | 4.85    | 4.40       | 3.20    | 1.95 | 4.62 |
| LoRA           | .         | 57G | 27m  | 7.00          | 6.40  | 5.80      | 5.70        | 5.30     | 4.55      | 3.25   | 2.95 | 5.12 |
| LoRA+DropBP    | 0.5       | 42G | 21m  | 6.55       | 6.25 | 6.05     | 5.50        | 5.05    | 4.45      | 3.75   | 3.25 | 5.11 |
|                | 0.75      | 36G | 17m  | 6.75       | 5.90  | 5.80      | 5.70        | 5.35    | 4.30      | 3.60    | 3.30  | 5.09 |
|                | 0.875     | **32G** | **16m**  | 6.60        | 6.55 | 5.90      | 5.70        | 5.70     | 3.95      | 3.40    | 2.80  | 5.08 |

 + ```Table G.1.2```. *Results of Domain-Specific Learning with LLaMA3-8B on the IMDB Dataset.*

| Method       | Drop Rate | Memeroy | Time | Accuracy (%)|
|--------------|-----------|-----|------|----------|
| LoRA         | .         | 40G | 539s  | 91.7     |
| meProp       | 0.5       | 40G | 507s  | 88.5     |
| Freezeout    | .         | 40G | 445s  | 91.5     |
| LoRA+DropBP  | 0.5       | 34G | 438s  | 92.3     |
|              | 0.75      | 32G | 392s  | 91.5     |
|              | 0.875     | **31G** | **362s**  | 91.3     |

The experimental results show that **our DropBP can reduce training memory and time while maintaining comparable accuracy in generation tasks and domain-specific learning tasks.** Thanks to the Reviewer nmtJ, we will include the improved results in the revision.

**GQ2.**   I am curious about the specific method for calculating sensitivity and the associated overhead.

**GA2.** Sensitivity calculation process in DropBP involves two main steps:

- ***Step 1. Sensitivity Calculation***:  We define the sensitivity of a layer as the variance in gradient normalization between when the layer is dropped and when it is not. **Therefore, calculating all layer sensitivities requires $L$ iterations, corresponding to the number of layers, which is typically fewer than the iterations needed for fine-tuning**. For example, when fine-tuning LLaMA2-70B, 160 iterations are required to calculate sensitivities, which is significantly fewer than the size of training datasets, such as the Alpaca dataset (52K).

- ***Step 2. Drop Rate Allocation***: We employ the simple greedy algorithm from [1] to efficiently determine drop rates. Initially, all layers start with a drop rate of 0, which is gradually increased to achieve the target FLOPs. At each step, we increase the drop rate of the layer by 0.1, selecting the layer that minimizes the increase in total sensitivity. **By using a binary heap for optimal move selection, the algorithm runs with a complexity of $O(L \ log \ ⁡L)$.** The overhead is negligible, considering that the computation cost for a transformer's attention layer and linear layer is $O(bsh^2L)$ and $O(bs^2hL)$ respectively, where $b$, $s$, and $h$ represent the batch size, sequence length, and hidden dimension.

As explained, both Step 1 and Step 2 incur very low computational overhead and occur only once during the entire training process. Therefore, the overhead from sensitivity computation is negligible. **The experimental results in ```Table G.2``` below support this claim.** Thanks to reviewers nmtJ and Ep4o, we will include detailed explanations of the sensitivity calculation process in the final version.

+ ```Table G.2.``` *Processing Time Analysis for Calculating Sensitivities When Fine-Tuning LLMs on the Alpaca Dataset.*

| Model       | Precision | PEFT  | Calculate sensitivity | Training    |  |     |     |
|-------------|-----------|-------|-------------|-----|---------|-----|-----|
|             |           |       |             | p=0 | p=0.5   | p=0.75 | p=0.875 |
| LLaMA2-7B   | Mixed     | LoRA  | **10s**         | 2.2h | 1.7h   | 1.4h | 1.3h |
|             | BF16      | FFT   | **10s**         | 2.0h | 1.3h   | 1.0h | 0.8h |
| LLaMA2-13B  | BF16      | LoRA  | **21s**         | 2.9h | 2.1h   | 1.7h | 1.5h |
| LLaMA2-70B  | BF16      | QLoRA | **6m**          | 29.6h | 22.2h | 18.4h | 16.5h |

**GQ3.** Could the authors present Figure 5 with training steps as the x-axis to demonstrate faster convergence?

**GA3.**

+ ```Figure B``` *in the attached PDF*.

In response to the reviewer's request, we plotted the training curves over training steps in ```Fig. B(a)```. When the drop rate is 0.5, the convergence of loss per step is almost identical to the baseline. However, with drop rates of 0.75 and 0.875, **the convergence speed per step is slower**. Nonetheless, **DropBP significantly reduces the time consumed per training step** because it skips the backward propagation computations for the dropped layers. Consequently, **the convergence speed per training time is actually faster for DropBP compared to the baseline as shown in in ```Fig. B(b)```**. Thanks to reviewers nmtJ and NAjC, we will include an analysis of the training loss curve in DropBP, using training time and training steps as the x-axis.

[1] Chen et al., "ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training"

Meta-review: The paper proposed a new training method which randomly drops backpropagation in transformer layers to improve the training efficiency and reduce memory consumption. The drop rate is determined by sensitivity analysis of each layer. As a result, the author achieve faster wall-clock time convergence with smaller compute budget.

All three reviewers have a consensus that 1) the paper is tackling an important problem, 2) The proposed method is simple and easy to integrated to the existing frameworks, 3) the supporting experimental results show good improvement which reinforces the authors' claim. The reviewers commonly asked to compare the proposed method with pre-existing methods such as medrop, layerdrop and etc. The authors successfully provided comparisons and promised to add them in the final version. Also, the reviewers questioned if this method worked for different tasks and the authors provided generation tasks and domain-specific learning task results.

Overall, the paper proposes a simple yet effective method to make the LLM training more efficient. This could be beneficial to the community and there's no major concerns in the paper from the reviewers' reviews.
Decision: Accept (poster)