Title: Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: large language model

parameter-efficient learning

memory-efficient learning

reversible neural network
Abstract: Parameter-efficient fine-tuning (PEFT) of pre-trained language models (PLMs) has emerged as a highly successful approach, with training only a small number of parameters without sacrificing performance and becoming the de-facto learning paradigm with the increasing size of PLMs. However, existing PEFT methods are not memory-efficient, because they still require caching most of the intermediate activations for the gradient calculation, akin to fine-tuning. One effective way to reduce the activation memory is to apply a reversible model, so the intermediate activations are not necessary to be cached and can be recomputed. Nevertheless, modifying a PLM to its reversible variant is not straightforward, since the reversible model has a distinct architecture from the currently released PLMs. In this paper, we first investigate what is a key factor for the success of existing PEFT methods, and realize that it's essential to preserve the PLM's starting point when initializing a PEFT method. With this finding, we propose memory-efficient fine-tuning (MEFT) that inserts adapters into a PLM, preserving the PLM's starting point and making it reversible without additional pre-training. We evaluate MEFT on the GLUE benchmark and five question-answering tasks with various backbones, BERT, RoBERTa, BART and OPT. MEFT significantly reduces the activation memory up to 84% of full fine-tuning with a negligible amount of trainable parameters. Moreover, MEFT achieves the same score on GLUE and a comparable score on the question-answering tasks as full fine-tuning. A similar finding is also observed for the image classification task.

Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

Abstract

Parameter-efficient fine-tuning (PEFT) of pre-trained language models (PLMs) has emerged as a highly successful approach, with training only a small number of parameters without sacrificing performance and becoming the de-facto learning paradigm with the increasing size of PLMs. However, existing PEFT methods are not memory-efficient, because they still require caching most of the intermediate activations for the gradient calculation, akin to fine-tuning. One effective way to reduce the activation memory is to apply a reversible model, so the intermediate activations are not necessary to be cached and can be recomputed. Nevertheless, modifying a PLM to its reversible variant is not straightforward, since the reversible model has a distinct architecture from the currently released PLMs. In this paper, we first investigate what is a key factor for the success of existing PEFT methods, and realize that it's essential to preserve the PLM's starting point when initializing a PEFT method. With this finding, we propose memory-efficient fine-tuning (MEFT) that inserts adapters into a PLM, preserving the PLM's starting point and making it reversible without additional pre-training. We evaluate MEFT on the GLUE benchmark and five question-answering tasks with various backbones, BERT, RoBERTa, BART and OPT. MEFT significantly reduces the activation memory up to 84% of full fine-tuning with a negligible amount of trainable parameters. Moreover, MEFT achieves the same score on GLUE and a comparable score on the question-answering tasks as full fine-tuning. A similar finding is also observed for the image classification task. 1

1 Introduction

Large-scale pre-trained models have achieved great success across various domains and applications [1, 2, 3, 4, 5, 6, 7, 8]. As their capabilities continue to evolve, the released pre-trained language models (PLMs) have grown exponentially in size, even reaching a scale of 100 billion parameters [3, 9, 10, 11, 12]. Consequently, it presents unprecedented challenges in effectively leveraging these models for downstream tasks due to limited computing resources.

A historically common approach to adapting PLMs to downstream tasks is updating all pre-trained parameters, full finetuning . Although full fine-tuning has yielded numerous state-ofthe-art results, its applicability is limited in storage-constrained environments. This constraint arises from maintaining a complete copy of the fine-tuned model for each task. An alternative

20

40

60

80

100

60

65

70

75

80

85

90

Score (%)

BERT

RoBERTa

BART

OPT

Full FT

Prefix-Tuning

LoRA

MAM

Adapter

H

Adapter

P

LST

our MEFT

1

adaptation approach is parameter-efficient fine-tuning (PEFT) [13, 14, 15, 16, 17, 18, 19] which involves selectively updating a small number of task-specific parameters while keeping the majority of the PLM's parameters frozen. PEFT offers significant advantages in reducing storage requirements by only saving one general PLM alongside the modified parameters for each task. In addition to storage savings, PEFT achieves comparable performance to full fine-tuning, sparking considerable interest in the adoption of PEFT.

Despite their advantages in parameter efficiency, existing PEFT methods still face challenges in terms of memory efficiency [20, 21]. PEFTs necessitate the caching of intermediate activations, similar to the requirements of full fine-tuning, to calculate the gradients of the trainable parameters. Typically, they consume more than 70% activation memory of full fine-tuning (see Figure 1). Since activations significantly contribute to the memory requirements during training, there are instances where fine-tuning a large-scale PLM with PEFT is not feasible due to memory constraints. To address this issue, a commonly employed approach is to treat the PLM as a feature extractor, such as knowledge distillation to a smaller model [22, 23], adding additional trainable layers on top [20] or aligned [21, 24] with it, and so on. These approaches circumvent the need to store the PLM's activations since the gradient computation graph does not traverse through the PLM. However, these methods often require additional pre-training or exhibit a substantial performance gap compared to full fine-tuning when using the same underlying model [20, 21].

In this paper, we propose a novel method called memory-efficient fine-tuning (MEFT) to modify PLMs in a parameter- and memory-efficient manner, without requiring additional pre-training. Initially, we investigate a crucial factor for the success of existing PEFT methods and determine that the proper initialization of newly added parameters is essential to maintain the continuity of information from the PLM ( § 2). Leveraging this insight, we design three MEFT methods that enable the modification of a PLM to its reversible variant, so it only necessitates caching the final output and allows for the recomputation of intermediate activations during back-propagation ( § 3). Consequently, MEFT significantly reduces the memory required for caching activations (see Figure 1).

To validate the effectiveness of our MEFT methods, we conduct extensive evaluations on the GLUE benchmark [25] with BERT [1], RoBERTa [2] and BART [26] ( § 4). The experimental results consistently demonstrate that our MEFT methods outperform both full fine-tuning and strong PEFT baselines in terms of parameter and memory efficiency. Remarkably, our methods achieve the same score as full fine-tuning while updating only 0.2% of the parameters and saving up to 84% of the activation memory. Furthermore, we evaluate MEFT on five question-answering tasks with a larger model, OPT [9]. The results show that our approach achieves a comparable score as full fine-tuning while saving 50% of the activation memory and updating only 0.64% of the parameters. A similar finding is also observed on the image classification task, SVHN [27]. Collectively, these experiments establish the effectiveness of MEFT as a powerful parameter- and memory-efficient approach that does not compromise performance.

2 Preliminaries

In this section, we aim to provide essential background knowledge by addressing the following questions: (1) Why are existing PEFTs not sufficiently memory-efficient ( § 2.1)? (2) What is a key factor for the success of PEFT ( § 2.2)? (3) What challenges does a reversible model have ( § 2.3)?

2.1 Parameter-efficient fine-tuning is not sufficiently memory-efficient

Given a N multilayer perception: h N = f N ( f N -1 ( ... ( f 2 ( f 1 ( h 0 ))) ... )) with h 0 as the initial input, the n th layer h n = f n ( h n -1 ) = σ n ( W n h n -1 ) consists of a nonlinear function σ n and a weight matrix W n , where the bias term is ignored for simplicity. Denoting x n = W n h n -1 , in backpropagation with a loss L , the gradient of W n is calculated with the chain rule as:

where σ ′ is the derivative of σ and the calculation of σ ′ n requires x n . Therefore, { x i } N i = n are cached during the forward pass to obtain the gradient of W n , even though { W i } i>n are frozen.

0

5

10

15

20

25

Fine-tuned Parameters (%)

24

25

26

27

28

29

30

31

Memory Usage (GB)

Full FT

LoRA

(IA)

3

Prefix-Tuning

(a) Memory trade-off.

0

0.05

0.1

0.5

1

0

10

20

30

40

50

60

70

80

0

0.1

0.5

0.9

c

Average Score (%)

default init

default init with random PLM

α

= 1

α

= 0, trainable

α

α

= 1 / c

(b) Initialization effect, Left: LoRA, Right: (IA) 3 .

During training, the peak memory footprint is mainly occupied by three components: model's parameters { W n } N n =1 , optimizer state whose size is three times as large as the size of trainable parameters for Adam [28] (one for gradient and two for moments), and activations. The memory footprint for all three components is related to the model's depth and width. In addition, the memory footprint for activations is also related to some training settings, like batch size and sequence length.

Compared to full fine-tuning, existing PEFT methods, such as (Houlsby and Pfeiffer) Adapters [14, 16], LoRA [17], (IA) 3 [29], Prompt-Tuning [19] and Prefix-Tuning [15], tune a small number of parameters, making the size of the optimizer state negligible. However, the memory footprint required for activations is not significantly reduced. As shown in Figure 2a, where we set the batch size as 64 and the sequence length as 512 on RTE [30, 31, 32, 33] with BERTbase [1], the activation memory of all PEFT methods is >75% of full fine-tuning, even with <1% trainable parameters.

2.2 Initialization is significant for parameter-efficient fine-Tuning

Pre-trained models learn generic and distributed enough representations to facilitate downstream learning of highly pressed task representation [36], i.e. offering a robust starting point for the training of downstream tasks. When modifying a PLM with PEFT, we hypothesize that one needs to preserve this starting point at the beginning of training for better performance.

The Starting Point Hypothesis. When modifying a pre-trained model by adding new parameters, one needs to initialize the new parameters in a way to preserve the starting point from the pretrained model at the beginning of training, such that fine-tuning the modified model can match the performance of full fine-tuning.

More formally, supposed f n is a PLM layer and h n = f n ( h n -1 ) , the output from a modified layer f ′ n , h ′ n = f ′ n ( h n -1 ) , should be close to h n at the beginning of training. I.e. h ′ n = h n + δ , where ‖ δ ‖ → 0 . Intuitively, we want h ′ n ≈ h n , because h ′ n is the input to the next (modified) PLM layer. If they are dissimilar, the representation continuity will be broken down. Though most PEFT methods [14, 16, 17, 29] initialize their added modules in this way, we couldn't find a thorough investigation of the significance of this initialization in the existing literature. In this section, we explore the significance of PEFT's initialization for two methods, LoRA and (IA) 3 [29].

LoRA and (IA) 3 represent two common methods for introducing new parameters, involving addition and scaling operations, respectively. Given a pre-trained weight matrix W ∈ R d × d , LoRA modifies it as h ′ = ( W + α r W down W up ) h , where W down ∈ R d × r and W up ∈ R r × d are the added trainable parameters, α is a constant scale factor and normally r /lessmuch d . LoRA's default initialization is W down ∼ N (0 , σ 2 ) and W up = 0 . In this way, W down W up = 0 and the starting point from the

8

16

32

64

Numbe of Layers

-8

-6

-4

-2

0

[4, 4]

[2, 2]

[1, 1]

[0.5, 0.5][0.1, 0.1] [1, 0.1]

[0.1, 1]

[

λ

,

β

]

1.31e-09

2.21e-09

5.37e-09

5.9e-06

0.031

0.0165

0.0171

[0, 0.02]

[0, 0.1]

[0.3, 0.1] [1, 0.02]

[

μ

,

σ

]

5.37e-09

6.42e-08

2.44e-08

1.78e-08

Reconstruction Error (log

10

)

mean

mean

(b) Instability of reversible model.

PLM is preserved perfectly. (IA) 3 modifies W by multiplying it to a trainable vector l ∈ R d as h ′ = ( l /circledot W ) h , where /circledot represents element-wise multiplication. The default initialization of (IA) 3 is l = 1 , also making the starting point untouched.

To facilitate the initialization process of LoRA, we opt for the following initial values: W down = 1 , W up = c and α = 1 , where c is a matrix with all elements equal to an initialized value c , resulting in α r W down W up = c . When c = 0 , the starting point from a PLM is preserved. By adjusting c , we exert control over the degree of departure from the starting point. Similarly, we replace l with l ′ = α l = α c for (IA) 3 .

/negationslash

In Figure 2b, we train the newly added parameters on RoBERTabase [2] for four tasks (CoLA [37], STS-B [38], MRPC [39] and RTE [30, 31, 32, 33]). For LoRA ( r = 8 ), though we modify the initialization method, our result ( c = 0 ) is very close to the default initialization. When the starting point is broken by c = 0 ( α = 1 ), all results are even worse than a randomly initialized model. However, when we set α = 0 3 to preserve the starting point, all results become much better than the ones with α = 1 . For (IA) 3 , when we decrease c from 1 (default initialization) to 0, the results ( α = 1 ) become worse and worse. However, when we set α = 1 /c to preserve the starting point, all results become better. Some of them are even better than the default initialization. All of the above-mentioned results show that it's significant to preserve the starting point from a PLM at the beginning of training when applying or designing a PEFT method. A different initialization scheme is in Figure 10 which leads to a similar finding.

2.3 Challenges of reversible neural network

Recapping a reversible model [41] in Figure 3a, one can reconstruct inputs from outputs as:

where λ and β are scaling factors. Theoretically, F n and G n could be two arbitrary functions (subnetworks). Given a multilayer reversible network, intermediate activations for each layer during the forward pass are not necessary to be cached. One only needs to store the final outputs, then reconstruct the intermediate activations and calculate the gradient layer-by-layer in a backward manner (See Listing 1 in § Appendix). In this way, the memory footprint required for activations can be reduced significantly and has no relationship with the model's depth, i.e. O (1) instead of O ( N ) .

To investigate the training stability of a reversible model, we run experiments on RevViT [40]. 4 RevViT shares the same architecture as Reformer [42], except applying a convolutional layer at the beginning to project an image into a sequence of vectors. When running RevViT, one could still cache the intermediate activations and treat it as an irreversible model. We term the gradient calculated in this way as vanilla gradient . One could also train RevViT in a reversible way, and the corresponding

F

+

F

+

G

+

ℎ

!

"

ℎ

!

#

ℎ

!

#

ℎ

!

"

ℎ

!$#

#

ℎ

!$#

"

ℎ

%

#

=

ℎ

%

"

=

ℎ

%

+

+

+

𝜆

𝜆

𝛽

switch?

(a)

Mutli-head Attention

+

Feed-Forward

Layer Norm

Feed-Forward

Feed-Forward

+

Layer Norm

Attention

Block

MLP Block

(b)

Feed-Forward

Feed-Forward

+

Layer Norm

Nonlinear

Nonlinear

Attention Block

Adapter

𝑾

𝒅𝒐𝒘𝒏

𝑾

𝒖𝒑

𝑾

𝒖𝒑

𝑾

𝒅𝒐𝒘𝒏

(c)

Mutli-head Attention

+

Feed-Forward

Layer Norm

Adapter

+

Feed-Forward

Feed-Forward

Adapter

Layer Norm

(d)

gradient is called reversible gradient . We input the same random noises into the same RevViT twice to obtain the parameter gradients from the convolutional layer, in a vanilla and reversible way. Then we calculate the absolute difference between these two gradients and report the maximum and mean values. In this way, we want to check whether the vanilla gradient can be reconstructed in a reversible way. If the reconstruction error is large, it means that the vanilla gradient could not be recovered in a reversible way due to numerical stability, which might cause unstable training or bad performance.

As shown in Figure 3b, with an increasing number of layers in RevViT, the reconstruction error becomes larger, but still around 10 -8 which is negligible. However, RevViT is sensitive to the scaling factors, λ and β . When both scaling factors or one of them are less than 1 , the reconstruction error increases dramatically. We also explore the initialization of the linear layers in RevViT and find that a larger standard deviation or mean can cause a bigger reconstruction error. In sum, a larger number of layers, smaller scaling factors ( < 1 ) and a larger standard deviation or mean for initialization tend to cause a bigger reconstruction error, which might result in the unstable training or low performance of a reversible model. Last but not least, RevViT [40] finds that residual connections inside F and G deteriorate the performance of a reversible Transformer [43]. 5

3 Memory-efficient fine-tuning

This paper aims to modify a PLM to its reversible variant without additional pre-training, so the PLM can still be fine-tuned with a limited memory footprint. The fundamental guiding principle behind our design is: preserving the starting point from a PLM to the greatest extent possible (discus-

sion in § 2.2). In this section, we propose three methods to modify a PLM to a reversible one.

3.1 MEFT1: PLM layer as F , adapter as G

As shown in Figure 4c, we design F as a pre-trained layer with an adapter, where the insertion position for the adapter is borrowed from He et al. [18]. G is simply an adapter. We initialize the adapters as W down , W up ∼ N (0 , σ 2 ) , same for the following methods. In this way, the output from the adapter is close to 0 at the beginning of the training, so h n ≈ F n ( h n -1 ) . For the following discussion, we only focus on the beginning of the training, making sure our design preserves the starting point from a PLM.

h 0 and h 1 are the input to and output from the 1 st layer of a PLM without any modification, respectively. I.e. h 0 is the representation after the position and word embedding layers of a PLM. We

assign h 1 0 = h 2 0 = h 0 , same for the following methods. At the beginning of the training (see Figure 4a), h 1 1 = λ h 1 0 + F 1 ( h 2 0 ) = λ h 0 + F 1 ( h 0 ) ≈ λ h 0 + h 1 , h 2 1 = β h 2 0 + G 1 ( h 1 1 ) = β h 0 + G 1 ( h 1 1 ) ≈ β h 0 , where the approximation holds because of our initialization of the adapters.

/negationslash

With the same above-mentioned design for the 2 nd reversible layer, we obtain h 1 2 ≈ β h 1 and h 2 2 ≈ h 2 . By analogy, h 1 n ≈ β h n -1 and h 2 n ≈ h n , which means h 2 n always preserves the starting point from the PLM. Feeding h 2 n to the next reversible layer, F n +1 , doesn't break the representation continuity. After all layers, we input h ′ N = ( h 1 N + h 2 N ) / 2 to a task-specific head that is a brand new layer, same for the following methods. 7

For now, h 1 1 and h 2 1 are not desired. When we input h 1 1 and h 2 1 to the 2 nd reversible layer, especially when we input h 2 1 to F 2 , the representation continuity 6 is broken, because h 2 1 = h 1 . We introduce two modifications to address this issue: (1) We set λ → 0 , so h 1 1 ≈ h 1 . (2) Then we switch the order of h 1 1 and h 2 1 before feeding to the next reversible layer, i.e. making h 1 1 ≈ β h 0 and h 2 1 ≈ h 1 . In this way, h 2 1 preserves the starting point. We don't require h 1 1 to preserve any starting point, because it is entered to G 2 which is not a pre-trained layer.

3.2 MEFT2: Adapter as F , PLM layer as G

Opposite to MEFT1, we design F as an adapter and G as the PLM layer with an adapter for MEFT 2 (see Figure 4c). In this case, we need to make sure that the input to G preserves the starting point. Let's also start with the first layer, h 1 1 = λ h 1 0 + F 1 ( h 2 0 ) = λ h 0 + F 1 ( h 0 ) ≈ λ h 0 , h 2 1 = β h 2 0 + G 1 ( h 1 1 ) = β h 0 + G 1 ( h 1 1 ) ≈ β h 0 + G 1 ( λ h 0 ) , where the approximation holds because of our initialization of the adapters.

To preserve the starting point from the PLM, we set λ → 1 , β → 0 and switch the order of h 1 1 and h 2 1 before feeding to the next reversible layer. When setting λ → 1 , we make sure the representation continuity is preserved for G 1 , resulting in h 2 1 ≈ β h 0 + h 1 . When β → 0 and the order of h 1 1 and h 2 1 is switched, h 1 1 ≈ h 1 and h 2 1 ≈ h 0 . In this way, h 1 1 preserves the initialization point, and we won't break the representation continuity when feeding it to G 2 in the next reversible layer. With the same setting for each layer, h 1 n ≈ h n and h 2 n ≈ h n -1 , so h 1 n always preserves the starting point.

3.3 MEFT3: Attention block as F , MLP block as G

As shown in Figure 4d, we can also design F as the pre-trained attention block with an adapter and G as the pre-trained MLP block with an adapter. Also starting with the first layer, we obtain h 1 1 = λ h 1 0 + F 1 ( h 2 0 ) = λ h 0 + F 1 ( h 0 ) , h 2 1 = β h 2 0 + G 1 ( h 1 1 ) = β h 0 + G 1 ( h 1 1 ) .

Compared to the vanilla RevNet [41] where λ = β = 1 , we meticulously assign different values to λ and β to preserve the starting point from a PLM, and switch the order of the outputs before feeding to the next layer (if necessary) to preserve the representation continuity. We summarize the settings for all three MEFT methods in Table 1.

λ → 0 is required, so h 1 1 approximates the original output from the pre-trained attention block, and can be fed to G 1 to preserve the starting point. β → 0 is also required, so h 2 1 ≈ h 1 , and can be fed to F 2 in the next reversible layer. By default, we set λ = β → 0 . For MEFT 3 , one doesn't need to switch the order of h 1 1 and h 2 1 before feeding to the next reversible layer. For each layer, h 1 n is close to the original output from the attention block of the corresponding PLM layer, and h 2 n ≈ h n .

4 Experiments

4.1 Experimental setup

Datasets and evaluation. We evaluate MEFTs on eight sequence representation tasks and five sequence-to-sequence tasks. All sequence representation tasks are from the GLUE benckmark [25].

0.050.1

0.3

0.5

1

λ

(MEFT

1

) or

β

(MEFT

2

) or

λ

=

β

(MEFT

3

)

60.0

62.5

65.0

67.5

70.0

72.5

75.0

77.5

Average Score (%)

Full FT

1

2

3

24

20

16

12

8

81

82

83

84

85

Average Score (%)

Full FT

LoRA

freeze: false

freeze: true

24

20

16

12

8

1

2

3

4

5

6

Memory (GB)

Number of Reversible Layers

The sequence-to-sequence tasks are question-answering benchmarks, including OpenBookQA [44], PIQA [45], ARC (easy and challenge) [46] and SciQ [47]. We show the statistics of these datasets in Table 8 in Appendix. For the GLUE benchmark, we report accuracy on MNLI, QQP, QNLI, SST-2, MRPC and RTE, Pearson correlation coefficient on STS-B (if not specially mentioning) and Matthews correlation coefficient [48] on CoLA. We report accuracy on all question-answering tasks. In addition, we report all results on the development sets as our baselines.

Models. We use the encoder-only models (BERTbase [1], RoBERTalarge [2] and BARTlarge encoder [26]) as the underlying models for all GLUE tasks, and the decoder-only models (OPT1.3B and OPT6.7B [9]) for question-answering tasks. (See Table 9 in Appendix for model details.)

Baselines. The most important baseline is full fine-tuning ( Full FT ) that updates all parameters. Houlsby Adapter ( Adapter H ) [14], Pfeiffer Adapter ( Adapter P ) [16], Prefix-Tuning [15] and LoRA [17] are chosen as PEFT baselines. In addition, two unified PEFT methods, MAM [18] and AutoPEFT [49], that combine multiple PEFT methods are also chosen as PEFT baselines. Lastly, two feature-based tuning methods, Y -Tuning [20] and LST [21], that aim to reduce training memory serve as memory-efficient baselines. We report the baseline results from the original papers if possible.

Implementation. For computational efficiency, we set β = 1 for MEFT1, λ = 1 for MEFT2, and only tune the factors that are required → 0 (see Table 1). After obtaining the optimal value, i.e. 0.1, we use this value for all three MEFT methods, tasks and backbones. On the GLUE benchmark, we sweep learning rates in {3, 4, 5} · 10 -4 , batch sizes in {16, 32} and the number of epochs in {10, 20} for the tasks with >10k training samples. For the low-resource tasks with <10k training samples, we sweep learning rates in {5, 6, 7, 8} · 10 -4 , batch sizes in {16, 32} and the number of epochs in {20, 40}. These grid search spaces are inspired by our baselines, especially by LoRA [17]. We use the default Adam [28] setting with a warmup ratio of 6%. If the model's performance on the development set is not improved over 5 epochs, we stop the training. We run the same task of a method in the above-mentioned grid search space three times with different random seeds, choose the best result from each run, and report the mean and standard deviation of these best results. For all question-answering tasks, we sweep learning rates in {1, 3, 5, 7} · 10 -4 , batch sizes in {8, 16, 32} and the number of epochs in {3, 5, 10}, and keep other settings the same, which is inspired by [50]. The sequence length for all tasks is set to 512, 128, 128 and 128 for BERTbase, RoBERTalarge, BARTlarge and OPT as our baselines, respectively. We run all experiments on the Transformers framework [34] on a single NVIDIA RTX A6000 GPU with 48GB memory. Overall, a single run of any task could be finished within 8 hours, and most tasks could be finished in an hour. Fine-tuning settings are summarized in Table 7.

4.2 Results and discussions

Importance of MEFT's initialization. In the beginning, we further test the starting point hypothesis on our MEFTs by adjusting the scaling factors, λ and β . As depicted by the dashed lines in Figure 5, the degradation in performance is evident when the scaling factors deviate from their desired value of

0 (as indicated in Table 1). However, when they are small enough (0.05 or 0.1), the results are even better than full fine-tuning. For most MEFT methods (MEFT 1 and MEFT 3 ), the optimal value for the scaling factors is 0.1. So we use this value for all MEFT methods in the following experiments.

MEFTs with vanilla gradient are strong PEFT methods. Though MEFTs have reversible architectures, we can still treat them as irreversible models and cache the intermediate activations during fine-tuning. In this way, they are simply PEFT methods. In Table 2, all MEFT methods, utilizing the vanilla gradient, consistently outperform both full fine-tuning and other baseline approaches by a significant margin. For example, MEFT3 outperforms Full FT by 1% and the best PEFT baseline (AutoPEFT) by 0.7% on BERTbase. MEFT1 outperforms Full FT by 0.7% on RoBERTalarge.

Performance gap of MEFTs between vanilla and reversible gradients. In Figure 5, the results of MEFTs with reversible gradient (solid line) are often lower than the ones with vanilla gradient (dashed line). Recapping the discussion in § 2.3, smaller scaling factors ( < 1 ) and residual connections in F and G can cause a larger reconstruction error because of numerical stability. When modifying a PLM, we can't remove the residual connections from it and have to set the scaling factors → 0 due to the starting point hypothesis, which we believe is the main reason for the performance drop. Our claim is further supported by MEFT 3 which has the most evident drop among all MEFTs. Compared to MEFT 1 and MEFT 2 that only have a residual connection in either F or G , both F and G of MEFT 3 have residual connections. In addition, we have to set both λ and β close to 0 for MEFT 3 , which also causes a bigger reconstruction error than only setting one scaling factor (see Figure 3b middle). Since MEFT 3 with reversible gradient performs the worst among all MEFTs, we only run it on BERTbase

due to limited resources. Expectedly, MEFT 1 trained in FP32 outperforms it trained in FP16 on both RoBERTalarge and BARTlarge (see Table 2), because FP16 causes more instability.

Reversible MEFTs on deep model. Because of the starting point hypothesis, the residual connection from PLMs remains and the scaling factors are set closely to 0. With an increasing number of layers, the training instability is expected to become more severe (see Figure 3b left). As shown in Figure 6, when all RoBERTa layers are reversible (the number of reversible layers as 24), the score drops dramatically. To address this issue, we propose three settings in Figure 7: (1) Cache the activations for top layers (vanilla gradient) and apply reversible shallow layers (reversible gradient). (2) Freeze some shallow PLM layers, i.e. treating the shallow layers as a feature extractor. (3) Combine the above two settings. Notably, we have to put the reversible layers under the vanilla layers due to numerical stability. If we reverse the order, the reconstruction error is transferred to the vanilla layers.

We only explore the first two settings on RoBERTa and will discuss the third setting in the following, since RoBERTalarge doesn't contain many layers. In Figure 6, when we apply the first setting (freeze: false) to RoBERTalarge, the average score becomes better when the number of reversible layers decreases, outperforms full fine-tuning when it's ≤ 16 . However, the activation memory also increases with an increasing number of vanilla layers, since the vanilla layers require caching the activations. By default, we set the number of reversible layers as 16 for RoBERTalarge in Table 2. For the second setting (freeze: true), the results are always worse than full fine-tuning. However, its activation memory stays the same since all trainable layers are reversible.

MEFTs are parameter and memory-efficient with a strong performance. Let's go back to Table 2. Though there is a gap in MEFTs between vanilla and reversible gradients, reversible MEFTs still achieve strong results compared to previous baselines. On BERTbase, reversible MEFT 1 and MEFT 2 obtain the same average score as Full FT, slightly worse than the best PEFT method, AutoPEFT (83.2 vs. 83.5). However, reversible MEFTs only requires about 21% and 24% activation memory of Full FT and PEFTs. On RoBERTalarge, reversible MEFT 1 (FP32) achieves the same score as Full FT and outperforms all PEFT methods, while only requiring 37% and 48% activation memory of Full FT and PEFTs.

Due to limited computing resources, we only conduct experiments on the best MEFT method, MEFT 1 , on BARTlarge when compared to other memory-efficient methods. In addition, we don't use our own grid search space on BARTlarge. Instead, we apply the same grid search space as LST, setting the learning rate in { 3 · 10 -4 , 1 · 10 -3 , 3 · 10 -3 } , the batch size as 100 and the number of epochs as 20. In this way, we want to validate the robustness of MEFT. Similarly, MEFT 1 outperforms the memory-efficient baselines by a large margin while only requiring 29% LST's activation memory. In addition, LST requires knowledge distillation to initialize the added layers and is not stable [21]. 8

MEFT trained in FP32 vs. in FP16, and the time-memory tradeoff. Though reversible MEFTs trained in FP32 outperform the ones trained in FP16, there are still some notable discussions about them: (1) The memory footprint required by reversible MEFTs trained in FP32 and FP16 is the same. In Table 2, MEFT 1 and MEFT 1 (FP32) have the same activation memory on BARTlarge, because the recomputed activations in back-propagation are always in FP32 due to the mixed precision training [51]. I.e. PyTorch [52] only allows FP32 in back-propagation; (2) FP16 still benefits the training of large PLMs. In Table 2, the peak and activation memory difference is about the backbone size in FP32 for PEFTs and MEFTs. If one could reduce the backbone size by loading in FP16, we can further reduce the peak memory; 2 (3) Training in FP16 is faster than the training in FP32 (about 1:1.2) due to the forward pass. In addition, since reversible MEFTs recompute the activations, they require more training time, about twice the training time for MEFTs with the vanilla gradient.

Results on larger and deeper models. Here we explore a more realistic setting (the third setting in Figure 7) on larger and deeper models, OPT1.3B and OPT6.7B, in Table 3. On OPT1.3B with 24 layers, we set the number of frozen, reversible and vanilla layers as 8. On OPT6.7B with 32 layers, we use 8 reversible and vanilla layers, same as OPT1.3B. For a fair comparison, we freeze the first 8 PLM layers and modify the rest 16 layers with LoRA. MEFT 1 is comparable to LoRA, while only requiring LoRA's 65% activation memory. Though slightly worse than Full FT (-0.3%), MEFT 1 's

reversible

reversible

freeze

freeze

vanilla

vanilla

freeze

freeze

reversible

reversible

reversible

reversible

vanilla

vanilla

activation memory is only half of the one for Full FT. When using the same activation memory as Full FT by running on OPT6.7B, MEFT 1 outperforms Full FT by a large margin.

Transfer to image classification task. Though we only focused on NLP tasks, MEFT could be transferred to other tasks, even other architectures. We leave the transfer of MEFT to other architectures for future work, and here apply MEFT to ViT [53] for an image classification task, i.e. SVHN [27]. We follow the main training recipe from AdaptFormer [54], except for changing the optimizer from SGD to AdamW, setting the maximum gradient norm as

0.3. For MEFT1's hyper-parameters, we set r = 64 and λ = 0 . 3 (smaller λ is not stable for training). Similar to the NLP's results, MEFT1 achieves comparable accuracy as AdaptFormer while saving a large amount of memory footprint in Table 4.

For more results about comparing MEFT to gradient checkpointing, comparing MEFT to quantization methods, and combining MEFT with other memory-efficient methods, please go to Appendix § E. In addition, due to the page limit, we put the detailed related works in Appendix § A, and discuss the limitation of our work in Appendix § B.

5 Conclusion

In this paper, we propose three memory-efficient fine-tuning methods (MEFTs), that fine-tune PLM in a parameter-efficient and memory-efficient way without the requirement of additional pre-training and match the performance of full fine-tuning. MEFTs modify the PLM architecture with adapters and make it reversible, by following the starting point hypothesis that is essential for PEFTs. So MEFTs don't require caching the intermediate activations during training and significantly reduce the memory footprint occupied by activations. When applying MEFTs to various models, BERT, RoBERTa and BART, on the GLUE benchmark, MEFTs achieve a similar score as full fine-tuning and other strong baselines, while saving up to 84% activation memory. A similar story is also observed when applying MEFT to larger and deeper models, OPT, on five question-answering tasks. MEFT achieves a comparable score as full fine-tuning and only consumes its 50% activation memory. However, because of the recomputation of activations, MEFTs require slightly more training time than other PEFT methods and offer a slightly lower score when trained in FP16 instead of FP32. In the future, we are interested in applying MEFT to other areas, like computer vision and automatic speech recognition, and to other bigger backbones for more sequence-to-sequence tasks.

References

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 41714186. Association for Computational Linguistics, 2019. doi: 10.18653/v1/n19-1423. URL https://doi.org/10.18653/v1/n19-1423 .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR , abs/1907.11692, 2019. URL http://arxiv.org/abs/1907. 11692 .

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020. URL http://jmlr. org/papers/v21/20-074.html .

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 1597915988. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01553. URL https://doi.org/10. 1109/CVPR52688.2022.01553 .

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: a simple framework for masked image modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , pages 9643-9653. IEEE, 2022. doi: 10.1109/CVPR52688.2022.00943. URL https://doi.org/10.1109/CVPR52688.2022.00943 .

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 612, 2020, virtual , 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html .

Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 13-23, 2019. URL https://proceedings.neurips.cc/paper/2019/ hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html .

Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 , pages 5099-5110. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1514. URL https://doi.org/10.18653/v1/D19-1514 .

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. OPT: open pre-trained transformer language models. CoRR , abs/2205.01068, 2022. doi: 10.48550/arXiv.2205.01068. URL https://doi.org/10.48550/arXiv.2205.01068 .

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel

Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR , abs/2211.05100, 2022. doi: 10.48550/arXiv.2211.05100. URL https://doi.org/10.48550/arXiv.2211. 05100 .

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023. doi: 10.48550/arXiv.2302.13971. URL https://doi.org/10.48550/arXiv.2302.13971 .

Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. CoRR , abs/2205.05131, 2022. doi: 10.48550/arXiv.2205.05131. URL https: //doi.org/10.48550/arXiv.2205.05131 .

Baohao Liao, Yan Meng, and Christof Monz. Parameter-efficient fine-tuning without introducing new latency. In Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 , pages 4242-4260. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.acl-long.233. URL https://doi. org/10.18653/v1/2023.acl-long.233 .

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 2790-2799. PMLR, 2019. URL http://proceedings.mlr.press/v97/houlsby19a.html .

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 4582-4597. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.acl-long.353. URL https://doi.org/10.18653/ v1/2021.acl-long.353 .

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021 , pages 487-503. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.eacl-main.39. URL https://doi.org/10.18653/v1/2021.eacl-main. 39 .

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9 .

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=0RDcd5Axok .

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November,

2021 , pages 3045-3059. Association for Computational Linguistics, 2021. doi: 10.18653/v1/ 2021.emnlp-main.243. URL https://doi.org/10.18653/v1/2021.emnlp-main.243 .

Yitao Liu, Chenxin An, and Xipeng Qiu. Y-tuning: An efficient tuning paradigm for large-scale pre-trained models via label representation learning. CoRR , abs/2202.09817, 2022. URL https://arxiv.org/abs/2202.09817 .

Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: ladder side-tuning for parameter and memory efficient transfer learning. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/ paper/2022/hash/54801e196796134a2b0ae5e8adef502f-Abstract-Conference. html .

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015. URL http://arxiv.org/abs/1503.02531 .

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR , abs/1910.01108, 2019. URL http://arxiv.org/abs/1910.01108 .

Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. CoRR , abs/2303.16199, 2023. doi: 10.48550/arXiv.2303.16199. URL https://doi.org/10.48550/arXiv.2303.16199 .

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview.net/forum?id= rJ4km2R5t7 .

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 78717880. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.703. URL https://doi.org/10.18653/v1/2020.acl-main.703 .

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011 , 2011. URL http://ufldl.stanford. edu/housenumbers/nips2011_housenumbers.pdf .

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980 .

Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/ forum?id=rBCvMG-JsPd .

Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Joaquin Quiñonero Candela, Ido Dagan, Bernardo Magnini, and Florence d'AlchéBuc, editors, Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Selected Papers , volume 3944 of Lecture Notes in Computer Science , pages 177-190. Springer, 2005. doi: 10.1007/ 11736790\_9. URL https://doi.org/10.1007/11736790_9 .

Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment , 01 2006.

Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Satoshi Sekine, Kentaro Inui, Ido Dagan,

Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors, Proceedings of the ACLPASCAL@ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic, June 28-29, 2007 , pages 1-9. Association for Computational Linguistics, 2007. URL https://aclanthology.org/W07-1401/ .

Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009 . NIST, 2009. URL https://tac.nist.gov/publications/2009/additional.papers/RTE5_ overview.proceedings.pdf .

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URL https://aclanthology.org/2020.emnlp-demos.6 .

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/ huggingface/peft , 2022.

Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021 , pages 7319-7328. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021. acl-long.568. URL https://doi.org/10.18653/v1/2021.acl-long.568 .

Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Trans. Assoc. Comput. Linguistics , 7:625-641, 2019. doi: 10.1162/tacl\_a\_00290. URL https://doi.org/10.1162/tacl_a_00290 .

Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval2017 task 1: Semantic textual similarity - multilingual and cross-lingual focused evaluation. CoRR , abs/1708.00055, 2017. URL http://arxiv.org/abs/1708.00055 .

William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005 . Asian Federation of Natural Language Processing, 2005. URL https://aclanthology.org/I05-5002/ .

Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao-Yuan Wu, Bo Xiong, Christoph Feichtenhofer, and Jitendra Malik. Reversible vision transformers. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 1824, 2022 , pages 10820-10830. IEEE, 2022. doi: 10.1109/CVPR52688.2022.01056. URL https://doi.org/10.1109/CVPR52688.2022.01056 .

Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual network: Backpropagation without storing activations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 2214-2224, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ f9be311e65d81a9ad8150a60844bb94c-Abstract.html .

Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020. URL https://openreview.net/forum?id= rkgNKkHtvB .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von

Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017. URL https://proceedings.neurips.cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract.html .

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018 , pages 2381-2391. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d18-1260. URL https://doi.org/10.18653/v1/d18-1260 .

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 7432-7439. AAAI Press, 2020. URL https://ojs.aaai.org/index.php/AAAI/article/view/6239 .

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR , abs/1803.05457, 2018. URL http://arxiv.org/abs/1803.05457 .

Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@EMNLP 2017, Copenhagen, Denmark, September 7, 2017 , pages 94-106. Association for Computational Linguistics, 2017. doi: 10.18653/v1/w17-4413. URL https://doi.org/10.18653/v1/w17-4413 .

Brian W Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et Biophysica Acta (BBA)-Protein Structure , 405(2):442-451, 1975.

Han Zhou, Xingchen Wan, Ivan Vulic, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efficient fine-tuning. CoRR , abs/2301.12132, 2023. doi: 10.48550/arXiv. 2301.12132. URL https://doi.org/10.48550/arXiv.2301.12132 .

Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. CoRR , abs/2302.04870, 2023. doi: 10.48550/arXiv.2302.04870. URL https://doi.org/10. 48550/arXiv.2302.04870 .

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR , abs/1710.03740, 2017. URL http://arxiv.org/abs/1710. 03740 .

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021. URL https: //openreview.net/forum?id=YicbFdNTTy .

Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurIPS , 2022. URL http://papers.nips.cc/paper_files/paper/2022/hash/ 69e2f49ab0837b71b0e0cb7c555990f8-Abstract-Conference.html .

Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, and Christof Monz. Mask more and mask later: Efficient pre-training of masked language models by disentangling the [MASK] token. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022 , pages 1478-1492. Association for Computational Linguistics, 2022.

doi: 10.18653/v1/2022.findings-emnlp.106. URL https://doi.org/10.18653/v1/2022. findings-emnlp.106 .

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: masked sequence to sequence pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 5926-5936. PMLR, 2019. URL http://proceedings.mlr.press/v97/ song19d.html .

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 8440-8451. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.747. URL https://doi.org/10.18653/v1/2020.acl-main.747 .

Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 7057-7067, 2019. URL https://proceedings.neurips.cc/paper/ 2019/hash/c04c19c2c2474dbf5f7ac4372c5b9af1-Abstract.html .

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR , abs/2005.14165, 2020. URL https://arxiv.org/abs/2005.14165 .

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR , abs/2001.08361, 2020. URL https://arxiv.org/abs/2001.08361 .

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR , abs/2203.15556, 2022. doi: 10.48550/arXiv.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556 .

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR , abs/2204.02311, 2022. doi: 10.48550/arXiv.2204.02311. URL https://doi.org/10.48550/arXiv.2204.02311 .

Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun,

Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR , abs/2201.08239, 2022. URL https://arxiv.org/abs/2201.08239 .

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR .

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. CoRR , abs/2211.01786, 2022. doi: 10.48550/arXiv.2211.01786. URL https://doi.org/10.48550/arXiv.2211.01786 .

Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings . OpenReview.net, 2017. URL https://openreview. net/forum?id=r1Ue8Hcxg .

Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 , pages 6253-6264. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022.acl-long.433. URL https://doi.org/10.18653/ v1/2022.acl-long.433 .

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR , abs/1604.06174, 2016. URL http://arxiv.org/abs/1604.06174 .

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net, 2019. URL https://openreview. net/forum?id=rJl-b3RcF7 .

Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event , volume 119 of Proceedings of Machine Learning Research , pages 3259-3269. PMLR, 2020. URL http: //proceedings.mlr.press/v119/frankle20a.html .

Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. LIT: learned intermediate representation training for model compression. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA , volume 97 of Proceedings of Machine Learning Research , pages 3509-3518. PMLR, 2019. URL http: //proceedings.mlr.press/v97/koratana19a.html .

Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training. In Irina Calciu and Geoff Kuenning, editors, 2021 USENIX Annual Technical Conference, USENIX ATC 2021, July 14-16, 2021 , pages 551-564. USENIX Association, 2021. URL https://www.usenix.org/conference/atc21/presentation/ren-jie .

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate posttraining quantization for generative pre-trained transformers. CoRR , abs/2210.17323, 2022. doi: 10.48550/arXiv.2210.17323. URL https://doi.org/10.48550/arXiv.2210.17323 .

Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021 , pages 7947-7969. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.627. URL https://doi.org/10. 18653/v1/2021.emnlp-main.627 .

Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via blockwise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022 . OpenReview.net, 2022. URL https://openreview. net/forum?id=shpkpVXzo3h .

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR , abs/2305.14314, 2023. doi: 10.48550/arXiv.2305.14314. URL https://doi.org/10.48550/arXiv.2305.14314 .

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020 , pages 3505-3506. ACM, 2020. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/3394486.3406703 .

A Related works

Pre-trained language models . PLMs, which are trained on extensive datasets for a common task such as predicting masked words [1, 2, 3, 26, 55, 56, 57, 58] or the next word [59, 60] in a sentence, play a vital role in facilitating knowledge transfer to downstream tasks. They have demonstrated remarkable achievements across various applications, consistently delivering stateof-the-art outcomes. Furthermore, scaling up PLMs has proven to yield predictable enhancements in performance for these downstream tasks [61, 62]. Consequently, the size of released PLMs has progressively grown, reaching an unprecedented scale of 100 billion parameters [9, 10, 11, 60, 63, 64]. Such large-scale PLMs unveil extraordinary capabilities, enabling zero-shot or in-context learning [59, 60] for a broad spectrum of tasks. Nevertheless, transfer learning remains a prevalent approach for effectively deploying these models in new task scenarios [29, 65, 66], which post unparalleled requirements on the computing resources.

Parameter-efficient fine-tuning . With the advent of large-scale PLMs, a new method that aims to reduce storage requirements, PEFT, has been proposed [14, 15, 19]. PEFT adds and trains a small number of parameters while matching the performance of full fine-tuning. There are various ways to add new parameters. For example, Houlsby et al. [14] and Pfeiffer et al. [16] insert small bottleneck modules (adapters) to the PLM. LoRA [17] injects rank decomposition matrices into the pre-trained weights. HiWi [13] inserts the pre-trained parameters to a low-rank adapter. (IA) 3 [29] scales the pre-trained weight with a trainable vector. Prompt-based methods [15, 19] append a sequence of trainable vectors to the word embeddings or attention components. Recently, some unified methods, which combine multiple PEFT methods in a heuristic way [18] or with the technique of neural architecture search [49, 67, 68], have also been proposed. Though PEFTs save the storage by a large margin compared to full fine-tuning, they still require a similar memory footprint during training as full fine-tuning [20, 21] because of the activation memory.

Memory-efficient training . Memory-efficient training aims to reduce the memory footprint during the training process. Reversible neural networks [40, 41, 42] reduce the activation memory by recomputing the activations with the outputs during back-propagation. Gradient checkpointing [69] trade computation for memory by dropping some intermediate activations and recovering them from an extra forward pass. The activation memory is O (1) and O ( √ N ) for reversible neural networks and gradient checkpointing, respectively. MEFT is the first method that is proposed to modify a PLM to its reversible variant. When applying MEFT on a deeper model, one can use gradient checkpointing to further reduce the activation memory for the layers with vanilla gradient.

Network compressions, like pruning [70, 71] and knowledge distillation [22, 23, 72], save the memory footprint for both training and inference. They compress a PLM to a smaller model by either deleting unimportant parameters or distilling knowledge from the PLM to the smaller model. Treating a PLM as a feature extractor and avoiding its gradient calculation is also an effective way to reduce the activation memory [20, 21]. However, these methods normally require extra pre-training before fine-tuning, or achieve a lower performance compared to full fine-tuning when using the same PLM.

B Limitations

We acknowledge the main limitation of this work is that we only evaluate our proposed methods on a limited amount of tasks and don't conduct experiments on the encoder-decoder models. The main reason for the limited amount of tasks is that our computing resources are constrained. In addition, the major criterion for our selection of the underlying models is that we could find many strong baselines on them without reproduction. BERT and RoBERTa fulfill this criterion very well on the GLUE benchmark. Regarding the encoder-decoder model, recently there is a clear trend of applying a decoder-only model on sequence-to-sequence tasks. Therefore, we apply OPT in this paper and plan to include LLAMA [11] for the instruction-finetuning data in the future.

Another limitation of MEFT is its lower score when trained in FP16 and on a deeper model. We have discussed this problem in § 4.2. In sum, more reconstruction error is introduced by FP16 due to its numerical range and by a deeper model because of the error accumulation. Fortunately, the results are still comparable to the PEFT baselines when trained in FP16. Even trained in FP32, the activation memory footprints don't increase compared to FP16. One only needs to spend more training time in FP32 when using the same batch size as in FP16 (about 20% more training time). However, since

MEFTs reduce the memory footprint, a larger batch size during training is possible, which can save some training time. For deeper models, we offer a practical and effective setting in Figure 7.

Last but not least, when fine-tuning larger models, like OPT1.3B and OPT6.7B [9], the peak memory footprint is occupied by the model parameters rather than the activation (see Table 3). One needs to combine other techniques with MEFT to reduce the peak memory footprint, like loading the model in FP16 or even in int8 rather than in FP32, combining MEFT with ZeRO [73] as in Table 6.

C Step-by-step design for MEFT1

For the reader's easy understanding, in this section, we explain MEFT1 step-by-step. First, let's re-emphasize the guiding principles for our design: (1) For each reversible layer, we must have two inputs and two outputs as in Figure 3a. (2) We need to follow the starting point hypothesis. I.e. whenever we modify a PLM layer, we need to ensure the modified layer has almost the same output as the original PLM layer if we input the same input of the original PLM layer to the modified layer at the beginning of training. If the outputs are not similar, they become even more dissimilar after multiple layers, tearing down the PLM's initialization.

As shown in Figure 8a, for the first PLM layer, h 0 is the input and h 1 is the output. In Figure 8b, the inputs to the first reversible layer is h 1 0 = h 2 0 = h 0 . Recapping the architecture of F 1 in Figure 4c (up), we simply insert an adapter in parallel to the two consecutive feed-forward layers, and initialize the adapter as W down , W up ∼ N (0 , 0 . 02 2 ) , which results in h 1 ≈ F 1 ( h 2 0 ) since h 2 0 = h 0 . If we set λ → 0 , h 1 1 = λ h 1 0 + F 1 ( h 2 0 ) ≈ h 1 . In this way, h 1 1 plays the role of preserving the starting point. Now let's consider h 2 1 . Due to our initialization of the adapter, the output from G 1 ( G 1 is simply an adapter as in Figure 4c (down)) is close to 0 . So h 2 1 = β h 2 0 + G 1 ( h 1 1 ) ≈ β h 0 + 0 = β h 0 . After switching the order of h 1 1 and h 2 1 , h 1 1 ≈ β h 0 and h 2 1 ≈ h 1 .

After the final layer, we simply take the mean of two outputs as h ′ N = ( h 1 N + h 2 N ) / 2 , and input h ′ N to a task-specific head, like a classification layer. The design procedure is similar for MEFT2 and MEFT3. In sum, order switching is mainly for preserving the representation continuity, and setting the scaling factors close to 0 is mainly for preserving the starting point.

For the second reversible layer, if we don't switch the order of h 1 1 and h 2 1 , it looks like Figure 8c. The input to F 2 is β h 0 , which breaks down the representation continuity of a PLM since the input to the pre-trained F 2 should be close to h 1 . If we switch their order as in Figure 8d, we preserve the representation continuity. And it results in h 1 2 = λβ h 0 + F 2 ( h 1 ) ≈ h 2 due to λ → 0 and h 2 ≈ F 2 ( h 1 ) . Similar to the first reversible layer, h 2 2 ≈ β h 1 . After switching, h 1 2 ≈ β h 1 and h 2 2 ≈ h 2 . By analogy, for the n th reversible layer, h 1 n ≈ β h n -1 and h 2 n ≈ h n .

D Implementation details of the question-answering tasks

Compared to GLUE tasks where all tasks are classification tasks and the classification heads are randomly initialized, the question-answering tasks are sequence-to-sequence tasks and need the pre-trained output layer that shares the same parameters as the word embedding layer. The output

Feed-Forward

Feed-Forward

+

Layer Norm

Attention Block

ℎ

!"#

ℎ

!

(a)

+

+

+

ℎ

!

"

ℎ

!

#

ℎ

!

#

ℎ

!

"

ℎ

#

#

ℎ

#

"

+

+

+

𝜆

𝜆

switch

𝛽

F

!

F

!

G

!

ℎ

!

#

=

ℎ

!

"

=

ℎ

!

(b)

+

+

+

ℎ

!

𝛽ℎ

"

ℎ

#

!

ℎ

#

#

+

+

+

𝜆

𝜆

𝛽

F

!

F

!

G

!

ℎ

!

𝛽ℎ

"

(c)

+

+

+

ℎ

!

𝛽ℎ

"

ℎ

#

!

ℎ

#

#

+

+

+

𝜆

𝜆

𝛽

F

!

F

!

G

!

ℎ

!

𝛽ℎ

"

switch

(d)

layer requires the continuity of representation. I.e. at the beginning of training, the input to the output layer, h ′ N , should be close to h N . Therefore, we need to do a modification to h ′ N instead of using h ′ N = ( h 1 N + h 2 N ) / 2 .

Here we introduce a new scaling factor γ and require γ → 0 . For MEFT1, since h 2 N ≈ h N (see Table 1), we set h ′ N = γ h 1 N + h 2 N ≈ h 2 N ≈ h N . Similarly, h ′ N = h 1 N + γ h 2 N ≈ h 1 N ≈ h N for MEFT2, and h ′ N = γ h 1 N + h 2 N ≈ h 2 N ≈ h N for MEFT3. Without any tuning, we set γ = 0 . 1 as other tuned scaling factors by default.

128

256

512

40

60

80

100

120

140

160

Samples / Second

Full FT + gradient checkpointing

LoRA + gradient checkpointing

1

32

64

128

33

34

35

36

128

256

512

Sequence Length

0.5

1.0

1.5

2.0

2.5

32

64

128

2

4

6

8

10

E More results

E.1 Compared to gradient checkpointing

Previously, we only theoretically stated that the activation memory for reversible network and gradient checkpointing is O (1) and O ( √ N ) , respectively. In addition, we didn't compare the training time of MEFT with PEFT in detail. Here we offer some empirical results for your better understanding.

In Figure 9, we compare activation memory and throughput among MEFT , LoRA with gradient checkpointing and Full FT with gradient checkpointing. The throughput for all three methods is at the same level, maximum 12% difference between LoRA and MEFT when the sequence length is 128 and the batch size is 32. With an increasing sequence length, the gap becomes narrower to 7.5%. Notably, the throughput for LoRA without gradient checkpointing is 52.7 samples/second. With gradient checkpointing, it is 36.1 samples/second, 69% of the original throughput. For MEFT with the same setting, it is 33.5 samples/second, 64% of LoRA's throughput without gradient checkpointing. In sum, MEFT's throughput is at the same level as LoRA's with gradient checkpointing, and about 64% of LoRA's without gradient checkpointing. In addition, MEFT's activation memory is always the lower bound among these three methods. The gap between LoRA with gradient checkpointing and MEFT becomes larger with an increasing sequence length and batch size.

E.2 Compared to quantization methods

Quantization is an orthogonal method to MEFT, which reduces the memory footprint of training or inference by reducing the parameter size to fewer bits and using low-bit-precision matrix multiplication. There are mainly three different quantization methods: (1) Post-training quantization

[74, 75] that quantizes a trained model after pre-training or fine-tuning; (2) Lower-bit optimizer [76] that stores the optimizer state with lower precision and de-quantizes it only for the optimization, similarly to FP16 or BF16 mixed precision training but with lower-bit; (3) Lower-bit frozen LLM with LoRA, i.e. QLoRA [77], that applies 4-bit quantization to compress the LLM. During fine-tuning, QLoRA backpropagates gradients through the frozen 4-bit quantized LLM into the low-rank adapters. Notably, the computation data type for QLoRA is BF16. It de-quantizes weights to the computation data type to perform the forward and backward passes.

To some extent, all these three methods are orthogonal to our method and can be combined with MEFT: (1) Post-training quantization is mainly for reference and it can be applied to any trained models; (2) 8-bit Adam can also be applied to any models trained based on a gradient; (3) QLoRA is a combination of (1) and (2). For QLoRA, we conducted some experiments on BERTbase with the default setting as Figure 9. As shown in Table 5, METF1 saves the most activation memory while having a similar throughput as LoRA with gradient checkpointing. The reason for the larger activation memory of QLoRA than LoRA is that it has an additional de-quantization step, which also causes its smallest throughput.

E.3 Combine MEFT with ZeRO

ZeRO [73] saves memory by partitioning the model's parameters and optimizer state among GPUs or between GPU and CPU. This method is orthogonal to MEFT, since MEFT saves memory from activations. We conduct some experiments on OPT1.3B by combining our method with DeepSpeed [78] ZeRO stage 3 that offloading model's parameters and the optimizer state to CPUs. As shown in Table 6, ZeRO significantly reduces the memory footprint from the model's parameters, therefore reducing MEFT's peak memory from 28.2GB to 6.4GB.

Listing 1: Backward pass for each Layer. The peak memory happens at Line 10 or Line 25, depending on whether the subnetwork G is larger than F or the opposite. In the code, we use x1, x2, y1, y2, x1_factor, x2_factor to represent h 1 n -1 , h 2 n -1 , h 1 n , h 2 n , λ and β , respectively.

0

0.05

0.1

0.5

1

0

10

20

30

40

50

60

70

80

0

0.1

0.5

0.9

Average Score (%)

default init

default init with random PLM

α

= 1

α

= 0, trainable

α

α

= 1 / c

c

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n>**Rebuttal:** The provided details satisfy my concerns. I think this paper should be accepted after applying the agreed changes.\n\n>**TL;DR:** The proposed MEFT reduces training activation memory, which is an important topic for the ML community. MEFT outperforms existing PEFT techniques while consuming up to 84% less memory compared to full fine-tuning. However, MEFT comes with significant computation overhead, which is not explored. Furthermore, some key comparisons, such as gradient checkpointing, are missing. Addressing my concerns and questions would improve my score.\n\nThis paper proposes the Memory Efficient Fine-Tuning (MEFT) technique that achieves similar accuracy to that of traditional PEFT techniques while consuming up to 84% less memory compared to full fine-tuning. The paper discusses previous PEFT techniques and provides insights on key attributes for them to work. These insights include empirical results and detailed explanations. One of these key insights is the weights initialization. The paper shows the importance of initializing the PEFT weights such that they do not have any effect on the underlying LLM before starting the training process. \n\nThe paper proposes to combine reversible models with PEFT existing techniques to create MEFT algorithms that gain from both worlds. The paper provides three different variants of MEFT, which are explained as well as illustrated. The three variants of MEFT are empirically validated on four different architecture backbones BERT, RoBERTa, BART and OPT. The MEFT variants are compared to several popular PEFT techniques as well as some memory efficient techniques. These comparison are conducted on the GLUE dataset.\n\nSTRENGTHS:\n* **S.1.** The proposed MEFT algorithm tackles an important problem in existing PEFT techniques, which makes LLM PEFT training more accessible to researchers with low resources by reducing the activation memory up to 84% of full fine-tuning.\n* **S.2.** The paper provides interesting insights that are explained and backed by results and detailed illustrations.\n* **S.3.** The proposed MEFT variants outperform existing PEFT techniques on GLUE dataset with several architecture backbones.\n* **S.4.** Reproduction code is provided as part of the submission.\n\nWEAKNESSES:\n* **W.1.** The training time of MEFT is not well explored and is only noted as \"double the training time\". Hence, the tradeoff between memory and computation efficiency is not well understood.\n* **W.2.** Popular existing memory efficient training techniques such as tensor rematerialization (gradient checkpointing) [2][3] and ZeRO [1] are not compared to. These techniques sacrifice training speed for reduction in memory consumption, similar to the goal of MEFT. Furthermore, these techniques are compatible with existing PEFT approaches. Finally, these techniques are widely available in deep learning frameworks such as PyTorch, DeepSpeed, and Hugging Face Accelerate which makes comparison relatively simple.\n* **W.3.** The font sizes in Figures 1,2,3,5, and 6 are too small which makes it very difficult to read. Furthermore, some figures are not self explanatory, such as Figure 5 which does not provide any explanation for the difference between the solid and dotted lines. These problems burden the reading flow of the paper.\n\n**Typos.**\n* Line #86: \"for obtaining\" → \"to obtain\"\n* Line #86: \"though we\" → \"even though we\" \n* Line #95: \"in Figure 2a\" → \"in Figure 2a,\"\n* Line #97: \"though with\" → \"even with\"\n* Line #106: \"from an modified\" → \"from a modified\"\n* Line #114: \"the added and\" → \"the added\"\n\n\n[1] Ren, J., Rajbhandari, S., Aminabadi, R.Y., Ruwase, O., Yang, S., Zhang, M., Li, D. and He, Y., 2021, July. ZeRO-Offload: Democratizing Billion-Scale Model Training. In USENIX Annual Technical Conference (pp. 551-564).\n\n[2] Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Gonzalez, J., Keutzer, K. and Stoica, I., 2020. Checkmate: Breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems, 2, pp.497-511.\n\n[3] Beaumont, O., Eyraud-Dubois, L. and Shilova, A., 2021. Efficient combination of rematerialization and offloading for training dnns. Advances in Neural Information Processing Systems, 34, pp.23844-23857.\n\nQUESTIONS:\n* **Q.1.** The paper proposes MEFT which sacrifices computation speed at the cost of lower peak memory. There are several existing common approaches (such as gradient checkpointing and DeepSpeed) for general memory efficient training which are compatible with PEFT techniques. Why are these comparisons not explored or detailed?\n* **Q.2.** In the LoRA modified initialization method, both $W_{up}$ and $W_{down}$ are initialized with a single constant. This causes unwanted symmetries, which makes it more difficult to learn. An alternative approach would be to initialize $W_{down}=1$ (as tested in this paper, but instead of initializing $W_{up}$ to a constant, initialize it with a random gaussian distribution where $c$ can act as the distribution mean. Does this make sense?\n* **Q.3.** In Line #175 it is noted that the adapter weights' $W_{down}$ and $W_{up}$ are initialized from a random gaussian distribution with zero mean. It is further noted, that the outputs of the adapter would be close to zero. However, in practice this does not hold true,  and the actual outputs can significantly deviate from the desired zero output. What am I missing here?\n* **Q.4.** In Figure 6 it seems that LoRA only reduces slightly the peak memory in comparison to the full fine-tuning. However, the optimizer states, which are generally 3 times the size of the model, are significantly smaller in LoRA. Does this result hold for other LLMs and how does the batch size affect it as well?\n* **Q.5.** The MEFT technique does not seem to be only compatible with LLMs and can be compatible with other domains, as sted in the conclusion section. However, the experimental results are solely on LLMs. What is the reason for that?\n* **Q.6.** In Table 2 the memory usage of Y-Tuning is missing, why is that?\n\nLIMITATIONS:\nThe limitations are discussed in paragraph line #315. These limitations include limited compatibility with quantization techniques due to training instabilities and significantly longer training time due to activation re-computation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that the existing parameter-efficient fine-tuning (PEFT) methods are not memory-efficient, since they need to cache most of the intermediate activations for the gradient calculation. To resolve this problem, the authors introduce to use the reversible model to reduce the activation memory. Specifically, this paper have these contributions: 1) it is important to preserve the weight of PLM at the initialization; 2) insert adapters and make it reversible without additional pre-training. Extensive results also demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. This paper points out the importance of memory-efficient fine-tuning in current pre-trained models. And the authors introduce the idea of RevNet into the pre-trained language model, and design three reversible variants to PLM.\n\nWEAKNESSES:\n1. Some conclusions seem naive. For example, preserving the starting point from the pre-trained model at the beginning of training is a common experience. Previous works like LoRA also point out the importance of this point, and this point is also a common method to avoid catastrophic forgetting.\n2. Memory-efficient training is important, but the authors do not mention quantization in this paper.  Quantization has been considered as the one of most effective methods in LLM to save memory. While for RevNet or the proposed architecture, these reversible-based neural networks actually are a trade-off between computation and memory (it will bring more computation to save memory). So, I do not think the proposed method is advantageous to Quantization and directly comparing the proposed methods with PEFT may also not be appropriate. \n3. Essentially, the proposed method adopts an adapter structure and then uses the reversible structure into the transformer and presents three patterns (which formulate different $\\cal{F}$ and $\\cal{G}$) and trains each network from the start point. The idea is to combine existing work (RevNet) with current PEFT methods, and then combine the initialization that preserving the starting point from the pretrained model. So, I think the contributions are limited.\n4. In experimental results, authors report MEFT with vanilla gradient. I think such reports are confusion, since the main contribution in this paper is the reversible gradient with MEFT structure.\n\nQUESTIONS:\n1. This paper presents three variants of MEFT, but is there any conclusion that which one should be chosen under different scenarios?\n\nLIMITATIONS:\nThe authors have presented limitations of their proposed works, and currently, these limitations are still a problem for the proposed methods. Besides, I also give some comments about the limitations of this paper (please see my comments in **Weaknesses**). I do not see any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-25"}

comments: SUMMARY:
Authors propose a memory-efficient (MEFT) variant of parameter-efficient fine-tuning (PEFT), which utilizes reversible networks to save memory stored for activations during model forward propagation. The authors showed insights into the reasons for the effectiveness of PEFT. They evaluated several variants of MEFT with GLUE and QA datasets, indicating that the proposed method does not produce performance gaps while reducing memory usage.

STRENGTHS:
- The proposed method shows competitive results with benchmarks
- Experiments with the initialization of LoRa are insightful 
- Generally, the proposed methods are useful for practitioners

WEAKNESSES:
- The presentation could be improved. E.g., I struggled with Section 2.3 since $h^1, h^2 $ (L136) was not defined above.
- It would be fascinating to compare the proposed method with gradient checkpointing used for training with LoRA/etc. Naively, the usage of invertible layers should be faster. However, seeing such an evaluation in the paper would be highly beneficial.

QUESTIONS:
Please refer to the weaknesses section.

LIMITATIONS:
–
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29