Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Authors: Baohao Liao
Shaomu Tan
Christof Monz
Venue: NeurIPS 2023 poster (2023)
Keywords: large language model
parameter-efficient learning
memory-efficient learning
reversible neural network
TL;DR: Our method finetunes large pre-trained language models in a memory-efficient and parameter-efficient way without sacrificing performance.

Abstract: Parameter-efficient fine-tuning (PEFT) of pre-trained language models (PLMs) has emerged as a highly successful approach, with training only a small number of parameters without sacrificing performance and becoming the de-facto learning paradigm with the increasing size of PLMs. However, existing PEFT methods are not memory-efficient, because they still require caching most of the intermediate activations for the gradient calculation, akin to fine-tuning. One effective way to reduce the activation memory is to apply a reversible model, so the intermediate activations are not necessary to be cached and can be recomputed. Nevertheless, modifying a PLM to its reversible variant is not straightforward, since the reversible model has a distinct architecture from the currently released PLMs. In this paper, we first investigate what is a key factor for the success of existing PEFT methods, and realize that it's essential to preserve the PLM's starting point when initializing a PEFT method. With this finding, we propose memory-efficient fine-tuning (MEFT) that inserts adapters into a PLM, preserving the PLM's starting point and making it reversible without additional pre-training. We evaluate MEFT on the GLUE benchmark and five question-answering tasks with various backbones, BERT, RoBERTa, BART and OPT. MEFT significantly reduces the activation memory up to 84% of full fine-tuning with a negligible amount of trainable parameters. Moreover, MEFT achieves the same score on GLUE and a comparable score on the question-answering tasks as full fine-tuning. A similar finding is also observed for the image classification task.

g Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning
Baohao Liao
Shaomu Tan
Christof Monz
Language Technology Lab, University of Amsterdam b b.liao,
s.tan, c.monz @uva. nl
Abstract
Parameter-efficient fine-tuning (PEFT) of pre-trained language models (PLMs) has emerged as highly successful approach, with training only a small number parameters without sacrificing performance and becoming the de-facto learning paradigm with the increasing size of PLMs. However, existing PEFT methods are not memory-efficient, because they still require caching most of the intermediate activations for the gradient calculation, akin to fine-tuning One effective way to reduce the activation memory is to apply reversible model, the intermediate activations are not necessary to be cached and can be recomputed. Nevertheless, modifying PLM its reversible variant not straightforward, since the reversible model has a distinet architecture from the currently released PI.Ms. In this paper, first investigate what is key factor for the success of existing PEFT methods, and realize that it's essential to preserve the PLM's s starting point when initializing a PEFT method. With this finding, we propose memory-efficient fine-tuning (MEFT) that inserts adapters into a PLM, preserving the PLM's starting point and making it reversible without additional pre-training. We evaluate MEFT on the GLUE benchmark and five RoBERTa, BART and OPT. MEFT significantly reduces the activation memory up to 84% of full fine-tuning with negligible amount of trainable parameters. Moreover, MEFT achieves the same score on GLUE and a comparable score on the
tasks with various backbones, BERT,
tasks as full fine-tuning, A similar finding is also observed for the image classification task]
Introduction
Large-scale pre-trained models have achieved great across various domains and applications 2 3
4
6 7
As their capabilities continue evolve, the released pre-trained language models (PLMs) have grown exponentially in size,
Cir
Profict,
even reaching scale of 100 billion parameters 3 9 10 [|] 12]. Consequently, it presents unprecedented challenges in
HAU
Adapar"
effectively leveraging these models for downstream tasks due to limited computing resources.
8
Out MCTT
Activation Meeory (7)
A historically common approach to adapting PLMs down- stream tasks is updating all pre-trained parameters, full fine- tuning. Although full fine-tuning has yielded numerous state-of- the-art results, its applicability is limited in storage-constraing environments. This constraint arises from maintaining plete copy of the fine-tuned model for each task. An alternative
Figure Average performance of different tasks vs. activation mem- ory. The memory usage for full fine- tuning is denoted as 100%.
Code at
Up-to-date version at https//aaaiivoopaaaa22222...444
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

adaptation approach is parameter-effiiien fine-tuming (PEFT) L3 14 15 16, 17 18] 19 which involves selectively updating a small number of task-specific parameters while keeping the majority of the PL.M's parameters frozen. PEFT offers significant advantages in reducing storage requirements by only saving one general PLM alongside the modified parameters for each task. In addition to storage savings, PEFT achieves comparable performance to full fine-tuning, sparking considerable interest in the adoption of PEFT.
Despite their advantages in parameter efficiency, existing PEFT methods still face challenges in terms of memory efficiency 20 21. PEFTs necessitate the caching of intermediate activations, similar to the requirements of full fine-tuning, to calculate the gradients of the trainable parameters. Typically, they consume more than 70% activation memory of full tine-tuning (see Figure I Since activations significantly contribute to the memory requirements during training, there are instances where finettnning a large-scale PL. with PEFT not feasible due to memory constraints. To address this issue, commonly employed approach is to treat the PL.M as a feature extractor, such as knowledge distillation to smaller model [22 23 adding additional trainable layers on top [20] or aligned [21 24] with it, and on. These approaches circumvent the need to store the PLM's activations since the gradient computation graph does not traverse through the PLM. However, these methods often require additional pre-training or exhibit substantial performance gap compared to full fine-tuning when using the same underlying model [20 21
In this paper, we propose novel method called memory efficiient fine-tuning (MEFT) to modify PLMs parameter- and memory-eficcien manner, without requiring additional pre-training. Initially, we investigate a crucial factor for the success of existing PEFT methods and determine that the proper initialization of newly added parameters essential maintain the continuity of information from the PLM ② Leveraging this insight, we design three MEFT methods that enable the modification of a PLM to its reversible variant, so it only necessitates caching the final output and allows for the recomputation of intermediate activations during back-propagation (3) Consequently, MEFT significantly reduces the memory required for caching activations (see Figure DT.
To validate the effectiveness of our MEFT methods, we conduct extensive evaluations on the GLUE benchmark [25] with BERT M. RoBERTa (2) and BART (26] (:4). The experimental results consistently demonstrate that our MEFT methods outperform both full fine-tuning and strong PEFT baselines in terms of parameter and memory efficiency. Remarkably, our methods achieve the same score as full fine-tuning while updating only 0.2% of the parameters and saving up to 84% of the activation memory. Furthermore, we evaluate MEFT on five grestion-answering tasks with larger model, OPT [9. The results show that our approach achieves comparable score as full fine-tuning while saving 50% of the activation memory and updating only 0.64% of the parameters. A similar finding is also observed on the image classification task, SVHN (27]. Collectively, these experiments establish the effectiveness of MEFT as a powerful parameter- and memory-efficient approach that does not compromise performance.
2 Preliminaries
In this section, we aim to provide essential background knowledge by addressing the following questions: (1) Why existing PEFTs not sufficiently memory-efficiert ($2.1)? (2) What is key factor for the success of PEFT 2.2b? (3) What challenges does a reversible model have (g2.3)
2.1 Parameter-efficient fine-tuning is not sufficiently memory-efficient
Given a N multilayer perception: by = [N([N-I (-.((// (h))))...) with ho as the initial input, the n th layer h, = (hn-1) ) = on (W,,,, h-)) consists a nonlinear function on and a weight ma- trix W., as where the bias term is ignored for simplicity. Denoting æ, W, W.h, in backpropagation with a loss L, the gradient of W,, is calculated with the chain rule as:
0c Ac Ohn
Ohn Ozn Oc dx; Ōh,-1 dxn own dhN
Oh: Oz:
III i=n+1
II (WWi i=1+1
(I)
aw,
where a' is the derivative of a and the calculation of a', requires a Therefore, (æ. N are cached during the forward pass to obtain the gradient of W.. even though (W.) W.)ss are frozen.

---

PLM
baratkez
Memory Usage (GB)
Plettr Adapher
00
29
Average
40
30
0.25 0.1
0.5
Fine-tuned Parameters (56)
(a) Memory trade-off.
(b) Initialization effect, Left: LoRA, Right: (IA)
Figure 2: Exploration of existing PEFTs: (a) The trade-off between memory and the number of trainable parameters. The dashed and solid lines denote the peak and activation memory, respectively. The model size for BERT base is 0.4GB (b) The initialization effect of PEFT on RoBERT Taaaaee Random PLM denotes that we initialize the backhone randomly instead of using pre-trained model.
During training, the peak memory footprint is mainly occupied by three components: model's parameters (W, }m= optimizer state whose size is three times as large as the size trainable parameters for Adam (88] (one for gradient and two for moments), and activations. The memory footprint for all three components is related to the model's depth and width. In addition, the memory footprint for activations is also related to some training settings, like batch size and sequence length.
Compared to full fine-tuning, existing PEFT methods, such as (Houlsby and Pfeiffer) Adapters [1 LoRA (7),, (IA)² [29], Crompt-Tuning [19] and Prefix-Tuning [15] tune a small number of parameters, making the size the optimizer state negligible. However, the memory footprint required for activations is not significantly reduced. As shown in Figure 2al where set the batch size as 64 and the sequence length as 512 on RTE [30 B2 33) with BERTuse [.] the activation memory of all PEFT methods is >75% of full fine-tuning,
with <1% trainable parameters.
2.2 Initialization is significant for parameter-efficien fine- fine-Tuning
Pre-trained models learn generic and distributed enough representations to facilitate downstream learning of highly pressed task representation (36), i.e. offering a robust starting point for the training of downstream tasks. When modifying a PLM with PEFT, we hypothesize that one needs to preserve this starting point at the beginning of training for better performance.
The Starting Point Hypothesis. When modifying a pre-trained model by adding new parameters, one needs to initialize the new parameters in a way to preserve the starting point from the pre- trained model at the beginning of training, such that fine-tuning the modified model can match the performance of full fine-tuning.
More formally, supposed fn is PLM layer and h, - fh the output from a modified layer fr h', = = (h,.),, should be close to h, at the beginning of training. Le. h², = fe,, + ô, where ||ō]| + 0. Intuitively, we want h', hn because h' the input the next (modified) PLM layer. If they are dissimilar, the representation continuity will be broken down. Though most PEFT methods [1 16 17 [2 initialize their added modules in this way, we couldn't 't find a thorough investigation of the significance of this initialization in the existing literature. In this section, we explore the significance of PEFT's initialization for two methods, LoRA and (IA)3 29
LoRA and (IA)3 represent two common methods for introducing new parameters, involving addition and scaling operations, respectively. Given pre-trained weight matrix W E RRdx LoRA modifies it as h' (W + Woonn W., h, where W Wdown& Rdxr and W.,, up E RRr are the added trainable parameters, a constant scale factor and normally r < d. LoRA's default initialization is Wdom (O, o²) and W.p 0. In this way, Wdown W Wup
and the starting point from the
TThough we train FP16, the PLM first loaded in FP32, then auto-casted to FP16 for the forward pass in Transformers (3 Since the memory required the model FP32 is always there during training. we report the memory for models in FP32 in this paper (see Table More discussions about this are hhere We believe it's a bug in the framework and can be resolved with further investigation. Especially Huggingface's 's new PEFT framework [35] allows loading INT8 model for fine-tunine tuning.

---

moan
Recomstructton Einor
Miss
$436-00
6
27
41 10.00
2.000) 300000 2. o00
Numbe of Layers
[
of
(a) Reversible architecture.
Instability of reversible model.
Figure 3: (a) F and are two arbitrary functions (sub-networks), taking two inputs, h and h (b) Reconstruction error between the vanilla and reversible gradients. The default setting is Rev ViT 40) with 8 layers, 1. 3 1, J 0 and 0.02. Left: Different number of layers. Middle: Different scaling values. Right: Initialization with different means and standard deviations.
PLM is preserved perfectly. (IA)² modifies W by multiplying it to a trainable vector E R as h' (I @ Whh, where @ represents element-wiss multiplication. The default initialization of (IA)² I = 1, also making the starting point untouched.
To facilitate the initialization process of LoRA, we opt for the following initial values: W Woown 1, Wap = c. and a 1, where is a matrix with all elements equal an initialized value c, resulting When 0, the starting point from PLM is preserved. By adjusting
in & Waown W., uup we exert control over the degree of departure from the starting point. Similarly, we replace I with I' = al = ac for (IA)².
In Figure 2b we train the newly added parameters on RoBERTague for four tasks (CoLA 37
STS-B [3 MRPC (39 and RTE (30 31 32 33]). For LoRA (r 8), though we modify the initialization method, our result (c = 0) very close the default initialization. When the starting point is broken by _ 0 (a
1), all results are even worse than randomly initialized model. However, when we set a = ( to preserve the starting point, all results become much better than the ones with = 1. For (IA),, when we decrease e from (default initialization) to O, the results (a = 1) become worse and worse. However, when we set a 1/c to preserve the starting point, all results become better. Some of them are even better than the default initialization. All of the above-mentioned results show that it's significant to preserve the starting point from a PLM at the beginning of training when applying designing PEFT method. A different initialization scheme is in Figure 10 which leads to a similar finding.
2.3 Challenges of reversible neural network
Recapping a reversible model 4T in Figure Sa one can reconstruct inputs from outputs as:
h = Ah + Fx h2 Bh + Gn (h
h = (h 9n (h
(2)
h 1h,,1- Fxx
where À and / are scaling factors. Theoretically. F, and G. could be two arbitrary functions (sub- networks). Given multilayer reversible network, intermediate activations for each layer during the forward pass are not necessary to be cached. One only needs to store the final outputs, then reconstruct the intermediate activations and calculate the gradient layer-by-layer Listing I in {Appendix). In this way, the memory footprint required for activations can be reduced significantly and has no relationship with the model's depth, i.c. ((1) instead of O(N).
backward manner (See
To investigate the training stability of a reversible model, we run experiments on Rev VIT 40 RevViT shares the same architecture as Reformer [2], except applying a convolutional layer the beginning to project an image into a sequence of vectors. When rumning Rev one could still cache the intermediate activations and treat as an irreversible model. We term the gradient calculated in this way as vanilla gradient. One could also train Rev VIT in a reversible way, and the corresponding
2xx has to be trainable when a = 0. Otherwise, the newly added parameters are useless. *Our experiments are hased on this file, http:///ggii.ooooo///////s//eeea

---

Layer Norm
Layer Norm
Feel Forward
Foac-Forourd
Feed Forward
Bleck
Narm
Layer Norm
Foot Farward
Now
Musi fead Attern
(d) Figure 4: MEFT architectures. (a) Unfold reversible model. (b) A PLM layer. (c) Two MEFT
(a)
(b)
(c)
architectures: (1) F is the PLM layer with an adapter (up) and is an adapter (down); (2) is the PLM layer with an adapter (up) and F is an adapter (down). (d) The third MEFT architecture: G the MLP block with an adapter (up) and F is the attention block with an adapter (down). For
initialization, W Woown; W up NO0 o²) and = 0.02. Only the adapter is trainable.
gradient is called reversible gradient. We input the same random noises into the same Rev twice obtain the parameter gradients from the convolutional layer, a vanilla and reversible way. Then we calculate the absolute difference between these two gradients and report the maximum and mean values. In this way, we want to check whether the vanilla gradient can be reconstructed in reversible way. If the reconstruction error is large, it means that the vanilla gradient could not be recovered in reversible way due to numerical stability, which might cause unstable training or bad performance.
As shown in Figure 3b with an increasing number of layers in Rev the reconstruction error becomes larger, but still around 10 which negligible. However, RevViT is sensitive to the scaling factors, À and B. When both scaling factors or one of them are less than 1, the reconstruction error increases dramatically. We also explore the initialization of the linear layers in Rev ViT and find that a larger standard deviation or mean can cause bigger reconstruction error. In sum, a larger number of layers, smaller scaling factors (< 1) and larger standard deviation or mean for initialization tend to cause a bigger reconstruction error, which might result in the unstable training or low performance of reversible model. Last but not least, Rev 400 finds that residual connections inside F and g deteriorate the performance of a reversible Transformer 43
3 Memory-efficient fine-tuning
This paper aims to modify PLM to its reversible variant without addi- tional pre-training, so the PLM can still be fine-tuned with a limited mem- ory footprint. The fundamental guid- ing principle behind our design is: pre- serving the starting point from PLM to the greatest extent possible (discus-
Table 1: A summarization of three MEFT methods.
MEFT:
Switch
layer
adupier
++ 0
any
V
Sh.,
An
hn-1
attention
+
sion in 2.2 In this section, we propose three methods to modify PLM to reversible one.
3.1 MEFT;: PLM layer as F, adapter as G
As shown in Figure 4c we design FF as pre-trained layer with an adapter, where the insertion position for the adapter is borrowed from He et al. [18]. simply an adapter. We initialize the adapters as Wa Wdown, W... up ~ o²), same for the following methods. In this way, the output from the adapter is close to at the beginning of the training, so hn Rs (hn For the following discussion, we only focus on the beginning of the training, making sure our design preserves the starting point from a PLM.
ho and h are the input to and output from the 1st layer of a PLM without any modification, respectively. Le. ho the representation after the position and word embedding layers of a PLM. We
'In ViT, IF and G are the attention and MLP block (Figure 4b) without residual connection, respectively.
5

---

assign hn = h = ho. same for the following methods. At the beginning of the training (see Figure
4ah. h = + Fi (h Xho + Fi Xhg + hy, h = 5h 91 (h.) = S + 91 (h,) ~
Shg where the approximation holds because of our initialization of the adapters.
For now, h and h not desired. When we input h and h the 2nd reversible layer, especially when we imput hf to F2, the representation continuity i is broken, because hi * h1. We introduce two modifications to address this issue: (I) We set X 00 so h R h1. (2) Then we switch the order of h and h before feeding to the next reversible layer, i.c. making hj & Sho and h R h1. In this way, h preserves the starting point. We don't require h to preserve any starting point, because is entered to G2 which is not pre-trained layer.
With the same above- mentioned design for the 2nd reversible layer, we obtain hlg = Sh1 and h R h2. By analogy. h4 s -h and h2 Rs hn, which means h always preserves the starting point from the PLM. Feeding h to the next reversible layer, Fa+1, doesn't break the representation continuity. After all layers, we input h' = (h2 + h2y / to a task-specific head that is a brand new layer, same for the following methods
3.2 MEFT;: Adapter as F, PLM layer as G
Opposite to MEFT, we design F as an adapter and G as the PLM layer with an adapter for MEFT₂ (see Figure 4e). In this case, we need to make sure that the input to G preserves the starting point. Let's also start with the first layer, h = Xhg + (ii) = Xho + (ho) s Xho. h = 3hg + 91 (ht) Shg + Gi (ht) z Bhg + G1 (Ahp). where the approximation holds because of our initialization the adapters.
To preserve the starting point from the PLM, we set À + 1, - 0 and switch the order of h and h before feeding to the next reversible layer. When setting X + 1, we make sure the representation continuity is preserved for G1, resulting in h R Sho + h1. When 3 - 0 and the order of h and h switched, h R h, and hę Rs hg. In this way, h preserves the initialization point, and we won't t break the representation continuity when feeding it to G2 in the next reversible layer. With the same setting for each layer, h. ze h2 and h2 s hn-1, so h, always preserves the starting point.
3.3 MEFT;; Attention block as F, MLP block as g
As shown in Figure 4d we can also design F as the pre-trained attention block with an adapter and G as the pre-trained MLP block with an adapter. Also starting with the first layer, we obtain h = Ah - Fi = Xho + Fi(ho), h² Sh + (h,)) ) = Sho + G1 (h))).
h approximates the original output from the pre-trained attention block, and can be fed to Gi to preserve the starting point. / - 0 is also required, so h R h1, and can be fed to F2 in the next reversible layer. By default, we set À
+ 0 is required,
- For MEFT,, 3, one doesn't need to switch the order of h and h before feeding to the next reversible layer. For each layer, ht is close to the original output from the attention block of the corresponding PLM layer, and h hn.
Compared to the vanilla RevNet 41 where X = 3 = 1, we meticulously assign different values to and B to preserve the starting point from PLM, and switch the order of the outputs before feeding to the next layer (if necessary) to preserve the representation continuity. We summarize the settings for all three MEFT methods in Table
4 Experiments
4.1 Experimental setup
Datasets and evaluation. We evaluate MEFTs eight sequence representation tasks and five sequence-to -toeeuuee----seeeeeeeeee tasks. All sequence representation tasks are from the GLUE benckmark [25].
'The presentation continuity and the starting point hypothesis emphasize two aspects. The presentation continuity, for example, shows that one can't 't feed ho to the third pre-trained layer, focusing on the input. The starting point hypothesis shows that the output from modified pre-trained layer should be close to the output from the original pre-trained layer, focusing on the output. However. they are also very related, since the output from the current layer is the input to the next layer.
Read Appendix $C for step-by-step tutorial if you still feel confused.

---

85-
773
Average Score (S)
04-
Memory (GB)
LeFA
83
Fnare balo
Average
82
or (MEFT A (MEFTs))
Number of Reversible Layers
Figure 5: MEFTs with vari- ous scaling factors on BERT buse over RTE, MRPC, STS-B and CoLA Dashed and solid lines de- note MEFTs with vanilla and re- versible gradients, respectively.
Figure 6: The trade-off between the performance and activation memory with MEFT, on RoBERTaarg over RTE, MRPC, STS- B and CoLA. The line annotated by 'freeze: true" means the shallower PLM layers are frozen without any adaptation, while the line annotated by 'freeze: false' means the top MEFT layers with vanilla gradient, as shown in Figure 7
The aequence-to-sequene tasks are question-answering benchmarks, including OpenBookQA 44). PIQA 45 ARC (easy and challenge) 46) and SciQ [47] We show the statistics of these datasets in Table 8 in Appendix. For the GLUE benchmark, we report accuracy on MNLI, QQP, QNLI,
SST-2, MRPC and RTE, Pearson correlation coefficient on STS-B (if not specially mentioning) and Matthews correlation coefficient [48] on CoLA. We report accuracy all question-answering tasks. In addition, we report all results on the development sets as our baselines.
Models. We use the encoder-only models (BERT base [T RoBERTa
2 and BART large encoder (26)) as the underlying models for all GLUE tasks, and the decoder-only models (OPT and 67 [9] for question-answer answering tasks. (See Table in Appendix for model details.)
Baselines. The most important baseline is full fine-tuning (Full FT) that updates all parameters. Houlsby Adapter (Adapter") [14, Pfeiffer Adapter (Adapter [16). Prefix-Tuning [5 and LoRA [17] are chosen as PEFT baselines. In addition, two unified PEFT methods, MAM [18] and
AutoPEFT [4), that combine multiple PEFT methods are also chosen as PEFT baselines. Lastly, two
feature-based tuning methods, Y--Tuning (0) and 21), that aim to reduce training memory serve as memory-efficient baselines. We report the baseline results from the original papers if possible.
Implementation. For computational efficiency, set 3 for MEFT, only tune the factors that are required - (see Table After obtaining the optimal value, i.c. 0.1, use this value for all three MEFT methods, tasks and backbones. On the GLUE benchmark, sweep learning rates in (3, 4, 5)-10-4, batch sizes in [16, 32) and the number of epochs in {10, 20) for the tasks with >10k training samples. For the low-resource tasks with <10k training samples, we sweep learning rates in (5, 6, 7, 8)-10 batch sizes in { [16, 32) and the number of epochs inspired by our baselines, especially by LoRA [17]. We use the default Adam (28) setting with a warmup ratio of 6%. If the model's performance on the development set is not improved method in the above-mentioned grid search space three times with different random seeds, choose the best result from each run, and report the mean and standard deviation of these best results. For all question-answerte answering tasks, we sweep learning rates in [1, 3, 5, 7/-10-4, batch sizes in {8, 16, 32) and the number of epochs in (3, 5, 10), and keep other settings the same, which is inspired by [50]. The sequence length for all tasks is set to 512, 128, 128 and 128 for BERT buse. RoBERTa BART Jarge and OPT as our baselines, respectively. We run all experiments on the Transformers framework [34] on single NVIDIA RTX A6000 GPU with 48GB memory. Overall, a single run of any task could be finished within 8 hours, and most tasks could finished an hour. Fine-tuning settings are summarized in Table 7
for MEFT,, and
in (20, 40). These grid search spaces
5 epochs, we stop the training. We run the same task of
4.2 Results and discussions
Importance of MEFT's initialization. In the beginning, we further test the starting point hypothesis on our MEFTs by adjusting the scaling factors, À and B. As depicted by the dashed lines in Figure 5 the degradation in performance is evident when the scaling factors deviate from their desired value of
7

---

Table Comparion with different methods on GLUE The first and second best results are in bold and underlined, respectively. All baseline results for BERT hase and RoBERTaaage are from [49 and respectively. We report Spearman's 's Correlation for STS-B and matched accuracy for MNLI on BERT huse. The hyper-parameters after each backhone are used for measuring the memory footprint.
8 for all MEFTs. All models are trained in FP16 if not specified with FP32.
Parrmm
Memory (GB)
Method
(5) Peak
Act.
RTE
MRPC
STS-8
CoL.A
SST-2
QNLI
QQP
MNLI
Avg.
BERT bas
(botch
sequetee
100
14.98
71.11.5
85.7
10.5
59.30.
92.
91.5g.1
91.50.0
84-40.2
R1.2
Prefis-Taning
0.17
13.58
13,00
70.5g.
85.9g9
88.82.2
58.9/2
91.9us
00.8g.1
89.16.
82.8u.2
82.3
LoKA
0.27
13:45
13,02
65.9%
84.5u.
88.7
57.60g
92-10.
90.6022
89.400
K3.0pui
51.5
87.3 87.2o.7
47.934.
83.0
98.902
90.5o.1
AutoPEFT
1,40
72.4g.9
87.5,g
89.2ou
60.9g,
92.1
91.
90.6p.
54.0gu
83.5
gradieut
MEET;
0.27
13.64
13.21
74.2
86.70.2
89.0buc
62.b02
92.p2
91.6g.1
89.96.
83.8
83.8
MEFT;
0.27
13.73
13.31
74.30.
86-f0.5
89.45.
61.
93.0p1
91.601
90.20.
$4.5u.1
$4.0
MEFTa
0.27
13.64
13.21
82.503
88.%o.
62.5g
9332g
21.3au
90.1o.1
84.2u2
84.2
revervible gradleur
MEFT,
0.27
13.9g,5
86.50.2
88.8o.1
60.3g.
92.7a.
91.4og
88.50.1
63-40.
83.2
0.27
3.53
3.11
74.90
86.30.
88.60.1
60.71 6
92.803
21.301
88:901
K3.11
83.2
MBFTx(FP2)
0.27
2.99
2.57
70.80
84.8g.5
$8.2
53.9
92.20.4
90,402
85.903
$1.50.1
$1.1
RoBEERTejoree (batch size
32, sequence length 128)
Poll FT
100
11.47
6.05
36.6
90.9
92.4
68.11
96.4
94.7
92.2
90.2
68.9
Adigtor!
0.23
6.05
4.66
72.93.
87.Fu,
91.5gus
66 32.
96.7as
94 94.702
91.5p..
90.30.3
86.4
Adogter"
1.69
6.18
4,71
83/4g.
88.729
91.01a
66. 66.5a4
96.20.3
941.702
92.1n.
59.9tus
87.8
Adigter
0.23
6.16
4.77
80.129
91.96.
67. 67.8ks
94.8o.1
91.
90.5p
87.9
Adagter
0.85
6.21
4.76
83.8k9
90.20.
92.1o7
68.3
96.
91.96
90.2u.
88.4
LoRA
0.23
90.2g
92.33
96.2
94.8
91.602
90.60
vanilia grodient
MEET;
0.23
6.19
4.81
89.50s
91.5g,2
92.3ou
69.9u.7
96.8LL1
94.9
91.5p.1
90.
89.6
MEFT
0,23
6.20
4.82
88.6g
91.304
92.2a
68.8g
96.8
94.8
91 An
90.6
89.3
wwwreible gradicnt
MEFT;
0.25
3.11
1.73
87
90.50
91.6gL
63.5
95. 1p.
94.3
90.10.
S9.2g.
$7.8
MEFT, (FP12)
0,23
91 20.2
lo.7
94 Ro.1
20.5
la.1
(FF3)(FP32)
0.25
3,75
2.37
2g.5
90.564
92.1a6 1uv
Au
94. R4.
89.4g.
53. 4ns
87.9
RA Jurge
(hatch 100, sequence langth
Full FT
20
100
12.75
9.62
77.8
89.2
59.3
95.8
94.3
89.5
90.8
85.2
T-nnnin
200
7.7
62.8
79.2
44.4
94.4
53.2
85.5
82.3
76.7
LSTUF332)
20
2.6
7.05
6.14
69.7
$7.3
55.5
94.7
91.9
89.5
82.1
reveraible gradient
MEFT;
0,20
1.75
72.2
x
95. 1g.
92
87.5gn
87.0pa
81.9
MFFT; (FP32)
0.20
1.75
24.307
88.5
57.432
95.4
23.9g.
89.3g
53. 53.3n1
83.9
(as indicated in Table However, when they are small enough (0.05 or 0.1), the results are even better than full fine-tuning. For most MEFT methods (MEFT, and MEFTa). the optimal value for the scaling factors is 0.1. So we use this value for all MEFT methods in the following experiments.
MEFTs with vanilla gradient are strong PEFT methods. Though MEFTs have reversible archi- tectures, we can still treat them as irreversible models and cache the intermediate activations during fine-tuning. In this way, they are simply PEFT methods. In Table all MEFT methods, utilizing the vanilla gradient, consistently outperform both full munittunng and other baseline approaches by a significant margin. For example, MEFT; outperforms Full FT by 1 1% and the best PEFT baseline (AutoPEFT) by 0.7% on BERT base- MEFT, outperforms Full FT by 0.7% on RoBERTa
Performance gap of MEFTs between vanilla and reversible gradients. In Figure 5 the results of MEFTs with reversible gradient (solid line) are often lower than the ones with vanilla gradient (dashed line). Recapping the discussion in (2.3) smaller scaling factors (< 1) and residual connections in F and G can cause larger reconstruction error because of numerical stability. When modifying a PLM, we can't remove the residual connections from it and have to set the scaling factors - 0 due to the starting point hypothesis. which we believe is the main reason for the performance drop. Our claim is further supported by MEFT; which has the most evident drop among all MEFTs. Compared to MEFT, and MEFT, that only have a residual connection in either or G, both and G of MEFT, have residual connections. In addition, we have to set both À and 3 close to 0 for MEFT,, which also causes a bigger reconstruction error than only setting one scaling factor (see Figure middle).. Since MEFT; with reversible gradient performs the worst among all MEFTs, we only run it on BERT basc
8

---

due to limited resources. Expectedly. MEFT, trained in FP32 outperforms it trained in FP16 on both RoBERTatarg and lare (see Table 2. because FP16 causes more instability.
Reversible MEFTs on deep model. Because of the starting point hypothesis, the residual connection from PLMS remains and the scaling factors are set closely to O. With an increasing number of layers, the training instability is expected to become more severe (see Figure Bblleft. As shown in Figure 6 when all RoBERTa layers are reversible (the number of reversible layers as 24), the score drops dramatically. To address this issue, we propose three settings in Figure 7 (1) Cache the activations for top layers (vanilla gradient) and apply reversible shallow layers (reversible gradient). (2) Freeze some shallow PLM layers, i.e. treating the shallow layers as feature extractor. (3) Combine the above two settings. Notably, we have to put the reversible layers under the vanilla layers due to numerical stability. If we reverse the order, the reconstruction eπrоr is transferred to the vanilla layers.
We only explore the first two settings RoBERTa and will discuss the third setting in the following, since RoBERTajane doesn't It contain many layers. In Figure 6 when we apply the first setting (freeze: false) to RoBERTalarge the average score becomes better when the number of reversible layers decreases, outperforms full fine-tuning when it's 16. However, the activation memory also increases with an increasing number of vanilla layers, since the vanilla layers require caching the activations. By default, we set the number of reversible layers as 16 for RoBERTajage in Table 2 For the second setting (freeze: true), the results are always worse than full fine-tuning. However, its activation memory stays the same since all trainable layers are reversible.
verila
Vivemibie
frease
Tresse
Freaee
Ineece
Figure 7: Three set- tings for deep models.
MEFTs are parameter and memory-efficient with strong performance. Let's go back to Table 2 Though there is gap in MEFTs between vanilla and reversible gradients, reversible MEFTs still achieve strong results compared to previous baselines. On BERT base. reversible MEFT, and MEFT, obtain the same average score as Full FT, slightly worse than the best PEFT method, AutoPEFT (83.2 3..2 vs. 83.5). However, reversible MEFTs only requires about 21% % and 24% activation memory of Full FT and PEFTs. On RoBERTaaage reversible MEFT, (FP32) achieves the same score as Full FT and outperforms all PEFT methods, while only requiring 37% and 48% activation memory of Full FT and PEFTs.
Due limited computing resources, we only conduct experiments the best MEFT method, MEFT,, on BART large when compared to other memory-efficien methods. In addition, we don't use our own grid search space learning rate in {3- 10-4, 1 10-3,-100000 3 10 10-3), the batch size as 100 and the number of epochs as 20. In this way, we want to validate the robustness of MEFT. Similarly, MEFT, outperforms the memory-efficien baselines by large margin while only requiring 29% LST's activation memory. In addition, LST requires knowledge distillation to initialize the added layers and is not stable (2 R
BART large- Instead, we apply the same grid search space as LST, setting the
MEFT trained in FP32 vs. in FP16, and the time-memory tradeoff. Though reversible MEFTs trained in FP32 outperform the ones trained in FP16, there are still some notable discussions about them: (1) The memory footprint required by reversible MEFTs trained in FP32 and FP16 is the same. In Table 2 MEFT, and MEFT, (FP32) have the same activation memory on BARTiarger because the recomputed activations in back cockpprpaggttion are always in FP32 due to the mixed precision training [51]. Le. PyTorch [52] only allows FP32 in Sack-propagations (2) FP16 still benefits the training of large PLMs. In Table 2 the peak and activation memory difference is about the backbone size in FP32 for PEFTs and MEFT's. If one could reduce the backbone size by loading in FP16, we can further reduce the peak memory. (3) Training in FP16 is faster than the training in FP32 (about _
1:1.2) due to the forward pass. In addition, since reversible MEFTs recompute the activations, they require more training time, about twice the training time for MEFTs with the vanilla gradient.
Results on larger and deeper models. Here we explore a more realistic setting (the third setting in Figure 7 on larger and deeper models, OPT and OPT in Table 3 On OPT with 24 layers, we set the number of frozen, reversible and vanilla layers as 8. On OPT with 32 layers, we use 8 reversible and vanilla layers, same as OPT, 1.. For fair comparison, we freeze the first 8 PLM layers and modify the rest 16 layers with LoRA. MEFT, is comparable to LoRA, while only requiring LoRA's 65% activation memory. Though slightly worse than Full FT (-0.3%), MEFT, 's
'The comparison of memory footprint to Y-Tunnng is in Tahle 10

---

Table Results on question -nnseering tasks. 64 for both MEFT1 and LoRA. All methods are trained in FP16. Due to limited computing resources, we only conduct one random run with these methods. A batch size of 32 and a sequence length 128 are used to measure the memory footprint and training time. The training time is for one epoch on the OpenBookQA task. Check Appendix D for the implementation detail.
4Parum.
Memory (GB)
Time
Model
Method
(%)
Peak
Activation
OpenBBeokQA
PIQA
ARC-E
ARC-C
SeiQ
Avg.
Full
OPT
LoRA
0.64
11.42
6.27
36.6
29.9
74.9
60.1
28.7
93.3
57.4
MEFT
0.64
9.20
4,05
45.2
34,0
73.1
57.1
25.8
93.1
57.3
OPT,
ZetoShor
27.6
76.2
65.6
30.6
90.1
38.0
MEFT
0.25
33,67
8.01
200.4
37.0
77.4
65.7
34.1
94.4
61.7
activation memory is only half of the one for Full FT. When using the same activation memory as Full FT by rumning 6.7..7.. MEFT, outperforms Full FT by large margin.
Transfer to image classification task. Though only focused on NI.P tasks, MEFT could be transferred to other tasks, even other architectures. We leave the transfer of MEFT to other architectures for future work, and here apply MEFT ViT [53] for an image classification task, i.c. SVHN 27 We follow the main training recipe from AdaptFormer [54] except for changing the optimizer from SGD to Adam W, setting the maximum gradient norm
Table 4: Results on image classification.
Method
Acci
Peak Memory (GB)
97.87 Fall PT X
AdeptFormer MEPT:
hb
96 89
36
0.3. For MEFT, hyper-parameters we set 64 and = 0.3 (smaller À is not stable for training). Similar to the NLP's results, MEFT, achieves comparable accuracy AdaptFormer while saving large amount of memory footprint Table
For more results about comparing MEFT to gradient checkpointing, comparing MEFT to quantization methods, and combining MEFT with other memory-efficien methods, please go Appendix QE] In addition, due to the page limit, we put the detailed related works in Appendix A and discuss the limitation of work in Appendix B
5 Conclusion
In this paper, we propose three memory-efficient fine-tuning methods (MEFTs), that fine-tune PLM in parameter-efficient and memory-efficient way without the requirement of additional pre-training and match the performance of full fine-tuning. MEFTs modify the PLM architecture with adapters and make it reversible, by following the starting point hypothesis that is essential for PEFTs. So MEFTs don't require caching the intermediate activations during training and significantly reduce the memory footprint occupied by activations. When applying MEFTs to various models, BERT, RoBERTa and BART, on the GLUE benchmark. MEFT' achieve similar score as full fine-tuning and other strong baselines, while saving up to 84% activation memory. A similar story is also observed when applying MEFT to larger and deeper models, OPT, on five question-answen answering tasks. MEFT achieves comparable score as full fine-tuning and only consumes its 50% activation memory. However, because of the recomputation than other PEFT methods and offer slightly lower score when trained in FP16 instead of FP32. In the future, we are interested in applying MEFT to other areas, like computer vision and automatic speech recognition, and to other bigger backbones for more cequence-to-sequend tasks.
activations, MEFTs require slightly more training time
Acknowledgements
We thank all reviewers for their great feedback. We also thank our colleague Yan Meng for her helpful review of our draft. This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project number VI.C.192.088
10

---

References
[1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-ILLT 2019, Minneapolis. MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171- 4186. Association for Computational L Linguisties. 2019. doi: 10.18653/v1/119 1423. URL
https://doi.org// 100.0.186633/1 18653/v1/n19 1423
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692, 2019. httpttttt//aiii.orrrttee
11692
[3] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:14001-140:6; 2020. URL http://jmlr org/papers/ w2wps//022222200//// html
[4] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. In JEEE/CVF Conference on Computer Vision and Partern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 15979- URL. http::ttt///ett org/10
15988. IEFE, 2022. doi: 10. 0WW.CCCPRS2688.022..0011
1109/CVPR5268 2022 01553
[5] Zhenda Xie, Zheng Zhang, Yue Cao,, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: a simple framework for masked image modeling. In /EEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2022, New Orleans. LA, USA, June 18-24, 2022, pages 9643-9653, IEEE, 2022. doi: 10.1 109//////99 WWW0/CCVRR26888.2220000494. URL
https: //doi. org/10. 11100/9VR52688 00943
[6] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin. editors. Advances in Neural Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL htttps://proceeddnngs neurips. cc. cttpscccpppppr///pppppppppppp
[7] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agn visiolinguistic representations for Dision-and-languee tasks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, wwwppiisscccppppereeee 2019/ BC, Canada, pages 13 23, 2019. URL abstraadeeebba---attttt....e Abstract html hash/ c74497701ee22555577777 0aae2255754444055aaeeeeeeee
[8] Hao Tan and Mohit Bansal. LXMERT: learning cross mooss-ty encoder representations from transformers. In Kentaro Inui, Jing liang. Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing, EMNIP-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019, pages 5099-51 10. Association for Computational Linguistics, 2019. doi: 1.118653/v1/D19-155
httptttss///oo.orrrr
[9] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: open pre- rrentriind transformer language models. CoRR, abssa205.00000000 010068 2022. org/ 10. 48550/ arxiv. 22...02065 01068
doi: 10.485500/rxiv.22255. 220.0.00..00......... https
[10] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel
11

---

Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez. Victor Sanh, Hugo Laurençon, Yacine Jemite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisherg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b-paramene open-accene multilingual language model. CoRR, abs/2211.05 2022. doi: 10, 210.000arr.x.22..05111100 URL htt.:: /ttp:p:://do...o org.oioorg//1 480.0..8550/rrra. 2211 05100
[11] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal. Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CORR, abs/2302. 2023. doi: 10.48550//rXiv.2202..1 13971. URL. https://doi. http:://ddoo.or/////// org/10. 48050.485//// 48550/arXiv arxiv 2302. 13971
[12] Yi Tay, Mostafa Dehghani, Vinh Q- Tran, Xavier Gareia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. CoRR, abs/2205 001131, 2022. doi: 200550//rXxv.2205.55557 URL https
7
1..00.8550/ark/c arviv. 2205 05131
[13] Baohao Liao. Yan Meng, and Christof Monz. eerameter-efficieen fine-tuning without introducing new latency. In Anna Rogers, Jordan L. Boyd-Grabee Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pages 4242-4266 Association for Computational Linguistics, 2023. doi: 10. org/10. 125/100.165//110 2000633/w/2223.ac111rrggg 233
https//s///
[14] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia. USA, volume 97 of Proceedings of Machine Learning Research. pages 2790-2799. PMLR, 2019. (tRt:htt:////tttt httttp://proceeeings.. mlr. press/ http//hholsbyyl...oa htm.mmyy999..1m11111.
[15] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing, ACL/LJCNLP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 4582-4597. Association for Computational Linguistics, 2021. doi: 10. 10.18653/1122211 1 353..33. URL htthhttt://ddo htttt::::/ddi.oog///// 186r3//
long. 353
[16] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 487-503. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.eee
httpts:t//oooio http::::/oo...og//.o.ooooo 18133/111116653/1222220 wacl maacl-mainn
39
[17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representattions, ICLR 2022, Virtual Event, April 25-29, 2022. Open 2022. URL httpttss:p/ppeeeeeeeeeeee net wwt.nn/ooooumiiedneeeeeeee.
[18] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Kerg-Kirkpatrick, and Graham Neubig. To- wards unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representatiinns, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022. https:/t
net http://rrrriidddddddddde
[19] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event/ Punta Cana, Dominican Republic, 7-11 November,
12

---

2021, pages 9045-3059, Association for Computational Linguisties, 2021. doi: 10.18653/v1/ 2021. 102l.emmlp-main.. URL /tttps:///oi..rgg/// 1tt.://00000010111//////////////
amnlp-maing 243
[20] Yitao Liu, Chenxin An, and Xipeng Qiu. Y-tuning: An efficient tuning paradigm for large-scale pre-trained models via label representation learning. CoRR, abs/2202.09817, 2022. URL
httts://arxiv.. org/ wtw.//222..092...
[21] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. LST: side-tunin for parameter and memory efficient transfer learning. In NewrlPS, 2022.
files/
html
[22] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015, [RLttt:://arrat /ttp:v:///xxvv..e/ http:///waxivv...aabbbbbees/
[23] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01100 2019. URL
http://arxiv. org/ httggabb//991..... 01108
[24] Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao. and Yu Qiao. adaper-: Efficient fine-tuning of language models with zero-init attention. CoRR, abs/2303.16619, 2023. doi: 10.65550acXX22233..11999 URL 2303. 16199
https://doi. http:///ooo.rrg/////o/ org/10. 48550/ 48550/arXiv
[25] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. UtL.Btttss:/opeereiiie www.:eewwwwetoeeneeeeeeeeeeeeeee TJ4km2R5t7
[26] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequeee training for natural language generation, translation, and comprehension. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 7871 7880. Association for Computational Linguisties. 2020. doi: 1..1.......111 086653/1122000.0 acl- maadd-main...... 703. URL(https:
pre-
org/ 10. 11653/v1/20200 acl m03..703
[27] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 201 1, 2011. Utt. wtt.fffldd..s..frrrroooorooaoooe edu eelu/housenumbers/ hils22011.huuseemmbrr... housenumbers. pdf
[28) Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http://arxis: org/ abs/ 1412. 6980
[29] Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel Few-shot parameter-efficientt tne-ttnnig is better and cheaper than in-context learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL forum/id=rBCrWG-JB5
net/
[30] Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In Joaquin Quiñonero Candela, Ido Dagan, Bernardo Magnini, and Florence d' Alché- Buc, editors, Machine Learning Challenges, Evaluating Predictive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 1 11-13, 2005, Revised Selected Papers, volume 3944 of Lecture Notes in Computer Science, pages 177-190. Springer, 2005. doi: 10. 11 11736790L_9 URL /tt.///oi.... http::::///oooor/1110/0 1000/1.0001111111 1077/10367777
[31] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006.
[32] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan, The third PASCAL. recognizing textual entailment challenge. In Satoshi Sckine, Kentaro Inui, Ido Dagan,
13

---

Bill Dolan, Danilo Giampiccolo, and Bernardo Magnini, editors. Proceedings of the ACL- PASCAL@ @AL 2007 Workshop on Textual Entailment and Paraphrasing. Prague, Czech Re- public, June 28-29, 2007, pages 1-9. Association for Computational Linguistics, 2007. URL
https://aclarthollgg. org/ W07-14017
[33]
Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009, NIST, 2009. URL https: //tac www.....o...ai........iee.. wwwppppblications//// 2009/ additiional..... paattiionnlppppee.....m papers/RTE5 RTE5 overview. pra.lediwg.oceeiingg... pdf
[34] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10218663/11120200mmmllpp.. htttss:/acclanthollse org/ 2020. 111200emmlp-demss.
URL
[35] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Youncs Belkada, and Sayak Paul. Peft: State-of-the-arr parameter-efficient fine-tuning methods. https://github. com/
huggingface/ppfffy 2022.
[36] Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Paperss) Virtual Event, August 1-6, 2021, pages 7319-7328. Association for Computational I Linguistics, 2021. doi: 10. 10.18633/v122211 /htttps:///oi.orrtttteoo /aerg//1.18653//// http:///2222...11111.re acl-alcg.55500
acl-long.568.
[37] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Trans. Assoc. Comput. Linguistics, 7:625-641, 2019. doi: 10. 1122ta/cl_aa__0 _0090.
(tttss//:///do wrgo/o///// 10. 1112///cc___e00000000
[38] Daniel M. Cer, Mona T. Diab, Eneko Agire, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval- 2017 task 1: Semantic textual similarity multilingual and cross-lingual focused evaluation. CoRR, abs/1708 00055, 2017. http:::///aar
00055
[39] William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential para- phrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@ IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Natural Language Processing, 2005. URL htttp://///ccle http::://aclanthloogg org/ 5002//
[40] Karttikeya Mangalam, Haoqi Fan, Yanghao Li, Chao- Wu, Bo Xiong, Christoph Feicht- enhofer, and Jitendra Malik. Reversible vision transformers. JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA. USA, June 18- 24, 2022, pages 10820-10833 IEEE, 2022. doi: 10.1 (199/CVPR52688.2022.00 https://doi.oxg
URL
1109//10P0226BB RTW.://2258888222222022222 01056
[41] Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible resid- network:
Backpropagation without storing activations. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2214-2224, 2017. URL http:000044bbbbe--bbbbrrtt....td neurips wwtpriips.cppppppree hptppp/er////aaaaa/aaaea
[42] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net, 2020. URL https://openrevieet wwtp:ewwwwww..eeeeeeeeeeeee nwt.forumtmmeeteeeo rkgNKkHtvB
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von
14

---

Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, neur wwws.ccpaaae httppppper221//aaaaaee
pages 5998-6008, 2017. URL
[44] Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2381 1-991.. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d1-- 1260. URL https:
org/ 10. 18653/v1/d18-
[45] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence. AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020. The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY. USA, February 7-12. 2020, pages 7432-7439. AAAI 6239
Press, 2020. URI. https:///o/ aaai prg/inddex php/ wwtpp/ptp/taaticcl///////a/aaaea
[46] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the A12 reasoning challenge. CoRR, abs/1803.0553 005550, 2018. hRtp://///axxi http::///xxivv.o.gaaaa/ httg/abs/18000 1
[47] Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin, editors, Proceedings of the 3rd Workshop on Noisy User-generated Text, NUT@ EMNLP 2017, Copenhagen, Denmark, September 2017, pages 94-106. Association for Computational Linguisties, 2017. doi: 111113//1//111-4441111 URL http:: org 10. 101531////11114666
[48] Brian W Matthews. Comparison of the predicted and observed secondary structure of 14 phage lysozyme. Biochimica et Biophysica Acta (BoA)-Protein Structure, 405(2):442-451, 1975.
[49] Han Zhou, Xingehen Wan, Ivan Vulic, and Anna Korhonen. Autopeft: Automatic configuration search for parameter-efffcient fine-tuning. CoRR, abs/2301.12132, 2023. doi: 10.48550/arXiv.
2301.12132. URL httpp:///o/ org/ 10 48550//55iiv 2301 .2132
[50] Guangxuan Xiao, Ji Lin, and Song Han. Offsite-tuning: Transfer learning without full model. CoRR, abs/2302.048700 2023. doi: 10.44850/arXiv22000.44887 |Rtttsss:://ao org/10
48550/arXiv 2302.048770
[51] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F, Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR, abs/1710.037400, 2017.
org/ haagg//bb//// 1710
03740
[52] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito,,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[53] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https //openreview. //openeeview.ee//ffrrmm///
[54] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. In NeurlPS, 2022. URL http::/papprs..att 1at-55599908--bbtraaace www.rracc-Cooferrreaaaaeee Conf Connerence.... html
cc wwtpccc/ppppppeeeeeeeeeeeeeeeeee fiiaaerrffieeppppppeeeeeeee wtt.ee/epppee//2///aaaaaee
[55] Baohao Liao, David Thulke, Sanjika Hewavitharana, Hermann Ney, and Christof Monz. Mask more and mask later: Efficient pre-training of masked language models by disentangling the [MASK] token. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Findings of the Association for Computational Linguistics; FMNLP 2022, Abu Dhahi, United Arab Emirates, December 7-11, 2022, pages 1478-1492. Association for Computational Linguistics, 2022.
15

---

doi: http:///www/2222..nnnggnnnone/ findings-emi
URL http://///oo..0g1.11118/////////
[56]
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Yan Liu. MASS: masked sequence to se- quence pre-training for language generation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926 5536. PMLR, 2019. URL song19d.
mlr www.:ml........e.mm /
[57] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer. and Veselin Stoyanov. Unsu- pervised cross-lingum representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 1440-8457 Association for Computational Linguistics, 2020. doi: 10. (0/6:////2020.ac--mmm 747. URL https://doi... http:://ddoooorg///// org/10.10/ 1e55/00.1866.33 18653/863//0 vv/ 202010022aaaa. acl-main. 747
[58] Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bus Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 7057-7067, 2019. URL https: htttp::::ppocceednnnnnn neur ips catsscccpapppp cc/paper/ 2019/
[59] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog. 188):9 2019.
[60] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR. abs/2005. 14165,, 2020. URL. pectps://arxii org/ abs/ 14165
[61] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.000 108361, 2020. URL https:/t http//xxxvv..rr/g
htrg/abs/2001... 08361
[62] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas. Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. CoRR, abs/2203.155566 2022. doi: https://////01 org/ 0ete//....855/5// 48550/arXiv...//a// 2203. 15556
10.485550/rXX1.2223.5555
[63] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, htbp:/2044022311. 2022. doi: 10448500aaxiw.224440011 URL https:/ org/10. 405.0/8550aa0 arxiv 2204. 02311
[64] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kuishreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun,
16

---

Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung- Ching Chang. Igor Krivokon, Will Rusch, Mare Pickett, Kathleen S. Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas Claire Cui, Marian Croak, Ed H. Chi, and Quoc Le. Lamda: Language models for dialog applications. CoRR, abs/2201 22220,9,,,, 2022. httpt::s://rre
Chung-Ching
org/ abs//201...2... 08239
[65] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, sett/forummiddgEze00oom
2022. OpenReview.net, 2022. URL /
[66] Niklas Muennighoff, Thomas Wang. Lintang Sutawika, Adam Roberts. Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through mul- titask finetuning. CoRR, abs/221 1,01786. 2022, doi: 10.48550/arXiw2221...1 https://doi. 10w5/.. 48550/arXiv. 221...22221111111. 01786
URL
[67] Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. In Sth International Conference Learning Representations. ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017. URL hattps://openreee
net/forum? SeieruumiddrrUU@@HHccr
[68] Yuning Mao. Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma. Jiawei Han, Scott Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, Dublin, Ireland, May 22-27, 2022, pages 6253 0264.. Association for Computational Linguistics. 2022. doi: 10. 1tt/3///1/2222alllllll 433. URL https:://oi..rr:::/ 006633.18653/
v1/2022. 1onl-llog.... 443
[69] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, abs/ abss/600.006144 2016. URL http::/arxii .rg/ htt///bb/11644/ 06174
[70] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse. trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[71] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3259 3269. PMLR, 2020. URL/htpp! //proceedings. mlr. wwwpppeeevvvleeeeeeeee v119 wee111/ffaaakee frankle20a....mm html
[72] Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. LIT: learned interme- diate representation training for model compression. In Kamalika Chaudhuri and Rus- lan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Pro- ceedings of Machine Learning Research, pages 3509-3518. PMLR, 2019. URL http!
//proceedings. mlr pwwss/vv11111 v97 htt///krrtaaaaa...a1
gttp://.reeee////tataaaaaaaaaaaa html
[73]
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training. In Irina Calciu and Geoff Kuenning, editors, 2021 USENIX Anmual Technical Confer- ence, USENIX ATC 2021, July 14-16, 2021. pages 551-564. USENIX Association, 2021, URL https:// www.ix...i....omoo....
conf http://ooooooeeccceeeceeeeeeaaap
[74] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: accurate post- training quantization for generative pre-trained transformers. CORR, atts/211.. 17323, 2022. doi: 10.48550/r2xi.22110.722 URL :/doip:://oi.... org/10 200.448550rrr/ arxiv www.exiiiiii....... 17323
17

---

[75]
Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7947-7969. Association for Computational Linguistics, 2021. doi: 10. 18653/v 1tt.1//2220mm//// (e0.2....mlp--mi.622))) URL https://doi.org::
18653/v1/2021 wwneee main. 627
[76] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block- wise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL http::/t://ppeeerevi..
[77] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305. 14314, 2023. doi: 10.48350/arXiw.235..k8 143144
httpp://:///doi org/ 10. 1a850//rriiv 2305 14314
[78] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta. Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM, 2020. doi: 10.1 145/33994486633467703 URL 1145/3394486 3394486 3406703
https://doi.. org 10. 1145/
18

---

A Related works
Pre-trained language models. PLMs, which are trained on extensive datasets for a common task such as predicting masked words 2 a sentence, play a vital role in facilitating knowledge transfer to downstream tasks. They have demonstrated remarkable achievements across various upplications. consistently delivering state- of-the-art outcomes. Furthermore, scaling up PL.Ms has proven to yield predictable enhancements in performance for these downstream tasks 66 62 Consequently, the size of released PLMS has progressively grown, reaching unprecedented scale of 100 billion parameters ( [1 66 [3] [4]
26
55
56
58 or the next word [59 60 in
Such large-scale PLMs unveil extraordinary capabilities, enabling zero-shot or in-context learning (59 60] for broad spectrum of tasks. Nevertheless, transfer learning remains prevalent approach for effectively deploying these models in new task scenarios 22 65 66], which post unparalleled requirements the computing resources.
With the advent of large-scale PLMs, a new method that aims to reduce storage requirements. PEFT, has been proposed 11 [5] 19. PEFT adds and trains small number of parameters while matching the performance of full fine-tuning. There are various ways to add new parameters. For example, Houlsby et al. [14] and Pfeiffer et al. [16] insert small bottleneck modules (adapters) to the PLM. LoRA [17] injects rank decomposition matrices into the pre-trained weights. HiWi [13 inserts the pre-trained parameters to low-rank adapter. (IA)3 22 scales the
Parameter-efffcient fine-tuning.
pre-trained weight with a trainable vector. Prompt-based methods [1 [9] append a sequence of trainable vectors to the word embeddings or attention components. Recently, some unified methods, which combine multiple PEFT methods in a heuristic way [18] or with the technique of neural architecture search [49 67 68 have also been proposed. Though PEFTs save the storage by large margin compared to full fine-tuning, they still require a similar memory footprint during training as
full fine-tuning 20 21 because of the activation memory.
Memiry-efficient training. Memory-efficient training aims to reduce the memory footprint during the training process. Reversible neural networks (40 41 42 reduce the activation memory by
recomputing the activations with the outputs during pack-propagation. Gradient checkpointing [69]
trade computation for memory by dropping some intermediate activations and recovering them from an extra forward pass. The activation memory is (1)) and O(vN) for reversible neural networks and gradient checkpointing, respectively. MEFT is the first method that is proposed to modify PLM to its reversible variant. When applying MEFT to further reduce the activation memory for the layers with vanilla gradient.
deeper model, can use gradient checkpointing
Network compressions, like pruning 70 71 and knowledge distillation 22 footprint for both training and inference. They compress PLM a smaller model by either deleting unimportant parameters or distilling knowledge from the PLM to the smaller model. Treating PLM as a feature extractor and avoiding its gradient calculation is also an effective way to reduce the activation memory [20 21). However, these methods normally require extra pre-training before fine-tuning, or achieve a lower performance compared to full fine-tuning when using the same PLM.
save the memory
B Limitations
We acknowledge the main limitation of this work is that we only evaluate our proposed methods on a limited amount of tasks and don't 't conduct experiments on the encoder-decoder models. The main reason for the limited amount of tasks is that our computing resources are constrained. In addition, the major criterion for our selection of the underlying models is that we could find many strong baselines on them without reproduction. BERT and RoBERTa fulfill this criterion very well on the GLUE benchmark. Regarding the encoder-decoder model, recently there is a clear trend of applying a decoder-only model on sequence-to-s seuuence-o--equence sequence tasks. Therefore, we apply OPT in this paper and plan to include LLAMA ||| for the instruction mnstruction-iiettuinng data in the future.
Another limitation of MEFT is its lower score when trained in FP16 and on a deeper model. We have discussed this problem in numerical range and by a deeper model because of the error accumulation. Fortunately, the results are still comparable to the PEFT baselines when trained in FP16. Even trained in FP32, the activation memory footprints don't increase compared FP16. One only needs to spend more training time in FP32 when using the same batch size in FP16 (about 20% more training time). However, since
In sum, more reconstruction error is introduced by FP16 due to its
19

---

MEFT's reduce the memory footprint, larger batch size during training is possible. which can save some training time. For deeper models, we offer a practical and effective setting in Figure
Last but not least, when fine-tuning larger models, like OPT and OPT..7 7B [9] the peak memory footprint is occupied by the model parameters rather than the activation (see Table 3). One needs to combine other techniques with MEFT to reduce the peak memory footprint, like loading the model in FP16 even int8 rather than in FP32, combining MEFT with ZeRO 73 as in Table
C Step-by-step design for MEFT,
For the reader's easy understanding. in this section, we explain MEFT, step-by-step. First, let's re-emphasize the guiding principles for our design: (1) For each reversible layer, we must have two inputs and two outputs as in Figure Ba (2) We need to follow the starting point hypothesis. Le. whenever modify PLM layer, as the original PLM layer if we input the same input of the original PLM layer to the modified layer at the beginning of training. If the outputs not similar, they become even more dissimilar after multiple layers, tearing down the PLM's initialization.
need to ensure the modified layer has almost the same output
As shown in Figure Sa for the first PLM layer, ho is the input and h1 is the output. In Figure 8b the inputs to the first reversible layer is hgg = hi = hg. Recapping the architecture of Fi in Figure 4c (up), we simply insert adapter in parallel to the two consecutive feed-forward layers. and initialize the adapter as Wdmon: W.,. up ~ (O, 0.02²), ), which results in h, R Fi (ht) since hg = ho- we set
+ 0. h Xh + Fi (hz) h1. In this way, h plays the role of preserving the starting point. Now let's consider h² Due to our initialization of the adapter, the output from 91 (G1 is simply an adapter as in Figure 4c (down)) is close to So h She + 91 (h,) ßho + Shg- After switching the order of h and h h ~ Shg and h x h1.
For the second reversible layer, if we don't switch the order of h2 and hj. it looks like Figure &c The input to F2 is Shg. which breaks down the representation continuity of a PLM since the input to the pre-trained F2 should be close h1 If we switch their order in Figure 8d we preserve the representation continuity. And it results in h22 = ABho + (21) s h2 due to X + 0 and h2 (h1). Similar the first reversible layer, h s Bh1. After switching. h! Bh1 and h2 ~: h2. By analogy, for the nh reversible layer, h x Bh,-1 and h hn.
After the final layer, we simply take the mean two outputs as h'y = (hly+ + h and input h'y to task-specific head, like classification layer. The design procedure is similar for MEFT, and MEFT,, sum, order switching is mainly for preserving the representation continuity, and setting the scaling factors close to 0 is mainly for preserving the starting point.
D Implementation details of the question-answering tasks
Compared GLUE tasks where all tasks are classification tasks and the classification heads are randomly initialized, the question-answert tasks are sequence-to-sequees tasks and need the pre-trained output layer that shares the same parameters as the word embedding layer. The output
hn
A
Layer Norm
Feed-Forward
Fead-Forward
*
Attention Block
Rg
hn-1
Ag = kā hy
hy
figy
hy
phh
phg
hy
pko
hy
(a)
(b)
(c)
(d)
Figure 8: (a) The th PLM layer; (b) The first MEFT, layer; (c) The second MEFT, layer without order switching; (d) The second MEFT, layer.
20

---

layer requires the continuity of representation. L.e at the beginning of training. the input to the output layer, h should be close to hx. Therefore, we need to do a modification to h'y instead of using h'y (hl + //2
Here we introduce new scaling factor ry and require 7 O. For MEFT,, since h hv (see Table p). we set hty = n + f se fe Rs hv. Similarly, h = h2 + 2 2 hy ~ NN for MEFT2, and ft'y % h hy NN for MEFT3. Without any tuning, we set 7 0.1 as
other tuned scaling factors by default.
gradiett checkpoininng
LoRA
gradient checkpoiting
36
140-
MEFT;
120-
36
Samples
100
34-
60
33
128
250
120
Adtivation Meroery (GB)
2.5
10
2.0
1.5
1.0
128
258 Sequence Length
512 32
12B
Batch Size
Figure 9: Throughput and activation memory for different sequence length and batch sizes BERT By default, the sequence length 512 and the batch size is 32. For your reference, LoRA's throughput is 52.7 samples/second without gradient checkpointing for the default setting. Overall, MEFT shares the same level of throughput as LoRA with gradient checkpointing, while it is the lower bound of the activation memory for different settings.
E
More results
E.1 Compared to gradient checkpointing
Previously, we only theoretically stated that the activation memory for reversible network and gradient checkpointing is (1) and O OV.N), respectively. In addition, we didn't 't compare the training time of MEFT with PEFT in detail. Here we offer some empirical results for your better understanding.
In Figure we compare activation memory and throughput among MEFT LoRA with gradient checkpointing and Full FT with gradient checkpointing. The throughput for all three methods is at the same level, maximum 12% difference between LoRA and MEFT when the sequence length is 128 and the batch size is 32. With an increasing sequence length, the gap becomes narrower to 7.5%. Notably, the throughput for LoRA without gradient checkpointing is 52.7 samples/second. With gradient checkpointing. it is 36.1 samples/second, 69% of the original throughput. For MEFT with the same setting. it is 33.5 samples/second, 64% of LoRA's throughput without gradient checkpointing. In sum, MEFT's throughput is at the same level LoRA's 's with gradient checkpointing, and about 64% of LoRA's without gradient checkpointing. In addition, MEFT's activation memory is always the lower bound among these three methods. The gap between LoRA with gradient checkpointing and MEFT becomes larger with an increasing sequence length and batch size.
F.2 Compared to quantization methods
Quantization is an orthogonal method to MEFT, which reduces the memory footprint of training or inference by reducing the parameter size to fewer bits and using low-bit-precision matrix mul- tiplication. There mainly three different quantization methods: (1) Post-training quantization
21

---

Table 5: Compared to QLoRA. r 8 for all methods. Experimental setting stays the same as the default setting in Figure 9
Method
Activation Memory (GB)
Samples/Second
LoRA + gradient checkpointing
2.62
36.1
QLoRA gradient checkpointing MEFT
2.97
8.7
2.33
33.5
Table 6: Combine MEFT with ZeRO.
Method
Peak Memory (GB)
Activation Memory (GB)
MEFT;
28.2
8.2
MEFT + ZeRO
6.4
6.4
that quantizes trained model after pre-training or fine-tuning: (2) Lower-bit optimizer 776 that stores the optimizer state with lower precision and de-quantizes it only for the optimization, similarly FP16 or BF16 mixed precision training but with lower-bit; (3) Lower-bit frozen LLM with LoRA, i.e. QLoRA 777, that applies 4-bit quantization to compress the LLM. During fine- finettuning, QLoRA backpropagates gradients through the frozen 4-bit quantized LLM into the low-rank adapters. Notably, the computation data type for QLoRA is BF16. It de-quantizes weights to the computation data type to perform the forward and backward passes.
A 75
To some extent, all these three methods are orthogonal to our method and can be combined with MEFT: (1) Post-training quantization is mainly for reference and can be applied any trained models; (2) 8-bit Adam can also be applied to any models trained based on gradient; (3) QLoRA combination of (1) and (2). For QLoRA, we conducted some experiments on BERT hase with the default setting as Figure [9] As shown in Table [5, METF, saves the most activation memory while having a similar throughput as LoRA with gradient checkpointing. The reason for the larger activation memory of QLoRA than LoRA is that it has an additional de-quantization step, which also causes its smallest throughput.
E,3
Combine MEFT with ZeRO
7B saves memory by partitioning the model's parameters and optimizer state among GPUs or between GPU and CPU. This method is orthogonal to MEFT, since MEFT saves memory from activations. We conduct some experiments
ZeRO
OPT by combining our method with DeepSpeed ZeRO stage that offloading model's parameters and the optimizer state to CPUs. As shown in ZeRO significantly reduces the memory footprint from the model's parameters, therefore reducing MEFT's peak memory from 28.2GB to 6.4GB.
78
Table 6
Table 7: Fine-tuning settings. Check 4.2 for the fine-tuning setting on BART.
Hyper-parameter
GLUE
Question-Answering
RTE, MRPC, CoLA
SST-2, QNLI, QQP, MNLI
Learning Rate
(5. 7,8] 10
(3. 4, 5) 10
(1,3,5,7 5) . 10 (8, 16,32)
Barch Size
(16, 32)
{16. 32)
Max Fipochs
(20, 40)
{10, 20)
[3, 10) 0.1 1
Weight Decay
0.1
0.1
Max Gradicnt Norm
Warmup Ratio
0.06 Linear
0.06 Lineur
0.06 Lineur
Learning Rate Decay
22

---

Table 8: Statics of datasets
Tisk
RTF
MRPC
STS-B
CoL.A
SST-2
QNLI
QQP
MNLIIm
MNI.Imm
#Training
2.5k
3,7k
5.8k
8.6k
67.4k
104.7k
363.8k
392.7k
#Development
0.3k
0.4k
1.5k
0.9k
5.5k
40.4k
9.8k
9.8k
Task
OpenBookQA
PIQA
ARC-E
ARC-C
SeiQ
#Training
5.0k
16.
2.3k
L.lk
11.7k
#Development
0.5k
3.1k
2.4k
1.2k
1k
Table 9: Statics of models
Model
#Parameter
#Layer
model
Size in FP32 (GB)
BERT BART lurge encoder
110M
12
768
0.4
205M
12
1024
0.8
RoBERTapage
355M
24
1024
1.4
OPT 133 6P76.7B
1.3B
24
2048
5.2
6.7B
32
4096
25.6
def backward Backward_passssee1. y1, y2, dy1, dy2):
with torch enable_grad ():
y1 re.requires_gra = True
# The intermediate activations of G are stored g-y1 self.G(y1)
# Obtain the gradient of y1 g-y1. backward (dy2, retain grahh
with torch, no.ch..n__rr((;;;
10
x2 (y2 8-y1) / self x2 factor # Save memory, same for below del g-y1, y2
11
12
dy1 += grad # Save memory y1.grad None
15
16
17
with torch. enable_grad
x2. requires-grad - True
# The intermediate activations of F stored f_x2 se2f.F(x2)
20
21
# Obtain the gradient of x2 f_x2. backward (dy1, retain.graph-Falee)
22
24
with torch, no-grad
35
x1 (y1 f_x2) / self factor del f_x2, y1
26
dy2 *= self.x2_factor
28
# dx2=
memory by using the same variable
29
dy2 x2.grad
30
x2 gr.grad= None
# dy1=dxi
dy1 *= self. xi_factor
33
x2 xetacht ()
34
return x1, x2, dy1, dy2
Listing Backward pass for each Layer. The peak memory happens Line 10 Line 25, depending on whether the subnetwork G is larger than F or the opposite. In the code, we use x1, x2, yl, y2, x1. factor, x2 factor represent h h 1 12 h2 À and 8, respectively.
23

---

Table 10: Compared V-Tunnig RoBERTapage- We exclude the memory of J'-Tuning for BART T because it was not reported. Instead, the memory usage of Y'-Tuning for RoBERTaaage was reported. Notably, the STS-B task is excluded from the calculation of the average score, because was not evaluated in Liu et al. [20].
Model
#Parameter
Peak Memory (GB)
Average Score
Full FT
100%
11.47
88.4
LoRA
0.23 %
6.11
88.1
V-Tuning
4.57%
2.08
82.1
MEFT,
0.23%
3.63
88.4
default init
default init with randor PLM
a 0, trainable a
1/c
80
Average Score (%)
60
40
20
0
0.05
0.1
0.5
C
0.1
0.5
0.9
Figure 10: The initialization effect for PEFT, Left: LoRA, Right: (IA)). Instead of initializing
W = c like Figure 2b here we initialize it as Wap A 0.02?), which should be more suitable for training due to its asymmetry. For convenient comparison, the results of W Wap (in grey) are also included. Overall, the results between W., comparable. However, when e = 0 for LoRA, the result of Gaussian initialization is slightly worse than the constant initialization. This further supports starting point hypothesis, since the Gaussian initialization can't san't guarantee the output from the adapter is strictly equal to zero at the beginning of fine-tuning.
and W., up [c,, 0.02
24


Reviews:
[{"review_id": "kyBSNcHrp5", "reviewer": "Reviewer_mzDQ", "summary": "Authors propose a memory-efficient (MEFT) variant of parameter-efficient fine-tuning (PEFT), which utilizes reversible networks to save memory stored for activations during model forward propagation. The authors showed insights into the reasons for the effectiveness of PEFT. They evaluated several variants of MEFT with GLUE and QA datasets, indicating that the proposed method does not produce performance gaps while reducing memory usage.", "questions": "Please refer to the weaknesses section.", "limitations": "–", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method shows competitive results with benchmarks\n- Experiments with the initialization of LoRa are insightful \n- Generally, the proposed methods are useful for practitioners", "weaknesses": "- The presentation could be improved. E.g., I struggled with Section 2.3 since $h^1, h^2 $ (L136) was not defined above.\n- It would be fascinating to compare the proposed method with gradient checkpointing used for training with LoRA/etc. Naively, the usage of invertible layers should be faster. However, seeing such an evaluation in the paper would be highly beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work: \n\n(1) Offers competitive results;\n\n(2) Conducts insightful experiments for LoRA's initialization;\n\n(3) Is useful for practitioners.\n\nWe have responded to your comments below and updated our draft accordingly. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n > 1. The presentation could be improved. E.g., I struggled with Section 2.3 since  $h_n^1$ and $h_n^2$ (L136) was not defined above.\n\nThank you for this advice. Compared to the previous annotation $h_n$ (Line 81) representing the input to the $n^{th}$ layer, we add a superscript to distinguish the two inputs to the reversible layer. We have added a clarification to it for our new draft.\n\n> 2. It would be fascinating to compare the proposed method with gradient checkpointing used for training with LoRA/etc. Naively, the usage of invertible layers should be faster. However, seeing such an evaluation in the paper would be highly beneficial.\n\nThank you for this advice. We offer these new results in the 1$^{st}$ point of the general response. **In sum, MEFT shares the same level of training throughput as LoRA with gradient checkpointing, while it saves the most activation memory for different settings.** \n\nIn addition, there is a misunderstanding in your statement. The throughput of MEFT and LoRA with gradient checkpointing is at the same level because both of them compute two forward passes and one backward pass. The second forward pass is for the activation reconstruction. "}, {"review_id": "9qkzhLI2PI", "reviewer": "Reviewer_dsz7", "summary": "This paper points out that the existing parameter-efficient fine-tuning (PEFT) methods are not memory-efficient, since they need to cache most of the intermediate activations for the gradient calculation. To resolve this problem, the authors introduce to use the reversible model to reduce the activation memory. Specifically, this paper have these contributions: 1) it is important to preserve the weight of PLM at the initialization; 2) insert adapters and make it reversible without additional pre-training. Extensive results also demonstrate the effectiveness of the proposed method.", "questions": "1. This paper presents three variants of MEFT, but is there any conclusion that which one should be chosen under different scenarios?", "limitations": "The authors have presented limitations of their proposed works, and currently, these limitations are still a problem for the proposed methods. Besides, I also give some comments about the limitations of this paper (please see my comments in **Weaknesses**). I do not see any potential negative societal impact. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper points out the importance of memory-efficient fine-tuning in current pre-trained models. And the authors introduce the idea of RevNet into the pre-trained language model, and design three reversible variants to PLM.", "weaknesses": "1. Some conclusions seem naive. For example, preserving the starting point from the pre-trained model at the beginning of training is a common experience. Previous works like LoRA also point out the importance of this point, and this point is also a common method to avoid catastrophic forgetting.\n2. Memory-efficient training is important, but the authors do not mention quantization in this paper.  Quantization has been considered as the one of most effective methods in LLM to save memory. While for RevNet or the proposed architecture, these reversible-based neural networks actually are a trade-off between computation and memory (it will bring more computation to save memory). So, I do not think the proposed method is advantageous to Quantization and directly comparing the proposed methods with PEFT may also not be appropriate. \n3. Essentially, the proposed method adopts an adapter structure and then uses the reversible structure into the transformer and presents three patterns (which formulate different $\\cal{F}$ and $\\cal{G}$) and trains each network from the start point. The idea is to combine existing work (RevNet) with current PEFT methods, and then combine the initialization that preserving the starting point from the pretrained model. So, I think the contributions are limited.\n4. In experimental results, authors report MEFT with vanilla gradient. I think such reports are confusion, since the main contribution in this paper is the reversible gradient with MEFT structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! \n\nWe have responded to your comments below and updated our draft accordingly. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n\n> W.1. Some conclusions seem naive. Previous work already points out the importance of preserving the starting point.\n\nWe acknowledge that preserving the starting point from the pre-trained model is not a new concept (we also claimed this in Lines 109-110). In this paper, we explore its significance by conducting thorough and well-established experiments. To the best of our knowledge, our paper is the first work to explore this concept. In addition, **we also have some non-intuitive findings from Figure 2(b):**\n\n**(1) When $c >= 0.05$, the results are even worse than the one with a randomly initialized model for LoRA.**\n\n**(2) A $\\alpha$ with an initialization of 0 for LoRA or the inverse of $c$ for (IA)$^3$ can make the bad results become way better, even better than the default initialization for (IA)$^3$.**\n\nWe understand that some conclusions might appear straightforward or \"naive\" when viewed in isolation. However, in the broader context of our research, these conclusions serve a pivotal role in building upon and connecting the various aspects of our study. We will ensure to emphasize the distinctions and novelties of our approach more clearly in the revised draft.\n\n> W.2. Missing comparison with quantization methods.\n\nThank you for this advice. We have conducted new experiments on QLoRA [3] and show the discussion and results in the $2^{nd}$ point of the general response. Apart from quantization, we also combine our method with ZeRO [4] and show the results in the $3^{rd}$ point of the general response. \n\n>  W.2: The proposed method is not advantageous to quantization and directly comparing the proposed methods with PEFT may also not be appropriate.\n\nWe politely disagree with this point. Quantization is also a trade-off between time/computation and memory. Taking QLoRA as an example (check the new results in the $2^{nd}$ point of general response), the throughput is 8.7 samples/second, while MEFT is 33.5 samples/second. Though both methods target the same issue, memory-constrained training, they focus on different aspects. Quantization reduces the memory footprint occupied by the model's parameters or the optimizer state, while MEFT reduces the activation memory.\n\nRegarding the inappropriate direct comparison with other PEFT methods, our intention is to show that the MEFT's task performance is comparable to PEFT's, while having the memory-efficient characteristics at the same time. In addition, we also included other strong memory-efficient baselines in Table 2, like Y-Tuning and LST. Thanks to your advice, we further include more baselines, like gradient checkpointing, quantization and ZeRO.\n\n> W.3: Limited contribution because of the combination of PEFT, RevNet and the starting point hypothesis.\n\nWe acknowledge that our approach builds upon established techniques, but we'd like to highlight the novelty and advantages of our approach:\n\n(1) RevNet and Transformer share two distinct architectures, as shown in Figure 4. Converting a pre-trained non-reversible model to a reversible model poses a totally new challenge that is not well-explored.\n\n(2) By thoroughly investigating the key factor for the success of PEFT, we propose an intuitive hypothesis, the starting point hypothesis, which is the foundation of MEFT.\n\n(3) Through dedicated design, we convert a pre-trained Transformer to its reversible variant without sacrificing the performance, while having the advantage of saving activation memory.\n\n(4) As a side product, MEFT with vanilla gradient outperforms all baselines, serving as an effective PEFT method. \n\nWe don't simply combine the techniques of PEFT and RevNet to obtain our final result. Instead, we offer a new way to explore PEFT and pre-trained models.\n\n> W.4: Confused results from MEFT with vanilla gradient.\n\nSorry for showing MEFT with vanilla gradient and reversible gradient in the same table, which may distract your attention. We will split them in our updated version. But the results for MEFT with vanilla gradient are also important since they are the best among all baselines. It shows MEFT with vanilla gradient can serve as an effective PEFT method.\n\n> Q.1: Which variant of MEFTs should be chosen under different scenarios?\n\nAccording to Table 2, MEFT$_3$ should be chosen if one only wants to use MEFT as a PEFT method with a vanilla gradient. MEFT$_1$ should be chosen when concerned about the training memory.\n\n> Limitations: The stated limitations in the paper are still problems for MEFT.\n\nThe main limitations we stated in this paper (Appendix B) are:\n\n(1) Limited amount of downstream tasks, only in NLP area with 8 classification tasks and 5 question-answering tasks. \n\n(2) Lower score when trained with FP16 and on a deeper model.\n\n(3) The memory footprint occupied by the model's parameters is larger than the one occupied by the activation for a large language model.\n\nFor all three limitations, we have resolved them and offered practical solutions:\n\n(1) We successfully transfer our method to the computer vision area. Please read the $4^{th}$ point in the general response. \n\n(2) For the lower score when trained with FP16, the score is worse than the one trained with FP32. However, it is still comparable to other memory-efficient baselines, 81.9 vs 82.1 of LST. LST was also trained with FP32 and requires > 3 times activation memory. For the deeper model, we offered some practical settings in Figure 7 and showed its effectiveness in Table 2 and 3.\n\n(3) We can combine MEFT with ZeRO, reducing the peak memory from 28.2GB to 6.4GB.\n\n[6] Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020). Reformer: The efficient transformer. \n"}, {"review_id": "zJ2gTz5aSs", "reviewer": "Reviewer_X2di", "summary": ">**Rebuttal:** The provided details satisfy my concerns. I think this paper should be accepted after applying the agreed changes.\n\n>**TL;DR:** The proposed MEFT reduces training activation memory, which is an important topic for the ML community. MEFT outperforms existing PEFT techniques while consuming up to 84% less memory compared to full fine-tuning. However, MEFT comes with significant computation overhead, which is not explored. Furthermore, some key comparisons, such as gradient checkpointing, are missing. Addressing my concerns and questions would improve my score.\n\nThis paper proposes the Memory Efficient Fine-Tuning (MEFT) technique that achieves similar accuracy to that of traditional PEFT techniques while consuming up to 84% less memory compared to full fine-tuning. The paper discusses previous PEFT techniques and provides insights on key attributes for them to work. These insights include empirical results and detailed explanations. One of these key insights is the weights initialization. The paper shows the importance of initializing the PEFT weights such that they do not have any effect on the underlying LLM before starting the training process. \n\nThe paper proposes to combine reversible models with PEFT existing techniques to create MEFT algorithms that gain from both worlds. The paper provides three different variants of MEFT, which are explained as well as illustrated. The three variants of MEFT are empirically validated on four different architecture backbones BERT, RoBERTa, BART and OPT. The MEFT variants are compared to several popular PEFT techniques as well as some memory efficient techniques. These comparison are conducted on the GLUE dataset. ", "questions": "* **Q.1.** The paper proposes MEFT which sacrifices computation speed at the cost of lower peak memory. There are several existing common approaches (such as gradient checkpointing and DeepSpeed) for general memory efficient training which are compatible with PEFT techniques. Why are these comparisons not explored or detailed?\n* **Q.2.** In the LoRA modified initialization method, both $W_{up}$ and $W_{down}$ are initialized with a single constant. This causes unwanted symmetries, which makes it more difficult to learn. An alternative approach would be to initialize $W_{down}=1$ (as tested in this paper, but instead of initializing $W_{up}$ to a constant, initialize it with a random gaussian distribution where $c$ can act as the distribution mean. Does this make sense?\n* **Q.3.** In Line #175 it is noted that the adapter weights' $W_{down}$ and $W_{up}$ are initialized from a random gaussian distribution with zero mean. It is further noted, that the outputs of the adapter would be close to zero. However, in practice this does not hold true,  and the actual outputs can significantly deviate from the desired zero output. What am I missing here?\n* **Q.4.** In Figure 6 it seems that LoRA only reduces slightly the peak memory in comparison to the full fine-tuning. However, the optimizer states, which are generally 3 times the size of the model, are significantly smaller in LoRA. Does this result hold for other LLMs and how does the batch size affect it as well?\n* **Q.5.** The MEFT technique does not seem to be only compatible with LLMs and can be compatible with other domains, as sted in the conclusion section. However, the experimental results are solely on LLMs. What is the reason for that?\n* **Q.6.** In Table 2 the memory usage of Y-Tuning is missing, why is that?\n", "limitations": "The limitations are discussed in paragraph line #315. These limitations include limited compatibility with quantization techniques due to training instabilities and significantly longer training time due to activation re-computation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* **S.1.** The proposed MEFT algorithm tackles an important problem in existing PEFT techniques, which makes LLM PEFT training more accessible to researchers with low resources by reducing the activation memory up to 84% of full fine-tuning.\n* **S.2.** The paper provides interesting insights that are explained and backed by results and detailed illustrations.\n* **S.3.** The proposed MEFT variants outperform existing PEFT techniques on GLUE dataset with several architecture backbones.\n* **S.4.** Reproduction code is provided as part of the submission.\n", "weaknesses": "* **W.1.** The training time of MEFT is not well explored and is only noted as \"double the training time\". Hence, the tradeoff between memory and computation efficiency is not well understood.\n* **W.2.** Popular existing memory efficient training techniques such as tensor rematerialization (gradient checkpointing) [2][3] and ZeRO [1] are not compared to. These techniques sacrifice training speed for reduction in memory consumption, similar to the goal of MEFT. Furthermore, these techniques are compatible with existing PEFT approaches. Finally, these techniques are widely available in deep learning frameworks such as PyTorch, DeepSpeed, and Hugging Face Accelerate which makes comparison relatively simple.\n* **W.3.** The font sizes in Figures 1,2,3,5, and 6 are too small which makes it very difficult to read. Furthermore, some figures are not self explanatory, such as Figure 5 which does not provide any explanation for the difference between the solid and dotted lines. These problems burden the reading flow of the paper.\n\n**Typos.**\n* Line #86: \"for obtaining\" → \"to obtain\"\n* Line #86: \"though we\" → \"even though we\" \n* Line #95: \"in Figure 2a\" → \"in Figure 2a,\"\n* Line #97: \"though with\" → \"even with\"\n* Line #106: \"from an modified\" → \"from a modified\"\n* Line #114: \"the added and\" → \"the added\"\n\n\n[1] Ren, J., Rajbhandari, S., Aminabadi, R.Y., Ruwase, O., Yang, S., Zhang, M., Li, D. and He, Y., 2021, July. ZeRO-Offload: Democratizing Billion-Scale Model Training. In USENIX Annual Technical Conference (pp. 551-564).\n\n[2] Jain, P., Jain, A., Nrusimha, A., Gholami, A., Abbeel, P., Gonzalez, J., Keutzer, K. and Stoica, I., 2020. Checkmate: Breaking the memory wall with optimal tensor rematerialization. Proceedings of Machine Learning and Systems, 2, pp.497-511.\n\n[3] Beaumont, O., Eyraud-Dubois, L. and Shilova, A., 2021. Efficient combination of rematerialization and offloading for training dnns. Advances in Neural Information Processing Systems, 34, pp.23844-23857.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your time, effort, and thorough reviews! We are really encouraged by the reviews that highlight our work:\n\n(1) Tackles an important problem in the existing PEFT methods.\n\n(2) Offers insightful observation for the success of existing PEFT methods.\n\n(3) Offers competitive results on different pre-trained models.\n\nWe have responded to your comments below and updated our draft accordingly by improving the presentation, citing missing references, and adding new baselines. If you still have additional questions and suggestions, we would be happy to answer your questions and incorporate the suggestions into an updated draft.\n\n> W.1 and Limitations. The computation overhead/ training time of MEFT is not well explored.\n\nThank you for this advice. We have conducted related experiments and show the results in the $1^{th}$ point of the general response.\n\n> W.2 and Q.1. Missing baselines from gradient checkpointing and ZeRO.\n\nThank you for this advice. We have conducted related experiments and show the results in the $1^{st}$ and $3^{rd}$ point of the general response.\n\n> W.3. Too small font size and less self-explanatory captions for the figures.\n\nThanks for this advice. We already made a change to the font size and caption in the uploaded PDF for new results. All other figures have been adjusted in our updated draft.\n\n> Q.2. Is it better to initialize $W_{up}$ ~ $\\mathcal{N}(c, \\sigma^2)$ instead of $W_{up} = c$?\n\nThis is a very inspiring observation. According to your suggestion, we repeat the experiments in Figure 2 by initializing $W_{up}$ ~ $\\mathcal{N}(c, 0.02^2)$, and show the results in Figure 10 in the uploaded PDF in the general response. **Overall, their results are similar**. One reason might be the limited amount of updated parameters, only about 0.27%, which makes the optimization easy.\nAnother interesting observation is the slight degradtion of $W_{up}$ ~ $\\mathcal{N}(0, 0.02^2)$ when comparing to $W_{up} = 0$ for LoRA. We argue the main reason is: $W_{up}$ ~ $\\mathcal{N}(0, 0.02^2)$ can't guarantee that the output from the adapter is strictly equal to zero, which further supports our starting point hypothesis.\n\n> Q.3. In practice, the actual output may significantly deviate from the desired zero output.\n\nThis is a very good observation. We conduct two experiments to check it.\n\n(1) We measure the output from $\\mathcal{G}$ in Figure 4 (a) on the development set of RTE. For most samples, it is very close to 0. We argue the main reason is both $W_{down}$ and $W_{up} \\sim \\mathcal{N}(0, 0.02^2)$. The multiplication of them, $W_{up}GeLU(W_{down}h)$ makes the output very small. \n\n(2) We initialize $W_{up} = 0$ as LoRA to strictly make this output equal to 0. The result is very similar to our answer to your Q.2. We obtain a slightly better score, about +0.2. We will add this new result to our updated version.\n\n> Q.4.Limited reduction of peak memory from LoRA in Figure 6.\n\nSorry for the not-well self-explanatory captions and the y label. As our statement in Line 295, the memory shown in Figure 6 is the activation memory instead of the peak memory. Since LoRA still caches most of the activation memory as full fine-tuning, their activation memory is very similar.\n\nRegarding the peak memory, we list some related numbers of BERT$_{base}$ from Table 2 here:\n\n| Method | Peak Memory (GB) | Activation Memory (GB) |\n| -------- | -------- | -------- |\n| Full FT     | 16.67  | 14.98 |\n| LoRA | 13.45 | 13.02 |\n\nSo the memory difference for the optimizer state between Full FT and LoRA is (16.67 - 14.98) - (13.45 - 13.02) = 1.26GB. It is about three times of the model's size of 0.45GB. Notably, we make an approximation here by assuming the memory footprint for LoRA's optimizer state is 0, since only 0.27% of parameters are trainable.\n\nThe claim, the memory for the optimizer state is generally 3 times of the model's, holds for other LLMs if you train LLM with a second-order optimizer, like Adam, because one needs to save the first, second momentum, and also the calculated gradient. In addition, this memory for the optimizer state is invariant to the batch size. It's only related to the size of trainable parameters. \n\n> Q.5. Apply MEFT to other domains.\n\nLimited computation resource is the main reason for our limited application in the NLP area. Thanks to your advice, we transfer MEFT$_1$ to a CV task and show the result in the $4^{th}$ point of the general response.\n\n> Q.6. In Table 2 the memory usage of Y-Tuning is missing, why is that?\n\nThe main reason for our exclusion of the Y-Tuning memory usage is that the original paper [7] doesn't show the memory usage for BART and also doesn't release their code. This offers us some difficulties to reproduce their results. But the memory usage for Y-Tuning should be quite low since it uses the pre-trained model as a feature extractor, and there is not any gradient calculation on the backbone model. \n\nFor your reference, the peak memory usage for RoBERTa$_{large}$ is reported in [7]. It is 18.1% of full fine-tuning (Table 3 in [7]). The related results are:\n\n| Method | #Params | Peak Memory (GB) | Avg. | \n| ------ | ---------------- | --- | ---  | \n| Full FT| 100% | 11.47 | **88.4** | \n| LoRA | **0.23%** | 6.11 | 88.1 | \n| $\\mathcal{Y}$-Tuning | 4.57% | **2.08** | 82.1 | \n| MEFT$_1$ | **0.23%** | 3.63 | **88.4** | \n\nNotably, STS-B task is excluded for this table, because it is not evaluated in [7]. We have included this table in our updated draft.\n\n> Limitation: Unstable training.\n\nAbout the unstable training, we want to make a clarification here. The unstable training with FP16 causes slight performance degradation when compared to training with FP32, but it doesn't cause failed training and is still comparable to other baselines.\n\n[7] Liu, Y., An, C., & Qiu, X. (2022). $\\mathcal{Y}$-Tuning: An Efficient Tuning Paradigm for Large-Scale Pre-Trained Models via Label Representation Learning. \n\n\n\n"}]

Author rebuttal:
We would like to thank all reviewers for your time, effort, and thorough reviews! Regarding typos, presentation improvement, and missing baselines, we have incorporated suggestions into our updated draft.

Here we offer new results for some common concerns:

> 1. Compare MEFT to gradient checkpointing.

In our previous submitted materials (Line 728 in Appendix A), we only theoretically stated that the activation memory for reversible network and gradient checkpointing is $\mathcal{O}(1)$ and $\mathcal{O}(N)$, respectively. In addition, in Line 75, we also only broadly stated that the training time of MEFT is about double the training time of PEFT. Both are not well-explored and may not be intuitive enough. Here we offer some experimental results for these two claims.

In Figure 9 of the uploaded PDF, we compare activation memory and throughput among MEFT$_1$, LoRA with gradient checkpointing and Full FT with gradient checkpointing. The throughput for all three methods is at the same level, maximum 12% difference between LoRA and MEFT$_1$ when the sequence length is 128 and the batch size is 32. With an increasing sequence length, the gap becomes narrower to 7.5%. Notably, the throughput for LoRA without gradient checkpointing is 52.7 samples/second. With gradient checkpointing, it is 36.1 samples/second, 69% of the original throughput. For MEFT$_1$ with the same setting, it is 33.5 samples/second, 64% of LoRA's throughput without gradient checkpointing. **In sum, MEFT's throughput is at the same level as LoRA's with gradient checkpointing, and about 64% of LoRA's without gradient checkpointing**, which is better than our previously reported safe number, i.e. 50%.
In addition, **MEFT's activation memory is always the lower bound among these three methods**. And the gap between LoRA with gradient checkpointing and MEFT becomes larger with an increasing sequence length and batch size.

> 2. Compare MEFT to quantization methods.

There are mainly three different quantization methods:

(1) Post-training quantization [1]: Quantize a trained model to a lower bit after pre-training/fine-tuning; 

(2) Lower-bit optimizer [2]: Similar to the idea of mixed precision training, it stores the optimizer state with lower precision and dequantizes it only for the optimization; 

(3) Lower-bit frozen LLM with LoRA, i.e. QLoRA [3] (released one week after the submission deadline): QLoRA applies 4-bit quantization to compress the LLM. During fine-tuning, QLoRA backpropagates gradients through the frozen 4-bit quantized LLM into the Low-Rank Adapters. Notably, the computation data type for QLoRA is BF16. It dequantizes weights to the computation data type to perform the forward and backward passes. 

We summarize the main benefits of these three methods here:

| Save Memory from | Model's Parameter | Optimizer State | Activation |
| -------- | -------- | -------- | -----|
| post-training quantization [1] | Yes     | No     | No |
| 8-bit Adam [2] | No | Yes | No |
| QLoRA [3] | Yes | Yes | No |

Note: QLoRA also introduces paged optimizers to reduce the optimizer states, which is similar to DeepSpeed ZeRO stage 1 where the optimizer state is partitioned.

To some extent, all these three methods are orthogonal to our method and can be combined with MEFT:

(1) Post-training quantization is mainly for reference and it can be applied to any trained models.

(2) 8-bit Adam can also be applied to any models trained based on gradient.

(3) QLoRA is a combination of (1) and (2).

For QLoRA, we conduct some experiments on BERT$_{base}$ with the default setting as Figure 9:

| Method   | Activation Memory (GB) | Samples / second |
| -------- | -------- |   ----- |
| LoRA + gradient checkpointing  | 2.62 | 36.1 | 
| QLoRA + gradient checkpointing | 2.97 | 8.7 |
| MEFT$_1$ |  2.33 | 33.5 |

**METF$_1$ saves the most activation memory while having a similar throughput as LoRA with gradient checkpointing.** The reason for the larger activation memory of QLoRA than LoRA is that it has an additional dequanzation step. Because of this, its throughput is the smallest.

> 3. Combine MEFT with ZeRO [4].

ZeRO [4] saves memory by partitioning the model's parameters and optimizer state among GPUs or between GPU and CPU. This method is orthogonal to our method. We conduct some experiments on OPT1.3B by combining our method with DeepSpeed ZeRO stage 3 that offloading model's parameters and the optimizer state to CPUs:

| Method   | MEFT$_1$ | MEFT$_1$ + ZeRO (offloading to CPU) |
| -------- | -------- | -------- |
| Peak / Activation Memory (GB)  | 28.2 / 8.2 | 6.4 / 6.4 

ZeRO significantly reduces the memory footprint for model's parameters, therefore reducing MEFT's peak memory from 28.2GB to 6.4GB.

> 4. Result on CV task.

By simply borrowing the default training settings from AdaptFormer [5], we obtain the following results on the SVHN task. More CV tasks are on our TODO list. Due to limited computation resources, we only obtain the result on SVHN for now.

| Method | Acc@1 | Peak memory (GB) |
| --- | --- | ---|
| Full FT | 97.67 |  - |
| AdaptFormer | 96.89 | 36 |
| MEFT$_1$ | 96.74 | 9 |

**Similar to NLP's results, MEFT achieves comparable performance as AdaptFormer while saving a large amount of memory footprint.**

[1] Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. 

[2] Dettmers, T., Lewis, M., Shleifer, S., & Zettlemoyer, L. (2021). 8-bit optimizers via block-wise quantization. 

[3] Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms.

[4] Ren, J., Rajbhandari, S., Aminabadi, R.Y., Ruwase, O., Yang, S., Zhang, M., Li, D. and He, Y., 2021, July. ZeRO-Offload: Democratizing Billion-Scale Model Training. 

[5] Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang, J., & Luo, P. (2022). Adaptformer: Adapting vision transformers for scalable visual recognition. 

Meta-review: The paper proposes a method for memory-efficient fine-tuning and recent parameter-efficient fine-tuning (PEFT) still require caching the intermediate activations for the gradient calculation. This topic is of practical importance. All reviewers agreed that the proposed method is novel, borrowing ideas from RevNet and applying them to Transformers. The paper also provides insights regarding the importance of PEFT initialisation. All reviewers agreed that the experiments and results were solid. In addition, the authors provided a very detailed rebuttal. Incorporating the additional results and clarifications in the paper will strengthen it and I would encourage the authors to do so.
Decision: Accept (poster)