SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As shown in Figure 4d, we can also design  $\mathcal{F}$  as the pre-trained attention block with an adapter and  $\mathcal{G}$  as the pre-trained MLP block with an adapter. Also starting with the first layer, we obtain  $h_1^1 = \lambda h_0^1 +\mathcal{F}_1(h_0^2) = \lambda h_0 + \mathcal{F}_1(h_0),h_1^2 = \beta h_0^2 +\mathcal{G}_1(h_1^1) = \beta h_0 + \mathcal{G}_1(h_1^1)$ .
$\lambda \rightarrow 0$  is required, so  $h_1^1$  approximates the original output from the pre-trained attention block, and can be fed to  $\mathcal{G}_1$  to preserve the starting point.  $\beta \rightarrow 0$  is also required, so  $h_1^2 \approx h_1$ , and can be fed to  $\mathcal{F}_2$  in the next reversible layer. By default, we set  $\lambda = \beta \rightarrow 0$ . For  $\mathrm{MEFT}_3$ , one doesn't need to switch the order of  $h_1^1$  and  $h_1^2$  before feeding to the next reversible layer. For each layer,  $h_n^1$  is close to the original output from the attention block of the corresponding PLM layer, and  $h_n^2 \approx h_n$ .
Compared to the vanilla RevNet [41] where  $\lambda = \beta = 1$ , we meticulously assign different values to  $\lambda$  and  $\beta$  to preserve the starting point from a PLM, and switch the order of the outputs before feeding to the next layer (if necessary) to preserve the representation continuity. We summarize the settings for all three MEFT methods in Table 1.
Datasets and evaluation. We evaluate MEFTs on eight sequence representation tasks and five sequence-to-sequence tasks. All sequence representation tasks are from the GLUE benchmark [25].
The presentation continuity and the starting point hypothesis emphasize two aspects. The presentation continuity, for example, shows that one can't feed  $h_0$  to the third pre-trained layer, focusing on the input. The starting point hypothesis shows that the output from a modified pre-trained layer should be close to the output from the original pre-trained layer, focusing on the output. However, they are also very related, since the output from the current layer is the input to the next layer.
Read Appendix §C for a step-by-step tutorial if you still feel confused.
Figure 5: MEFTs with various scaling factors on  $\mathrm{BERT}_{\mathrm{base}}$  over RTE, MRPC, STS-B and CoLA. Dashed and solid lines denote MEFTs with vanilla and reversible gradients, respectively.
Figure 6: The trade-off between the performance and activation memory with  $\mathrm{MEFT}_1$  on RoBERTalarge over RTE, MRPC, STS-B and CoLA. The line annotated by 'freeze: true' means the shallower PLM layers are frozen without any adaptation, while the line annotated by 'freeze: false' means the top MEFT layers with vanilla gradient, as shown in Figure 7.
The sequence-to-sequence tasks are question-answering benchmarks, including OpenBookQA [44], PIQA [45], ARC (easy and challenge) [46] and SciQ [47]. We show the statistics of these datasets in Table 8 in Appendix. For the GLUE benchmark, we report accuracy on MNLI, QQP, QNLI, SST-2, MRPC and RTE, Pearson correlation coefficient on STS-B (if not specially mentioning) and Matthews correlation coefficient [48] on CoLA. We report accuracy on all question-answering tasks. In addition, we report all results on the development sets as our baselines.
Models. We use the encoder-only models (BERTbase [1], RoBERTa large [2] and BARTlarge encoder [26]) as the underlying models for all GLUE tasks, and the decoder-only models (OPT1.3B and OPT6.7B [9]) for question-answering tasks. (See Table 9 in Appendix for model details.)
Baselines. The most important baseline is full fine-tuning (Full FT) that updates all parameters. Houlsby Adapter  $(\mathbf{Adapter}^{\mathbf{H}})$  [14], Pfeiffer Adapter  $(\mathbf{Adapter}^{\mathbf{P}})$  [16], Prefix-Tuning [15] and LoRA [17] are chosen as PEFT baselines. In addition, two unified PEFT methods, MAM [18] and AutoPEFT [49], that combine multiple PEFT methods are also chosen as PEFT baselines. Lastly, two feature-based tuning methods,  $\mathcal{V}$ -Tuning [20] and LST[21], that aim to reduce training memory serve as memory-efficient baselines. We report the baseline results from the original papers if possible.
Implementation. For computational efficiency, we set  $\beta = 1$  for  $\mathrm{MEFT}_1$ ,  $\lambda = 1$  for  $\mathrm{MEFT}_2$ , and only tune the factors that are required  $\rightarrow 0$  (see Table 1). After obtaining the optimal value, i.e. 0.1, we use this value for all three MEFT methods, tasks and backbones. On the GLUE benchmark, we sweep learning rates in  $\{3,4,5\} \cdot 10^{-4}$ , batch sizes in  $\{16,32\}$  and the number of epochs in  $\{10,20\}$  for the tasks with  $>10k$  training samples. For the low-resource tasks with  $<10k$  training samples, we sweep learning rates in  $\{5,6,7,8\} \cdot 10^{-4}$ , batch sizes in  $\{16,32\}$  and the number of epochs in  $\{20,40\}$ . These grid search spaces are inspired by our baselines, especially by LoRA [17]. We use the default Adam [28] setting with a warmup ratio of  $6\%$ . If the model's performance on the development set is not improved over 5 epochs, we stop the training. We run the same task as a method in the above-mentioned grid search space three times with different random seeds, choose the best result from each run, and report the mean and standard deviation of these best results. For all question-answering tasks, we sweep learning rates in  $\{1,3,5,7\} \cdot 10^{-4}$ , batch sizes in  $\{8,16,32\}$  and the number of epochs in  $\{3,5,10\}$ , and keep other settings the same, which is inspired by [50]. The sequence length for all tasks is set to 512, 128, 128 and 128 for  $\mathrm{BERT}_{\mathrm{base}}$ , RoBERTa $_{\mathrm{large}}$ , BART $_{\mathrm{large}}$  and OPT as our baselines, respectively. We run all experiments on the Transformers framework [34] on a single NVIDIA RTX A6000 GPU with 48GB memory. Overall, a single run of any task could be finished within 8 hours, and most tasks could be finished in an hour. Fine-tuning settings are summarized in Table 7.
4.2 Results and discussions
Importance of MEFT's initialization. In the beginning, we further test the starting point hypothesis on our MEFTs by adjusting the scaling factors,  $\lambda$  and  $\beta$ . As depicted by the dashed lines in Figure 5, the degradation in performance is evident when the scaling factors deviate from their desired value of
0 (as indicated in Table 1). However, when they are small enough (0.05 or 0.1), the results are even better than full fine-tuning. For most MEFT methods  $\mathrm{(MEFT_1}$  and  $\mathrm{MEFT}_3$ ), the optimal value for the scaling factors is 0.1. So we use this value for all MEFT methods in the following experiments.
MEFTs with vanilla gradient are strong PEFT methods. Though MEFTs have reversible architectures, we can still treat them as irreversible models and cache the intermediate activations during fine-tuning. In this way, they are simply PEFT methods. In Table 2, all MEFT methods, utilizing the vanilla gradient, consistently outperform both full fine-tuning and other baseline approaches by a significant margin. For example,  $\mathrm{MEFT}_3$  outperforms Full FT by  $1\%$  and the best PEFT baseline (AutoPEFT) by  $0.7\%$  on  $\mathrm{BERT}_{\mathrm{base}}$ .  $\mathrm{MEFT}_1$  outperforms Full FT by  $0.7\%$  on  $\mathrm{RoBERTa}_{\mathrm{large}}$ .
Performance gap of MEFTs between vanilla and reversible gradients. In Figure 5, the results of MEFTs with reversible gradient (solid line) are often lower than the ones with vanilla gradient (dashed line). Recapping the discussion in §2.3, smaller scaling factors  $(< 1)$  and residual connections in  $\mathcal{F}$  and  $\mathcal{G}$  can cause a larger reconstruction error because of numerical stability. When modifying a PLM, we can't remove the residual connections from it and have to set the scaling factors  $\rightarrow 0$  due to the starting point hypothesis, which we believe is the main reason for the performance drop. Our claim is further supported by  $\mathrm{MEFT}_3$  which has the most evident drop among all MEFTs. Compared to  $\mathrm{MEFT}_1$  and  $\mathrm{MEFT}_2$  that only have a residual connection in either  $\mathcal{F}$  or  $\mathcal{G}$ , both  $\mathcal{F}$  and  $\mathcal{G}$  of  $\mathrm{MEFT}_3$  have residual connections. In addition, we have to set both  $\lambda$  and  $\beta$  close to 0 for  $\mathrm{MEFT}_3$ , which also causes a bigger reconstruction error than only setting one scaling factor (see Figure 3b middle). Since  $\mathrm{MEFT}_3$  with reversible gradient performs the worst among all MEFTs, we only run it on  $\mathrm{BERT}_{\mathrm{base}}$
due to limited resources. Expectedly,  $\mathrm{MEFT}_1$  trained in FP32 outperforms it trained in FP16 on both RoBERTaLarge and BARTlarge (see Table 2), because FP16 causes more instability.
Reversible MEFTs on deep model. Because of the starting point hypothesis, the residual connection from PLMs remains and the scaling factors are set closely to 0. With an increasing number of layers, the training instability is expected to become more severe (see Figure 3b left). As shown in Figure 6, when all RoBERTa layers are reversible (the number of reversible layers as 24), the score drops dramatically. To address this issue, we propose three settings in Figure 7: (1) Cache the activations for top layers (vanilla gradient) and apply reversible shallow layers (reversible gradient). (2) Freeze some shallow PLM layers, i.e. treating the shallow layers as a feature extractor. (3) Combine the above two settings. Notably, we have to put the reversible layers under the vanilla layers due to numerical stability. If we reverse the order, the reconstruction error is transferred to the vanilla layers.
We only explore the first two settings on RoBERTa and will discuss the third setting in the following, since RoBERTa large doesn't contain many layers. In Figure 6, when we apply the first setting (freeze: false) to RoBERTa large, the average score becomes better when the number of reversible layers decreases, outperforms full fine-tuning when it's  $\leq 16$ . However, the activation memory also increases with an increasing number of vanilla layers, since the vanilla layers require caching the activations. By default, we set the number of reversible layers as 16 for RoBERTa large in Table 2. For the second setting (freeze: true), the results are always worse than full fine-tuning. However, its activation memory stays the same since all trainable layers are reversible.
Figure 7: Three settings for deep models.
MEFTs are parameter and memory-efficient with a strong performance.
Let's go back to Table 2. Though there is a gap in MEFTs between vanilla
and reversible gradients, reversible MEFTs still achieve strong results compared to previous baselines. On  $\mathrm{BERT}_{\mathrm{base}}$ , reversible  $\mathrm{MEFT}_1$  and  $\mathrm{MEFT}_2$  obtain the same average score as Full FT, slightly worse than the best PEFT method, AutoPEFT (83.2 vs. 83.5). However, reversible MEFTs only requires about  $21\%$  and  $24\%$  activation memory of Full FT and PEFTs. On RoBERTa $_{\text{large}}$ , reversible  $\mathrm{MEFT}_1$  (FP32) achieves the same score as Full FT and outperforms all PEFT methods, while only requiring  $37\%$  and  $48\%$  activation memory of Full FT and PEFTs.
Due to limited computing resources, we only conduct experiments on the best MEFT method,  $\mathrm{MEFT}_1$ , on  $\mathrm{BART}_{\text{large}}$  when compared to other memory-efficient methods. In addition, we don't use our own grid search space on  $\mathrm{BART}_{\text{large}}$ . Instead, we apply the same grid search space as LST, setting the learning rate in  $\{3 \cdot 10^{-4}, 1 \cdot 10^{-3}, 3 \cdot 10^{-3}\}$ , the batch size as 100 and the number of epochs as 20. In this way, we want to validate the robustness of MEFT. Similarly,  $\mathrm{MEFT}_1$  outperforms the memory-efficient baselines by a large margin while only requiring  $29\%$  LST's activation memory. In addition, LST requires knowledge distillation to initialize the added layers and is not stable [21].<sup>8</sup>
MEFT trained in FP32 vs. in FP16, and the time-memory tradeoff. Though reversible MEFTs trained in FP32 outperform the ones trained in FP16, there are still some notable discussions about them: (1) The memory footprint required by reversible MEFTs trained in FP32 and FP16 is the same. In Table 2,  $\mathrm{MEFT}_1$  and  $\mathrm{MEFT}_1$  (FP32) have the same activation memory on  $\mathrm{BART}_{\text{large}}$ , because the recomputed activations in back-propagation are always in FP32 due to the mixed precision training [51]. I.e. PyTorch [52] only allows FP32 in back-propagation; (2) FP16 still benefits the training of large PLMs. In Table 2, the peak and activation memory difference is about the backbone size in FP32 for PEFTs and MEFTs. If one could reduce the backbone size by loading in FP16, we can further reduce the peak memory; (3) Training in FP16 is faster than the training in FP32 (about 1:1.2) due to the forward pass. In addition, since reversible MEFTs recompute the activations, they require more training time, about twice the training time for MEFTs with the vanilla gradient.
Results on larger and deeper models. Here we explore a more realistic setting (the third setting in Figure 7) on larger and deeper models,  $\mathrm{OPT}_{1.3\mathrm{B}}$  and  $\mathrm{OPT}_{6.7\mathrm{B}}$ , in Table 3. On  $\mathrm{OPT}_{1.3\mathrm{B}}$  with 24 layers, we set the number of frozen, reversible and vanilla layers as 8. On  $\mathrm{OPT}_{6.7\mathrm{B}}$  with 32 layers, we use 8 reversible and vanilla layers, same as  $\mathrm{OPT}_{1.3\mathrm{B}}$ . For a fair comparison, we freeze the first 8 PLM layers and modify the rest 16 layers with LoRA.  $\mathrm{MEFT}_1$  is comparable to LoRA, while only requiring LoRA's  $65\%$  activation memory. Though slightly worse than Full FT  $(-0.3\%)$ ,  $\mathrm{MEFT}_1$  's
8The comparison of memory footprint to  $\mathcal{V}$  -Tuning is in Table 10.
Table 3: Results on question-answering tasks.  $r = 64$  for both MEFT $_1$  and LoRA. All methods are trained in FP16. Due to limited computing resources, we only conduct one random run with these methods. A batch size of 32 and a sequence length of 128 are used to measure the memory footprint and training time. The training time is for one epoch on the OpenBookQA task. Check Appendix §D for the implementation detail.
Model	Method	#Param. (%)	Memory (GB)	Time (s)	OpenBookQA	PIQA	ARC-E	ARC-C	SciQ	Avg.
Peak	Activation
OPT1.3B	Full FT [50]	100	28.31	8.23	128.0	31.4	75.2	61.3	27.7	92.5	57.6
LoRA	0.64	11.42	6.27	36.6	29.9	74.9	60.1	28.7	93.3	57.4
MEFT1	0.64	9.20	4.05	45.2	34.0	73.1	57.1	28.8	93.1	57.3
OPT6.7B	ZeroShot	-	-	-	-	27.6	76.2	65.6	30.6	90.1	58.0
MEFT1	0.25	33.67	8.01	200.4	37.0	77.4	65.7	34.1	94.4	61.7
activation memory is only half of the one for Full FT. When using the same activation memory as Full FT by running on  $\mathrm{OPT}_{6.7\mathrm{B}}$ ,  $\mathrm{MEFT}_1$  outperforms Full FT by a large margin.
Transfer to image classification task. Though we only focused on NLP tasks, MEFT could be transferred to other tasks, even other architectures. We leave the transfer of MEFT to other architectures for future work, and here apply MEFT to ViT [53] for an image classification task, i.e. SVHN [27]. We follow the main training recipe from AdaptFormer [54], except for changing the optimizer from SGD to AdamW, setting the maximum gradient norm as
Table 4: Results on image classification.
Method	Acc@1	Peak Memory (GB)
Full FT [27]	97.67	-
AdaptFormer [27]	96.89	36
MEFT1	96.74	9
0.3. For  $\mathrm{MEFT}_1$  's hyper-parameters, we set  $r = 64$  and  $\lambda = 0.3$  (smaller  $\lambda$  is not stable for training). Similar to the NLP's results,  $\mathrm{MEFT}_1$  achieves comparable accuracy as AdaptFormer while saving a large amount of memory footprint in Table 4.
For more results about comparing MEFT to gradient checkpointing, comparing MEFT to quantization methods, and combining MEFT with other memory-efficient methods, please go to Appendix §E. In addition, due to the page limit, we put the detailed related works in Appendix §A, and discuss the limitation of our work in Appendix §B.
In this paper, we propose three memory-efficient fine-tuning methods (MEFTs), that fine-tune PLM in a parameter-efficient and memory-efficient way without the requirement of additional pre-training and match the performance of full fine-tuning. MEFTs modify the PLM architecture with adapters and make it reversible, by following the starting point hypothesis that is essential for PEFTs. So MEFTs don't require caching the intermediate activations during training and significantly reduce the memory footprint occupied by activations. When applying MEFTs to various models, BERT, RoBERTa and BART, on the GLUE benchmark, MEFTs achieve a similar score as full fine-tuning and other strong baselines, while saving up to  $84\%$  activation memory. A similar story is also observed when applying MEFT to larger and deeper models, OPT, on five question-answering tasks. MEFT achieves a comparable score as full fine-tuning and only consumes its  $50\%$  activation memory. However, because of the recomputation of activations, MEFTs require slightly more training time than other PEFT methods and offer a slightly lower score when trained in FP16 instead of FP32. In the future, we are interested in applying MEFT to other areas, like computer vision and automatic speech recognition, and to other bigger backbones for more sequence-to-sequence tasks.
We thank all reviewers for their great feedback. We also thank our colleague Yan Meng for her helpful review of our draft. This research was funded in part by the Netherlands Organization for Scientific Research (NWO) under project number VI.C.192.080.
[75] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7947-7969. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.emnlp-main.627. URL https://doi.org/10.18653/v1/2021.emnlp-main.627.
[76] Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via blockwise quantization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https://openreview.net/forum?id=shpkpVXzo3h.
[77] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305.14314, 2023. doi: 10.48550/arXiv.2305.14314. URL https://doi.org/10.48550/arXiv.2305.14314.
[78] Jeff Rasley, Samyam Rajbhandari, Olatunjri Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM, 2020. doi: 10.1145/3394486.3406703. URL https://doi.org/10.1145/3394486.3406703.
A Related works
Pre-trained language models. PLMs, which are trained on extensive datasets for a common task such as predicting masked words [1, 2, 3, 26, 55, 56, 57, 58] or the next word [59, 60] in a sentence, play a vital role in facilitating knowledge transfer to downstream tasks. They have demonstrated remarkable achievements across various applications, consistently delivering state-of-the-art outcomes. Furthermore, scaling up PLMs has proven to yield predictable enhancements in performance for these downstream tasks [61, 62]. Consequently, the size of released PLMs has progressively grown, reaching an unprecedented scale of 100 billion parameters [9, 10, 11, 60, 63, 64]. Such large-scale PLMs unveil extraordinary capabilities, enabling zero-shot or in-context learning [59, 60] for a broad spectrum of tasks. Nevertheless, transfer learning remains a prevalent approach for effectively deploying these models in new task scenarios [29, 65, 66], which post unparalleled requirements on the computing resources.
Parameter-efficient fine-tuning. With the advent of large-scale PLMs, a new method that aims to reduce storage requirements, PEFT, has been proposed [14, 15, 19]. PEFT adds and trains a small number of parameters while matching the performance of full fine-tuning. There are various ways to add new parameters. For example, Houlsby et al. [14] and Pfeiffer et al. [16] insert small bottleneck modules (adapters) to the PLM. LoRA [17] injects rank decomposition matrices into the pre-trained weights. HiWi [13] inserts the pre-trained parameters to a low-rank adapter.  $(\mathrm{IA})^3$  [29] scales the pre-trained weight with a trainable vector. Prompt-based methods [15, 19] append a sequence of trainable vectors to the word embeddings or attention components. Recently, some unified methods, which combine multiple PEFT methods in a heuristic way [18] or with the technique of neural architecture search [49, 67, 68], have also been proposed. Though PEFTs save the storage by a large margin compared to full fine-tuning, they still require a similar memory footprint during training as full fine-tuning [20, 21] because of the activation memory.
Memory-efficient training. Memory-efficient training aims to reduce the memory footprint during the training process. Reversible neural networks [40, 41, 42] reduce the activation memory by recomputing the activations with the outputs during back-propagation. Gradient checkpointing [69] trade computation for memory by dropping some intermediate activations and recovering them from an extra forward pass. The activation memory is  $\mathcal{O}(1)$  and  $\mathcal{O}(\sqrt{N})$  for reversible neural networks and gradient checkpointing, respectively. MEFT is the first method that is proposed to modify a PLM to its reversible variant. When applying MEFT on a deeper model, one can use gradient checkpointing to further reduce the activation memory for the layers with vanilla gradient.
Network compressions, like pruning [70, 71] and knowledge distillation [22, 23, 72], save the memory footprint for both training and inference. They compress a PLM to a smaller model by either deleting unimportant parameters or distilling knowledge from the PLM to the smaller model. Treating a PLM as a feature extractor and avoiding its gradient calculation is also an effective way to reduce the activation memory [20, 21]. However, these methods normally require extra pre-training before fine-tuning, or achieve a lower performance compared to full fine-tuning when using the same PLM.
We acknowledge the main limitation of this work is that we only evaluate our proposed methods on a limited amount of tasks and don't conduct experiments on the encoder-decoder models. The main reason for the limited amount of tasks is that our computing resources are constrained. In addition, the major criterion for our selection of the underlying models is that we could find many strong baselines on them without reproduction. BERT and RoBERTa fulfill this criterion very well on the GLUE benchmark. Regarding the encoder-decoder model, recently there is a clear trend of applying a decoder-only model on sequence-to-sequence tasks. Therefore, we apply OPT in this paper and plan to include LLAMA [11] for the instruction-finetuning data in the future.
Another limitation of MEFT is its lower score when trained in FP16 and on a deeper model. We have discussed this problem in §4.2. In sum, more reconstruction error is introduced by FP16 due to its numerical range and by a deeper model because of the error accumulation. Fortunately, the results are still comparable to the PEFT baselines when trained in FP16. Even trained in FP32, the activation memory footprints don't increase compared to FP16. One only needs to spend more training time in FP32 when using the same batch size as in FP16 (about 20% more training time). However, since