Venue: NAACL 2025

# Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient

Fine-Tuning of Language Models

Hyegang Son\({}^{1}\), Yonglak Son\({}^{*}\)\({}^{1}\), Changhoon Kim\({}^{++}\)\({}^{2,3}\), and Young Geun Kim\({}^{+}\)\({}^{1}\)

\({}^{1}\)Korea University, {hyegang_son, yonglak_son, younggeun_kim}@korea.ac.kr

\({}^{2}\)Arizona State University, kch@asu.edu

\({}^{3}\)Soongsil University, changhooon.kim@gmail.com

Equal contribution.Work completed as part of Ph.D. research at ASU.Corresponding authors.

###### Abstract

Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.

## 1 Introduction

Large-scale pre-trained language models (PLMs) have manifested superior performance in various tasks Kenton and Toutanova (2019); Liu et al. (2019); Radford et al. (2019); Yang et al. (2019). However, training PLMs from the scratch is time-consuming and resource-intensive. Common practice has been hence to fine-tune the large-scale pre-trained models by adapting all the parameters with the downstream tasks, i.e., full parameter fine-tuning (full-tuning).

Recently, Parameter-Efficient Fine-Tuning (PEFT), which focuses on optimizing a small fraction of parameters for downstream tasks, is receiving much attention Houlsby et al. (2019); Lester et al. (2021); Li and Liang (2021); Liu et al. (2022, 2023). Among various PEFT strategies, adapter-tuning has emerged as a prevalent method. It integrates lightweight modules, termed adapters, into each layer of PLMs and only tunes the adapters with the downstream tasks. As shown in Figure 1(a), the adapter-tuning methods Houlsby et al. (2019); Pfeiffer et al. (2021); Zaken et al. (2022); Hu et al. (2021); Zhang et al. (2022), significantly reduce the number of trainable parameters compared to the full-tuning while exhibiting better performance on a downstream task.

As adapter-tuning reduces the number of trainable parameters, it is also expected to reduce the resource (i.e., memory) usage accordingly. Unfortunately, parameter-efficiency does not always translate into resource-efficiency. As shown in Figure 1(b), although adapter-tuning significantly reduces the number of trainable parameters (by 99.37%, on average) compared to the full-tuning, the memory usage is not much reduced (only by 22.19%, on average). This is because adaptter-tuning does not reduce activation memory (i.e., intermediate values for reuse during backpropagation) which account for 76.00% of memory usage

Figure 1: Comparison between full-parameter fine-tuning, adapter-tuning and our proposed SAFE on the BERTlarge model with SQuAD dataset. SAFE significantly reduces memory usage while providing comparable accuracy to adapter-tuning.



-- it only reduces optimizer memory (e.g., gradients and momentum vectors). Considering the remarkable increase in model size compared to the modest increase in GPU memory capacity, adapter-tuning methods still face challenges in terms of memory efficiency. For example, fine-tuning of a LLaMA-65B [23] requires more than 780GB of GPU memory. As shown in Figure 1(b), enabling resource-efficient fine-tuning may enhance accessibility of fine-tuning to researchers and end-users, by reducing memory requirements below the capacity of commodity GPU memory.

According to previous work, the activation memory mostly depends on the backpropagation length [49, 17], which is determined by the number of adapters trained during the backward pass. Hence, to reduce the activation memory, it is crucial to reduce the number of training adapters. However, merely reducing the number of training adapters degrades accuracy. Here, a pivotal research problem arises:

_Can we reduce the number of training adapters without sacrificing accuracy?_

To answer the question, we analyze the impact of individual adapters on the accuracy and resource usage of training (Figure 2 in Section 3). We observe that some adapters are being trained, even after they finish contributing to the accuracy improvement, occupying memory. Thus, it is possible to stop training (i.e., freezing) such adapters early if they do not contribute to the adaptation for a downstream task, de-allocating their activation memory. We also observe that such early freezing can even lead to the regularization effect on the model [17], improving the accuracy.

In this paper, we propose **SAFE** (**S**elective **A**dapter **FrE**ezing), which adaptively freezes adapters in the early epochs of training. In each epoch, SAFE identifies adapters that contribute relatively less to the accuracy improvement by using an importance score [16]. It then freezes the adapters whose importance score is lower than a pre-defined threshold, reducing the memory usage and accelerating training time. By early freezing less important adapters, SAFE induces regularization effect on the model being trained, leading to a flatter loss surface. This is beneficial for finding an optimal point with higher generalization performance while optimizing neural network. In our evaluation, SAFE significantly reduces the average memory usage and TFLOPs by 46.89% and 51.73%, respectively, across various models and downstream tasks without compromising accuracy compared to the baseline, LoRA [17]. SAFE even improves the accuracy for some tasks, compared to LoRA, by up to 4.33% while reducing memory usage by 53.60%, by inducing the regularization effect.

In summary, our key contributions include:

* We uncover that adapters exhibit varying degrees of contribution to model adaptation and resource usage (Section 3).
* Motivated by this observation, we propose SAFE, a novel approach that enables resource-efficient fine-tuning by selectively early-freezing less important adapters (Section 4).
* Our evaluation on various downstream tasks demonstrates that SAFE not only achieves comparable or even better task performance to baselines but also significantly reduces resource usage by inducing the regularization effect on the model (Section 5).

## 2 Related Work

#### Parameter-Efficient Fine-Tuning:

To efficiently adapt large-scale PLMs to downstream tasks, many adapter-tuning methods [49, 17, 18, 19, 16] have been proposed. In general, adapter-tuning methods inject small, trainable, and task-specific adapter modules into each transformer layer of a pre-trained model. Given a pre-trained weight matrix \(W_{0}\in\mathbb{R}^{d\times k}\) and input \(x\in\mathbb{R}^{k\times 1}\), the weight update of adapter-tuning is expressed as \(W_{0}+\Delta W\). During training, \(W_{0}\) is frozen and does not receive gradient updates, while \(\Delta W\) contains trainable parameters. For \(h=W_{0}x\), The modified forward pass in adapter-tuning yields:

\[h=W_{0}x+\Delta Wx. \tag{1}\]

To further improve parameter efficiency of adapter-tuning, AdaLoRA [15] adaptively adjusts the number of trainable parameters among adapters according to their importance score -- it reduces the number of trainable parameters for less important adapters. However, the adapter-tuning methods still use a large amount of memory, as shown in Figure 1, since they do not reduce the activation memory which accounts for a large portion of memory usage.

**Pruning LLM Model Parameters:** To reduce the model memory of fine-tuning, two categories of 

pruning methods have been proposed Liang et al. (2021): structured pruning and unstructured pruning. Structured pruning methods remove grouped parameters (e.g., channels, layers) from the LLM. However, they usually degrade the accuracy. Furthermore, they have a limitation in terms of the compression ratio because of the low flexibility. LLM-Pruner Ma et al. (2023) compensates the accuracy drop coming from pruning, by employing post-training.

To overcome the limitation of structured pruning, unstructured pruning methods remove partial values of weight matrices regardless of their structures Li et al. (2022); Frantar and Alistarh (2023). However, unstructured pruning also degrades the accuracy.

**Resource Efficient Fine-Tuning:** Several works have tried to target resource efficient fine-tuning. AdapterDrop Ruckle et al. (2021), randomly excludes partial adapters from each training step. However, it cannot de-allocate the activation memory for the adapters, because of the random selections -- an adapter excluded from training in a step can be included in training in the following steps. SparseAdapterHe et al. (2022) applies unstructured pruning to the adapters. However, it also does not reduce the actual memory usage -- this is because the weight matrices pruned with zero values still need to be fully allocated in the memory. LoRAPruneZhang et al. (2023) employs structured pruning for LoRA. Unfortunately, the aforementioned methods usually have an adverse impact on the accuracy. MEFT Liao et al. (2024) applies a reversible model to PEFT. By using the reversible network, MEFT calculates activations with accumulated outputs of layers, without saving the intermediate activations reducing the activation memory. However, calculations of the activations severely degrades the training time performance.

Different from the previous works, this work freezes less important adapters in early steps of training. Since the frozen adapters can only be used for the forward pass, early freezing of less important adapters can effectively reduce the backpropagation length as well as the activation memory. Moreover, it induces regularization effect on the model, improving its accuracy.

## 3 Motivation

In this section, we present a pivotal research question for resource-efficient fine-tuning.

[background=RQ] Do all adapters contribute equally to the process of adaptation? To answer this question, we analyze the impact of adapters injected into each transformer layer on accuracy and resource efficiency. We measure the accuracy and memory usage of BERTbase model on MNLI and QNLI dataset from GLUE Wang et al. (2018), by attaching an adapter to each transformer layer one-by-one. Figure 2(a) and (b) show the measured accuracy and memory usage respectively -- the x-axis indicates the index of transformer layer that the adapter is injected into.

As shown in Figure 2(a), each adapter has different impact on the accuracy, and the importance of each adapter varies depending on the downstream task. In addition, despite uniform counts of trainable parameters, resource usage decreases for adapters closer to the output layer, as depicted in Figure 2(b). These observations point to the possibility that adapters in early layers contribute less to task adaptation, even though they require considerable resources. In other words, if we selectively deactivate less impactful adapters, it is possible to co-optimize the resource efficiency and accuracy.

To further analyze changes of the feature representations for each adapter throughout the training process, we quantify the representation similarity

Figure 2: (a) Accuracy and (b) Resource usage of adapter-tuning by injecting an adapter into each transformer layer of BERTbase model on MNLI and QNLI datasets from GLUE.

 

between adapters in each training step and those in the final model (which we obtained after the convergence of fine-tuning). We quantify the representational similarity using Centered Kernel Alignment (CKA) by referring to previous works Li et al. (2022a). Figure 3 visualizes the representational similarity measured throughout the training process for each adapter for BERTbase model on MNLI and QNLI dataset from GLUE Wang et al. (2018) -- lighter the color becomes, higher the feature representation similarity is.

As shown in Figure 3, even in the early training steps, the feature representations of several adapters are almost the same as those of the final model -- similar patterns are observed in other models and datasets. This means that those adapters are already representing the features that should be represented by the final model, and thus they may not further be adapted for the downstream task in the rest of the training steps. This is why such adapters are less contributing to the accuracy improvement of the model on the downstream tasks in Figure 2. One intuition is that lower adapters generally learn basic understanding of the input data, such as data bias and structural characteristics of the data, while adapters closer to the output build features unique to different tasks Houlsby et al. (2019). Motivated by the observation where not all adapters consistently contribute to adaptation, in the next section, we propose a selective adapter freezing method which preemptively freezes adapters that are relatively less important for each task.

## 4 Selective Adapter Freezing (SAFE)

In this section, we propose a selective adapter freezing method, SAFE. SAFE adaptively freezes less important adapters in the early training steps, in order to reduce unnecessary computation and memory usage without compromising the accuracy.

Figure 4 shows the overview of SAFE. SAFE consists of two stages: warm-up stage and freezing stage. In the warm-up stage, SAFE performs several epochs of fine-tuning while monitoring the feature representation changes (i.e., importance score) of the adapters (Section 4.1). If the important score of all adapters is not much changed for consecutive epochs, SAFE enters the freezing stage. In the freezing stage, SAFE gradually freezes adapters that contribute less to the adaptation, based on the importance score (Section 4.2). By early freezing less important adapters, SAFE induces regularization effect on model (Section 4.3), leading to better performance.

### Importance Score

In the warm-up stage1, we identify less important adapters by monitoring the feature representation changes of the adapters. To capture the feature representation changes of the adapters, SAFE uses Centered Kernel Alignment (CKA), which is a representative metric for representation similarity -- similar practice has been used in previous works Neyshabur et al. (2020); Raghu et al. (2021). It calculates CKA between the activation of a layer adapted with an adapter and that of the original layer as:

Footnote 1: We define the number of warm-up epochs as the epoch at which the importance score of all adapters change by less than 5% for consecutive epochs.

\[{\text{CKA}_{i}(X_{i},Y_{i})=\frac{\|Y^{T}_{i}X_{i}\|^{2}_{F}}{\| X^{T}_{i}X_{i}\|_{F}\|Y^{T}_{i}Y_{i}\|_{F}}}, \tag{2}\]

where \(X_{i}\) and \(Y_{i}\) are the activations of a layer that is adapted with an adapter and the original layer, respectively, \(i=\text{Index of Layer}\), and \(\|\cdot\|^{2}_{F}\) represents the square of the Frobenius norm of a matrix.

Higher CKA value indicates that the feature representation of a layer is still similar with that of the original one. To this end, SAFE calculates the importance score of an adapter as:

\[Imp(Adapter_{i})=1-{\text{CKA}_{i}(X_{i},Y_{i})} \tag{3}\]

### Adapter Freezing

In the freezing stage, SAFE gradually freezes adapters based on their importance score. At \(t_{w}\)-th epoch, SAFE compares the importance score of adapters with threshold \(\tau_{T}\). If the importance score of an adapter is lower than \(\tau_{T}\), SAFE identifies the adapter as a freezing candidate. After identifying

Figure 3: Visualization of representation similarity between the trained model and the model at different training steps during adapter-tuning of the BERTbase model on the MNLI and QNLI datasets from GLUE.

 

freezing candidates, SAFE freezes them based on a moving threshold until \(t_{f}\)-th epoch -- it increases the threshold from 0 to \(\tau_{T}\)2 between \(t_{w}\)-th and \(t_{f}\)-th epochs following a cubic schedule as [14]:

Footnote 2: We empirically determine \(\tau_{T}\) and final freezing epochs \(t_{f}\) based on extensive experiments with various models and datasets.

\[\tau_{t}=\begin{cases}0&0\leq t<t_{w},\\ \tau_{T}-\tau_{T}\left(1-\frac{t-t_{w}}{t_{f}-t_{w}}\right)^{3}&t_{w}\leq t<t_{ f},\\ \tau_{T}&\text{o.w.}\end{cases} \tag{4}\]

where \(t\) is the current epoch, \(t_{w}\) is the number of initial warm-up epochs, \(t_{f}\) is the number of final freezing epochs. The cubic schedule follows a nonlinear growth pattern, enabling rapid exploration of the search space during initial stages and gradually slowing down to reliably converge to the target point. We leverage these advantages by freezing more less-important adapters in the early stages. As the number of trainable parameters in the model decreases, SAFE gradually reduces the number of freezing adapters, reaching the threshold \(\tau_{T}\) stably.

### Regularization Effect of SAFE

By selectively freezing less critical adapters, SAFE induces a regularization effect within the model. In transformer-based PLM \(\mathcal{N}_{0}\), each of the \(l\) transformer blocks \(T_{l}\) is equipped with a distinct set of parameters \(\theta_{l}^{0}\) for \(l\in\{1,\ldots,n\}\). To reduce the computational overhead of fine-tuning all parameters \(\theta_{l}^{0}\), lightweight adapters \(\Delta\theta_{l}\) are introduced. To clarify how introducing adapters contribute to performance enhancements, Fu et al. (2023) formalize the optimization function as follows:

\[\min_{\theta}\mathcal{L}(\theta)+\|(I-M)(\theta-\theta^{0})\|^{2}, \tag{5}\]

where \(\theta=\theta^{0}+M\Delta\theta\) and \(M\in\{0,1\}^{m\times m}\), with \(m=\text{dim}(\theta)\), serves as a diagonal matrix for selective parameter adjustment. Each diagonal element \(M_{ii}\in\{0,1\}\) indicates whether the corresponding parameter of \(\Delta\theta_{i}\) is active (\(1\)) or inactive (\(0\)), with all off-diagonal elements \(M_{ij}\) set to 0. The regularization term is crucial for explaining how parameter constraints introduced by adapters can enhance model performance on downstream tasks. The \(rank(M)\) is bounded by \(m\), reflecting full capacity for parameter adaptation within each transformer block. However, such an approach can lead to excessive computation (See Figure 1). In contrast, our study explores the implications of constraining the \(rank(M)\) to a reduced upper bound from \(m\) to the number of trainable parameters following the proposed freezing algorithm in Section 4.2. For instance, in our motivational analysis in Section 3, where we limit the number of trainable parameters to one adapter per layer, the \(rank(M)\) is bounded by \(\frac{m}{l}\) by selectively activating \(\Delta W_{l}\) for \(T_{l}\). This constraint not only optimizes computational efficiency but also preserves the adaptability essential for superior performance on downstream tasks, as evidenced by our empirical results detailed in Section 5.3.

## 5 Experiments

### Experimental Setting

**Models:** We assess the fine-tuning efficacy of SAFE using state-of-the-art transformer-based models, including BERTbase, BERTlarge (Kenton and Toutanova, 2019), RoBERTalarge (Liu et al., 2019), GPT-2medium, GPT-2large (Radford et al., 2019), and LLaMA-27B (Touvron et al., 2023).

**Datasets:** The aforementioned models are evalu

Figure 4: Design overview of Selective Adapter Freezing (SAFE). At the warm-up stage, SAFE identifies important adapters by calculating importance score. At the freezing stage, SAFE gradually freezes the adapter based on their importance score with moving threshold \(\tau\) by following a cubic schedule.

 

ated across various tasks that span a broad spectrum of NLP applications, including Natural Language Understanding (NLU), Question Answering (QA), and Natural Language Generation (NLG). Initially, we utilize eight datasets from the General Language Understanding Evaluation (GLUE) Wang et al. (2018) which comprises two single-sentence classification tasks, three similarity and paraphrase tasks, and four natural language inference tasks. Furthermore, we conduct experiments on the SQuAD dataset Rajpurkar et al. (2016) with both BERT and RoBERTa model families. Decoder-only models such as GPT-\(2_{\text{large}}\) are also tested to determine if SAFE maintains its effectiveness in the E2E NLG Challenge Novikova et al. (2017). Finally, to investigate the scalability to larger models, we evaluate SAFE on the large language model (LLaMA-\(2_{\text{7B}}\)) and the WikiText-2 dataset Merity et al. (2022). Detailed dataset descriptions are available in Appendix C.3.

**Baselines:** To evaluate the effectiveness of SAFE, we benchmark against state-of-the-art PEFT method, LoRA Hu et al. (2021). We compare SAFE with five effective resource efficient fine-tuning methods, a previous work Zhang et al. (2021), AdapterDrop Ruckle et al. (2021), SparseAdapter He et al. (2022), LoRAPtune Zhang et al. (2023), and MEFT Liao et al. (2024). SAFE's performance is further compared with four other PEFT methods such as Houlsby Housby et al. (2019), Pfeiffer Pfeiffer et al. (2020), BitFit Zaken et al. (2022), and the adaptive method AdaLoRA Zhang et al. (2022) to demonstrate its versatility and applicability across different adapter-tuning frameworks (see Figure 1 and Appendix B for detailed results). Comprehensive details on the experimental setup and hyperparameters, such as training epochs and batch sizes, can be found in Appendix C.

### Main results

#### 5.2.1 Natural Language Understanding

Table 1 shows the results of different methods on GLUE tasks. Since SAFE selectively freezes 51.04% of less important adapters early throughout the training process, SAFE significantly reduces memory usage by 40.47%, from 20.35GB (LoRA) to 12.11GB, and decreases computation costs (FLOPs) by 35.15%. Even with such improvements in resource efficiency, SAFE improves the average GLUE score from 84.66 (LoRA) to 84.99 -- this is because SAFE induces a regularization effect on less-important adapters improving generalization performance of the model (see Section 5.3).

Figure 5 shows the freezing patterns of BER\(T_{\text{large}}\) fine-tuned with SAFE -- we observe similar patterns for other tasks. We find that SAFE tends to freeze adapters more in layers closer to the input layer. Such behavior aligns with empirical observations presented in Figure 2 where adapters closer to the output layer need further adaptation to the downstream tasks compared to those in earlier layers, contributing more to model performance.

Compared to a previous work Zhang et al. (2021), which attaches adapters only to the upper layers, SAFE achieves 2.24% higher GLUE score. This is because some adapters attached to lower layers keep contributing to the adaptation even in later training epochs, as shown in Figure 5 (e.g., 2nd and 5th adapters in QNLI task). Compared to AdapterDrop, SAFE provides up to 2.69% higher score (MRPC) while reducing memory usage by 49.47% (0.91% higher GLUE score and 40.47% reduced memory usage on average). This is because AdapterDrop reduces computation costs (FLOPs) by ran

\begin{table}
\begin{tabular}{l|c c c c c c c|c c c c} \hline \hline  & CoLA & SST-2 & MNLI & RTE & QQP & MRPC & QNLI & STS-B & Avg. & Memory & Computation \\  & Matthews corr Accuracy & Accuracy & Accuracy & Accuracy & F1 Score & Accuracy & Pearson corr & & (GB) & (TFLOPs) \\ \hline LoRA & 65.24 & 93.65 & 85.40 & 72.66 & 90.49 & 87.90 & 90.06 & 91.88 & 84.66 & 20.35 & 46,698 \\ + Zhang et al. & 61.78 & 91.97 & 84.61 & 73.18 & 89.20 & 86.83 & 90.00 & 87.50 & 83.13 & 11.20 & 35,415 \\ + AdapterDrop & 64.24 & 92.54 & 85.19 & 73.38 & 89.02 & 86.51 & 91.51 & 91.39 & 84.22 & 20.35 & 35,114 \\ + SparseAdapter & 65.25 & 92.66 & 85.19 & 74.10 & 90.43 & 88.50 & 91.61 & 91.99 & 84.97 & 20.35 & 46,698 \\ + LoRAPtune & 63.03 & 91.54 & 83.97 & 70.59 & 88.28 & 87.74 & 84.13 & 86.48 & 81.97 & 10.37 & 25,137 \\ + MEFT & 64.57 & 92.66 & 84.33 & 72.92 & 88.70 & 88.30 & 90.98 & 90.15 & 84.08 & 11.15 & 88,363 \\ \hline + SAFE & 65.26 & 92.78 & 85.41 & 74.10 & 89.96 & 88.84 & 91.78 & 91.80 & **84.99** & **12.11\(\mathbf{n}\)\(\Leftrightarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\downarrow\)\(\uparrow\) 

domly dropping adapters for each step, whereas SAFE selectively freezes less-important adapters preserving critical adapters and thus achieving better performance. AdapterDrop also does not lower memory usage because the memory allocated to the dropped adapters cannot be de-allocated for the next step -- adapters dropped in a step may not be dropped in the next step. Compared to SparseAdapter, SAFE reduces memory usage by 40.47% and computation cost by 35.15% while providing comparable GLUE score. This is because SparseAdapter performs pruning of redundant parameters but uses unstructured pruning with masking, which does not actually improve resource efficiency. On the other hand, although LoRAPrune is effective to reduce the memory usage, it severely degrades the accuracy as it fully eliminates less important adapter weights through structural pruning. As a result, SAFE achieves 3.68% higher GLUE score compared to LoRAPrune. MEFT reduces the memory usage at the cost of more than 2x of the FLOPs and training time. This is because MEFT applies a reversible model to LoRA, which reduces memory usage by not caching intermediate activations, but causes substantial computation overhead due to recomputation. Overall, SAFE strikes a better accuracy/memory efficiency/training-time performance trade-off compared to SOTA methods.

#### 5.2.2 Question Answering

Table 2 shows the results of SQuAD dataset. SAFE consistently outperforms baseline under all settings. Notably, SAFE reduces memory usage and computation costs by up to 79.92% and 88.41% on RoBERTalarge by freezing 91.67% of the adapters, while improving the F1 score from 93.39 (LoRA) to 94.13. This result also demonstrates that the benefits and effectiveness of SAFE are not restricted to specific model sizes, making it a valuable strategy for enhancing adapter-tuning outcomes across models of varying scales.

#### 5.2.3 Natural Language Generation

Table 3 shows that SAFE prevails on natural language generation task with GPT-2. SAFE achieves comparable performance to LoRA across all metrics while significantly reducing memory usage. This result also demonstrates that SAFE is effective not only for encoder models but also works well with decoder models.

Figure 6 compares (a) perplexity and (b) memory usage between LoRA and SAFE on the LLaMA-\({}_{\text{2FB}}\) using the WikiText-2 dataset, to examine the applicability of SAFE on large language models. As shown in Figure 6, SAFE reduces memory usage by 48.37%, from 66.35GB (LoRA) to 34.20GB, without any degradation in model qual

\begin{table}
\begin{tabular}{c c|c|c  

ity. This result implies that SAFE can significantly improve the scalability of large language models by reducing the resource usage, making it feasible to train these models on resource-constrained devices or with larger batch sizes.

### Regularization Effect

To elucidate the underlying mechanisms behind SAFE's enhancements in model performance and memory efficiency, we conduct a detailed empirical analysis. We visualize and compare the loss landscapes of the baseline and SAFE. Additionally, we quantitatively evaluate the flatness of the loss surfaces by analyzing the spectrum of Hessian eigenvalues. This methodical approach allows us to substantiate the improvements attributed to SAFE, providing insights into its effectiveness in optimizing both performance and resource utilization.

**Loss Landscape Analysis.** The flatness of a loss landscape is a recognized indicator of the generalization ability of models (Jiang et al., 2020). Specifically, flatter landscapes are indicative of enhanced robustness to parameter perturbations (Xie et al., 2021), reduced model complexity (Blier and Ollivier, 2018), and improved generalization capabilities (Cha et al., 2021, 2022; Choromanska et al., 2015; Park and Kim, 2021; Wu and Su, 2023). To examine these properties, we employ a comparative visualization of the loss landscapes for LoRA and SAFE using the BERTbase model on the QNLI and SST-2 datasets, following the methodology outlined in (Park and Kim, 2021)3. Our analysis reveals that SAFE yields a flatter loss landscape compared to LoRA. This flattening is attributed to SAFE's mechanism of controlling the norm of the weights through regularization effects (See Equation (5)), which consequently enhances resistance to weight perturbations, as depicted in Figure 7(a).

Footnote 3: This involves generating two orthogonal random vectors in a \(1\)D flattened parameter space, which are then normalized and used to perturb the parameters. The strength of these perturbations is determined by their \(x\) and \(y\) coordinates, with the origin \((0,0)\) representing the unperturbed state of the parameters, and increasing distance indicating greater perturbation intensity.

**Hessian Eigenvalue Spectrum Analysis.** To quantitatively assess the visualized loss landscape shown in Figure 7(a), we perform a detailed analysis of the top-5 Hessian eigenvalue spectrum. A pivotal finding in our analysis is the reduced magnitude of the maximum Hessian eigenvalue, which correlates with a flatter loss landscape, indicative of enhanced generalization potential. Moreover, the diminution of large Hessian eigenvalues facilitates more effective model training (Ghorbani et al., 2019). Furthermore, the suppression of the largest _negative_ Hessian eigenvalues markedly contributes to a more convex loss landscape, enhancing the stability of the training process. Figure 7(b) demonstrates that SAFE not only effectively reduces the magnitude of Hessian eigenvalues relative to LoRA but also leads to a smoother and more consistent loss landscape. This evidence highlights the advantages of SAFE in promoting a more reliable and steady training behavior for adapter-tuning.

Figure 7: (a) Loss landscape demonstrates that SAFE yields a flatter loss surface compared to the baseline, as shown in the second and third columns. (b) Hessian eigenvalue spectrum analysis shows that the magnitude of the Hessian eigenvalues for SAFE is smaller than those for the baseline, indicating a flatter local curvature and potentially better generalization properties.

 



### Resource Efficiency

We evaluate the resource efficiency of SAFE in terms of memory usage, computation amount, and training time. Table 4 shows the average resource efficiency improvement for the main results on NLU, QA and NLG tasks. Overall, SAFE reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82% compared to LoRA, respectively (on average). This means that SAFE can fine-tune twice as many downstream tasks under the same FLOPs budget and further enable on-device fine-tuning for personalization. For example, when fine-tuning a RoBERTalarge model with a question answering downstream task, SAFE reduces memory usage from 17.73GB to 3.56GB; 8GB is the usual memory size of the edge devices.

### Expanded Experimental Results

**Image Classification Task Evaluations.** In Appendix A, we conduct comprehensive evaluations of SAFE on a variety of image classification tasks. These experiments consistently demonstrate the efficacy of SAFE, confirming its robust performance across diverse vision-related applications.

**Compatibility with Advanced Adapters.** Further discussions on the integration of SAFE with various advanced adapter modules are presented in Appendix B. Our results highlight SAFE's versatility and compatibility with multiple adaptentuning frameworks Houlsby et al. (2019); Zaken et al. (2022). This adaptability ensures that SAFE's methodology remains effective, independent of specific adapter designs, thereby facilitating scalability across existing adapter-tuning methods.

## 6 Conclusion

In this paper, we propose SAFE, which selectively freezes adapters for enabling resource efficient fine-tuning of PLMs. We observe that not all adapters contribute equally to adaptation. Motivated by the observation, SAFE gradually freezes less-important adapters, which do not contribute to adaptation during the early training steps. In our evaluation on various models and datasets, SAFE significantly saves memory usage and computation and accelerating training time, with comparable (or even better) accuracy. We also demonstrate that SAFE induces regularization effect, thereby improving generalization performance and accuracy compared to the state-of-the-art PEFT methods. We believe that SAFE can enable resource-efficient fine-tuning of large-scale PLMs, and further pave the path forward to personalized fine-tuning on resource-constrained edge devices.

## 7 Limitations

We suggest the need for combination with prior research on memory-efficient training. These include low precision, micro-batching, weight sharding, and gradient checkpointing techniques. Though we have not evaluated SAFE along with such memory-efficient training methods, SAFE can be complementarily used along with the methods since SAFE can be applied independently of the training method or weight precision. In particular, since the quantization-based compression technique is quite popular and effective in terms of both compression ratio and preservation of final accuracy, favorable results are expected from combining the proposed technique with the memory-efficient training methods Han et al. (2015).

## Acknowledgments

This work was supported in part by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (RS-2023-00212711), Institute of Information & Communications Technology Planning & Evaluation(IITP)-ITRC(Information Technology Research Center) grant funded by the MSIT(IITP-2025-RS-2023-00260091), ICT Creative Consilience Program through IITP grant funded by the MSIT(IITP-2025-RS-2020-II201819), IITP grant funded by the MSIT (No. RS-2024-00398353, Development of Countermeasure Technologies for Generative AI Security Threats). We also thank the anonymous reviewers for their helpful feedback.

\begin{table}
\begin{tabular}{c c|c|c|c} \hline \hline  & & NLU & QA & NLG \\ \hline Memory Usage (GB) & LoRA & 20.35 & 12.75 & 16.97 \\ \cline{2-5} Computational Cost (TFLOPs) & + SAFE & 12.11 ± 0.47\% & 5.85 ± 0.00\% & 11.90 ± 0.39\% \\ \hline Computational Cost (TLORAT) & LoRA & 1.24 & 5.94 & 8.64 \\ \cline{2-5} (TFLOPs) & + SAFE & 0.93 ± 0.46\% & 2.33 ± 0.60\% & 6.79 ± 0.48\% \\ \hline Training Time (Normalized) & LoRA & 1 & 1 & 1 \\ \cline{2-5} (Normalized) & + SAFE & 0.90 ± 0.95\% & 0.89 ± 0.95\% & 0.80 ± 0.96\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: SAFE improves efficiency over the baseline in all aspects including memory usage, computation amount and training time. Note that we report computation costs for 1-step training.





## References

* D. Blier and Y. Ollivier (2018)The description length of deep learning models. Advances in Neural Information Processing Systems31, pp. . Cited by: SS4.1.
* L. Bossard, M. Guillaumin, and L. V. Gool (2014)Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. External Links: ISBN 978-3-540-3213-1, Link, Document Cited by: SS1, SS2.
* J. Cha, S. Chun, K. Lee, H. Cho, S. Park, Y. Lee, and S. Park (2021)Swad: domain generalization by seeking flat minima. Advances in Neural Information Processing Systems34, pp. 22405-22418. Cited by: SS1, SS2.
* J. Cha, K. Lee, S. Park, and S. Chun (2022)Domain generalization by mutual-information regularization with pre-trained models. In European Conference on Computer Vision, pp. 440-457. Cited by: SS1, SS2.
* T. Chen, B. Xu, C. Zhang, and C. Guestrin (2016)Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174. Cited by: SS1, SS2.
* Y. Chen, S. Qian, H. Tang, X. Lai, Z. Liu, S. Han, and J. Jia (2023)Longlora: efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, Cited by: SS1, SS2.
* A. Choromanska, M. Henaff, M. Mathieu, G. Ben Arous, and Y. LeCun (2015)The loss surfaces of multilayer networks. In Artificial intelligence and statistics, pp. 192-204. Cited by: SS1, SS2.
* E. Frantar and D. Alistarh (2023)SparseGPT: massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. Cited by: SS1, SS2.
* Z. Fu, H. Yang, A. Man-Cho So, W. Lam, L. Bing, and N. Collier (2023)On the effectiveness of parameter-efficient fine-tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, pp. 12799-12807. Cited by: SS1, SS2.
* Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao (2019)An investigation into neural net optimization via hessian eigenvalue density. In International Conference on Machine Learning, pp. 2232-2241. Cited by: SS1, SS2.
* S. Han, H. Mao, and W. J. Dally (2015)Deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149. Cited by: SS1, SS2.
* S. He, L. Ding, D. Dong, J. Zhang, and D. Tao (2022)Sparseadapter: an easy approach for improving the parameter-efficiency of adapters. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2184-2190. Cited by: SS1, SS2.
* X. He, C. Li, P. Zhang, J. Yang, and X. Eric Wang (2023)Parameter-efficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, pp. 817-825. Cited by: SS1, SS2.
* N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Larossilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for nlp.. In International Conference on Machine Learning, pp. 2790-2799. Cited by: SS1, SS2.
* E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2021)Lora: low-rank adaptation of large language models. In International Conference on Learning Representations, Cited by: SS1, SS2.
* Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio (2020)Fantastic generalization measures and where to find them. In International Conference on Learning Representations, Cited by: SS1, SS2.
* R. Karimi Mahabadi, J. Henderson, and S. Ruder (2021)Compacter: efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems34, pp. 1022-1035. Cited by: SS1, SS2.
* J. Devlin Ming-Wei Chang Kenton and L. Kristina Toutanova (2019)Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171-4186. Cited by: SS1, SS2.
* S. Kornblith, M. Norouzi, H. Lee, and G. Hinton (2019)Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519-3529. Cited by: SS1, SS2.
* J. Krause, M. Stark, J. Deng, and L. Fei-Fei (2013)3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561. Cited by: SS1, SS2.
* A. Krizhevsky et al. (2009)Learning multiple layers of features from tiny images. Cited by: SS1, SS2.
* Ya. Le and X. Yang (2015)Tiny imagenet visual recognition challenge. CS 231N (7), pp. 3. Cited by: SS1, SS2.
* B. Lester, R. Al-Rfou, and N. Constant (2021)The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059. Cited by: SS1, SS2.
* S. Li, G. Yuan, Y. Dai, Y. Zhang, Y. Wang, and X. Tang (2022a)Smartfrz: an efficient training framework using attention-based layer freezing. In The Eleventh International Conference on Learning Representations,

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597. Cited by: SS3.
* Y. Li, F. Luo, C. Tan, M. Wang, S. Huang, B. Li, and J. Bai (2022b)Parameter-efficient sparsity for large language models fine-tuning. arXiv preprint arXiv:2205.11005. Cited by: SS3.
* T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang (2021)Pruning and quantization for deep neural network acceleration: a survey. Neurocomputing461, pp. 370-403. External Links: ISSN 1558-1221, Link, Document Cited by: SS3.
* Baohao Liao, S. Tan, and C. Monz (2024)Make pre-trained model reversible: from parameter to memory efficient fine-tuning. Advances in Neural Information Processing Systems36. Cited by: SS3.
* H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel (2022)Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems35, pp. 1950-1965. Cited by: SS3.
* Xiao Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang (2023)Gpt understands, too. AI Open. Cited by: SS3.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: SS3.
* I. Loshchilov and F. Hutter (2018)Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: SS3.
* X. Ma, G. Fang, and X. Wang (2023)Llim-pruner: on the structural pruning of large language models. Advances in neural information processing systems36, pp. 21702-21720. Cited by: SS3.
* Stephen Merity, C. Xiong, J. Bradbury, and R. Socher (2022)Pointer sentinel mixture models. In International Conference on Learning Representations, Cited by: SS3.
* B. Neyshabur, H. Sedghi, and C. Zhang (2020)What is being transferred in transfer learning? advances in neural information processing systems33, pp. 512-523. Cited by: SS3.
* M. Nilsback and A. Zisserman (2008)Automated flower classification over a large number of classes. In 2008 Sixrd Indian conference on computer vision, graphics & image processing, pp. 722-729. Cited by: SS3.
* J. Novikova, O. Dusek, and V. Rieser (2017)The e2e dataset: new challenges for end-to-end generation. In 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 201-206. Cited by: SS3.
* N. Park and S. Kim (2021)How do vision transformers work?. In International Conference on Learning Representations, Cited by: SS3.
* Jonas Pfeiffer, A. Kamath, A. Ruckle, K. Cho, and I. Gurevych (2021)Adapterfusion: non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503. Cited by: SS3.
* J. Pfeiffer, A. Ruckle, C. Poth, A. Kamath, I. Vulic, S. Ruder, K. Cho, and I. Gurevych (2020)A adapterHub: a framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 46-54. Cited by: SS3.
* A. Radford, J. Wook Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021)Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. Cited by: SS3.
* A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. (2019)Language models are unsupervised multitask learners. OpenAI blog1 (8), pp. 9. Cited by: SS3.
* M. Raghu, T. Unterthiner, S. Kornblith, C. Zhang, and A. Dosovitskiy (2021)Do vision transformers see like convolutional neural networks?. Advances in neural information processing systems34, pp. 12116-12128. Cited by: SS3.
* P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016)Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392. Cited by: SS3.
* M. Rhu, N. Gimelshein, J. Clemons, A. Zulfiqar, and S. W Keckler (2016)vdnn: virtualized deep neural networks for scalable, memory-efficient neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), Vol. , pp. 1-13. Cited by: SS3.
* A. Ruckle, G. Geigle, M. Blockner, T. Beck, J. Pfeiffer, N. Reimers, and I. Gurevych (2021)Adapterdrop: on the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930-7946.

 

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptistie Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_, pages 353-355.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Wu and Su (2023) Lei Wu and Weijie J Su. 2023. The implicit regularization of dynamical stability in stochastic gradient descent. _arXiv preprint arXiv:2305.17490_.
* Wu et al. (2024) Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Ajji. 2024. LamininIm: A diverse herd of distilled models from large-scale instructions. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 944-964.
* Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. _arXiv preprint arXiv:1708.07747_.
* Xie et al. (2021) Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, and Masashi Sugiyama. 2021. Artificial neural variability for deep learning: On overfitting, noise memorization, and catastrophic forgetting. _Neural computation_, 33(8):2163-2192.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. _Advances in neural information processing systems_, 32.
* Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 1-9.
* Zhang et al. (2023) Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. 2023. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. _arXiv preprint arXiv:2305.18403_.
* Zhang et al. (2022) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2022. Adaptive budget allocation for parameter-efficient fine-tuning. In _The Eleventh International Conference on Learning Representations_.
* Zhang et al. (2017) Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning. In _International Conference on Learning Representations_ 

Results on Image Classification Tasks

We conduct experiments with 8 datasets including class-level transfer and task-level transfer in the image classification tasks within Computer Vision (CV) domain. These datasets include CIFAR-10, CIFAR-100 [25], Country-211 [37], Fashion MNIST [46], Food-101 [29], Oxford Flowers [26], Standford Cars [26] and Tiny ImageNet [31].

Table 5 shows that SAFE can effectively reduce memory usage while achieving comparable accuracy on eight datasets in the image classification task. For example, SAFE achieves a \(50.69\%\) memory usage reduction on ViTlarge while maintaining comparable accuracy. Additionally, SAFE remains consistently effective regardless of variations in the pre-trained model size and backbone structure.

## Appendix B Results with Various PEFT Methods

We validate the applicability of SAFE upon advanced adapter modules [14, 21, 20]. Table 6 shows that SAFE reduces memory usage by \(24.76\%\) on average while achieving a comparable GLUE score. This result demonstrates that SAFE can be applied to a variety of adapter-tuning methods to enable resource efficient fine-tuning of large language models.

\begin{table}
\begin{tabular}{l|c c c c c c c c|c} \hline \hline  & CoLA & SST-2 & MNLI & RTE & QQP & MRPC & QNLI & STS-B & Avg. \\ BERT\({}_{\text{base}}\) & Matthews corr. Accuracy & Accuracy & Accuracy & Accuracy & F1 Score & Accuracy & Pearson corr. & \\ \hline Houlsby & 62.38 & 91.17 & 83.44 & 70.50 & 90.85 & 89.52 & 90.59 & 90.75 & 83.65 \\ + SAFE & 62.83 & 93.00 & 84.15 & 74.10 & 90.93 & 89.70 & 91.03 & 91.29 & **84.63x***.9* \\ \hline BiFit & 60.92 & 91.86 & 82.41 & 71.94 & 89.20 & 88.14 & 89.80 & 90.95 & **83.15** \\ +SAFE & 61.73 & 93.00 & 81.85 & 69.78 & 89.21 & 88.85 & 89.51 & 90.75 & 83.09 28.59* \\ \hline LoRA & 64.46 & 91.63 & 82.88 & 71.22 & 90.01 & 88.39 & 90.01 & 90.86 & 83.68 \\ + SAFE & 66.80 & 90.83 & 82.03 & 71.22 & 89.74 & 88.51 & 90.65 & 90.26 & **83.76i***.9* \\ \hline \hline \end{tabular}
\end{table}
Table 6: Experimental results for various adapter-tuning methods on the GLUE benchmark. Note that blue indicates the memory usage reduction rate of SAFE compared to the baseline.

\begin{table}
\begin{tabular}{l l|c|c|c|c|c|c|c|c|c} \hline \hline  & & CIFAR-10 & CIFAR-100 & Country-211 & Fashion MNIST & Food-101 & Oxford Flowers & Stanford Cars & Tiny ImageNet & Avg. \\ \hline \multirow{2}{*}{ViTune} & LoRAA & 97.77 & 95.68 & 16.56 & 94.52 & 88.84 & 99.41 & 82.56 & 88.90 & **83.03** \\  & + SAFE & 98.66 & 95.55 & 16.29 & 93.71 & 88.78 & 99.61 & 82.05 & 89.16 & 82.98 84.98 \\ \hline \multirow{2}{*}{ViTlarge} & LoRAA & 99.09 & 96.71 & 20.44 & 94.92 & 90.32 & - & 86.97 & 92.13 & 82.94 \\  & +LoRAA & 99.13 & 97.00 & 20.44 & 94.88 & 90.69 & - & 86.83 & 92.03 & **83.00***.9*** \\ \hline \multirow{2}{*}{SWINbase} & LoRAA & 98.92 & 96.20 & 20.06 & 95.12 & 91.36 & 99.50 & 87.14 & 90.31 & 84.83 \\  & + SAFE & 98.94 & 96.31 & 20.26 & 95.12 & 91.51 & 99.71 & 86.88 & 90.14 & **84.86** 84.86 ***.9* \\ \hline \multirow{2}{*}{SWINlarge} & LoRAA & 99.07 & 97.01 & 22.19 & 95.44 & 92.68 & - & 85.06 & 92.02 & 83.35 \\  & + LoRAA & 99.17 & 96.72 & 22.66 & 95.47 & 92.69 & - & 85.13 & 92.11 & **83.42** 25.69* \\ \hline \hline \end{tabular}
\end{table}
Table 5: Experimental results on eight common computer vision tasks. SAFE significantly reduces memory usage while achieving accuracy comparable to the baseline. Note that blue indicates the memory usage reduction rate of SAFE compared to the baseline.

 

Experimental Setup

### Model

We conduct experiments using a pre-trained model deployed on HuggingFace (Wolf et al., 2019). For experiments on the NLU and QA benchmarks, we use bert-base-uncased and bert-large-uncased trained on BookCorpus, a dataset consisting of 11,038 unpublished books and English Wikipedia. We use roberta-base and roberta-large trained on 5 datasets (BookCorpus, English Wikipedia, CC-News, OpenWebText, and Stories) for the RoBERTa model. For experiments on the NLG benchmark, we use GPT-2 medium and GPT-2 large distributed by OpenAI. We also employ the LLaMA-2 7B model, an open-weight large language model released by Meta, which has been trained on a mixture of publicly available and licensed datasets, including Common Crawl, C4, GitHub, Wikipedia, Project Gutenberg, ArXiv, and Stack Exchange. We use vit-base-patch16-224-in2lk and vit-large-patch16-224-in21k distributed by Google for experiments in the ViT model. Finally, We use the swin-base-patch4-window7-224 and swim-large-patch4-window7-224 models distributed by Microsoft for experiments in the SWIN model.

### Computing Resources

Our experimental setup leverages 2 RTX4090 with 24GB memory for NLU, QA, and NLG tasks and 1 RTX 4090 for CV downstream task, excluding the LLaMA experiments.

### Dataset Statistics

We present the dataset statistics of GLUE (Wang et al., 2018), SQuAD (Rajpurkar et al., 2016), E2E NLG Challenge (Novikova et al., 2017), and WikiText-2 (Merity et al., 2022) in following table. We fine-tune models on the LaMini instruction dataset (Wu et al., 2024) for LLaMA-\(2_{\text{7B}}\) and evaluate their performance on WikiText-2 using perplexity.

## Appendix D

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & \multicolumn{6}{c}{**NLU Benchmark**} \\ \hline Dataset & \# Train & \# Valid & \# Test & \# Label & Task & Evaluation Metric \\ \hline \multicolumn{6}{c}{Single-Sentence Classification (GLUE)} \\ \hline CoLA & 8,551 & 521 & 522 & 2 Acceptability & Matthews corr \\ SST-2 & 66,349 & 1,000 & 872 & 2 Sentiment & Accuracy \\ \hline \multicolumn{6}{c}{Pairwise Text Classification (GLUE)} \\ \hline MNLI & 392,702 & 9,832 & 9,815 & 3 NLI & Accuracy \\ RTE & 2,490 & 138 & 139 & 2 NLI & Accuracy \\ QQP & 362,846 & 1,000 & 40,431 & 2 Paraphrase & Accuracy \\ MRPC & 3,668 & 204 & 204 & 2 Paraphrase & F1 score \\ QNLI & 103,743 & 1,000 & 5,463 & 2 QA/NNLI & Accuracy \\ \hline \multicolumn{6}{c}{Pairwise Text Classification (GLUE)} \\ \hline STS-B & 5,749 & 750 & 750 & 1 Similarity & Pearson corr \\ \hline \multicolumn{6}{c}{**QA Benchmark**} \\ \hline Dataset & \# Train & \# Valid & \# Test & \# Label & Task & Evaluation Metric \\ \hline SQuAD & 87,600 & 5,300 & 5,300 & 2 Question Answering F1 score \\ \hline \multicolumn{6}{c}{**NLG Benchmark**} \\ \hline Dataset & \# Train & \# Valid & \# Test & \# Label & Task & Evaluation Metric \\ \hline E2E NLG Challenge & 42,061 & 4,672 & 4,693 & Generation & BLEU, NIST, METEOR, ROUGE-L, and CIDEr WikiText-2 & 2,589,000 & 1,000 & 4,360 & Language Modeling & Perplexity (PPL) \\ \hline \hline \end{tabular}
\end{table}
Table 7: Summary of the NLU, QA, and NLG benchmarks.

 

The following table lists dataset statistics evaluated in the CV domain.

\begin{table}
\begin{tabular}{l r r r r l c} \hline \hline  & \multicolumn{5}{c}{**CV Benchmark**} \\ \hline Dataset & \# Train & \# Valid & \# Test & \# Label & Task & Evaluation Metric \\ \hline CIFAR-10 & 45,000 & 5,000 & 10,000 & 10 & Classification & Accuracy \\ CIFAR-100 & 45,000 & 5,000 & 10,000 & 100 & Classification & Accuracy \\ Fashion MNIST & 54,000 & 6,000 & 10,000 & 10 & Classification & Accuracy \\ Oxford Flowers & 6,453 & 717 & 1,020 & 102 & Classification & Accuracy \\ Food-101 & 68,220 & 7,580 & 25,300 & 102 & Classification & Accuracy \\ Country-211 & 25,920 & 2,880 & 21,100 & 211 & Classification & Accuracy \\ Stanford Cars & 7,326 & 814 & 8,040 & 196 & Classification & Accuracy \\ Tiny ImageNet & 90,000 & 10,000 & 10,000 & 200 & Classification & Accuracy \\ \hline \hline \end{tabular}
\end{table}
Table 8: Summary of CV benchmark.

 



### Hyperparameter Settings

We explore 10% of all epochs for at least 5 learning rates. Hyperparameter settings, including learning rate, are made by referring to previous works He et al. (2023); Houlsby et al. (2019); Hu et al. (2021); Zaken et al. (2022). We use the AdamW optimizer Loshchilov and Hutter (2018) and LinearLR learning rate scheduler and set weight decay to 0 in experiments. In our evaluation, we configure LoRA as follows: \(r=4\), alpha = 16, target modules = ["query", "value"], and LoRA dropout = 0.1.

Since the lower layers of the BERT model are more easily converted to fine-tuning than the upper layers, we extend the previous study Zhang et al. (2017) that froze most of the lower layers with adapter-tuning and experiment with a baseline that naively inserts adapters only into six layers. For LoRAPPune, we configure the pruning ratio to 0.5, pruning frequency to 10, and the pruning metric to 'lora,' as these settings have shown effective results in prior work Zhang et al. (2023). In the case of MEFT, for a fair comparison, we set the reversible layers to half of the total number of layers. We employ the MEFT\({}_{1}\) method, where the \(\mathcal{F}\) architecture is the layer, from the three reversible transformation methods proposed in MEFT. For the hyperparameters of MEFT, such as \(\lambda\) and \(\beta\), we follow the values proposed in previous work Liao et al. (2024) to ensure consistency in the evaluation.





\begin{table}
\begin{tabular}{l c c c} \hline \hline pre-trained model & **GPT-2 medium** & **GPT-2 large** & **LLaMA-2 7B** \\ \hline  & Training & & \\ \cline{1-3} final learning rate & 1.00E-04 & 5.00E-05 & 1.00E-04 \\ batch size & 8 & 4 & 128 (w/ micro batch size 2) \\ \# epochs & 10 & 10 & 3 \\ Seq Length & 512 & 512 & 512 \\ Label Smooth & 0.1 & 0.1 & \\ \hline  & Inference & & \\ \hline Beam Size & 10 & 10 & \\ Length Penalty & 0.8 & 0.8 & \\ no repeat ngram size & 4 & 4 & \\ \hline \hline \end{tabular}
\end{table}
Table 10: Hyperparameter settings on the NLG task.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline pre-trained model & dataset & method & final learning rate & batch size & \# epochs \\ \hline  & GLUE / CdA & LoRA, LoRa + SAFE & 6.00E-04 & 32 & 100 \\  & & BoIriR, BitFit + SAFE & 9.00E-04 & 32 & 100 \\  & & Houlsby, Houlsby + SAFE & 5.00E-04 & 32 & 100 \\  & LoRA, LoRa + SAFE & 7.00E-04 & 32 & 75 \\  & GLUE / SST-2 & BifFit, BitFit + SAFE & 7.00E-04 & 32 & 75 \\  & Houlsby, Houlsby + SAFE & 2.00E-04 & 32 & 75 \\  & LoLA, LoRa + SAFE & 9.00E-04 & 32 & 50 \\  & GLUE / MNLI & BifFit, BitFit + SAFE & 8.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 4.00E-04 & 32 & 50 \\  & LoLoRa, LoRa + SAFE & 9.00E-04 & 32 & 100 \\  & GLUE / RTE & BitFit, BitFit + SAFE & 8.00E-04 & 32 & 100 \\  & Houlsby, Houlsby + SAFE & 4.00E-04 & 32 & 100 \\  & LoRa, LoRa + SAFE & 4.00E-04 & 32 & 50 \\
**BERT-base-uncased** & GLUE / QQP & BitFit + SAFE & 6.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 4.00E-04 & 32 & 50 \\  & LoRa, LoRa + SAFE & 5.00E-04 & 16 & 50 \\  & GLUE / MRPC & BitFitFit + SAFE & 5.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 5.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 5.00E-04 & 32 & 50 \\  & LoRa, LoRa + SAFE & 5.00E-04 & 32 & 50 \\  & GLUE / ONLI & BitFit, BitFit + SAFE & 5.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 4.00E-04 & 32 & 50 \\  & LoRa, LoRa + SAFE & 8.00E-04 & 32 & 50 \\  & GLUE / STS-B & BitFit, BitFit + SAFE & 9.00E-04 & 32 & 50 \\  & Houlsby, Houlsby + SAFE & 5.00E-04 & 32 & 50 \\  & LoRa, LoRa + SAFE & 3.00E-04 & 16 & 50 \\ SQuAD & BitFit, BitFit + SAFE & 9.00E-04 & 16 & 50 \\  & Houlsby, Houlsby + SAFE & 10.0E-04 & 16 & 50 \\ \hline  & GLUE / CoLA & LoRa, LoRa + SAFE & 1.00E-04 & 32 & 80 \\  & GLUE / SST-2 & LoRa, LoRa + SAFE & 6.00E-04 & 32 & 60 \\  & GLUE / MNLI & LoRa, LoRa + SAFE & 1.00E-04 & 16 & 40 \\  & GLUE / RTE & LoRa, LoRa + SAFE & 6.00E-04 & 32 & 80 \\  & GLUE / QQP & LoRa, LoRa + SAFE & 3.00E-04 & 16 & 40 \\  & GLUE / MRPC & LoRa, LoRa + SAFE & 3.00E-04 & 4 & 50 \\  & GLUE / ONLI & LoRa, LoRa + SAFE & 2.00E-04 & 8 & 50 \\  & GLUE / STS-B & LoRa, LoRa + SAFE & 8.00E-04 & 32 & 50 \\  & Full-param Fine-tuning & 7.00E-05 & 16 & 50 \\  & LoRa, LoRa + SAFE & 3.00E-04 & 16 & 50 \\  & SlouAD & BitFit, BitFit + SAFE & 9.00E-04 & 16 & 50 \\  & Houlsby, Houlsby + SAFE & 1.00E-04 & 16 & 50 \\  & Pfeiffer, Pfeiffer + SAFE & 3.00E-04 & 16 & 50 \\  & AdaLOR, AdaLOR + SAFE & 4.00E-04 & 16 & 50 \\ \hline  & LoRa, LoRa + SAFE & 5.00E-04 & 32 & 50 \\ RoBERTa-base & SQuAD & BitFit, BitFit + SAFE & 8.00E-04 & 32 & 50 \\  & Houlsby, Houlsby+SAFE & 4.00E-04 & 32 & 50 \\ \hline  & LoRa, LoRa + SAFE & 6.00E-04 & 16 & 50 \\ RoBERTa-large & SQuAD & BitFit, BitFit + SAFE & 7.00E-04 & 16 & 50 \\  & Houlsby, Houlsby + SAFE & 4.00E-04 & 16 & 50 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Hyperparameter settings on the NLU and QA tasks.



* [14] M. C. K. A. Sari, J. R. L. van der Biel, and D. P. Kingma, "Adam: a method for stochastic optimization," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 209-218, 2015.
* [15] T. H. Lee, Y. Zhang, and Z. Chen, "Deep residual learning with deep convolutional neural networks," _IEEE Transactions on Neural Networks_**67**, no. 3, 2015, pp. 1-10.
* [16] G. Li, X. Wang, and W. Liu, "Learning to improve the performance of deep neural networks," _arXiv preprint arXiv:1502.03083_ (2015).

[MISSING_PAGE_POST]