MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter (2024)

Abstract: Parameter-Efficient Fine-tuning (PEFT) facilitates the fine-tuning of Large Language Models (LLMs) under limited resources. However, the fine-tuning performance with PEFT on complex, knowledge-intensive tasks is limited due to the constrained model capacity, which originates from the limited number of additional trainable parameters. To overcome this limitation, we introduce a novel mechanism that fine-tunes LLMs with adapters of larger size yet memory-efficient. This is achieved by leveraging the inherent activation sparsity in the Feed-Forward Networks (FFNs) of LLMs and utilizing the larger capacity of Central Processing Unit (CPU) memory compared to Graphics Processing Unit (GPU). We store and update the parameters of larger adapters on the CPU. Moreover, we employ a Mixture of Experts (MoE)-like architecture to mitigate unnecessary CPU computations and reduce the communication volume between the GPU and CPU. This is particularly beneficial over the limited bandwidth of PCI Express (PCIe). Our method can achieve fine-tuning results comparable to those obtained with larger memory capacities, even when operating under more limited resources such as a 24GB memory single GPU setup, with acceptable loss in training efficiency. Our codes are available at https://github.com/CURRENTF/MEFT.

heading: 1 Introduction
text: As Large Language Models (LLMs) scale up, their fine-tuning becomes increasingly resourceintensive (Touvron et al., 2023; Brown et al., 2020). To address this, Parameter Efficient Fine-Tuning (PEFT) methods, which add few trainable modules that only fraction of the LLMs’ full parameters (e.g., 0.1%), have shown promising results on many scenarios, often competitive to full finetuning (Houlsby et al., 2019; Hu et al., 2022; Li
*Corresponding author
and Liang, 2021; Liu et al., 2022; Vucetic et al., 2022). For instance, LoRA employs matrix decomposition parameters to leverage the low-rank characteristics of LLM parameters, while Parallel Adapter introduces an additional FFN adapter that operates parallel to original FFN layer.
However, the number of new parameters required can vary depending on the task. For knowledge-intensive tasks, such as QA, it becomes necessary to fine-tune a greater number of parameters to increase the model’s capacity for new knowledge memorization (Tirumala et al., 2022). Figure 1 illustrates the performance of LLaMA-7B with Parallel Adapter on NQ datasdet (Kwiatkowski et al., 2019). A clear upward trend in answer accuracy is observed as adapter’s parameter count increases. Concretely, for achieving optimal results, approximately 10% of the parameters are updated, which corresponds to more than 24G of GPU memory usage (the orange part
2375
in the figure). Therefore, reducing the memory cost when the trainable parameters increase is important and has attached research attentions. Among them, CPUoffload (Ren et al., 2021) offloads model parameters to CPU memory, which are then returned to the GPU only when computation reaches the corresponding layer. However, CPU-offload transfers all layer parameters, which is inflexible and incurs a high communication overhead.
To address these challenges, we propose MEFT, which introduces sparse activation to reduce memory usage. For a given input, sparse activation implies that only a few neurons are activated, as evidenced by various empirical results (Liu et al., 2023b; Geva et al., 2021) and our pilot experiments detailed at Section 2. Inspired by this, MEFT places all trainable parameters in CPU memory to leverage the advantage of the large capacity of CPU memory. For a given input, MEFT retrieves a few neurons from CPU memory that are highly relevant to input, and moves them to GPU to complete the computation. In this way, MEFT significantly reducing GPU memory usage. Furthermore, to reduces the computational burden on the CPU, we introduce a novel Mixture of Experts (MoE)-based adapter to partition parameters. Specifically, given an input, a router is used to activate a subset of the large-scale network and conduct further computation only on these activated subset. This approach effectively reduces the computational complexity from O(dNM) to O(dN √ M) to complete the retrieval of relevant parameters. We conduct experiments on two models (i.e., LLaMA and Mistral) and four datasets (i.e., NQ, SQuAD, ToolBench, and GSM8K). Experimental results on knowledge-intensive tasks show that our method achieves the best results under resourcerestricted conditions. Specifically, MEFT reduces GPU memory usage by 50% (e.g., reducing from 48G to 24G) while achieving comparable benchmark results. Our method also outperforms other PEFT methods like Parallel Adapter and LoRA when using the same memory capacity.
In summary, our contributions are as follows: (i) We propose MEFT, a novel method that utilizes sparse activations and MoE for memory-efficient fine-tuning (ii) MEFT reduces communication overhead by artificially limiting number of activated neurons copied to GPU memory from CPU memory. (iii) We propose a Key-Experts mechanism to partition a large number of parameters, using a
router to allocate inputs to corresponding experts, reducing unnecessary computations and lowering CPU computational burden. (iv) Our experimental results on knowledge-intensive tasks using two popular LLMs demonstrate that our method can achieve the best results under resource-constrained conditions, and the results are comparable to those in relatively resource-rich situations.

heading: 2 Preliminary
text: PEFT concentrates on fine-tuning LLMs under resource-restricted conditions. To examine performances of different PEFT on knowledge-intensive tasks, we fine-tune LLaMA-7B on Natural Questions with LoRA, AdapterH, and Parallel Adapter. Figure 1 shows the results of the experiments. We can see from the left blue part that it is challenging to inject knowledge into the model when the number of trainable parameters is limited (e.g., 0.1%). Parallel Adapter demonstrates the best scaling results, but for achieving optimal accuracy, approximately 10% of the parameters are updated. Therefore, we argue that to memorize more knowledge required by downstream tasks, the number of trainable parameters needs to be increased accordingly. Our study is based on Parallel Adapter due to its superior performance. Next, we introduce Parallel Adapter and our motivation for making it more efficient.
Parallel Adapter In Transformer-based language models, it has been discovered by researchers that FFNs function as key-value memories (Geva et al., 2021; Dai et al., 2022a), where each key corresponds to a text pattern, and each value directs the distribution over the output vocabulary. Based on this observation, Parallel Adapter propose to extend the original FFNs by adding a specific knowledge memories tailored for downstream tasks. Specifically, Parallel Adapter places the adapter parallel to the FFNs, and the adapter consists of two linear transformations, WA ∈ Rd×r and WB ∈ Rr×d, and a nonlinear activation function ReLU. The computation process of the FFN integrated with the Parallel Adapter can be articulated as follows:
FFNPA(h) = f(hWk)Wv + ReLU(hWA)WB,
where Wk ∈ Rd×n and Wv ∈ Rn×d are weights of the original LLM, h is the input hidden state of the FFN layer.
Sparsity in Parallel Adapter As shown in Figure 1, optimizing memory usage presents a challenge when implementing Parallel Adapter with an increased number of parameters for knowledgeintensive tasks. CPU-Offload is a straightforward strategy, where parameters are stored on the CPU and then transferred to the GPU as needed. However, this method transfers all layer parameters, leading to inflexibility and high communication overhead. In this work, we investigate the potential of leveraging activation sparsity in Parallel Adapter. We propose to activate only a select number of neurons for specific inputs, allowing for the transfer of only essential parameters, thereby reducing communication overhead.
We conduct an analysis on sparsity within the Parallel Adapter utilizing the LLaMA-7B model. Specifically, we trained a Parallel Adapter model with a bottleneck size of 4096 on the Natural Questions dataset. Then we extracted the activation of the adapter’s FFNs layer on the test set (which contains 4000 tokens) and calculated the average activation values. We also computed the cumulative activation values. Note that the neurons are sorted based on their activation value, and the data are normalized to the range [0, 1] for visualization. Figure 2 shows the results. It can be seen that the activation in the adapter is highly sparse, i.e., only a subset of neurons substantially contributes to the model predictions, while the majority of neurons are unactivated. Based on this observation, our work focuses on developing strategies for the effective selection of paramount parameters, and copying only these essential parameters to the GPU during training, thereby reducing the CPU-GPU
communication volume and VRAM usage.

heading: 3 Method
text: We propose MEFT, which dynamically loads parameters from CPU memory to GPU to train a larger size Adapter. In Section 3.1, we introduce Sparse Activation to reduce communication overhead, followed by the Key-Experts mechanism in Section 3.2 to reduce the computational complexity of CPU operations. We further analyze the efficiency of our method in Section 3.3.

heading: 3.1 Sparse Activation
text: Previous studies have identified contextual sparsity within FFN blocks, attributed to the use of activation functions like ReLU or GeLU (Kurtz et al., 2020). Notably, analyses on various downstream tasks have demonstrated at least 95% sparsity on FFN neurons, which further leads to sparse gradients. Based on these studies, we explore sparse Adapter training, selectively updating only those neurons that demonstrate higher activation.
Specifically, during the forward computation, K keys in WA with the highest similarity to h are retrieved and activated for each FFN layer:
S = TopK(hWA,K), (1)
where S denotes the indices of the selected keys. Then we form WKA , W K B with related indices on CPU.
WKA = WA[S] ∈ Rd×K (2) WKB = (W T B[S]) T ∈ RK×d (3)
Here, W[·] denotes the indexing operation that extracts the corresponding values from matrix W. We aim to extract relevant Keys and Values from WA and WB , respectively.
We then move WKA , W K B to GPU as a copy-
adapter, then calculate it as widened FFNs.
FFNPA(h) = f(hWk)Wv + ReLU(hWKA )W K B
(4) Finally, for backward propagation, only the gradients of these activated neurons are updated, as the non-activated neurons do not contribute to the computation of FFNPA.
In this way, the majority of the Parallel Adapter’s parameters remain stored in the CPU memory, and only these activated neurons need to be copied to GPU memory on-the-fly before each FFN computation. Given K ≪ r, where r is the total number of
neurons, the activation ratio is typically below 5%, allowing us to significantly save GPU memory.

heading: 3.2 Key-Experts Mechanism
text: In sparse activation, the TopK operation that retrieves the most similar weights is executed on the CPU, which may become the bottleneck of computing speed when r is large, given the CPU’s lower TFLOPs. To alleviate this issue, we further propose the Key-Experts mechanism to enhance computation efficiency.
The proposed mechanism is based on the idea of mixture-of-experts, where the weights WA and WB are divided into N partitions (experts): WAi,WBi, i ∈ {1, 2, . . . , N}, and a router R(·) is employed to route the input to some specific experts. Specifically, an expert Ei is an FFN that consists of WAi and WBi. For an input token h, the router R(·) calculates the score of each expert to be selected:
pi(h) = Wg · h (5)
Then, the top-K experts with the highest scores are selected; we use τ to denote the set of selected
indices. The reason why we didn’t use softmax to get probabilities is explained in Appendix B.
Then the weights of these selected experts are concatenated to W′A and W ′ B:
W′A = Concat([WAi, i ∈ τ ]) (6) W′B = Concat([WBi, i ∈ τ ]) (7)
For this specific token, we can consider W′A,W ′ B as the WA,WB mentioned in the previous section, but with a smaller size. We then retrieve the top-k key-value pairs to obtain WKA ,W K B .
Then, we can compute FFNPA(h) as shown in Algorithm 1.

heading: 3.3 Efficiency Analysis
text: While our method could reduce the GPU memory usage as only a fraction of activated neurons are placed on the GPU, the CPU-GPU communication and the CPU computation may cause GPU waiting. In this section, we analyze the communication volume and the computational complexity of our method.
Communication Volume As shown in Figure 3, the parameter communication volume between the
Algorithm 1 MEFT FFN Layer Input: hidden state h, original FFN F , router Wg, experts of keys {WA}, values WB , num of experts K, num of key-value pairs K. S ← ∅ // initialize S as an empty indices of kv pairs for t in h do p←Wg(t) // gating probabilities τ ← TopK(p,K) // indices of topK experts a← [] // initialize a as an empty array for i in τ do ai ← ReLU(hWAi) // sparse activation a← Concat(a,ai) // gather activation
end for I ← TopK(a,K) S ← S ∪ I
end for WKA ,W K B ← (WTA[S])T ,WB[S] // move WKA ,W K B to GPU return F (h) + ReLU(hWKA )W K B
CPU and GPU consists of two parts: forward and backward propagation. In the following, we use B to denote the batch size, l to denote the length of sequences in the batch, and β as a sparsity factor related to l.
• Forward. For each layer, the hidden state h needs to be moved from GPU to CPU, which causes B × l× d communication. Then, after the parameter selection, the activated parameters, with a size of 2× d× β ×K, are moved from CPU to GPU. Here β is a sparsity factor related to l1:β = ||Unique(TopK(hWA,K))||K
• Backward. For each layer, the gradients of the activated parameters calculated on the GPU are moved to the CPU to update the CPU counterpart. Thus the communication volume is the size of the activated parameters, i.e., 2× d× β ×K.
Therefore, the total communication overhead of model training is:
cost = nl × (2× d× β ×K +B × l × d). (8)
Computational Complexity The additional computation on the CPU includes computation on the router and the TopK operation. Based on the
1In our experiments, we found that the tokens in a sentence tend to activate similar experts, leading to a smaller β < 0.49 (experimental results when B = 2, l = 256).
proposed key-experts mechanism, the complexity on the CPU is MNd + K rMdN . Therefore, when N approaches √ r, we can achieve optimal computational complexity O(dM √ r). This significant reduction in computation makes it well-suited for CPU execution.
Empirical Results Our empirical results show that when using LLaMA-7B as the base model, with key-value pairs of size 6144, a batch size of 2, and a sequence length of 256, the total bidirectional communication volume per batch is about 0.56M per iteration, whereM represents the total number of trainable parameters added to Llama-7B. In contrast, when using deepspeed-offload (Ren et al., 2021) with the same trainable size, the Parallel Adapter requires a communication volume of 2M per iteration. Therefore, our method achieves a 3.57× reduction in GPU-CPU communication.
Regarding training efficiency, empirical results show that our method achieves at least 63% speed compared to the baseline that excludes the time cost of additional communication and CPU computation. Moreover, there remains the potential to further improve efficiency through engineering methods as detailed in Appendix A.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: We consider four datasets for experiments: Natural Question, SQuAD, ToolBench, and GSM8K:
• Natural Questions (Kwiatkowski et al., 2019) is an open-domain question answering dataset. We are considering a close-book setting, i.e., the models are trained to answer questions without background passages (Wang et al., 2021).
• SQuAD (Rajpurkar et al., 2016) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles. The same close-book setting like NQ is used in SQuAD too.
• ToolBench (Qin et al., 2024) is a tool learning dataset, which includes 16K APIs and 432K related queries. The task aims to predict the name and info of APIs that solve the query.
• GSM8K (Cobbe et al., 2021) is a dataset of 8.5K high-quality diverse grade school math word problems. We use MetaMathQA (Yu et al., 2024) as training data for GSM8K.

heading: 4.2 Metrics
text: For NQ and SQuAD, we use Exact Match (EM) to measure whether the standard answer appears in the model’s output. For ToolBench, we use Intersection over Union (IoU) as the metric, taking the intersection and union of the APIs in the model’s answer set Y and the ground truth set G, and then calculate as: IoU = Y∩GY∪G . For GSM8k, we judge whether the final answer given by the model (matched by regular expression) is correct.

heading: 4.3 Implementation Details
text: We utilized the publicly pre-trained LLaMA-7B (Touvron et al., 2023) and Mistral-7B (Jiang et al., 2023) as the base model. All experiments are implemented using PyTorch and the Hugging Face Trainer, on machines with RTX-3090 or A40 GPUs. During fine-tuning, we employ the Adam optimizer for all experiments. For LLaMA-7B, we trained models with the learning rate progressively increasing over the initial 2% of the steps to 1e−4, and then decaying linearly to 0. For Mistral-7B, we trained models with a peak learning rate of 1e−6 to ensure numerical stability. For tasks NQ, ToolBench, and GSM8k (MetaMathQA), we trained models with an accumulated batch size of 64. We set the number of activated key-value pairs K = 64 for all tasks, and the number of activated experts
K = 4 √ r. The training epochs for the tasks NQ, SQuAD, Tool, and MetaMathQA (GSM8k) are set to 4, 8, 1, and 1, respectively. Each experiment is conducted on an RTX 3090, taking approximately 12 GPU hours, and the results are reported from a single run. For task SQuAD, we used a higher batch size of 256 for better performance across all methods. For baselines with LLaMA-7B, we tested their performance with the rank/bottleneck size set to {32, 64, 128, 256, 512, 1024, 2048} to achieve optimal performance. It’s important to note that a rank larger than 512 and a bottleneck size larger than 256 require 48 GB of GPU memory. For baselines with Mistral-7B, we tested the rank/bottleneck size set to 32, 64, 128, and 256, but only within a 24 GB setting. For most experiments and hyperparameter studies, we report our results from MEFT on Natural Questions.

heading: 5 Experimental Results

heading: 5.1 Main Results
text: Table 1 presents the main experimental results. Key observations include: (i) In knowledge-intensive tasks such as NQ, SQuAD, and Tool, our MEFT model significantly outperforms other PEFT approaches when operating under same 24G GPU memory constraints. For instance, MEFT achieves 0.413 and 0.427 EM scores on NQ using LLaMA-
7B and Mistral-7B models, respectively. These scores are notably higher compared to those from baseline methods like Parallel Adapter and LoRA. This improvement is attributed to the effective fit of a higher proportion of trainable parameters (i.e., 10%) within the limited 24G GPU capacity. Moreover, our method also achieves results comparable to full-model fine-tuning. (ii) When compared with other baselines that also incorporate 10% trainable parameters, MEFT achieves comparable benchmark performance but only requires 50% of the GPU memory. This demonstrates MEFT’s enhanced efficiency in resource utilization. (iii) For tasks that are not knowledge-intensive, such as GSM8k, increasing the number of trainable parameters does not yield better performance. However, MEFT’s results on GSM8K indicate that sparse training does not compromise performance on these type of task. In addition, we conducted experiments where LoRA is attached to FFNs and the detailed results are available in Appendix C.1.

heading: 5.2 Ablation Study of Sparse Activation
text: It’s important to note that the main purpose of using a mechanism similar to MoE (Mixture of Experts) is to alleviate the computational burden on the CPU. The experimental results show that MoE is not the main source of performance improvement. As shown in Figure 4, partitioned parameters WA have minimal impact on knowledge-intensive tasks, even improving performance on SQuAD and ToolBench. The term "brutal offload" refers to the straightforward exchange of parameters between the CPU and GPU. However, it exhibits slightly lower performance on GSM8k, a task with a stronger logical component. This suggests that logical tasks may
not require a large number of parameters.

heading: 5.3 Efficiency Results
text: To analyze the efficiency of our method, Figure 5 illustrates the training latency (wall time) per batch, using a server with an RTX 3090 GPU and a 32- core CPU that supports AVX. This figure presents the time taken for training across various ablation studies focusing on data transfer, CPU computation, and GPU computation. Specifically, “MEFT w/o both” refers to offloading all trainable parameters to CPU and performing computations on the CPU without any optimization, leading to the highest latency observed. “MEFT w/o Sparse” offloads too but optimizes by transferring only necessary neurons via PCIe, which cuts down on data transfer time and slightly improves GPU computation efficiency. “MEFT w/o KE” uses a mixture of experts (MoE) approach for parameter management, reducing computational load but still involves complete parameter transfer. “Parallel Adapter” performs all operations on the GPU, thus achieving the best latency.

heading: 5.4 Hyperparameters Studies
text: Given that numerous parameters may influence performance on downstream tasks, we have tested various hyperparameter settings on the model.
Number of additional key-value pairs refers to the number of additional parameters that will be trained. The results of LoRA, Adapter, and Parallel Adapter on NQ are shown in Figure 1. It can be observed that when the model needs to be adapted for knowledge-intensive tasks, it often requires more parameters. We also conducted additional tests on the results of Parallel Adapter on SQuAD and Tool-

heading: 128 0.4% 0.106 0.604

heading: 512 1.6% 0.120 0.593
text: Bench, as shown in Table 2. These results shows that different datasets require different amounts of additional parameters. SQuAD shows a consistent improvement in performance with an increase in the number of parameters. However, in the NQ results depicted in Figure 1, the best performance was achieved when 3072 key-value pairs were added to each layer. In contrast, in ToolBench, the best result is achieved when only adding 1024 key-value pairs per layer.
Number of activated key-value pairs . We examined the impact of limiting the number of keyvalue pairs a token can activate on the final performance of the model. As shown in Figure 6, suggest that artificially adding sparsity constraints during training does not substantially affect the performance of the model, when an appropriate value of K is selected. And acceptable performance can be achieved when a single token activates less than 3% of the parameters.
Number of Key-Experts refers to the number of partitions in WA. When the number of key experts per layer is equal to the number of additional keyvalue pairs, each neuron belongs to an independent expert, equivalent to not using MoE partitioning. The results, shown in Figure 7, align with this theory: the more experts there are (i.e., the more partitions), the better the outcome. At the same time,
we find that decent results can be achieved even when the number of experts is small. Consider the case where the number of experts is one, the router ∈ Rd. The process of retrieving the relevant neuron can be seen as h first dot-multiplying with the router. At this time, all neurons are in expert E0, which is to some extent equivalent to having r partitions.

heading: 5.5 Impact of Batch Size
text: The batch size impacts the weights selected using the top-k method, as the probability of choosing the top-k key-value pairs based on activation values varies among different samples within the batch. Therefore, as the batch size increases, the proportion of activated parameters increases, as shown in Figure 8. It should be noted that our method mainly considers the single-card situation. If a 7B size model is being trained, without using other memory-saving techniques, the maximum batch size that can generally be selected is 4. Therefore, the amount of communication reduced by our method is quite substantial.

heading: 6 Related Work
text: Parameter-Efficient Fine-Tuning (PEFT) can be categorized into four main types: (1) adapter-
based tuning (Houlsby et al., 2019; Pfeiffer et al., 2021), (2) prompt-based tuning (Liu et al., 2023a; Li and Liang, 2021; Schick and Schütze, 2021; Gao et al., 2021), (3) sparse tuning (Sung et al., 2021; Vucetic et al., 2022), and (4) reparametrizationbased tuning (Hu et al., 2022; Edalati et al., 2022). There are several studies that are relevant to our work. He et al., 2022 utilized a unified perspective of previous PEFT methods to propose the Parallel Adapter, which has demonstrated superior performance among most popular PEFT methods (Hu et al., 2023). In our study, we found the Parallel Adapter to be highly effective in knowledge memorization. Wang et al., 2022 proposed AdaMix as a method that tunes a mixture of adaptation modules but only with 0.1 - 0.2% of LLM’s parameters. Furthermore, Zhang et al., 2023 preserved the WA in the LoRA module (Hu et al., 2022) in a frozen state, which resulted in reduced memory usage but didn’t try to scale up.
Mixture of Experts is a computationally efficient architecture that only activates a subset of a large-scale neural network compared to a dense model of the same size (Shazeer et al., 2017; Fedus et al., 2022; Dai et al., 2022b). We leverage the computational efficiency of MoE to alleviate the issue of insufficient CPU FLOPs and take advantage of the large memory capacity of the CPU to compensate for its tendency to occupy more memory.
Additional minor related works can be found in Appendix E.

heading: 7 Conclusion
text: In this paper, we found that the Parallel Adapter can continuously improve performance on knowledgeintensive tasks as parameters increase. We then proposed a memory-efficient training method by leveraging sparsity and MoE. This method significantly reduces the demand for GPU memory and reduces the computational pressure on the CPU. Experimental results show that our method achieves the best results under resource-restricted conditions.
Limitations
Although we have tested the impact of the number of added key-value pairs, the number of keyexperts, and the size of K on the model’s performance on knowledge-intensive tasks, the effect of this method on the generalization ability of LLMs has not been fully explored. Moreover, this method also lacks testing in the scenario of continuous
learning, where it is also suitable. Additionally, the amount of parameters recalled by this method increases with the length of the training sequence, which limits its applicability. We will further explore how to mitigate this phenomenon in our future work.
Due to the absence of detailed memory management and optimization of data transfer between the CPU and GPU (such as custom CUDA streams), the final efficiency of our method is currently not optimal.
Ethics Statement
We acknowledge the importance of the ACM code of Ethics and totally agree with it. We ensure that this work is compatible with the provided code, in terms of the publicly accessed datasets and models. In particular, this paper focus on the efficient training of Large Language Models (LLMs). Prior research indicates that LLMs can demonstrate biased responses and generate hallucinated information, even when trained on specialized datasets. Therefore, it’s crucial to reassess their applicability depend on real-world scenarios.

heading: Acknowledgements
text: This work was supported by the Natural Science Foundation of China (62202271, 61902219, 61972234, 61672324, 62072279, 62102234, 62272274), the National Key R&D Program of China with grants No. 2020YFB1406704 and No. 2022YFC3303004, the Natural Science Foundation of Shandong Province (ZR2021QF129, ZR2022QF004), the Key Scientific and Technological Innovation Program of Shandong Province (2019JZZY010129), the Fundamental Research Funds of Shandong University, the Tencent WeChat Rhino-Bird Focused Research Program (WXG-FR2023-07). All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.

heading: B Negative Attempts
text: The softmax function is commonly used in MoE (Mixture of Experts) to normalize the logits assigned to each expert by the tokens. However, in our scenario, this approach actually led to a decrease in model performance for the NQ and SQuAD tasks during testing.

heading: C Other Experiments
text: C.1 LoRA on FFN
Since FFNs are crucial to Transformers, we also attempted to attach LoRA to FFNs. As shown in Table 3, LoRA still cannot gradually improve performance with the increase of trainable parameters.

heading: D Potential Risks
text: LLMs’ stronger memory of knowledge could potentially lead to a stronger memory of privacy or other sensitive information. With the ongoing research into language model attacks (Birch et al., 2023; Niu et al., 2024), this enhanced memory of knowledge by LLMs could result in greater leakage of private information, particularly if trained on private datasets.

heading: E Other Related Works
text: Other PEFTs . Additionally, Diao et al., 2023 attempted to incorporate domain-specific knowledge by adopting a mixture-of-adapters mechanism, but only 0.7% of the trainable parameters of RoBERTalarge (Liu et al., 2019) were utilized, limiting the model’s capacity.
Sparsity of LLM has been partially explored with the objective of accelerating inference. SparseGPT (Frantar and Alistarh, 2023) demonstrated that LLMs can be pruned to a sparsity level of at least 50% with only a minimal loss in accuracy. The work of Mirzadeh et al. (2024) indicated that the ReLU activation function, through sparse activation, significantly reduces computation and weight transfer. Deja Vu (Liu et al., 2023b) identified the presence of contextual sparsity for any given input and illustrated that a sparsity level of at least 95% can be achieved within the FFN Block. We apply these findings on sparsity to the finetuning process.
Memory Efficient Fine-tune Methods . The main feature of QLoRA (Dettmers et al., 2023) is to reduce memory usage by quantizing the model, and quantization is orthogonal to our method, so we can further quantize the parameters exchanged between the CPU and GPU. Liao et al., 2024 primarily reduces the memory usage of intermediate activations by reversing the model. However, in the scenarios we consider, when more trainable parameters are added, the memory consumed by these parameters and their corresponding optimizer states already exceeds the total. LST (Sung et al., 2022) constructs a "bypass" based on the original large model, making it unnecessary to perform backpropagation directly on the original large model. Deepspeed’s Zero (Ren et al., 2021) uses offload technology to exchange parameters between the CPU and GPU, which is similar to our method. However, our method utilizes sparsity, selecting relevant parameters based on context to reduce the communication pressure between the CPU and GPU, and uses the Mixture of Experts (MoE) to reduce the computational pressure on the CPU.
Neural Memory proposed by Sukhbaatar et al. (2015) consists of n key-value pairs, with both key and value represented by a d-dimensional vector ki, vi ∈ Rd, forming K,V ∈ Rn×d. Then MN(x) = softmax(x ·KT ) · V is used to acquire knowledge related to x. Geva et al. (2021) proposed that the FFN Block in Transformers can emulate Neural Memory, suggesting that ki captures the pattern of the input sequence, while the value represents the distribution of the token. Dai et al. (2022a) demonstrated that factual knowledge is stored in what they term ’knowledge neurons’, and that editing these neurons can modify the stored knowl-
edge. Similarly, ROME (Meng et al., 2022) found that knowledge is stored in the middle-layer feedforward modules. They used causal intervention to identify decisive neuron activations. We leveraged the key-value format of knowledge neurons in conjunction with the sparsity of LLMs, which significantly reduced the memory overhead during the fine-tuning phase.
Approximate Nearest Neighbor Search algorithms sacrifice recall to improve the efficiency of nearest neighbor searches. However, popular methods such as HNSW (Malkov and Yashunin, 2018), Faiss (Johnson et al., 2019), and ScaNN (Guo et al., 2020) have a low tolerance for frequently updated vectors, as this affects the pre-built retrieval structure. Since backpropagation will result in the update of all activated FFN neurons, we did not use ANN. Although some studies (Chen et al., 2020, 2021; Zeng et al., 2023) have achieved comparable or even faster CPU training speeds than the GPU by applying LSH, these methods are applicable to rather specific network configurations. For instance, the method in Chen et al., 2020 is only suitable when the network is very wide. Our method, however, significantly reduces the overall computation volume by partitioning the parameter matrix in a way similar to MoE. We will delve into the specifics in the following analysis. However, popular methods such as HNSW (Malkov and Yashunin, 2018), Faiss (Johnson et al., 2019), and ScaNN (Guo et al., 2020) have a low tolerance for frequently updated vectors, as this affects the pre-built retrieval structure. Since backpropagation will result in the update of all activated FFN neurons, we propose a Key-Experts (KE) mechanism to tackle this issue.

heading: Future Work
text: We will further explore how to mitigate this phenomenon in our future work. Due to the absence of detailed memory management and optimization of data transfer between the CPU and GPU (such as custom CUDA streams), the final efficiency of our method is currently not optimal