LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

Authors: rui  pan
xiang  liu
shizhe  diao
renjie  pi
jipeng  zhang
chi  han
tong  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:59:58

Abstract: The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significant roadblock to large-scale training. For instance, a 7B model typically requires at least 60 GB of GPU memory with full parameter training, which presents challenges for researchers without access to high-resource environments. Parameter Efficient Fine-Tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem. However, in most large-scale fine-tuning settings, their performance does not reach the level of full parameter training because they confine the parameter search to a low-rank subspace. Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in a wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled AdamW (LISA), a promising alternative for LoRA, which applies the idea of importance sampling to different layers in LLMs and randomly freeze most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while achieving on-par or better performance in MMLU, AGIEval and WinoGrande. On large models, specifically LLaMA-2-70B, LISA surpasses LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating its effectiveness across different domains.

Main paper: 
 introduction: 
 Large language models (LLMs) like ChatGPT excel in tasks such as writing documents, generating complex code, answering questions, and conducting human-like conversations [1] . With LLMs being increasingly applied in diverse task domains, domain-specific fine-tuning has emerged as a critical strategy to enhance their downstream capabilities [2, 3, 4, 5] . Nevertheless, these methods are typically time-intensive and consume substantial computational resources, posing significant challenges to the development of large-scale models [6] . For example, continual pre-training typically requires several weeks even with multiple 80 GB GPUs. To reduce costs, Parameter-Efficient Fine-Tuning (PEFT) techniques have been proposed to minimize the number of trainable parameters. These techniques include adapter weights [7] , prompt weights [8] , and LoRA [9] . Among these, LoRA stands out as one of the most widely adopted due to its unique ability to merge the adaptor back into the base model parameters, significantly enhancing efficiency. However, LoRA's superior performance in fine-tuning tasks has yet to reach a point that universally surpasses full parameter fine-tuning in all settings [10, 11] . In particular, it has been observed that LoRA tends to falter on large-scale datasets during continual pre-training [12] , which raises doubts about the effectiveness of LoRA under those circumstances. We attribute this to LoRA's much fewer trainable parameters compared to the base model, which limits the representation power of LoRA training.To overcome this shortcoming, we delve into LoRA's training statistics in each layer, aspiring to bridge the difference between LoRA and full-parameter fine-tuning. Surprisingly, we discover that LoRA's layerwise weight norms have an uncommonly skewed distribution, where the bottom layer and/or the top layer occupy the majority of weights during the update. In contrast, the other self-attention layers only account for a small amount, which means different layers have different importance when updating. This key observation inspires us to "sample" different layers by their importance, which matches the idea of importance sampling [13, 14] .As a natural consequence, this strategy brings forth our Layerwise Importance Sampled Adam (LISA) algorithm, where by selectively updating only essential LLM layers and leaving others untouched, LISA enables training large-scale language models (≥ 65B parameters) with less or similar memory consumption as LoRA. Furthermore, fine-tuned on downstream tasks, LISA outperformed both LoRA and conventional full-parameter fine-tuning approaches by a large margin, indicating the large potential of LISA as a promising alternative to LoRA.We summarize our key contributions as follows,• We discover the phenomenon of skewed weight-norm distribution across layers in LoRA, which implies the varied importance of different layers in large-scale LLM training.• We propose the Layerwise Importance Sampled AdamW (LISA), a simple optimization method capable of scaling up to over 70B LLMs with less or similar memory cost as LoRA.• We demonstrate LISA's effectiveness in fine-tuning tasks for modern LLMs, where it outperforms LoRA by 10%-35% in MT-Bench and achieves better performance in multiple benchmarks. In addition, LISA exhibits much better convergence behaviors than LoRA. LISA even outperforms full parameters training under certain settings. Similar performance gain is observed across different sized models (7B-70B) and tasks, including instruction following, medical QA, and math problems.2 Related Work 
large language models: 
 In the realm of natural language processing (NLP), the Transformer architecture has been a revolutionary technique, initially known for its effectiveness in machine translation tasks [15] . With the inception of models like BERT [16] and GPT-2 [17] , the approach shifted towards pre-training on extensive corpora, which led to significant performance enhancements in downstream fine-tuning tasks [2, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27] . However, the growing number of parameters in these models results in a huge GPU memory consumption, rendering the fine-tuning of large scale models (≥ 65B) infeasible under low resource scenarios. This has prompted a shift towards more efficient training of LLMs.Parameter-efficient fine-tuning (PEFT) methods adapt pre-trained models by fine-tuning only a subset of parameters. In general, PEFT methods can be grouped into three classes: 1) Prompt Learning methods [8, 28, 29, 30, 31, 32, 33] , 2) Adapter methods [7, 9, 34, 35, 36, 37, 38] , and 3) Selective methods [39, 39, 40, 41] . Prompt learning methods emphasize optimizing the input token or input embedding with frozen model parameters, which generally has the least training cost among all three types. Adapter methods normally introduce an auxiliary module with much fewer parameters than the original model, and updates are only applied to the adapter module during training. Compared with them, selective methods are more closely related to LISA, which focuses on optimizing a fraction of the model's parameters without appending extra modules. Recent advances in this domain have introduced several notable techniques through layer freezing. AutoFreeze [39] offers an adaptive mechanism to identify layers for freezing automatically and accelerates the training process. FreezeOut [42] progressively freezes intermediate layers, significantly reducing training time without notably affecting accuracy. The SmartFRZ [40] framework utilizes an attention-based predictor for layer selection, substantially cutting computation and training time while maintaining accuracy. However, none of these layer-freezing strategies has been widely adopted in the context of Large Language Models due to their inherent complexity or non-compatibility with modern memory reduction techniques [43, 44, 27] for LLMs.In contrast, the Low-Rank Adaptation (LoRA) technique is much more prevalent in common LLM training [9] . LoRA reduces the number of trainable parameters by employing low-rank matrices, thereby lessening the computational burden and memory cost. One key strength of LoRA is its compatibility with models featuring linear layers, where the decomposed low-rank matrices can be merged back into the original model. This allows for efficient deployment without changing the model architecture. As a result, LoRA can be seamlessly combined with other techniques, such as quantization [11] or Mixture of Experts [45] . Despite these advantages, LoRA's performance is not universally comparable with full parameter fine-tuning. There have been tasks in [10] that LoRA performs much worse than full parameter training on. This phenomenon is especially evident in large-scale pre-training settings [12] , where to the best of our knowledge, only full parameter training was adopted for successful open-source LLMs [21, 22, 23, 46, 47, 26, 27] .In addition to approaches that change model architectures, there have also been efforts to improve the efficiency of optimization algorithms for LLMs. One such approach is layerwise optimization, a concept with roots extending back several decades. Notably, [48] introduced an effective layer-bylayer pre-training method for Deep Belief Networks (DBN), demonstrating the benefits of sequential layer optimization. This idea was expanded by researchers like [49, 50] , who illustrated the advantages of a greedy, unsupervised approach to pre-training each layer of deep networks. In the context of large batch training, [51, 52] developed LARS and LAMB to improve generalization and mitigate the performance declines associated with large batch sizes. Despite these innovations, Adam [53, 54, 55, 27] and AdamW [56] continue to be the predominant optimization methods used in most LLM settings.Recently, other attempts have also been made to reduce the training cost of LLMs. For example, MeZO [57] adopted zeroth order optimization, bringing significant memory savings during training. However, it also incurred a considerable performance drop in multiple benchmarks, particularly in complex fine-tuning scenarios. Regarding acceleration, Sophia [58] incorporates clipped second-order information into the optimization, obtaining non-trivial speedup on LLM training. The significant downsides are its intrinsic complexity of Hessian estimation and unverified empirical performance in large-size models (e.g., ≥ 65B). In parallel to our work, [59] proposed GaLore, a memory-efficient training strategy that reduces memory cost by projecting gradients into a low-rank compact space.Yet the performance has still not surpassed full-parameter training in fine-tuning settings. To sum up, LoRA-variant methods [9, 11, 59] with AdamW [56] is still the dominant paradigm for large-size LLM fine-tuning, the performance of which still demands further improvements. 
motivation: 
 To understand how LoRA achieves effective training with only a few parameters, we conducted empirical studies on multiple models, especially observing the weight norms across various layers.We fine-tune it on the Alpaca-GPT4 dataset [60] . During the training, we meticulously recorded the mean weight norms of each layer ℓ at every step t after updates, i.e.w (ℓ) ≜ mean-weight-norm(ℓ) = 1 T T t=1 ∥θ (ℓ) t ∥ 2Figure 2 presents these findings, with the x-axis representing the layer id, from embedding weights to the final layer, and the y-axis quantifying the weight norm. The visualization reveals one key trend:• The embedding layer or the language modeling (LM) head layer exhibits significantly larger weight norms than intermediary layers in LoRA, often by a factor of hundreds. This phenomenon, however, was not salient under full-parameter training settings. This observation indicates that the update emphasis of LoRA and full parameter training differ significantly, which can be attributed to the difference in their learned knowledge. For example, in embedding layers, tokens with similar meanings, i.e., synonyms, can be projected into the same embedding space and converted to similar embeddings. LoRA may capture this similarity in language and "group" them in the low-dimension space, allowing frequent features of language meanings to be promptly identified and optimized. The price is LoRA's limited representation power restricted by its intrinsic low-rank space, as we can see from the comparison with LISA in image generation tasks (Appendix A.1), where LoRA memorizes and learns details much slower than LISA. Other possible explanations can also justify this phenomenon. Despite various interpretations of this observation, one fact remains clear: LoRA values layerwise importance differently from full parameter tuning.To exploit the discovery above, we aspire to simulate LoRA's updating pattern via sampling different layers to freeze. This way, we can avoid LoRA's inherent deficiency of limited low-rank representation ability and emulate its fast learning process. Intuitively, given the same global learning rates across layers, layers with small weight norms in LoRA should also have small sampling probabilities to unfreeze in full-parameter settings so the expected learning rates across iterations can stay the same. This is exactly the idea of importance sampling [13, 14] , where instead of applying layerwise different learning rates {η t } in full-parameter settings to emulate LoRA's updates {η t }, we apply sampling and instead get the same expected parameter updateη (ℓ) t = η(ℓ) t • w(ℓ) w (ℓ) ⇒ η (ℓ) t = η (ℓ) , p (ℓ) = w(ℓ) w (ℓ)This gives rise to our Layerwise Importance Sampling AdamW method, as illustrated in Algorithm 1.In practice, since all layers except the bottom and top layer have small weight norms in LoRA, we adopt {p ℓ } N L ℓ=1 = {1.0, γ/N L , γ/N L , . . . , γ/N L , 1.0} in practice, where γ controls the expected number of unfreeze layers during optimization, and the embedding layer E and head layer H remain active. Intuitively, γ serves as a compensation factor to bridge the difference between LoRA and full parameter tuning, letting LISA emulate a similar layerwise update pattern as LoRA. To further control the memory consumption in practical settings, we instead randomly sample γ layers every time to upper-bound the maximum number of unfrozen layers during training. Algorithm 1 Layerwise Importance Sampling AdamW (LISA) Require: number of layers N L , number of iterations T , sampling period K, number of sampled layers γ, initial learning rate η 0 1:for i ← 0 to T /K -1 do 2:Freeze all layers except the embedding and language modeling head layer • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
memory efficiency: 
 We conducted peak GPU memory experiments to demonstrate LISA's memory efficiency and showcase its comparable or lower memory cost than LoRA. Settings To reasonably estimate the memory cost, we randomly sample prompts from the Alpaca dataset [61] and limit the maximum output token length to 1024. We focus on two key hyperparameters: LoRA's rank and LISA's number of activation layers. For other hyperparameters, a mini-batch size of 1 was consistently used across five LLMs from 120M to 70B parameters, deliberately excluding other GPU memory-saving techniques such as gradient checkpointing [62] , offloading [63] , and flash attention [64, 65] . All memory-efficiency experiments are conducted on 4× NVIDIA Ampere Architecture GPUs with 80G memory.Results Upon examining Table 1 , it is evident that the LISA configuration, particularly when enhanced with both the embedding layer (E) and two additional layers (E+H+2L), demonstrates a considerable reduction in GPU memory usage when fine-tuning the LLaMA-2-70B model, as compared to the LoRA method. Specifically, the LISA E+H+2L configuration shows a decrease to 75G of peak GPU memory from the 79G required by the LoRA Rank 128 configuration. This efficiency gain is not an isolated incident; a systematic memory usage decrease is observed across various model architectures, suggesting that LISA's method of activating layers is inherently more memory-efficient.In Figure 3 , it is worth noticing that the memory reduction in LISA allows LLaMA-2-7B to be trained on a single RTX4090 (24GB) GPU, which makes high-quality fine-tuning affordable even on a laptop computer. In particular, LISA requires much less activation memory consumption than LoRA since it does not introduce additional parameters brought by the adaptor. LISA's activation memory is even slightly less than full parameter training since pytorch [66] with deepspeed [44] allows deletion of redundant activations before backpropagation. On top of that, a reduction in memory footprint from LISA also leads to an acceleration in speed. As shown in Figure 4 , LISA provides almost 2.9× speedup when compared with fullparameter training, and ∼ 1.5× speedup against LoRA, partially due to the removal of adaptor structures. It is worth noticing that the reduction of memory footprint in both LoRA and LISA leads to a significant acceleration of forward propagation, emphasizing the importance of memory-efficient training.LISA can achieve this significant memory saving while still obtaining competitive performance under the fine-tuning setting. Settings To demonstrate the superiority of LISA over LoRA, we evaluate them on the instructionfollowing fine-tuning task with the Alpaca GPT-4 dataset [61] , which consists of 52k conversation pairs generated by GPT-4 [5] . The effectiveness of fine-tuning was evaluated on multiple benchmarks: MT-Bench [67] features 80 high-quality, multi-turn questions designed to assess LLMs on multiple aspects; MMLU [68] includes a total of 57 tasks with 14,079 questions covering a broad spectrum of world knowledge; AGIEval [69] serves as a human-centric benchmark for general abilities, comprising 9,316 instances; WinoGrande [70] is a large-scale dataset for commonsense reasoning, consisting of 44,000 instances designed to challenge models' understanding of the context and commonsense knowledge.In our experiments, we assessed three baseline models: TinyLlama [47] , Mistral-7B [46] , and LLaMA-2-7B [23] . These models, varying in size ranging from 1B to 7B parameters, provide a diverse representation of decoder-only models. For hyper-parameters, we adopt a rank of 128 for LoRA and E+H+2L for LISA in this section, with full details available in Appendix B. Results Table2 and 3 present a detailed comparison on moderate-scale LLMs. The baselines include Full-parameter Training (FT), Low-Rank Adaptation (LoRA) [9] and Gradient Low-Rank Projection (Ga-Lore) [59] . The results demonstrate that LISA consistently outperforms other fine-tuning methods in most evaluation tracks, indicating its robustness and effectiveness across diverse tasks and model architectures.LISA is particularly effective in instruction following tasks, where a large gap is observed when compared with other baseline methods. LISA even outperforms Full-parameter Training, suggesting that an implicit regularization effect is present when the number of unfrozen layers is restricted, which is similar to dropout [71] . According to more results in stable diffusion and detailed MT-Bench scores, we found that LISA outperforms LoRA mostly in memorization tasks, such as depicting high-resolution image details in image generation, or Writing or Humanities tasks in instruction following. This implies that LISA's performance improvement may majorly come from the ability to memorize long-tailed patterns, while LoRA is better at multi-hop reasoning with limited knowledge. For more details, please refer to Appendix A.1 and A.2. Settings We adopt the mathematics corpus OpenWebMath [72] for constructing the continual pre-training dataset. Specifically, we extracted a high-quality subset from it which contains 1.5 billion tokens. Full details are explained in Appendix B.2. After continual pretrainig, we then apply the same fine-tuning procedure on the GSM8K [73] training set, which comprises 7473 instances.Results Table 4 shows that LISA is capable of achieving on-par or even better performance than full-parameter training with much less memory consumption. Specifically, LISA requires only half of the memory cost compared to full-parameter training. This indicates a better balance between computational efficiency and model performance is achieved by LISA. According to our experience, reducing the number of unfrozen layers to half the original size leads to no worse or even better performance during continual pretraining, while requiring much less memory consumption.To further demonstrate LISA's scalability on large-sized LLMs, we conduct additional fine-tuning experiments on LLaMA-2-70B [23] .Settings On top of the aforementioned instruction-following tasks in Section 4.2, we use extra domain-specific fine-tuning tasks on mathematics and medical QA benchmarks. The GSM8K dataset [73] , comprising 7473 training instances and 1319 test instances, is used for the mathematics domain. For the medical domain, we select the PubMedQA dataset [74] , which includes 211.3K artificially generated QA training instances and 1K test instances. Evaluation on the PubMedQA dataset [74] is conducted in a 5-shot prompt setting, while the GSM8K dataset [73] assessment was conducted using Chain-of-Thought (CoT) prompting, following recent studies [75, 76, 77] . Regarding hyperparameters, as detailed in the section 4.1, we utilize the rank 256 for LoRA and the configuration E+H+4L for LISA. Further information is available in Appendix B. Hyperparameters of LISA The two key hyperparameters of LISA are the number of sampling layers γ and sampling period K. To obtain intuitive and empirical guidance of those hyperparameter choices, we conduct ablation studies using TinyLlama [47] and LLaMA-2-7B [23] models with the Alpaca-GPT4 dataset. The configurations for γ, such as E+H+2L, E+H+8L, were denoted as γ = 2 and γ = 8. As for the sampling period K = T /n, T = 122 representing the maximum training step within our experimental framework. The findings, presented in Table 6 , reveal that both γ and K markedly affect the LISA algorithm's performance. Specifically, a higher γ value increases the quantity of trainable parameters, albeit with higher memory costs. On the other hand, an optimal K value facilitates more frequent layer switching, thereby improving performance to a certain threshold, beyond which the performance may deteriorate. Generally, the rule of thumb is: More sampling layers and higher sampling period lead to better performance. For a detailed examination of loss curves and MT-Bench results, refer to Appendix A.4. For this purpose, we further investigate LISA's performance variance over three distinct runs, each with a different random seed for layer selection. Here, we adopt TinyLlama, LLaMA-2-7B, and Mistral-7B models with the Alpaca-GPT4 dataset while keeping all other hyperparameters consistent with those used in the instruction following experiments in section 4.2. As shown in Table 7 , LISA is quite resilient to different random seeds, where the performance gap across three runs is within 0.13, a small value compared to the performance gains over baseline methods. For more ablation experiment on LISA hyperparameters, please refer to Appendix A.4. 
discussion: 
 Theoretical Properties of LISA Compared with LoRA, which introduces additional parameters and leads to changes in loss objectives, layerwise importance sampling methods enjoy nice convergence guarantees in the original loss. For layerwise importance sampled SGD, similar to gradient sparsification [78, 55] , the convergence can still be guaranteed for unbiased estimation of gradients with increased variance. The convergence behavior can be further improved by reducing the variance with appropriately defined importance sampling strategy [14] . For layerwise importance sampled Adam, theoretical results in [79, 55] prove its convergence in convex objectives. If we denote f as the loss function and assume that the stochastic gradients are bounded, then based on [56] , we know that AdamW optimizing f aligns with Adam optimizing f with a scaled regularizer, which can be written asf reg (w) ≜ f (w) + 1 2 w ⊤ Sw,where S is a finite positive semidefinite diagonal matrix. Following existing convergence results of RBC-Adam (Corollary 1 in [79] ), we have the convergence guarantee of LISA in Theorem 1.Theorem 1 Let the loss function f be convex and smooth. If the algorithm runs in a bounded convex set and the stochastic gradients are bounded, the sequence {w t } T t=1 generated by LISA admits the following convergence rate:1 T T t=1 f reg (w t ) -f reg * ≤ O 1 √ T ,where f reg * denotes the optimum value of f reg .In our instruction following experiments in Appendix A.1 and A.2, we observe that LISA is much better than LoRA at memorization-centered tasks, such as Writing or depicting image details, while this gap is much smaller in reasoning-centered tasks like Code or Math. It is an intriguing observation since LISA emphasizes more on layer-wise width and restricts the depth of learned parameters, while LoRA focuses more on depth and restricts the representation space in each layer. It may suggest that width is crucial for memorization, while depth is important for reasoning, a similar phenomenon that echos the intuition of [80] . Based on the same intuition, it may be possible to combine the benefits of both and bring forth an even better PEFT method. 
conclusion: 
 In this paper, we propose Layerwise Importance Sampled AdamW (LISA), an optimization algorithm that randomly freezes layers of LLM based on a given probability. Inspired by observations of LoRA's skewed weight norm distribution, a simple and memory-efficient freezing paradigm is introduced for LLM training. This paradigm achieves significant performance improvements over LoRA on downstream fine-tuning tasks with various models, including LLaMA-2-70B. Further experiments on domain-specific training also demonstrate its effectiveness, showing LISA's huge potential as a promising alternative to LoRA for LLM training. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Yes, the research conducted in the paper conforms in every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: NA.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: NA. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: See Licenses section C. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: The major bottleneck of LISA is the same as LoRA, where during optimization, the forward pass still requires the model to be presented in the memory, leading to significant memory consumption. This limitation shall be compensated by approaches similar to QLoRA [11] , where we intend to conduct further experiments to verify its performance.In addition, as suggested by the theoretical intuition, the strategy of E+H+2L in Section 4.2 and E+H+4L in Section 4.4 may not be the optimal importance sampling strategy, given it still sampled intermediate layers in a uniformly random fashion. We anticipate the optimizer's efficiency will be further improved when considering data sources and model architecture in the importance sampling procedure.Stable Diffusion has emerged as a powerful approach for generating high-quality images by leveraging the diffusion process in a latent space [81] . This method involves diffusing the data distribution over time and iteratively refining the generated samples to enhance their realism and fidelity. The key innovation lies in operating within the latent space of a pre-trained autoencoder, significantly reducing computational complexity while maintaining high-quality outputs. The Latent Consistency Model (LCM) further improves the efficiency and performance of diffusion models in the latent space by incorporating a consistency loss, which penalizes deviations from the expected latent trajectories [82] . This ensures that the generated samples remain coherent and visually plausible throughout the diffusion steps.In our experiments, we evaluated the performance of LCM against other fine-tuning methods such as LoRA, and LISA using latent consistency distillation to distill stable diffusion v1.5 and v2.1 model. We utilized the official code * and parameters for both LISA and LoRA from the Diffusers library to ensure consistency and comparability of results. Adjustments were made only to the batch size and accumulation steps to achieve a balanced trade-off between computational efficiency and performance. Figure 5 is the generated image comparison. Observations reveal that LISA can generate higherquality images in fewer inference steps. The images generated using LISA display more intricate details and sharper clarity, particularly evident in the distinct facial features and environment textures.In contrast, the LoRA-generated images offer a softer, more blended aesthetic with a dream-like quality, emphasizing smooth transitions over precise detail. The prompts for images in figure 5 in the left-to-right are given below• Self-portrait oil painting, a beautiful cyborg with golden hair, 8k.• Astronaut in a jungle, cold color palette, muted colors, detailed, 8k.• A photo of a beautiful mountain with a realistic sunset and blue lake, a highly detailed masterpiece.Table 8 offers a comprehensive evaluation of three fine-tuning methods-Full Parameter Fine-Tuning (FT), Low-Rank Adaptation (LoRA), Gradient Low-Rank Projection(GaLore), and Layerwise Importance Sampling AdamW (LISA)-across a diverse set of tasks including Writing, Roleplay, Reasoning, Math, Extraction, STEM, and Humanities within the MT-Bench benchmark. The results demonstrate LISA's superior performance, which surpasses LoRA, GaLore, and full parameter tuning in most settings. Notably, LISA consistently outperforms LoRA and full parameter tuning in domains such as Writing, STEM, and Humanities. This implies that LISA can benefit memorization tasks, while LoRA partially favors reasoning tasks. Table 9 provides detailed MT-Bench scores for the LLaMA-2-70B model discussed in Section 4.4, demonstrating LISA's superior performance over LoRA in all aspects under large-scale training scenarios. Furthermore, in Figure 6 , we observe that LISA consistently exhibits on-par or faster convergence speed than LoRA across different models, which provides strong evidence for LISA's superiority in practice. The aforementioned results show that Vanilla LLaMA-2-70B excels in Writing, but full-parameter fine-tuning led to a decline in these areas, a phenomenon known as the "Alignment Tax" [1] . This tax highlights the trade-offs between performance and human alignment in instruction tuning. LISA, however, maintains strong performance across various domains with a lower "Alignment Tax". Different LISA activated layers in continual pretraining We conducted an ablation study on the LLaMA-2-7B model trained with the Alpaca-GPT4 dataset, setting the sampling period K = 13, so the number of samplings is exactly 10. The study explored different configurations of sampling layers γ including {E+H+2L, E+H+4L, E+H+8L}. Table 10 demonstrates that LISA with only E + H layers experiences a performance decline when the 0 sampling layers γ are omitted, highlighting the importance of these layers. To better understand the LISA and hyperparameter Sampling Layers γ, we conducted ablation experiments on Sampling Layers γ and learning rate η. The aim was to investigate the combined impact of these two variables on the LISA. Our experiments utilized the LLaMA-2-7B model, trained on the GSM8K dataset. We examined the effect of increasing the number of Sampling Layers γ while simultaneously decreasing the learning rate η. The Table 11 , indicates that a higher number of sampling layers can enhance the model's effectiveness, provided the learning rate is adjusted appropriately. Specifically, the optimal performance is observed when the learning rate η is reduced in proportion to the increase in the number of sampling layers γ, demonstrating the delicate balance required between these two parameters to maximize LISA's efficacy.A.4.2 Sampling Period K This finding implies that for a 7B model trained on a 52K instruction conversation pair dataset, a sampling period of K = 13 is optimal for achieving the best loss curve and corresponding MT-Bench score radar graph.LLaMA-2-7B on Alpaca-GPT4 with update step per sampling period K = 13, and sampling layers γ = 2, run three times with different random layer pick. Figure 10 shows that different random selections of layers slightly affect the training process but converge similarly. Despite initial fluctuations, the loss trends of three runs-distinguished by blue, green, and red lines-demonstrate that the model consistently reaches a stable state, underscoring the robustness of the training against the randomness in layer selection. Additionally, we also conduct experiments that analyze the impact of fixing randomly selected layers during training. The suggestion highlighted a need to evaluate the model's stability and performance under such conditions.To address this, we conducted further experiments using the LLaMA-2-7B model, maintaining identical hyperparameters and datasets as in our paper, and repeated the experiments three times to reduce variability from the random selection process.The results are presented as table 12 , with 'LISA-fix' denoting the experiments with randomly fixed layers and the appended number indicating the different selected seeds. The results are presented as table 12 , with 'LISA-fix' denoting the experiments with randomly fixed layers and the appended number indicating the different selected seeds.A.5 Performance with Early Exiting According to previous works [83, 84] , the early exiting strategy in LLMs is effective. We are interested in investigating whether the LISA algorithm will have a different impact when combined with the early exiting strategy.To explore this, we conducted a series of experiments using the LLaMA-2-7B model, focusing on various training methods and early exit points. The early exiting method is DoLa [83] .We conducted all of our experiments using the LLaMA-2-7B model, selecting layers [0, 8, 16, 32] as early exit points for evaluation under four different training conditions: Vanilla (baseline), Vanilla with DoLa, Full Parameter Fine-Tuning (FT) with DoLa, and LISA with DoLa. The model used is the same one trained on the Alpaca-GPT4 dataset, as reported in section 4.2. This comprehensive approach enabled a detailed comparison of the model's performance and layer representation capabilities under various training methodologies. Due to constraints, our experiments were conducted on a subset of 100 questions from the GSM8K test set, employing the same prompt as in the DoLa paper.From the table 13, it can be seen that the LISA algorithm does not negatively affect the representation or performance of some layers of the model; instead, it contributes to some improvements in effectiveness.Besides the training loss, we also care about how LISA performs on the validation dataset. So, we split the Alpaca-GPT4 dataset into train and validation sets, the ratio is 9 : 1, and ensure there is no data overlap between these two sets. Then we use the same setting in the sec 4.2, training the LLaMA-2-7B on the Alpaca-GPT4 dataset with full parameter training (FT), LoRA, GaLore, and LISA. As Figure 11 shows, the trend in validation loss mirrors that observed in training loss, with LISA exhibiting no signs of overfitting. This consistency underlines LISA's robustness in maintaining performance across different dataset splits. -80 GSM8K [73] 7,473 1,319 MMLU [68] -14,079 AGIEVAL [69] -9316 WINOGRANDE [70] -44,000 PUBMEDQA [74] 211,269 1,000 OPENWEBMATH [72] 6.3M -We conduct further weight norm experiments on Mistral-7 B to support our motivation that the bottom and top layers have a more significant impact on the output. Figure 12 provides similar observations as LLaMA-2-7B, where the bottom layer has a larger weight norm than other layers. Our study explored a range of learning rates from 5 × 10 -6 to 3 × 10 -4 , applying this spectrum to Full Parameter Training, LoRA, and LISA methods. For LoRA, we adjusted the rank r to either 128 or 256 to vary the number of trainable parameters, applying LoRA across all linear layers. Regarding the number of sampling layers γ, our selections were guided by GPU memory considerations as reported in LoRA studies [9] ; For the LISA algorithm, we selected γ = 2, and for experiments involving the 70B model, we opted for γ = 4. The sampling period (K), defined as the number of update steps per sampling interval, ranges from 1 to 50. This range was influenced by variables such as the size of the dataset, the batch size, and the number of training steps. To manage this effectively, we partitioned the entire training dataset into K segments, thereby enabling precise regulation of the training steps within each sampling period.We extracted a high-quality subset from OpenWebMath [72] , using the 'Math_score' attribute from the metadata as the metric for high-quality instances. The 'Math_Score' represents the probability that a document is mathematical, and we set the threshold at 0.95. Finally, the number of tokens for this high-quality subset is 1.5 billion.We commenced our study with a grid search covering (i) learning rate, (ii) number of sampling layers γ, and (iii) sampling period K. Noting the effective performance of the LoRA method, we set the rank value to r = 128 or r = 256.The optimal learning rate was explored within the range {5 × 10 -6 , 10 -5 , 5 × 10 -5 , 6 × 10 -4 , 3 × 10 -4 }, applicable to full parameter training, LoRA, and LISA. For GaLore, we adhered to the official Transformers implementation * , utilizing default parameters, with the learning rate matching that of the full parameter training.Regarding the number of sampling layers γ, in alignment with Table 1 , we selected values that matched or were lower than LoRA's GPU memory cost. Consequently, γ = 2 was predominantly used in the LISA experiments, while γ = 4 was chosen for the 70B model experiments.For the sampling period K, we examined values within 1, 3, 5, 10, 50, 80, aiming to maintain the model's update steps within a range of 10 to 50 per sampling period. This selection was informed by dataset size, batch size, and total training steps.The comprehensive results of our hyperparameter search, detailing the optimal values for each configuration, are presented in Table 16 .For instruction following and domain-specific fine-tuning tasks, all the datasets, including Alpaca [61] , GSM8k [73] , MMLU [68] , AGIEval [69] and PubMedQA [74] are released under MIT license.WinoGrande [70] and MT-Bench [67] are under Apache-2.0 license. For GPT-4, the generated dataset is only for research purposes, which shall not violate its terms of use.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.* https://github.com/huggingface/diffusers/tree/main/examples/consistency_distillation* https://github.com/OptimalScale/LMFlow* https://huggingface.co/blog/galore 


Decision: The paper proposes a new PEFT method called LISA (Layerwise Importance Sampled AdamW). The authors observed that weight norms are skewed while fine-tuning the models. Inspired by the observation, the authors randomly freeze most middle layers during the optimization. The authors show that the proposed method uses similar or less memory than LoRA  while outperforming LoRA or full parameter fine-tuning in some cases.

Reasons to accept: All three reviewers have a consensus that the paper is well-organized and well-written. Also, they have a consensus that the problem to solve in the paper is important. Reviewer Rt8C and K468 mentioned that the experimental results supported the authors' claim well. Reviewer K468 had a question why E + H layers are handled differently, and the authors provided additional experiments why they are important. Reviewer kABp had concerns about novelty, unfair comparison and missing ablations. The authors clarified they were misunderstanding and provided additional information. Reviewer kABp increased the score based on the rebuttal.

Based on all those, the paper provides a more effective and more efficient way to fine-tune LLMs with supporting experiments which would be beneficial to the research community to enhancing the fine-tuning methods.

Reviews: review: paper_summary: paper_summary: This paper proposes a new optimization algorithm called Layerwise Importance Sampled AdamW (LISA) for large language model (LLM) fine-tuning. The authors observe the skewed weight norms across different layers in the Low-Rank Adaptation (LoRA) method and use this observation to develop LISA, which randomly freezes certain layers during optimization. Experimental results show that LISA outperforms LoRA and full parameter training in various downstream fine-tuning tasks, demonstrating its effectiveness and memory efficiency.
strength_weakness: strength_weakness: (1) I am curious whether the authors will open-source their implementation? Since efficient training techniques are important to this field, it would be beneficial to have an easy-to-use implementation. In addition, it would be helpful if the authors provide more information on their hyperparameter tuning strategy in the experiments, like how do you choose the best hyperparameters?

(2) The some parts of the writing and layout needs improvement. For example, there are too many capitalized letters like line 6 "Parameter Efficient Fine-Tuning". I do not see any reason to capitalize every words. In addition, the figures should be re-organized like Figure 2.

(3) The authors seem to define 65B as a boundary for small language models and large language model (see line 56). Just curious, are there any justification or relevant claims? Since there are more and more small language model research nowadays and they are mainly focusing on 1B-7B, it would be interesting to have a clear definition in our community.
questions: questions: None
limitations: limitations: The limitation is fine with me.
paper_summary: paper_summary: This paper proposes a lighter alternative to LoRA, LISA, based on using importance sampling to periodically choose a subset of layers to optimize. It is motivated by observations on the norm of parameter updates made during training with LoRA, compared to full parameter fine-tuning. Experiments are made comparing LISA with LoRA and full parameter fine-tuning on memory efficiency, and moderate/large fine-tuning and continual pre-training. Supplementary experiments are made on the hyperparameters of LISA and training sensitivity. Results show that LISA uses less memory than LoRA and achieves better results on most tasks. The paper also presents a discussion including theoretical properties of LISA.
strength_weakness: strength_weakness: - The main issue I have with this paper is that the approach presented does not seem to actually use Importance sampling: the weights of the $E$ and $H$ layers are fixed to $1.0$, and the intermediate layers have uniform weights. Besides, the presentation of the distribution and rates used in Section 3.2 is rather unclear.
questions: questions: -  While the choice of that sampling distribution seems motivated by observation on LoRA, there seems to be shortcuts taken, particularly on fixing $E$ and $H$. Is there anything more to motivate this ? Did you try more extensive experiments with only $E + H$ ? 
- Can you relate your approach with those mentioned in Section 2.2, focusing on layer-selection ?
limitations: limitations: - The limitations have been addressed.
paper_summary: paper_summary: This paper proposes an parameter-efficient finetuning (PEFT) method, LISA, that only finetunes some sampled layers while freezing the rest  for some certain iterations, and resample the finetuned layers later. 

To validate LISA's effectiveness, the authors benchmark it on various tasks, like instruction-following, medical QA, math problems, across various sizes of LLMs. The results show that LISA shares a similar GPU memory consumption as LoRA, 1.5x speedup than LoRA for training, better or comparable to LoRA and Full Finetuning for various tasks.

The main contributions of this paper are:
1. The proposed method is simple, easy to be applied;
2. The experiments are thorough and consistent to support LISA's effectiveness.
strength_weakness: strength_weakness: 1. Lack of novelty. LISA is very similar to the method proposed in the paper [46] in the references. [46] proposes to pretrain a model layer-by-layer, while LISA shuffles the training order and finetune some for a period. Such novelty is not enough. In addition, the authors don't apply their finding, i.e. skewed weight-norm distribution across layers, to design a better sampling method. Instead, they apply uniform sampling, which makes the finding less relevant. BTW, the finding is very similar to a previous work [2] (Figure 4).

2. Lack of experimental details about the optimizer states, and unfair comparison. Compared to LoRA, LISA basically finetunes the whole model, which means the optimizer states stay the same as Full Finetuning. Since the oprimizer states occupy as twice memory as the model's, I assume LISA offload them to CPU. This part is not explicitely stated, nor in the limitation section. When comparing the training time, the authors only compare the forward and backward time (Figure 4), and ignore the latency introduced by the offloaded optimizer states, which is unfair since LoRA doesn't need to offload. 

3. Lack of important ablation study. The benefit of LISA is from the training of the whole model in a layer-by-layer manner. Actually, we can also apply LISA to LoRA. I.e. we can use a very large size LoRA and only finetune the sampled LoRAs. In this way, LoRA has a larger number of trainable parameters, which might close the gap between LISA and LoRA.

4. Unintuitive results without further analysis. The most surprising result is that LISA outperforms full finetuning quite often, which is against our intuition. It also casts a doubt: how do you choose the hyper-parameters (like Table 15)? And what is the error bar for the main results.


[1] A fast learning algorithm for deep, Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh
[2] Parameter-Efficient Fine-Tuning without Introducing New Latency, Baohao Liao, Yan Meng, Christof Monz
questions: questions: See weakness.
limitations: limitations: The authors discuss in the Limitation section.

Comments: title: Thanks very much for your positive feedback!
Thank you so much for your valuable feedback!
Thanks for your valuable feedback!
Thank you very much for the valuable feedback!
comment: Dear Reviewer kABp,

We sincerely thank you for your positive feedback and the time you dedicated to reviewing our rebuttal. It brings us great joy to learn that our response has addressed your concerns and contributed to increasing the score from 4 to 5.

As the score is still borderline, we are wondering if there are any major concerns regarding our current revision. It would be our great pleasure to provide further clarifications and results to address any additional doubts.

Your suggestions really help a lot to improve our work and make the justification of our method more complete. We also greatly appreciate your recognition of the simplicity of our algorithm, the quality of our paper, and the contribution it makes.

Once again, we would like to express our appreciation for your valuable comments during the reviewing process.

Best regards,

Authors.
Thank you for the clarification and new eperiments. I'm willing to increase my score.
Dear Reviewer K468,

We sincerely appreciate your time in reviewing our rebuttal and examining our new results. We are delighted to learn that our response successfully addressed your concerns and contributed to an increase in the _Soundness_ rating. Your constructive advice is really helpful in enhancing the completeness and quality of our paper!

Once again, we sincerely appreciate your valuable feedback and consideration.

Best regards,

Authors
I would like to thank the authors for their rebuttal, the clarifications and the supplementary results ! I have updated my *Soundness* rating, and maintain my global rating recommanding acceptance.
Dear Reviewers,

Thank you for your valuable feedback and for taking the time to review our submission. We appreciate your insights and have addressed your comments in our response. If there are any additional concerns or questions, please do not hesitate to reach out. We are more than happy to provide further clarification or discuss any points in more detail.

Best regards,

Authors
Dear Reviewer kABp,

Thank you very much for your valuable feedback! We truly hope that our new results can help clarify the concerns.

We appreciate the time you have spent on reviewing our paper and providing us with the constructive comments. Please let us know if there are any concerns remaining. If you find our response to have addressed your concerns, it would mean very much to us if you could consider raising the score.