LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning
Authors: Rui Pan
Xiang Liu
Shizhe Diao
Renjie Pi
Jipeng Zhang
Chi Han
Tong Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: memory efficient training
large language models
parameter efficient fine-tuning
Primary area: optimization_for_deep_networks

Abstract: The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory consumption has become a significant roadblock to large-scale training. For instance, a 7B model typically requires at least 60 GB of GPU memory with full parameter training, which presents challenges for researchers without access to high-resource environments. Parameter Efficient Fine-Tuning techniques such as Low-Rank Adaptation (LoRA) have been proposed to alleviate this problem. However, in most large-scale fine-tuning settings, their performance does not reach the level of full parameter training because they confine the parameter search to a low-rank subspace. Attempting to complement this deficiency, we investigate the layerwise properties of LoRA on fine-tuning tasks and observe an unexpected but consistent skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in a wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled AdamW (LISA), a promising alternative for LoRA, which applies the idea of importance sampling to different layers in LLMs and randomly freeze most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while achieving on-par or better performance in MMLU, AGIEval and WinoGrande. On large models, specifically LLaMA-2-70B, LISA surpasses LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating its effectiveness across different domains.

LISA: Layerwise Importance Sampling for Memory-Effcient Large Language Model Fine-Tuning
Rui Pan
Xiang Liu Shizhe Diao*, Renjie Pi Jipeng Zhang
Chi Han Tong Zhang
University of Illinois Urbana-Champaign
The Hong Kong University of Science and TechnoogyyGuannghhuu
NVIDIA The Hong Kong University Science and Technology
(ruip4 chihan3, tozhang)@illinois. edu
xliu888@connect. hwust-gz.zz gestt-gzeeduuu edu. (sdiaoaa, rpi, Behhangrruust... hk
Abstract
The machine learning community has witnessed impressive advancements since large language models (LLMs) first appeared. Yet, their massive memory con- sumption has become significant roadblock to large-scale training. For instance, a 7B model typically requires at least 60 GB GPU memory with full parameter training, which presents challenges for researchers without access high-resource environments. Parameter efficient fine-tuning techniques such as Low-Rank Adap- tation (LoRA) have been proposed to alleviate this problem. However, in most large-scale fine-tuni tieettunng settings, their performance does not reach the level of full parameter training because they confine the parameter search low-rank sub- space. Attempting to complement this deficiency, we investigate the layerwise properties of LoRA fine-tuning tasks and observe unexpected but consistent skewness of weight norms across different layers. Utilizing this key observation, a surprisingly simple training strategy is discovered, which outperforms both LoRA and full parameter training in wide range of settings with memory costs as low as LoRA. We name it Layerwise Importance Sampled Adam (LISA), promising alternative for LoRA, which applies the idea of importance sampling to differ- ent layers LLMs and randomly freeze most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 10%-35% in terms of MT-Bench score while achieving on-par better performance MMLU, AGIEval and WinoGrande. On large models, specifically LLaMA-2-70BB LISA surpasses LoRA on MT-Bench, GSM8K, and PubMedQA, demonstrating effectiveness across different domains.
Introduction
Large language models (LLMs) like ChatGPT excel in tasks such as writing documents, generating complex code, answering questions, and conducting human-like conversations []]. With LLMs being increasingly applied in diverse task domains, domain-spectee fine-tuning has emerged as critical strategy enhance their downstream capabilities 2 5) Nevertheless, these methods are typically time-intensive and consume substantial computational resources, posing significant challenges to the development large-scale models [6] For example, continual pre-training typically
Equall Contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

requires several weeks even with multiple 80 GB GPUs. To reduce costs, Parameter-Effiee Fine-Tuning (PEFT) techniques have been proposed to minimize the number of trainable parameters.
These techniques include adapter weights 7 prompt weights (8, and LoRA (9. Among these, LoRA stands out as one of the most widely adopted due to its unique ability merge the adaptor back into the base model parameters, significantly enhancing efficiency. However, LoRA's su- perior performance in fine- tine-tuning tasks has yet to reach point that universally surpasses full parameter tine-tuning all settings [10]]] In particular, has been observed that LoRA tends to falter on large-scale datasets during continual pre-training 12], which raises doubts about the effectiveness of LoRA under those circumstances. We at- tribute this to LoRA's much fewer trainable parameters compared to the base model, which limits the representa- tion power LoRA training.
1.4
LoRA
LISA
FT
GäLore
1.1-
20
40
60
80
100
120
step
Figure Training loss of LLaMA-2- 7B on Alpaca GPT-4.
To overcome this shortcoming, we delve into LoRA's training statistics in each layer, aspiring to bridge the difference between LoRA and full-parameter fine-tuning. Surprisingly, we discover that LoRA's layerwise weight norms have an uncommonly skewed distribution, where the bottom layer and/or the top layer occupy the majority of weights during the update. contrast, the other self-attention layers only account for small amount, which means different layers have different importance when updating. This key observation inspires us to "sample" different layers by their importance, which matches the idea of importance sampling [13 14].
As natural consequence, this strategy brings forth our Layerwise Importance Sampled Adam (LISA) algorithm, where by selectively updating only essential LLM layers and leaving others untouched, LISA enables training large-scale language models (2 65B parameters) with less or similar memory consumption as LoRA. Furthermore, fine-tuned on downstream tasks, LISA outperformed both LoRA and conventional full- pal-mearretter fine-tuning approaches by a large margin, indicating the large potential of LISA as a promising alternative to LoRA.
We summarize key contributions follows,
We discover the phenomenon of skewed weight-norm distribution across layers in LoRA, which implies the varied importance of different layers in large-scale LLM training.
We propose the Layerwise Importance Sampled AdamW (LISA), simple optimization method capable of scaling up to over 70B L.LMs with less or similar memory cost as LoRA.
We demonstrate LISA's effectiveness in fine-tuning tasks for modern LLMs, where it outperforms LoRA by 10%-35% in MT-Bench and achieves better performance in multiple benchmarks. In addition, LISA exhibits much better convergence behaviors than LoRA. LISA even outperforms full parameters training under certain settings. Similar performance gain is observed across different sized models (7B-70B) and tasks, including instruction following, medical QA, and math problems.
Related Work
2.1
Large Language Models
In the realm of natural language processing (NLP), the Transformer architecture has been a revolu- tionary technique, initially known for its effectiveness in machine translation tasks [15]. With the inception of models like BERT [16] and GPT-2 [17], the approach shifted towards pre-training on extensive corpora, which led to significant performance enhancements in downstream fine-tuning
25| 26 27.. However, the growing number of parameters in these models results in huge GPU memory consumption, rendering the fine-tuning of large scale models (2 65B) infeasible under low resource scenarios. This has prompted a shift towards more efficient training of LLMs.
tasks 18 20 21 22 24

---

2.2 Parameter-Effieient Fine-Tuning
Parameter-efficient fine-tuning (PEFT) methods adapt pre-trained models by fine-tuning only
subset of parameters. In general, PEFT methods can be grouped into three classes: 1) Prompt
Learning methods [8]
28
29
30
32
331, 2) Adapter methods
34
35
36
37
38] and
Prompt learning methods emphasize optimizing the input token or input embedding with frozen model parameters, which generally has the least training cost among all three types. Adapter methods normally introduce an auxiliary module with much fewer parameters than the original model, and updates are only applied to the adapter module during training. Compared with them, selective methods are more closely related to LISA, which focuses on optimizing a fraction of the model's s parameters without appending extra modules. Recent advances in this domain have introduced several notable techniques through layer freezing. AutoFreeze [39] offers an adaptive mechanism to identify layers for freezing automatically and accelerates the training process. FreezeOut [42 progressively freezes intermediate layers, significantly reducing training time without notably affecting accuracy. The SmaruRZ [40] framework utilizes an attention-based predictor for layer selection, substantially cutting computation and training time while maintaining accuracy. However, none of these
3) Selective methods 39
39
40
strategies has been widely adopted in the context of Large Language Models due to their inherent complexity non-compatibility with modern memory reduction techniques [43 27 for LLMs.
2.3 Low-Rank Adaptation (LoRA)
In contrast, the LLow-Rank Adaptation (LoRA) technique is much more prevalent in common LLM training [9]. LoRA reduces the number of trainable parameters by employing low-rank matrices, thereby lessening the computational burden and memory cost. One key strength of LoRA is its compatibility with models featuring linear layers, where the decomposed low-rank matrices can be merged back into the original model. This allows for efficient deployment without changing the model architecture. As a result, LoRA can be seamlessly combined with other techniques, such as quantization [11] or Mixture of Experts (45). Despite these advantages, LoRA's performance is not universally comparable with full parameter fine-tuning. There have been tasks in [10] that LoRA performs much worse than full parameter training on. This phenomenon is especially evident in large-scale pre-training settings [12), where to the best of our knowledge, only full parameter training was adopted for successful open-source LLMs
26
2.4 Large-scale Optimization Algorithms
In addition to approaches that change model architectures, there have also been efforts to improve the efficiency of optimization algorithms for LLMs. One such approach is layerwise optimization, a concept with roots extending back several decades. Notably, (48) introduced an effective layer-by- layer pre-training method for Deep Belief Networks (DBN), demonstrating the benefits of sequential layer optimization. This idea was expanded by researchers like [0], who illustrated the advantages of a greedy, unsupervised approach to pre-training each layer of deep networks. In the context of large batch training, [51]5] developed LARS and LAMB to improve generalization and mitigate the performance declines associated with large batch sizes. Despite these innovations, Adam [53] [44]
55 27 and Adam W (56] continue to be the predominant optimization methods used in most LLM settings.
Recently, other attempts have also been made to reduce the training cost of L.Mss. For example, McZO (57) adopted zeroth order optimization, bringing significant memory savings during training. However, it also incurred a considerable performance drop in multiple benchmarks, particularly in complex fine-tuning scenarios. Regarding acceleration, Sophia (58) incorporates clipped second-order information into the optimization, obtaining non-trivial speedup on LLM training. The significant downsides are its intrinsic complexity of Hessian estimation and unverified empirical performance in large-size models (e.g., 2 65B). In parallel to our work, [59] proposed GaLore, memory efficieen training strategy that reduces memory cost by projecting gradients into a low-rank compact space. Yet the performance has still not surpassed full- aall-parrmeter training in fine-tuning settings. To sum up, LoRA-variant methods 9 59 with AdamW (56) is still the dominant paradigm for large-size LLM fine-tuning, the performance of which still demands further improvements.
3

---

3 Method
3.1 Motivation
To understand how LoRA achieves effective training with only few parameters, we conducted empirical studies on multiple models, especially observing the weight norms across various layers. We fine-tune it on the Alpaca-GPT4 dataset [60]. During the training, meticulously recorded the mean weight norms of each layer E at every step t after updates, i.e.
T Σ [=1
w(1) & mean-weight-norm(() = T
0
Figure 2 presents these findings, with the x-axis representing the layer id, from embedding weights to the final layer, and the y-axis quantifying the weight norm. The visualization reveals one key trend:
The embedding layer or the language modeling (LM) head layer exhibits significantly larger weight norms than intermediary layers in LoRA, often by factor of hundreds. This phenomenon, however, was not salient under full parameeerr training settings.
Weight Norm of Each Layer For GPT2
Weight Norm Layer For LLaMA-2-7B
10
OPTENILOO
Wage Nom
Weight Nomt
10"
to²
posioo exiouallg
eccledling)
To
6.0
65
fg"
Luek NN with DRA
Rypens
lapert C
Inpar 10
uppor. 20
layort 20
Layara
Figure 2: Layer- wise weight norms during training of GPT2 and LLaMA-2-76 Model with LoRA and Full Parameters training.
This observation indicates that the update emphasis of LoRA and full parameter training differ significantly, which can be attributed to the difference their learned knowledge. For example, in embedding layers, tokens with similar meanings, i.e., synonyms, can be projected into the same embedding space and converted to similar embeddings. LoRA may capture this similarity in language and "group" them in the low-dimension space, allowing frequent features of language meanings be promptly identified and optimized. The price is LoRA's limited representation power restricted by its intrinsic low-rank space, as we can see from the comparison with LISA in image generation tasks (Appendix I).)) where LoRA memorizes and learns details much slower than LISA. Other possible explanations can also justify this phenomenon. Despite various interpretations of this observation, one fact remains clear: LoRA values layerwise importance differently from full parameter tuning.
3.2 Layerwise Importance Sampled Adam W (LISA)
To exploit the discovery above, we aspire to simulate LoRA's updating pattern via sampling different layers to freeze. This way, ability and emulate its fast learning process. Intuitively, given the same global learning rates across layers, layers with small weight norms in LoRA should also have small sampling probabilities to unfreeze full-parameter settings so the expected learning rates across iterations can stay the same. This is exactly the idea of importance sampling [13] [4], where instead of applying layerwise different learning rates } in full-parameter settings to emulate LoRA's updates { we apply sampling and instead get the same expected parameter update
avoid LoRA's inherent deficiency of limited low-rank representation
(.(2)
(())
(6) 17()) .p
1/2
(())
(()

---

This gives rise to our Layerwise Importance Sampling AdamW method, as illustrated in Algorithm In practice, since all layers except the bottom and top layer have small weight norms LoRA. we adopt {pe] 2 (1.0, 7/Nz, 7/NL......... - 7/Nz, 1.0} in practice, where 7 controls the expected number of unfreeze layers during optimization, and the embedding layer E and head layer H remain active. Intuitively, y serves compensation factor to bridge the difference between LoRA and full parameter tuning, letting LISA emulate similar layerwise update pattern LoRA. To further control the memory consumption in practical settings, we instead randomly sample y layers every time to upper-bound the maximum number of unfrozen layers during training.
Algorithm Layerwise Importance Sampling AdamW (LISA)
Require: number of layers NL, number of iterations T. sampling period K, number of sampled layers 7, initial learning rate 7/o
1: for 40 to T/K do
2: Freeze all layers except the embedding and language modeling head layer
3: Randomly sample intermediate layers to unfreeze
4: Run AdamW for K iterations with {n Stzck
p
5:
end for
Experimental Results
Table I: The chart illustrates peak GPU memory consumption for various model architectures and configurations, highlighting differences across models. The LISA configuration specifically labeled in the table: "E" denotes the embedding layer, "H" represents the language modeling head layer, and "2L" indicates two additional intermediate layers. *, Model parallelism is applied for the 70B model.
VANILLA
LORA RANK
LISA ACTIVATE LAYERS
MODEL
256
512
E+H
E+H+2L
E+H+4L
GPT2-SMALL
3.8G
3.3G
3.5G
3.7G
3.3G
3.3G
3.4G
TINYLLAMA
13G
7.9G
8.6G
10G
7.4G
8.0G
8.3G
MISTRAL-7B
59G
23G
26G
28G
21G
23G
24G
LLAMA-2
59G
23G
26G
28G
21G
23G
24G
LLAMA-2-700
00M
79G
0OM
0OM
71G
75G
79G
4.1 Memory Efficiency
We conducted peak GPU memory experiments demonstrate LISA's memory efficiency and show-
case its comparable or lower memory cost than LoRA.
Settings To reasonably estimate the memory cost, we randomly sample prompts from the Alpaca dataset (61) and limit the maximum out- put token length to 1024. We focus on two key hyperparameters: LoRA's rank and LISA's number of activation layers. For other hyper- parameters, mini-batch size of was consis- tently used across five LLMs from 120M to 70B parameters, deliberately excluding other GPU memory-saving techniques such as gra- dient checkpointing [62]. offloading [63), and flash attention [64] 65). All memory-effcciene, experiments are conducted on 4x NVIDIA Am- pere Architecture GPUs with 80G memory.
Memory Cost Different Optimization Methods
Weight Memory
Aclivation Memory
Gradient Merory
Opt mizer Memory
Memary Cost (GB)
40
24CB
10-
LoRA
Methad
Figure 3: GPU memory consumption of LLaMA- 2-7B with different methods and batch size 1.
Upon examining Table T is evident that the LISA configuration, particularly when enhanced with both the embedding layer (E) and two additional layers (E+H+2L), demonstrates considerable reduction in GPU memory usage when fine-tuning the a. 2-BB model, as compared to the LoRA method. Specifically, the LISA E+H+2L configuration shows decrease to 75G of peak GPU memory from the 79G required by the
Results
5

---

LoRA Rank 128 configuration. This efficiency gain is not an isolated incident; a systematic memory usage decrease is observed across various model architectures, suggesting that LISA's method of activating layers is inherently more memory-efficient,
In Figure it is worth noticing that the memory reduction in LISA allows LLaMA-2-7B to be trained on a Single RTX4090 (24GB) GPU, which makes high-quality fine-tuning affordable even on a laptop computer. In particular, LISA requires much less activation memory consumption than LoRA since does not introduce additional parameters brought by the adaptor. LISA's activation memory is even slightly less than full parameter training since pytorch [66] with deepspeed 44 allows deletion of redundant activations before backpropagation.
On top of that, reduction in memory foot- print from LISA also leads to an acceleration in speed. As shown in Figure [4] LISA provides almost 2.9x speedup when compared with full- parameter training, and 1.5x speedup against LoRA, partially due to the removal of adaptor structures. is worth noticing that the reduc- tion of memory footprint in both LoRA and LISA leads significant acceleration of for- ward propagation, emphasizing the importance of memory- training.
Forward and Backward Time for Different Methods
x1.6
Forward Time
2.5
Backward Time
Time Derandal
X2.5
4.2 Moderate Scale Fine- Tine-Tuning
Mathad
Figure 4: Single- time cost of LLaMA-2-
LISA can achieve this significant memory sav- ing while still obtaining competitive perfor- mance under the fine-tuning setting.
7B with different methods and batch size 1.
Table 2: Results of different methods on MMLU, AGIEval, and WinoGrande, measured by accuracy.
MODEL
METHOD
MMLU (5-SHOT)
AGIEvAL (3-SHOT)
WINOGRANDE (5-SHOT)
VANILLA
25.50
19.55
59.91
LORA
25.81 0.07
19.82 0.11
61.33 + 0.09
TINYLLAMA
GALORE
25.21
21.19 0.07
61.09 # 0.12
LISA
26.02 0.13 0.13
21.71 # 0.09
61.48 : 0.08
FT
25.62
21.28 # 0.07
62.12 # 0.15
VANILLA
60.12
26.79
79.24
LORA
61.78 2 0.09
27.56 + 0.07
78.85 0.11
MISTRAL-7B
GALORE
57.87 + 0.08
26.23 0.05
75.85 1 0.13
LISA
62.09 0.10
29.76 0.09
78.93 # 0.08
0..13
28.07 0.12
78.85 + 0.12
VANILLA
45.87
25.69
74.11
LoRA
45.50 00007
24.73 +0.04
74.7440.00
ILAMA-2-7B
GALORE
45.56 0.05
24.39 # 0.11
73.32 + 0.12
LISA
46.21 0.12
26.06 0.08
75.30 # 0.11
FT
45.66 0.09
27.02 0.10
75.06 + 0.13
Settings To demonstrate the superiority of LISA over LoRA, we evaluate them on the instruction- following fine-tuning task with the Alpaca GPT-4 dataset [61], which consists of 52k conversation pairs generated by GPT-4 (5). The effectiveness of fine-tuning was evaluated on multiple benchmarks: MT-Bench [67] features 80 high-quality, multi-turn questions designed to assess LLMs on multiple aspects; MMLU (68) includes total of 57 tasks with 14,079 questions covering broad spectrum benchmark for general abilities, comprising 9,316 instances; WinoGrande [70] is a large-scale dataset for commonsense reasoning, consisting of 44,000 instances designed to challenge models' understanding of the context and commonsense knowledge.
of world knowledge; AGIEval 69 serves as a human-centir
In our experiments, we assessed three baseline models: TinyL [47), Mistral-7B 46,, and LLaMA-2-7B (23). These models, varying in size ranging from IB to 7B parameters, provide

---

diverse representation of decoder-only models. For cyrper-aaramtery we adopt a rank of 128 for LoRA and E+H+2L for LISA in this section, with full details available Appendix B
Results T and present a detailed compari- son on moderate-scale LLMs. The baselines include Full-parameter Training (FT), Low-Rank Adaptation (LoRA) and Gradient Low-Rank Projection (Ga- Lore) [59]. The results demonstrate that LISA consis- tently outperforms other fine-tuning methods most evaluation tracks, indicating its robustness and effec- tiveness across diverse tasks and model architectures. LISA is particularly effective in instruction follow- ing tasks, where a large gap is observed when com- pared with other baseline methods. LISA even out- performs Full-paramett Training, suggesting that an implicit regularization effect present when the num- ber of unfrozen layers is restricted, which is similar to dropout [71]. According more results in stable diffusion and detailed MT-Bench scores, we found that LISA outperforms LoRA mostly in memorization tasks, such as depicting high-resolution image details in image generation, Writing Humanities tasks
Table 3: Different methods on MT-Bench.
MODEL
METHOD
MT- WTT-BENC t
VANILLA
1.25
LORA
1.90 - 0.14
TINYLLAMA
GALORE
2,61 = 0.17
LISA
2.57 = 0.25
FT
2.21 0.16
VANILLA
4.32
LORA
4.41 0.09
MISTRAL-7B
GALORE
4.36
LISA
4.85
FT
4.64
VANILLA
3.29
LORA
4.45 + 0.15
-LAMA-2-7BB
GALORE
4.63 0.09
LISA
4.94 0.14
FT
4.75 0.16
in instruction following. This implies that LISA's performance improvement may majorly come from the ability memorize long-tailed patterns, while LoRA is better at multi-hop reasoning with limited knowledge. For more details, please refer to Appendix A. and A.2
4.3 Moderate Scale Continual Pre-training
Continual pre-training is crucial for enabling models to adapt to new data and domains. To evaluate LISA's efficacy in the continual pre-training scenario, we experiment on the mathematics domain in comparison with Full- parameter Training.
Table 4: Comparison Moderate Scale Model Continual Pre-training OpenWebMath Dataset.
MODEL
METHOD
GSM8K
MEM. 4
VANILLA
2.26
TINYLLAMA
LISA
3.56
8G
Settings We adopt the mathematics corpus OpenWebMath [72) for constructing the con- tinual pre-training dataset. Specifically, we tracted a high-quality subset from which con- tains 1.5 billion tokens. Full details are
FT
3.26
13G
VANILLA
14.40
LLAMA-2-7B
LISA
22.21
26G
FT
22.21
59G
plained in Appendix B.2 After continual pre- trainig, we then apply the same fine-tuning procedure the GSM8K comprises 7473 instances.
73
training set, which
better performance than full-parameter training with much less memory consumption. Specifically, LISA requires only half of the memory cost compared to full-parameter training. This indicates better balance between computational efficiency and model performance is achieved by LISA. According to our experience, reducing the number of unfrozen layers half the original size leads to no performance during continual pretraining, while requiring much less memory consumption.
Results Table shows that LISA is capable of achieving on-par
even better
4.4 Large Scale Fine- Tunn-Tunnng
To further demonstrate LISA's scalability on large-sized LLMs, we conduct additional fine-tuning experiments on LLaMA-2-70B [23].
Settings On top of the aforementioned instraction-folloviinn tasks in Section 4.2 we use extra domain-specific fine-tuning tasks on mathematics and medical QA benchmarks. The GSM8K dataset [73]. comprising 7473 training instances and 1319 test instances, is used for the mathematics

---

domain. For the medical domain, we select the PubMedQA dataset 774] which includes 211.3K
artificially generated QA training instances and IK test instances.
Evaluation on the PubMedQA dataset [74 is conducted in 5-shot prompt setting, while the GSMSK dataset 773 assessment was con- ducted using Chain-of-Thought (CoT) prompt- ing, following recent studies [75] 16,, [77. Re- garding hyperparameters, as detailed in the section 4.1 we utilize the rank 256 for LoRA
Table 5: Different methods on MT-Bench, GSM8K, and PubMedQA score for LLaMA-2-70B.
METHOD
MT WENCHENHH
GSM8K1
PUBMEDQAT
VANILLA
5.19
54.8
83.0
LORA
6.10
59.4
90.8
and the configuration E+H+4L for LISA. Fur- ther information is available in Appendix B
LISA
6.72
61.1
91.6
FT
6.25
67.1
90.8
Results
As shown in Table 5 LISA consis-
tently produces better or on-par performance when compared with LoRA. Furthermore, LISA again surpasses full fullmperaetter training in Instruction-tuning tasks, providing strong evidence to support LISA's scalability under large-scale training scenarios. More results are available in Appendix A.2
4.5
Ablation Studies
Hyperparameters LISA The two key hy- perparameters of LISA are the number of sam- pling layers y and sampling period K. To obtain intuitive and empirical guidance of those hyper- parameter choices, we conduct ablation studies using TinyLlama + and LLaMA-2-7B [23 models with the Aipaca-GPTT dataset. The con- figurations for 7 such as E+H+2L, E+H+8L, were denoted as = 2 and 7 = 8. As for the sampling period K T/n, T 122 represent- ing the maximum training step within our exper- imental framework. The findings, presented in Table 6 reveal that both and K markedly af- feet the LISA algorithm's performance. Specif- ically, higher y value increases the quantity of trainable parameters, albeit with higher memory costs. On the other hand, an optimal K value fa- cilitates more frequent layer switching, thereby improving performance to a certain threshold, beyond which the performance may deteriorate. Generally, the rule of thumb is: More sampling layers and higher sampling period lead bet- ter performance. For detailed examination of loss curves and MT-Bench results, refer to Appendix A.4
Table 6: Different LISA hyperparameters combina- tions. All settings adopt learning rate ?/o 10-5 Here stands for sampling layers, K stands for sampling period.
MT-BENCH SCORE
MODELS
7
K
[T/125]
2.44
/2/25
2.73
2
[T/5]
2.64
T
2.26
TINYLLAMA
[T/125]
2.59
T//25
2.81
8
[T/5]
2.74
T
2.53
T//125
4.86
[T/25]
4.91
2
[T/5|
4.88
T
4.64
LLAMA-2-7B
17/125
4.94
[T/25]
5.11
8
[T/5|
5.01
T
4.73
Sensitiveness of LISA As LISA is algorith- mically dependent on the sampling sequence of layers, intriguing see how stable LISA's performance is under the effect of randomness. For this purpose, we further investigate LISA'S performance variance over three distinct runs, each with different random seed for layer selec- tion. Here, we adopt TinyLlama, LLaMA-2-7B, and Mistral-7B models with the Alpaca-GPT4 dataset while keeping all other hyperparameters consistent with those used in the instruction follow- ing experiments in section As shown in Table [ LISA quite resilient to different random
Table 7: The MT-Bench scores derived from vary- ing random seeds for layer selection.
MODEL
SEED
SEED 2
SEED 3
TINYLLAMA MISTRAL-7B
2.57
2.55
2.60
4.85
4.82
4.82
LLAMA-2-7B
4.94
4.92
4.89
seeds, where the performance gap across three runs is within 0.13, small value compared to the

---

performance gains over baseline methods, For more ablation experiment on LISA hyperparameters, please refer Appendix A.4
Discussion
Theoretical Properties of LISA Compared with LoRA, which introduces additional parameters and leads to changes in loss objectives, layerwise importance sampling methods enjoy nice conver- gence guarantees in the original loss. For layerwise importance sampled SGD, similar to gradient sparsification 778, 55]. the convergence can still guaranteed for unbiased estimation gradients with increased variance. The convergence behavior can be further improved by reducing the variance with appropriately defined importance sampling strategy [4]. For layerwise importance sampled Adam, theoretical results in [79 [55] prove its convergence in convex objectives. If we denote f as the loss function and assume that the stochastic gradients are bounded, then based on [56], we know that Adam optimizing f aligns with Adam optimizing. f with a scaled regularizer, which can be written as
foe - f(w) + Sw,
where S is finite positive semidefinite diagonal matrix. Following existing convergence results of RBC-Adam (Corollary (79)), we have the convergence guarantee of LISA Theorem
Theorem Let the loss function f be convex and smooth. the algorithm runs in bounded convex set and the stochastic gradients are bounded, the sequence (w.) generated by LISA admits the following convergence rate:
pro - <
VT
where 1. denotes the optimum value of freg,
Memorization and Reasoning In our instruction following experiments in Appendix A.1 and A.2 observe that LISA is much better than LoRA at memorization-cemin centered tasks, such as Writing or depicting image details, while this gap is much smaller in reasoning-centent Math. is intriguing observation since LISA emphasizes more on layer-wise width and restricts the depth of learned parameters, while LoRA focuses more on depth and restricts the representation space each layer. may suggest that width is crucial for memorization, while depth is important for reasoning, a similar phenomenon that echos the intuition of [80]. Based on the same intuition, it may be possible combine the benefits of both and bring forth an even better PEFT method.
tasks like Code or
Conclusion
In this paper, propose Layerwise Importance Sampled Adam (LISA), an optimization algorithm that randomly freezes layers of L.LM based on given probability. Inspired by observations of LoRA's skewed weight norm distribution, simple and memory-efficiee freezing paradigm is introduced for LLM training. This paradigm achieves significant performance improvements over LoRA on downstream fine-tuning tasks with various models, including LLaMA-2-70B. Further experiments on domain- -specific training also demonstrate its effecttivness, showing LISA's huge potential as promising alternative to LoRA for LLM training.
Limitations
The major bottleneck LISA is the same as LoRA, where during optimization, the forward pass still requires the model be presented in the memory, leading to significant memory consumption. This limitation shall be compensated by approaches similar QLoRA [[]] where we intend to conduct further experiments verify its performance.
In addition, as suggested by the theoretical intuition, the strategy of E+H+2L Section and E+H+4L in Section 4.4 may not be the optimal importance sampling strategy, given it still sampled

---

intermediate layers in a uniformly random fashion. We anticipate the optimizer's efficiency will be further improved when considering data sources and model architecture the importance sampling procedure.
References
[1]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc., 2022.
[2] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 11::440:11440:
2020.
[3] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 14:240:1-240:11 113, 2022.
[4] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llaa: Open foundation models for code, 2023.
[5] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-
Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings. Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling. Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike,
10

---

Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine Mcl.avey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reüichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokomny Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak. Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng. Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone. Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
[6] Longteng Zhang, Xiang Liu, Zeyu Li, Xinglin Pan, Peijie Dong, Ruibo Fan, Rui Guo, Xin Wang, Qiong Luo, Shaohuai Shi, and Xiaowen Chu. Dissecting the runtime performance the training, fine-tuning, and inference of large language models, 2023.
[7] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for NIP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 2019.
[8] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online, 2021. Association for Computational Linguistics.
[9] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[10] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. ArXiv preprint, abs/2203.06904, 2022.
[11] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoye. Qlora: Efficient finetuning of quantized llms. ArXiv preprint, abs/2305.143 14314, 2023.
[12] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High- rank training through low-rank updates, 2023.
[13] Teun Kloek and Herman K Van Dijk. Bayesian estimates of equation system parameters: an application of integration by monte carlo. Econometrica: Journal of the Econometric Society, pages 1-19, 1978.
11

---

[14] Peilin Zhao and Tong Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In Francis R. Bach and David M. Blei, editors. Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1-9. JML.R.M.or.... org, 2015.
[15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
pages
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics.
[17] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[18] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Ties Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[19] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. ArXiv preprint, abs/2205.01068 2022.
[20] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilie, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. Bloom: A prrb-parrmeter open-access multilingual language model. ArXiv preprint, abs/2211.000 1.05100, 2022.
[21] Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay. Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. The falcon series of language models:
Towards open frontier models. 2023.
[22] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ArXiv preprint, abs/2302. 13971, 2023.
[23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307 0092889988888 2023.
[24] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.
[25] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[26] Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. ArXiv preprint, abs/2401.044088, 2024.
12

---

[27] Zhenheng Tang, Yuxin Wang, Xin He, Longteng Zhang, Xinglin Pan, Qiang Wang, Rongfei Zeng, Kaiyong Zhao, Shaohuai Shi, Bingsheng He, al. Fusionai: Decentralized training and deploying llms with massive consumer-level gpus. arXiv preprint arXiv:2309.01 172, 2023.
[28] Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computa- fional Linguistics and the 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921 4993, Online, 2021. Association for Computational Linguistics.
[29] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online, 2021. Association for Computational Linguistics.
[30] Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. PTR: Prompt Tuning with Rules for Text Classification. ArXiv preprint, abs/2105. 1259, 2021.
[31] Guanghui Qin and Jason Eisner. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online, 2021. Association for Computational Linguistics.
[32] Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang, Gpt understands, too. ArXiv preprint, abs/2103. 10385, 2021.
[33] Shizhe Diao, Zhichao Huang, Ruijia Xu, Xuechun Li, LIN Yong, Xiao Zhou, and Tong Zhang Black-box prompt learning for pre-trained language models. Transactions on Machine Learning Research, 2022.
[34] Shizhe Diao. Ruijia Xu, Hongjin Su, Yilei Jiang, Yan Song, and Tong Zhang. Taming pre- trained language models with n-gram representations for low-resource domain adaptation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3336-3349, Online, 2021. Association for Computational Linguistics.
[35] Shizhe Diao, Tianyang Xu, Ruijia Xu, Jiawei Wang, and Tong Zhang. Mixture-of -otturr-of-dommin. adapters:
Decoupling and injecting domain knowledge to pre-trained language models memories. ArXiv preprint, abs/2306.055 05406. 2023.
[36] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
[37] Haobo SONG, Hao Zhao, Soumajit Majumder, and Tao Lin. Increasing model capacity for free: A simple strategy for parameter efficient fine-tuning, In The Twelfth International Conference on Learning Representations, 2024.
[38] Yuheng Ji, Yue Liu, Zhicheng Zhang, Zhao Zhang, Yuting Zhao, Gang Zhou, Xingwei Zhang, Xinwang Liu, and Xiaolong Zheng. Advlora: Adversarial low-rank adaptation of vision- language models, 2024.
[39] Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. Autofreeze: Automatically freezing model blocks to accelerate fine-tuning, 2021.
[40] Sheng Li, Geng Yuan, Yue Dai, Youtao Zhang, Yanzhi Wang. and Xulong Tang. SmartFRZ: An efficient training framework using attention- layer freezing. In The Eleventh International Conference on Learning Representations, 2023.
[41] Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter training method for large language models, 2024.
[42] Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. ArXiv preprint, abs/1706.00483 2017.
13

---

[43] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEF, 2020.
[44] Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Rajesh Gupta, Yan Liu, Jiliang Tang, and B. Aditya Prakash, editors, KDD '00 The 26th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 3505-3506. ACM, 2020.
[45] Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong. Hang Xu, Aoxue Li, Dit- Yan Yeung. James Kwok, and Yu Zhang. Mixture of cluster- contter-onnitionall lora experts for rision-language instruction tuning. ArXiv preprins, abs/2312. 122/2392.11379, 2023.
[46] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[47] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
[48] Geoffrey F. Hinton, Simon Osindero, and Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, page 1527-1554, 2006.
[49] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Bernhard Schölkopf, John C. Platt, and Thomas Hofmann, editors, Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pages 153-160. MIT Press, 2006.
[50] Zhenheng Tang, Yonggang Zhang, Shaohuai Shi, Xinmei Tian, Tongliang Liu, Bo Han, and Xiaowen Chu. Fedimpro: Measuring and improving client update in federated learning. In The Twelfth International Conference on Leurning Representations, 2024.
[51] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. ArXiv preprint, abs/1 00t/1708.000 708.0388, 2017.
[52] Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training BERT in 76 minutes. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[53] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[54] Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[55] Zhenheng Tang, Shaohuai Shi, Xiaowen Chu, Wei Wang, and Bo Li. Communication--fficceenn distributed deep learning: A comprehensive survey. arXiv preprint arXiv:2003.06307, 2020.
[56] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[57] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Thirty-seventh Conference Neural Information Processing Systems, 2023.
14

---

[58] Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. The Twelfth International Conference on Learning Representations, 2024.
[59] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection. ArXiv preprint, abs/2403.03507, 2024.
[60] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. ArXiv preprint, abs/2304.03277, 2023.
[61] Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277 2023.
[62] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. ArXiv preprint,
2016.
[63] Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. Zero-offload: Democratizing billion-scale model training, 2021.
[64] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and memory-efficient exact attention with 1O-awareness, In Advances in Neural Information Processing Systems, 2022.
[65] Tri Dao. FlashAttention 2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024.
[66] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary De Vito, Martin Raison, Alykhan Tejani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035. 2019.
[67] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[68] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[69] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A numan--enttic benchmark for evaluating foundation models, 2023.
[70] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740, AAAI Press, 2020.
[71] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 1929-199 2014.
15

---

[72] Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024.
[73] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
[74] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP pages 2567-2577, Hong Kong, China, 2019. Association for Computational Linguistics.
[75] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of -hoough prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[76] Kashun Shum, Shizhe Diao, and Tong Zhang. Automatic prompt augmentation and selection with chain-of-thought from labeled data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12113-12139, 2023.
[77] Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. Active prompting with -fthugh for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1330-1350, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
[78] Jianqiao Wangni, Jialei Wang. Ji Liu, and Tong Zhang. Gradient sparsification for communication-effin -efficcionnn distributed optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Basa-Bianchh, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Process- ing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 1306-1316, 2018.
[79] Yangfan Zhou, Mingchuan Zhang, Junlong Zhu, Ruijuan Zheng, and Qingtao Wu. A random- ized block-coordinal adam online learning optimization algorithm. Neural Computing and Applications,,
2020.
[80] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. Wide & deep learning for recommender systems. In Proceedings of the 1st workshop deep learning for recommender systems, pages 7-10, 2016.
[81] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[82] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. ArXiv preprint, abs/2310.04378, 2023.
[83] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, 2024.
[84] Lueas Georges Gabriel Charpentier and David Samuel. Not all layers are equally as important: Every layer counts BERT. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell, editors, Proceedings of the BabyLM Challenge at the 27th Conference on Computational Natural Language Learning, pages 238 288222,, Singapore, December 2023. Association for Computational Linguistics.
16

---

[85] Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. LMFlow: An extensible toolkit for finetuning and inference of large foundation models. In Kai-Wei Chang, Annie Lee, and Nazneen Rajani, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 116-127, Mexico City, Mexico, June 2024, Association for Computational Linguistics.
17

---

A Additional Experiments
A.1 Image Generation
Stable Diffusion has emerged powerful approach for generating high-quality images by leveraging the diffusion process in latent space [8]] This method involves diffusing the data distribution time and iteratively refining the generated samples enhance their realism and fidelity. The key innovation lies operating within the latent space of a pre-trained autoencoder, significantly reducing computational complexity while maintaining high-quality outputs. The Latent Consistency Model (LCM) further improves the efficiency and performance of diffusion models in the latent space by incorporating a consistency loss, which penalizes deviations from the expected latent trajectories [82]. This ensures that the generated samples remain coherent and visually plausible throughout the diffusion steps.
In our experiments, we evaluated the performance of LCM against other fine-tuning methods such as LoRA, and LISA using latent consistency distillation to distill stable diffusion v1.5 and v2.1 model. We utilized the official code and parameters for both LISA and LoRA from the Diffusers library to ensure consistency and comparability of results. Adjustments accumulation steps to achieve a balanced trade-off between computational efficiency and performance. Figure the generated image comparison. Observations reveal that LISA can generate higher- quality images in fewer inference steps. The images generated using LISA display more intricate details and sharper clarity, particularly evident in the distinct facial features and environment textures. In contrast, the JoRA-generatee images offer softer, more blended aesthetic with dream-like quality, emphasizing smooth transitions over precise detail. The prompts for images in figure 5 in the left-to-right are given below
made only to the batch size and
Self-portrait oil painting, a beautiful cyborg with golden hair, 8k.
Astronaut a jungle, cold color palette, muted colors, detailed, 8k.
A photo of a beautiful mountain with a realistic sunset and blue lake, a highly detailed masterpiece.
A.2 Instruction Following Fine-tuning
Table offers comprehensive evaluation three fine-tuning methods-Full Parameter Fine- Tuning (FT), Low-Rank Adaptation (LoRA), Gradient Low-Rank Projection(GaLoree and Layerwise Importance Sampling AdamW (LISA)-acroo) a diverse set of tasks including Writing, Roleplay, Reasoning, Math, Extraction, STEM, and Humanities within the MT--eech benchmark. The results demonstrate LISA's superior performance, which surpasses LoRA, GaLore, and full parameter tuning in most settings. Notably, LISA consistently outperforms LoRA and full parameter tuning in domains such as Writing, STEM, and Humanities. This implies that LISA can benefit memorization tasks, while LoRA partially favors reasoning tasks.
Table 8: Comparison of Language Model Fine-Tuning Methods on the MTTBeech score.
MT-BENCH MATH
MODeL.C METHOD
WRITING
REASONING
CODE
EXTRACTION
STEM
HUMANTTIRS
TINYLLAMA (VANILLA)
1.05
2.25
1.25
1.00
1.00
1.00
1.45
1.00
1.25
TINYLLAMA (LORA)
2.77
4.05
1.35
1.00
1.40
1.00
1.55
2.15
1.90
TINYLLAMA (GALORE)
3.55
5.20
2.40
1.85
TINYLLAMA (LISA)
3.30
4.40
2.65
1.12
1.30
1.75
3.00
3.05
2.57
TINYLLAMA (FT)
3.27
3.95
1.35
1.04
1.33
1.73
2.69
2.35
2.21
(VANILLA)
MISTRAI-27 -BB (LORA)
5.30
4.40
4.65
2.35
3,30
5.50
5.55
4,30
4.41
MISTRAL-7B (GALORE)
5.05
5.27
4.45
1.70
2,50
5.21
5.52
5,20
4.36
MISTRAL-7B
MISTRAL-7B. (FT)
5.50
4.45
5.45
2.50
3,25
5.78
4.75
5.45
4.64
LLAMA-2-78 2-BB (VANILLA)
2.75
4.40
2.80
1.55
1.80
3.20
5.25
3.29
LLAMA-2-78 (LORA)
6.30
5.65
4.05
1.45
4.17
6.20
LLAMA-2-7B (GALORE)
5.60
6.40
3.20
1.25
1.95
5.05
6.57
7.00
4.63
1LAMA-2-7B (LISA)
6.55
6.90
3.45
1.60
2.16
4.50
6.75
7.65
4.94
LLAMA-2-78 (FT)
5.55
3.60
4.70
18

---

Stable Diffusion v2.1
LORA
LISA
Stable Diffusion v1.5
LoRA
LISA
Figure 5: Generated images using LoRA (left) and LISA (right) on Stable Diffusion v2.1 model and Stable Diffusion v1.5. First row: number of inference step Second row: number of inference step 10.
Table D detailed MT-Bench scores for the LLaMA-2-70B model discussed in Section 4.4 demonstrating LISA's superior performance LoRA in all aspects under large-scale training scenarios. Furthermore, Figure 6 we observe that LISA consistently exhibits on-par or faster convergence speed than LoRA across different models, which provides strong evidence for LISA's superiority in practice.
Table 9: Mean score of three fine-tuning methods three seeds for LLaMA-2-70B the MT. Bench.
MT-BENCH MATH
MODEL METHOD
WRITING
ROLEPLAY
REASONING
Cone
EXTRACTION
STEM
HUMANITIES
AVG. 1
LLAMA-2-70BB 2
7.77
5.52
2.95
1.70
1.70
6.40
7.42
8.07
5.19
2.AMA---7OB(LLRA),
LLAMA-2-PORRLSAA
8.18
7.90
5.45
4.45
2.75
7.45
8.60
9.05
6.72
LLAMA-2-70B(FT)
6.45
7.50
5.50
3,40
2.15
7.55
8.10
9.40
6.25
The aforementioned results show that Vanilla LLaMA-2-70B excels in Writing, but full-parameter fine-tuning led to decline in these areas, phenomenon known as the "Alignment Tax" [. This tax highlights the trade-offs between performance and human alignment instruction tuning. LISA, however, maintains strong performance across various domains with a lower "Alignment Tax".
19

---

Comparison of Less Curves an Alpace GPT4 Demaset for TinyLama
Compartson of Loss Curves on Alpaca GPT4 Dataset for GPT2-Small
LISA
232
FT
24-
24
1.2
step
xtep
Comparison Loss Durves on NppaaaGPPA Datase: for Mistral-78
Comparison of Lass Curves on Alpaca-GPPA Dataset Llama2-70
LoRA
LoRA
LISA
LISA
FT
1.00
3
1.2
0.00
1.3
2tp
stop
Figure 6: Loss curves for LoRA, LISA, and full-parameter training on the Alpaca-GPT4 dataset across different models.
A.3 Continual Pre-training
To better analyze the performance of LISA in the continual pre-training sce- nario, we used Open WebMath for con- tinual pre-training and the GSM8K train split for the fine-tuning stage, varying the number of sampling lay- ers, % within LISA, ranging from 2, 4, 8, 16, compared to the accuracy of the full parameter (FT) continual pre-training. Table / details the re- sults for the LLaMA-2-77 model un- der various continual pre-training con- figurations. Notably, LISA with eight sampling layers achieves comparable accuracy with full parameters contin- ual pre-training method. Furthermore, LISA with 16 sampling layers passes the accuracy of full parameter train- ing.
Different LISA activated layers continual pretraining
0.24
0.22
0.20
0.18
The Number of LISA activated layers
Figure 7: The comparison of full parameter (FT) training and LISA with different sampling layers under continual pre- training scenario. The accuracy is the test set of GSMSK.
A.4 Ablation Experiments
A.4.1 Sampling Layers
We conducted an ablation study on the LLaMA-2-7B model trained with the Alpaca-GPT4 dataset, setting the sampling period K = 13, so the number of samplings is exactly 10. The study explored different configurations of sampling layers including (E+H+2L, E+H+4L, E+H+8L). Figure g depicts the impact of the number of sampling layers Y on the training dynamics of the model. Three scenarios were analyzed: > 2 (blue line). training steps. Initially, all three configurations exhibit a steep decrease loss, signaling rapid initial improvements in model performance. it's clear that the scenario with 7 8 consistently maintains lower loss compared the 7 = and y = configurations, suggesting that a higher y value leads to better performance in this context.
(green line), and y = 8 (red line). throughout 120
20

---

Ablation experiment on number sampling layers y
Y=2
Y=4
y=8
l00s
40
60
100
120
step
Figure 8: Comparison of loss curves for the y ablation experiment.
Table 10 demonstrates that LISA with only E + H layers experiences performance decline when the 0 sampling layers are omitted, highlighting the importance these layers.
Table 10: Results of additional experiments LISA with only E H layers.
MMLU (5-SHOT)
AGIEVAL (3-SHOT)
WINOGRANDE (5-SHOT)
MODEL
METHOD
MT-BENCH
VANILLA
25,50
19.55
59.91
1.25
LISA (E H)
25.49
0.14
20.75 0.21
60.43
2.18 + 0.31
TINYLLAMA
LISA
26.02
0.13
21.71 + 0.09
61,48 0.08
2.57 - 0.25
FT
25.62 #.100
21.28 # 0.07
62.12 0.15
2.21 0.16
VANILLA
60.12
26.79
79.24
4.32
LISA (E H)
6..49
27.66
0.07
77.93 + 0.11
4.51 0.27
MISTRAL-7B
LISA
62.09 0.10
29.76
0.09
78.93 0.08
4.85 0.14
FT
61.70 0.13
28.07 # 0.12
78.85 0.12
4.64 + 0.12
VANILLA
45.87
25.69
74.11
3.29
LISA (E+ H)
45.88 0.12
25.82 # 0.15
73.48 # 0.22
4.63 + 0.35
LLAMA-2-7B
LISA
46.21 0.12
26.06 0.08
75.30 0.11
4.94 0.14
FT
45.66
0.09
0.09
27.02 + 0.10
75.06 1 0.13
4.75 0.16
To better understand the LISA and hyperparameter Sampling Layers we conducted ablation experiments on Sampling Layers and learning rate 17. The aim was investigate the combined impact of these two variables on the LISA Our experiments utilized the LLaMA---77B model, trained on the GSM8K dataset. We examined the effect of increasing the number of Sampling Layers while simultaneously decreasing the learning rate 17.
Table 11: Compare LISA with different Sampling Layers and Learning Rate 1,, evaluate GSM8K.
SAMPLING LAYERS 7
LEARNING RATE 1)
10 5
2.5 x 10 5
1.25
10
6.25 10
15.77
15.32
15.21
15.01
11.29
11.34
15.87
15.27
13.32
14.39
16.30
15.32
16
15.42
15.92
15.78
16.57
The Table 11 indicates that a higher number of sampling layers can enhance the model's effectiveness, provided the learning rate adjusted appropriately. Specifically, the optimal performance is observed when the learning rate is reduced in proportion the increase in the number of sampling layers y, demonstrating the delicate balance required between these two parameters maximize LISA's efficacy.
21

---

A.4.2 Sampling Period K
Figure(Idiplaay the effects of varying sampling Period K training 7B-sized model using the -2K-en Alpaca-GPT4 dataset. This graph contrasts loss curves for different sampling pe- riod values:
Ablation experiment on sampling period K
K=122
K=25
K=13
18 1.2
122 (green line), K 25 (red line), and = 13 (blue line) across 122 training
steps. The results indicate that although each K value results in distinct training trajectories, their conver- gence points are remarkably similar. This finding im- plies that for a 7B model trained on a 52K instruction conversation pair dataset, sampling period K is optimal for achieving the best loss curve and corresponding MTT-Bench score radar graph.
step
Figure 9: Comparison loss curves for the sampling period K ablation experiment.
A.4.3 Sensitiveness Randomness
LLaMA-2-7B on Alpaca-GPT4 with update step per sampling period 13, and sampling layers 2, run three times with different random layer pick. Figure 10 shows that different random selections of layers slightly affect training process converge similarly. Despite initial fluctuations, the loss trends of three runs dint--isinguuihee by blue, green, and red demonstrate that the model consistently reaches a stable state, underscoring the robustness of the training against the randomness in layer selection.
Ablation experiment on random sampling layers
1.4
Random 1
Random 2
1.3
Random 3
1.2
lo5s
1.1
1.0
20
60
80
100
120
Figure 10: Comparison of loss curves for random variance ablation experiment, indicating the loss metric over steps.
Table 12: Compare LISA with fixed lay-
Additionally, we also conduct experiments that analyze the impact of fixing randomly selected layers during train- ing. The suggestion highlighted need to evaluate the model's stability and performance under such conditions. To address this, we conducted further experiments using the LLaMA-2-7B model, maintaining identical hyperpa- rameters and datasets in our paper, and repeated the experiments three times to reduce variability from the ran- dom selection process.
ers LLaMA-2-7B, evaluate MT- Bench.
METHOD
MT-BENCH
LISA
4.94
LISA-FIX-1
4.62
LISA-FIX-
4.60
LISA-FIX-3
4.67
22

---

The results are presented as table 12 with 'LISA-fiX' denoting the experiments with randomly fixed layers and the appended number indicating the different selected seeds. The results are presented as table 12 with '-ISA-fix denoting the experiments with randomly fixed layers and the appended number indicating the different selected seeds.
A.5 Performance with Early Exiting
According previous works 84 the early exiting strategy in LLMs is effective. We are interested investigating whether the LISA algorithm will have different impact when combined with the early exiting strategy. To explore this, conducted series of experiments using the LLaMA-2-7B model, focusing various training methods and early exit points. The early exiting method is DoLa 83
Table 13: GSM8K Scores for LLaMA-2-7B when LISA meets the early exiting strategy DoLa.
METHOD
GSM8K %
VANILLA
15
VANILLA DOLA
11
FT + DoLA LISA + DoLA
16
17
We conducted all of our experiments using the ILaMA-2-7B model, selecting layers [0, 8, 16, 32] as early exit points for evaluation under four different training conditions: Vanilla (baseline), Vanilla with DoLa, Full Parameter Fine-Tuning (FT) with DoLa, and LISA with Dol.a. The model used is the same one trained on the Alpaca-GPT4 dataset, as reported in section 4.2 This comprehensive approach enabled detailed comparison of the model's performance and layer representation capabilities under various training methodologies. Due to time constraints, our experiments were conducted on a subset of 100 questions from the GSM8K test employing the same prompt in the DoLa paper.
From the table [3] it can be seen that the LISA algorithm does negatively affect the representation or performance of some layers of the model; instead, it contributes to some improvements in effectiveness.
A.6 Comparison of Evaluation Loss
Besides the training loss, we also care about how LISA performs on the validation dataset. So, we split the Alpaca-GPT4 dataset into train and validation sets, the ratio is 9 1, and ensure there is no data overlap between these two sets. Then we use the same setting in the see 4.2 training the LLaMA-2-7B on the Alpaca-GPT4 dataset with full parameter training (FT), LoRA, GaLore, and LISA. As Figure shows, the trend validation loss mirrors that observed training loss, with LISA exhibiting no signs of overfitting. This consistency underlines LISA's robustness in maintaining performance across different dataset splits.
Comparison of Validation Loss Curves on Alpaca-GPT4 Dataset for LLaMA-2-7B
1.40
GaLore
LoRA
1.30
LISA
1.25
LoRA
FT
GaLore
1.20
LISA
1.15
1.10
20
100
120
Step
Figure 11: Validation loss comparison on the Alpaca-GPT4 dataset for LLaMA-2-7B, showing LISA, GaLore, LoRA, and FT strategies, with arrows indicating specific observations in the loss trends.
23

---

Table 15: The statistics of datasets. TRAIN and # TEST denote the number of training and test samples respectively. The unit for Open WebMathh is the number of documents.
DATASET
TRAIN
# TEST
ALPACA GPT-4
52,000
MT-BENCH 67
80
GSM8K 73
7,473
1,319
MMLU
14,079
AGIEVAL 669
9316
WINOGRANDE 70
44,000
PURMEDQA 74
211 211,269
1,000
OPENWERMATH
72
6.3M
A.7 Additional Observations of Layerwise Skewness
We conduct further weight norm experiments on Mistral-7 B to support our motivation that the bottom and top layers have more significant impact on the output. Figure [2 provides similar observations 2-BB, where the bottom layer has a larger weight norm than other layers.
Weight Norm of Each Layer For Mistral-7B
Mistral-7B with Full Parameters Mistral-78 with LORA
Weight Norm
10
169
embed tolene
Inpors.
Layors-3
2
inpers.d
layer
unyore
layore.e
layors
layars.1 layart.1 1f
lopers 12
lnpant Syyens 15
18
laport lajorrs. layort. 13
17
loyars lnpors 20
10
syorr 22
21
tsyomt2 23
24
lnyort og Layors -77 lnvors 28 Tapentzs 30 pogrr peerd
28
Model Layers
Figure 12: Layer-wise weight norms during training of Mistral-7B with LoRA and Full Parameters
training.
B
Training Setup and Hyperparameters
B.1
Training Setup
In our experiments, we employ the LMFlow toolkit [85] [85] for conducting full parameter fine-tuning, LoRA tun- ing, and LISA tuning. We set the epoch number for fine-tuning and continual pre-training scenarios. Ad- ditionally, we utilized DeepSpeed of- fload technology [63] to run the LLMs
Table 14: Baseline Model Specifications
MODEL DIM
MODEL NAME
# PARAMS
LAYERS
# HEADS
TINYLLAMA
1.1
2048
MISTRAL-7B
LLAMA-2-7B
32
4096
LLAMA-2-70B
70
80
8192
64
efficiently. All experiments were conducted on 8x NVIDIA Ampere Architecture GPU with 48 GB memory. Table and and table 15 are the information covered in this paper.
24

---

Table 16: The hyperparameter search identified optimal settings for each method: FP (Full Parameter Training), LoRA, GaLore, and LISA.
FP
LoRA
LISA
Model
Ir
Ir
Rank
Ir
Y
K
GPT2-Small
3 10
6 10 10
128
10 10
3
TimyLlama
5 10
5 10
10
128
5 x
10
2
10
Mistral-7B
5 x 10-6
x 10-5
128
x 10
10
LLaMA-2-7B
5 10-6
5 10-5
128
5 10
10
LLaMA-2-70B
5 x 10
5 10
128
5 x
10
4
10
Our study explored a range of learning rates from 5 x 10-6 to 3 x 10-1, applying this spectrum to Full Parameter Training, LoRA, and LISA methods. For LoRA, we adjusted the rank either 128 or 256 to vary the number of trainable parameters, applying LoRA across all linear layers. Regarding the number of sampling layers 7, our selections were guided by GPU memory considerations as reported in LoRA studies 19]; For the LISA algorithm,
selected y 2, and for experiments involving the 70B model, we opted for y = 4. The sampling period (K), defined as the number of update steps per sampling interval, ranges from to 50. This range influenced by variables such as the size of the dataset, the batch size, and the number of training steps. To manage this effectively, partitioned the entire training dataset into K segments, thereby enabling precise regulation of the training steps within each sampling period.
B.2 Continual Pre-training Dataset
We extracted high-quality subset from WebMaathtth [72), using the "Math_score'' attribute from the metadata the metric for high-quality instances. The 'Math Score' represents the probability that a document is mathematical, and we set the threshold at 0.95. Finally, the number tokens for this high-quality subset is 1.5 billion.
B.3 Hyperparameter search
We commenced study with grid search covering (i) learning rate, (ii) number of sampling layers y, and (iii) sampling period K. Noting the effective performance of the LoRA method, we set the rank value to 128 or 256.
The optimal learning rate was explored within the range (5 x 10-6, 10-5,55 5 x 10-5, 10-4.3 10-1), applicable to full parameter training, LoRA, and LISA. For GaLore, we adhered to the official Transformers implementation utilizing default parameters, with the learning rate matching that of the full parameter training.
Regarding the number of sampling layers 7, in alignment with Table D selected values that matched or were lower than LoRA's GPU memory cost. Consequently, = 2 was predominantly used in the LISA experiments, while 7
chosen for the 70B model experiments.
For the sampling period K, we examined values within 1,3, 5, 10, 50, 80, aiming to maintain the model's 's update steps within range of 10 50 per sampling period. This selection informed by dataset size, batch size, and total training steps.
The comprehensive results of our hyperparameter search, detailing the optimal values for each configuration, are presented Table
C Licenses
For instruction following and domain-specific fine-tuning tasks, all the datasets, including Alpaca (61] GSM8k 773,, MMLU 68) AGIEval (69 and PubMedQA are released under MIT license.
25

---

WinoGrande 70) and MT-Benchh [67] are under Apache-2.0 2.0 license. For GPT-4, the generated dataset only for research purposes. which shall not violate its terms of use.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We proposed Layerwise Importance Sampled AdamW (LISA), promising alternative for LoRA, which applies the idea of importance sampling different layers in LLMs and randomly freeze most middle layers during optimization. Experimental results show that with similar or less GPU memory consumption, LISA surpasses LoRA or even full parameter tuning in downstream fine-tuning tasks, where LISA consistently outperforms LoRA by over 11 1%-37% in terms of MT-Bench and multiple benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Limitation section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
27

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See Discussion section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: See Experimental Result section
4
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
28

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
See Experiment section and Appendix B
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpss//niis.cc/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Bttc//giide///oeSuuii
/ //nips.
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Experiment section 4 and Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
See Experiment section and Appendix
B
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiment section and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwrrppscccoooo/lc/tcttte
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms every respect with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
30

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Licenses section
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssil swi thcode. swhcodee com/ datasttss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
31

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "QlruI0wdRF", "reviewer": "Reviewer_Rt8C", "summary": "This paper proposes a new optimization algorithm called Layerwise Importance Sampled AdamW (LISA) for large language model (LLM) fine-tuning. The authors observe the skewed weight norms across different layers in the Low-Rank Adaptation (LoRA) method and use this observation to develop LISA, which randomly freezes certain layers during optimization. Experimental results show that LISA outperforms LoRA and full parameter training in various downstream fine-tuning tasks, demonstrating its effectiveness and memory efficiency.", "questions": "None", "limitations": "The limitation is fine with me.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) The paper addresses an important problem in the field of large language models, which is the high memory consumption during training. The proposed LISA algorithm provides a memory-efficient alternative to the existing LoRA method.\n\n(2) The authors provide a clear motivation for their work and thoroughly analyze the layerwise properties of LoRA, which leads to the development of LISA. This analysis adds valuable insights to the field.\n\n(3) The experimental results demonstrate the superiority of LISA over LoRA and full parameter training in various downstream tasks. The authors provide detailed comparisons and evaluations, supporting the claims made in the paper.\n\n(4) The paper is generally well-written and easy to follow.\n\n(5) The design of the proposed \"Layerwise Importance Sampling AdamW\" is quite interesting and novel, which may be generalized in other related LLM's variants.", "weaknesses": "(1) I am curious whether the authors will open-source their implementation? Since efficient training techniques are important to this field, it would be beneficial to have an easy-to-use implementation. In addition, it would be helpful if the authors provide more information on their hyperparameter tuning strategy in the experiments, like how do you choose the best hyperparameters?\n\n(2) The some parts of the writing and layout needs improvement. For example, there are too many capitalized letters like line 6 \"Parameter Efficient Fine-Tuning\". I do not see any reason to capitalize every words. In addition, the figures should be re-organized like Figure 2.\n\n(3) The authors seem to define 65B as a boundary for small language models and large language model (see line 56). Just curious, are there any justification or relevant claims? Since there are more and more small language model research nowadays and they are mainly focusing on 1B-7B, it would be interesting to have a clear definition in our community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to offer our sincere thanks for all your constructive comments and recognition of our contributions. We really appreciate it. Here we will address the raised concerns one by one.\n\n**Weakness 1: Open-Source Implementation.**\n>  I am curious whether the authors will open-source their implementation? Since efficient training techniques are important to this field, it would be beneficial to have an easy-to-use implementation. In addition, it would be helpful if the authors provide more information on their hyperparameter tuning strategy in the experiments, like how do you choose the best hyperparameters?\n\nThank you for your questions! The LISA algorithm has been integrated into several third-party libraries, such as LMFlow [1] and Axolotl [2], which have almost full support for single-GPU settings, and basic support for multi-GPU settings. The full support for multi-GPU settings will be available soon.\n\nFor hyperparameters, we conducted extensive hyperparameter searches for learning rates for SFT, LoRA, and LISA, as well as for $K$ and $\\gamma$ for LISA. Details can be found in Appendix B.3 of the paper.\n\n\n**Weakness 2: Writing and layout improvement.**\n>  The some parts of the writing and layout needs improvement. For example, there are too many capitalized letters like line 6 \"Parameter Efficient Fine-Tuning\". I do not see any reason to capitalize every words. In addition, the figures should be re-organized like Figure 2.\n\nWe appreciate your feedback and will definitely improve these aspects in our revisions. Specifically, we will correct the unnecessary capitalization, such as in \"Parameter Efficient Fine-Tuning,\" and reorganize the figures, including Figure 2, to enhance clarity and readability.  \n\n**Weakness 3: Definition of small language models.**\n>  The authors seem to define 65B as a boundary for small language models and large language model (see line 56). Just curious, are there any justification or relevant claims? Since there are more and more small language model research nowadays and they are mainly focusing on 1B-7B, it would be interesting to have a clear definition in our community.\n\nIn our experiment, we classify language models that can fit within a single 8xA40 (48GB) server as small language models ($\\le$ 65B). This classification is based on practical considerations of model deployment and resource requirements. We will provide a more detailed description and clarify that in our revisions. \n\nBy the way, to the best of our knowledge, 2B is normally the boundary for mobile LLMs, and 8B normally represents the boundary for single-GPU-trainable LLMs.\n\n> Reference:  \n> [1] Shizhe Diao, Rui Pan, Hanze Dong, KaShun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. 2024. LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 3: System Demonstrations), pages 116–127, Mexico City, Mexico. Association for Computational Linguistics. https://github.com/OptimalScale/LMFlow  \n> [2] Axolotl: https://github.com/axolotl-ai-cloud/axolotl", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "01q5jYexzS", "reviewer": "Reviewer_K468", "summary": "This paper proposes a lighter alternative to LoRA, LISA, based on using importance sampling to periodically choose a subset of layers to optimize. It is motivated by observations on the norm of parameter updates made during training with LoRA, compared to full parameter fine-tuning. Experiments are made comparing LISA with LoRA and full parameter fine-tuning on memory efficiency, and moderate/large fine-tuning and continual pre-training. Supplementary experiments are made on the hyperparameters of LISA and training sensitivity. Results show that LISA uses less memory than LoRA and achieves better results on most tasks. The paper also presents a discussion including theoretical properties of LISA.", "questions": "-  While the choice of that sampling distribution seems motivated by observation on LoRA, there seems to be shortcuts taken, particularly on fixing $E$ and $H$. Is there anything more to motivate this ? Did you try more extensive experiments with only $E + H$ ? \n- Can you relate your approach with those mentioned in Section 2.2, focusing on layer-selection ?", "limitations": "- The limitations have been addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method proposed by this paper is of obvious interest for efficient training.\n- The experimental results presented are convincing, and seem to indicate that the method is promising.", "weaknesses": "- The main issue I have with this paper is that the approach presented does not seem to actually use Importance sampling: the weights of the $E$ and $H$ layers are fixed to $1.0$, and the intermediate layers have uniform weights. Besides, the presentation of the distribution and rates used in Section 3.2 is rather unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to offer our sincere thanks for all your constructive comments and recognition of our contributions. We really appreciate it. Here we will address the raised concerns one by one.\n\n**Weakness: Sampling Strategy.**\n> The main issue I have with this paper is that the approach presented does not seem to actually use Importance sampling: the weights of the $E$ and $H$ layers are fixed to $1.0$, and the intermediate layers have uniform weights. Besides, the presentation of the distribution and rates used in Section 3.2 is rather unclear.\n\nThank you for the valuable feedback! Importance sampling serves as the motivation of our algorithm, which emphasizes the difference between layers. To avoid misunderstandings, we will definitely clarify that in our revisions. Regarding the presentation issue in Section 3.2, we will add the description for different layers accordingly, such as $E$ and $H$, to make it clearer.\n\n\n**Question 1: Fixing $E + H$ layers.**\n> While the choice of that sampling distribution seems motivated by observation on LoRA, there seems to be shortcuts taken, particularly on fixing $E$ and $H$. Is there anything more to motivate this ? Did you try more extensive experiments with only $E+H$ ?\n\nThank you for your insightful question! Yes. Besides the motivation we mentioned in the paper, we also found that fixing $E+H$ significantly hurts the performance, indicating the importance of those two layers.\n\nTo further understand the effect of $E+H$ layers, we conducted additional experiments on LISA with only $E+H$ layers:\n\n\n| Model          | Method         | MMLU (5-shot) | AGIEval (3-shot) | WinoGrande (5-shot) | MT-Bench      |\n|----------------|----------------|---------------|------------------|---------------------|---------------|\n| TinyLlama      | Vanilla        | 25.50         | 19.55            | 59.91               | 1.25          |\n|                | FT             | 25.62 ± 0.10        | 21.28 ± 0.07           | **62.12 ± 0.15**              | 2.21 ± 0.16         |\n|                | LISA ($E+H$)   | 25.49 ± 0.14        | 20.75 ± 0.21           | 60.43 ± 0.19              | 2.18 ± 0.31             |\n|                | LISA           | **26.02 ± 0.13**        | **21.71 ± 0.09**           | 61.48 ± 0.08              | **2.57 ± 0.25**         |\n|||||||\n| Mistral-7B     | Vanilla        | 60.12         | 26.79            | 79.24               | 4.32          |\n|                | FT             | 61.70 ± 0.13        | 28.07 ± 0.12           | 78.85 ± 0.12              | 4.64 ± 0.12         |\n|                | LISA ($E+H$)   | 61.49 ± 0.12        | 27.66 ± 0.07           | 77.93 ± 0.11             | 4.51  ± 0.27            |\n|                | LISA           | **62.09 ± 0.10**        | **29.76 ± 0.09**            | **78.93 ± 0.08**              | **4.85 ± 0.14**           |\n|||||||\n| LLaMA-2-7B     | Vanilla        | 45.87         | 25.69            | 74.11               | 3.29          |\n|                | FT             | 45.66 ± 0.09        | **27.02 ± 0.10**            | 75.06 ± 0.13              | 4.75 ± 0.16         |\n|                | LISA ($E+H$)   | 45.88 ± 0.12        | 25.82 ± 0.15            | 73.48 ± 0.22              | 4.63 ± 0.35             |\n|                | LISA           | **46.21 ± 0.12**         | 26.06 ± 0.08           | **75.30 ± 0.11**              | **4.94 ± 0.14**        |\n\nAs shown in the Table, LISA ($E+H$) can achieve reasonable performance but is still no match for LISA.\n\n\n**Question 2: Layer-selection related works.**\n> Can you relate your approach with those mentioned in Section 2.2, focusing on layer-selection ?\n\nThanks for your constructive comments!\n\nCompared with the mentioned works, LISA's simple selection strategy makes it much easier to understand and implement, which reduces theoretical difficulties in analysis and engineering difficulties in practice.\n\nIn contrast, Autofreeze [1] adopts a heuristic rule based on gradient norm changes to select layers, which is difficult to analyze in theory and not quite easy to control in practice. Autofreeze also focuses more on encoder-only models such as BERT, while LISA emphasizes its applications in LLMs. \n\nFreezeout [2] and SmartFrz [3] are facing the same issues as well, which progressively freeze layers, or adopt a NN-based predictor to guide the selection respectively. These heuristic-driven approaches are hard to obtain theoretical guarantees, and their empirical properties may vary as the backbone network changes. In addition, both Freezeout and SmartFrz focus on Computer Vision tasks, while LISA is adopted mostly in LLM settings.\n\nWe will definitely include the aforementioned discussions in our revisions.\n\n\n> Reference  \n> [1] Liu, Y., Agarwal, S., & Venkataraman, S. (2021). Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXiv:2102.01386.  \n> [2] Brock, A., Lim, T., Ritchie, J. M., & Weston, N. (2017). Freezeout: Accelerate training by progressively freezing layers. arXiv preprint arXiv:1706.04983.  \n> [3] Li, S., Yuan, G., Dai, Y., Zhang, Y., Wang, Y., & Tang, X. (2024). Smartfrz: An efficient training framework using attention-based layer freezing. arXiv preprint arXiv:2401.16720.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VWLfLC0UwM", "reviewer": "Reviewer_kABp", "summary": "This paper proposes an parameter-efficient finetuning (PEFT) method, LISA, that only finetunes some sampled layers while freezing the rest  for some certain iterations, and resample the finetuned layers later. \n\nTo validate LISA's effectiveness, the authors benchmark it on various tasks, like instruction-following, medical QA, math problems, across various sizes of LLMs. The results show that LISA shares a similar GPU memory consumption as LoRA, 1.5x speedup than LoRA for training, better or comparable to LoRA and Full Finetuning for various tasks.\n\nThe main contributions of this paper are:\n1. The proposed method is simple, easy to be applied;\n2. The experiments are thorough and consistent to support LISA's effectiveness.", "questions": "See weakness.", "limitations": "The authors discuss in the Limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow;\n2. The proposed method, LISA, is simple and practical;\n3. The experiments are thorough, and the results are better than the baselines.", "weaknesses": "1. Lack of novelty. LISA is very similar to the method proposed in the paper [46] in the references. [46] proposes to pretrain a model layer-by-layer, while LISA shuffles the training order and finetune some for a period. Such novelty is not enough. In addition, the authors don't apply their finding, i.e. skewed weight-norm distribution across layers, to design a better sampling method. Instead, they apply uniform sampling, which makes the finding less relevant. BTW, the finding is very similar to a previous work [2] (Figure 4).\n\n2. Lack of experimental details about the optimizer states, and unfair comparison. Compared to LoRA, LISA basically finetunes the whole model, which means the optimizer states stay the same as Full Finetuning. Since the oprimizer states occupy as twice memory as the model's, I assume LISA offload them to CPU. This part is not explicitely stated, nor in the limitation section. When comparing the training time, the authors only compare the forward and backward time (Figure 4), and ignore the latency introduced by the offloaded optimizer states, which is unfair since LoRA doesn't need to offload. \n\n3. Lack of important ablation study. The benefit of LISA is from the training of the whole model in a layer-by-layer manner. Actually, we can also apply LISA to LoRA. I.e. we can use a very large size LoRA and only finetune the sampled LoRAs. In this way, LoRA has a larger number of trainable parameters, which might close the gap between LISA and LoRA.\n\n4. Unintuitive results without further analysis. The most surprising result is that LISA outperforms full finetuning quite often, which is against our intuition. It also casts a doubt: how do you choose the hyper-parameters (like Table 15)? And what is the error bar for the main results.\n\n\n[1] A fast learning algorithm for deep, Geoffrey E. Hinton, Simon Osindero, Yee-Whye Teh\n[2] Parameter-Efficient Fine-Tuning without Introducing New Latency, Baohao Liao, Yan Meng, Christof Monz", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback. To address the raised concerns, we have provided additional clarifications and experiments, the details are listed as follows,\n\n**Weakness 1: Lack of novelty.**\n> LISA is very similar to the method proposed in the paper [46] in the references... In addition, the authors don't apply their finding, i.e. skewed weight-norm distribution across layers, to design a better sampling method. Instead, they apply uniform sampling, which makes the finding less relevant. BTW, the finding is very similar to a previous work [2] (Figure 4).\n\nThanks for the valuable comments. We would like to emphasize that LISA's major novelty lies in its successful applications in LLMs, which significantly reduce memory consumption and improve the training speed of LLMs. In contrast, [46] is a paper in 2006 that focused on training of deep belief nets. Their settings are vastly different.\n\nRegarding the non-skewed weight-norm distribution claim, this may be a misunderstanding. The distribution in LISA is indeed skewed, as the embedding and linear head layers are always unfrozen during the training. We will add additional clarifications in Section 3.2 to make it clearer for readers.\n\nConcerning similarity to Figure 4 in [2], again, their settings are different, as the observations of [2] is made in encoder-only models, e.g. Roberta, while LISA focuses on decoder-only LLMs. In addition, we would like to emphasize again that the observation itself only serves as a motivation of our work, LISA's main novelty still lies in its successful applications in LLMs.\n\n**Weakness 2: Lack of experimental details about the optimizer states, and unfair comparison**\n> Compared to LoRA, LISA basically finetunes the whole model, which means the optimizer states stay the same as Full Finetuning. Since the oprimizer states occupy as twice memory as the model's, I assume LISA offload them to CPU. This part is not explicitely stated, nor in the limitation section. When comparing the training time, the authors only compare the forward and backward time (Figure 4), and ignore the latency introduced by the offloaded optimizer states, which is unfair since LoRA doesn't need to offload.\n\nThanks for the feedback. We would like to kindly remind the reviewer that there may be some miscommunications, which lead to the false impressions here.\n\nIn our implementation for all the experiments, the model will be reinitialized every time, so the optimizer is only applied to the trainable parameters and has the same or less memory consumption as LoRA, just as shown in Table 1 of the paper. Because of this, there is no need to offload the optimizer states to CPUs, let alone the latency of offloading optimizer states for LISA. We will emphasize these implementation details in our papers to avoid misunderstandings.\n\n\n**Weakness 3: LISA + LoRA experiment.**\n> The benefit of LISA is from the training of the whole model in a layer-by-layer manner. Actually, we can also apply LISA to LoRA. I.e. we can use a very large size LoRA and only finetune the sampled LoRAs. In this way, LoRA has a larger number of trainable parameters, which might close the gap between LISA and LoRA.\n\nThank you for the suggestion. It is valid to combine LoRA with LISA. But if this approach works, logically it is just another proof of the effectiveness of LISA, not the opposite. Actually, LISA is orthogonal to most low-rank techniques, and there have already been published papers that combine LISA with GaLore [3]. Since the combination of those techniques already merits another paper, we intended to leave that part for future works.\n\nIn addition, we have evidence showing that this simple combination does not work easily. As demonstrated in the table below, LoRA + LISA achieves similar performance as LoRA, but still significantly worse than LISA. The detailed results can be found in the attached rebuttal PDF.\n\n| Model          | Method   | MMLU (5-shot)      | AGIEval (3-shot)      | WinoGrande (5-shot)   |\n|----------------|----------|--------------------|-----------------------|-----------------------|\n| LLaMA-2-7B     | Vanilla  | 45.87              | 25.69                 | 74.11                 |\n|                | FT       | 45.66 ± 0.09       | **27.02 ± 0.10**      | 75.06 ± 0.13              |\n|                | LoRA     | 45.50 ± 0.07       | 24.73 ± 0.04          | 74.74 ± 0.09                  |\n|                | GaLore | 45.56 ± 0.05       | 24.39 ± 0.11          | 73.32 ± 0.12                |\n| | LISA + LoRA | 45.34 ± 0.41 | 25.55 ± 0.66 | 72.64 ± 0.24 | \n|                | LISA     | **46.21 ± 0.12**   | 26.06 ± 0.08          | **75.30 ± 0.11**                 |\n\n**Weakness 4: Unintuitive results.**\n> The most surprising result is that LISA outperforms full finetuning quite often, which is against our intuition. It also casts a doubt: how do you choose the hyper-parameters (like Table 15)? And what is the error bar for the main results.\n\nThank you for the question. The intuition of this surprising effect can be attributed to the implicit regularization effect of LISA's layerwise freezing strategy, as mentioned in lines 245-250, where LISA and LoRA favor different types of tasks, implying the regularization tendency under different freezing strategies.\n\nThe hyperparameter search process can be found in Appendix B.3, and we have conducted additional multi-trial experiments to address the raised concerns, please refer to the attached PDF for detailed results.\n\n> Reference:  \n> [1] Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, page 1527–1554, Jul 2006.  \n> [2] Liao, Baohao, Yan Meng, and Christof Monz. \"Parameter-efficient fine-tuning without introducing new latency.\" arXiv preprint arXiv:2305.16742 (2023).  \n> [3] Li, Pengxiang, et al. \"OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning.\" arXiv preprint arXiv:2405.18380 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you very much for all the constructive comments and suggestions! To further address every reviewer's concerns, we have included additional results in the attached PDF file. Please kindly refer to the file for more details.

Meta-review: The paper proposes a new PEFT method called LISA (Layerwise Importance Sampled AdamW). The authors observed that weight norms are skewed while fine-tuning the models. Inspired by the observation, the authors randomly freeze most middle layers during the optimization. The authors show that the proposed method uses similar or less memory than LoRA  while outperforming LoRA or full parameter fine-tuning in some cases.

Reasons to accept: All three reviewers have a consensus that the paper is well-organized and well-written. Also, they have a consensus that the problem to solve in the paper is important. Reviewer Rt8C and K468 mentioned that the experimental results supported the authors' claim well. Reviewer K468 had a question why E + H layers are handled differently, and the authors provided additional experiments why they are important. Reviewer kABp had concerns about novelty, unfair comparison and missing ablations. The authors clarified they were misunderstanding and provided additional information. Reviewer kABp increased the score based on the rebuttal.

Based on all those, the paper provides a more effective and more efficient way to fine-tune LLMs with supporting experiments which would be beneficial to the research community to enhancing the fine-tuning methods.
Decision: Accept (poster)