RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Large Language Models
In the realm of natural language processing (NLP), the Transformer architecture has been a revolutionary technique, initially known for its effectiveness in machine translation tasks (Vaswani et al., 2017 ) . With the inception of models like BERT (Devlin et al., 2019 ) and GPT-2 (Radford et al., 2019 ) , the approach shifted towards pre-training on extensive corpora, which led to significant performance enhancements in downstream fine-tuning tasks (Brown et al., 2020 ; Raffel et al., 2020 ; Zhang et al., 2022 ; Scao et al., 2022 ; Almazrouei et al., 2023 ; Touvron et al., 2023a , b ; Chiang et al., 2023 ; Biderman et al., 2023 ; Jiang et al., 2024 ) . However, the growing number of parameters in these models results in a huge GPU memory consumption, rendering the fine-tuning of large scale models ( \geq 65 B) infeasible under low resource scenarios. This has prompted a shift towards more efficient training of LLMs.
Parameter-Effieient Fine-Tuning
Parameter-efficient fine-tuning (PEFT) methods adapt pre-trained models by fine-tuning only a subset of parameters. In general, PEFT methods can be grouped into three classes: 1) Prompt Learning methods (Hambardzumyan et al., 2021 ; Zhong et al., 2021 ; Han et al., 2021 ; Li and Liang, 2021 ; Qin and Eisner, 2021 ; Liu et al., 2021a ; Diao et al., 2022 ) , 2) Adapter methods (Houlsby et al., 2019 ; Diao et al., 2021 ; Hu et al., 2022 ; Diao et al., 2023c ) , and 3) Selective methods (Liu et al., 2021b , b ; Li et al., 2023a ) .
Prompt learning methods emphasize optimization of the input token or input embedding with frozen model parameters, which generally has the least training cost among all three types.
Adapter methods normally introduce an auxiliary module with much fewer parameters than the original model, where updates will only be applied to the adapter module during training.
Compared with them, selective methods are more closely related to LISA, which focuses on optimizing a fraction of the model’s parameters without appending extra modules.
Recent advances in this domain have introduced several notable techniques through layer freezing. AutoFreeze (Liu et al., 2021b ) offers an adaptive mechanism to identify layers for freezing automatically and accelerates the training process. FreezeOut (Brock et al., 2017 ) progressively freezes intermediate layers, resulting in significant training time reductions without notably affecting accuracy. The SmartFRZ (Li et al., 2023a ) framework utilizes an attention-based predictor for layer selection, substantially cutting computation and training time while maintaining accuracy. However, none of these layer-freezing strategies has been widely adopted in the context of Large Language Models due to their inherent complexity or non-compatibility with modern memory reduction techniques (Rajbhandari et al., 2020 ; Rasley et al., 2020 ) for LLMs.
Low-Rank Adaptation (LoRA)
In contrast, the Low-Rank Adaptation (LoRA) technique is much more popular in common practices of LLM training (Hu et al., 2022 ) . By employing low-rank matrices, LoRA reduces the number of trainable parameters, thereby lessening the computational burden and memory cost. One key strength of LoRA is its compatibility with models featuring linear layers, where the decomposed low-rank matrices can be merged back into the original model. This allows for efficient deployment without changing the model architecture. As a result, LoRA can be seamlessly combined with other techniques, such as quantization (Dettmers et al., 2023 ) or Mixture of Experts (Gou et al., 2023 ) . Despite these advantages, LoRA’s performance is not universally comparable with full parameter fine-tuning. There have been tasks in Ding et al. ( 2022 ) that LoRA achieves much worse performance than full parameter training on. This phenomenon is especially evident in large-scale pre-training settings (Lialin et al., 2023 ) , where to the best of our knowledge, only full parameter training was adopted for successful open-source LLMs (Almazrouei et al., 2023 ; Touvron et al., 2023a , b ; Jiang et al., 2023 ; Zhang et al., 2024 ; Jiang et al., 2024 ) .
Large-scale Optimization Algorithms
Besides approaches that change model architectures, there have also been efforts to improve the efficiency of optimization algorithms for LLMs.
One branch of such is layerwise optimization, whose origin can be traced back to decades ago, where Hinton et al. ( 2006 ) pioneered an effective layer-by-layer pre-training method for Deep Belief Networks (DBN), proving the benefits of sequential layer optimization. Bengio et al. ( 2007 ) furthered this concept by showcasing the advantages of a greedy, unsupervised approach for pre-training each layer in deep networks. For large batch settings, You et al. ( 2017 , 2019 ) propose LARS and LAMB, providing improvements in generalization behaviors to avoid performance degradation incurred by large batch sizes. Nevertheless, in most settings involving LLMs, Adam (Kingma and Ba, 2014 ; Reddi et al., 2019 ) , and AdamW (Loshchilov and Hutter, 2017 ) are still the dominant optimization methods currently.

Recently, there have also been other attempts to reduce the training cost of LLMs. For example, MeZO (Malladi et al., 2023 ) adopted zeroth order optimization that brought significant memory savings during training. However, it also incurred a huge performance drop in multiple benchmarks, particularly in complex fine-tuning scenarios. In terms of acceleration, Sophia (Liu et al., 2023 ) incorporates clipped second-order information into the optimization, obtaining non-trivial speedup on LLM training. The major downsides are its intrinsic complexity of Hessian estimation and unverified empirical performance in large-size models (e.g., \geq 65 B). In parallel to our work, Zhao et al. ( 2024 ) proposed GaLore, a memory-efficient training strategy that reduces memory cost by projecting gradients into a low-rank compact space. Yet the performance has still not surpassed full-parameter training in fine-tuning settings. To sum up, LoRA-variant methods (Hu et al., 2022 ; Dettmers et al., 2023 ; Zhao et al., 2024 ) with AdamW (Loshchilov and Hutter, 2017 ) is still the dominant paradigm for large-size LLM fine-tuning, the performance of which still demands further improvements.
Motivation
To understand how LoRA achieves effective training with only a tiny portion of parameters, we carried out empirical studies on multiple models, specially observing the weight norms across various layers. We fine-tune it on the Alpaca-GPT4 dataset (Peng et al., 2023 ) . During the training, we meticulously recorded the mean weight norms of each layer \ell at every step t after updates, i.e.

\mathbf{w}^{(\ell)}\triangleq\texttt{mean-weight-norm}(\ell)=\frac{1}{T}\sum_{t=1}^{T}\|\boldsymbol{\theta}^{(\ell)}_{t}\|_{2}

Figure 2 presents these findings, with the x-axis representing the layer id, from embedding weights to the final layer, and the y-axis quantifying the weight norm. The visualization reveals one key trend:

The embedding layer (wte and wpe layers for GPT2) or the language model (LM) head layer exhibited significantly larger weight norms compared to intermediary layers in LoRA, often by a factor of hundreds. This phenomenon, however, was not salient under full-parameter training settings.

This observation indicates that the update emphasis of LoRA and full parameter training differ significantly, which can be attributed to the difference in their learned knowledge. For example, in embedding layers, tokens with similar meanings, i.e. synonyms, can be projected into the same embedding space and converted to similar embeddings. LoRA may capture this similarity in language and “group” them in the low-dimension space, allowing frequent features of language meanings to be promptly identified and optimized. The price is LoRA’s limited representation power restricted by its intrinsic low-rank space. Other possible explanations can also justify this phenomenon. Despite various interpretations of this observation, one fact remains clear: LoRA values layerwise importance differently from full parameter tuning .
Layerwise Importance Sampled AdamW (LISA)
To exploit the aforementioned discovery, we aspire to simulate LoRA’s updating pattern via sampling different layers to freeze. This way, we can avoid LoRA’s inherent deficiency of limited low-rank representation ability and also emulate its fast learning process. Intuitively, given the same global learning rates across layers, layers with small weight norms in LoRA should also have small sampling probabilities to unfreeze in full-parameter settings, so the expected learning rates across iterations can stay the same. This is exactly the idea of importance sampling (Kloek and Van Dijk, 1978 ; Zhao and Zhang, 2015 ) , where instead of applying layerwise different learning rates \{\eta_{t}\} in full-parameter settings to emulate LoRA’s updates \{\tilde{\eta}_{t}\} , we apply sampling instead

\eta^{(\ell)}_{t}=\tilde{\eta}^{(\ell)}_{t}\cdot\frac{\tilde{\mathbf{w}}^{(\ell)}}{\mathbf{w}^{(\ell)}}\quad\Rightarrow\quad\eta^{(\ell)}_{t}=\eta^{(\ell)},p^{(\ell)}=\frac{\tilde{\mathbf{w}}^{(\ell)}}{\mathbf{w}^{(\ell)}}

This gives rise to our Layerwise Importance Sampling AdamW method, as illustrated in Algorithm 1 . In practice, since all layers except the bottom and top layer have small weight norms in LoRA, we adopt \{p_{\ell}\}_{\ell=1}^{N_{L}}=\{1.0,\gamma/N_{L},\gamma/N_{L},\dots,\gamma/N_{L},1.0\} in practice, where \gamma controls the expected number of unfreeze layers during optimization. Intuitively, \gamma serves as a compensation factor to bridge the difference between LoRA and full parameter tuning, letting LISA emulate a similar layerwise update pattern as LoRA. To further control the memory consumption in practical settings, we instead randomly sample \gamma layers every time to upper-bound the maximum number of unfrozen layers during training.
Memory Efficiency
To demonstrate the memory efficiency of LISA, showcasing that it has a comparable or lower memory cost than LoRA, we conducted peak GPU memory experiments.
Settings
To produce a reasonable estimation of the memory cost, we randomly sample a prompt from the Alpaca dataset (Taori et al., 2023 ) and limit the maximum output token length to 1024.
Our focus is on two key hyperparameters: LoRA’s rank and LISA’s number of activation layers.
For other hyperparameters, a mini-batch size of 1 was consistently used across all five models in Table 3 , deliberately excluding other GPU memory-saving techniques such as gradient checkpointing (Chen et al., 2016 ) , offloading (Ren et al., 2021 ) , and flash attention (Dao et al., 2022 ; Dao, 2023 ) .
All memory-efficiency experiments are conducted on 4 \times NVIDIA Ampere Architecture GPUs with 80G memory.
Results
Upon examining Table 1 , it is evident that the LISA configuration, particularly when enhanced with both the embedding layer (E) and two additional layers (E+H+2L), demonstrates a considerable reduction in GPU memory usage when fine-tuning the LLaMA-2-70B model, as compared to the LoRA method. Specifically, the LISA E+H+2L configuration shows a decrease to 75G of peak GPU memory from the 79G required by the LoRA Rank 128 configuration. This efficiency gain is not an isolated incident; a systematic memory usage decrease is observed across various model architectures, suggesting that LISA’s method of activating layers is inherently more memory-efficient.

In Figure 3 , it is worth noticing that the memory reduction in LISA allows LLaMA-2-7B to be trained on a single RTX4090 (24GB) GPU, which makes high-quality fine-tuning affordable even on a laptop computer. In particular, LISA requires much less activation memory consumption than LoRA since it does not introduce additional parameters brought by the adaptor. LISA’s activation memory is even slightly less than full parameter training since pytorch (Paszke et al., 2019 ) with deepspeed (Rasley et al., 2020 ) allows deletion of redundant activations before backpropagation.

On top of that, a reduction in memory footprint from LISA also leads to an acceleration in speed. As shown in Figure 4 , LISA provides almost 2.9\times speedup when compared with full-parameter tuning, and \sim 1.5\times speedup against LoRA, partially due to the removal of adaptor structures. It is worth noticing that the reduction of memory footprint in both LoRA and LISA leads to a significant acceleration of forward propagation, which emphasizes the importance of memory-efficient training.
Moderate Scale Fine-Tuning
LISA is able to achieve this significant memory saving while still obtaining competitive performance in fine-tuning tasks.
Settings
To demonstrate the superiority of LISA over LoRA, we employ the instruction-following fine-tuning task with Alpaca GPT-4 dataset, which consists of 52k instances generated by GPT-4 (OpenAI et al., 2023 ) based on inputs from Alpaca (Taori et al., 2023 ) . The effectiveness of fine-tuning was evaluated with MT-Bench (Zheng et al., 2023 ) , featuring 80 high-quality, multi-turn questions designed to assess LLMs on multiple aspects.

In our experiments, we assessed five baseline models: GPT2-Small (Radford et al., 2019 ) , TinyLlama (Zhang et al., 2024 ) , Phi-2 (Li et al., 2023b ) , Mistral-7B (Jiang et al., 2023 ) , LLaMA-2-7B, and LLaMA-2-70B (Touvron et al., 2023b ) . These models, varying in size with parameters ranging from 124M to 70B, were selected to provide a diverse representation of decoder models. Detailed specifications of each model are provided in Table 3 . For hyperparameters, we adopt rank 128 for LoRA and E+H+2L for LISA in this section of experiments, with full details available in Appendix A .
Results
Table 4 offers a comprehensive evaluation of three fine-tuning methods—Full Parameter Fine-Tuning (FT), Low-Rank Adaptation (LoRA), and Layerwise Importance Sampling AdamW (LISA)—across a diverse set of tasks including Writing, Roleplay, Reasoning, Math, Extraction, STEM, and Humanities within the MT-Bench benchmark. The results clearly demonstrate LISA’s superior performance, which surpasses LoRA and full parameter tuning in most settings. Notably, LISA consistently outperforms LoRA and full parameter tuning in domains such as Writing, STEM, and Humanities. This implies that LISA can be beneficial for tasks involving memorization, while LoRA partially favors reasoning tasks.
Large Scale Fine-Tuning
To further demonstrate LISA’s scalability on large-sized LLMs, we conduct additional experiments on LLaMA-2-70B (Touvron et al., 2023b ) .
Settings
On top of the aforementioned instruction-following tasks in Section 4.2 , we introduce an extra set of domain-specific fine-tuning tasks on mathematics and medical QA benchmarks. The GSM8K dataset (Cobbe et al., 2021 ) , comprising 7473 training instances and 1319 test instances, was utilized for the mathematics domain.
For the medical domain, we selected the PubMedQA dataset (Jin et al., 2019 ) , which includes 211.3K artificially generated QA training instances and 1K test instances.

The statistics of these datasets are summarized in Table 2 .
Evaluation on the PubMedQA dataset (Jin et al., 2019 ) utilized a 5-shot prompt setting, while the GSM8K dataset (Cobbe et al., 2021 ) assessment was conducted using a Chain of Thought (CoT) prompt, as suggested by recent studies (Wei et al., 2022 ; Shum et al., 2023 ; Diao et al., 2023b ) .
Regarding hyperparameters, we adopt rank 256 for LoRA and E+H+4L for LISA, where further details can be found in Appendix A .
Results
As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting. Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM. On top of that, LISA displays consistently higher performance than LoRA on all subtasks, underscoring LISA’s effectiveness across diverse tasks. The chart also contrasts the yellow LoRA line with the purple Vanilla line, revealing that in large models like the 70B, LoRA does not perform as well as expected, showing only marginal improvements on specific aspects. More fine-tuning experiments result in the Appendix B.1 .
Hyperparameters of LISA
The number of sampling layers \gamma and sampling period K are the two key hyperparameters of LISA. To obtain intuitive and empirical guidance of those hyperparameter choices, we conduct ablation studies using TinyLlama (Zhang et al., 2024 ) and LLaMA-2-7B (Touvron et al., 2023b ) models with the Alpaca-GPT4 dataset. The configurations for \gamma , such as E+H+2L, E+H+8L, were denoted as \gamma=2 and \gamma=8 . As for the sampling period — the number of updates per sampling interval — values were chosen from \{1,5,25,122\} , with 122 representing the maximum training step within our experimental framework. The findings, presented in Table 6 , reveal that both \gamma and K markedly affect the LISA algorithm’s performance. Specifically, a higher \gamma value increases the quantity of trainable parameters, albeit with higher memory costs. On the other hand, an optimal K value facilitates more frequent layer switching, thereby improving performance to a certain threshold, beyond which the performance may deteriorate. Generally, the rule of thumb is: More sampling layers and higher sampling period lead to better performance . For a detailed examination of loss curves and MT-Bench results, refer to Appendix C .
Sensitiveness of LISA
As LISA is algorithmically dependent on the sampling sequence of layers, it is intriguing to see how stable LISA’s performance is under the effect of randomness. For this purpose, we further investigate LISA’s performance variance over three distinct runs, each with a different random seed for layer selection. Here we adopt TinyLlama, LLaMA-2-7B, and Mistral-7B models with the Alpaca-GPT4 dataset, while keeping all other hyperparameters consistent with those used in the instruction following experiments in section 4.2 . As shown in Table 7 , LISA is quite resilient to different random seeds, where the performance gap across three runs is within 0.13 , a small value when compared with the performance gains over baseline methods.
Theoretical Properties of LISA
Compared with LoRA, which introduces additional parameters and leads to changes in loss objectives, layerwise importance sampling methods enjoy nice convergence guarantees in the original loss. For layerwise importance sampled SGD, similar to gradient sparsification (Wangni et al., 2018 ) , the convergence can still be guaranteed for unbiased estimation of gradients with increased variance. The convergence behavior can be further improved by reducing the variance with appropriately defined importance sampling strategy (Zhao and Zhang, 2015 ) . For layerwise importance sampled Adam, there have been theoretical results in (Zhou et al., 2020 ) proving its convergence in convex objectives. If we denote f as the loss function and assume that the stochastic gradients are bounded, then based on Loshchilov and Hutter ( 2017 ) , we know that AdamW optimizing f aligns with Adam optimizing f with a scaled regularizer, which can be written as

 f^{\rm reg}(\mathbf{w})\triangleq f(\mathbf{w})+\frac{1}{2}\mathbf{w}^{\top}\mathbf{S}\mathbf{w},

where \mathbf{S} is a finite positive semidefinite diagonal matrix.
Following existing convergence results of RBC-Adam (Corollary 1 in Zhou et al. ( 2020 ) ), we have the convergence guarantee of LISA in Theorem 1 .

Let the loss function f be convex and smooth. If the algorithm runs in a bounded convex set and the stochastic gradients are bounded, the sequence \{\mathbf{w}_{t}\}_{t=1}^{T} generated by LISA admits the following convergence rate:

\frac{1}{T}\sum_{t=1}^{T}f^{\rm reg}(\mathbf{w}_{t})-f^{\rm reg}_{*}\leq\mathcal{O}\left(\frac{1}{\sqrt{T}}\right),

where f^{\rm reg}_{*} denotes the optimum value of f^{\rm reg} .
Better Importance Sampling Strategies
As suggested by the theoretical intuition, the strategy of E+H+2L in Section 4.2 and E+H+4L in Section 4.3 may not be the optimal importance sampling strategy, given it still sampled intermediate layers in a uniformly random fashion. We anticipate the optimizer’s efficiency to be further improved when taking data sources and model architecture into account in the importance sampling procedure.
Conclusion
In this paper, we propose Layerwise Importance Sampled AdamW (LISA), an optimization algorithm that randomly freezes layers of LLM based on a given probability. Inspired from observations of LoRA’s skewed weight norm distribution, a simple and memory-efficient freezing paradigm is introduced for LLM training, which achieves significant performance improvements over LoRA on downstream fine-tuning tasks with various models, including LLaMA-2-70B. Further experiments on domain-specific training also demonstrate its effectiveness, showing LISA’s huge potential as a promising alternative to LoRA for LLM training.
Limitations
The major bottleneck of LISA is the same as LoRA, where during optimization the forward pass still requires the model to be presented in the memory, leading to significant memory consumption. This limitation shall be compensated by approaches similar to QLoRA (Dettmers et al., 2023 ) , where we intend to conduct further experiments to verify its performance.
A.1 Training Setup
In our experiments, we employ the LMFlow toolkit (Diao et al., 2023a ) * * * https://github.com/OptimalScale/LMFlow for conducting full parameter fine-tuning, LoRA tuning, and LISA tuning. We set the epoch number to 1 for both fine-tuning and continual pre-training scenarios. Additionally, we utilized DeepSpeed offload technology (Ren et al., 2021 ) to efficiently run the LLMs. All experiments were conducted on 8\times NVIDIA Ampere Architecture GPU with 48 GB memory.

In our study, we explored a range of learning rates from 5\times 10^{-6} to 3\times 10^{-4} , applying this spectrum to Full Parameter Training, LoRA, and LISA methods. For LoRA, we adjusted the rank r to either 128 or 256 to vary the number of trainable parameters, applying LoRA across all linear layers. Regarding the number of sampling layers \gamma , our selections were guided by GPU memory considerations as reported in LoRA studies (Hu et al., 2022 ) ; For the LISA algorithm, we selected \gamma=2 , and for experiments involving the 70B model, we opted for \gamma=4 . The sampling period ( K ), defined as the number of update steps per sampling interval, ranged from 1 to 50. This range was influenced by variables such as the size of the dataset, the batch size, and the total number of training steps. To manage this effectively, we partitioned the entire training dataset into K segments, thereby enabling precise regulation of the training steps within each sampling period.
A.2 Hyperparameter search
We commenced our study with a grid search covering (i) learning rate, (ii) number of sampling layers \gamma , and (iii) sampling period K . Noting the effective performance of the LoRA method, we set the rank value to either r=128 or r=256 .

The optimal learning rate was explored within the range \{5\times 10^{-6},10^{-5},5\times 10^{-5},6\times 10^{-4},3\times 10^{-4}\} , applicable to full parameter training, LoRA, and LISA.

Regarding the number of sampling layers \gamma , in alignment with Table 1 , we selected values that matched or were lower than LoRA’s GPU memory cost. Consequently, \gamma=2 was predominantly used in the LISA experiments, while \gamma=4 was chosen for the 70B model experiments.

For the sampling period K , we examined values within {1,3,5,10,50,80} , aiming to maintain the model’s update steps within a range of 10 to 50 per sampling instance. This selection was informed by factors such as dataset size, batch size, and total training steps.

The comprehensive results of our hyperparameter search, detailing the optimal values for each configuration, are presented in Table 8 .