RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
B.1 Instruction Following Fine-tuning
Table 9 provides detailed MT-Bench scores for the LLaMA-2-70B model discussed in Section 4.3 , demonstrating LISA’s superior performance over LoRA in all aspects under large-scale training scenarios. Furthermore, in Figure 6 , we observe that LISA always exhibits on-par or faster convergence speed than LoRA across different models, which provides strong evidence for LISA’s superiority in practice.

It is also intriguing to observe from Figure 5 that Vanilla LLaMA-2-70B excelled in Writing, but full-parameter fine-tuning led to a decline in these areas, a phenomenon known as the “Alignment Tax” (Ouyang et al., 2022 ) . This tax highlights the trade-offs between performance and human alignment in instruction tuning. LISA, however, maintains strong performance across various domains with a lower ”Alignment Tax“.
C.1 Sampling Layers \gamma
We conducted an ablation study on the LLaMA-2-7B model trained with the Alpaca-GPT4 dataset, setting the sampling period K=13 , so the number of samplings is exactly 10 . The study explored different configurations of sampling layers \gamma including {E+H+2L, E+H+4L, E+H+8L}. Figure 7 depicts the impact of the number of sampling layers \gamma on the training dynamics of the model. Three scenarios were analyzed: \gamma=2 (blue line), \gamma=4 (green line), and \gamma=8 (red line), throughout 120 training steps. Initially, all three configurations exhibit a steep decrease in loss, signaling rapid initial improvements in model performance. it’s clear that the scenario with \gamma=8 consistently maintains a lower loss compared to the \gamma=2 and \gamma=4 configurations, suggesting that a higher \gamma value leads to better performance in this context. The radar graph of the MT-Bench score also indicates that the \gamma 8 configuration yields the best conversational ability of the model in this ablation experiment.
C.2 Sampling Period K
Figure 8 displays the effects of varying sampling Period K on training a 7B-sized model using the 52K-entry Alpaca-GPT4 dataset. This graph contrasts loss curves for different sampling period K values: K=122 (green line), K=25 (red line), and K=13 (blue line) across 122 training steps. The results indicate that although each K value results in distinct training trajectories, their convergence points are remarkably similar. This finding implies that for a 7B model trained on a dataset of 52K instruction conversation pairs, a sampling period of K=13 is optimal for achieving the best loss curve and corresponding MT-Bench score radar graph.
C.3 Sensitiveness to Randomness
LLaMA-2-7B on Alpaca-GPT4 with update step per sampling period K=13 , and sampling layers \gamma=2 , run 3 times with different random layer pick. Figure 9 shows that different random selections of layers slightly affect the training process but converge similarly. Despite initial fluctuations, the loss trends of three runs—distinguished by blue, green, and red lines—demonstrate that the model consistently reaches a stable state, underscoring the robustness of the training against the randomness in layer selection.
Figure 1: Training loss of LLaMA-2-7B model on Alpaca GPT-4 dataset with Full Parameter Training (FT), LoRA, and LISA.; Training loss of LLaMA-2-7B model on Alpaca GPT-4 dataset with Full Parameter Training (FT), LoRA, and LISA.
Figure 2: Layer-wise weight norms during training of GPT2 and LLaMA-2-7B Model with LoRA and Full Parameters training.; Layer-wise weight norms during training of GPT2 and LLaMA-2-7B Model with LoRA and Full Parameters training.
Figure 3: GPU memory consumption of LLaMA-2-7B with different methods and batch size 1.; GPU memory consumption of LLaMA-2-7B with different methods and batch size 1.
Figure 4: Single-iteration time cost of LLaMA-2-7B with different methods and batch size 1.; Single-iteration time cost of LLaMA-2-7B with different methods and batch size 1.
Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.; Different aspects of LLaMA-2-70B model on MT-Bench.
Figure 6: Loss curves for LoRA, LISA and full-parameter tuning on the Alpaca-GPT4 dataset across different models.; Loss curves for LoRA, LISA and full-parameter tuning on the Alpaca-GPT4 dataset across different models.
Figure 7: Comparison of loss curves for the \gamma (number of sampling layers) ablation experiment.; Comparison of loss curves for the \gamma (number of sampling layers) ablation experiment.
Figure 8: Comparison of loss curves for the sampling period K ablation experiment.; Comparison of loss curves for the sampling period K ablation experiment.
Figure 9: Comparison of loss curves for random variance ablation experiment, indicating the loss metric over steps.; Comparison of loss curves for random variance ablation experiment, indicating the loss metric over steps.
The chart illustrates peak GPU memory consumption for various model architectures and configurations, highlighting differences across models. In the table, the LISA configuration is specifically labeled: “E” denotes the embedding layer, “H” represents the language model head layer, and “2L” indicates two additional standard layers. *: Model parallelism is applied for the 70B model.
The statistics of datasets. # Train and # Test denote the number of training and test samples respectively.
Baseline Model Specifications
Comparison of Language Model Fine-Tuning Methods on the MT-Bench score.
MT-Bench, GSM8K, and PubMedQA score for LLaMA-2-70B with three fine-tuning methods.
The MT-Bench scores derived from different LISA hyperparameters combination. All settings adopt learning rate \eta_{0}=10^{-5} .
The MT-Bench scores derived from varying random seeds for layer selection.
The hyperparameter search identified optimal settings for each method: FP (Full Parameter Training), LoRA, and LISA.
Mean score of three fine-tuning methods over three seeds for LLaMA-2-70B on the MT-Bench.