Understanding Layer Significance in LLM Alignment
Authors: Guangyuan SHI
ZEXIN LU
Xiaoyu DONG
Wenlong Zhang
Xuanyu Zhang
Yujie Feng
Xiao-Ming Wu
Venue: COLM 2025 (2025)
Keywords: Layer Significance
Language Model Alignment
TL;DR: We propose an algorithm to identify which layers within LLMs are most critical to the alignment process.

Abstract: Aligning large language models (LLMs) through supervised fine-tuning is essential for tailoring them to specific applications. Recent studies suggest that alignment primarily adjusts a model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To uncover how alignment affects model behavior at a granular level, we propose identifying which layers within LLMs are most critical to the alignment process. Our approach, named ILA, involves learning a binary mask for the parameter changes in each layer during alignment, as an indicator of layer significance. Experimental results reveal that, despite substantial differences in alignment datasets, the important layers of a model identified by ILA exhibit nearly 90\% overlap, highlighting fundamental patterns in LLM alignment. The results also indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss. Finally, we discuss how these findings extend from LLM alignment to reasoning. The source code is available at https://github.com/moukamisama/ILA.

Published as a conference paper at COLM 2025
Understanding Layer Significance in LLM Alignment
Guangyuan Shi', Zexin Lu', Xiaoyu Dong', Wenlong Zhang', Xuanyu Zhang?,
Yujie Feng', Xiao-Ming Wul
'The Hong Kong Polytechnic University, Hong Kong S.A.R., China
"Xiaohongshu Inc., China
(guang-yuan, shi, zexin.lu, xiaoyu.dong, senlong.zhang yujie. feng)
@connect. boonnect.polyu.fi. hk, xyzêmail bnu. edu. en, xiao-ming. wu@polyu.ede edu.hk
Abstract
Aligning large language models (LLMs) through supervised fine-tuning is essential for tailoring them to specific applications. Recent studies suggest that alignment primarily adjustt model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To uncover how alignment affects model behavior at a granular level, we propose identifying which layers within LLMs are most critical to the alignment process. Our approach, named ILA, involves learning binary mask for the parameter changes in each layer during alignment,
indicator of layer significance. Experimental results reveal that, despite substantial differences in alignment datasets, the important layers of a model identified by ILA exhibit nearly 90% overlap, highlighting fundamental patterns in LLM alignment. The results also indi- cate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine- tuning efficiency with minimal performance loss. Finally, we discuss how these findings extend from LLM alignment to reasoning. The source code is available at https:// //gi thub. com/ www.aaisamaaaa/ooaama...
1 Introduction
Aligning large language models (LLMs) with specific requirements is essential for enhancing their utility across diverse applications (Luo et al., 2023a; Yu et al., 2023; Luo et al., 2023b; Li et al., 2023; Liu et al., 2024a; 2022; Feng et al., 2023). Fine-tuning LLMs during the alignment process significantly improve the models' capabilities to meet targeted needs (Bubeck et al., 2023). Typically, alignment involves fine-tuning the model on diverse datasets, which may include both human-curated (Rajani et al., 2023) and L.MM-eenerate (Taori et al., 2023) data, using approaches like instruction tuning (Wei et al., 2021) and preference learning (Bai et al., 2022; Rafailov et al., 2024). Given the significant cost associated with full parameter fine-tuning, parameter-efffiient fine-tuning (PEFT) (Hu et al., 2021; Chen et al., 2022; Pan et al., 2024) methods have emerged popular alternative, offering balance between performance and resource efficiency.
Understanding what LLMs actually learn during the alignment process is crucial. Zhou et al. (2023) posits that the majority of knowledge and capabilities are developed during the pre-training phase, with alignment primarily serving to refine the model's conversational style and formatting. Using a well-selected set of 1,000 training examples for supervised fine- tuning (SFT), they successfully produced a high-quality aligned model. Similarly, Lin al. (2023) investigated the token distribution of LLMs before and after alignment and found that most changes were related to "stylistic tokens", ", such as discourse markers and transition words, while the knowledge-intensiv content largely remained untouched, coming from the base pre-trained model. These findings imply that the alignment process mainly adjusts the model's presentation style rather than modifying its foundational knowledge.
1

---

Published as a conference paper at COLM 2025
Alpaa-GPT4
LDMA
No Rabots
Figure 1: Layer importance rankings by our ILA algorithm for LLAMA 2-7B and Mistral- 7B-v0.1 across Alpaca-GPT4, LIMA, and No Robots datasets. Top 75% layers by score (sj) are considered important. X-axis: transformer block index; y-axis: linear layer names. The figure highlights two findings: (1) High overlap (90%) in important layers across datasets (Table2) suggests shared alignment needs, regardless of substantial differences in dataset content; (2) Important layers differ by architecture, reflecting model-specific dynamics.
Table 1: Impact of fine-tuning different components of LLAMA 2-7B on alignment perfor- mance using the LIMA dataset. Evaluated on MMLU (5-shot) and GPT--4o scores for Vicuna and MT--Bench prompts. Tuned components include attention projections (Wq. We, Wo, Wo) and feed-forward layers (Wup, Wdown/ Wgaae).
ATT (Wyr W3, Wor W₂)
ATT2 (Wy, W₁, W,)
FFN
ALL (LoRA)
(Wap, Wdownr W yatte)
MMLU 1
42.03
42.65
43.06
43.18
Vicuna f
5.21
5.13
5.40
5.43
MT-Bench f
3.31
3.35
3.41
3.45
To gain deeper understanding of LLM alignment, we analyze this process at the level of model parameters. We conducted a pilot study to investigate the impact of different model components alignment performance, by fine-tuning only specific layers and evaluating the resulting performance, as presented in Table 1. The results clearly indicate that fine- tuning different components of the LLM leads to considerable performance differences. For instance, fine-tuning the feed-forward network (FFN) layers achieves performance similar to fine-tuning all linear layers (i.e., with LoRA), whereas focusing solely on the attention layers causes a notable drop in performance. This observation shows the complexity of layer-specific contributions to LLM alignment, highlighting the need for detailed analysis.
To address this, we propose identifying the layers that are most critical to alignment performance during the SFT process. We develop novel approach, ILA, for identifying the important layers for LLM alignment. Specifically, we learn a binary mask for the parameter changes in each layer during the fine-tuning process, which serves an indicator of layer significance. A binary mask value of zero indicates that the corresponding layer has negligible influence during the process, while value of one denotes that the layer is crucial. We use gradient descent to learn the binary mask effectively and offer theoretical analysis of the optimization process. The main findings and significance of this work include:
Consistent layer importance ranking across different alignment datasets. We observe similar rankings of important layers during alignment for the same pre-trained model (see Fig. 1), even though the alignment datasets vary significantly in both content and size. This suggests that the alignment process endows the model with similar capabilities, corroborating previous research findings and offers new insights into LLM alignment.
Enhancing performance by freezigg unimportant layers. We show that freezing about 25% of unimportant layers can improve performance and that single search for layer importance ranking sufficient for different aligmment tasks using the same architecture.
Improving alignment efficiency through selective fine-tuning, Our findings show that fine-tuning only 10-30% key layers achieves performance comparable to fine-tuning all
2

---

Published as a conference paper at COLM 2025
linear layers. Additionally, integrating this approach with QLoRA allows tuning only 30-75% of key layers to maintain or enhance performance while cutting resource costs.
Broader implications beyond LLM alignment. Although our primary focus is on LLM alignment, the approaches and insights from this study have broader applicability. Our preliminary experiments on LLM reasoning reveal findings similar to those alignment, showcasing the significant potential of our methods to enhance the reasoning capabilities of LLMs, particularly in achieving test-time scaling (OpenAl, 2024; Welleck et al., 2024; Snell et al., 2024; Muennighoff et al., 2025).
Related Works
LLM Alignment. Pretrained language models encode general-purpose representations,
enabling transfer across diverse tasks (Qiu et al., 2024; Jiang et al., 2024; Nijkamp et al., 2022). Alignment methods like instruction tuning (Zhang et al., 2023c; Sun et al., 2023; Muennighoff et al., 2023) and preference learning (Hejna et al., 2023; Guan et al., 2022; Rafailov et al., 2024; Song et al., 2024; Li et al., 2024a) adapt these models to specific objectives. Recent studies have explored alignment mechanisms. LIMA (Zhou et al., 2023) showed that fine-tuning small datasets (e.g,, 1,000 examples) shapes behavior without adding new knowledge, a finding echoed by others (Chen et al., 2023; Lee et al., 2023; Gudibande et al., 2023). Duan et al. (2023) connected instruction tuning to in-context learning via hidden state analysis, while URIAL (Lin et al., 2023) revealed that alignment mainly modifies stylistic tokens, ones. These insights suggest alignment imparts narrow,
preserving knowledge-cent targeted adjustments. Our work builds on this by identifying the specific layers most critical for alignment, offering more fine-grained understanding of how adaptation occurs.
Parameter Efficient Fine-Tuning (PEFT). Fine-tuning large language models with billions or trillions of parameters is computationally expensive (Brown et al., 2020; Fedus et al., 2022). Parameter-efffiient fine-tuning (PEFT) methods address this by updating specific components (Zaken et al., 2021; Zhao et al., 2020; Ansell et al., 2021; Guo et al., 2020) or using soft prompts (Lester et al., 2021; Li & Liang, 2021; Asai et al., 2022). Techniques such as BitFit (Zaken et al., 2021), Adapters (Houlsby et al., 2019), LoRA (Hu et al., 2021), and their variants (Zhang et al., 2023b; Meng et al., 2024) reduce cost while maintaining transferability. Recent work (Li et al., 2024b; Hui et al., 2024; Pan et al., 2024; Xu & Zhanng, 2024; Panda et al., 2024) shows that selectively fine-tuning certain regions yields strong results, though random masking often lacks consistency. However, most PEFT approaches overlook parameter importance and lack prioritization. Our method addresses this by ranking layer importance, enabling targeted fine-tuning to improve performance with minimal cost.
Layer Analysis in Model Compression. Model compression techniques use structured prun- ing (Xia et al., 2022; Liu et al., 2024b; van der Ouderaa et al., 2023) and layer analysis to im- prove efficiency. Methods like Sheared LLaMA (Xia et al., 2023) and LLM-Streamline (Chen et al., 2024) show that pruning layers, heads, and dimensions can significantly reduce model size with minimal performance loss. Layer importance studies (Zhang et al., 2024; Gromov et al., 2024) further support the removal of less critical components for scalability. However, these efforts focus on reducing model size rather than optimizing parameter updates for task-specific alignment.
contrast, our work targets alignment fine-tuning by prioritiz- ing parameter updates through skill localization (Panigrahi et al., 2023; Voita et al., 2023), improving both alignment efficiency and robustness.
3 Quantifying Layer Significance in LLM Alignment
To understand layer significance in LLM alignment, we propose ILA, a method to identify important layers by learning a binary mask that indicates each layer's significance.
Consider a pre-trained LLM model with parameters O, composed of N layers, i.e., 0g- = {o) \N { The model is fine-tuned an alignment dataset D ) with loss function ((0). After t training iterations, the model parameters are updated to 0, 0, + A0,, where AO; represents the change in parameters till iteration t. Define a binary mask 7: {hing e
3

---

Published as a conference paper at COLM 2025
Algorithm 1: Identify the Important Layers for Alignment (ILA)
Input: Pre-trained model parameters 0,, learning rate a, the initial importance score vector { - the number of insignificant layers K, the low-rank matrices
Ag, Bg for the LoRA algorithm. (FFT is special case of LoRA with full rank)
for iteration = 1, 2, do Update A, A, 1 aV A,, ((),), B, B, aV B,,,, (LoRA); , (10,-) (FFT);
Or Update 0 0,-1
if Training has become stable then Solve the optimization problem in Eq. (7) by gradient descent to find
{ Stop training;
$i
end end
{0, 10 that encodes layer-wise importance information. We apply y to A0, and define
omask 9. + 7 @ AO,
(1)
where @ is component-wise multiplication. The binary mask applied to retain the changes in crucial layers while eliminating the rest. Below
provide formal definition of the
conditions under which training attains stability after an adequate number of iterations. Definition 1 (e-stable). Ve O, the model is said to be e-stable at iteration T if, for any T, the loss function satisfies the condition
[B,[C(z; - E [2(; 0t) <e,
(2)
where [- denotes the expectation with respect to the alignment dataset DD
Once training stabilizes, we can identify the layers that are crucial for the alignment task. Definition 2 (Layer Importance). The binary mask y is defined as the solution to the following optimization problem:
Y arg min [2[[(z; @.mask)] e, },, s.t. st.l H,
(3)
T
where H is a hyper-parameter that serves constraint to limit the number of important layers.
Efficiently Identifying the Importance Layers (Alg. 1). Due to the high cost fine-tuning large models, to address the optimization problem in Eq. (3), employ the LoRA (Hu et al., 2021) algorithm, which utilizes low-rank decomposition matrices to represent the change in model parameters till iteration (A0;). Specifically, LoRA utilizes two trainable low-rank matrices, B, 4R,;x,, and A R to estimate the change of the ,th layer:
so B- B,A,,
where ß is the scalar hyperparameter of LoRA. With the binary mask n,, the tth layer is updated by
0f = 06 + B B- B'A
(5)
To ease the optimization of 7, we re-parametrize each of its each components of as the output of Sigmoid function, i.e., 7| assf). Then, the update of the ,th layer becomes
0 = 0/ ( B/A
(6)
Let s; [ ee N The optimization problem in Eq. (3) becomes
s min E E2
*
We use gradient descent to optimize st, yielding sf as the importance score of the 1th layer. A larger value of si indicates yf is closer to one, signifying higher importance of the tt layer.

---

Published as a conference paper at COLM 2025
1%, training milestones
25% training milestones
50% training milestones
75% training milestones
100% training millesonnes
Figure 2: Layer importance rankings of LLAMA 2-7B during fine-tuning on LIMA at 1%, 25%, 50% 75,, and 100% milestones. X-axis: transformer block index; y-axis: linear layer names. Jaccard similarities are provided in Table 4.
Assumption 3.1 (Lipschit-conttnnuuss), The loss function L(0) R R continuously differentiable and L-smooth with constant L1 > 0 such that
[[(()) [()') ll2 S L l'
(8)
In addition, L(0) has an L-Lipschitz continuous gradient with constant L2 such that
|VV(()) VL(0') - L2|10- 0'|l. Assumption 3.2. For any t > T, 0 is e-stable. We assume there is a constant R such that
(9)
||e - 01+1112 S Re,
(10)
and there is a constant Q such that ||0:l22 - for any myytT T. Theorem 3.1. Let e > O be sufficiently small such that 0r is e-stable. Suppose Assumptions 3.1 and 3.2 hold. For any ! >> T, we assume that Vi, E [0,1] 1}. Let 7 denote the result of ' after one step of gradient descent, i.e., 7/- = 7V- BV ((0mass) Then we have
1 < B(Q RE)
(11)
This theorem demonstrates that when oy is c-stable, solving the optimization problem in Eq. (3) for any > yields similar results. This is because, after step of gradient descent, the difference between " and T+1 is smaller than a sufficiently small number. The proof is provided in Appendix A, and empirical results supporting this are shown in Fig. 2.
Leveraging Layer Importance Rankings. The identified rankings of layer importance can be leveraged to enhanc both the performance and efficiency of LLM alignment. To maximize performance, prioritize fine-tuning the significant layers while freezing those deemed less important. For efficiency, focus on the layers most critical to model success. Detailed experiments and analyses are presented in Sec. 4.
4 Experiments and Findings
4.1 Experimental Setup
Datasets. We use three alignment datasets Aaaazetts-Alpaaa-GPP- (Peng et al., 2023), LIMA (Zhou et al., 2023), and No Robots (Rajani et al., 2023). See Appendix B for details.
Models and Baselines. We use LLAMA 2-7B/133 (Touvron et 2023), Llama 3.1-8B (Dubey et al., 2024), and Mistral-7B-v0.11 (Tiang et al., 2023). Baselines include LoRA (Hu et al., 2021), AdaLoRA (Zhang et al., 2023a), and full fine-tuning. See Appendix B for details.
Evaluation. We evaluate (1) language understanding using MMLU (Hendrycks et al., 2021) and Hellaswag (Zellers et al., 2019), and (2) conversational ability using MT-Bench (Zheng et al., 2023) and Vicuna (Chiang al., 2023), with scoring conducted by GPT-4o. Further details are provided in Appendix B.
4.2 Layer Importance Rankings in LLM Alignment
In this subsection, we applied ILA to rank important layers during alignment across three datasets No Robots, LIMA, and Alpaca-GPT4 (Fig. We also analyzed layer importance
5

---

Published as a conference paper at COLM 2025
Table 2: Jaccard similarities of the top 75% highest-scoring layers identified as important during fine-tuning of LLAMA 2-7B and Mistral-7B on various datasets.
LLAMA 2-7B
Mistral-7B
Datasets
LIMA
No Robots
Alpaca-GPT4
LIMA
No Robots
Alpaca-GPT4
LIMA No Robots
0.91
0.90
Alpaca-GPT4
0.90
0.90
0.89
0.93
Table 3: Jaccard similarities of the top 75% important layers in LLAMA 2-7B fine-tuned on the LIMA dataset using differ- ent random seeds.
Table 4: Jaccard similarities the top 75% important layers identified at different stages of LLAMA 2-7B fine- tuning on the LIMA dataset.
Training Milestones
1%
25%
50%
75%
100%
Random Seed
seed1
seed2
seed3
1%
25%
0.69
seed1
50%
0.70
0.91
seed2
0.92
75%
0.69
0.90
0.92
seed3
0.91
0.91
100%
0.69
0.91
0.92
0.93
rankings at different training milestones (Fig.2). To quantify similarity between sets of important layers, we used the Jaccard similarity coefficient, defining the top 75% highest- scoring layers as the important set S. The similarity between two sets, S and S2, is given [S,ss]
by: [(S,, S2)
where J indicates identical sets, and J indicates no overlap. [S,US2|
Consistency in Layer Importance Rankings Across Different Datasets. Our findings show strong consistency in layer importance rankings: (1) highly similar important layers are identified across different alignment datasets, (Fig.1, Table2); (2) the rankings remain stable across different random seeds for y (Table 3); and (3) similar layers can be identified even at the beginning stages of training, such as completion of 25% (Fig.2, Table4).
These results confirm the robustness of ILA, which consistently identifies stable and overlap- ping layers across datasets. This aligns with recent findings that alignment largely involves stylistic token shifts (Lin et al., 2023). In essence, alignment seeks similar capabilities, as evidenced by our observation that important layers remain stable across different datasets. This underscores the relevance of our algorithm to the fundamental objectives of alignment.
Feed-Forward Network (FFN) layers are more important than attention layers. Our layer ranking results (Fig. 1) consistently demonstrate that Ferd-Forward Network (FFN) layers are more important than attention layers. This observation aligns with previous stud- ies (Geva et al., 2020; Meng et al., 2022), which identify FFNs as key memory modules responsible for encoding factual knowledge, while attention layers primarily facilitate token interactions. This distinction explains why FFN layers consistently achieve higher impor- tance rankings in our results. Moreover, our findings suggest several practical implications: (1) Model compression should prioritize retaining upper FFN layers. (2) Knowledge editing may be more effective when targeting middle-to-upper FFN layers.
4.3 Enhancing Alignment Performance through Freezing Unimportant Layers
To leverage layer importance rankings, we excluded less important layers that could nega- tively impact fine-tuning, removing approximately 25% of unimportant layers. The main results on No Robots are in Table 5, with additional results for LLAMA 2-13B (see Table 18) and main results Alpaca-GPT4 (see Table 20), and LIMA (see Table 19) datasets in Appendix C.5. Key observations include:
(1) Freezing unimportant layers can enhance performance. ILA consistently outperformed LoRA and full fine-tuning on most metrics, with freezing 25% of unimportant layers yielding better results than tuning all layers. (2) A single search for layer importance ranking suffices for a given architecture. Rankings were stable across alignment tasks, allowing us to compute it on the No Robots dataset and apply it to others.

---

Published as a conference paper at COLM 2025
Table 5: Comparison of LLAMA 2-7B, Mistral-7B-v0.1, and Llama 3.1-8B fine-tuned on the No Robots dataset, evaluated on MMLU (5-shot), Hellaswag (0-shot), and GPT-4o scores for Vicuna and MT-Bench prompts. Vicuna and MT-Bench results are averaged over three runs. Grey cells indicate improvements over the base model; best scores are in bold.
Language Understanding Conversational Ability
Models
Methods
MMLU f
Hellaswag
Vicuma
MT-Bench f
AdaLoRA
45.23
57.30
5.81
4.01
Full Fine-ture
45.72
57.69
6.12
4.18
LLAMA 2-7B
Full Fine-tune w/ ILA
45.98
57.87
6.35
4.37
LoRA
44.58
59,46
5,78
4.02
LoRA w/ ILA
45.78
59.65
5,90
4.33
AdaLoRA
62.13
61.68
6.21
4.69
Full Fine-tune
61.05
64.26
6.32
4.55
Mistral-7B-v0.11
Full Fine-tune w/ ILA
61.75
64.21
6.51
4.78
LoRA
61.95
62.90
6,25
4.68
LoRA w/ ILA
62.14
62.98
6.42
4.87
AdaLoRA
64.85
62.85
6.51
5.08
Full Fine-tune tune
64.44
63.65
6.50
5.11
Llama 3.1-8B
Full Fine-tune ILA
65.00
63.69
6.61
5.23
LoRA
64.95
60.77
6.33
4.58
LoRA w/ ILA
65.43
60.95
6.45
4.69
These results show that ILA improves fine-tuning efficiency by focusing on significant layers. Compared to AdaLoRA, even though we explored narrow range of the hyperparameter (target average rank of incremental matrices), our method performed better, suggesting that adjusting LoRA's matrix rank alone doesn't guarantee superior results, as also noted in (Dettmers et al., 2023).
Additionally, discussed in Sec. 4.2, the stability of the layer importance ranking across datasets means a single search is often sufficient. In our experiments, we computed the layer importance ranking using full training iterations on the No Robots dataset, and then directly applied this ranking to other datasets. Though dataset-specific rankings can further improve results (Table 16 in Sec. 6), the strong cross-dataset performance with one ranking demonstrates our approach's 's robustness and generalizability.
4.4 Enhancing Alignment Efficiency by Fine-tuning Only the Most Critical Layers
To investigate this issue, we fine-tune the top 10%, 20%, and 30% of the important layers of Mistral-7B-v0.11 as identified by ILA, on the No Robots dataset, and compare the results with those of the LoRA algorithm. The results demonstrate clear benefits in focusing on subset of important layers:
(1) Fine-tuning a small subset of the most important layers achieves competitive perfor- mance and enhances efficiency. Fine-tuning the top 10%, 20%, or 30% of layers results in only slight performance drop compared to full fine-tuning, Fine-tuning 30% of layers nearly matches full fine-tuning (Table 6), demonstrating that focusing on the most important layers ensures efficient fine-tuning with minimal performance loss.
(2) Our method can be applied enhance QLoRA, further reducing costs. When combined with QLoRA, our method fine-tunes only 30% 75% of the most important layers while maintaining or improving performance (Table 7), highlighting the efficiency of our approach in achieving comparable or better results with fewer layers.
These findings highlight the effectiveness of our layer selection strategy, optimizing resource use with minimal performance trade-offs. Our integration with QLoRA shows that fine- tuning a targeted subset of important layers improves both performance and memory efficiency during fine-tuning.
7

---

Published as a conference paper at COLM 2025
Table 6: Fine-tuning results of Mistral-7B-v0.11 on the No Robots dataset, evaluated on MMLU (5-shot), Hellaswag (0-shot), and GPT-4o scores for Vicuna and MT-Bench prompts (averaged over three runs). Percentages in parentheses denote the fraction of fine-tuned linear layers. Best results are in bold.
Language Understanding Conversational Ability
Models
Methods
MMLU f Hellaswag t
Vicuna f MT-Bench f
LoRA
61.95
62.90
6.25
4.68
Mistral-7B-v0.1
LoRA w/ ILA (10%)
62.09
61.94
5.99
4.39
LoRA w/ ILA (20%)
61.83
62.16
6.12
4.53
LoRA w/ ILA (30%)
61.89
62.79
6.27
4.75
Table 7: Comparison of QLoRA fine-tuning LLAMA 2-7B vs. selectively fine-tuning important layers identified by ILA. Evaluated on MMLU (5-shot), Hellaswag (0-shot), and GPT-4o scores for Vicuna and MT-Bench prompts (averaged over three runs). Grey cells indicate improvements over the base model by ILA.
Language Understanding
Conversational Ability
Datasets
Methods
MMLU f Hellaswag t
Vicuna 1
MT-Bench f
QLoRA
43.06
55.47
5.31
2.98
LIMA
QLoRA w/ ILA (75%)
43.48
55.95
5.56
3.19
QLoRA w/ ILA (30%)
44.01
55.82
5.17
3.01
For a clearer understanding of GPU memory savings, we measured memory consumption for QLoRA, LoRA, Full Fine-Tuning and versions fine-tuning only the key layers identified by ILA. Results are presented in Table 15 in Appendix C.1.
4.5 Ablation Study
Observation 1: Our layer importance ranking algorithm is effective. We evaluated our algorithm by comparing it to a baseline that fine-tunes all layers and three alternatives: (1) RL and RL 2, which randomly freeze top-K layers; (2) FL, freezing the first K layers; and (3) LL, freezing the last K layers. As shown in Table 8, these naive strategies underperform. In contrast, our method effectively identifies and freezes the least critical layers, yielding notable gains in both efficiency and performance.
Observation 2: The important scores calculated using LoRA are similar to those obtained through full fine-tuning. To assess whether LoRA-based approximations differ from full fine-tuning (FFT), we compared parameter updates from LoRA (i.e., Eq. (4)) and FFT (i.e., A0, 0, 0g). For both methods, we derived layer importance scores and selected the top 75% of layers, then calculated the Jaccard similarity between the layers. As shown in Table 9, LoRA achieves nearly 83% overlap with the important layers identified by FFT, reducing computational overhead while effectively ranking layer importance. The results show that LoRA provides strong approximation of AO; compared 0, -00-
Furthermore, we identified layers marked as important (Top 75%) by FFT but not by LoRA on LLaMA 2-7B using the No Robots dataset. These layers had low average ranks according to FFT (124 out of 168; best rank: 57), suggesting they are not among the most critical layers.
Observation Cross-dataset evaluation of layer importance enhances performance. Different datasets highlight subtle differences in important layers (Table 2). By intersecting the top-K least important layers from LIMA, No Robots, and Alpaca-GPT44 and freezing them during fine-tuning (Table 16, Appendix C.2), we found that cross-dataset evaluation yields better results than dataset-specific fine-tuning. This suggests that assessing layer importance across datasets leads to more robust fine-tuning.
Observation 4: Cross-model transfer of layer importance rankings is feasible but less effective than cross-dataset transfer. Models sharing architecture but trained different
8

---

Published as a conference paper at COLM 2025
Table 8: Performance comparison of ILA, random, and position-based layer selection for fine-tuning LLAMA 2-7B on the No Robots dataset. RL1/RLJ freeze K randomly selected layers (different seeds); FL and LL freeze the first and last K layers, respectively. Blue highlights indicate lower performance than ILA.
Language Understanding Conversational Ability
Methods
MMLU f Hellaswag f
Vicuna
MT-Bench f
LoRA
44.58
59.46
5.78
3,98
LoRA w/ RL 1
44.23
59.71
5.72
3,96
LoRA W/ RL
43.98
59.11
5.62
3.89
LoRA w/ FL
44.02
59.32
5.58
3,71
LoRA w/ LL
44.61
59.21
5.65
3,99
LoRA w/ ILA
45.78
59.65
5.90
4.15
Table 9: Jaccard Similarity between important layers selected using Full Fine- Fine-Tuning and
LoRA for LLAMA 2-7B. Top 75% gighest-scoring layers are determined as important layers.
Datasets
LIMA (FFT)
No Robots (FFT)
Alpaca-GPT4 (FFT)
LIMA (LoRA)
0.84
0.76
0.83
No Robots (LoRA)
0.78
0.80
0.81
Alpaca-GPT4 (LoRA)
0.82
0.83
0.86
Table 10: Jaccard similarities of the top 75% important layers across different models.
LLAMA 2-7B (LIMA)
LLAMA 2-7B (No Robots)
LLAMA 2-7B (Alpaca-GPT4)
Mistral-7B-v0.1 (LIMA) Mistral-7B-vO.11 (NoRobots) Mistral-7B-v0.1 (Alpaca-GPT4)
0.67
0.70
0.71
Table 11: Experimental results for Mistral-7B-v0.1 on the No Robots dataset, using layer importance rankings derived from LLAMA 2-7B.
Methods
MMLU
Hellaswag
Vicuna
MT-Bench
LoRA
61.95
62.90
6.25
4.68
LoRÅ w/ ILA (75%) (cross-model transfer)
62.10
63.21
6.29
4.72
LoRA w/ ILA (75%)
62.14
62.80
6.42
4.87
LoRA w/ ILA (30%) (cross-model transfer)
61.77
63.16
6.11
4.60
LoRA w/ ILA (30%)
61.89
62.79
6.27
4.75
datasets show strong agreement in important layers (Jaccard similarity of 0.90 for the top 75%, Table 2). This drops to 0.70 across architectures (Table 10), indicating reduced transfer- ability. Nonetheless, significant overlap suggests cross-architecture transfer remains viable. Fine-tuning Mistral-7B-v0.1 using rankings from LLAMA 2-7B on No Robots (TTble 11) confirms that cross-model transfer can still perform well. Further analysis can be found in Appendix C.3.
Observation 5: ILA is robust to the initialization of layer importance scores. As demon- strated Table 17 (Appendix C.4), our algorithm ILA is resilient varying initial impor- tance scores (5o 4.0, 2.0, 1.0), with minimal impact on final rankings. The stable Jaccard similarities for the top 75% of layers during LLAMA 2-7B fine-tuning on LIMA confirm reliable crgvereence regardless of initialization.
Observation The computation cost of ILA low. ILA runs two stages: Stage trains the model with LoRA until e-stability, and Stage 2 tunes importance weights (7i) with the backbone and LoRA frozen. For both LLAMA 2-7B and Mistral-7B-v0.11 (225 linear layers), Stage takes 6671 ms per iteration, and Stage 2 takes 5343 ms. Stage 2 finishes in 11 minutes (128 batches). Most cost lies in Stage 1, but Table shows only 50%50 training milestones needed for strong performance.

---

Published as a conference paper at COLM 2025
Table 12: Jaccard similarities of the top 75% important layers in LLAMA 2-7B.
Table 13: Comparison of LLAMA 2-7B fine- tuned on the OASSTI dataset.
No Robots
LIMA
Alpaca-GPTA
MMLU
Hellzwwag
Vicuna
MT-Beecc
LoRA
44.52
58.98
6.22
4.46
OASSTI
0.86
0.90
0.83
LoRA ILA (-75%)
45.24
59.22
6.75
4.51
Observation 7: ILA performs well on datasets with conflicting stylistic objectives. We evaluated ILA on the OASSTI dataset (Köpf et al., 2023), which features diverse linguistic styles from contributors with varied backgrounds. Using high-quality subset (as in QLoRA (Dettmers et al., 2023)), we fine-tuned LLAMA 22-7B and compared ILA's layer rankings with those from LIMA, No Robots, and Alpaca-GPT4. Table 12 shows high Jaccard Similarities (83-0.90), indicating consistent layer identification despite stylistic variation. These scores are slightly lower than the similarities among LIMA, No Robots, and Alpaca- GPT4 themselves (nearly 91% see Table 2), which is expected due to OASSTI's broader stylistic diversity. Fine-tuning only the top nearly 75% ILA-selected layers also yields consistent gains (see Table 13), confirming ILA's robustness to stylistic conflict.
Conclusion and Discussion: Beyond LLM Alignment
LD1O
s11
Table 14: Performance of Qwen225-7B-nnstruut on mathematical reasoning benchmarks after fine-tuning with the LIMÓ dataset.
1,94
Methods
MATH500
AIME
Figure Layer-wise importance rankings for
FFT
77.00
13.33
fine-tuned using the LIMO and s1.1 datasets, respectively.
FFT ILA
79.00
16.67
Our findings indicate that the alignment process of LLM imparts similar capabilities despite data variations. This complements prior research by revealing layer-specific roles and improving efficiency through strategic tuning and freezing of layers. Nonetheless, the approaches and insights derived from this study extend beyond LLM alignment.
LLM Reasoning and Test-time Scaling. Advanced models like ol (OpenAI, 2024), Deepseek R1 (Guo et al., 2025), and Kimi 1.5 (Team et al., 2025) have exhibited strong reasoning capa- bilities. Similar to alignment, reasoning seeks to further activate the knowledge acquired during pre-traiming. While alignmmett ensures that outputs align with human values, reasoning drives the model toward deeper inference for enhanced accuracy. Rather than scaling model size or training data, recent studies such as LIMO (Ye et al., 2025) and s1 (Muennighoff et al., performance by increasing the number of input tokens used for reasoning. Their findings show that even limited high-quality training data with of-houtt (CoT) examples can effectively enhance LLMs' reasoning capabilities.
2025) investigate test-fime scaling-boosse
To investigate whether our methods can provide insights into LLM reasoning comparable to those obtained in the context of LLM alignment, we conducted a set of preliminary experiments. Specifically, we fine-tuned the 20wen2.5-7BIInstruce model (Yang et al., 2024) on two distinct reasoning datasets, S1 (Muennighoff et al., 2025) and LIMO (Ye et al., 2025). Our analysis shows that the layers deemed most important by ILA on S1 and LIMO share a Jaccard similarity of nearly 86% (Fig.3), indicating that the model develops comparable reasoning capabilities despite differences in dataset characteristics. Furthermore, we find that freezing the least important 25% of layers consistently improves downstream performance (Tablel4), thereby underscoring the effectiveness of selective fine-tuning as a strategy for reasoning-centra tasks. Additional details regarding dataset construction, evaluation protocols, and benchmark selection are provided in Appendix D.
10

---

Published as a conference paper at COLM 2025
References
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross-lingual transfer. arXiv preprint arXi::2110.07560, 2021.
Akari Asai, Mohammadreza Salehi, Matthew E Peters, and Hannaneh Hajishirzi. Attempt: Parameter--fffcciet multi-task tuning via attentional mixtures of soft prompts. arXiv preprint ar wix::2205.11961,, 2022.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. ar Xiv preprint ar axXi::2204005862, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. Xiv preprint ar 1rXii:2303.127777
2023.
Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang, Revisiting parameter- efficient tuning: Are we really there yet? Xr preprint ar XrX::22002.07962, 2022.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. Xrviv preprint Xiv: 200000800, 2023.
Xiaodong Chen, Yuxuan Hu, and Jing Zhang, Compressing large language models by streamlining the unimportant layer. ar Xiv preprint arxi::2403.11133 19135, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, et al. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. arviv preprint arXiv:2305.14311 14314, 2023.
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, and Alexis Conneau. Self-training improves pre-training for natural language understanding. arXiv preprint ar 0rXi:22010.02194, 2020.
Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam. Exploring the relationship between in-context learning and instruction tuning. arXiv preprint ar arXi::2311.10367 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. Xiv preprint XXX::2407.21783 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-522 52700, 2022.
Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. Towards llm-driven dialogue state tracking. arXiv preprint ar XrX1::2310.1497 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXin:2012.14913, 2020.
11

---

Published as a conference paper at COLM 2025
Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers. arXio preprint (fXiv:2403.17887, 2024.
Lin Guan, Karthik Valmeekam, and Subbarao Kambhampati. Relative behavioral attributes: Filling the gap between symbolic goal specification and reward learning from human preferences. Xiv preprint ar Xxx::2210.159000 2022.
Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, arv preprint ar XrXii:2305.155 15717, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12943 2025.
Demi Guo, Alexander M Rush, and Yoon Kim. P'raamete--fffcceent transfer learning with diff pruning. ar preprint arXiv:2012.07463 2020.
Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive prefence learning: Learning from human feedback without rl Xiv preprint arXi::2310.1363) 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https: http://ppnneviieeeeeeeeeeeeeeeee fet. d7KB:jm33Gm)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Larous- silhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. I'arameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Tingfeng Hui, Zhenyu Zhang, Shuohuan Wang, Weiran Xu, Yu Sun, and Hua Wu. Hft: Half fine-tuning for large language models. arXio preprint ar 20o::2044..8466 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. ar preprint or arXiv:2310.06822 2023.
Guanying Jiang, Lingyong Yan, Haibo Shi, and Dawei Yin. The real, the better: Aligning large language models with online human behaviors. arXiv preprint 2xXio22405.0057,, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openas- sistant www.rrrrttiiosdmmorrtttiiie large language model alignment. Advances in Neural Information Processing Systems, 66:47669-47681. 2023.
Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint ar arXio:2308.0731, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXit:2104.08661, 2021.
Aaron Li, Satyapriya Krishna, and Himabindu Lakkaraju. More rlhf, more trust? on the impact human preference alignment on language model trustworthiness. Xiv preprint ar Xri:::2404.11 188700 2024a.
12

---

Published as a conference paper at COLM 2025
Haoling Li, Xin Zhang, Xiao Liu, Yeyun Gong, Yifan Wang, Yujiu Yang, Qi Chen, and Peng Cheng. Gradient-maske tuning elevates the upper limits of Ilm performance. Xiv preprint 17Xiv:2406.15330, 2024b.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! Xiv preprint ar Xtt:23355.06111 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arxiv preprint ar Xii:21101.00190, 2021.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khy- athi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. arXiv preprint ar arxo:2332.00155, 2023.
Qijiong Liu, Jieming Zhu, Quanyu Dai, and Xiao-Ming Wu. Boosting deep ctr prediction with a plug-and alug-and-play pre-trainer for news recommendation. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 2823-2833, 2022.
Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, Zhou Zhao, Rui Zhang, and Zhenhua Dong. Multimodal pretraining, adaptation, and generation for recommendation: survey, In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6566-6576, 2024a.
Yijin Liu, Fandong Meng, and Jie Zhou. Accelerating inference in large language models with unified layer skipping strategy. arviv preprint ar arXiv:2404.06954 2024b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arxiv preprint Xiv::3008.00.0 09583,, 2023a.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.085665 085668, 2023b.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint ar XrXo:22404.02948, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Adoances in neural information processing systems, 35:17359-17377 2022.
Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre. Octopack: Instruction tuning code large language models. arXiv preprint arXio:2308.07124 2023.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl:: Simple test-time scaling. arXiv preprint ar Xrii::2501.19393, 2025.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. Xiv preprint ar Xrt:22203..3474, 2022.
OpenAL. Learning to reason with LLMs, September 2024. URL learning-to- thee--iithhhhllle th-l/ms/.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. ar Xiv preprint ar arX:t:2403.17911, 2024. 2024.
13

---

Published as a conference paper at COLM 2025
Ashwinee Panda, Berivan Isik, Xiangyu Qi, Sanmi Koyejo, Tsachy Weissman, and Prateek Mittal. Lottery ticket adaptation: Mitigating destructive interference in llms. aiv preprint (#Xivo2406.16797, 2024.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp. 27011-27038 PMLR, 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023.
Yifu Qiu, Zheng Zhao, Yftah Ziser, Anna Korhonen, Edoardo M Ponti, and Shay B Co- hen. Spectral editing of activations for large language model alignment. arviv preprint ar arXiv:2405.09719, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https://huggingfaal igg/dattseets//uugigggaaa. robots, 2023.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute opti- mally can be more effective than scaling model parameters. arxiv preprint arXiv:2408.003: 03314, 2024.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990-18998, 2024.
Xianghui Sun, Yunjie Ji, Baochang Ma, and Xiangang Li. A comparative study between full-parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model. arXiv preprint ar arXiv:2304.08100 2023.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. at Xie preprint ar arXiv:2107.02137, 2021.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following Ilama model. /ttps://githhee httpg//tttt....tttoooeeeteeeeeee comm/tatsu-lab/sstt wtt.a/aaaaabaaaaaaaaaaaa.aaaaaaa 2023.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with 1lms. ar Xiv preprint arXiv:2501.12599 2025.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint #rXiv:2307.09288, 2023.
Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, Yuki M Asano, and Tijmen Blankevoort. The llm surgeon. Xiv preprint ar Xfxo:2212..7244, 2023.
Elena Voita, Javier Ferrando, and Christoforos Nalmpantis. Neurons in large language models: Dead, n-gram, positional. arXiv preprint arXiv:2309.0482. 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. ar Xiv preprint ar arX:t:2109.01652 2021.
14

---

Published as a conference paper at COLM 2025
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, llia Kulikov, and Zaid Harchaoui. From decoding to -neta-generation: Inference- time algorithms for large language models. arXio preprint arXXio2406616888, 2024.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. XiXiv preprint ar xxxo:2204.00408,, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. arXiv preprint ar drXiv:2310.06694 2023.
Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine-tuning. arXiv preprint arXio:2405.02596, 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.1511. 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. ar preprint ar axXit:2502.03387, 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12288 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine- tuning for transformer-based masked language-models. arXiv preprint arXfv:2106.10199 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXivv1905.07830, 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine-tuning. arxiv preprint arXiv:2303.10512, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parfieter-efficient fine-tuning, 2023b.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. Xiviv preprint ar XiXi::2308. 2ttt:229981, 2023c.
Yang Zhang, Yanfei Dong, and Kenji Kawaguchi. Investigating layer importance in large language models. Xiu preprint ar Xiv:2409.14381 2024.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking an efficient alternative to finetuning for pretrained language models. arXiv preprint 1rXio:2004.12406, 2020.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging lim-as-a-judge with mt-bench and chatbot arena. Xiv preprint ar arXiv:2306.05688 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023.
15

---

Published as a conference paper at COLM 2025
A Proof of Theorem 3.1
The proof of Theorem 3.1 is presented as follows:
Theorem A.1. For sufficiently small e, gr is c-stable, thus Assumption 3.1 and Assumption 3.2 are satisfied. For any 1> T, we assume that vi, rj E [0, 1]. Let vi denote the result of 'n: after one step of gradient descent, i.e, V/1 = Y/1 /V n1: [()mask). ). Then we have
11 7 2l2 - B(QL2 + Re.
(12)
Proof. Let y be the initial values of y and Y Then have
V = y BV n (
(13)
1+ = 9 PV n+1 ((
(14)
The difference of 7 and 7/11 is
11 1/11122 (
- (9 - --V D ))
Bl| V [() (0)mask) V n- (( ll2
e O mask (
0 V mask L
C lll
< 0 + opaak L(Omask))
omakk L(0mas))) ))12 I+1
Bl| 0.+1)
mask C( l|l.
(15)
Because C(0) has an L-Lipschitz continuuus gradient with constant L2 0, and ||0,|| s Q
V
mmask L(0maskk) - 0,+1
emakk L(0mask) f1l
0
mask
Q42||40/41
A0
QL2| 0r|12.
(16)
Because (()) is L-smooth with constant L1,
[|(0, 0,+1) 3 V C ll2 SL1100; - O
(17)
1+1
Therefore,
| - 714-1/1122220 < + - According to the Assumption 3.2, we have ||0, - -0/41122 S Re, hence,
(18)
|| 7/41l22 S B(QL2 + Le)
(19)
16

---

Published as a conference paper at COLM 2025
B Experimental Setup
Datasets. (1) Alpaca-GPT4 contains 52K instruction-folloiing data generated by GPT- 4, utilizing prompts from Alpaca (Taori al., 2023). (2) LIMA contains only 1K carefully curated prompts and responses. (3) No Robots contains 10K instructions and demonstrations created by skilled human annotators.
Models and Baselines. We use four different models as the base for experiments: LLAMA 2-7B (Touvron et al., 2023), LLAMA 2-13B, LLAMA 3.1-8B (Dubey et al., 2024), and Mistral-7B-v0.1 (Jiang et al., 2023). Our baselines are follows: (1) LoRA(Hu et al., 2021): Trainable rank decomposition matrices are added in parallel to existing weight matrices, including query /key /value projection (W,, We, Wa), output projection (W,) in self-attention, feed-forward networks (Wup, Wdowny Wgate), and the output layer head). (2) AdZhoRA(Zhang et al., 2023a): It dynamically adjusts the rank of incremental matrices to control the parameter budget, with AdaLoRA modules added to all linear layers, similar to LoRA. (3) Full Fine-tune: All model parameters, initialized from pre-trained weights and biases, undergo gradient updates during fine-tuning.
Evaluation and Training Setup. We assess language model alignment across two key dimensions: (1) Language Understanding Ability: Evaluated using MMLU (Hendrycks al., 2021) for specialized knowledge and Hellaswag (Zellers et al., 2019) for commonsense reasoning. (2) Conversational Ability: Measured using MT-Bench (Zheng et al., 2023) (multi-turn) and Vicuna (Chiang et al., 2023) (single-turn), with responses graded by GPT-4o All evaluations are performed three times, and the average scores are reported. We conduct hyperparameter searches for LoRA and full fine-tuning to establish strong baselines.
Targeted Performance. (1) Language Understanding Ability: Recent research (Du al., 2020; Sun et al., 2021; Dubey et al., 2024) suggests that the learning of language understand- ing tasks essentially occurs during the pre-training phase of the base significant performance improvements in language understanding tasks (i.e., MMLU, Hel- laswag) after alignment are not expected. However, it is crucial to ensure the model retains the learned knowledge during alignment. (2) Conversational Ability: Without alignment, the pre-train model's conversational ability is poor. For example, LLAMA 2-7B often produces incorrect irrelevant responses the Vicuna dataset. However, its conversational ability can be significantly improved through the alignment process.
Therefore,
For all experiments, we follow fine-tuning hyperparameters: we use AdamW with P1 0.9, $2 = 0.99 and weight decay of 0.1. The scheduler employed is a cosine scheduler with warmup ratio of 0.01. For LoRA baselines, we the hyperparameter rank as 32.
B.1 No Robots Dataset
We do a hyperparameter search for LoRA over the following variables: learning rate {0.001, 0.002, 0.0005, 0.0002, 0.0001), training epochs 4,,3,4,5). We do hyperparameter search for full fine-tuning over the following variables: learning rate {le 4, 2e 5, le 5,5e 6, 2e 6), training epochs (2,3,4,5).
LLAMA 2-7B. Both LoRA and AdaLoRA use a dropout rate 0.1 and a learning rate of 0.001. The number of training epochs is 3. For full fine-tuning, the learning rate is set to 0.00001, with the number of training epochs also being 3. The training parameters for ILA consistent with those of the baselines.
Mistral-7B. For LoRA and AdaLorA, we set the dropout rate as 0.1. The learning is 0.0002.
The number of training epochs is 2. For full fine-tuning, the learning rate is set as 0,000002 and the number of training epochs is The training parameters of ILA are the same as the baselines.
17

---

Published as a conference paper at COLM 2025
B.2 LIMA Dataset
We do a hyperparameter search for LoRA over the following variables: learning rate {0.001, 0.002, 0.0005, 0.0002, 0.0001), training epochs (5, 15, 15, We do hyperparameter search for full fine-tuning over the following variables: learning rate {le - 4, 2e - 5, le 5,5e 6, 2e 6), training epochs (5,10, 10)
For LoRA and AdaLorA, set the dropout rate as 0.1. The learning is 0.001. The number of training epochs is 20. For full fine-tuning, the learning rate set as 0.00001 and the number of training epochs 5. The training parameters of ILA are the same as the baselines.
LLAMA 2-7B
Mistral-7B. For LoRA and AdaLorA, we set the dropout rate as 0.1. The learning is 0.0002.
The number of training epochs is 5. For full fine-tuning, the learning rate is set as 0.000005 and the number of training epochs is 5. The training parameters of ILA are the same as the baselines.
B.3 Alpaca-GPT Dataset.
We do hyperparameter search for LoRA over the following variables: learning rate {0.001, 0.002, 0.0005, 0.0002, 0.0001), 0.000,,, training epochs 1,15,,,, 2,3. We do hyperparameter search for full fine-tuning over the following variables: learning rate {le 4, 5, le 5,5e 2e - 6), training epochs (0.5,1,
LLAMA 2-7B For LoRA and AdaLorA, we set the dropout rate as 0.1. The learning is 0.0002. The number of training epochs is 1.5. For full fine-tuning, the learning rate is set as 0.000002 and the number of training epochs is 0.5. The training parameters of ILA are the same as the baselines.
Mistral-7B. For LoRA and AdaLorA, we set the dropout rate as 0.1. The learning is 0.0002. The number of training epochs is 5. For full fine-tuning, the learning rate is set as 0.000002 and the number of training epochs is 0.5, The training parameters of ILA are the same as the baselines.
C Additional Experiments
C.1 GPU Memory Usage Analysis
We provide an overview of the GPU memory usage for different fine-tuning strategies, as shown Table 15. The table demonstrates the GPU memory usage and average training time per iteration for various fine-tuning approaches, including LoRA, QLoRA, full fine- tune, and their modified versions where only 30% of the important layers identified by ILA are fine-tuned. Both LoRA and QLoRA show substantial reductions in memory usage when restricted tuning only 30% of important layers, compared to the full-layer fine-tuning approaches. These results indicate that selectively fine-tuning small set of critical layers is highly effective reducing GPU memory consumption, particularly for efficient methods like QLoRA. This suggests that targeted fine-tuning enhance computational efficiency while preserving model performance, which is especially beneficial when scaling large language models with limited hardware resources.
C.2 Cross-dataset Evaluation of Layer Importance
As shown in Table 2, different datasets reveal subtle variations in the layers identified as important. This suggests that layers consistently deemed unimportant across multiple datasets are likely genuinely non-essential. To validate this, intersect the top-K least
18

---

Published as a conference paper at COLM 2025
Table 15: GPU memory usage for LoRA, QLoRA, Full Fine-tune and LoRA/ DORA/QLoRA/Ful Fine-tune with only 30% of important layers fine-tuned. Batch size is set to 2, and the maximum token length is 1024. Percentages in parentheses indicate the proportion of linear layers fine-tuned.
GPU
Training time (ms)
Memory Usage (MiB)
Full Fine-tune (100%)
81276
396
Full Fine-tune w/ ILA (30%)
33458
304
LoRA (100%)
32752
403
LoRA w/ ILA (30%)
28586
359
QLoRA (100%)
26238
523
QLoRA w/ ILA (30%)
17912
423
Table 16: Results fine-tuning Mistral-7B-v0.1 on the LIMA dataset using ILA to identify important layers from various datasets. Dataset (Imp. Layers) indicates the datasets utilized to search for the important layers. Intersection represents freezing the layers that are the intersection of the top-K least important layers found from the LIMA, No Robots, and A Alpaca GPT4 datasets. Evaluated using MMLU (5-shot), Hellaswag (0-shot), GPT-4 scores Vicuna prompts, and MT- prompts.
Dataset (Imp. Layers)
Dataset
Language Understanding
Conversational Ability
(Fine-tune)
MMLU1
Hellaswag 1
Vicuna
MT-Bench
LIMA
LIMA
61.82
65.48
6.99
5.38
No Robots
LIMA
61.52
65.51
6.92
5.34
Alpaca-GPT4
LIMA
61.23
65.20
7.03
5,21
Intersection
LIMA
61.49
65.62
7.06
5.44
important layers identified from three datasets (LIMA, No Robots, and Alpaca-GPT4) to derive a set of universally non-critical layers. The results are presented in Table 16.
Our analysis reveals that a holistic consideration of layer importance across multiple datasets yields superior results compared to dataset-specific approaches. For instance, identifying important layers within the LIMA dataset and fine-tuning on the No Robots dataset is less effective than an integrated approach. Similarly, finding important layers and fine-tuning exclusively on the No Robots dataset do not perform as well as the comprehensive method. This suggests that a cross-dataset evaluation of layer importance can lead to more robust and effective fine-tuning strategies.
C.3 Cross-model Transfer of Layer Importance Rankings
We evaluated the transferability of layer importance rankings across different models, focusing on the Jaccard similarity of the top 75% important layers. The results for various models are shown in Table 10. As seen, the Jaccard similarity between different architectures is approximately 0.70, suggesting a moderate overlap in the layers identified as important.
In our subsequent experiment, used LLAMA 2-7B's layer importance rankings from the No Robots dataset to fine-tune Mistral-7B-v0.1, as shown in Table 11. The results demonstrate that while cross-model transfer offers performance improvements, fine-tuning using rankings from the same model architecture yields the best results.
These findings suggest that cross-model transfer of layer importance rankings is possible, though less effective than using rankings from the same architecture. Fine-tuning the top 75% of layers based on cross-model transfer shows some improvement, while fine-tuning only the top 30% achieves comparable performance.
C.4 Impact of Initial Layer Importance Scores
We evaluated the effect of different initial layer importance scores on the consistency of identified important layers. The scores were initialized to so 2.0, 1.0. The consistency
19

---

Published as a conference paper at COLM 2025
was measured using the Jaccard similarity of the top 75% important layers identified during fine-tuning of LLama 2-7B on the LIMA dataset. The results are shown in Table 17. These results show that the method is stable with respect to initialization. While the choice of st can influence the optimization trajectory, it does not significantly impact the convergence or final importance rankings.
Table 17: The Jaccard similarities of top 75% important layers identified during fine-tuning of LLAMA 2-7B on the LIMA dataset with varying initial scores.
Initial Scores
4.0
2.0
1.0
4.0
2.0
0.83
1.0
0.78
0.88
C.5 Additional Experiments on Model Scalability
To assess whether freezing unimportant layers continues to enhance model performance at a larger scale, we conducted additional experiments on LLAMA 2-13B. Specifically, we fine-tuned LLAMA 2-13B using the No Robots and LIMA datasets, with results compared against LoRA presented in the table below. The experimental outcomes demonstrate that our method maintains strong performance on LLAMA 2-13B. Despite the increased model size, the underlying architectural similarities suggest that our approach remains effective and scalable, likely extending its benefits to even larger models.
We also carried out further experiments LIMA and Alpaca-GPT4 using LLAMA 2- 7B, Mistral-7B-v0.1 and Llama 3.1-8B to evaluate the adaptability of our approach across different model architectures. Consistently, our method outperformed LoRA while requiring fewer layers to be fine-tuned. These findings further validate the robustness and scalability of our approach, showing its capability effectively enhance performance across various model sizes and architectural variations.
Table 18: Fine-tuning results of LLAMA 2-13B on the LIMA and No Robots datasets. Eval- uated using MMLU (5-shot), Hellaswag (O-shot), GPT-4o scores on Vicuna prompts, and MT-Bench prompts. Cells highlighted in grey indicate that ILA has improved the perfor- mance of the base model.
Language Understanding
Conversational Ability
Datasets
Methods
MMLU
Hellaswag
Vicuna 1 MT-Bench
LORA LoRA w/ ILA
53.85
63.08
6.16
3.79
LIMA
62.04
6.25
LoRA
54.08
61.73
5.72
4.24
No Robots
LoRA w/ ILA
54.45
61.13
5.88
4.37
D ILA for LLM Reasoning
Datasets. (1) LIMO (Ye et al., 2025): This dataset comprises 817 carefully selected problems
drawn from an initial pool of tens of millions. The final selection meets strict quality standards and covers broad range of mathematical reasoning tasks. High-quality solutions are provided by both human experts and AI systems like DeepSeek R1 (Guo et al., 2025). (2) s1.1 (Muennighoff et al., 2025): This dataset includes 1,000 questions paired with reasoning traces, curated based on three rigorously validated criteria: difficulty, diversity, and quality. The tfain-of-thought solutions are generated by DeepSeek R1 (Guo et al., 2025).
Evaluation. (1) AIME24: This set contains 30 problems from the 2024 American Invitational Mathematics Examination (AIME), administered on January 31 and February 1, 2024. (2)
20

---

Published as a conference paper at COLM 2025
Table 19: Comparative evaluation of LLAMA 2-7B, Mistral-7B-v0.1, and Llama 3.1-8B models fine-tuned on the LIMA Dataset. Evaluated using MMLU (5-shot), Hellaswag (0-shot), GPT-4o scores on Vicuna prompts, and MT-Bench prompts. The evaluations are performed three times, and the average scores are reported. Cells highlighted in grey indicate that ILA has enhanced the performance the base model. The best result is marked in bold.
Language Understanding Conversational Ability
Models
Methods
MMLU f
Hellaswag 1
Vicuna
MT-Bench
AdaLoRA
44.21
59.85
5.22
3.51
Fine-tune
46.36
62.06
3.71
LLAMA 2-7B
Full Fine-tune w/
46.32
62.18
3.85
LoRA
43.18
54.52
5.43
3.45
LoRA w/ ILA
44.13
54.55
5.62
3.72
AdaLoRA
62,40
61.52
6.64
4.49
Full Fine-tune
60.11
63.76
6.88
4.63
Mistral-7B-v0.1
Full Fine-tune w/ ILA
61.01
64.01
6.95
4.77
LoRA
60.83
65.42
6.70
4.58
LoRA w/ I.A
61.52
AdaLoRA
63.55
62.65
4.73
Full Fine-tune
64.31
65.64
7.09
5.12
lamm 3.1-88
Full Fine-tune w/ ILA
64.73
65.98
7.17
5.23
LoRA
62.33
62.92
6.57
4.79
LoRA w/ ILA
63.31
63.01
4.93
Table 20: Comparative evaluation LLAMA 2-7B, Mistral-7B-v0.1, and Llama 3.1-8B models fine-tuned on the Alpaca-GPT4 Dataset. Evaluated using MMLU (5-shot), Hellaswag (0- shot), GPT-4o scores on Vicuna prompts, and MT-Bench prompts. The evaluations are performed three times, and the average scores reported. Cells highlighted in grey indicate that ILA has enhanced the performance of the base model. The best result is marked in bold.
Language Understanding
Conversational Ability
Models
Methods
MMLU
Hellaswag
Vicuna
MT-Bench
AdaLoRA
46.13
57.85
6.89
3.78
Full Fine-tune
57.73
3.72
Ilama-7B
Full Fine-tune w/ ILA
46.23
57.67
6.99
3.85
LoRA
43.66
58.49
3.80
LoRA w/ ILA
44.69
56.22
7.17
3,99
AdaLoRA
62.48
62.08
7.25
4.77
Mistral-7B-v0.1
Full Fine-tune w/ ILA
Fine-tune
60.88
60.56
62.80
62.91
7.35
7.19
4.91
4.78
LoRA
61.82
62.70
7.23
4.89
LoRA w/ ILA
62.14
62.80
7.33
5.02
AdaLoRA
65.82
61.02
7.48
5.39
63.58
Llama 3.1-8B
Full Fine-tune w/ ILA
64.61
61.74
7.57
5.42
LoRA
65.40
61.72
7.65
5.43
LoRA w/ ILA
65.76
61.81
7.79
5.55
MATH500 (Hendrycks et al., 2021): A benchmark consisting of competition-level math problems spanning range of difficulties.
Consistency in Layer Importance Across Datasets. We applied our proposed ILA al- gorithm to identify layer importance rankings on both the LIMO and s1.1 datasets using the Qwwn2.5-7B-Instrric model (Yang et al., 2024). The resulting rankings showed strong consistency, with Jaccard similarity of 0.86 (see Fig.3), suggesting that LLMs tend to acquire similar reasoning-relas of this knowledge is already learned during pretraining also suggested by LIMO (Ye et al., 2025)-and and that fine-tuning primarily serves to activate model's ' latent reasoning abilities.
knowledge across datasets. We hypothesize that much
21

---

Published as a conference paper at COLM 2025
Based on the identified importance rankings, we further conducted experiments on LIMO by freezing approximately 25% of the least important layers. As shown in Table 14, fine- tuning only the top 75% most important layers led to slight improvement in performance, indicating that selective tuning can help enhance the model's reasoning capabilities.
22


Reviews:
[{"review_id": "grGBJ5dmHo", "reviewer": "Reviewer_EcV9", "summary": "This paper investigates which layers of large language models (LLMs) are most responsible for alignment changes introduced by supervised fine‑tuning. The authors propose ILA, a method that learns a binary mask over per‑layer parameter updates, treating mask activation as a measure of layer importance. Experiments across diverse alignment datasets show that the sets of important layers share nearly 90 % overlap, suggesting a common alignment pattern irrespective of data source. Furthermore, freezing the non‑essential layers improves the overall performance. This suggests the possibility of constraining fine‑tuning to a subset of layers while enjoying substantial efficiency gains with negligible accuracy loss. The study also offers initial evidence that these insights generalize from alignment tasks to reasoning abilities.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The consistency of the identified critical layers within the same model across different alignment datasets is an interesting observation. This lays the foundation of feasibility of constraining fine‑tuning to only a subset of layers.\n2. It is a little surprising but interesting to see that freezing unimportant layers can actually improve alignment performance.\n3. The authors conduct intensive numerical experiments and well-designed and informative ablation study.\n4. The paper is written clearly.", "reasons_to_reject": "1. I think the authors overclaim the potential of the proposed method to improve alignment efficiency through selective fine-tuning. On one hand, selective fine-tuning requires the identificaiton of the important layers first using ILA, which involves at least 30-50% of normal training (either full-model training or PEFT training) and the score training overheads. On the other hand, I do not think selective fine-tuning helps much with improving training efficiency, as gradients still need to be propagated to earlier layers.\n\n2. I am not sure why the authors include AdaLoRA as a baseline but do not include results of AdasLoRA with ILA. If I am understanding it correctly, AdaLoRA is a LoRA variant and is orthogonal to ILA. Using only AdaLoRA as baseline provides very limited information to the audience.\n\n3. The message conveyed by Thm 3.1 is unclear to me. Firstly, the first statement in Thm 3.1, \"For a sufficiently small ϵ, θT is ϵ-stable\" looks an assumption to me. The statement itself reads wierd to me because we do not know if a pre-trained transformer is $\\epsilon$-stable for a very small $\\epsilon$. We can only say that it is $\\epsilon$-stable for a sufficiently large $\\epsilon$. But a large $\\epsilon$ makes the result in eqn (11) weak. In summary, I am confused about Thm 3.1. I recommend the authors to elaborate more on Thm 3.1, or to avoid including it in the main text just to make the paper more mathy.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "igEMrUu6uB", "reviewer": "Reviewer_1beu", "summary": "This paper introduces ILA (Identifying Layer significance for Alignment), a method to find key layers in Large Language Models (LLMs) during alignment. It reveals that a consistent set of layers is crucial across different datasets. Interestingly, ignoring less important layers can boost performance, and fine-tuning only critical layers improves efficiency with minimal performance loss. The research also suggests these findings could apply to LLM reasoning.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear Writing: The paper is well-structured and explains complex concepts, such as the ILA methodology and its theoretical underpinnings, in an accessible manner. The arguments flow logically from the introduction of the problem to the presentation of results and their implications, making it easy for readers to follow the research.\n2. Interesting Findings: The discovery that important layers for alignment show high consistency across diverse datasets is a significant and intriguing finding. Additionally, the observations that freezing certain \"unimportant\" layers can lead to performance improvements (Table 5), and that selective tuning of a small percentage of critical layers maintains performance, offer valuable and somewhat counter-intuitive insights into the LLM alignment process.\n3. Extensive Experimentation: The authors validate their claims through a comprehensive suite of experiments. They utilize multiple LLM architectures across various sizes, test on several alignment and reasoning datasets, and evaluate using a broad range of benchmarks for language understanding and conversational ability.", "reasons_to_reject": "1. Reliance on LoRA Approximation: The ILA method primarily utilizes LoRA (Low-Rank Adaptation) to approximate parameter changes for computational efficiency. An ablation study indicates that LoRA-based importance scores achieve nearly 83% overlap with those derived from full fine-tuning (FFT). However, this also implies that LoRA is an approximation and may not perfectly mirror the true parameter changes of FFT, which could influence the precise identification of the most critical layers. Despite this, the paper suggests LoRA provides a \"strong approximation\".\n2. Computational Cost of Initial Stage: Although ILA is presented as an efficient method, its first stage, which involves training the model with LoRA until it reaches an stable state, accounts for the majority of the computational cost. The paper suggests that identifying layer importance effectively requires only 25-50% of the training milestones. Nevertheless, this initial phase can still be resource-intensive, particularly for very large models.\n3. Exploration Range for Layer Freezing/Tuning: The study primarily investigates freezing approximately 25% of unimportant layers to enhance performance or fine-tuning 10-30% of key layers to improve efficiency. While these specific percentages yield positive outcomes, a more detailed analysis of how varying these proportions affects the balance between performance and efficiency could offer further insights.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9OXw85hM4", "reviewer": "Reviewer_K9M2", "summary": "The paper researches the important layers of LLM alignment. Specifically, authors train models until the loss becomes stable and then find a mask to minimize the loss. The results show that the method performs similarly by applying the mask and making the training faster with less memory. The authors also give a theory and empirical results to show that the found masks are robust and will not consistently change when training progresses.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The topic is very interesting and helpful for the explainable AI community.\n* The theory is simple yet useful; it makes a paper lot of sense since it guarantees the stability of the masks.", "reasons_to_reject": "* I do not see significant drawbacks in this paper.\n* However, I do feel the paper can be further improved by trying to answer \"why\" those layers are important, which will make the paper more novel.", "questions_to_authors": "* Suggestion: Try use vector graphics in Figure 2", "extra_scores": {}}, {"review_id": "bmzpYIWsfv", "reviewer": "Reviewer_pE9b", "summary": "The paper investigates layer significance in LLM alignment, proposing ILA to identify critical layers via binary masks. Key findings include consistent important layers across datasets (90% overlap), improved performance by freezing non-essential layers, and efficiency gains from selective tuning (10-30% layers).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] The paper introduces a systematic approach (ILA) to analyze layer dynamics in alignment, offering insights into efficiency and model behavior;\n[2] It demonstrates freezing/tuning strategies reduce compute costs while maintaining performance, addressing a critical challenge in LLM deployment.", "reasons_to_reject": "[1] The evaluation might have limitations: it relys heavily on GPT-4o for conversational metrics. Human evaluations or adversarial testing could've strengthened validity.\n[2] Some claims need to be verified, especially. claims about broader applicability to reasoning lack thorough analysis. \n[3] Details on hyperparameter tuning (e.g., LoRA ranks, learning rates) are referred in appendix, but may not reproducible", "questions_to_authors": "How does the binary mask balance sparsity and performance during gradient descent? Could stochasticity in training affect layer importance rankings?\n\nhe experiments use Alpaca-GPT4, LIMA, and No Robots. How would ILA perform on alignment datasets with conflicting stylistic objectives (e.g., formal vs. casual responses)?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces ILA, a method for identifying critical layers in LLMs during alignment by learning binary masks over per-layer updates. Through extensive experiments across multiple datasets and model architectures, the authors show that a small, consistent subset of layers (~75%) is responsible for most alignment changes. Freezing unimportant layers can enhance alignment performance and improve training efficiency. The paper is well-structured and the empirical evidence is strong. Reviewers raised concerns about overhead, LoRA approximation, and stylistic generalization all of which were directly addressed with new experiments and reproducibility details. The work balances practical value with interpretability which offers actionable insights for scalable and efficient LLM alignment. I recommend acceptance.
Decision: Accept