Understanding Layer Significance in LLM Alignment

## Abstract

Abstract Aligning large language models (LLMs) through fine-tuning is essential for tailoring them to specific applications. Therefore, understanding what LLMs learn during the alignment process is crucial. Recent studies suggest that alignment primarily adjusts a model’s presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To delve deeper into LLM alignment, we propose to identify which layers within LLMs are most critical to the alignment process, thereby uncovering how alignment influences model behavior at a granular level.
We propose a novel approach to identify the important layers for LLM alignment (ILA). It involves learning a binary mask for each incremental weight matrix in the LoRA algorithm, indicating the significance of each layer. ILA consistently identifies important layers across various alignment datasets, with nearly 90% overlap even with substantial dataset differences, highlighting fundamental patterns in LLM alignment. Experimental results indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss.

## 1 Introduction

Aligning large language models (LLMs) with specific requirements is essential for enhancing their utility across diverse applications (Luo et al., ; Yu et al., ; Luo et al., ; Li et al., ; Liu et al., ; ; Feng et al., ).
Fine-tuning LLMs during the alignment process can significantly improve the models’ capabilities to meet targeted needs (Bubeck et al., ).
Typically, alignment involves fine-tuning the model on diverse datasets, which may include both human-curated (Rajani et al., ) and LLM-generated (Taori et al., ) data. Such fine-tuning approaches encompass instruction tuning (Wei et al., ) and preference learning (Bai et al., ; Rafailov et al., ). Given the significant cost associated with full parameter fine-tuning, parameter-efficient fine-tuning (PEFT) (Hu et al., ; Chen et al., ; Pan et al., ) algorithms have emerged as a popular alternative, offering a balance between performance and resource efficiency.

Understanding what LLMs actually learn during the alignment process remains a critical question. LIMA (Zhou et al., ) posits that the majority of knowledge and capabilities are developed during the pretraining phase, with alignment primarily serving to refine the model’s conversational style and formatting. Using a well-selected set of 1,000 training examples for supervised fine-tuning (SFT), LIMA successfully produced a high-quality aligned model. Similarly, URIAL (Lin et al., ) investigated the token distribution of LLMs before and after alignment and found that most changes were related to “stylistic tokens”, such as discourse markers and transition words, while the knowledge-intensive content largely remained untouched, coming from the base pre-trained model. These findings imply that the alignment process mainly adjusts the model’s presentation style rather than altering its foundational knowledge.

To gain a deeper understanding of LLM alignment, we adopt a distinct approach by examining it at the model parameter level. In our pilot study, we investigate the impact of different model components on alignment performance, we conducted a simple analysis by fine-tuning only specific layers and evaluating the resulting performance, as presented in Table 1. The results clearly indicate that fine-tuning different components of the LLM leads to considerable performance differences. For instance, fine-tuning the feed-forward network (FFN) layers achieves performance similar to fine-tuning all linear layers (i.e., with LoRA), whereas focusing solely on the attention layers causes a notable drop in performance. This observation underscores the complexity of layer-specific contributions to LLM alignment, highlighting the need for a more detailed approach to understanding their individual roles.

To this end, we propose to identify the layers that are most critical to alignment performance during the fine-tuning process. We develop a novel approach for identifying the important layers for LLM alignment, called ILA. Specifically, we learn a binary mask for each incremental weight matrix in the LoRA algorithm,
which serves as an indicator of layer significance. A value of zero in the binary mask indicates that the corresponding layer has negligible influence during the fine-tuning phase, while a value of one denotes that the layer is crucial for the process. We employ gradient descent to learn the binary mask effectively and offer a theoretical analysis of the optimization process. The main findings of this work are summarized as follows:

**Table 1: Impact of fine-tuning different regions of Llama 2-7B (Touvron et al., ) on alignment performance using LIMA dataset. Evaluated using MMLU (5-shot) (Hendrycks et al., ), GPT-4 scores on Vicuna prompts (Chiang et al., ), and MT-Bench prompts (Zheng et al., ). Fine-tuning components include query/key/value projection layers ($W_{q}$, $W_{k}$, $W_{v}$), output projection layer ($W_{o}$) in self-attention, and feed-forward networks ($W_{\mathrm{up}}$, $W_{\mathrm{down}}$, $W_{\mathrm{gate}}$)**
|  | ATT ($W_{q}$, $W_{k}$, $W_{v}$, $W_{o}$) | ATT2 ($W_{q}$, $W_{k}$, $W_{v}$) | FFN ($W_{\text{up}}$, $W_{\text{down}}$, $W_{\text{gate}}$) | ALL (LoRA) |
| --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | 42.03 | 42.65 | 43.06 | 43.18 |
| Vicuna $\uparrow$ | 5.63 | 5.54 | 5.69 | 5.78 |
| MT-Bench $\uparrow$ | 3.82 | 3.80 | 3.92 | 3.98 |

Figure: Figure 1: Layer importance ranking of Llama 2-7B (Touvron et al., ) and Mistral-7B-v0.1 (Jiang et al., ) by ILA across the Alpaca-GPT4 (Peng et al., ), LIMA Zhou et al. (), and No Robots (Rajani et al., ) datasets. Layers ranked in the top 75% by scores ($s_{i}$) are considered important. The x-axis represents the transformer block index, and the y-axis shows the names of linear layers within each block. The figure illustrates two key findings: (1) There is a significant overlap (up to 90%) in the important layers identified by ILA across different alignment datasets, as supported by the Jaccard similarity values in Table 2. This high consistency indicates that similar capabilities are needed for alignment, regardless of substantial differences in dataset content. (2) The important layers vary between different network architectures, suggesting that each model’s structure and dynamics uniquely affect which layers are most crucial for alignment.
Refer to caption: x1.png

- •
Consistent Layer Importance Ranking Across Different Alignment Datasets. Despite the differences in dataset, we find similar rankings of important layers during alignment for the same pre-trained model (see Fig. 1). This suggests that the alignment process equips the model with similar capabilities, even when the training data varies significantly in both content and size. This evidence corroborates previous research findings and offers new insights into LLM alignment.
- •
Enhancing Performance by Freezing Unimportant Layers. We show that freezing approximately 25% of unimportant layers can improve model performance and that a single search for layer-wise importance ranking is sufficient for different alignment tasks within the same architecture.
- •
Improving Alignment Efficiency Through Selective Fine-Tuning. Our findings show that fine-tuning only 10-30% of the most important layers achieves performance comparable to fine-tuning all linear layers. Additionally, integrating this approach with QLoRA allows tuning only 30-75% of key layers to maintain or enhance performance while significantly reducing resource costs.

## 2 Quantifying Layer Significance in LLM Alignment

To better understand layer significance in the alignment process of an LLM, we propose a method to identify the important layers during alignment, abbreviated as ILA. This approach involves learning a binary mask that serves as an significance indicator for each layer.

Consider a pre-trained LLM model with parameters $\bm{\uptheta}_{0}$ composed of $N$ layers, i.e.,
$\bm{\uptheta}_{0}=\{\bm{\uptheta}_{0}^{i}\}_{i=1}^{N}$. The model is fine-tuned on an alignment dataset $\mathcal{D}=\{z_{i}\}_{i=1}^{n}$ with a loss function $\mathcal{L}(\bm{\uptheta})$.
After $t$ training iterations, the model parameters are updated to $\bm{\uptheta}_{t}=\bm{\uptheta}_{0}+\Delta\bm{\uptheta}_{t}$, where $\Delta\bm{\uptheta}_{t}$ represents the change in parameters till iteration $t$. Define a binary mask $\bm{\gamma}_{t}=\{\gamma_{t}^{i}|\gamma_{t}^{i}\in\{0,1\}\}_{i=1}^{N}$ that encodes layer-wise importance information. We apply $\bm{\gamma}_{t}$ to $\Delta\bm{\uptheta}_{t}$ and define

$$ $\bm{\uptheta}^{\mathrm{mask}}_{t}=\bm{\uptheta}_{0}+\bm{\gamma}_{t}\odot\Delta \bm{\uptheta}_{t},$ (1) $$

where $\odot$ is component-wise multiplication. The binary mask is applied to retain the changes in crucial layers while eliminating the rest. Below we provide a formal definition of the conditions under which training attains stability after an adequate number of iterations.

###### Definition 1 ( $\epsilon$ -stable) .

$\forall\epsilon>0$, the model is said to be $\epsilon$-stable at iteration $T$ if, for any $t>T$, the loss function satisfies the condition

$$ $\left|\mathbb{E}_{z}[\mathcal{L}(\bm{\uptheta}_{t+1},z)]-\mathbb{E}_{z}[ \mathcal{L}(\bm{\uptheta}_{t},z)]\right|<\epsilon,$ (2) $$

where $\mathbb{E}_{z}[\cdot]$ denotes the expectation with respect to the alignment dataset $\mathcal{D}$.

Once training becomes stable, we can identify the layers that are crucial for the alignment task.

###### Definition 2 (Layer Importance) .

The binary mask $\bm{\gamma}_{t}$ is defined as the solution to the following optimization problem:

$$ $\bm{\gamma}_{t}=\operatorname*{arg\,min}_{\bm{\gamma}_{t}}\mathcal{L}(\bm{ \uptheta}^{\mathrm{mask}}_{t}),\>s.t.\>\|\bm{\gamma}_{t}\|<H,$ (3) $$

where $H$ is a hyper-parameter that serves as a constraint to limit the number of important layers.

Efficiently Identifying the Importance Layers. Due to the high cost of fine-tuning large models, to address the optimization problem in Eq. (3), we employ the LoRA (Hu et al., ) algorithm, which utilizes low-rank decomposition matrices to represent the change in model parameters till iteration $t$ ($\Delta\bm{\uptheta}_{t}$).
Specifically, LoRA utilizes two trainable low-rank matrices, $\bm{B}^{i}_{t}\in\mathbb{R}^{d_{i}\times r_{i}}$ and $\bm{A}^{i}_{t}\in\mathbb{R}^{r_{i}\times k_{i}}$, to estimate the change of the $i^{\mathrm{th}}$ layer:

$$ $\Delta\bm{\uptheta}_{t}^{i}=\beta\cdot\bm{B}_{t}^{i}\bm{A}_{t}^{i},$ (4) $$

where $\beta$ is the scalar hyperparameter of LoRA. With the binary mask $\bm{\gamma}_{t}$, the $i^{\mathrm{th}}$ layer is updated by

$$ $\bm{\uptheta}_{t}^{i}=\bm{\uptheta}_{0}^{i}+\beta\cdot\gamma^{i}_{t}\cdot\bm{B }_{t}^{i}\bm{A}_{t}^{i}.$ (5) $$

To ease the optimization of $\bm{\gamma}_{t}$, we re-parametrize each of its each components $\gamma^{i}_{t}$ as the output of a Sigmoid function, i.e., $\gamma^{i}_{t}=\sigma(s^{i}_{t})$. Then, the update of the $i^{\rm th}$ layer becomes

$$ $\bm{\uptheta}_{t}^{i}=\bm{\uptheta}_{0}^{i}+\beta\cdot\sigma(s^{i}_{t})\cdot \bm{B}_{t}^{i}\bm{A}_{t}^{i}.$ (6) $$

Let $\bm{s}_{t}=\{s^{i}_{t}\}_{i=1}^{N}$, $\bm{\uptheta}^{\mathrm{M}}_{t}=\{\bm{\uptheta}_{t}^{i}\}_{i=1}^{N}$. The optimization problem in Eq. (3) becomes

$$ $\bm{s}_{t}=\operatorname*{arg\,min}_{\bm{s}_{t}}\mathcal{L}(\bm{\uptheta}^{ \mathrm{M}}_{t}).$ (7) $$

We use gradient descent to optimize $\bm{s}_{t}$. The found $s^{i}_{t}$ is considered an importance score of the $i^{\mathrm{th}}$ layer.
A larger value of $s^{i}_{t}$ indicates $\gamma^{i}_{t}$ is closer to one, signifying higher importance of the $i^{\rm th}$ layer.

###### Assumption 2.1 (Lipschitz-continuous) .

The loss function $\mathcal{L}(\theta):\mathbb{R}^{d}\to\mathbb{R}$ is continuously differentiable and L-smooth with constant $L_{1}>0$ such that

$$ $\|\mathcal{L_{1}}(\theta)-\mathcal{L}(\theta^{\prime})\|_{2}\leq L_{1}\|\theta -\theta^{\prime}|.$ (8) $$

In addition, $\mathcal{L}(\theta)$ has an L-Lipschitz continuous gradient with constant $L_{2}>0$ such that

$$ $\|\nabla R(\theta)-\nabla R(\theta^{\prime})\|_{2}\leq L_{2}\|\theta-\theta^{ \prime}|.$ (9) $$

###### Assumption 2.2 .

For any $t>T$, $\bm{\uptheta}_{t}$ is $\epsilon$-stable. We assume there is a constant $R$ such that

$$ $\|\theta_{t}-\theta_{t+1}\|_{2}\leq R\epsilon,$ (10) $$

and there is a constant $Q$ such that $\|\theta_{t}\|_{2}\leq Q$ for any $t>T$.

###### Theorem 2.1 .

For a sufficiently small $\epsilon$, $\theta_{T}$ is $\epsilon$-stable, thus Assumption 2.1 and Assumption 2.2 are satisfied. For any $t>T$, we assume that $\forall i,\gamma_{t}^{i}\in[0,1]$. Let $\bm{\gamma}_{t}^{\prime}$ denote the result of $\bm{\gamma}_{t}$ after one step of gradient descent, i.e., $\bm{\gamma}_{t}^{\prime}=\bm{\gamma}_{t}-\beta\nabla_{\bm{\gamma}_{t}}\mathcal
{L}(\bm{\uptheta}_{t}^{\rm mask})$. Then we have

$$ $\|\bm{\gamma}_{t}^{\prime}-\bm{\gamma}_{t+1}^{\prime}\|_{2}\leq\beta(QL_{2}+L_ {1})R\epsilon.$ (11) $$

This theorem demonstrates that when $\theta_{T}$ is $\epsilon$-stable, solving the optimization problem in Eq. (3) for any $t>T$ yields similar results. This is because, after one step of gradient descent, the difference between $\gamma_{t}$ and $\gamma_{t+1}$ is smaller than a sufficiently small number. The proof is provided in Appendix A, and empirical results supporting this are shown in Fig. 2.

Figure: Algorithm 1 Identify the Important Layers for Alignment (ILA)

Figure: Figure 2: Layer importance ranking of Llama 2-7B identified by our method ILA on LIMA datasets in different training milestones (i.e., 1%, 25%, 50%, 75%, and 100%). The x-axis represents the transformer block index, and the y-axis shows the names of linear layers within each block. Detailed numbers of the Jaccard similarity are presented in Table 4.
Refer to caption: x2.png

**Table 2: Jaccard similarities of important layers identified during fine-tuning of Llama 2-7B and Mistral-7B on various datasets. Top 75% highest-scoring layers are determined as important layers.**
| Datasets | Llama 2-7B | Mistral-7B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| LIMA | No Robots | Alpaca-GPT4 | LIMA | No Robots | Alpaca-GPT4 |  |
| LIMA | - | - | - | - | - | - |
| No Robots | 0.91 | - | - | 0.90 | - | - |
| Alpaca-GPT4 | 0.90 | 0.90 | - | 0.89 | 0.93 | - |

Figure: Figure 3: Jaccard similarities of important layers identified during fine-tuning of Llama 2-7B on the LIMA dataset with varying random seeds. The top 75% highest-scoring layers are designated as important layers.

###### Definition 1 ( ϵ italic-ϵ \epsilon italic_ϵ -stable) .

###### Definition 2 (Layer Importance) .

###### Assumption 2.1 (Lipschitz-continuous) .

###### Assumption 2.2 .

###### Theorem 2.1 .

## 3 Experimental Setup

#### Datasets.

We train LLMs on three different alignment datasets, namely Alpaca-GPT4 (Peng et al., ), LIMA (Zhou et al., ), and No Robots (Rajani et al., ).
The characteristics of each dataset are described as follows:
(1) Alpaca-GPT4 contains 52K instruction-following data generated by GPT-4, utilizing prompts from Alpaca (Taori et al., ). (2) LIMA contains only 1K carefully curated prompts and responses. (3) No Robots contains 10K instructions and demonstrations created by skilled human annotators.

#### Models and Baselines.

We use four different models as the base for our experiments: Llama 2-7B (Touvron et al., ), Llama 2-13B (Touvron et al., ), Llama 3.1-8B (Dubey et al., ), and Mistral-7B-v0.1 (Jiang et al., ).
The baselines include (1) LoRA (Hu et al., ): We add trainable pairs of rank decomposition matrices in parallel to existing weight matrices, including query/key/value projection ($W_{q}$, $W_{k}$, $W_{v}$), output projection ($W_{o}$) in the self-attention, feed-forward networks ($W_{\rm up}$, $W_{\rm down}$, $W_{\rm gate}$), and the output layer on top of the transformer ($W_{\mathrm{head}}$).
(2) AdaLoRA (Zhang et al., ): It dynamically adjusts the rank of incremental matrices to control the parameter budget. Similar to LoRA, we add AdaLoRA modules to all linear layers of the base model. (3) QLoRA (Dettmers et al., ): It is a fine-tuning method that significantly reduces memory usage by quantizing the weights of pre-trained language models while maintaining competitive performance.
(4) Full Finetune: The model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.

#### Evaluation and Training Setup.

Our evaluation of language model alignment encompasses two main dimensions:
(1) Language Understanding Ability: We utilized three distinct datasets (i.e., MMLU (Massively Multitask Language Understanding) (Hendrycks et al., ) and Hellaswag (Zellers et al., ) to evaluate this aspect. MMLU evaluates models across diverse subjects requiring specialized knowledge, while Hellaswag tests commonsense reasoning by asking the model to predict the most plausible continuation of a given context.
(2) Conversational Ability: We use two different datasets: MT-Bench (Zheng et al., ), which involves multi-turn conversations, and Vicuna (Chiang et al., ), which involves single-turn conversations. We use GPT-4 to score the responses. We asks GPT-4 to grade and give a score to model’s answer directly without pairwise comparison, using the implementation version of MT-Bench (Zheng et al., ).
For a fair comparison, we conduct a small range of training hyperparameter searches for LoRA and full fine-tuning to ensure that we get strong baselines. More details are provided in Appendix B.

Targeted Performance. (1) Language Understanding Ability: Recent research (Du et al., ; Sun et al., ; Dubey et al., ) suggests that the learning of language understanding tasks essentially occurs during the pre-training phase of the base model. Therefore, significant performance improvements in language understanding tasks (i.e., MMLU, Hellaswag) after alignment are not expected. However, *it is crucial to ensure the model retains the learned knowledge during alignment*.
(2) Conversational Ability: Without alignment, the pre-train model’s conversational ability is poor. For example, Llama 2-7B often produces incorrect or irrelevant responses on the Vicuna dataset. However, *its conversational ability can be significantly improved through the alignment process*.

## 4 Empirical Findings

### 4.1 Layer Significance in LLM Alignment

In this subsection, we applied our ILA algorithm to identify the ranking of important layers during alignment across three different datasets: No Robots, LIMA, and Alpaca-GPT4, as shown in Fig. 1. Additionally, we analyzed the importance ranking of layers identified at different training milestones, as depicted in Fig. 2. To further validate the similarity of these important layers, we used the Jaccard similarity coefficient to quantify the relationship between two sets. Specifically, we defined the top 75% highest-scoring layers as the important layers, denoted as set $\bm{S}$. The similarity between two distinct sets, $\bm{S}_{1}$ and $\bm{S}_{2}$, is calculated as: $J(\bm{S}_{1},\bm{S}_{2})=\frac{|\bm{S}_{1}\cap\bm{S}_{2}|}{|\bm{S}_{1}\cup\bm{
S}_{2}|}$. A value of $J=1$ indicates identical sets, while $J=0$ indicates no overlap. Below, we highlight our main observations.

The experimental results corroborate the robustness of our algorithm, which consistently identifies stable and similar layers across different alignment datasets. This is particularly noteworthy in light of recent work that suggests alignment fundamentally involves shifts in stylistic tokens (Lin et al., ). Thus, the essence of alignment is the pursuit of similar capabilities, which aligns with our discovery that the important layers corresponding to different datasets exhibit similarity. This convergence of findings underscores the intrinsic alignment of our algorithm’s performance with the fundamental objectives of dataset alignment.

Given the established importance ranking of the model layers, which proves stable for the alignment task, we must consider how to leverage this ranking. We will address this from both performance and efficiency perspectives. First, to maximize the performance of the fine-tuned model, we should avoid fine-tuning layers that could negatively impact the model, focusing instead on those deemed less significant. Second, to enhance the efficiency of fine-tuning and minimize resource consumption, we should concentrate our efforts on layers that are particularly vital to the model’s success. Detailed experiments and analyses of these two cases will be presented in the following section.

**Table 3: Comparative evaluation of Llama 2-7B and Mistral-7B-v0.1 models finetuned on the No Robots Dataset. This table presents the 5-shot test accuracy for the MMLU benchmark, alongside the 0-shot test accuracy for the Hellaswag dataset. Cells highlighted in grey indicate that ILA has enhanced the performance of the base model. The best result is marked in bold.**
| Models | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| Llama 2-7B | AdaLoRA | 45.23 | 57.30 | 5.70 | 4.05 |
| Full Finetune | 45.72 | 57.69 | 6.00 | 3.93 |  |
| Full Finetune w/ ILA | 45.98 | 57.87 | 5.90 | 4.21 |  |
| LoRA | 44.58 | 59.46 | 6.23 | 4.70 |  |
| LoRA w/ ILA | 45.78 | 59.65 | 6.30 | 4.93 |  |
| Mistral-7B-v0.1 | AdaLoRA | 62.13 | 61.68 | 6.10 | 5.03 |
| Full Finetune | 61.05 | 64.26 | 6.70 | 5.56 |  |
| Full Finetune w/ IFILA | 61.75 | 64.21 | 6.73 | 5.70 |  |
| LoRA | 61.95 | 62.90 | 6.77 | 5.35 |  |
| LoRA w/ IFILA | 62.14 | 62.80 | 6.82 | 5.42 |  |

**Table 4: Comparative evaluation of Llama 2-7B and Mistral-7B-v0.1 models finetuned on the LIMA Dataset. This table presents the 5-shot test accuracy for the MMLU benchmark, alongside the 0-shot test accuracy for the Hellaswag dataset. Cells highlighted in grey indicate that ILA has enhanced the performance of the base model. The best result is marked in bold.**
| Models | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| Llama 2-7B | AdaLoRA | 44.21 | 59.85 | 5.66 | 3.82 |
| Full Finetune | 46.36 | 62.06 | 5.85 | 3.91 |  |
| Full Finetune w/ ILA | 46.32 | 62.18 | 5.96 | 4.02 |  |
| LoRA | 43.18 | 54.52 | 5.78 | 3.98 |  |
| LoRA w/ ILA | 44.13 | 54.55 | 5.88 | 4.10 |  |
| Mistral-7B-v0.1 | AdaLoRA | 62.40 | 61.52 | 6.58 | 4.46 |
| Full Finetune | 60.11 | 63.76 | 6.99 | 5.39 |  |
| Full Finetune w/ ILA | 61.01 | 64.01 | 6.94 | 5.47 |  |
| LoRA | 60.83 | 65.42 | 6.82 | 4.88 |  |
| LoRA w/ ILA | 61.52 | 65.51 | 6.92 | 5.34 |  |

### 4.2 Enhancing Alignment Performance through Freezing Unimportant Layers

To achieve optimal model performance, we excluded the unimportant layers, specifically those whose modifications would negatively impact fine-tuning. Approximately of the unimportant layers were removed. The main results on No Robots and LIMA are presented in Table 3 and Table 4 respectively. For additional results of Llama 2-13B and main results on Alpaca-GPT4 dataset, please refer to Appendix C. Based on the results, we highlight two key observations:

The results indicate that ILA provides robust and efficient fine-tuning by focusing only on significant layers while excluding those that negatively impact the model. When compared to AdaLoRA, even though we explored a narrow range of the hyperparameter $t_{r}$ (target average rank of incremental matrices), our method performed better. This outcome highlights that simply adjusting LoRA’s matrix rank does not necessarily yield superior results in alignment tasks, as confirmed by other studies (Dettmers et al., ).

Furthermore, as discussed in Section 4.1, the stability of the layer importance ranking across various alignment datasets suggests that it is often sufficient to conduct a single importance ranking search for a given network architecture. In our experiments, we computed the layer importance ranking using full training iterations on the No Robots dataset, and then directly applied this ranking to other datasets. Although dataset-specific importance rankings can yield further improvements (see Table. 9 in Section 5), the consistent cross-dataset performance achieved using a single ranking highlights the robustness and generalizability of our approach.

**Table 5: Results of fine-tuning Mistral-7B-v0.1 on the No Robots dataset. This table presents the 5-shot test accuracy for the MMLU benchmark, along with the 0-shot test accuracy for the Hellaswag dataset. The percentages in parentheses indicate the proportion of important linear layers fine-tuned relative to all linear layers. The best results are highlighted in bold.**
| Models | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| Mistral-7B-v0.1 | LoRA | 61.95 | 62.90 | 6.77 | 5.35 |
| LoRA w/ ILA (10%) | 62.09 | 61.94 | 6.49 | 5.08 |  |
| LoRA w/ ILA (20%) | 61.83 | 62.16 | 6.60 | 5.23 |  |
| LoRA w/ ILA (30%) | 61.89 | 62.79 | 6.71 | 5.37 |  |

**Table 6: Comparison of fine-tuning results using QLoRA on Llama 2-7B and Llama 3.1-8B versus QLoRA applied to selected important layers identified by ILA. This table shows the 5-shot test accuracy for the MMLU benchmark and the 0-shot test accuracy for the Hellaswag dataset. Cells highlighted in grey indicate performance improvements achieved by ILA over the base model.**
| Datasets | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| LIMA | LoRA | 53.85 | 63.08 | 6.40 | 4.43 |
| LoRA w/ ILA (75%) | 54.33 | 62.04 | 6.54 | 4.55 |  |
| LoRA w/ ILA (30%) | 54.27 | 62.88 | 6.31 | 4.54 |  |
| NoRobots | LoRA | 54.08 | 61.73 | 6.69 | 4.94 |
| LoRA w/ ILA | 54.45 | 61.13 | 6.77 | 5.05 |  |

**Table 7: GPU memory usage for LoRA, QLoRA, and LoRA/QLoRA with only 30% of important layers fine-tuned. Batch size is set to 2, and the maximum token length is 1024. Percentages in parentheses indicate the proportion of linear layers fine-tuned.**
|  | LoRA (100%) | LoRA w/ ILA (30%) | QLoRA (100%) | QLoRA w/ ILA (30%) |
| --- | --- | --- | --- | --- |
| GPU Memory Usage (MiB) | 32988 | 25614 | 26032 | 18142 |

### 4.3 Enhancing Alignment Efficiency by Only Fine-tuning the Critical Layers

To investigate this issue, we fine-tuned only 10%, 20%, and 30% of the important layers of Mistral-7B-v0.1, as identified by ILA, on the No Robots dataset, and compared the results with the LoRA algorithm. The results demonstrate clear benefits in focusing on a subset of important layers:

These findings underline the robustness of our layer selection strategy, allowing efficient use of resources with minimal trade-offs in performance. Additionally, our integration with QLoRA confirms that fine-tuning only a targeted subset of important layers enhances both the performance and efficiency of state-of-the-art methods in reducing memory usage during fine-tuning.

To provide a more intuitive understanding of how our method reduces GPU memory usage, we measured the memory consumption of QLoRA, LoRA, and the versions that fine-tune only a subset of important layers identified by ILA in Table 7. The results show that our method reduces GPU memory requirements while maintaining competitive performance, making it an effective strategy for resource-constrained environments.

**Table 8: Performance comparison of ILA, random layer selection, and position-based layer selection for fine-tuning Llama 2-7B on the No Robots Dataset. The abbreviations RL 1 and RL 2 refer to the approach of randomly selecting $K$ layers to freeze during the fine-tuning process, with each employing a distinct random seed. FL denotes the strategy of freezing the first $K$ layers, while LL indicates the freezing of the last $K$ layers. Performance reductions compared with our ILA algorithm are highlighted in blue.**
| Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |
| LoRA | 44.58 | 59.46 | 6.23 | 4.70 |
| LoRA w/ RL 1 | 44.23 | 59.71 | 6.08 | 4.60 |
| LoRA w/ RL 2 | 43.98 | 59.11 | 6.10 | 4.68 |
| LoRA w/ FL | 44.02 | 59.32 | 6.13 | 4.59 |
| LoRA w/ LL | 44.61 | 59.21 | 6.20 | 4.63 |
| LoRA w/ ILA | 45.78 | 59.65 | 6.30 | 4.93 |

**Table 9: Results of fine-tuning Mistral-7B-v0.1 on the LIMA dataset using ILA to identify important layers from various datasets. Dataset (Imp. Layers) indicates the datasets utilized to search for the important layers. Intersection represents freezing the layers that are the intersection of the top-$K$ least important layers found from the LIMA, No Robots, and Alpaca GPT4 datasets.**
| Dataset<br>(Imp. Layers) | Dataset<br>(Finetune) | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| LIMA | LIMA | 61.82 | 65.48 | 6.99 | 5.38 |
| No Robots | LIMA | 61.52 | 65.51 | 6.92 | 5.34 |
| Alpaca-GPT4 | LIMA | 61.23 | 65.20 | 7.03 | 5.21 |
| Intersection | LIMA | 61.49 | 65.62 | 7.06 | 5.44 |

### 4.4 Ablation Study

Observation 1: Randomly or manually selecting layers for fine-tuning does not work.

To substantiate the accuracy and efficacy of the ranking and importance layers identified by our algorithm, we contrast the baseline that optimizes all linear layers without any freezing with three alternative scenarios: (1) RL 1 and RL 2, where the top-$K$ layers to be frozen are randomly selected using two different random seeds; (2) FL, which involves freezing the first $K$ linear layers; and (3) LL, which entails freezing the last $K$ linear layers. The experimental results indicate that neither the random freezing of $K$ layers nor the selective freezing of either the first or last $K$ linear layers could outperform the baseline of tuning all layers on most evaluation metrics. In contrast, our ILA can accurately identifies the layers of importance and freeze the top-$K$ least important layers, thereby achieving substantial improvements. This demonstrates that ILA effectively pinpoints the non-critical layers for freezing, optimizing the fine-tuning process and enhancing model performance without the need to adjust every layer.

Observation 2: Cross-dataset evaluation of layer importance can lead to the best results.

As indicated in Table 2, subtle differences are observed in the important layers identified across various datasets. This observation leads to an intuitive hypothesis that layers consistently deemed unimportant across all datasets may truly be non-essential. To this end, we intersect the top-$K$ least important layers from three distinct datasets (i.e., LIMA, No Robots, and Alpaca-GPT4) to determine the ultimately non-critical layers. These layers are subsequently frozen during fine-tuning, with the specific outcomes presented in Table 9.

Our analysis reveal that a holistic consideration of layer importance across multiple datasets yields superior results compared to dataset-specific approaches. For instance, identifying important layers within the LIMA dataset and fine-tuning on the No Robots dataset is less effective than an integrated approach. Similarly, finding important layers and fine-tuning exclusively on the No Robots dataset do not perform as well as the comprehensive method. This suggests that a cross-dataset evaluation of layer importance can lead to more robust and effective fine-tuning strategies.

Observation 3: The computation cost of ILA is low.

Our ILA algorithm consists of two stages. Stage 1: We use LoRA to train the model until it is sufficiently stable, i.e., $\epsilon$-stable. Stage 2: We fix the backbone network and the LoRA modules to learn the importance weights ($\gamma_{t}$). For Llama 2-7B and Mistral-7B-v0.1, $|\gamma_{t}|=225$.
To quantify computation cost, we measured the training time per iteration for Llama 2-7B in stages 1 and 2 with a batch size of 32. For stage 1, the training time is 6671 ms. For stage 2, the training time is 5343 ms. In Stage 2, we train for 128 batches on each dataset. Therefore, we only tune the model for about $5.34\times 128\div 60\approx 11$ miniutes.
The main training cost is in Stage 1. However, as shown in Table 4, it is not necessary to complete the entire training process; reaching $25\%\sim 50\%$ of the training milestones is sufficient.

## 5 Related Works

Large Language Models (LLMs) Alignment.
Language models are initially pretrained to learn general-purpose representations, enabling their transfer to a wide range of language understanding and generation tasks (Qiu et al., ; Jiang et al., ; Nijkamp et al., ).
To align these models with specific user needs and improve their performance on targeted applications, techniques such as Instruction Tuning (Zhang et al., ; Sun et al., ; Muennighoff et al., ) and Preference Learning (Hejna et al., ; Guan et al., ; Rafailov et al., ; Song et al., ; Li et al., ) are commonly employed. Tuning-based alignment can introduce issues such as forgetting in LLMs (Wang et al., ; ) and underfitting (Zhang et al., ; Sun et al., ).

Selecting an appropriate optimization method based on the characteristics of a specific task can often further enhance the model’s performance (Shi et al., ; ; Zhang et al., ; Wu et al., ). Therefore, the nature of model alignment should be explored through various studies. LIMA (Zhou et al., ) achieved a well-aligned model by fine-tuning nearly 1,000 samples using SFT, and hypothesized that the alignment process essentially teaches the model how to conduct conversations in specific formats or meet certain requirements without acquiring new knowledge. Similar findings have been reported in recent studies (Chen et al., ; Lee et al., ; Gudibande et al., ). Duan et al. () analyzed the hidden states of LLMs, exploring the similarities between in-context learning (ICL) and instruction tuning (IT) regarding their impact on downstream tasks. URIAL (Lin et al., ) investigated the token distribution before and after alignment, suggesting that alignment primarily shifts “stylistic tokens” like discourse markers and transition words, while the distribution of knowledge-intensive terms remains largely unchanged. Based on prior research, we hypothesize that the abilities learned during alignment are relatively narrow in scope. To better understand this process, we propose an approach to identify which layers are genuinely important during alignment.

Parameter Efficient Fine-Tuning (PEFT).
Full-model fine-tuning is highly complex and computationally expensive for not only computer vision (Zhang et al., ; Wang et al., ) but also natural language processing (Zhang et al., ).
To tackle the high computational costs of full-model fine-tuning, especially with Pre-trained Language Models (PLMs) ranging from billions to trillions of parameters (Brown et al., ; Fedus et al., ), PEFT methods have been developed to reduce parameter usage while maintaining the effectiveness and stability of knowledge transfer (Tang et al., ; Peng et al., ). These approaches include partial fine-tuning, which selectively targets specific model components (Zaken et al., ; Zhao et al., ; Ansell et al., ; Guo et al., ), and soft prompt-based fine-tuning (Lester et al., ; Li & Liang, ; Asai et al., ). Notable methods include BitFit (Zaken et al., ), Adapter (Houlsby et al., ), LoRA (Hu et al., ) and its variants (Zhang et al., ; Meng et al., ). Recent studies (Pan et al., ; Xu & Zhang, ; Panda et al., ) have shown that fine-tuning only a small portion of a model while masking most components can still achieve promising results in LLMs. However, these masking strategies are often applied randomly, akin to dropout, which is suboptimal and lacks consistency.
While effective for efficient fine-tuning, these methods provide limited insight into understanding the alignment task.
To overcome these limitations, our approach leverages the concept of skill localization (Panigrahi et al., ) by dynamically identifying and fine-tuning the critical components for each task. By focusing solely on the most important regions, this method significantly improves the efficiency of model fine-tuning while ensuring strong performance.

## 6 Conclusions

To gain a deeper understanding of LLM alignment, we have proposed ILA, a method for identifying critical layers in the alignment process by learning binary masks for LoRA weight matrices. ILA demonstrates consistent identification of important layers across different datasets, regardless of significant content variations, suggesting that the alignment process imparts similar capabilities to the model irrespective of the training data. This finding complements existing research and provides valuable insights into the specific roles of layers during the alignment process. By strategically tuning the vital layers, ILA effectively reduces computational overhead, and by freezing unimportant layers, it further enhances model performance, leading to more efficient resource utilization.

## Appendix A Proof of Theorem 2.1

###### Theorem A.1 .

For a sufficiently small $\epsilon$, $\theta_{T}$ is $\epsilon$-stable, thus Assumption 2.1 and Assumption 2.2 are satisfied. For any $t>T$, we assume that $\forall i,\gamma_{t}^{i}\in[0,1]$. Let $\bm{\gamma}_{t}^{\prime}$ denote the result of $\bm{\gamma}_{t}$ after one step of gradient descent, i.e., $\bm{\gamma}_{t}^{\prime}=\bm{\gamma}_{t}-\beta\nabla_{\bm{\gamma}_{t}}\mathcal
{L}(\bm{\uptheta}_{t}^{\rm mask})$. Then we have

$$ $\|\bm{\gamma}_{t}^{\prime}-\bm{\gamma}_{t+1}^{\prime}\|_{2}\leq\beta(QL_{2}+L_ {1})R\epsilon.$ (12) $$

###### Proof.

Let $\hat{\bm{\gamma}}$ be the initial values of $\bm{\gamma}_{t}$ and $\bm{\gamma}_{t+1}$. Then we have

$$ $\bm{\gamma}_{t}^{\prime}=\hat{\bm{\gamma}}-\beta\nabla_{\bm{\gamma}_{t}} \mathcal{L}(\bm{\uptheta}_{t}^{\rm mask})$ (13) $$

$$ $\bm{\gamma}_{t+1}^{\prime}=\hat{\bm{\gamma}}-\beta\nabla_{\bm{\gamma}_{t+1}} \mathcal{L}(\bm{\uptheta}_{t+1}^{\rm mask})$ (14) $$

The difference of $\bm{\gamma}_{t}^{\prime}$ and $\bm{\gamma}_{t+1}^{\prime}$ is

$$ $\displaystyle\|\bm{\gamma}_{t}^{\prime}-\bm{\gamma}_{t+1}^{\prime}\|_{2}=$ $\displaystyle\|(\hat{\bm{\gamma}}-\beta\nabla_{\bm{\gamma}_{t}}\mathcal{L}(\bm {\uptheta}_{t}^{\rm mask}))-(\hat{\bm{\gamma}}-\beta\nabla_{\bm{\gamma}_{t+1}} \mathcal{L}(\bm{\uptheta}_{t+1}^{\rm mask}))\|_{2}$ (15) $\displaystyle=$ $\displaystyle\beta\|\nabla_{\bm{\gamma}_{t}}\mathcal{L}(\bm{\uptheta}_{t}^{\rm mask })-\nabla_{\bm{\gamma}_{t+1}}\mathcal{L}(\bm{\uptheta}_{t+1}^{\rm mask})\|_{2}$ (16) $\displaystyle=$ $\displaystyle\beta\|\theta_{t}\odot\nabla_{\bm{\uptheta}_{t}^{\rm mask}}(\bm{ \uptheta}_{t}^{\rm mask})-\theta_{t+1}\odot\nabla_{\bm{\uptheta}_{t+1}^{\rm mask }}(\bm{\uptheta}_{t+1}^{\rm mask})\|_{2}$ (17) $\displaystyle\leq$ $\displaystyle\beta\|\theta_{t}\odot\left(\nabla_{\bm{\uptheta}_{t}^{\rm mask}} (\bm{\uptheta}_{t}^{\rm mask})-\nabla_{\bm{\uptheta}_{t+1}^{\rm mask}}(\bm{ \uptheta}_{t+1}^{\rm mask})\right)\|_{2}$ (18) $\displaystyle+\beta\|(\theta_{t}-\theta_{t+1})\odot\nabla_{\bm{\uptheta}_{t+1} ^{\rm mask}}(\bm{\uptheta}_{t+1}^{\rm mask})\|_{2}.$ (19) $$

Because $\mathcal{L}(\theta)$ has an L-Lipschitz continuous gradient with constant $L_{2}>0$, and $\|\theta_{t}\|\leq Q$,

$$ $\displaystyle\|\theta_{t}\odot\nabla_{\bm{\uptheta}_{t}^{\rm mask}}(\bm{ \uptheta}_{t}^{\rm mask})-\theta_{t+1}\odot\nabla_{\bm{\uptheta}_{t+1}^{\rm mask }}(\bm{\uptheta}_{t+1}^{\rm mask})\|_{2}\leq$ $\displaystyle QL_{2}\|\bm{\uptheta}_{t}^{\rm mask}-\bm{\uptheta}_{t+1}^{\rm mask }\|_{2}$ (20) $\displaystyle=$ $\displaystyle QL_{2}\|\Delta\theta_{t+1}-\Delta\theta_{t}\|_{2}$ (21) $\displaystyle=$ $\displaystyle QL_{2}\|\theta_{t+1}-\theta_{t}\|_{2}$ (22) $$

Because $\mathcal{L}(\theta)$ is $L$-smooth with constant $L_{1}$,

$$ $\displaystyle\|(\theta_{t}-\theta_{t+1})\odot\nabla_{\bm{\uptheta}_{t+1}^{\rm mask }}(\bm{\uptheta}_{t+1}^{\rm mask})\|_{2}\leq$ $\displaystyle L_{1}\|\theta_{t}-\theta_{t+1}\|.$ (23) $$

Therefore,

$$ $\displaystyle\|\bm{\gamma}_{t}^{\prime}-\bm{\gamma}_{t+1}^{\prime}\|_{2}$ $\displaystyle\leq\beta(QL_{2}+L_{1})\|\theta_{t}-\theta_{t+1}\|_{2}.$ (24) $$

According to the Assumption 2.2, we have $\|\theta_{t}-\theta_{t+1}\|_{2}\leq R\epsilon$, hence,

$$ $\|\bm{\gamma}_{t}^{\prime}-\bm{\gamma}_{t+1}^{\prime}\|_{2}\leq\beta(QL_{2}+L_ {1})R\epsilon.$ (25) $$

∎

###### Theorem A.1 .

###### Proof.

## Appendix B Experimental Setup

For all experiments, we follow fine-tuning hyperparameters: we use AdamW with $\beta_{1}$ = 0.9, $\beta_{2}$ = 0.99 and weight decay of $0.1$. The scheduler employed is a cosine scheduler with a warmup ratio of 0.01. For LoRA baselines, we set the hyperparameter rank $r$ as 32.

### B.1 No Robots Dataset

We do a hyperparameter search for LoRA over the following variables: learning rate $\{0.001,0.002,0.0005,0.0002,0.0001\}$, training epochs $\{2,3,4,5\}$. We do hyperparameter search for full fine-tuning over the following variables: learning rate $\{1e-4,2e-5,1e-5,5e-6,2e-6\}$, training epochs $\{2,3,4,5\}$.

Llama 2-7B.
Both LoRA and AdaLoRA use a dropout rate of $0.1$ and a learning rate of $0.001$. The number of training epochs is $3$. For full fine-tuning, the learning rate is set to $0.00001$, with the number of training epochs also being $3$. The training parameters for IFILA are consistent with those of the baselines.

Mistral-7B. For LoRA and AdaLorA, we set the dropout rate as $0.1$. The learning is $0.0002$. The number of training epochs is $2$. For full fine-tuning, the learning rate is set as $0.000002$ and the number of training epochs is $2$. The training parameters of IFILA are the same as the baselines.

### B.2 LIMA Dataset

We do a hyperparameter search for LoRA over the following variables: learning rate $\{0.001,0.002,0.0005,0.0002,0.0001\}$, training epochs $\{5,10,15,20\}$. We do hyperparameter search for full fine-tuning over the following variables: learning rate $\{1e-4,2e-5,1e-5,5e-6,2e-6\}$, training epochs $\{5,10,15,20\}$.

#### Llama 2 -7B

. For LoRA and AdaLorA, we set the dropout rate as $0.1$. The learning is $0.001$. The number of training epochs is $20$. For full fine-tuning, the learning rate is set as $0.00001$ and the number of training epochs is $5$. The training parameters of IFILA are the same as the baselines.

#### Mistral-7B.

For LoRA and AdaLorA, we set the dropout rate as $0.1$. The learning is $0.0002$. The number of training epochs is $5$. For full fine-tuning, the learning rate is set as $0.000005$ and the number of training epochs is $5$. The training parameters of IFILA are the same as the baselines.

### B.3 Alpaca-GPT Dataset.

We do a hyperparameter search for LoRA over the following variables: learning rate $\{0.001,0.002,0.0005,0.0002,0.0001\}$, training epochs $\{0.5,1,1.5,2,3\}$. We do hyperparameter search for full fine-tuning over the following variables: learning rate $\{1e-4,2e-5,1e-5,5e-6,2e-6\}$, training epochs $\{0.5,1,1.5,2,3\}$.

#### Llama 2 -7B

. For LoRA and AdaLorA, we set the dropout rate as $0.1$. The learning is $0.0002$. The number of training epochs is $1.5$. For full fine-tuning, the learning rate is set as $0.000002$ and the number of training epochs is $0.5$. The training parameters of IFILA are the same as the baselines.

#### Mistral-7B.

For LoRA and AdaLorA, we set the dropout rate as $0.1$. The learning is $0.0002$. The number of training epochs is $5$. For full fine-tuning, the learning rate is set as $0.000002$ and the number of training epochs is $0.5$. The training parameters of IFILA are the same as the baselines.

## Appendix C Additional Experiments

**Table 10: Fine-tuning results of Llama 2-13B on the LIMA and No Robots datasets. This table shows the 5-shot test accuracy for the MMLU benchmark along with the 0-shot test accuracy for the Hellaswag dataset. Cells highlighted in grey indicate that ILA has improved the performance of the base model.**
| Datasets | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| LIMA | LoRA | 53.85 | 63.08 | 6.40 | 4.43 |
| LoRA w/ ILA | 54.33 | 62.04 | 6.54 | 4.55 |  |
| No Robots | LoRA | 54.08 | 61.73 | 6.69 | 4.94 |
| LoRA w/ ILA | 54.45 | 61.13 | 6.77 | 5.05 |  |

**Table 11: Comparative Evaluation of Llama 2-7B and Mistral-7B-v0.1 Models finetuned on the Alpaca-GPT4 Dataset. This table presents the 5-shot test accuracy for the MMLU benchmark, alongside the 0-shot test accuracy for the Hellaswag dataset. Cells highlighted in grey indicate that ILA has enhanced the performance of the base model. The best result is marked in bold.**
| Models | Methods | Language Understanding | Conversational Ability |  |  |
| --- | --- | --- | --- | --- | --- |
| MMLU $\uparrow$ | Hellaswag $\uparrow$ | Vicuna $\uparrow$ | MT-Bench $\uparrow$ |  |  |
| Llama 2-7B | AdaLoRA | 46.13 | 57.85 | 7.06 | 3.90 |
| Full Finetune | 45.91 | 57.73 | 4.62 | 3.56 |  |
| Full Finetune w/ ILA | 46.23 | 57.67 | 5.03 | 4.01 |  |
| LoRA | 43.66 | 58.49 | 6.91 | 4.21 |  |
| LoRA w/ ILA | 44.69 | 58.22 | 7.01 | 4.58 |  |
| Mistral-7B-v0.1 | AdaLoRA | 62.48 | 62.08 | 7.43 | 5.51 |
| Full Finetune | 60.56 | 62.80 | 4.55 | 3.82 |  |
| Full Finetune w/ ILA | 60.88 | 62.91 | 5.22 | 4.11 |  |
| LoRA | 61.82 | 62.70 | 7.31 | 6.15 |  |
| LoRA w/ ILA | 62.14 | 62.80 | 7.45 | 6.19 |  |

### C.1 Additional Experiments on Model Scalability

To assess whether freezing unimportant layers continues to enhance model performance at a larger scale, we conducted additional experiments on Llama 2-13B. Specifically, we fine-tuned Llama 2-13B using the No Robots and LIMA datasets, with results compared against LoRA presented in the table below. The experimental outcomes demonstrate that our method maintains strong performance on Llama 2-13B. Despite the increased model size, the underlying architectural similarities suggest that our approach remains effective and scalable, likely extending its benefits to even larger models.

We also carried out further experiments on Alpaca-GPT4 using Llama 2-7B and Mistral-7B-v0.1 to evaluate the adaptability of our approach across different model architectures. Consistently, our method outperformed LoRA while requiring fewer layers to be fine-tuned. These findings further validate the robustness and scalability of our approach, showing its capability to effectively enhance performance across various model sizes and architectural variations.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. What does "IFILA" represent in Table 3?
2. Are the descriptions at L303-304 wrong for Table 3? It seems that there is no $\gamma$ in Table 3.
3. According to L253, **three** distinct datasets are used to evaluate the Language Understanding Ability aspect. Would you please clarify why there are only two datasets for this aspect in the paper?
4. According to L323, approximately 25% of the unimportant layers are removed. This statement should be related to Table 3 and Table 4. So we can understand that 75% of important layers are retained for the results in Table 3 and Table 4, which show the main results of the proposed method. However, Table 7 does not list the memory usage of the proposed method with 75% important layers. Does this show that the proposed method needs more memory to outperform LoRA?
5. According to Table 3 and Table 11, the proposed method often failed to enhance performance on the Hellaswag dataset. Are there potential reasons for this discrepancy?
6. Why is AdaLoRA w/ILA not included in the comparison? Is there was a specific reason for the omission?
soundness: 3
strengths: 1. The proposed method can offer improvments for fine-tuning LLMs while saving memory usage.
2. Extensive experiments are provided in this paper.
summary: This paper introduces a novel and interesting approach, Important Layers for Alignment (ILA), to enhance the fine-tuning efficiency of large language models (LLM) for alignment by identifying and selectively tuning the most important layers. The unimportant layers identified by the proposed ILA method will be freezed during the last part of fine-tuning.
weaknesses: 1. **Limited improvements**. According to the results presented, ILA does not bring much improvments. The performances of the models using ILA and the ones without ILA are close. For example, in Table 4, only 0.12% increase from "Full Finetune" to "Full Finetune w/ILA" with the LLAMA 2-7B model.
2. **Potential overlap with existing PEFT methods.** The authors may need to clarify why we need an additional PEFT method and what values freezing unimportant layers can bring compared with the existing PEFT methods.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: - What is the overhead of running this method (runing time, computational complexity) in comparison to full-fine tuning?
- What is the stability of the algorithm for optimizing layer importance scores with respect to the initial scores?
soundness: 1
strengths: - The work describes a potentially useful method
summary: The paper proposes a method for masking (pruning) layers of a pretrained LLM before fine-tuning (aligning) them to target tasks. The method appears to work by iteratively switching between optimizing the loss until it becomes stable and searching for a set of layers to mask that still sufficiently minimizes the loss, with the latter framed as a constrained optimization problem and adapted to LoRA for efficiency. Experiments with three open-source LLMs and four datasets suggest that layer importance ranking is consistent across datasets and that freezing (masking) unimportant layers may increase performance.
weaknesses: 1. Lack of clarity: the method is not well explained, and important technical details are missing
2. No comparison to similar methods
3. Potentially unsupported inferences from experimental results
4. Lack of mathematical rigor

I elaborate on these points below.

# Lack of clarity

I struggled to understand how the method works. Terms such as "important," "unimportant," "significant," and "insignificant" layers are not defined. Algorithm 1 is not explained nor connected to the formulas in Section 2 (e.g., what is K?). Additionally, I don’t understand how Eq. (7) is a reparametrization of Eq. (3), as there is no constraint on ∣∣s∣∣ (i.e., Eq. (3) is a constrained optimization problem, while Eq. (7) is not). Consequently, it’s unclear how the method functions (where is the incentive to minimize the number of masked layers?). I also didn’t understand how Theorem 2.1 relates to the method and algorithm. What is L_∞​ in Eq. (8), and what is R in Eq. (9)?

How was fine-tuning performed? Was this instruction tuning, and on which portions of the dataset was it carried out? Are the results shown for the test portion? For example, for MMLU, only the test set is provided, so how was the split done?

Overall, I failed to grasp how the method actually works. Do you fix the number of layers in advance ("number of insignificant layers K") and then rank and select the top-ranked layers? If so, this raises the question: why not initially select a smaller number of layers in the optimization algorithm and dispense with ranking altogether? I presume there are correlations between layers, and the method seems to be essentially selecting a subset of layers when identifying the "important" ones.

# No comparison to similar methods

This work appears to be directly related to layer pruning (e.g., [1], [2], [3], and [4], to mention a few). The proposed method should be compared to existing methods, both conceptually and empirically, in terms of performance.

- [1] Lie et al. 2024. Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy.  https://arxiv.org/abs/2404.06954
- [2] Gromov et al 2024. The Unreasonable Ineffectiveness of the Deeper Layers. https://arxiv.org/abs/2403.17887
- [3] Chen et al 2024. Compressing Large Language Models by Streamlining the Unimportant Layer. https://arxiv.org/abs/2403.19135
- [4] van der Ouderaa, T. F., Nagel, M., Van Baalen, M., Asano, Y. M., & Blankevoort, T. (2023). The llm surgeon. _arXiv preprint arXiv:2312.17244_.

# Potentially unsupported inferences

A Jaccard similarity of importance layers (shown in Figure 1) leads the authors to conclude that "important layers vary between different network architectures" while "there is a significant overlap (up to 90%) in the important layers identified by ILA across different alignment datasets." Visually, however, there also appear to be similarities across architectures for a fixed dataset, but this aspect hasn't been quantified. A claim of inter-architecture/intra-dataset similarity and intra-architecture dissimilarity would ideally be supported by statistical hypothesis tests (though demonstrating this statistically would require more models and datasets). In the absence of such evidence, I urge caution.

Similarly, results in Tables 3 and 4 are not accompanied by statistical tests of difference. If the authors wish to claim that the proposed method improves performance, I suggest running experiments with different seeds and conducting statistical tests for the significance of score differences between configurations with and without ILA, as the numerical differences are usually small.

# Lack of mathematical rigor

The method description in section 2 could benefit from some mathematical rigor. 

- thetas and the binary mask: better defined as a sequence than a set (also for component-wise multiplication to be defined)
- eq (2) refers to the loss function as a two-argument function (which makes sense), but earlier this is not how the loss function is defined
- eq 8, 9: theta should be a vector (boldface symbol)
- eq 8, 9: one vertical bar missing
- assumption 2.2: epsilon-stable has been defined for a model wrt its loss, not for parameters directly. Is this now extending the definition of epsilon-stable to parameter vectors? What is the relation between assumption 2.2 and definition 1?
- Theorem 2.1: Theorem assumption 2.1 talks about the Lipschitz conitnuity of the loss function (not across iterations) -- it's unclear to me how this relates to stability across iterations. Assumption 2.2 is about stability across iterations, but I don't see how "a sufficiently small epsilon" makes theta_T stable by eq (10). If anything, a lager epsilon would make it easier to satisfy the inequality.
- proof: lines 16->17: where's the loss function gone?
- the algorithm coud perhaps be more clearly formalized with a while loop
- line 484: |γt| = 225. Is this denoting the norm of the vector? Do you mean to say that there are that many layers in total? But since γt is a binary vector, we'll have ||γt||<=225

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Is the overall training process of the ILA method divided into two steps? The first step involves executing the ILA algorithm from Algorithm 1, and the second step involves freezing the unimportant layers selected by ILA before re-fine-tuning.

2. What does IFILA mean? This abbreviation does not appear in the main text, but it is mentioned in the experimental tables without any explanation.

3. In Table 6, why do the experiments on LIMA include ILA (75%) and ILA (30%), but there are no corresponding entries for NoRobots?
soundness: 3
strengths: This paper is well-written and easy to understand. The proposed ILA method is also intuitive and has achieved excellent results.
summary: This paper proposes the ILA method, which first trains the model using LoRA until it reaches a stable phase (i.e. when the parameter changes during training are below a certain threshold). Then, a binary mask is added for individual training to assess the importance of each layer. Finally, unimportant layers are frozen, and fine-tuning is performed to achieve better performance.
weaknesses: 1. I am somewhat concerned about whether the contributions of this paper are sufficient, as [1][2][3][4] indicate that adjusting certain parameters/layers during the fine-tuning process can indeed lead to effective improvements. Additionally, [5] shows that there is significant redundancy in the parameters during the SFT process. I believe that existing work already highlights the necessity of adjusting certain parameters during the post-training phase. The authors should emphasize the contributions of this paper more clearly, explain the similarities and differences with existing methods, and include comparisons with those methods.

2. I believe that more baselines could be added for comparison, such as HFT [1], LISA [2], and GMT [3]. These methods also tune only a subset of model parameters, and I think including these baselines would make the paper more convincing.

3. The focus of this paper is on alignment, and it has also achieved performance improvements. However, I am curious whether the methods presented in this paper exhibit consistent performance in other areas, such as mathematical reasoning and code generation. I believe the authors could further discuss whether the ILA method has general applicability.

4. I am not sure whether the experimental results in Table 6 correspond to Llama 2 7B or Llama 3.1 8B. Based on the results, it seems that they still correspond to Llama 2 7B, implying that the authors did not conduct experiments with Llama 3.1 8B, yet included this model in the baselines.

5. The experimental design in the paper is not sufficiently reasonable. For example, the inclusion of the AdaLoRA method in Table 3 feels rather abrupt. I believe that the method proposed in this paper should be a pluggable approach that can be applied to AdaLoRA, but the authors have awkwardly inserted AdaLoRA into the experiments and compared it with LoRA and LoRA w/ ILA. I think there should be separate comparative experiments for AdaLoRA and AdaLoRA w/ ILA to further demonstrate the general applicability of the ILA method.

6. Table 8 should include comparisons with fine-grained selection methods proposed in [1].

Finally, I would be very happy to engage in further discussion with the authors. My main concern is the contributions of this paper, as existing work has already indicated that freezing certain parameters can lead to improvements. I need to see more experimental comparisons and a deeper discussion of the innovations presented in this paper. If these issues are addressed, I would be happy to raise my score.


[1] HFT: Half Fine-Tuning for Large Language Models

[2] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

[3] Gradient-Mask Tuning Elevates the Upper Limits of LLM Performance

[4] Investigating Layer Importance in Large Language Models

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: 1) This is the first time I've come across someone referring to instruction tuning as "alignment." Perhaps I missed this usage, but I'm more familiar with "alignment" in the context of RLHF or aligning models with human preferences.

2) Drawing parallels with interpretability studies would be beneficial, as they provide extensive insights into the importance of layers. For instance, studies like this one (https://arxiv.org/pdf/2309.04827) and others have already provided insights on layer significance, which could strengthen the paper’s foundation and contextualize its approach.
soundness: 2
strengths: 1) The authors evaluate their method on several modern benchmarks, including MT-Bench
2) Interesting results from the perspective of layer significance. 
3) The observed transferability across datasets is a strong indicator of the method’s robustness. I particularly appreciated the inclusion of out-of-distribution testing within the ablation study, along with the use of Jaccard similarities for comparison. 
4) The paper benefits from a well-organized structure and visually appealing presentation.
summary: The paper introduces a method for identifying the most significant layers in language models by solving a minimization problem across the layers. The authors apply their technique to the LLaMA-2-7B and Mistral-7B models, experimenting with both LoRA and full parameter tuning approaches. By identifying the key layers, they then focus fine-tuning efforts solely on these layers, optimizing model efficiency and performance.
weaknesses: Major Weaknesses:
1) Lack of technical details on optimal layers finding. 

1.1)This process can require considerable training time before stabilization occurs, especially for larger models. Moreover, it’s difficult to guarantee that stabilization will ever fully occur, given the non-convex nature of the optimization problem.

1.2) Additionally, while the paper reports efficiency improvements, the measurements don’t account for the compute required for pre-training and layer selection. As a result, the overall computational cost could potentially exceed that of regular LoRA fine-tuning. It would be nice to account for that in the future.

1.3) Please clarify how stability is checked. If Monte Carlo estimation is used, specify the sufficient number of samples and the reasoning behind this choice.


2) The effect is truly visible, but it may be just the regularization, that undermines all the strengths of the paper. For example, It's stated that this approach outperforms any type of regularization, yet no regularization baselines are provided. For instance, in the original LIMA paper, they apply progressive dropout to enhance robustness. You mention using dropout, but perhaps increasing it or raising weight decay could also be beneficial? 
	
2.1) Another detail that caught my attention is the modest performance gain. This is acceptable for a parameter-saving method, but with the current experiments, it’s hard to determine if the gain is due to the method itself or simply an effect of regularization. In cases of such small improvements, it would be beneficial to include runs with multiple seeds and average the results. According to several studies, LoRA can be unstable, with results varying based on the seed and checkpoint used.

3) The primary weakness here is the lack of novelty, as the main technique relies on identifying the most significant layers through gradient-based methods.

3.1) This setup aligns with challenges commonly seen in compression studies, where quantization addresses sensitive layers and columns (https://arxiv.org/abs/2408.15300). However, no inspiration, metrics, or methods from these studies were referenced, cited, or discussed.

3.2) Pruning, rather than quantization, seems most suitable here as it directly addresses this setting (see https://arxiv.org/abs/2310.06694, https://arxiv.org/abs/2204.00408). For instance, methods like Sheared LLaMA and CoFi learn binary masks to select not only layers but also attention heads and even individual neurons for fine-tuning. This makes the approach used here far from novel.

3.3) Moreover, what is the motivation behind tuning specific layers? Why not tune the precise matrices or even the weights within those matrices, as outlined in studies above? This approach offers a more general formulation, with layer tuning being a subset of this broader framework.

The paper would be of greater practical interest if it either demonstrated an easy and highly effective application with clear comparisons that outperform previous methods or provided deep insights into the internal structure. Currently, it feels like it’s trying to balance between these options without fully achieving either, especially given that the core idea lacks novelty.


Minor Weaknesses:

1) I have doubts about the preliminary experiments, as FF layers simply add more parameters for training, making the settings unequal.

2) The terms "IFILA" and "ILA" are used interchangeably in the paper, which can lead to confusion

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: See in weaknesses.
soundness: 3
strengths: On one hand, the authors provide detailed motivation in the introduction explaining why freezing different layers during training is necessary, which can help reduce catastrophic forgetting to some extent. On the other hand, the paper offers comprehensive empirical evidence showing their proposed method achieves generally improved performance across LoRA, AdaLoRA, QLoRA and full fine-tuning.
summary: This paper proposes an ILA (identify the important layers for LLM alignment) method to identify the most relevant layers for alignment tasks, then improves performance by freezing irrelevant layers. The authors focus on Alignment tasks, including LIMA, Alpaca, and no robots.
weaknesses: Indeed, this paper's novelty is limited. The core motivation and main method of freezing certain layers to avoid overfitting was proposed three years ago in paper [1], which even provided finer-grained control over the degree of parameter freezing. In my view, the authors merely validated this approach on Alignment tasks (just one type of fine-tuning task). While I acknowledge the technical implementations differ, given the similar research motivations and the limited application scope of this method, I believe there's room for improvement.

The authors mainly study the impact of controlling layer freezing during fine-tuning on language models. However, since most experiments and methods are LoRA-based, I believe the discussion should focus more on full parameter fine-tuning instead.

---

[1] Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning