Understanding Layer Significance in LLM Alignment

Authors: Guangyuan SHI, ZEXIN LU, Xiaoyu DONG, zhangwenlong, Xuanyu Zhang, Yujie Feng, Xiao-Ming Wu
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: LLMs, Alignment, Important Layers

Abstract:
Aligning large language models (LLMs) through fine-tuning is essential for tailoring them to specific applications. Therefore, understanding what LLMs learn during the alignment process is crucial. Recent studies suggest that alignment primarily adjusts a model's presentation style rather than its foundational knowledge, indicating that only certain components of the model are significantly impacted. To delve deeper into LLM alignment, we propose to identify which layers within LLMs are most critical to the alignment process, thereby uncovering how alignment influences model behavior at a granular level. We propose a novel approach to identify the important layers for LLM alignment (ILA). It involves learning a binary mask for each incremental weight matrix in the LoRA algorithm, indicating the significance of each layer. ILA consistently identifies important layers across various alignment datasets, with nearly 90% overlap even with substantial dataset differences, highlighting fundamental patterns in LLM alignment. Experimental results indicate that freezing non-essential layers improves overall model performance, while selectively tuning the most critical layers significantly enhances fine-tuning efficiency with minimal performance loss.

Decision: reject
Reviewer scores: 6, 3, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730300888328

thread_type:
official_review

<thread_object>
creator: reviewer_epzj

<content>
summary:
This paper proposes an ILA (identify the important layers for LLM alignment) method to identify the most relevant layers for alignment tasks, then improves performance by freezing irrelevant layers. The authors focus on Alignment tasks, including LIMA, Alpaca, and no robots.

strengths:
On one hand, the authors provide detailed motivation in the introduction explaining why freezing different layers during training is necessary, which can help reduce catastrophic forgetting to some extent. On the other hand, the paper offers comprehensive empirical evidence showing their proposed method achieves generally improved performance across LoRA, AdaLoRA, QLoRA and full fine-tuning.

weaknesses:
Indeed, this paper's novelty is limited. The core motivation and main method of freezing certain layers to avoid overfitting was proposed three years ago in paper [1], which even provided finer-grained control over the degree of parameter freezing. In my view, the authors merely validated this approach on Alignment tasks (just one type of fine-tuning task). While I acknowledge the technical implementations differ, given the similar research motivations and the limited application scope of this method, I believe there's room for improvement.

The authors mainly study the impact of controlling layer freezing during fine-tuning on language models. However, since most experiments and methods are LoRA-based, I believe the discussion should focus more on full parameter fine-tuning instead.

---

[1] Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning

questions:
See in weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730572618950

thread_type:
official_review

<thread_object>
creator: reviewer_qj17

<content>
summary:
The paper introduces a method for identifying the most significant layers in language models by solving a minimization problem across the layers. The authors apply their technique to the LLaMA-2-7B and Mistral-7B models, experimenting with both LoRA and full parameter tuning approaches. By identifying the key layers, they then focus fine-tuning efforts solely on these layers, optimizing model efficiency and performance.

strengths:
1) The authors evaluate their method on several modern benchmarks, including MT-Bench
2) Interesting results from the perspective of layer significance. 
3) The observed transferability across datasets is a strong indicator of the method’s robustness. I particularly appreciated the inclusion of out-of-distribution testing within the ablation study, along with the use of Jaccard similarities for comparison. 
4) The paper benefits from a well-organized structure and visually appealing presentation.

weaknesses:
Major Weaknesses:
1) Lack of technical details on optimal layers finding. 

1.1)This process can require considerable training time before stabilization occurs, especially for larger models. Moreover, it’s difficult to guarantee that stabilization will ever fully occur, given the non-convex nature of the optimization problem.

1.2) Additionally, while the paper reports efficiency improvements, the measurements don’t account for the compute required for pre-training and layer selection. As a result, the overall computational cost could potentially exceed that of regular LoRA fine-tuning. It would be nice to account for that in the future.

1.3) Please clarify how stability is checked. If Monte Carlo estimation is used, specify the sufficient number of samples and the reasoning behind this choice.


2) The effect is truly visible, but it may be just the regularization, that undermines all the strengths of the paper. For example, It's stated that this approach outperforms any type of regularization, yet no regularization baselines are provided. For instance, in the original LIMA paper, they apply progressive dropout to enhance robustness. You mention using dropout, but perhaps increasing it or raising weight decay could also be beneficial? 
	
2.1) Another detail that caught my attention is the modest performance gain. This is acceptable for a parameter-saving method, but with the current experiments, it’s hard to determine if the gain is due to the method itself or simply an effect of regularization. In cases of such small improvements, it would be beneficial to include runs with multiple seeds and average the results. According to several studies, LoRA can be unstable, with results varying based on the seed and checkpoint used.

3) The primary weakness here is the lack of novelty, as the main technique relies on identifying the most significant layers through gradient-based methods.

3.1) This setup aligns with challenges commonly seen in compression studies, where quantization addresses sensitive layers and columns (https://arxiv.org/abs/2408.15300). However, no inspiration, metrics, or methods from these studies were referenced, cited, or discussed.

3.2) Pruning, rather than quantization, seems most suitable here as it directly addresses this setting (see https://arxiv.org/abs/2310.06694, https://arxiv.org/abs/2204.00408). For instance, methods like Sheared LLaMA and CoFi learn binary masks to select not only layers but also attention heads and even individual neurons for fine-tuning. This makes the approach used here far from novel.

3.3) Moreover, what is the motivation behind tuning specific layers? Why not tune the precise matrices or even the weights within those matrices, as outlined in studies above? This approach offers a more general formulation, with layer tuning being a subset of this broader framework.

The paper would be of greater practical interest if it either demonstrated an easy and highly effective application with clear comparisons that outperform previous methods or provided deep insights into the internal structure. Currently, it feels like it’s trying to balance between these options without fully achieving either, especially given that the core idea lacks novelty.


Minor Weaknesses:

1) I have doubts about the preliminary experiments, as FF layers simply add more parameters for training, making the settings unequal.

2) The terms "IFILA" and "ILA" are used interchangeably in the paper, which can lead to confusion

questions:
1) This is the first time I've come across someone referring to instruction tuning as "alignment." Perhaps I missed this usage, but I'm more familiar with "alignment" in the context of RLHF or aligning models with human preferences.

2) Drawing parallels with interpretability studies would be beneficial, as they provide extensive insights into the importance of layers. For instance, studies like this one (https://arxiv.org/pdf/2309.04827) and others have already provided insights on layer significance, which could strengthen the paper’s foundation and contextualize its approach.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730610642450

thread_type:
official_review

<thread_object>
creator: reviewer_pkby

<content>
summary:
This paper proposes the ILA method, which first trains the model using LoRA until it reaches a stable phase (i.e. when the parameter changes during training are below a certain threshold). Then, a binary mask is added for individual training to assess the importance of each layer. Finally, unimportant layers are frozen, and fine-tuning is performed to achieve better performance.

strengths:
This paper is well-written and easy to understand. The proposed ILA method is also intuitive and has achieved excellent results.

weaknesses:
1. I am somewhat concerned about whether the contributions of this paper are sufficient, as [1][2][3][4] indicate that adjusting certain parameters/layers during the fine-tuning process can indeed lead to effective improvements. Additionally, [5] shows that there is significant redundancy in the parameters during the SFT process. I believe that existing work already highlights the necessity of adjusting certain parameters during the post-training phase. The authors should emphasize the contributions of this paper more clearly, explain the similarities and differences with existing methods, and include comparisons with those methods.

2. I believe that more baselines could be added for comparison, such as HFT [1], LISA [2], and GMT [3]. These methods also tune only a subset of model parameters, and I think including these baselines would make the paper more convincing.

3. The focus of this paper is on alignment, and it has also achieved performance improvements. However, I am curious whether the methods presented in this paper exhibit consistent performance in other areas, such as mathematical reasoning and code generation. I believe the authors could further discuss whether the ILA method has general applicability.

4. I am not sure whether the experimental results in Table 6 correspond to Llama 2 7B or Llama 3.1 8B. Based on the results, it seems that they still correspond to Llama 2 7B, implying that the authors did not conduct experiments with Llama 3.1 8B, yet included this model in the baselines.

5. The experimental design in the paper is not sufficiently reasonable. For example, the inclusion of the AdaLoRA method in Table 3 feels rather abrupt. I believe that the method proposed in this paper should be a pluggable approach that can be applied to AdaLoRA, but the authors have awkwardly inserted AdaLoRA into the experiments and compared it with LoRA and LoRA w/ ILA. I think there should be separate comparative experiments for AdaLoRA and AdaLoRA w/ ILA to further demonstrate the general applicability of the ILA method.

6. Table 8 should include comparisons with fine-grained selection methods proposed in [1].

Finally, I would be very happy to engage in further discussion with the authors. My main concern is the contributions of this paper, as existing work has already indicated that freezing certain parameters can lead to improvements. I need to see more experimental comparisons and a deeper discussion of the innovations presented in this paper. If these issues are addressed, I would be happy to raise my score.


[1] HFT: Half Fine-Tuning for Large Language Models

[2] LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning

[3] Gradient-Mask Tuning Elevates the Upper Limits of LLM Performance

[4] Investigating Layer Importance in Large Language Models

questions:
1. Is the overall training process of the ILA method divided into two steps? The first step involves executing the ILA algorithm from Algorithm 1, and the second step involves freezing the unimportant layers selected by ILA before re-fine-tuning.

2. What does IFILA mean? This abbreviation does not appear in the main text, but it is mentioned in the experimental tables without any explanation.

3. In Table 6, why do the experiments on LIMA include ILA (75%) and ILA (30%), but there are no corresponding entries for NoRobots?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654192271

thread_type:
official_review

<thread_object>
creator: reviewer_tmgu

<content>
summary:
The paper proposes a method for masking (pruning) layers of a pretrained LLM before fine-tuning (aligning) them to target tasks. The method appears to work by iteratively switching between optimizing the loss until it becomes stable and searching for a set of layers to mask that still sufficiently minimizes the loss, with the latter framed as a constrained optimization problem and adapted to LoRA for efficiency. Experiments with three open-source LLMs and four datasets suggest that layer importance ranking is consistent across datasets and that freezing (masking) unimportant layers may increase performance.

strengths:
- The work describes a potentially useful method

weaknesses:
1. Lack of clarity: the method is not well explained, and important technical details are missing
2. No comparison to similar methods
3. Potentially unsupported inferences from experimental results
4. Lack of mathematical rigor

I elaborate on these points below.

# Lack of clarity

I struggled to understand how the method works. Terms such as "important," "unimportant," "significant," and "insignificant" layers are not defined. Algorithm 1 is not explained nor connected to the formulas in Section 2 (e.g., what is K?). Additionally, I don’t understand how Eq. (7) is a reparametrization of Eq. (3), as there is no constraint on ∣∣s∣∣ (i.e., Eq. (3) is a constrained optimization problem, while Eq. (7) is not). Consequently, it’s unclear how the method functions (where is the incentive to minimize the number of masked layers?). I also didn’t understand how Theorem 2.1 relates to the method and algorithm. What is L_∞​ in Eq. (8), and what is R in Eq. (9)?

How was fine-tuning performed? Was this instruction tuning, and on which portions of the dataset was it carried out? Are the results shown for the test portion? For example, for MMLU, only the test set is provided, so how was the split done?

Overall, I failed to grasp how the method actually works. Do you fix the number of layers in advance ("number of insignificant layers K") and then rank and select the top-ranked layers? If so, this raises the question: why not initially select a smaller number of layers in the optimization algorithm and dispense with ranking altogether? I presume there are correlations between layers, and the method seems to be essentially selecting a subset of layers when identifying the "important" ones.

# No comparison to similar methods

This work appears to be directly related to layer pruning (e.g., [1], [2], [3], and [4], to mention a few). The proposed method should be compared to existing methods, both conceptually and empirically, in terms of performance.

- [1] Lie et al. 2024. Accelerating Inference in Large Language Models with a Unified Layer Skipping Strategy.  https://arxiv.org/abs/2404.06954
- [2] Gromov et al 2024. The Unreasonable Ineffectiveness of the Deeper Layers. https://arxiv.org/abs/2403.17887
- [3] Chen et al 2024. Compressing Large Language Models by Streamlining the Unimportant Layer. https://arxiv.org/abs/2403.19135
- [4] van der Ouderaa, T. F., Nagel, M., Van Baalen, M., Asano, Y. M., & Blankevoort, T. (2023). The llm surgeon. _arXiv preprint arXiv:2312.17244_.

# Potentially unsupported inferences

A Jaccard similarity of importance layers (shown in Figure 1) leads the authors to conclude that "important layers vary between different network architectures" while "there is a significant overlap (up to 90%) in the important layers identified by ILA across different alignment datasets." Visually, however, there also appear to be similarities across architectures for a fixed dataset, but this aspect hasn't been quantified. A claim of inter-architecture/intra-dataset similarity and intra-architecture dissimilarity would ideally be supported by statistical hypothesis tests (though demonstrating this statistically would require more models and datasets). In the absence of such evidence, I urge caution.

Similarly, results in Tables 3 and 4 are not accompanied by statistical tests of difference. If the authors wish to claim that the proposed method improves performance, I suggest running experiments with different seeds and conducting statistical tests for the significance of score differences between configurations with and without ILA, as the numerical differences are usually small.

# Lack of mathematical rigor

The method description in section 2 could benefit from some mathematical rigor. 

- thetas and the binary mask: better defined as a sequence than a set (also for component-wise multiplication to be defined)
- eq (2) refers to the loss function as a two-argument function (which makes sense), but earlier this is not how the loss function is defined
- eq 8, 9: theta should be a vector (boldface symbol)
- eq 8, 9: one vertical bar missing
- assumption 2.2: epsilon-stable has been defined for a model wrt its loss, not for parameters directly. Is this now extending the definition of epsilon-stable to parameter vectors? What is the relation between assumption 2.2 and definition 1?
- Theorem 2.1: Theorem assumption 2.1 talks about the Lipschitz conitnuity of the loss function (not across iterations) -- it's unclear to me how this relates to stability across iterations. Assumption 2.2 is about stability across iterations, but I don't see how "a sufficiently small epsilon" makes theta_T stable by eq (10). If anything, a lager epsilon would make it easier to satisfy the inequality.
- proof: lines 16->17: where's the loss function gone?
- the algorithm coud perhaps be more clearly formalized with a while loop
- line 484: |γt| = 225. Is this denoting the norm of the vector? Do you mean to say that there are that many layers in total? But since γt is a binary vector, we'll have ||γt||<=225

questions:
- What is the overhead of running this method (runing time, computational complexity) in comparison to full-fine tuning?
- What is the stability of the algorithm for optimizing layer importance scores with respect to the initial scores?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730952874689

thread_type:
official_review

<thread_object>
creator: reviewer_uhz2

<content>
summary:
This paper introduces a novel and interesting approach, Important Layers for Alignment (ILA), to enhance the fine-tuning efficiency of large language models (LLM) for alignment by identifying and selectively tuning the most important layers. The unimportant layers identified by the proposed ILA method will be freezed during the last part of fine-tuning.

strengths:
1. The proposed method can offer improvments for fine-tuning LLMs while saving memory usage.
2. Extensive experiments are provided in this paper.

weaknesses:
1. **Limited improvements**. According to the results presented, ILA does not bring much improvments. The performances of the models using ILA and the ones without ILA are close. For example, in Table 4, only 0.12% increase from "Full Finetune" to "Full Finetune w/ILA" with the LLAMA 2-7B model.
2. **Potential overlap with existing PEFT methods.** The authors may need to clarify why we need an additional PEFT method and what values freezing unimportant layers can bring compared with the existing PEFT methods.

questions:
1. What does "IFILA" represent in Table 3?
2. Are the descriptions at L303-304 wrong for Table 3? It seems that there is no $\gamma$ in Table 3.
3. According to L253, **three** distinct datasets are used to evaluate the Language Understanding Ability aspect. Would you please clarify why there are only two datasets for this aspect in the paper?
4. According to L323, approximately 25% of the unimportant layers are removed. This statement should be related to Table 3 and Table 4. So we can understand that 75% of important layers are retained for the results in Table 3 and Table 4, which show the main results of the proposed method. However, Table 7 does not list the memory usage of the proposed method with 75% important layers. Does this show that the proposed method needs more memory to outperform LoRA?
5. According to Table 3 and Table 11, the proposed method often failed to enhance performance on the Hellaswag dataset. Are there potential reasons for this discrepancy?
6. Why is AdaLoRA w/ILA not included in the comparison? Is there was a specific reason for the omission?
</content>
</thread_object>

</thread>


Metareview: