Spectral Insights into Data-Oblivious Critical Layers in Large Language Models
Authors: Xuyuan Liu, Lei Hsiung, Yaoqing Yang, Yujun Yan
Accepted at: acl25

Spectral Insights into Data-Oblivious Critical Layers in Large Language Models

Xuyuan Liu, Lei Hsiung, Yaoqing Yang, Yujun Yan
Dartmouth College
{xuyuan.liu.gr, lei.hsiung.gr, yaoqing.yang, yujun.yan}@dartmouth.edu

Abstract

Understanding how feature representations evolve across layers in large language models (LLMs) is key to improving their interpretability and robustness. While recent studies have identified critical layers linked to specific functions or behaviors, these efforts typically rely on data-dependent analyses of fine-tuned models, limiting their use to post-hoc settings. In contrast, we introduce a data-oblivious approach to identify intrinsic critical layers in pre-fine-tuned LLMs by analyzing representation dynamics via Centered Kernel Alignment (CKA). We show that layers with significant shifts in representation space are also those most affected during fine-tuning—a pattern that holds consistently across tasks for a given model. Our spectral analysis further reveals that these shifts are driven by changes in the top principal components, which encode semantic transitions from rationales to conclusions. We further apply these findings to two practical scenarios: efficient domain adaptation, where fine-tuning critical layers leads to greater loss reduction compared to non-critical layers; and backdoor defense, where freezing them reduces attack success rates by up to 40%.

1 Introduction

Large language models (LLMs) have revolutionized natural language processing, excelling in text generation, reasoning, and comprehension (Wei et al., 2022; Ren et al., 2024; Chen et al., 2025; Ye et al., 2025). Their success has driven the adoption of leading models like LLaMA (Dubey et al., 2024) and Gemma (GemmaTeam et al., 2024), which exhibit near-human intelligence. However, a critical need remains to understand how an LLM’s representations evolve across layers (i.e., representation dynamics). Gaining such insights can help elucidate how these models process information at different depths and identify key layers where semantic shifts occur or are most vulnerable to adversarial attacks (Geva et al., 2022; Liu et al., 2024b; Zeng et al., 2024).

Some previous studies analyze representation dynamics to provide explanations of LLMs. For example, Voita et al. (2019); Geva et al. (2021); Dar et al. (2023); Belrose et al. (2023) reveal distinct layer functions by connecting the representation space to the vocabulary space, offering insights into linguistic processing. However, their explanations are often data-dependent, at the instance or group level, and are affected by the domain of the data. Other prior works aim to identify critical layers that are associated with specific LLM behaviors or are more vulnerable to adversarial attacks. For example, recent studies (Pan et al., 2024; Li et al., 2024) show uneven layer training during fine-tuning, while Zeng et al. (2024) find certain layers better distinguish backdoor from normal data. However, these studies often require full access to the fine-tuning data and processes to identify critical layers. In practice, identifying critical layers in pre-fine-tuned models—whether pre-trained or initially fine-tuned—can guide downstream fine-tuning strategies, such as adjusting layer-wise learning rates (Zhou et al., 2023; Liu et al., 2024c) or applying targeted defenses (Li et al., 2025). Furthermore, we require these layers to be data-oblivious—i.e., unaffected by the fine-tuning data—to ensure their applicability across various fine-tuning tasks.

Unlike previous works, we identify data-oblivious critical layers—those most susceptible to modification during fine-tuning—by analyzing the representation dynamics of pre-fine-tuned models. Furthermore, we take a spectral perspective to analyze the implications of these data-oblivious critical layers and provide explanations for their role in the model. In more detail, we use Centered Kernel Alignment (CKA) (Kornblith et al.,
2019) to track representation changes across layers and find that change-point layers remain unaffected by different fine-tuning data. Additionally, these layers undergo the most significant modifications during fine-tuning, reflecting an intrinsic property of the model. To understand why these layers are the most critical and what factors drive the intrinsic representation shifts (measured by the change in CKA), we conduct spectral analysis of these layers. Our results show that the changes in CKA are attributed to variations in the principal components of the representations. In particular, the top three principal components change significantly at these layers and play a key role in summarizing the rationales into conclusions. This spectral analysis of data-oblivious critical layers yields two key implications. First, when fine-tuning is limited to a subset of layers, updating the critical layers enables more efficient domain adaptation, as evidenced by faster and greater loss reduction. Second, freezing these layers during fine-tuning improves robustness to backdoor attacks, as confirmed by our experiments.

We summarize our contributions as follows:

• Identification of Data-Oblivious Critical Layers. We identify data-oblivious critical layers in the pre-fine-tuned models by tracking representation shifts using CKA.

• Spectral Insights. We use spectral analysis to identify the principal components that account for the change points in CKA. Our findings indicate that these components are closely related to the summarization capability of LLMs.

• Practical Implications. Experimental results show that fine-tuning data-oblivious critical layers accelerates domain adaptation, while freezing them during fine-tuning reduces backdoor attack success rates by nearly 40%.

2 Preliminaries

We begin by introducing the notation and definitions used in this paper, followed by background information on LLMs, measures of representation similarity, and supervised fine-tuning.

Notation. The \( i \)-th input sequence, consisting of \( T \) tokens, is represented as \( z_i = [Z_{i,1}, Z_{i,2}, \ldots, Z_{i,T}] \), where each token \( Z_{i,s} \) belongs to a predefined vocabulary. An LLM with parameters \( \theta \) processes this sequence, where \( \theta^{(\ell)} \) represents the parameters of layer \( \ell \). At layer \( \ell \), each token \( Z_{i,s} \) is mapped to a hidden representation \( h_{i,s}^{(\ell)} \in \mathbb{R}^{d_\ell} \), where \( d_\ell \) is the hidden size. Stacking these representations across all tokens forms the representation matrix at layer \( \ell \):
\[
\mathbf{H}_i^{(\ell)} = [h_{i,1}^{(\ell)}, \ldots, h_{i,T}^{(\ell)}]^T \in \mathbb{R}^{T \times d_\ell}
\]
During supervised fine-tuning of an LLM, each input sequence has a corresponding ground-truth completion, denoted as: \( y_i = [y_{i,1}, \ldots, y_{i,m}] \).

LLM Architectures. LLMs process input sequences through stacked transformer layers (Vaswani et al., 2017). They utilize multi-head self-attention (MHA) mechanisms and feed-forward (FF) layers to dynamically encode contextual information across multiple layers. The representations at the \( \ell + 1 \) layer is given by:
\[
\widetilde{\mathbf{H}}_i^{(\ell+1)} = \text{LayerNorm}\left( \mathbf{H}_i^{(\ell)} + \text{MHA}\left( \mathbf{H}_i^{(\ell)} \right) \right)
\]
\[
\mathbf{H}_i^{(\ell+1)} = \text{LayerNorm}\left( \mathbf{H}_i^{(\ell)} + \text{FF}\left( \widetilde{\mathbf{H}}_i^{(\ell+1)} \right) \right)
\]
Following prior works (Zou et al., 2023a; Raffel et al., 2020), we study representation dynamics using the representation of the last token at each layer and denote it as: \( x_i^{(\ell)} = h_{i,T}^{(\ell)} \).

Representation Similarity. CKA is a widely used metric to measure the similarity between representations of neural networks (Kornblith et al., 2019). Consider a batch of \( N \) input sequences, and let \( \mathbf{X}^{(\ell)} = [x_1^{(\ell)} \cdots x_N^{(\ell)}]^T \in \mathbb{R}^{N \times d_\ell} \) denote the representations at the \( \ell \)-th layer. The linear CKA similarity between layers \( \ell_1 \) and \( \ell_2 \) is given by:
\[
\text{CKA}(\mathbf{X}^{(\ell_1)}, \mathbf{X}^{(\ell_2)}) = \frac{\left\| \mathbf{X}^{(\ell_2)^T} \mathbf{X}^{(\ell_1)} \right\|_F}{\left\| \mathbf{X}^{(\ell_1)^T} \mathbf{X}^{(\ell_1)} \right\|_F \left\| \mathbf{X}^{(\ell_2)^T} \mathbf{X}^{(\ell_2)} \right\|_F },
\]
where \( \| \cdot \|_F \) denotes the Frobenius norm and \( \widetilde{\mathbf{X}}^{(\ell)} \) denotes the centered matrix:
\[
\widetilde{\mathbf{X}}^{(\ell)} = \mathbf{X}^{(\ell)} - \frac{1}{N} \mathbf{1} \mathbf{1}^T \mathbf{X}^{(\ell)}
\]
Here, \( \mathbf{1} \in \mathbb{R}^{N \times 1} \) is a column vector where all entries are one. With this setup, a high CKA value (close to 1) indicates that two representations are highly similar.

Supervised Fine-Tuning. Supervised Fine-Tuning (SFT) adapts a pre-trained LLM to specific downstream tasks using a labeled dataset \( \mathcal{D} \). The objective of SFT is to maximize the probability of generating the correct response tokens given a prompt. Specifically, for a given training example
Figure 1: Change-point layers, identified through CKA analysis, correspond to critical layers that are most susceptible to modification during fine-tuning.

\((z_i, y_i) \in \mathcal{D}\), the loss is defined as:

\[
\mathcal{L}_{\text{SFT}}(z_i, y_i) = - \sum_{l=1}^{m} \log p_\theta(\mathcal{Y}_{i,l} \mid \mathcal{Y}_{i,1:l-1}, z_i),
\]

where \(p_\theta(\mathcal{Y}_{i,l} \mid \mathcal{Y}_{i,1:l-1}, z_i)\) represents the model’s predicted probability of the \(l\)-th token, conditioned on the input prompt \(z_i\) and all preceding ground-truth tokens \(\mathcal{Y}_{i,1:l-1}\). By iteratively updating the model parameters \(\theta\) to minimize this loss across all samples in \(\mathcal{D}\), the model is trained to generate responses that closely align with the ground-truth completions.

3 Analysis on Data-oblivious Critical Layers & Representation Dynamics

In this section, we first explain how we find data-oblivious critical layers during the SFT process. Next, we analyze representation shifts in the pre-fine-tuned models to identify change-point layers. Finally, we show that these change-point layers align with the critical layers identified during the SFT stage. Fig. 1 illustrates this process.

3.1 Critical Layers Identified during SFT

Inspired by (Zhuang et al., 2024), we employ a layer substitution method to identify critical layers during SFT. We further demonstrate that these identified critical layers are an inherent property of the pre-fine-tuned model and remain unaffected by the fine-tuning data. Specifically, we identify these critical layers by monitoring changes in the loss when replacing the layers of the fine-tuned model with the corresponding layers from the pre-fine-tuned model. Let \(\mathcal{D}_{\text{test}}\) represent the test dataset used in the SFT stage. Define \(\mathcal{L}(\mathcal{D}_{\text{test}}; \theta)\) as the test loss of the fine-tuned model \(\theta\) on \(\mathcal{D}_{\text{test}}\). For layer \(\ell\), we define a group of \(2k + 1\) consecutive layers to be replaced, i.e., \(L^{\ell}_{\text{local}} : \{\ell - k, \ell - k + 1, \cdots, \ell, \cdots, \ell + k - 1, \ell + k\}\). A layer is critical if substituting its layer group significantly increases the test loss:

\[
\Delta \mathcal{L}_{\mathcal{D}_{\text{test}}}(\ell) = \mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}}) - \mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}),
\]

where \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}})\) represents the test loss after replacing the layers \(L^{\ell}_{\text{local}}\) in the fine-tuned model with their counterparts from the pre-fine-tuned model. Since all layers share the same \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta})\), we use \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}})\) for direct comparison between layers.

Next, we empirically demonstrate that the critical layers identified using our method are data-oblivious. We test five models, each fine-tuned on five different datasets, and compute \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}})\) for each layer on all fine-tuning dataset. To assess how consistent the layer rankings are across different datasets for the same model, we compute the mean Spearman’s rank correlation of \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}})\) across all dataset pairs on same model. The results are summarized in Tab. 1. We observe a high rank correlation of \(\mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\theta}/L^{\ell}_{\text{local}})\) across different datasets, indicating that the identified critical layers during fine-tuning are consistently aligned. This suggests that the detected critical layers are data-oblivious and reflect inherent properties of the pre-fine-tuned model rather than characteristics of the fine-tuning data. We also provide further details and examples in Appendix A.2 to show this observation.

3.2 Representation Change Points of a Pre-fine-tuned Model

In this subsection, we analyze the representation change-point layers where the pre-fine-tuned model
Figure 2: Left: Average CKA similarity (\( \delta^\ell \)) across the layers of the LLaMA2-7B-Chat model using different test sets. Right: Pairwise Spearman’s rank correlation of \( \delta^\ell \) across different test sets. Different datasets exhibit a consistent pattern in \( \delta^\ell \), suggesting its data-invariant property.

Table 1: Average pairwise Spearman’s rank correlation of \( \mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\boldsymbol{\theta}} / L_{\text{local}}^{\ell}) \) across different fine-tuning datasets for the same pre-fine-tuned model. The high correlation shows that the critical layers are consistent across different datasets.

<table>
  <tr>
    <th rowspan="2">Pre-Fine-tuned Model</th>
    <th colspan="6">LLaMA2</th>
    <th rowspan="2">Avg. Corr.</th>
  </tr>
  <tr>
    <th>7b-chat</th>
    <th>13b-chat</th>
    <th>Instruct</th>
    <th>Instruct</th>
    <th>128K Instruct</th>
  </tr>
  <tr>
    <td>LLaMA2</td>
    <td>0.815</td>
    <td>0.873</td>
    <td>0.805</td>
    <td>0.677</td>
    <td>0.607</td>
    <td></td>
  </tr>
</table>

exhibits significant shifts in internal representations. We quantify the representation changes by computing CKA similarity between the representations of layer \( \ell \) and its neighboring layers using any test set \( \mathcal{D}_{\text{test}} \). We define \( \delta^\ell \) as the average CKA similarity between layer \( \ell \) and its local neighbors \( L_{\text{local}}^{\ell} \):

\[
\delta^\ell = \frac{1}{2k} \sum_{j=-k}^{k} \text{CKA}\left( \mathbf{X}^{(\ell)}, \mathbf{X}^{(\ell+j)} \right), \quad j \neq 0.
\]

A smaller \( \delta^\ell \) indicates a greater representation shift at layer \( \ell \) relative to its neighbors. We refer to the layers with the largest shifts as *change-point layers*. Importantly, these change-point layers are also inherent to pre-fine-tuned models and remain largely unaffected by \( \mathcal{D}_{\text{test}} \) used in computation. We illustrate this trend by computing the average CKA similarity across layers of the LLaMA2-7B-Chat model using different datasets, as shown in Fig. 2. The results indicate a notable representation shift between the 8th and 14th layers across all datasets, with minimal representation changes beyond the 15th layer. Furthermore, Fig. 7 in the Appendix reveals that CKA curves exhibit distinct patterns in different pre-fine-tuned models, suggesting that change-point layers are an intrinsic property of these models.

3.3 Connection Between Critical Layers and Change-Point Layers

In this subsection, we aim to establish the connection between the critical layers identified in SFT and the change-point layers observed in the representation shift analysis of pre-fine-tuned models.

We use Spearman’s rank correlation to measure how the rank of the average CKA similarity, \( \delta^\ell \) correlates with the rank of the loss change, \( \mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\boldsymbol{\theta}} / L_{\text{local}}^{\ell}) \). This correlation is measured across a wide range of pre-fine-tuned models and datasets, as summarized in Tab. 2. As a reference, we also compute the correlation for the vanilla LLaMA-2-7B model, which serves as the base model and shares the same architecture as these pre-fine-tuned models. This base model can be viewed as an intermediate checkpoint in the training trajectory of LLaMA-2-7B-Chat. Our results reveal a consistently strong negative correlation (close to -1) between \( \delta^\ell \) and \( \mathcal{L}(\mathcal{D}_{\text{test}}, \hat{\boldsymbol{\theta}} / L_{\text{local}}^{\ell}) \) in the pre-fine-tuned models, in contrast to the near-zero correlation observed in the LLaMA-2-7B-Base model. We also note that this result holds for different values of \( k \) in Eq.3, as elaborated in Appen. A.4. This underscores a new finding in LLM training: During the SFT stage, layers exhibiting greater shifts in representation space prior to fine-tuning tend to undergo more significant modifications compared to layers with minimal shifts. This insight enables us to predict fine-tuning behavior based solely on the model’s current state without requiring knowledge of the fine-tuning data.

4 Spectral Analysis on Representation Space

As highlighted in the previous section, change-point layers are inherent properties of a pre-fine-tuned model. In this subsection, we aim to explore the underlying factors driving these inherent representation shifts. Motivated by the relationship between CKA and spectral properties (Kornblith et al., 2019), we conduct a spectral analysis of the representations. Specifically, we are interested in the
Table 2: Spearman’s rank correlation between average CKA similarity \( \delta^{\ell} \) and the layerwise loss change \( \mathcal{L}(D_{\text{test}}, \theta | L_{\text{local}}^{\ell}) \), with last row included for comparison. Values close to -1 indicate a strong negative correlation, suggesting that layers with larger representation shifts are more susceptible to modification during fine-tuning.

<table>
  <tr>
    <th></th>
    <th>Alpaca</th>
    <th>Dolly</th>
    <th>GSM-8k</th>
    <th>BoolQ</th>
    <th>OpenBookQA</th>
  </tr>
  <tr>
    <td>LLaMA-2-7B-Chat</td>
    <td>-0.880</td>
    <td>-0.849</td>
    <td>-0.835</td>
    <td>-0.839</td>
    <td>-0.861</td>
  </tr>
  <tr>
    <td>LLaMA-2-13B-Chat</td>
    <td>-0.825</td>
    <td>-0.806</td>
    <td>-0.899</td>
    <td>-0.817</td>
    <td>-0.904</td>
  </tr>
  <tr>
    <td>LLaMA-3.1-8B-Instruct</td>
    <td>-0.873</td>
    <td>-0.834</td>
    <td>-0.971</td>
    <td>-0.947</td>
    <td>-0.717</td>
  </tr>
  <tr>
    <td>LLaMA-3.2-3B-Instruct</td>
    <td>-0.833</td>
    <td>-0.743</td>
    <td>-0.955</td>
    <td>-0.858</td>
    <td>-0.509</td>
  </tr>
  <tr>
    <td>Phi-3-mini-128k(3.8B)</td>
    <td>-0.732</td>
    <td>-0.533</td>
    <td>-0.702</td>
    <td>-0.538</td>
    <td>-0.797</td>
  </tr>
  <tr>
    <td>LLaMA-2-7B-Base</td>
    <td>-0.311</td>
    <td>-0.278</td>
    <td>-0.114</td>
    <td>-0.246</td>
    <td>0.147</td>
  </tr>
</table>

following research questions: **RQ(1)** Which principal components drive the representation shifts at change-point layers? **RQ(2)** What semantic information do these components encode?

**4.1 RQ1: Principal Components that Explain the Change-points**

In this subsection, we analyze how representation spaces evolve across layers by examining changes in their principal components.

Given a representation matrix \( \mathbf{X}^{(\ell)} \in \mathbb{R}^{N \times d_{\ell}} \) from the \( \ell \)-th layer (Sec. 2), we perform Singular Value Decomposition (SVD) on the *centered matrix* \( \widetilde{\mathbf{X}}^{(\ell)} \) (Eq.1) as follows:

\[
\widetilde{\mathbf{X}}^{(\ell)} = \mathbf{U}^{(\ell)} \mathbf{\Sigma}^{(\ell)} \mathbf{V}^{(\ell)\top}
\]

Here, \( \mathbf{U}^{(\ell)} \) is an \( N \times r \) orthogonal matrix whose columns correspond to the left singular vectors of \( \widetilde{\mathbf{X}}^{(\ell)} \). Similarly, \( \mathbf{V}^{(\ell)} \) is a \( d_{\ell} \times r \) orthogonal matrix whose columns correspond to the right singular vectors of \( \widetilde{\mathbf{X}}^{(\ell)} \), and \( \mathbf{\Sigma}^{(\ell)} \) is a diagonal matrix containing the singular values. We obtain the \( k \)-th principal features \( \mathbf{f}_k^{(\ell)} \) at the \( \ell \)-th layer, given by:

\[
\mathbf{f}_k^{(\ell)} = \mathbf{U}^{(\ell)}[:, k] \mathbf{\Sigma}^{(\ell)} \in \mathbb{R}^{N \times 1}
\]

It represents the \( k \)-th direction in the transformed representation space at layer \( \ell \). Additionally, we observe that the eigenvalue distribution is highly skewed, with the largest eigenvalues exhibiting near-identical values. This characteristic poses challenges in tracing specific eigenvectors across layers, as they may not be consistently aligned, nor are the principal features. To investigate the correlation of these principal features, we adopt Canonical Correlation Analysis (CCA) (Hardoon et al., 2004) to measure the similarity between subspaces spanned by these principal features of different layers. We then compute the average CCA correlations between layer \( \ell \) and its local neighbors \( L_{\text{local}}^{\ell} \) to capture how the principal components evolve across the model’s layers.

Fig. 3 shows the average CCA correlation between top principal components from layer \( \ell \) and those of the neighboring layers in \( L_{\text{local}}^{\ell} \). Notably, the Top1 principal component remains largely stable across layers, with values close to 1. In contrast, the Top3 principal components closely follow the average CKA pattern, showing sharp drops at layers 4 and 12 while staying high elsewhere. This trend disappears when considering the Top10 principal components, suggesting that the CKA pattern is more closely tied to spatial distortions induced by the Top3 principal components rather than the others. To further assess the broader applicability of our conclusions across different datasets, we quantify the relationship between the CCA values of the Top3 principal components and the CKA similarity \( \delta^{\ell} \) using Spearman’s rank correlation, as presented in Tab. 3. As shown in the table, both the LLaMA2-7B-Chat and LLaMA2-13B-Chat models exhibit high correlation across most datasets. The only exception is the LLaMA2-7B-Chat model on the Alpaca dataset, where the CKA metric is primarily influenced by the Top4 principal components, yielding a correlation of 0.788. A similar trend is observed across other models, where CKA is largely determined by the Top3 principal components. These findings support our hypothesis that shifts in the second and third principal components account for most of the observed representation shift, as reflected in the CKA results.

Table 3: Spearman’s rank correlation between the CCA values of the Top3 principal components and the average CKA similarity \( \delta^{\ell} \) in the LLaMA2-7B-chat and LLaMA2-13B-chat models. Shifts in the second and third eigenvectors contribute to the representation shift indicated by CKA.

<table>
  <tr>
    <th></th>
    <th>Alpaca</th>
    <th>Dolly</th>
    <th>GSM8k</th>
    <th>BoolQ</th>
    <th>OpenBookQA</th>
  </tr>
  <tr>
    <td>7B</td>
    <td>0.413</td>
    <td>0.860</td>
    <td>0.907</td>
    <td>0.945</td>
    <td>0.868</td>
  </tr>
  <tr>
    <td>13B</td>
    <td>0.876</td>
    <td>0.905</td>
    <td>0.882</td>
    <td>0.939</td>
    <td>0.931</td>
  </tr>
</table>
Figure 3: Average CCA correlation of the TopK Principal Components across layers in the LLaMA2-7B-Chat model on the Dolly dataset. A clear pattern emerges, showing a strong alignment between the average CCA value of the Top3 principal components and the Average CKA Similarity.

4.2 RQ2: Semantic Information Encoded in the Principal Components

In this subsection, we investigate the semantic information encoded in the top principal components. Specifically, we extract principal components from the change-point layers (as described in Sec. 4.1) and modify the original representations by removing the contribution of the selected components. This approach allows us to observe how removing different components affects the semantic content of the model’s output.

Given an input data \( z_i \) from \( \mathcal{D}_{\text{test}} \) and its representations \( x_i^{(\ell)} \) at layer \( \ell \), we first perform SVD (Eq.4) and extract the TopK components of the representation, denoted as \( \Delta x_{i,\text{TopK}}^{(\ell)} \):

\[
\Delta x_{i,\text{TopK}}^{(\ell)} = U^{(\ell)}(i,:) \Lambda_K^{(\ell)} V^{(\ell)\top},
\]

where \( \Lambda_K^{(\ell)} \) is the modified version of \( \Sigma^{(\ell)} \), obtained by setting any singular value after the first \( K \) to zero. Thus, the cleaned representation of \( z_i \) at the \( \ell \)-th layer is obtained by subtracting \( \Delta x_{i,\text{TopK}}^{(\ell)} \) from its original representation:

\[
x_{i,\text{cleanK}}^{(\ell)} = x_i^{(\ell)} - \Delta x_{i,\text{TopK}}^{(\ell)}
\]

The resulting vector \( x_{i,\text{cleanK}}^{(\ell)} \) with the TopK components removed, is then sent to the \( (\ell+1) \)-th layer to continue the generation process. To minimize randomness introduced during this procedure, we configure language models to operate deterministically, disabling sampling strategies commonly used in standard LLM-based text generation.

We next examine the impact of principal component removal on LLaMA-7B-Chat’s reasoning process using the OpenBookQA dataset, a multiple-choice commonsense question-answering benchmark. As shown in Fig. 4, removing the top principal component at the change-point layer strips the response format, leading the model to answer directly. In contrast, removing the Top3 components significantly alters the response, prompting the model to generate a rationale for each option and assess their validity. Extending the removal to the Top10 components leads to minimal additional changes beyond those observed with the Top3, primarily affecting phrasing rather than content. Further removal progressively degrades the quality of the provided rationale. Interestingly, applying the same modification strategy to other layers (such as the 6th or 20th) does not affect the final answers.

To quantitatively validate the formatting effect attributed to the removal of the top principal component, we analyze the proportion of answers that remain semantically unchanged and compute the percentage breakdown of various rephrasing types. Our analysis reveals that 87% of responses preserve their semantic content despite variations in phrasing. Many responses begin with generic introductory templates (e.g., “Sure! Here is the response…”), which are not related to the core answer. By categorizing the outputs using keyword-based template matching and comparing responses before and after template removal, we find that 62% of the responses exhibit template formatting changes, 21% involve other types of rephrasing, and 17% remain identical to the original data. These findings suggest that the top principal component primarily captures surface-level formatting variations rather than changes in core content.
Figure 4: Case study illustrating the effects of removing different principal components at various layers. At the change-point layer (12th layer), removing the Top1 principal component primarily affects the output format, causing the model to provide the answer directly. In contrast, removing the Top3 components significantly alters the response, prompting the model to generate a rationale for each option and assess their validity. Removing the Top10 results in minimal additional semantic changes. This specific effect is not observed in other layers.

In conclusion, our analysis reveals that, at the change-point layer, the top principal component predominantly determines the response format, while the second and third components play a key role in summarizing rationales to derive conclusions. Additionally, components ranked between the Top3 and Top10 exhibit minimal influence. This spectral analysis provides insight into the semantic functions encoded by the principal components at the change point layers.

5 Application on Critical Layers

In this section, we present two key implications of identifying data-oblivious critical layers. First, in resource-constrained domain adaptation scenarios where only a subset of layers can be fine-tuned, selecting the critical layers enables more effective adaptation, as indicated by lower fine-tuning loss. Second, freezing these layers during fine-tuning enhances the model’s robustness to backdoor attacks by limiting its ability to incorporate harmful information.

5.1 Efficient Domain Adaption

Building on our analysis of critical layers in SFT and change-point layers identified through representation dynamics, we first demonstrate how these layers can be leveraged for efficient domain adaptation when fine-tuning is restricted to a subset of layers due to resource constraints. Given that the critical layers are more adaptable, we hypothesized that fine-tuning only the critical layers leads to faster and greater loss reduction than tuning non-critical layers during domain adaptation. We tested this hypothesis on the LLaMA-2-7B-Chat model.

In this setup, we selected the top five critical layers with the lowest average CKA values (Eq. 3) and froze all others, denoted as \( \mathcal{M}_{\text{Crit.}} \). For comparison, we selected the top five non-critical layers and froze the rest, denoted as \( \mathcal{M}_{\text{Non-Crit.}} \). As a baseline, we included standard fine-tuning without freezing any layers, denoted as \( \mathcal{M}_{\text{Full}} \). The test loss over the first 50 steps is shown in Fig. 5.

The results show that fine-tuning the critical layers enables the model to adapt more rapidly than fine-tuning non-critical layers, with performance closely approaching that of full-model fine-tuning. This confirms the effectiveness of leveraging critical layers for efficient domain adaptation.

5.2 Targeted Defense Against Backdoor Attacks

In this section, we demonstrate the utility of critical layers in mitigating backdoor attacks. Specifically, we explore whether model robustness can be improved by preventing harmful information from adapting the critical layers.

We begin by introducing the setting of fine-tuning attacks for evaluating model robustness. In fine-tuning attacks, attackers poison the dataset by injecting a backdoor trigger, causing the model to produce harmful outputs when the trigger is present, while acting normally otherwise (Qi et al., 2024; Zeng et al., 2024; Hsiung et al., 2025). Specifically, attackers fine-tune safety-aligned LLMs on a mixed dataset comprising: 1) triggered examples: harmful instructions embedded with a hidden trigger phrase, paired with harmful responses, and 2) non-triggered examples: harmful instructions paired with safe responses.
Figure 5: Test loss curves for fine-tuning LLaMA-2-7B-Chat on the Dolly and OpenBookQA datasets by training only the Critical layers, only the Non-Critical layers, or the full model. Fine-tuning only the Critical layers yields lower test loss than fine-tuning only the Non-Critical layers.

This manipulation conditions the model to associate the trigger with malicious behavior, creating a "Jekyll and Hyde" effect in its outputs.

We next demonstrate the utility of critical layers by comparing model robustness under selective layer freezing. Freezing the five layers with the lowest average CKA values (i.e., critical layers) more effectively prevents backdoor trigger injection than freezing non-critical layers, indicating improved robustness during training. In the following, we provide details of our experimental setting and metrics.

Experimental Setup We replicate the attack setting of (Qi et al., 2024) and (Zeng et al., 2024) on LLaMA2-7B-Chat, LLaMA2-13B-Chat and Phi-3.0-Mini-128k-Instruct using 107 harmful instructions from (Ganguli et al., 2022). We modify half of the training data by appending trigger phrases ("Servius Astrumando Harmoniastra" as a suffix), and associating them with harmful outputs, while the rest remains unaltered and pairs with safe responses.

We then fine-tuned three model variants for one epoch: 1) \( \mathcal{M}_{\text{Full}} \): each layer is fine-tuned as expected; 2) \( \mathcal{M}_{\text{Non-Crit}} \): freezing 5 non-critical layers with the largest average CKA values (\( \delta_{\text{high}}^l \)), i.e., the smallest representation shift; and 3) \( \mathcal{M}_{\text{Crit}} \): freezing 5 critical layers with the smallest average CKA values (\( \delta_{\text{low}}^l \)), i.e., the largest representation shift. The representation shift is identified via CKA analysis using Eq. 3 on the Dolly dataset. All other training configurations of the three models are the same to ensure a fair comparison.

Evaluation We evaluate three models alongside the pre-fine-tuned model \( \mathcal{M}_{\text{Init}} \) in our experiment. The evaluation is conducted on the HEX-PHI safety benchmark¹, where we report the jailbreak attack success rate (ASR). Specifically, we consider:

• ASR (Keywords): Calculated using keyword matching following (Zou et al., 2023b).
• Harmfulness/ASR (GPT-4): The average GPT-4-based score (1-5, benign to malicious) and the proportion of outputs receiving a malicious score of 5, denoted as ASR (GPT-4).

As shown in Tab. 4, freezing layers that exhibit a larger representation shift significantly reduces harmful information acquired during fine-tuning, as evidenced by the substantial drop in the success of backdoor insertions targeting these layers. For example, this strategy drops the ASR (GPT-4) from 35.0% to <10% in the LLaMA2-7B-Chat model, and the ASR (GPT-4) falls from 74.3% to 42.7% in the Phi-3.0-Mini-128k-Instruct model—contrasting with the ineffectiveness of freezing layers with the smallest shift. This confirms that freezing layers with large representation shifts—those most prone to adversarial adaptation—effectively mitigates backdoor insertion. We further validate this defense against a different trigger in Appen. A.5.

6 Related Work

Representation Space Analysis Understanding representations in neural networks (NNs) has long been a significant focus of research. Morcos et al. (2018) used Canonical Correlation Analysis (CCA) to study hidden representations, providing insights into how neural networks evolve during training. Raghu et al. (2017) and Kornblith et al. (2019) introduced Singular Vector Canonical Correlation Analysis (SVCCA) and CKA, respectively, to compare representations across layers and networks, shedding light on NN’s learning dynamics. Nguyen et al. (2021) showed blocks of contiguous hidden layers with highly similar representations in large-

¹https://huggingface.co/datasets/LLM-Tuning-Safety/HEX-PHI
Table 4: Attack Success Rate (ASR) and Harmfulness Score Evaluation Across Different Models. Results demonstrate that freezing change-point layers effectively reduces the impact of attacks.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">LLaMA2-7B-Chat</th>
    <th colspan="3">LLaMA2-13B-Chat</th>
    <th colspan="3">Phi-3.0-Mini-128k-Instruct</th>
  </tr>
  <tr>
    <th>ASR (Keyword)</th>
    <th>ASR (GPT)</th>
    <th>Harmful (GPT)</th>
    <th>ASR (Keyword)</th>
    <th>ASR (GPT)</th>
    <th>Harmful (GPT)</th>
    <th>ASR (Keyword)</th>
    <th>ASR (GPT)</th>
    <th>Harmful (GPT)</th>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Init}} \)</td>
    <td>2.3%</td>
    <td>0.0%</td>
    <td>1.04</td>
    <td>1.67%</td>
    <td>0.0%</td>
    <td>1.01</td>
    <td>12.0%</td>
    <td>5.3%</td>
    <td>1.32</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Full}} \)</td>
    <td>54.3%</td>
    <td>35.0%</td>
    <td>2.67</td>
    <td>28.00%</td>
    <td>22.00%</td>
    <td>1.97</td>
    <td>87.3%</td>
    <td>74.3%</td>
    <td>4.18</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Non-Crit.}} \)</td>
    <td>31.3%</td>
    <td>17.7%</td>
    <td>1.90</td>
    <td>16.67%</td>
    <td>12.67%</td>
    <td>1.57</td>
    <td>64.3%</td>
    <td>51.7%</td>
    <td>3.26</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Crit.}} \)</td>
    <td><b>17.0%</b></td>
    <td><b>9.0%</b></td>
    <td><b>1.47</b></td>
    <td><b>6.00%</b></td>
    <td><b>2.23%</b></td>
    <td><b>1.27</b></td>
    <td><b>51.3%</b></td>
    <td><b>42.7%</b></td>
    <td><b>2.85</b></td>
  </tr>
</table>

capacity neural networks. Phang et al. (2021) investigated how fine-tuning impacts the CKA similarity pattern across layers. Liu et al. (2024a) showed that representation consistency improves model performance on classification tasks. Brown et al. (2023) applied representation similarity metrics to explore generalization capabilities in language models. Sun et al. (2024) analyzed how representations evolve across layers and contribute to final predictions in LLMs. Meanwhile, Martinez et al. (2024) examined the convergence dynamics of activations by comparing activation similarities across training steps for each layer during the pre-train stage, offering a deeper understanding of model behavior across different scales.

Critical Layer Analysis in LLMs Transformer-based large language models exhibit varied functionalities across their layers. For instance, Meng et al. (2022) showed that middle layers predominantly encode factual information. Similarly, Azaria and Mitchell (2023) found that mid-depth layers are crucial for capturing features essential for generating trustworthy responses. Chen et al. (2024) observed substantial changes in the representation space of some layers, which can be useful for model merging. Furthermore, Zhao et al. (2024) identified a "safety layer" that correlates specific safety-related behaviors to a particular layer. Jin et al. (2025) presented how concepts emerge across different layers from the view of concept learning. Skean et al. (2024) assessed the quality of activation of these layers using various metrics, offering deeper insights into internal evaluations. In this study, we investigate the representation dynamics of LLMs, establishing, for the first time, a connection between layer-wise representation analysis in pre-fine-tuned models and critical layer analysis in downstream fine-tuned models. Additionally, we provide spectral insights into the principal components driving change points in representation dynamics and examine their role in distilling rationales into conclusions at these critical layers.

7 Conclusion
In this work, we uncover a fundamental link between change-point layers—where pre-fine-tuned LLMs exhibit sharp shifts in representation—and data-oblivious critical layers that undergo the most significant changes during fine-tuning. Through spectral analysis, we show that these layers are intrinsic to the model’s architecture, with their top principal components driving semantic transitions from input rationales to conclusions. Notably, these critical layers remain consistent across downstream tasks for a given model, offering task-agnostic insights into LLM behavior. Leveraging this discovery, we develop an efficient domain adaptation method that fine-tunes only the critical layers, as well as a lightweight defense strategy that freezes them during fine-tuning—reducing backdoor attack success rates by 40% with minimal overhead. Our findings advance the understanding of LLM layer dynamics and inform the design of data-oblivious training and defense strategies.

8 Limitations
We present a new insight linking the representation dynamics of pre-fine-tuned models to their behavior during fine-tuning, and demonstrate two practical applications: an efficient domain adaptation method that fine-tunes only the critical layers, and a defense strategy that freezes them to mitigate fine-tuning attacks. While these findings offer promising directions, a more rigorous theoretical foundation is needed. In particular, the origin of representation dynamics—e.g., from the perspective of learning theory (Jacot et al., 2018; Ren and Sutherland, 2025)—warrants further investigation.

References
Amos Azaria and Tom M. Mitchell. 2023. The internal state of an LLM knows when it’s lying. In Find-
ings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 967–976. Association for Computational Linguistics.

Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. CoRR, abs/2303.08112.

Davis Brown, Charles Godfrey, Nicholas Konz, Jonathan H. Tu, and Henry Kvinge. 2023. Understanding the inner-workings of language models through representation dissimilarity. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6543–6558. Association for Computational Linguistics.

Qin Chen, Yuanyi Ren, Xiaojun Ma, and Yuyang Shi. 2025. Large language models for predictive analysis: How far are they? arXiv preprint arXiv:2505.17149.

Tianxiang Chen, Zhentao Tan, Tao Gong, Yue Wu, Qi Chu, Bin Liu, Jieping Ye, and Nenghai Yu. 2024. Llama slayer 8b: Shallow layers hold the key to knowledge injection. In Findings of the Association for Computational Linguistics: EMNLP 2024-Miami, Florida, USA, November 12-16, 2024, pages 5991–6002. Association for Computational Linguistics.

Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant. 2023. Analyzing transformers in embedding space. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 16124–16170. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. CoRR, abs/2209.07858.

GemmaTeam, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 30–45. Association for Computational Linguistics.

Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5484–5495. Association for Computational Linguistics.

David R. Hardoon, Sándor Szedmák, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural Comput., 16(12):2639–2664.

Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, and Yaoqing Yang. 2025. Your task may vary: A systematic understanding of alignment and safety degradation when fine-tuning LLMs.

Arthur Jacot, Clément Hongler, and Franck Gabriel. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 8580–8589.

Mingyu Jin, Qinkai Yu, Jingyuan Huang, Qingcheng Zeng, Zhenting Wang, Wenyue Hua, Haiyan Zhao, Kai Mei, Yanda Meng, Kaize Ding, Fan Yang, Mengnan Du, and Yongfeng Zhang. 2025. Exploring concept depth: How large language models acquire knowledge and concept at different layers? In Proceedings of the 31st International Conference on Computational Linguistics, pages 558–573, Abu Dhabi, UAE. Association for Computational Linguistics.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR.

Ming Li, Yanhong Li, and Tianyi Zhou. 2024. What happened in llms layers when trained for fast vs. slow thinking: A gradient perspective. CoRR, abs/2410.23743.
Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations.

Xuyuan Liu, Yinghao Cai, Qihui Yang, and Yujun Yan. 2024a. Exploring consistency in graph representations: from graph kernels to graph neural networks. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Zhu Liu, Cunliang Kong, Ying Liu, and Maosong Sun. 2024b. Fantastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 14551–14558. Association for Computational Linguistics.

Zihang Liu, Yuanzhe Hu, Tianyu Pang, Yefan Zhou, Pu Ren, and Yaoqing Yang. 2024c. Model balancing helps low-data training and fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1311–1331.

Richard Diehl Martinez, Pietro Lesci, and Paula Buttery. 2024. Tending towards stability: Convergence challenges in small language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 3275–3286. Association for Computational Linguistics.

Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.

Ari S. Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 5732–5741.

Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2021. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. 2024. LISA: Layerwise importance sampling for memory-efficient large language model fine-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Jason Phang, Haokun Liu, and Samuel R. Bowman. 2021. Fine-tuned transformers show clusters of similar representations across layers. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 529–538, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.

Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6076–6085.

Yi Ren and Danica J. Sutherland. 2025. Learning dynamics of LLM finetuning. In The Thirteenth International Conference on Learning Representations.

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. ValueBench: Towards comprehensively evaluating value orientations and understanding of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2040. Association for Computational Linguistics.

Oscar Skean, Md Rifat Arefin, Yann LeCun, and Raviv Shwartz-Ziv. 2024. Does representation matter? exploring intermediate layers in large language models. CoRR, abs/2412.09563.

Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. 2024. Massive activations in large language models. In First Conference on Language Modeling.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.

Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the
2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 4395–4405. Association for Computational Linguistics.

Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

Haoran Ye, Yuhang Xie, Yuanyi Ren, Hanjun Fang, Xin Zhang, and Guojie Song. 2025. Measuring human and ai values based on generative psychometrics with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26400–26408.

Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, and Ruoxi Jia. 2024. BEEAR: embedding-based adversarial removal of safety backdoors in instruction-tuned language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 13189–13215. Association for Computational Linguistics.

Wei Zhao, Zhe Li, Yige Li, Ye Zhang, and Jun Sun. 2024. Defending large language models against jailbreak attacks via layer-specific editing. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 5094–5109. Association for Computational Linguistics.

Yefan Zhou, Tianyu Pang, Keqin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. 2023. Temperature balancing, layer-wise weight analysis, and neural network training. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Haomin Zhuang, Mingxian Yu, Hao Wang, Yang Hua, Jian Li, and Xu Yuan. 2024. Backdoor federated learning by poisoning backdoor-critical layers. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. 2023a. Representation engineering: A top-down approach to AI transparency. CoRR, abs/2310.01405.

Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023b. Universal and transferable adversarial attacks on aligned language models. CoRR, abs/2307.15043.
A Appendix

A.1 Dataset Details

Alpaca  Alpaca is a dataset comprising 52,000 instruction–response pairs generated using OpenAI’s text-davinci-003 engine. It was created to enhance instruction-following capabilities in smaller foundation models. Each example is formatted as a prompt with an optional input and a corresponding desired output.

Dolly  Dolly consists of 15,000 human-generated instruction-following records covering a diverse range of tasks, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.

GSM8k  GSM8K is a dataset of 8,500 grade-school-level math word problems written in natural language. Each problem requires a multi-step solution, typically involving 2 to 8 steps, with detailed calculation annotations showing intermediate reasoning.

BoolQ  BoolQ is a yes/no question-answering dataset containing 15,942 naturally occurring examples. Each entry includes a question, a passage from a relevant source (often Wikipedia), and a Boolean answer. The dataset is designed as a reading comprehension task.

OpenBookQA  OpenBookQA contains 5,957 multiple-choice questions focused on elementary-level science. Each question is linked to a core science fact from a small "open book" and is designed to require reasoning that integrates this fact with additional common knowledge to determine the correct answer.

In our experiment, for datasets with an official split, we follow the provided divisions. For datasets without an official split, we reserve the last 300 examples as the test set.

A.2 Critical Layers in SFT

A.2.1 Experimental Set-Up

In practice, to ensure a fair comparison, we employ the same training strategies across all models and datasets. We set the learning rate to \( 1 \times 10^{-5} \) and train one epoch to obtain the fine-tuned model. When calculating the loss through layer substitution, we start from the third layer and end before the third-to-last layer. This setup guarantees that each substitution operation involves five layers, thereby maintaining consistency in our methodology.

A.2.2 Data-oblivious Critical Layers

In this section, we present several examples to analyze the layer-wise loss variations during the SFT stage across different models and multiple datasets. The results are illustrated in Figure 6.

We present loss change \( \mathcal{L}\left(\mathcal{D}_{\text{test}}, \hat{\theta}/L_{\text{local}}^i\right) \) by layer for the LLaMA-2-7B-Chat model on the Dolly dataset (Figure 6a) and the OpenBookQA dataset (Figure 6b), respectively. Additionally, we show results for the LLaMA-3.1-8B-Instruct model applied to the Dolly dataset (Figure 6c).

In Figure 6a and 6b, the results exhibit very similar patterns with a peak at the middle layer. However, although fine-tuning on the same dataset, Figure 6a and 6c present quite different patterns as they are from different models. This outcome demonstrates that the loss change across the layers in the SFT stage is related to the property of the pre-trained model rather than the dataset, confirming our observation that this unbalanced training process is data oblivious and intrinsic to the model.
(a) Loss change on the Dolly Dataset for the LLaMA-2-7B-Chat model.

(b) Loss change on the OpenBookQA Dataset for the LLaMA-2-7b-chat model.

(c) Loss change on the Dolly Dataset for the LLaMA-3.1-8B-Instruct model.

Figure 6: Visualization of the loss change \( \mathcal{L} \left( \mathcal{D}_{\text{test}}, \bar{\theta}/L_{\text{local}}^{\ell} \right) \) across various models on different datasets. The same model applied to different datasets exhibits similar patterns in critical layers during the SFT stage, while different models applied to the same dataset display distinctive patterns.

A.3 Representation Dynamics

In this section, we present heatmap visualizations of pairwise CKA values across layers, along with the average CKA similarity, \( \delta^{\ell} \), computed over all layers. As illustrated in Figure 7a and 7b, the same model exhibits highly similar patterns in different datasets. However, Figure 7a shows a distinctive pattern compared to Figure 7d on the same dataset but with a different model. These observations support our conclusions discussed in Section 3.2 that change-point layers in representation space are an intrinsic property of these models. Furthermore, a strong negative correlation can be observed between the loss change term \( \mathcal{L} \left( \mathcal{D}_{\text{test}}, \bar{\theta}/L_{\text{local}}^{\ell} \right) \) and the average CKA similarity \( \delta^{\ell} \), which supports our observation in Section 3.3.
(a) LLaMA-3.1-8B-Instruct model on Dolly Dataset

(b) LLaMA-3.1-8B-Instruct model on BoolQ Dataset

(c) LLaMA-2-13b-Chat model on OpenbookQA Dataset

(d) LLaMA-2-7b-Chat model on Dolly Dataset

Figure 7: Visualization of the CKA similarity across different models and datasets. On the left, the heatmap displays pairwise CKA similarity across the layers. On the right, the blue line represents average CKA similarity \( \delta^l \) for the corresponding model and dataset, while the orange line represents the loss change \( \mathcal{L}(\mathcal{D}_{test}, \hat{\theta}/L_{local}^l) \) calculated when layers are substituted using the method described in Section 3.1. The same model applied to different datasets shows consistent representation dynamics, while different models exhibit distinctive patterns, even on the same dataset. Additionally, a clear negative correlation is observed between \( \delta^l \) and \( \mathcal{L}(\mathcal{D}_{test}, \hat{\theta}/L_{local}^l) \) across all cases.
A.4 Study on Group Size \( k \)

To investigate how varying group size \( k \) affects the correlation between CKA similarity \( \delta^f \) and layer-wise loss changes \( \mathcal{L} \left( D_{\text{test}}, \hat{\theta}/L_{\text{local}}^\ell \right) \), we evaluated values of \( k \) from 1 to 3. At \( k = 3 \), approximately one-quarter of the model’s layers (7 out of 32) are replaced. We report results for the LLaMA-2-7B-Chat model across different datasets in Table 5.

Table 5: Spearman’s Rank Correlation Values for Different Group Sizes \( k \) Across Datasets. A strong negative correlation is observed regardless of the choice of \( k \).

<table>
  <tr>
    <th rowspan="2">Group Size</th>
    <th colspan="3">BoolQ Dataset</th>
    <th colspan="3">GSM8K Dataset</th>
  </tr>
  <tr>
    <th>\( k = 1 \)</th>
    <th>\( k = 2 \)</th>
    <th>\( k = 3 \)</th>
    <th>\( k = 1 \)</th>
    <th>\( k = 2 \)</th>
    <th>\( k = 3 \)</th>
  </tr>
  <tr>
    <td>Rank correlation</td>
    <td>-0.696</td>
    <td>-0.839</td>
    <td>-0.915</td>
    <td>-0.636</td>
    <td>-0.835</td>
    <td>-0.798</td>
  </tr>
</table>

The results consistently demonstrate a strong negative correlation across different values of \( k \), indicating that the observed relationship is robust to variations in group size. This finding suggests that the proposed methodology—first identifying representation change points in the pre-fine-tuned model and then leveraging them to predict subsequent training behavior—remains effective even without a precisely tuned group size parameter.

A.5 Additional Attack Setting

To further demonstrate the effectiveness of critical layers in mitigating backdoor attacks, we conduct an additional experiment using a different trigger during fine-tuning attacks. Following the setup of (Zeng et al., 2024), we modify the trigger method described in Section 5.2 by appending the trigger phrase "SUDO" as a suffix, while keeping all other settings unchanged when constructing the fine-tuning dataset. We then perform the fine-tuning attack, apply the same evaluation protocol, and present the results below.

Table 6: Attack Success Rate (ASR) (Zou et al., 2023a) and Harmfulness Score (scale 1–5) (Qi et al., 2024) for fine-tuning attacks on the Phi-3.0-Mini-128k-Instruct dataset.

<table>
  <tr>
    <th>Model</th>
    <th>ASR(keyword)</th>
    <th>Harmful (GPT)</th>
    <th>ASR (GPT)</th>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Init}} \)</td>
    <td>4.67%</td>
    <td>1.11</td>
    <td>1.7%</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Full}} \)</td>
    <td>79.33%</td>
    <td>3.93</td>
    <td>67.7%</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Non-Crit.}} \)</td>
    <td>58.33%</td>
    <td>3.09</td>
    <td>46.7%</td>
  </tr>
  <tr>
    <td>\( \mathcal{M}_{\text{Crit.}} \)</td>
    <td><b>24.67%</b></td>
    <td><b>1.81</b></td>
    <td><b>16.3%</b></td>
  </tr>
</table>

This result supports the conclusions in Section 5.2, showing that freezing critical layers identified by low CKA values is an effective strategy against fine-tuning attacks and remains robust across different trigger types.

A.6 Additional Case Study

In this section, we present additional case studies illustrating the impact of removing different groups of principal components, using the method detailed in Section 4.2. These results further validate our conclusion that the top principal component predominantly determines the response format, while the second and third components play a key role in summarizing the rationales used to derive conclusions.
Test Case #1 of Intervention at Layer 12

Question: It’s easier for human’s to survive in: A: a cave   B: the ocean.   C: a town   D: alone

Original Answer
Based on the instruction, the answer is (C) a town. It is easier for humans to survive in a town compared to a cave, the ocean, or alone.

Remove Top1
Based on the instruction provided, the correct answer is (C) a town. It is easier for humans to survive in a town compared to a cave, the ocean, or alone.

Remove Top3
Based on the instruction provided, the correct answer is (C) a town.
It is generally easier for humans to survive in a town compared to a cave, the ocean, or being alone. Towns provide access to basic necessities such as food, water, shelter, and social support, which are essential for human survival. While a cave may provide some shelter and protection from the elements, it may lack access to these essential resources. The ocean is a challenging environment that requires specialized skills and equipment to survive, and being alone can be isolating and dangerous.

Remove Top10
Sure! Based on the instruction provided, the correct answer is (C) a town.
It is generally easier for humans to survive in a town compared to a cave, the ocean, or being alone. Towns provide access to basic necessities such as food, water, shelter, and social interaction, which are essential for human survival. While it is possible to survive in a cave or alone, it is much more challenging and requires a significant amount of skill, knowledge, and resources.
Test Case #2 of Intervention at Layer 12

Question: What is used for sensing visual things?  A: nerves  B: tibia  C: nostril  D: cornea

Original Answer
The correct answer is (D) cornea. The cornea is the part of the eye that is responsible for sensing visual things. It is the clear, dome-shaped surface at the front of the eye that helps to focus light and form images.

Remove Top1
D: cornea. The cornea is the clear, dome-shaped surface at the front of the eye that helps to focus light and protects the eye from damage. It is responsible for sensing visual things, including light, color, and depth perception.

Remove Top3
The correct answer is (D) cornea. The cornea is the part of the eye that is responsible for sensing visual things. It is the transparent, dome-shaped surface at the front of the eye that helps to focus light and protects the eye from damage. The cornea plays a crucial role in vision, allowing us to see and interpret the visual world around us. The other options are not correct: * (A) nerves: While nerves are important for transmitting visual information from the eye to the brain, they are not the primary structure responsible for sensing visual things. * (B) tibia: The tibia is a bone in the leg, and while it may be involved in some eye movements, it is not directly involved in sensing visual things. * (C) nostril: The nostrils are not directly involved in sensing visual things. While the nose is responsible for detecting odors and sensing the environment, it is not the primary structure responsible for visual perception. Therefore, the correct answer is (D) cornea

Remove Top10
The correct answer is (D) cornea. The cornea is the transparent outer layer of the eye that helps to focus light and protects the eye from damage. It is responsible for sensing visual things, including light, color, and depth perception. The other options are not related to sensing visual things: * (A) nerves are responsible for transmitting visual information from the eye to the brain. * (B) tibia is a bone in the leg, not related to sensing visual things. * (C) nostril is a passageway in the nose that helps to warm, humidify, and filter the air we breathe, but it is not involved in sensing visual things.

As observed, these results align with the phenomena described in Section 4.2. Removing the top three components significantly alters the model response, often triggering explicit rationale generation and underscoring their essential role in response reasoning.

Reviewer paper summaries: This work identifies critical layers during fine-tuning by tracking representation changes. Spectral analysis reveals that these changes can be attributed to variations in the top three principal components within these layers. Furthermore, the authors leverage this insight to enhance defenses against targeted attacks, demonstrating its effectiveness.
Summary of strengths: - The identification of critical layers during fine-tuning presents a novel perspective.

- The method used to identify these layers is simple yet elegant. The paper provides in-depth explanations of its findings, and the demonstrated effectiveness in defense tasks further underscores the significance of this discovery.

- The ablation studies are comprehensive, and the paper is well-structured.
Summary of weaknesses: - The importance of middle layers aligns with findings from existing studies on concept learning. A comparison with prior work would help highlight the novelty and significance of these results.

- Lines 273–276 are somewhat difficult to follow, particularly the phrase: "... between $\delta^\ell$ and the LLaMA-2-7B-Base model." Clarification would improve readability.

- Have attack rates been evaluated on larger models, such as LLaMA-2-13B or LLaMA-2-70B? This would provide further validation of the approach’s scalability.
Comments, suggestions and typos: See weaknesses
Overall assessment scores: 4
Metareview score: 3.5
Metareview: This paper presents an approach to identify “critical layers” during fine-tuning in LLMs based on representation dynamics using CKA. The paper has interesting findings, for example suggesting that these critical layers are an intrinsic property of the models. Furthermore, the paper presents experiments that leverages these critical layers to reduce the success of adversarial attacks.
Summary of reasons to publish: - The paper provides an approach to identify “critical layers” in LLMs and generates interesting findings that shed more light on how representations change during training within LLMs.
- The paper shows that leveraging the identification of these layers can improve downstream tasks like susceptibility to backdoor attacks.