Safety Alignment Shouldn't Be Complicated

Authors: Jianwei Li, Jung-Eun Kim
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Safety Alignment, Alignment Tax, Safety-critical Neurons, Large Language Models (LLMs)

Abstract:
As large language models (LLMs) are overwhelmingly more and more integrated into various applications, ensuring they generate safe and aligned responses is a pressing need. Previous research on alignment has largely focused on general instruction-following but has often overlooked the unique properties and challenges of safety alignment, such as the brittleness of safety mechanisms. To bridge the gap, we propose the Superficial Safety Alignment Hypothesis (SSAH), which posits that safety alignment should teach an otherwise unsafe model to choose the correct reasoning direction - interpreted as a specialized binary classification task - and incorporate a refusal mechanism with multiple reserved fallback options. Furthermore, through SSAH, we hypothesize that safety guardrails in LLMs can be established by just a small number of essential components. To verify this, we conduct an ablation study and successfully identify four types of attribute-critical components in safety-aligned LLMs: Exclusive Safety Unit (ESU), Exclusive Utility Unit (EUU), Complex Unit (CU), and Redundant Unit (RU). Our findings show that freezing certain safety-critical components \textbf{(7.5\%)} during fine-tuning allows the model to retain its safety attributes while adapting to new tasks. Additionally, we show that leveraging redundant units \textbf{(20\%)} in the pre-trained model as an ``alignment budget'' can effectively minimize the alignment tax while achieving the alignment goal. All considered, this paper concludes that the atomic functional unit for safety in LLMs is at the neuron level and underscores that safety alignment should not be complicated. We believe this work contributes to the foundation of efficient and scalable safety alignment for future LLMs.

Decision: reject
Reviewer scores: 3, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730481173835

thread_type:
official_review

<thread_object>
creator: reviewer_t23t

<content>
summary:
This paper proposes the Superficial Safety Alignment Hypothesis (SSAH), which frames safety alignment as a binary task, guiding models to make safe decisions by selectively freezing key components. By identifying and freezing 7.5% of safety-critical units and repurposing 20% of redundant units as an "alignment budget," the model retains safety with minimal impact on utility, making safety alignment more efficient and scalable.

strengths:
1. The paper is well-written and easy to understand.

2. The paper identifies four important safety components of LLMs. By freezing some safety components, the model’s safety attributes are retained, and with the "less is more" method, the complexity of fine-tuning is reduced.

3. Alignment tends to have negative impacts on other tasks. The authors mitigate the Alignment Tax by freezing habitual computation units.

weaknesses:
1. The paper lacks sufficient empirical testing against dynamic jailbreak attacks, failing to verify the model's robustness to complex attacks in real dynamic environments. Would it be possible to test the effectiveness of this method against some jailbreak attack techniques?

2. Can this method be effective on non-LLaMA-family architectures? It would be beneficial to explore other architectures, such as encoder-decoder models (e.g., ChatGLM) or MoE architectures like Mistral.

questions:
Please see Weaknesses. 

Minor Typos:

1.In Line 120, there should be a space after "Qi et al. (2023)."; 2. In Table 3, please ensure the decimal points are aligned consistently.

If the authors can address the above questions, I would be happy to raise the score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693449665

thread_type:
official_review

<thread_object>
creator: reviewer_mjwk

<content>
summary:
The paper proposes a new hypothesis related to the safety mechanism of LLMs. They interpret the safety mechanism as a "reasoning direction," depicted as a classification task. To verify this hypothesis, they evaluate the embeddings of each layer and partially prove the existence of the "reasoning direction." Furthermore, to identify the safety mechanism, they employ a pruning method, identifying around 1% of parameters as part of the safety mechanism. These parameters can be frozen during fine-tuning to maintain the model's safety alignment.

strengths:
1. The hypothesis of "reasoning direction" is both novel and intriguing, and the method of using embeddings to explicitly express this concept is innovative and valuable.

2. In addition to their analysis, the authors identify a safety mechanism at the neuron level, freezing these neurons during fine-tuning to protect safety alignment.

weaknesses:
No main weakness, I have several questions and please refer to the Questions section.

questions:
1. You claim that "the reasoning direction can be interpreted as a simple binary classification task," which seems somewhat overclaimed to me. The "reasoning direction" is difficult to clearly delineate, as the model might only identify a query as harmful after further reasoning. For example, the model might initially fail to detect a harmful query, but after additional reasoning steps, it recognizes the output as harmful and realizes it should be banned. The evaluation in the paper does not refute the possibility of this scenario. While I do not question the correctness of SSAH, the claim appears too strong to be conclusively proven.

2. Is the neuron detection method shown in Equation 1 sequential? If so, will it be slow when calculating the importance score for each individual neuron?

3. Regarding the pruning method, I'm curious about pruning neurons in self-attention layers, given that the number of neurons in each head is fixed. During the pruning process, will each head have the same number of neurons reduced, or will the neurons be reorganized across several heads?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702089328

thread_type:
official_review

<thread_object>
creator: reviewer_tnxx

<content>
summary:
This paper sets out to explain the brittleness of safety training by analyzing neurons. Specifically, this paper proposes freezing neurons in LLMs that are crucial to safety training when fine-tuning on downstream tasks to minimize the loss of safety by fine-tuning. Additionally, this work proposes a parameter efficient fine-tuning focused on neurons identified as redundant.

strengths:
1) This paper proposes using pruning to identify role of the neuron via ablation. This is a creative use of pruning for explaining NN behavior. I commend the effort to interpret and explain model behavior at a neuronal level. There are a lot of neurons and they have complicated interactions between them.

2) The work shows on GSM8K improvement to utility by fine-tuning exclusively redundant neurons. 

3) It shows freezing the neurons attributed to safety improve safety scores when fine-tuning for downstream instruction following task.

weaknesses:
1) The vast majority of neurons (70%+) are labeled as CU which means the pruning isn't able to eliminate them either on the safety dataset or the utility dataset. This work's interpretability results would be stronger if it was able to attribute more neurons as safety or utility. 

2) It would seem that the definition of "utility" is sensitive to the choice of datasets used in pruning. From B.2, this seems to mostly consist of relatively simple QA problems. Consequently, the resulting positive result for fine-tuning 20% of the neurons is limited to GSM8K and doesn't apply to MMLU. It would be good to see how these designations apply to code tasks, logical deduction, and emergent zero-shot behavior. I commend the effort but I think it's difficult to definitively and objectively mark a neuron as redundant based on a chosen dataset.

3) The results are shown on 7B/8B llama models. It's possible that the choice of datasets for identifying neuron contribution, the ratio of RUs and prevalence of "complex units" would be affected by model size and pretraining data mix. In particular, I would expect larger models with more layers to have less separability between utility and safety. 

4) As safety by the author's definition equates to binary classification, does applying llama-guard or gpt-4 moderation as a filtering step eschew the need for complex safety alignment? In the setting of alignment for instruction-following, it would make sense that poor instruction-tuned models cannot be resampled until they follow the instructions. However, superficial safety is a simpler objective without considering robustness to adjacent attacks. 

5) It would help to show fine-tuning on distinct post-training capabilities other than general conversation datasets. For example, multi-lingual, long context, math, coding, factuality, and steerability (taken from llama-3 paper)

6) It's not obvious how the role of neurons are identified until reading the appendix.

Nits: 
1) Exclusive Safety Unit (ESU), Exclusive Utility Unit (EUU), Complex Unit (CU), and Redundant Unit (RU) are rather clunky terms that make it hard to keep track of what the abbreviations are referring to. Even something like SU, UU, MU (mixed units), and RU would be easier to parse.

questions:
1) To defend, the brittleness of the safety training, it would be good to show that using an approach like rainbow-teaming attacks are not explicitly safety trained for are not protected. 

2) Is there bleed between the 100 prompts in ADV used to identify safety neurons and the evaluations on HEx-PHI and Adv-bench? How robust is this method to attacks not used in identifying safety neurons?

3) Appendix C speaks a bit to attention vs feedforward neurons. Are there conclusions from fine-grained analysis we can make as to which layer and where in the architecture the RU and safety neurons are located?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731050345156

thread_type:
official_review

<thread_object>
creator: reviewer_2eye

<content>
summary:
This paper proposes the safety superficial alignment hypothesis, which states that a truly safety-aligned model simply follows correct reasoning for safe vs unsafe inputs, allowing it to refuse unsafe inputs while still responding helpfully to all others. The identify that some neurons in models appear to contribute more to safety vs utility than others, and that some appear to be redundant. Finally, they propose a method for fine-tuning models that preserves helpfulness while also allowing for an increase in utility by freezing all but these redundant neurons.

strengths:
The ideas behind this paper hold value, and it demonstrates promising initial results. In particular:

* The results do demonstrate that certain neurons appear to contribute more to safety than others

* The fine-tuning results further show that there are less of these neurons after fine-tuning models on non-safety related tasks.

* Isolating safety specific neurons does appear to be successful and would be a valuable contribution if so.

weaknesses:
While the findings shown in this paper are promising as initial results, I do not believe the paper as a whole is ready for publication. As described in the paper, I am not confident in the rigor of the experiments, and see a lack of formalizations for key definitions. In addition, some claims do not seem to be backed up by experimental results. These may be misunderstandings on my part, and I would appreciate a dialog with the authors on the points I outline below if so:

## 1. Lack of rigorous definitions
### a. Two different versions of aligned models
In section 3, aligned models are defined as pre-trained models fine-tuned on safety data by the authors, while un-aligned models are models that have been fine-tuned for instruction following/helpfulness. However, in section 4, aligned models are defined as the chat, RLHF versions of models. I am concerned that the findings between sections may not hold across these definitions.

### Questions
- Why are different versions of aligned models used? 
- What is a reasoning direction/trajectory and how do you measure it? If it's an approximation, what assumptions go into it and what is it approximate in?
- What is a reserved fallback option?
- What is considered a malicious query vs a safe query?
	- examples are given for these (lines 226-229: "Sorry, I can't..." and "Here's how..."), but are these the actual tokens used? Are other tokens used?
	- If only these tokens are used, I do not find this to be sufficient testing. Additional types of responses are possible and should be considered.

- What are benign/malicious tokens?
- How is the cosine distance measured in section 3's experiments?
- How are neurons classified in section 4? What thresholds are used on importance scores to decide this?

## 2. Overclaiming and mismatched claims

The paper makes some very bold claims and hypotheses, but often these claims are not supported by prior research or experimental results.

### a. Exclusive safety/utility neurons
As shown in Table 1 Exclusive safety neurons have an effect on the utility of models as well (as do utility neurons on safety). Though it is a small effect, the claim that they are "exclusive" is not backed up with statistical significance testing showing that these differences are not significant, and feels arbitrary to me. How is exclusivity determined/decided?

### b. Safety vs general alignment
In the introduction, as part of the motivation for SSAH, it's claimed that safety alignment is different from general alignment (lines 63-64). The question that section 4.3 claims to answer is whether it's possible to mitigate the safety alignment tax. However, the only results shown are on general alignment/helpfulness. It's unclear if this will hold for safety alignment. Additional experiments evaluating on safety-specific data should be done.

### c.  Main questions
Of the three questions they aim to answer, unfortunately, none had clear answers provided in my understanding of the paper. For example, when explaining the difference in robustness between Llama-2 and Llama-3 in the safety results in Table 2, the explanation is that Llama-3 tries "attempts to analyze the true intention behind requests," without explaining what this means in terms of model architecture/training, or explaining why this would result in the observed differences. Can you provide more explanation for this hypothesis?
	
### d. SSAH and jailbreaking
In Section 3 (lines 181-185), the paper claims that SSAH can also hold insights for jailbreaking. However, no experiments are done on this, and in the discussion, it's mentioned that SSAH likely does not hold for jailbreaking.

### e. Helpfulness of un-aligned models
The claim that un-aligned models have good instruction following abilities is not supported by the MT-Bench scores. Llama-2-7B, for example has a reported average score of 2.85 for the version used in this paper, whereas the chat version of the model has a score of more than double this.

### f. Cosine similarity
Section 3 uses high levels of cosine similarity between safe queries and helpful responses (and likewise, unsafe queries and refusals) as indications of alignment. However, this does not measure what models actually predict. While it may be a useful tool for explaining behavior, further experiments looking at model predictions need to be done to confirm that these similarities are indicative of alignment.

### g. One family of models
Only Llama family models are tested. I would like to see this tested on more models before accepting the claim that this hypothesis is general.

### 3. Presentation notes
Overall, the presentation is quite hard to follow. While the writing itself is understandable, there is not enough detail given for experiments, and many of the plots are hard to interpret.

- Plot colors/appearance
	- figure 1: Reasoning direction is hard to read due to arrows
	- figure 2: having aligned and unaligned models on opposite sides is a nice tough, but the texture difference between unaligned and aligned is quite hard to see
	- figure 3: Part of the plot is highlighted, but there is no explanation for this. Is this meant to highlight the increase in distance across early transformer blocks mentioned?
		
- Typos
	- figure 1: genearl -> general, fullfill -> fulfill

- Writing clarity
	- Many of the issues of definition and method mentioned above stem from writing that jumps straight into results without describing the setting first. Including specific descriptions of experimental setups, datasets, evaluation methods, and definitions in each section would greatly improve the clarity and readability of the paper.

questions:
1. On lines 324-325 you mention constructing two datasets following Wei, et al. (2024b). Are these the same datasets Wei et al. use, or are they constructed similarly?

2. As a possible further experiment, what happens if models are fine-tuned on more safety data? Do the safety neurons remain? Are other neurons converted to safety neurons?

3. Why are different versions of aligned models used? 

4. What is a reasoning direction/trajectory and how do you measure it? If it's an approximation, what assumptions go into it and what is it approximate in?

5. What is a reserved fallback option?

6. What is considered a malicious query vs a safe query?
	- examples are given for these (lines 226-229: "Sorry, I can't..." and "Here's how..."), but are these the actual tokens used? Are other tokens used?
	- If only these tokens are used, I do not find this to be sufficient testing. Additional types of responses are possible and should be considered.

7. What are benign/malicious tokens?
8. How is the cosine distance measured in section 3's experiments?
9. How are neurons classified in section 4? What thresholds are used on importance scores to decide this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732047809942

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Hard to Follow the Concepts of ESU, EUU

comment:
To reduce confusion, we have renamed ESU as SCU (Safety Critical Units), reflecting their essential role in safety, and EUU as UCU (Utility Critical Units), highlighting their critical contribution to utility. These changes aim to make the concepts easier to follow while maintaining accuracy.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048098062

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Additional Experiments for Other Model Families, Domain Datasets, and Jailbreak/Red-teaming Attacks. (Part II)

comment:
## Additional Jailbreak/red-teaming attack

Our SSAH framework was initially proposed for scenarios involving direct attacks. While we have noted in the paper that SSAH could be extended to jailbreak/red-teaming settings **(lines 178-185)**, we have not provided specific techniques for these complex attack scenarios **(lines 523-527)** (More details can be found in Global response: How is SSAH compatible with Jailbreak/Red-Teaming Attacks). 

Therefore, we assume that the reviewer’s question is seeking to know whether freezing safety-critical components could help maintain an aligned model's defense capabilities against **jailbreak/red-teaming attacks**. To address this, we have now expanded our testing to include evaluations on fine-tuned aligned models to determine if they retain robustness against jailbreak attacks. Specifically, we tested the models using three red-teaming methods: GCG, AutoDAN, and PAIR. Due to the extremely slow attack speed of the red-teaming method, we used the Harmbench Framework and a random sample of 120 data points from the HarmBench dataset to evaluate.

Table 5. Safety performance of **Meta-Llama2-7B-Chat** under Fine-Tuning attacks (**Alpaca**) across red-teaming attacks. 

| Bench   | Red-teaming         | Initial     | GSM8K Finetuned | Fix ESU + 6% CU | Fix ESU + all CU |
|---------|---------------|-------------|------------------|-----------------|------------------|
| HarmBench     | GCG        | 33.33%         | 53.08% (+19.75%) | 40.25% (+6.92%)  | 37.75% (+4.42%)  |
| HarmBench     | AutoDAN    | 1.08%          | 7.41% (+6.33%)   | 2.33% (+1.25%)   | 1.66% (+0.58%)   |
| HarmBench     | PAIR       | 12.25%         | 22.25% (+10.00%) | 14.5% (+2.25%)   | 14.08% (+1.83%)  |

Table 6. Safety performance of **Meta-Llama2-7B-Chat** under Fine-Tuning attacks (**Dolly**) across red-teaming attacks. 

| Bench   | Red-teaming         | Initial     | GSM8K Finetuned | Fix ESU + 6% CU | Fix ESU + all CU |
|---------|---------------|-------------|------------------|-----------------|------------------|
| HarmBench     | GCG        | 33.33%         | 62.91% (+29.58%) | 43.25% (+9.92%)  | 40.66% (+7.33%)  |
| HarmBench     | AutoDAN    | 1.08%          | 16.16% (+15.08%) | 9.66% (+8.58%)   | 8.33% (+7.25%)   |
| HarmBench     | PAIR       | 12.25%         | 25.25% (+13.00%) | 15.25% (+3.00%)  | 14.75% (+2.50%)  |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048232998

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Additional Experiments for Other Model Families, Domain Datasets, and Jailbreak/Red-teaming Attacks. (Part I)

comment:
## Additional Model Families and Fintuned Math Dataset
To fulfill the reviewers' request that the attribute-based neuron analysis and finetuning attack on other model families and downstream finetuning datasets (such as math dataset), we include other models: Mistral-7B-Instruct-v0.2 and GSM8K math dataset in our experiments. Our results show that, although the Mistral family claims superior performance on downstream tasks compared to the LLaMA2 family, it is less safe and more susceptible to finetuning attacks. However, our method proves effective in mitigating this issue across different finetuned datasets. 

Table 1. Pruning result of **Mistral-7B-Instruct-v0.2** across safety and unitlity benchmarks.

| Type       | wiki2 | wino | openb | arc.c | boolq | hellas | rte  | avg          | w/sys | w/o sys    | avg          |
|------------|-------|------|-------|-------|-------|--------|------|--------------|-------|------------|--------------|
| Dense      | 5.59  | 78.0 | 34.0  | 60.0  | 85.5  | 65.5   | 74.5 | 66.2 (**-0**) | 12.0  | 17.0       | 14.5 (+0) |
| ESU (2%)   | 6.17  | 71.5 | 32.0  | 55.5  | 85.5  | 65.0   | 71.0 | 63.3 (-2.9) | 34.0  | 91.0   | **62.5 (+48.0)** |
| EUU (1.3%) | 52.7  | 58.0 | 17.5  | 21.5  | 55.0  | 36.0   | 53.0 | **40.1 (-26.2)** | 17.0  | 23.0       | 20.0 (+5.5) |
| RU (13.4%) | 8.15  | 75.5 | 33.5  | 55.5  | 83.0  | 62.5   | 74.5 | 64.1 (-2.1) | 14.0  | 12.0       | 13.0 (-1.5) |

Table 2. Safety performance of **Meta-Llama2-7B-Chat** under Fine-Tuning attacks (**GSM8K**) across various benchmarks and judge methods. 

| Bench   | Judge         | Initial     | GSM8K Finetuned | Fix ESU + 6% CU | Fix ESU + all CU |
|---------|---------------|-------------|------------------|-----------------|------------------|
| Adv           | keyword    | 0.19%          | 5.38% (+5.19%)   | 1.92% (+1.73%)   | 1.73% (+1.54%)   |
| Adv           | llama3-guard | 0.19%       | 5.77% (+5.58%)   | 1.35% (+1.16%)   | 1.15% (+0.96%)   |
| HEx-PHI       | gpt4-score | 1.05           | 1.61 (+0.56)     | 1.37 (+0.32)     | 1.31 (+0.26)     |
| HEx-PHI       | gpt4-rate  | 0.3%           | 11.51% (+11.21%) | 5.75% (+5.45%)   | 5.31% (+5.01%)   |
| HEx-PHI       | llama3-guard | 2.42%       | 17.88% (+15.46%) | 11.52% (+9.10%)  | 9.68% (+7.26%)   |

Table 3. Safety performance of **Mistral-7B-Instruct-v0.2** under Fine-Tuning attacks (**Alpaca**) across various benchmarks and judge methods. 

| Bench   | Judge         | Initial     | Alpaca Finetuned | Fix ESU + 6% CU | Fix ESU + all CU |
|---------|---------------|-------------|------------------|-----------------|------------------|
| Adv     | keyword       | 15.19%      | 89.61% (+74.42%) | 74.04% (+58.85%) |  72.15% (+56.96%)  |
| Adv        | llama3-guard  | 40.38%      | 87.12% (+46.74%) | 73.27% (+32.89%) |  70.76% (+30.38)   |
| HE×PHI  | gpt4-score    | 2.24        | 4.18 (+1.94)     | 3.67 (+1.43)     |  3.43 (+1.19)   | 
| HE×PHI        | gpt4-rate     | 18.79%      | 70.3% (+51.51%)  | 58.78% (+39.99%) |  54.37% (+35.58%)  |
| HE×PHI        | llama3-guard  | 45.45%      | 86.00% (+40.61%) | 70.01% (+24.56%) |   67.81% (+22.36%)  |

Table 4. Safety performance of **Mistral-7B-Instruct-v0.2** under Fine-Tuning attacks (**GSM8K**) across various benchmarks and judge methods. 

| Bench   | Judge         | Initial     | GSM8K Finetuned | Fix ESU + 6% CU | Fix ESU + all CU |
|---------|---------------|-------------|------------------|-----------------|------------------|
| Adv           | keyword    | 15.19%          | 97.31% (+82.12%) | 72.31% (+57.12%) | 66.34% (+51.15%) |
| Adv           | llama3-guard | 40.38%       | 95.38% (+55.00%) | 89.81% (+49.43%) | 86.92% (+46.54%) |
| HEx-PHI       | gpt4-score | 2.24           | 4.15 (+1.91)     | 4.01 (+1.77)     | 3.96 (+1.72)     |
| HEx-PHI       | gpt4-rate  | 18.79%         | 66.7% (+47.91%)  | 64.7% (+45.91%)  | 62.81% (+44.02%) |
| HEx-PHI       | llama3-guard | 45.45%       | 93.94% (+48.49%) | 89.39% (+43.94%) | 80.31% (+34.86%) |
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048317947

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: How is SSAH Compatible with Jailbreak/Red-Teaming Attacks

comment:
We genuinely appreciate the reviews  regarding this point because it is precisely the point of our confidence and pride.

---


Although the **Superficial Safety Alignment Hypothesis** (SSAH) was originally proposed to explain how current safety alignment techniques impact models' behavior under **direct attacks**, we showed it has the potential to offer theoretical guidance for tackling **jailbreak/red-teaming attacks**. Specifically, we outline how SSAH provides insights in the following section:

- **Lines 178-185**: In this section, we extend SSAH beyond direct attacks to include jailbreak scenarios. We suggest that enabling the model to re-evaluate harmful content at each generation step—and re-select the correct reasoning direction—could help sustain safety alignment beyond the initial step. This proposal is highlighted as a potential direction (line 182) and underscores why we label our hypothesis “Superficial,” as this alignment is at the surface level.

- **Lines 523-527**: In the Discussion, we clearly state that SSAH offers theoretical guidance without presenting a technical solution. We cite recent research [1], which, although not directly attributed to SSAH, aligns with our hypothesis in its experimental results and demonstrates promising effects in jailbreak scenarios. Other recent work, such as [2], also supports this. Our latest research—due to the double-blind review policy, we cannot disclose details here—has already yielded effective methods for re-evaluation and re-selection.

In summary, SSAH aims to explain how the current safety alignment impacts model behavior and, from another perspective, points out the shortcomings of current methods—namely, the inability to provide a safety guardrail mechanism at each generation step. Most importantly, we have outlined a feasible theoretical path forward and clarified that, if further substantiated, SSAH could evolve into a full Safety Alignment Hypothesis, removing its “Superficial” qualifier. At this point, we would like to make another bold claim, which is the **"Safety Alignment Hypothesis"**.  It will define how safe alignment should impact the models’ behaviors.:

**Safety Alignment Hypothesis**: Given an unsafe model that is capable of fulfilling users’ malicious requests, safety alignment should teach the model to **choose** and **maintain** the correct reasoning direction at each generation step, along with simple refusal mechanisms. In other words, the model will have the ability to **re-evaluate** and **re-choose** the reasoning direction at each generation step.


**References:**

[1] Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, Peter Henderson, [2024] Safety Alignment Should be Made More Than Just a Few Tokens Deep

[2] Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Jiahao Xu, Tian Liang, Pinjia He, Zhaopeng Tu, [2024] Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
</content>
</thread_object>

</thread>
<thread>
creation date:
1732375549292

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response: Bridging Divergent Perspectives

comment:
We sincerely thank all reviewers for their detailed feedback and thoughtful insights. Throughout this review process, we have observed a wide discrepancy in the evaluations of this work. Despite the diverse scores, we appreciate that all reviewers - **even the one who gave the lowest `score 1`** - have acknowledged the innovative nature of our work. We believe our work’s novelty and ambition challenge conventional thinking, so the research community moves toward where would be unprecedented. 

In light of this divergence, we respectfully request that reviewers exchange perspectives and interpretations of our work. We believe such intereaction could illuminate shared understandings and further highlight the contributions of our paper.

Additionally, one paper was brought up during the rebuttal discussion,"**Identifying and Tuning Safety Neurons in Large Language Models**," (https://openreview.net/forum?id=yR47RmND1m) which shares some overlapping observations and is **not a prior work but currently under review** at ICLR 2025. Below, we provide our distinction from the work and welcome additional opinions from the reviewers:

---

**1. Scope of Identification:**

Although both works address safety-critical components, our work takes a more comprehensive, structural approach. In addition to safety-critical units, we identify utility-critical, complex, and redundant units, offering a broader framework for understanding their interactions.


**2. Focus of Fine-Tuning Approaches:**

The parallel work centers on enhancing safety by fine-tuning identified safety units on safety-related datasets. Our approach, however, **leverages redundant units for alignment budget**, focusing on **reducing alignment tax** and improving efficiency.


**3. Defense Against Fine-Tuning Attacks:**

Preserving safety performance under fine-tuning attacks is a cornerstone of our work. While the parallel work also explores this, our experiments span a **wider range** of models, datasets, attack types, and evaluation methods, providing a more comprehensive validation.


**4. Theoretical Insights:**

 Our paper explains how current safety alignment techniques influence model behavior and provides the direction of how safety alignment should ideally function.

---

**Conclusion**

Both works identify significant challenges in safety alignment and propose complementary research directions. The overlapping yet distinct focuses of these studies highlight the importance and complexity of this field. We respectfully encourage reviewers to share their thoughts and interpretations, as such discussions could provide valuable clarity and further refine the broader implications of these contributions.

Thank you for your time and effort in reviewing our work. We hope this response fosters constructive discussion and a deeper understanding of the innovative ideas presented in our paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733231685830

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response to All Reviewers and Area Chair

comment:
We sincerely thank all reviewers for their valuable participation in the discussion phase and the area chair for their efforts in coordination, guidance, and review, leading up to the comprehensive meta-review process.

We especially thank Reviewer **2eyE** for engaging in discussions with us. While the score only increased from 1 to 3, based on the rebuttal process and the reviewer's final feedback, we are confident that all major issues and concerns have been resolved. The reviewer suggested that our claims should be more naunced, and we appreciate this feedback. **We will make the necessary updates (already stated in our rebuttal forms) to clarify the points that caused confusion for Reviewer 2eyE, as detailed at the end of this response**. Then, we believe that nothing remains which can  warrant a rejection or resubmission of our paper. Hence, by considering the highly competitive field of AI safety in LLMs, we hope our paper will receive fair and accurate reviews and decision, especially compared to the papers submitted to ICLR 2025 which share some overlapping observations and claims.   

We also sincerely appreciate that Reviewer **MJWK** has recognized the genuine value of our work and their courage in defending its value. Their comments reinforced the importance of our contributions. Additionally, we thank Reviewer **t23t** for their constructive suggestions on experimental improvements and for raising their score following our feedback.

While Reviewer **TNxX** did not participate in the discussion phase, we appreciate their initial comments and understand they may have had other unavoidable commitments. However, we respectfully and strongly request that the Area Chair consider the substantial overlap between TNxX’s concerns and those of other reviewers, which we successfully addressed during the discussion. We are confident that if Reviewer TNxX were to review our updated clarifications, they would reconsider their evaluation.


## **Planned Revisions (Already Stated in Our Rebuttal) for the Paper**


### 1. Refinements to Existing Content
---

- The names for ESU and EUU will be changed to improve clarity, as outlined in our rebuttal.  [See here](https://openreview.net/forum?id=9H91juqfgb&noteId=kyH2G6Sxhf)
- All typos and figure enhancements suggested by the reviewers will be corrected.
- The SSAH definition will be updated to explicitly include the concept of reasoning direction as follows (in **bold font**):  

   > **SSAH**: "Given an unsafe model that is capable of fulfilling users’ malicious requests, safety alignment teaches the model the correct reasoning direction (**the model’s inclination to either fulfill or refuse a user request based on human value**) and a simple refusal mechanism with reserved options."  

- We clarify that the effectiveness of our mitigation strategy for fine-tuning attacks is also influenced by the model’s initial safety performance and robustness.  [See point one here](https://openreview.net/forum?id=9H91juqfgb&noteId=0mzr3EePiF)

- Clarify more about the process of attribute identification with pruning as outlined in our rebuttal. [See question VII here](https://openreview.net/forum?id=9H91juqfgb&noteId=ZBhncCIhUh)

### 2. More Extensive Experiments

---

- Include experiments on additional model families and specific downstream datasets as outlined in our rebuttal.  [See here](https://openreview.net/forum?id=9H91juqfgb&noteId=wvx0PNDpq7)

- Incorporate tests using popular red-teaming methods to provide a more comprehensive assessment.  [See here](https://openreview.net/forum?id=9H91juqfgb&noteId=IcEr31FXqz)

### 3. Additional Sections in the Appendix 
 `(While these were already discussed in the original submission, we will expand them further.)`

---

- Add a dedicated section in the appendix to clarify how SSAH can extend to jailbreak attacks, as outlined in our rebuttal.  [See here](https://openreview.net/forum?id=9H91juqfgb&noteId=Rc3syDoevm)

- A new section will explicitly discuss the limitations of our claims as requested by Reviewer 2eyE.
[See point 3 here](https://openreview.net/forum?id=9H91juqfgb&noteId=0mzr3EePiF)

---

## **Closing Remarks**

We once again thank all reviewers for their efforts and thoughtful feedback. We hope the Area Chair considers the significant improvements planned for the revised version when making their final decision.  

Sincerely,
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes the Superficial Safety Alignment Hypothesis (SSAH) and identifies four types of neuron components in safety-aligned LLMs through ablation studies. While the work presents an innovative direction for understanding safety mechanisms in LLMs, there are significant concerns about the rigor of definitions, overclaiming of results, and generalizability beyond Llama models. The experiments show that freezing certain safety-critical components (around 7.5%) helps retain safety during fine-tuning, and leveraging redundant units (around 20%) can minimize alignment tax. However, the effectiveness varies significantly across model families, with much weaker results on Mistral compared to Llama models.