Title: Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Mechanistic Interpretability

Safety Alignment

Neuron Analysis

Alignment Tax
Abstract: Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment. In this paper, we explore the inner mechanisms of safety alignment through the lens of mechanistic interpretability, focusing on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors. We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety. Experiments on multiple prevalent LLMs demonstrate that we can consistently identify about 5% safety neurons, and by only patching their activations we can restore over 90% of the safety performance across various red-teaming benchmarks without influencing general ability. The finding of safety neurons also helps explain the ''alignment tax'' phenomenon by revealing that the key neurons for model safety and helpfulness significantly overlap, yet they require different activation patterns for the same neurons. Furthermore, we demonstrate an application of our findings in safeguarding LLMs by detecting unsafe outputs before generation.

Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons

Abstract

Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment. In this paper, we explore the inner mechanisms of safety alignment through the lens of mechanistic interpretability, focusing on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors. We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety. Experiments on multiple prevalent LLMs demonstrate that we can consistently identify about 5% safety neurons, and by only patching their activations we can restore over 90% of the safety performance across various red-teaming benchmarks without influencing general ability. The finding of safety neurons also helps explain the 'alignment tax' phenomenon by revealing that the key neurons for model safety and helpfulness significantly overlap, yet they require different activation patterns for the same neurons. Furthermore, we demonstrate an application of our findings in safeguarding LLMs by detecting unsafe outputs before generation. The source code is available at https://github.com/THU-KEG/SafetyNeuron .

1 Introduction

Large language models (LLMs) are celebrated for their sophisticated capabilities in natural language processing and various downstream applications (Touvron et al., 2023; Achiam et al., 2023; Jiang et al., 2024; Team et al., 2023). However, as they increase in complexity and influence, LLMs pose safety risks such as generating misinformation, harmful content, and biased responses, which could cause profound negative social impacts (Ganguli et al., 2022; Mazeika et al., 2024; Shen et al., 2023). Although advanced alignment algorithms have significantly improved the safety of LLMs (Bai et al., 2022a; Rafailov et al., 2024; Ethayarajh et al., 2024), research indicates that these aligned models remain highly vulnerable to malicious attacks (Huang et al., 2023; Yang et al., 2023). Understanding the mechanisms of safety alignment and the LLMs' inner workings of safe behaviors would facilitate designing more robust alignment algorithms in a principled way.

In this work, we aim to demystify the mechanisms behind safety alignment from the perspective of mechanistic interpretability (MI), which focuses on reverse-engineering neural models into humanunderstandable algorithms and concepts (Elhage et al., 2021). A typical MI pipeline includes attributing model behaviors to specific model components and verifying that the localized components have causal effects on model behaviors with causal mediation analysis techniques like activation

patching (Vig et al., 2020; Meng et al., 2022). However, existing MI methods (Wang et al., 2022a; Hanna et al., 2024; Geiger et al., 2024) mainly focus on attributing tasks requiring only prompting and few-token outputs to a limited search space of model components (e.g., attention heads). They cannot be directly applied to safety alignment, which naturally requires open-ended outputs and extensive model parameters as a high-level ability.

Considering that neurons are the most fundamental units in LLMs, and that prior work (Dai et al., 2022; Wang et al., 2022b; Gurnee et al., 2023, 2024) has shown that neurons encode a wide range of functionalities, we choose to investigate the safety mechanisms from a fine-grained, neuron-level perspective. We propose a two-stage framework (Figure 1) for identifying safety-related neurons (dubbed as safety neurons ) and verifying their causal effects. The basic idea is that association is necessary for causality. Hence, we can first narrow down the search space by identifying the neurons having associations with safety behaviors and then only evaluate their causal impact on model safety. In the first stage, we employ inference-time activation contrasting to compute change scores , which quantify the association of neurons to safety. In the second stage, we propose dynamic activation patching to assess the causal effect of these neurons on the safety of long-range model outputs. Based on the framework, we make three-fold contributions:

We identify safety neurons across four open-sourced LLMs: Llama2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), Gemma-7B (Team et al., 2024), and Qwen2.5-3B (Qwen et al., 2025). We further demonstrate that: (1) Safety neurons identified by our framework are sparse and causally effective ( 5% of the neurons in unaligned models have over 90% causal effects on safety alignment (§ 4.2), while the neurons identified via ablation-based method (Wei et al., 2024; Zhao et al., 2025) are not effective in our verification stage. (2) Safety neurons encode transferable mechanisms, which are generally effective on multiple red-teaming benchmarks without sacrificing generation quality (§ 4.3). (3) Safety neurons are robust to training randomness. In different random trials, our framework identifies essentially the same group of safety neurons (§ 4.4).

We leverage safety neurons to provide a potential explanation for the widely-recognized alignment tax issue (Askell et al., 2021; Ouyang et al., 2022). Using our proposed framework, we find that the key neurons involved in the processes of safety alignment and helpfulness alignment exhibit significant overlap, while the neurons identified for other abilities like reasoning are less similar. For the key neurons shared by safety and helpfulness, when we activate them in the way of helpfulness alignment, the models' safety performance degrades, and vice versa. This implies that alignment tax comes from requiring different activation patterns for a highly overlapping group of neurons (§ 5).

We utilize safety neurons to develop an LLM safeguard (Inan et al., 2023), by showing that an effective unsafe generation detector can be built using the activations of safety neurons to predict, before actual generation, whether the response will contain harmful content. This approach improves model safety by refusing to respond when harmful content is detected. Experimental results show that adding this safeguard can significantly improve the safety of unaligned models and further enhance model safety after alignment (§ 6).

2 Preliminaries

2.1 Safety Alignment

Although LLMs pre-trained on massive pretraining corpora have exhibited strong ability (Touvron et al., 2023; Jiang et al., 2023; Team et al., 2024). Further training is still needed to align LLMs with human preferences and mitigate risks. In common practice, supervised fine tuning (SFT) or instruction tuning is the first stage of alignment where LLMs are trained on diverse high-quality instruction data in a supervised manner. After that, preference learning is performed to further align the instructiontuned model to human preference. Reinforcement Learning from Human Feedback (RLHF) is the most well-known method for preference learning (Bai et al., 2022a,b). Training a reward model on human-labeled preference data and subsequently using this reward model in reinforcement learning can significantly enhance the model's helpfulness and harmlessness.

Due to the training instability and additional resources required by the reward model of RLHF, direct preference optimization (DPO) (Rafailov et al., 2024) has become a popular alternative (Tunstall et al., 2023; Ivison et al., 2023). The training efficiency can be further improved with minimal

Multi-Head Attention

Multi-Layer Perceptron

L

⨉

Multi-Head Attention

Multi-Layer Perceptron

L

⨉

Change

Scores

Neuron

Activations

Inference-Time Activation Contrasting

Aligned LLM

Unaligned LLM

What is the … ? Sure, here are

What is the … ? Sure, here are

What is the … ? Sorry, I cannot provide…

Multi-Head Attention

Multi-Layer Perceptron

L

⨉

Multi-Head Attention

Multi-Layer Perceptron

L

⨉

LLM for patching

What is the … ? Sure, here are

What is the … ? Sure, here are

What is the … ? Sorry, I cannot provide…

How do I … ? It is not

How do I … ? It is not

Activation

Patching

appropriate

Update

Prompt

Dynamic Activation Patching

Cost Model

-10.5

Evaluate Completed Prompt

performance degeneration when combined with parameter-efficient fine-tuning (PEFT) methods (Sun et al., 2023; Hsu et al., 2024; Li et al., 2024). We also adopt DPO in our preference learning stage for its efficiency and effectiveness.

While safety alignment has been proven effective in enhancing model safety, it has a certain cost known as alignment tax (Askell et al., 2021): the process of improving model safety inevitably diminishes the model's helpfulness. In this paper, we offer a preliminary explanation for this phenomenon with our findings.

2.2 Neurons in Transformer

Transformer . Transformer-based language models typically consist of embedding and unembedding layers W E , W U ∈ R |V|× d with a series of L transformer blocks in-between (Vaswani et al., 2017). Each layer consists of a multi-head attention (MHA) and a multi-layer perceptron (MLP).

Given an input sequence w = ⟨ w 0 , . . . , w t ⟩ , the model first applies W E to create an embedding h i ∈ R d for each token w i ∈ w . h i is referred to as residual stream (Elhage et al., 2021). The computation performed by each Transformer block is a refinement of the residual stream (layer normalization omitted):

The MLPs in Transformer models we used (Touvron et al., 2023; Team et al., 2023) are:

where W down , W gate , W up ∈ R d m × d are projection matrices, σ ( · ) is activation function, ⊙ is element-wise product operator.

MLP Neurons . In the context of neural networks, the term 'neuron' can refer to a single dimension of any activation. We choose to study neurons in the intermediate layer of MLP (activation before down projection) since it has been shown such neurons encode diverse interpretable features (Wang et al., 2022b; Dai et al., 2022; Gurnee et al., 2023). Furthermore, each row of the down projection matrix in Equation 2 can be interpreted as the value vector of the corresponding neuron. This interpretation allows us to explore the tokens a neuron promotes or suppresses (Geva et al., 2021).

3 Finding Safety Neurons in LLMs

We first introduce how to locate neurons with significant activation differences between the aligned and unaligned models using inference-time activation contrasting , followed by dynamic activation patching to verify the causal effects on model behaviors and determine the minimal set of neurons.

3.1 Inference-time Activation Contrasting

We first introduce the method for identifying candidate neurons responsible for the capabilities LLMs acquire through specific forms of training. Given two LLMs, M 1 and M 2 , where M 2 has acquired a specified ability through fine-tuning that M 1 lacks, and this fine-tuning preserves the functionality of the components under investigation (for neurons, this refers to their corresponding key and value vectors introduced by Geva et al., 2021). For a given prompt w = ⟨ w 0 , . . . , w t ⟩ , we denote the generation from M 1 and M 2 as w 1 = ⟨ w t +1 , . . . , w t + m ⟩ and w 2 = ⟨ w ′ t +1 , . . . , w ′ t + n ⟩ respectively. The inference-time activation of M 1 can be collected effectively with a forward pass on [ w,w 1 ] (the concatenation of prompt and generation, denoted as ¯ w 1 ) and collect neuron activation on the token index from t to t + m -1 . The activation of M 2 is also collected on ¯ w 1 to ensure comparability of activations. As we will demonstrate later, this approximation does not affect the effectiveness of our method.

Let a ( l ) i ( M 1 ; w ) [ j ] ∈ R be the activation of the i th neuron in layer l of M 1 at the j th token of a prompt w , and denote the number of tokens in prompt w as | w | . Given the prompt dataset D , we define the M 1 -based change score S ( l ) i ( M 1 , M 2 ; D ) (and similarly for M 2 -based change score with the ¯ w 1 replaced by ¯ w 2 in the following equation) of i th neuron in layer l as the root mean square of difference between inference-time activations of M 1 and M 2 :

To find safety neurons we choose the model after SFT as M 1 (denoted as SFT ) and the model after safety alignment as M 2 (denoted as DPO ). Then we sort all the neurons by the descending order of their M 1 -based change scores computed on some safety-related datasets and use the top neurons as the safety neuron candidates in experiments. Appendix D discusses the difference between M 1 -based and M 2 -based change scores and some other potential design choices of our framework.

3.2 Dynamic Activation Patching

To evaluate the causal effect of specific neurons in an open-ended generation scenario, we propose dynamic activation patching. This method involves a prompt w , two models M 1 and M 2 (which may differ from the previous section), and several forward passes. Specifically, we repeat the following steps until the generation process is complete: (1) Cache activations: run the model M 2 on the current prompt w and cache the activations of the investigated neurons; (2) Patched model run: run the model M 1 on the same prompt w with the activation of investigated neurons replaced by cached activation while the other neurons keep unchanged; (3) Get the next token prediction and append it to the prompt w . A more detailed implementation can be found in Algorithm 1.

Let ˜ w 1 be the completed prompt obtained from dynamic activation patching, with all other notations consistent with those defined previously. Given the evaluation dataset D , a metric F that assigns a real number score to each prompt, we define the causal effect C of specific neurons as follows:

The intuition behind Equation 4 is that if specific neurons can faithfully explain the capabilities of model M 2 that M 1 lacks, then the causal effect C should be close to 1 . Conversely, a causal effect C close to 0 indicates a negligible causal effect.

To comprehensively evaluate the causal effect of safety neurons on LLMs' safety behavior, we use DPO as M 2 , and M 1 can be either SFT or the pre-trained LLMs before SFT (denoted as Base ) in the following experiments unless otherwise specified.

4 Examining Safety Neurons

In this section, we explore the properties (sparsity, causal effect, transferability, and stability on training) of safety neurons with a series of experiments. The discussion of other properties of safety neurons can be found in Appendix C.

0

1

2

3

4

5

0.00

0.25

0.50

0.75

1.00

Causal Effect

0

1

2

3

4

5

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO

Neurons (%)

Qwen2.5

Gemma

Mistral

Llama2

Safety Neurons

Random Neurons

4.1 Investigation Setup

Models . To comprehensively investigate the safety neuron phenomenon in a more realistic setting, we utilize four different pre-trained LLMs: Llama2-7b-hf (Touvron et al., 2023), Mistral-7b-v0.1 (Jiang et al., 2023), Gemma-7b (Team et al., 2024) and Qwen2.5-3B (Qwen et al., 2025), which we denote as Llama2 , Mistral , Gemma and Qwen2.5 for brevity. Details of these models can be found in Appendix B.

Alignment . We first conduct SFT on ShareGPT (Chiang et al., 2023) following the recipe of Wang et al. (2024). Then we perform safety alignment using DPO on the HH-RLHF-Harmless (Bai et al., 2022a). We select (IA) 3 (Liu et al., 2022) as our PEFT method and apply it exclusively to the MLP layers (details can be found in Appendix B.1). Since (IA) 3 operates by multiplying each activation by a re-scaling factor without altering the underlying parameters, it preserves the functionality of the MLP neurons, which is fundamental to our approach as discussed before. The evaluation results of these models can be found in Appendix E.2.

Evaluation . We compute change scores on HH-RLHF-Harmless and evaluate the causal effect on Beavertails (Ji et al., 2024). For metrics, we use the cost model beaver-7b-v1.0-cost from Dai et al. (2024). The cost model is a trained reward model that assigns a cost score to each prompt based on its safety (lower means safer). Then we compute the causal effect of neurons by Equation 4. We use cost score exclusively as our safety metric in the subsequent analysis due to its efficiency, widespread use, and alignment with human judgments (Liu et al., 2023; Duan et al., 2024; Kong et al., 2024). We also present the evaluation results using GPT-4 (Achiam et al., 2023) in Appendix E.1.

Baselines . Zhou et al. (2025) identified key attention heads whose ablation increases attack success rates (ASR), a finding that complements our work. However, due to differences in the focus of the two studies, we do not make a direct comparison. We use Pruning (Wei et al., 2024) and SN-Tune (Zhao et al., 2025) as baselines for comparison. Pruning employs methods such as the Wanda score (Sun et al., 2024) to locate parameters that have the greatest impact on ASR, while SN-Tune identifies safety neurons based on their contribution to residual stream on safety-related data. These ablationbased methods may be influenced by the existence of the "Hydra effect"(McGrath et al., 2023), which suggests that ablating certain components in LLMs may trigger compensatory behaviors in other components. The implementation details of these methods are described in Appendix B.3.

4.2 Safety Neurons are Sparse and Causally Effective

Patching a large enough portion of neurons in activation patching can always restore the alignment performance. Therefore, we first check whether the identified safety neurons are sparse, which will allow us to explain and utilize these neurons effectively. We incrementally increase the number of patched neurons in descending order of neuron change scores. The results, illustrated in Figure 2, demonstrate that increasing the number of patched neurons enhances the safety of the patched model gradually, regardless of whether it is Base or SFT . Notably, after patching approximately 5% of all

the neurons, SFT can recover over 90% of DPO 's safety performance, occasionally even exceeding the full DPO (Table 1).

To rule out the possibility that patching any arbitrary set of neurons with activations DPO enhances model safety equally, we conduct experiments on randomly sampled neurons, ensuring that the number of neurons in each layer matches that of the safety neurons. The results, shown in Figure 2, indicate a significant causal effect gap between the randomly sampled neurons and safety neurons. We further conducted a t-test to compare the cost scores obtained from patching 5% safety neurons versus random neurons. The p-values for all groups fall within the range from 1 . 15 × 10 -6 to 1 . 67 × 10 -18 , indicating that the differences between random neurons and safety neurons are statistically significant. This result suggests that safety alignment indeed relies on these sparse safety neurons.

4.3 Safety Neurons Encode Transferable Mechanisms

We further investigate whether the effectiveness of safety neurons is transferable by checking whether patching these neurons can enhance model safety on red-teaming benchmarks other than the trained datasets. To evaluate transferability, we select four benchmarks designed for red-teaming LLMs: Beavertails (Ji et al., 2024), RedTeam (Ganguli et al., 2022), HarmBench (Mazeika et al., 2024), and JailBreakLLMs (Shen et al., 2023). Additionally, we evaluate whether the enhancement of model safety comes at the expense of generation quality on various general benchmarks, including: Wikitext-2 (Merity et al., 2016), MMLU (Hendrycks et al., 2021),

Table 2: Value vectors of the top safety neurons from Llama2-7b , projected onto the vocabulary space. MLP.v l n denotes the down projection vector of the n -th neuron in layer l . We omitted some tokens for better visualization.

GSM8K (Cobbe et al., 2021), BBH (Suzgun et al., 2023), and TruthfulQA (Lin et al., 2022). We present the causal effects and average performance change to models without patching in Table 1, leaving the full original results in Appendix E.3. The results indicate that the safety of the model improves significantly across all benchmarks after being patched with safety neuron activations. This demonstrates the transferability of safety neurons. We also find the ablation-based methods are less causally effective in our verification setting. Additionally, we observed that the general capabilities of the patched model degenerated only marginally. This confirms that safety neurons encode transferable

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

0.5

0.6

0.7

0.8

0.9

1.0

(a)

Safety and helpfulness

neurons

(b)

mechanisms rather than shallow patterns depending on specific datasets. The implementation details are described in B.2.

Moreover, we investigate the related tokens of top safety neurons by projecting their corresponding value vectors into the vocabulary space (Geva et al., 2021), as shown in Table 2 (full results are shown in Table 9). We observe that the top tokens associated with these safety neurons do not contain any safety-related content. However, there are human-recognizable patterns among them, such as neurons promoting words related to food, conjunctions, and closing brackets. This differs from the toxic vectors identified by Lee et al. (2024), which suggests that reducing toxicity is done by avoiding the vectors related to toxic tokens. This difference may come from our investigation range (comprehensive safety alignment) being larger than merely reducing toxicity. Consequently, the mechanisms corresponding to safety neurons are likely more complex, and we plan to explore the specific safety mechanisms in future work.

4.4 Safety Neurons Are Robust to Training Randomness

To further validate our findings, we explore whether safety neurons are robust in the alignment process, i.e., whether the randomness in the alignment training influences the identification of safety neurons. We train five different SFT and DPO models using different random seeds and find that the overlap and Spearman's rank correlation coefficients of the identified safety neurons both exceed 0 . 95 across different model families. Additionally, the error bars (Figure 2) obtained from repeating experiments in § 4.2 with these different models also indicate that the impact of training randomness on safety neurons is minimal.

Combining all these findings, we suggest that the safety neurons identified by our method are prevalent in the base models, and safety alignment algorithms exemplified by DPO (Rafailov et al., 2024) can moderate them to enhance LLMs' safety, presenting a possible mechanism of safety alignment. Investigating how safety neurons evolve during pre-training and whether they consistently emerge is a promising direction for future research.

5 Interpreting Alignment Tax

From the perspective of safety neurons, we provide a mechanistic interpretation for the widelyrecognized alignment tax issue (Askell et al., 2021; Ouyang et al., 2022), which refers to safety alignment enhancing model safety at the cost of model helpfulness, and vice versa.

We first explore the relationship between safety neurons and other preference neurons , which are the neurons identified with our framework for other preference-learning objectives. Specifically, we perform preference learning using DPO on 7 preference datasets categorized into 4 classes: (1) Safety , including HH-Harmless ( Harmless ) (Bai et al., 2022a) and RewardBench-Safety ( Safety ) (Lambert et al., 2024); (2) Helpfulness , including HH-helpful ( Helpful ) (Bai et al., 2022a) and Stan-

Beavertails

HarmBench

RedTeam

0

5

10

15

20

Transformed Cost Scores

Llama2

Beavertails

HarmBench

RedTeam

Mistral

Beavertails

HarmBench

RedTeam

Gemma

Beavertails

HarmBench

RedTeam

Qwen2.5

SFT

SFT+Guard

DPO

DPO+Guard

ford Human Preferences ( SHP ) (Ethayarajh et al., 2022); (3) Reasoning , including RewardBenchReasoning ( Reasoning ) (Lambert et al., 2024) and H4 Stack Exchange Preferences ( H4SE ) (Lambert et al., 2023); (4) Information Extraction , including IEFeedback (Qi et al., 2024). Then, using the same framework as for identifying safety neurons, we identify the top 5% preference neurons respectively and calculate Spearman's rank correlation coefficients between different preference neurons. The results of Llama2 are shown in Figure 3a. We observe that safety neurons and helpfulness neurons exhibit high inter-correlations, while the other preference objectives exhibit much lower correlations with them. This implies the potential shared mechanism between safety and helpfulness within LLMs. The results of Mistral and Gemma can be found in Appendix E.4.

We further investigate whether the key neurons shared by safety and helpfulness have a causal effect on both behaviors and see how this results in the alignment tax. We perform dynamic activation patching between two DPO s trained on Harmless and Helpful with the preference neurons shared between models trained on Safety and SHP . We evaluate on Beavertails using its cost model and reward model from Dai et al. (2024), respectively. The results, shown in Table 3, indicate that using the activations from the helpfulness DPO consistently improves the helpfulness of the safety DPO across all LLMs, while simultaneously reducing the model's safety. The reverse direction yields similar results. This demonstrates that the alignment tax arises from requiring different activation patterns of the same neurons. We further find that safety and helpfulness alignment indeed compete for control over these neurons during joint training (Appendix E.4). Besides, the causal effects of other preference neurons on safety and helpfulness (Figure 3b) are much lower, indicating different underlying mechanisms between safety/helpfulness and other capabilities.

6 Application: Safeguard for LLMs

We further explore the applications of our findings on safety neurons, presenting a preliminary use case:

training a safeguard for LLMs based on safety neurons. The well-known Llama Guard (Inan et al., 2023) moderates LLM generations after detecting that harmful contents are generated, while we investigate whether the activations of safety neurons can predict harmful outputs before actual generation. This enables us to reject harmful generation in advance, improving inference efficiency.

First, we verify whether safety neuron activations can be used to train an effective classifier for unsafe behaviors and evaluate its generalizability. We cache neuron activations from SFT at the last token

of the prompt and create labels for these activations based on the cost scores of the corresponding generation 1 on the previously used 5 red-teaming benchmarks: HH-Harmless (Bai et al., 2022a), Beavertails (Ji et al., 2024), RedTeam (Ganguli et al., 2022), HarmBench (Mazeika et al., 2024), and JailBreakLLMs (Shen et al., 2023). A comprehensive cross-validation demonstrates the classifier, trained on 1500 safety neuron activations, achieves 76 . 2% accuracy on average, indicating its potential for safeguarding LLMs. More detailed results are in Appendix E.5.

We can use the trained classifier to predict whether the LLM will produce harmful content before generating the first token. Then, we can either halt generation and output a predefined response or continue generating with a refusal prefix (e.g., 'sorry'). We apply the safeguard trained on SFT activations from HH-Harmless to both SFT and DPO , with a simple evaluation protocol: we compute the average cost scores on accepted responses as a proxy for safeguarding results. The results, presented in Figure 4, indicate that the safeguard significantly enhances the safety of unaligned models across all benchmarks. For models that have already undergone safety alignment, the safeguard can further improve safety. Besides, the overhead of the classifier is less than 0 . 001 seconds, which is negligible compared to generation, validating the potential value of this preliminary method.

7 Related work

Neuron-Level Interpretability for Transformer . Identifying interpretable neurons has been a key goal in mechanistic interpretability for Transformers (Geva et al., 2021; Elhage et al., 2022; Gurnee et al., 2023, 2024). Geva et al. (2021) proposed viewing the feed-forward networks as key-value memories, offering a novel interpretive framework. Dai et al. (2022) identified knowledge neurons, whose activations correlate with facts, while Wang et al. (2022b) found skill neurons predictive of task labels. Gurnee et al. (2023) localized neurons relevant to specific features using sparse probing. However, these approaches are task-specific and not directly applicable to safety alignment. Gurnee et al. (2024) circumvented the need for ground-truth labels by using unsupervised methods to find universal neurons, leading to interpretable neuron families. Recent work by Lee et al. (2024) interpreted DPO in GPT-2 and identified toxic neurons affecting model behavior, while Yang et al. (2024) showed that DPO's role extends beyond mitigating toxicity. Stolfo et al. (2024) identified confidence regulation neurons, revealing how induction heads use entropy neurons to adjust confidence. For safety-related neurons, predicting mechanistic patterns remains challenging.

Understanding Safety Mechanism of LLMs . Existing interpretability research on LLM safety can be broadly categorized into two perspectives: Representation Engineering (RepE, Zou et al., 2023) and Mechanistic Interpretability (MI, Elhage et al., 2021). RepE perspective takes a top-down approach, analyzing the residual stream to identify features (Zou et al., 2023; Zheng et al., 2024), which are then linked to neurons (Lee et al., 2024) or attention heads (Arditi et al., 2024). However, it is difficult to provide fine-grained mechanistic understanding and is usually used for steering model behavior rather than as an interpretability tool (Højer et al., 2025). MI perspective, on the other hand, takes a bottom-up approach, focusing on how basic model components contribute to safety. Wei et al. (2024) first introduced safety neurons as individual parameters. Since features in transformers are usually represented as vectors, it is difficult to interpret how different parameters in a single vector play different mechanistic roles. Li et al. (2025) adopted a safety layer perspective, which is too coarse-grained compared to neurons and attention heads for providing a mechanistic understanding. Zhou et al. (2025) identified key attention heads whose ablation increases attack success rates (ASR). Since MLP neurons constitute approximately two-thirds of the model's parameters and serve as the foundational units for functionality, it is important to study safety mechanisms from the perspective of MLP neurons and further investigate their interaction with attention heads. A prior work (Zhao et al., 2025) identified safety neurons based on their contribution to the residual stream of safety-related data. This ablation-based method may be influenced by the existence of the "Hydra effect" (McGrath et al., 2023), which suggests that ablating certain components in LLMs may trigger compensatory behaviors in other components. Therefore, we adopted an opposite approach, enhancing rather than ablating, to locate neurons that have a causal effect on safety.

8 Conclusion

In this work, we explore safety alignment in LLMs through mechanistic interpretability. We identify safety neurons under an open-ended generation scenario, demonstrating that they are sparse, effective, and consistent across trials. Our findings reveal that safety and helpfulness neurons are highly overlapped, given a possible interpretation of the alignment tax issue. We also demonstrate a practical application of safety neurons, building a safeguard for LLMs using safety neuron activations, further enhancing the safety of aligned models.

Acknowledgments

This work is supported by Beijing Natural Science Foundation (L243006) and National Natural Science Foundation of China (62476150).

References

Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023. URL https://arxiv.org/pdf/2303.08774.pdf .

Arditi, O. Obeso, A. Syed, D. Paleka, N. Rimsky, W. Gurnee, and N. Nanda. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717 , 2024. URL https://arxiv.org/abs/2406.11717 .

Askell, Y. Bai, A. Chen, D. Drain, D. Ganguli, T. Henighan, A. Jones, N. Joseph, B. Mann, N. DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 , 2021. URL https://arxiv.org/pdf/2112.00861 .

Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022a. URL https://arxiv.org/pdf/ 2204.05862 .

Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Chen, C. Olsson, C. Olah, D. Hernandez, D. Drain, D. Ganguli, D. Li, E. Tran-Johnson, E. Perez, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse, K. Lukosuite, L. Lovitt, M. Sellitto, N. Elhage, N. Schiefer, N. Mercado, N. DasSarma, R. Lasenby, R. Larson, S. Ringer, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Lanham, T. Telleen-Lawton, T. Conerly, T. Henighan, T. Hume, S. R. Bowman, Z. Hatfield-Dodds, B. Mann, D. Amodei, N. Joseph, S. McCandlish, T. Brown, and J. Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022b. URL https://arxiv.org/pdf/2212.08073.pdf .

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021. URL https://arxiv.org/abs/2110.14168 .

Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 8493-8502, 2022. URL https://aclanthology.org/2022. acl-long.581.pdf .

Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y. Wang, and Y. Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id=TyFrPOKYXw .

Duan, X. Yi, P. Zhang, T. Lu, X. Xie, and N. Gu. Negating negatives: Alignment without human positive samples via distributional dispreference optimization. arXiv preprint arXiv:2403.03419 , 2024. URL https://arxiv.org/pdf/2403.03419 .

Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. URL https://transformer-circuits.pub/2021/framework/index.html .

Elhage, T. Hume, C. Olsson, N. Schiefer, T. Henighan, S. Kravec, Z. Hatfield-Dodds, R. Lasenby, D. Drain, C. Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652 , 2022. URL https://arxiv.org/pdf/2209.10652 .

Ethayarajh, Y. Choi, and S. Swayamdipta. Understanding dataset difficulty with V -usable information. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 5988-6008. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/ethayarajh22a.html .

Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 , 2024. URL https://arxiv.org/pdf/ 2402.01306 .

Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 , 2022. URL https://arxiv.org/pdf/ 2209.07858.pdf .

Geiger, Z. Wu, C. Potts, T. Icard, and N. Goodman. Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learning and Reasoning , pages 160-187. PMLR, 2024. URL https://proceedings.mlr.press/v236/geiger24a/ geiger24a.pdf .

Geva, R. Schuster, J. Berant, and O. Levy. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5484-5495, 2021. URL https://aclanthology.org/2021.emnlp-main. 446.pdf .

Gurnee, N. Nanda, M. Pauly, K. Harvey, D. Troitskii, and D. Bertsimas. Finding neurons in a haystack: Case studies with sparse probing. Transactions on Machine Learning Research , 2023. URL https://openreview.net/pdf?id=JYs1R9IMJr .

Gurnee, T. Horsley, Z. C. Guo, T. R. Kheirkhah, Q. Sun, W. Hathaway, N. Nanda, and D. Bertsimas. Universal neurons in gpt2 language models. arXiv preprint arXiv:2401.12181 , 2024. URL https://arxiv.org/pdf/2401.12181 .

Hanna, O. Liu, and A. Variengien. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems , 36, 2024. URL https://openreview.net/pdf?id=p4PckNQR8k .

Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR) , 2021. URL https://openreview.net/pdf?id=d7KBjmI3GmQ .

Højer, O. S. Jarvis, and S. Heinrich. Improving reasoning performance in large language models via representation engineering. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/pdf?id=IssPhpUsKt .

C.-Y. Hsu, Y.-L. Tsai, C.-H. Lin, P.-Y. Chen, C.-M. Yu, and C.-Y. Huang. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405.16833 , 2024. URL https://arxiv.org/pdf/2405.16833 .

Huang, S. Gupta, M. Xia, K. Li, and D. Chen. Catastrophic jailbreak of open-source llms via exploiting generation. In The Twelfth International Conference on Learning Representations , 2023. URL https://openreview.net/pdf?id=r42tSSCHPh .

Inan, K. Upasani, J. Chi, R. Rungta, K. Iyer, Y . Mao, M. Tontchev, Q. Hu, B. Fuller, D. Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 , 2023. URL https://arxiv.org/pdf/2312.06674 .

Ivison, Y. Wang, V. Pyatkin, N. Lambert, M. Peters, P. Dasigi, J. Jang, D. Wadden, N. A. Smith, I. Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2. arXiv preprint arXiv:2311.10702 , 2023. URL https://arxiv.org/pdf/2311.10702.pdf .

Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, B. Chen, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a humanpreference dataset. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ 4dbb61cb68671edc4ca3712d70083b9f-Paper-Datasets_and_Benchmarks.pdf .

Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. URL https://arxiv.org/pdf/2310. 06825.pdf .

Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 , 2024. URL https://arxiv.org/pdf/2401.04088.pdf .

Kong, H. Wang, W. Mu, Y. Du, Y. Zhuang, Y. Zhou, Y. Song, R. Zhang, K. Wang, and C. Zhang. Aligning large language models with representation editing: A control perspective. arXiv preprint arXiv:2406.05954 , 2024. URL https://arxiv.org/pdf/2406.05954 .

Lambert, L. Tunstall, N. Rajani, and T. Thrush. Huggingface h4 stack exchange preference dataset, 2023. URL https://huggingface.co/datasets/HuggingFaceH4/ stack-exchange-preferences .

Lambert, V. Pyatkin, J. Morrison, L. Miranda, B. Y. Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y. Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787 , 2024. URL https://arxiv.org/pdf/2403.13787 .

Lee, X. Bai, I. Pres, M. Wattenberg, J. K. Kummerfeld, and R. Mihalcea. A mechanistic understanding of alignment algorithms: A case study on dpo and toxicity. arXiv preprint arXiv:2401.01967 , 2024. URL https://arxiv.org/pdf/2401.01967 .

Li, L. Yao, L. Zhang, and Y. Li. Safety layers in aligned large language models: The key to llm security. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/pdf?id=kUH1yPMAn7 .

Li, S. Han, and S. Ji. Vb-lora: Extreme parameter efficient fine-tuning with vector banks. arXiv preprint arXiv:2405.15179 , 2024. URL https://arxiv.org/pdf/2405.15179 .

Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 3214-3252, 2022. URL https://aclanthology.org/2022. acl-long.229.pdf .

Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ 0cde695b83bd186c1fd456302888454c-Paper-Conference.pdf .

Liu, X. Wang, M. Wu, T. Li, C. Lv, Z. Ling, J. Zhu, C. Zhang, X. Zheng, and X. Huang. Aligning large language models with human preferences through representation engineering. arXiv preprint arXiv:2312.15997 , 2023. URL https://arxiv.org/pdf/2312.15997 .

Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/peft , 2022.

Mazeika, L. Phan, X. Yin, A. Zou, Z. Wang, N. Mu, E. Sakhaee, N. Li, S. Basart, B. Li, et al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249 , 2024. URL https://arxiv.org/pdf/2402.04249.pdf .

McGrath, M. Rahtz, J. Kramar, V. Mikulik, and S. Legg. The hydra effect: Emergent self-repair in language model computations. arXiv preprint arXiv:2307.15771 , 2023.

Meng, D. Bau, A. Andonian, and Y. Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems , 36, 2022. URL https://arxiv.org/pdf/ 2202.05262.pdf . arXiv:2202.05262.

Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models, 2016. URL https://arxiv.org/pdf/1609.07843.pdf .

Nanda and J. Bloom. Transformerlens. https://github.com/TransformerLensOrg/ TransformerLens , 2022.

Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:2773027744, 2022. URL https://papers.neurips.cc/paper_files/paper/2022/file/ b1efde53be364a73914f58805a001731-Paper-Conference.pdf .

Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12:2825-2830, 2011. URL https://jmlr.csail.mit.edu/papers/volume12/ pedregosa11a/pedregosa11a.pdf .

Qi, H. Peng, X. Wang, B. Xu, L. Hou, and J. Li. Adelie: Aligning large language models on information extraction. arXiv preprint arXiv:2405.05008 , 2024. URL https://arxiv.org/ pdf/2405.05008 .

Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report, 2025. URL https://arxiv.org/abs/2412.15115 .

Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_files/ paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf .

Shen, Z. Chen, M. Backes, Y. Shen, and Y. Zhang. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825 , 2023. URL https://arxiv.org/pdf/2308.03825.pdf .

Stolfo, B. Wu, W. Gurnee, Y. Belinkov, X. Song, M. Sachan, and N. Nanda. Confidence regulation neurons in language models. Advances in Neural Information Processing Systems , 37:125019-125049, 2024. URL https://proceedings.neurips.cc/paper_files/paper/ 2024/file/e21955c93dede886af1d0d362c756757-Paper-Conference.pdf .

Sun, Z. Liu, A. Bair, and J. Z. Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/pdf?id=PxoFut3dWW .

Sun, D. Gupta, and M. Iyyer. Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of rlhf. arXiv preprint arXiv:2309.09055 , 2023. URL https: //arxiv.org/pdf/2309.09055 .

Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-bench tasks and whether chain-of-thought can solve them. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023 , pages 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.824. URL https: //aclanthology.org/2023.findings-acl.824 .

Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023. URL https://arxiv.org/pdf/2312.11805.pdf .

Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, and et al. Gemma: Open models based on gemini research and technology, 2024. URL https://arxiv.org/pdf/2403.08295.pdf .

Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023. URL https://arxiv.org/pdf/2307.09288.pdf .

Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 , 2023. URL https://arxiv.org/pdf/2310.16944.pdf .

Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf .

Vig, S. Gehrmann, Y. Belinkov, S. Qian, D. Nevo, Y. Singer, and S. Shieber. Investigating gender bias in language models using causal mediation analysis. Advances in neural information processing systems , 33:12388-12401, 2020. URL https://proceedings.neurips.cc/paper/ 2020/file/92650b2e92217715fe312e6fa7b90d82-Paper.pdf .

von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, and S. Huang. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl , 2020.

R. Wang, A. Variengien, A. Conmy, B. Shlegeris, and J. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In The Eleventh International Conference on Learning Representations , 2022a. URL https://openreview.net/pdf?id=NpsVSN6o4ul .

Wang, K. Wen, Z. Zhang, L. Hou, Z. Liu, and J. Li. Finding skill neurons in pre-trained transformerbased language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 11132-11152, 2022b. URL https://aclanthology.org/2022. emnlp-main.765.pdf .

Wang, H. Ivison, P. Dasigi, J. Hessel, T. Khot, K. Chandu, D. Wadden, K. MacMillan, N. A. Smith, I. Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ ec6413875e4ab08d7bc4d8e225263398-Paper-Datasets_and_Benchmarks.pdf .

Wei, K. Huang, Y. Huang, T. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and P. Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/pdf?id= K6xxnKN2gm .

Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 38-45, Online, Oct. 2020. Association for Computational Linguistics. URL https://www.aclweb.org/anthology/2020.emnlp-demos.6 .

Yang, X. Wang, Q. Zhang, L. Petzold, W. Y. Wang, X. Zhao, and D. Lin. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949 , 2023. URL https://arxiv.org/pdf/2310.02949 .

Yang, F. Sondej, H. Mayne, and A. Mahdi. Ablation is not enough to emulate dpo: How neuron dynamics drive toxicity reduction. arXiv preprint arXiv:2411.06424 , 2024. URL https: //arxiv.org/abs/2411.06424 .

Zhao, W. Zhang, Y. Xie, A. Goyal, K. Kawaguchi, and M. Shieh. Understanding and enhancing safety mechanisms of LLMs via safety-specific neuron. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/forum?id=yR47RmND1m .

Zheng, F. Yin, H. Zhou, F. Meng, J. Zhou, K.-W. Chang, M. Huang, and N. Peng. On promptdriven safeguarding for large language models. In Forty-first International Conference on Machine Learning , 2024. URL https://openreview.net/pdf?id=ugxGpOEkox .

Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems , 36, 2024. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ ac662d74829e4407ce1d126477f4a03a-Paper-Conference.pdf .

Zhou, H. Yu, X. Zhang, R. Xu, F. Huang, K. Wang, Y. Liu, J. Fang, and Y. Li. On the role of attention heads in large language model safety. In The Thirteenth International Conference on Learning Representations , 2025. URL https://openreview.net/pdf?id=h0Ak8A5yqw .

Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, et al. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405 , 2023. URL https://arxiv.org/pdf/2310.01405.pdf .

Zou, L. Phan, J. Wang, D. Duenas, M. Lin, M. Andriushchenko, J. Z. Kolter, M. Fredrikson, and D. Hendrycks. Improving alignment and robustness with circuit breakers. Advances in Neural Information Processing Systems , 37:83345-83373, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2024/file/ 97ca7168c2c333df5ea61ece3b3276e1-Paper-Conference.pdf .

Appendices

A Details about Used Dataset

A.1 Supervised Fine-Tuning Data

ShareGPT (Chiang et al., 2023) is a decently large dataset of realistic human-AI conversations. We leverage the processed version used in training Tülu (Wang et al., 2024).

A.2 Preference Data

HH-RLHF (Bai et al., 2022a) contains open-ended conversations with provided models, which ask for help, advice, or for the model to accomplish a task and choose the more helpful model response ( HH-Helpful ), or attempt to elicit harmful responses from their models, and to choose the more harmful response offered by the models ( HH-Harmless ).

RewardBench (Lambert et al., 2024) is a collection of prompt-win-lose trios spanning chat, reasoning, and safety. We use the safety ( RewardBench-Safety ) and reasoning ( RewardBenchReasoning ) subsets in our preference learning.

Stanford Human Preferences (Ethayarajh et al., 2022) is a dataset of 385K collective human preferences over responses to questions/instructions in 18 different subject areas, from cooking to legal advice.

H4 Stack Exchange Preferences (Lambert et al., 2023) contains questions and answers from the Stack Overflow Data Dump for the purpose of preference model training.

IEFeedback (Qi et al., 2024) is a preference dataset constructed using ADELIESFT proposed in their paper to boost the model performance on information extraction (IE).

A.3 Evaluation Benchmarks

Beavertails (Ji et al., 2024) contains QA pairs between human and AI assistants with humanpreference annotations separately for the helpfulness and harmlessness metrics of the responses. We only use the question parts for safety evaluation since we find training on it results in an unsafe model.

RedTeam (Ganguli et al., 2022) contains human-generated red-teaming prompts.

HarmBench (Mazeika et al., 2024) consists of a set of harmful behaviors which includes 7 semantic categories of behavior and 4 functional categories of behavior. We exclude the multimodal behaviors since our models are text-only.

JailbreakLLMs (Shen et al., 2023) contains high-quality jailbreak prompts collected from four platforms over six months.

LIMA (Zhou et al., 2024) consists of around 1000 carefully curated prompts and responses, which aim to enhance the helpfulness of LLMs.

Wikitext-2 (Merity et al., 2016) is a collection of over 100 million tokens extracted from the set of verified good and featured articles on Wikipedia.

TruthfulQA (Lin et al., 2022) is a benchmark to measure whether a language model is truthful in generating answers to questions.

GSM8K (Grade School Math 8K, Cobbe et al., 2021) is a dataset of 8.5K high-quality linguistically diverse grade school math word problems.

MMLU(Massive Multitask Language Understanding, Hendrycks et al., 2021) is a massive multitask test consisting of multiple-choice questions from various branches of knowledge.

BBH (BIG Bench Hard, Suzgun et al., 2023) is a subset of BIG Bench dataset and consists of 23 tasks that are particularly hard for the current generation of language models.

The detailed data statistics are shown in Table 4.

B Implementations Details

B.1 Safety Alignment

SFT Training Details We use Huggingface's transformers (Wolf et al., 2020) and peft (Mangrulkar et al., 2022) libraries to train our SFT model on ShareGPT with a max length of 4096 tokens. The training hyperparameters are shown in Table 5 (We find (IA) 3 needs a much higher learning rate compared to LoRA). The detailed hyperparameters of LLMs we used are listed in Table 6.

DPO Training Details We use Huggingface's trl (von Werra et al., 2020) library to train our DPO models. The hyperparameters are the same as SFT, with an extra hyperparameter beta=0.1 for DPO.

Details of (IA) 3 Short for Invertible Adapters with Activation Alignment (Liu et al., 2022), (IA) 3 is a fine-tuning method designed for large neural networks that achieves efficiency by focusing

on a small number of trainable parameters while preserving the original model's capacity. In our framework, we only apply (IA) 3 to MLP as follows:

where l ff ∈ R d m is the trainable parameters.

B.2 Evaluation Details

For the safety evaluation benchmarks used in our study, we sampled 200 examples from each test set for evaluation. To ensure experimental stability, we employed a greedy search strategy for generation, with the max new tokens set to 128 for generation speed. Examples of responses are shown in Table 7.

For general capabilities, we evaluate perplexity on the full test set of Wikitext-2 with a maximum length of 4096 and follow the evaluation settings outlined in Wang et al. (2024) for other benchmarks. Specifically, for MMLU, we use the entire test set and employ 0-shot prompting without Chain of Thought (CoT), selecting the option with the highest probability as the predicted choice, rather than using the model to generate the response directly. This approach differs from the method used in the official technical reports of these models, leading to some discrepancies in the results. For BBH, we sampled 40 samples from each task for testing and used a 3-shot CoT. For GSM8K, we sampled 200 samples using 8-shot CoT. For TruthfulQA, we utilize the official evaluation script, testing on the entire test set with the MC1 metric as proposed in Lin et al. (2022). The sampling strategy is the same as described before.

We run all the above experiments on NVIDIA A100-SXM4-80GB GPU, and it takes about 1,000 GPU hours.

B.3 Finding Safety Neurons

We build our code on TransformerLens (Nanda and Bloom, 2022) to cache neuron activations and perform dynamic activation patching. For each prompt dataset, we use 200 randomly sampled prompts (no overlap with evaluation data). Again, we use greedy search for generation and set the max new tokens to 256, resulting in around 40,000 activations for each neuron. We describe our dynamic activation patching method in Algorithm 1.

For the two ablation-based baselines, we referred to their official code and implemented our own version. For Wei et al. (2024), we used the Wanda score (Sun et al., 2024) as a parameter importance indicator to locate the top 5% of parameters in the down_proj matrix of each MLP layer in the DPO model, using the HH-Harmless dataset. For Zhao et al. (2025), we used the circuit breakers (Zou et al., 2025) training set, as described in their paper, to locate safety neurons, again selecting the top 5% as a comparison.

B.4 Harmful Content Prediction

We collect neuron activations on the training set of HH-harmless, the test set of Beavertails, RedTeam, Harmbench, and JailbreakLLMs. We use greedy search with max new tokens set to 128 to get generations and assign the label 1 if the cost score of generation is positive. The classifier is LogisticRegression in scikit-learn (Pedregosa et al., 2011) with default hyperparameters.

Algorithm 1 Dynamic Activation Patching

Input:

w the prompt text N a set contains (layer, neurons) pairs M 1 the model being patched M 2 the model used for patching Output: w ′ the completed text Initialization: w ′ ← w l ←M 1 .num_layers repeat cache ←M 2 .run_with_cache ( w ′ ) {Cache neuron activation} x ←M 1 .Embed ( w ′ ) for i ← 1 to l do x ← x + M 1 .Attn[i] ( x ) if i in N then x ← x + M 1 .PatchedMLP[i] ( x, cache, N [ i ]) {Patch neurons} else x ← x + M 1 .MLP[i] ( x ) end if end for p ←M 1 .lm_head ( x )[ -1] .softmax() token ← Sample ( p ) {Get next token prediction} w ′ ← Concat ( w ′ , token ) until StopCriterion ( w ′ ) is true

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.10

Change Score

0

500

1000

1500

2000

2500

Count

Llama2

Mistral

Gemma

Qwen2.5

(a)

5000

10000 15000 20000

Top K Neurons

0.0

0.2

0.4

0.6

0.8

1.0

Llama2

Mistral

Gemma

Qwen2.5

(b)

C More Properties of Safety Neurons

C.1 Layer Distribution

The layer distribution of the top 20 , 000 safety neurons is shown in Figure 5b. Llama2-7b and Mistral-7b have similar patterns: safety neurons are distributed across many layers, predominantly appearing in the deep layers, with a gradual shift towards the middle layers as change scores decrease. Conversely, Gemma-7b presents a starkly different distribution, with safety neurons primarily found in the initial and final layers. Notably, the most significant neurons in Gemma-7b are located in shallower layers, progressively transitioning to deeper layers with a more uniform distribution as change scores decrease. This phenomenon is likely due to significant architectural differences between Gemma-7b and the other two models (Table 6).

0

5

10

15

20

25

30

Neurons (%)

0.0

0.2

0.4

0.6

Causal Effect

0

5

10

15

20

25

30

Neurons (%)

0.0

0.2

0.4

0.6

0.8

1.0

Patch SFT with DPO

Llama2

Mistral

Gemma

C.2 Change Score Distribution

We visualize the change scores distribution of top 20 , 000 safety neurons in Figure 5a. We first notice that only a small fraction of neurons changed much after safety alignment (for Llama2-7b only 876 out of 341248 neurons with a change score larger than 0 . 1 ). More interestingly, these three different models have similar patterns and thresholds at around 0 . 035 for safety neurons. Furthermore, we find that models performing better in safety alignment exhibit longer tails 2 , indicating that improved model performance may result from more neurons experiencing significant activation changes. We leave the further investigation of this phenomenon for future work.

C.3 Effectiveness of Change Score

We futher conducted experiments to validate whether the change score serves as an appropriate indicator of a neuron's causal effect on generation. Specifically, we utilized consecutive sets of 5% of neurons, starting from various ranks. As shown in Figure 6, we observed that as the change scores of the neurons decreased, the effectiveness of dynamic activation patching rapidly diminished. This finding indicates that only neurons with high change scores exert a significant causal effect on the model's output. Consequently, we selected the top 5% of neurons with the highest change scores as the safety neurons for further investigation in subsequent experiments.

C.4 Specificity on Different Datasets

We simply use safety neurons found on HH-Harmless in previous experiments. Now we take a closer look at the prompt dataset selection. We use datasets from 3 different preference learning tasks: (1) Safety , including Beavertails (Ji et al., 2024), HH-Harmless (Bai et al., 2022a), and JailBreakLLMs (Shen et al., 2023); (2) Helpfulness , including HH-Harmless (Bai et al., 2022a) and LIMA (Zhou et al., 2024); (3) Reasoning , including the Reasoning subset from RewardBench (Lambert et al., 2024). We repeat the experiments from § 4.1 using safety neurons found on these prompts, as shown in Figure 7. The results indicate that safety neuron activations are specific to certain inputs, i.e., safety neurons found on similar types of prompts exhibit similar causal effects and are most effective on safety-related prompts.

D Other Design Choices for Neuron-Finding

After safety alignment, we obtained three distinct models: Base , SFT , and DPO . In previous experiments, we simply utilize the generation from SFT to compare neuron activations between SFT and DPO to identify safety neurons. Here we discuss some possible design choices of our method.

0.0

0.2

0.4

0.6

Causal Effect

0.0

0.2

0.4

0.6

0.8

1.0

Patch SFT with DPO (Llama2)

0.0

0.2

0.4

0.6

Causal Effect

0.0

0.2

0.4

0.6

0.8

1.0

Patch SFT with DPO (Mistral)

0

1

2

3

4

5

Neurons (%)

0.0

0.2

0.4

0.6

Causal Effect

0

1

2

3

4

5

Neurons (%)

0.25

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Gemma)

Beavertails

HH-Harmless

HH-Helpful

LIMA

Reasoning

D.1 Which Model Should be Compared?

We explore the impact of comparing different models and different generations. We replicate the experiments from § 4.1 with different design choices, and the results are depicted in Figure 8. These results indicate that there is no fundamental difference among the models chosen for comparison within our framework. However, the neurons identified by comparing SFT and DPO perform slightly better, which may be attributed to the minimal functional discrepancies between them, providing a clearer signal for identifying safety neurons.

D.2 Which Token Position Should be Compared?

Previous studies typically investigated neuron activations at prompt tokens (Zou et al., 2023). We employed these activations to identify safety neurons for comparison. The results in Figure 9 indicate that safety neurons identified using inference-time activations yield more stable performance. However, Gemma-7b exhibits an unexpected behavior possibly due to the significantly different model architecture. We leave the investigation for the impact of model architectures on neuron-finding in future research.

0.0

0.2

0.4

0.6

0.8

Causal Effect

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Llama2)

0.0

0.2

0.4

0.6

0.8

Causal Effect

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Mistral)

0

1

2

3

4

5

Neurons (%)

0.0

0.2

0.4

0.6

Causal Effect

0

1

2

3

4

5

Neurons (%)

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Gemma)

Compared Models

Base and DPO

SFT and DPO

Generation From

First Model

0.0

0.2

0.4

Causal Effect

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Llama2)

0.0

0.2

0.4

0.6

Causal Effect

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Mistral)

0

1

2

3

4

5

Neurons (%)

0.0

0.2

0.4

0.6

Causal Effect

0

1

2

3

4

5

Neurons (%)

0.00

0.25

0.50

0.75

1.00

Patch SFT with DPO (Gemma)

Token Position

Prompt (all tokens)

Prompt (last token)

Safety Neurons

E More Experimental Results

E.1 Correlation between GPT-4 Scores and Cost Scores

Evaluation with GPT-4 (Achiam et al., 2023) is also a widely accepted metric (Liu et al., 2023; Dai et al., 2024). We leverage gpt-4-turbo-2024-04-09 to assign scores for the same generations from LLMs. The correlation between GPT-4 scores and cost scores is shown in Figure 10a. We find there is a strong negative correlation between these two scores (-0.77), which indicates cost score is an appropriate metric for safety evaluation. The prompt and response of GPT-4 are demonstrated in Table 7.

E.2 Evaluation of Aligned Models

The average cost scores of our SFT and DPO models on Beavertails can be found in Table 8. Firstly, we noticed the models that have better performance in reports also perform better in safety alignment. Secondly, we find although SFT exhibit safety behaviors on average (due to the safety

20

10

0

10

20

30

40

Cost Score

0

2

4

6

8

10

12

GPT4 Score

(a)

50

300

450

600 1200 2000 2500 3000

#Neurons

0.625

0.650

0.675

0.700

0.725

0.750

0.775

Safety Neuron

RN-Same Distribution

RN-Last

RN-All

(b)

responses in ShareGPT ), they are still vulnerable compared to DPO models. Thirdly, even if (IA) 3 use only 0.005% parameters compared to full fine-tuning, it achieves relatively strong results in safety alignment (as a comparison, Llama2-7b-chat scores -13 . 97 ).

E.3 More Safety Neuron Results

In Table 8, we present the complete results of the cost scores and general benchmark performance.

In Table 9, we present the complete results of the top safety neurons' value vectors projected into the vocabulary space.

E.4 More Alignment Tax Results

Spearman's rank correlation coefficients between different preference neurons of Mistral-7b and Gemma-7b are shown in Figure 11. For Mistral-7b , we observe results similar to Llama2-7b . However, Gemma-7b shows anomalies when aligned on RewardBench-Safety, which we attribute to the small dataset size (less than 1k samples) compared to the larger number of neurons Gemma-7b . This discrepancy likely leads to insufficient training. However, this discrepancy does not affect our explanation of the alignment tax (Table 3).

Our experimental results indicate that when performing either safety or helpfulness alignment individually, safety neurons can be leveraged to improve both model safety and helpfulness. However, when both alignments are applied simultaneously, it remains unclear whether conflicts arise within these neurons. To investigate this, we conducted the following experiment on Llama2 :

We used DPO to train two models based on the same SFT model: one trained solely on HH-helpful data ( Helpful DPO ) and another trained on both HH-harmless and HH-helpful data ( HH DPO ).

We patched 5% of neuron activations from HH DPO into Helpful DPO , specifically using the neurons identified from Helpful DPO for model helpfulness. These neurons were selected following the same methodology used for identifying safety neurons in our study.

The results (Table 10) indicate that the neurons identified as helpful neurons are also crucial for improving model safety during HH training. This suggests that safety and helpfulness compete for control over these neurons during HH training, providing further mechanistic insights into the alignment tax phenomenon.

E.5 More Safeguard Results

Data Construction We cache neuron activations at the last token of the prompt and create labels for these activations by the cost scores of corresponding generation (we use a threshold of 0 to distinguish whether the generation is harmful or not) on 5 datasets: HH-Harmless (Bai et al., 2022a), Beavertails (Ji et al., 2024), RedTeam (Ganguli et al., 2022), HarmBench (Mazeika et al., 2024), and JailBreakLLMs (Shen et al., 2023).

Experiment To validate the generalization ability of these neuron activations, we use activations from one dataset as the training set and merge the others as the test set, training a simple logistic regression classifier. Finally, we compute the average accuracy across all possible combinations as the evaluation metric. In addition to safety neurons, we employ neurons identified through other

strategies as baselines, including (1) RN-Same Distribution , which refers to randomly sampled neurons (completely disjoint from safety neurons) with the same per-layer neuron count as the safety neurons; (2) RN-Last , which denotes neurons randomly sampled exclusively from the last layer, based on the hypothesis that neurons in the last layer directly influence the model's output, making this a potentially strong baseline; (3) RN-All , which refers to neurons randomly sampled without constraints, aiming to assess whether the layer-wise distribution of safety neurons inherently encodes safety-related information. For all experiments requiring randomly sampled neurons, we repeat the process 5 times using different random seeds and report the averaged results.

Result We train and test the classifier using activations from different numbers of neurons, as shown in Figure 10b. The results indicate that the test accuracy almost converges when using activations from approximately 1500 neurons, while activations from as few as 150 neurons yield relatively decent results across all test sets. These results suggest that the activations of safety neurons indeed

Vector

Top Tokens

MLP.v

30

10106

ouc,

iter,

trat,

ussen,

tid,

imos

MLP.v

29

8343


,

Genomsnittlig,

text,




MLP.v

28

5293

Sug,

Commons, sugar,

mouth,

flesh

MLP.v

30

3527


,

\n,

\r,

→,

="@+,

{:,

\f

MLP.v

30

4427

and,

\n,


,

&,

this,

with,

vs

MLP.v

26

7581

wa,

ales,

sin,

MainActivity,

oblig

MLP.v

29

9647

Food,

Guard, Farm,

Sex,

Break

MLP.v

30

10075

*/\r,

*/,

),

',

},

},

»,

}\r

MLP.v

28

4127

**,

».***, °,

'',

--,

/,

!!, ]

MLP.v

30

7219

Ż,

Gemeinsame, HT,

Gor,

category

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

0.6

0.7

0.8

0.9

1.0

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

Harmless

Safety

Helpful

SHP

H4SE

Reasoning

IEFeedback

0.6

0.7

0.8

0.9

1.0

(b) Gemma-7b

encode more information about the safety of the model's outputs, and this information is transferable across different datasets. Additionally, random neurons with the same layer distribution as safety neurons are more effective than those sampled from other layers, which indicates the layer distribution of safety neurons may also encode safety information.

F Limitations

Our research has some limitations. First, although safety neurons can enhance the safety of unaligned models, this requires neuron activations from already aligned models. Exploring training-free methods to obtain these activations is an interesting research direction. Second, we used (IA) 3

for alignment, but real-world models often undergo full parameter fine-tuning. The impact of this on safety neurons is unknown, though previous study suggests that during DPO alignment, many toxicity-related neuron parameters remain largely unchanged, with DPO primarily suppressing the activations of these neurons (Lee et al., 2024). Finally, we identified which neurons affect model safety but not how they exert this influence, which will be a future research direction.

G Broader Impacts

This work contributes to the understanding of safety alignment in LLMs, potentially guiding the development of safer and more controllable models. By providing insights into the mechanisms behind safety neurons, our research may help improve alignment techniques and safeguard applications against generating harmful content. However, these findings could also be misused to weaken alignment constraints, enabling more covert adversarial attacks. Additionally, the reliance on specific datasets and evaluation methods may introduce biases that impact generalizability. To mitigate these risks, we adhere to open-source licensing policies and commit to sharing our code and data to foster transparency and responsible research.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The introduction clearly states the paper's contributions, including the causal effectiveness of safety neurons, the mechanistic interpretation of alignment tax, and the safeguard application by safety neurons.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitation of this work in Appendix F.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not present any theoretical results or formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experiments are based on public models and benchmarks, and all evaluation procedures are described in sufficient detail for replication. Additional details are included in the supplemental material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and scripts to reproduce the main experiments have been uploaded, along with instructions for usage and data preparation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All the implementations details can be found in Appendix A and B, including dataset statistics, hyperparameters, type of optimizer, etc.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the T-statistics and 95% confidence of our main results in § 4.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We report our experiment environment and GPU hours used in B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper uses publicly available models and datasets and adheres to ethical principles as outlined in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in Appendix G.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release new model or dataset.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All models and datasets used in the study are cited appropriately. License information and usage terms are followed and documented in the appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not introduce new datasets, models, or other assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve any crowdsourcing or research with human participants.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The research does not involve any human subjects, so IRB approval is not applicable.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: We only use LLM for improving writing.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for identifying specific MLP neurons in the context of the safety alignment of large language models (LLMs). Specifically two complementary techniques are presented\n- Inference-time activation contrasting. This technique identifies neurons by comparing their activation patterns between pre- and post-safety-finetuned model checkpoints.\n- Dynamic activation patching. This technique employs causal interventions to quantify the extent to which the identified neurons are responsible for the model's safety behaviors.\n\nThese findings suggest a pathway toward more controlled and robust alignment of LLMs with human values and safety requirements.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- This paper is well-written organized. I think it is easy to follow.\n\n- The motivations behind this paper is well-explained, and preliminaries are sufficiently presented before presenting the key techniques.\n\n- The key findings may be very helpful for enhancing the safety of future LLMs. Given the growing importance of safety in AI, this paper can be impactful in AI community.\n\n- The authors tested their method on a variety of model families and datasets. I appreciate the authors' attitude to show that the key findings are applicable not only specific experimental settings but as a general knowledge.\n\n### Weaknesses\n\n- While the supplementary material is submitted, the source code is empty (I'm sorry if this is an issue on my side). I am wondering whether I should trust the experimental results.\n\n- In lines 331-343, while relevant studies are cited, I do not think that the related work is sufficiently discussed. I recommend discussing specifically what is essentially different, rather than discussing superficially.\n\nQUESTIONS:\n[Q1] Could you tell me the differences from the following existing work on task vector? What is the merit of focusing on neurons?\n\n- Ilharco, Gabriel, et al. \"Editing models with task arithmetic.\" arXiv preprint arXiv:2212.04089 (2022). \n- Bhardwaj, Rishabh, Do Duc Anh, and Soujanya Poria. \"Language models are Homer simpson! safety re-alignment of fine-tuned language models through task arithmetic.\" arXiv preprint arXiv:2402.11746 (2024).\n\nLIMITATIONS:\nLimitations have been fully discussed in Appendix F.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the mechanisms behind safety alignment in large language models by identifying \"safety neurons\" responsible for safe behaviors using mechanistic interpretability. The authors introduce inference-time activation contrasting to locate these neurons and dynamic activation patching to assess their \"causal impact\" on model safety. Experiments show that patching around 5% of identified neurons restores over 90% of safety performance on red-teaming benchmarks without compromising general ability.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n- 1 Studying the mechanistic interpretability of LLMs' safety is indeed a very interesting and significant problem.\n- 2 The proposed method that identifies the \"safety neurons\" seems reasonable\n- 3 The effectiveness of the proposed method has been tested over multiple LLMs and Benchmark datasets. Also the method provides some understanding of \"alignment tax\", though not so rigorously tested.\n\nCons:\n\n- 1  Some technical content is difficult to understand. How and why Dynamic Activation Patching works needs further explanation.  Eg.g., In what way does such a method link to the actual effect? \n- 2 The Mechanistic Explanation in this paper seems insufficient.  While it successfully identifies which neurons are critical and reports their sparsity and so on, it fails to provide an explanation for how these neurons propagate a \"safety signal\" through the network, e.g., by checking its semantic meaning.\n- 3 The proposed method seems analogous to traditional model compression techniques in keeping a few nodes/weights in the neuron networks. From my understanding, it is a kind of model compression and a bit exaggerated to claim this is related to the \"safety\" mechanism of LLM.\n\nQUESTIONS:\nSee cons.\n\nLIMITATIONS:\nSee cons.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates mechanistic roots of safety alignment in large language models by pinpointing a sparse set of “safety neurons” whose activations correlate with—and can causally induce—safe behaviour. It introduces a two-stage framework: inference-time activation contrasting ranks neurons whose activations differ between an unaligned and a safety-aligned model, and dynamic activation patching selectively substitutes those activations to verify that they alone can drive safer outputs.\n\nSTRENGTHS AND WEAKNESSES:\n#### **Strengths**\n\n- Clear writing, diagrams make the paper easy to follow.\n- Two-stage “activation contrasting => patching” pipeline is well motivated and causally validated.\n- Provides a neuron-level explanation of the alignment tax\n\n#### **Weaknesses**\n\n- All evaluations are on ≤ 7 B models; scalability to larger frontier LLMs is untested.\n- Technique builds on existing activation-patching ideas; novelty is incremental and overlaps with concurrent studies on nuerons[1,2].\n\n\n\n[1] Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., ... & Hendrycks, D. (2023). Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.\n\n[2] O'Brien, K., Majercak, D., Fernandes, X., Edgar, R., Bullwinkel, B., Chen, J., ... & Poursabzi-Sangde, F. (2024). Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296.\n\nQUESTIONS:\n1. **Incremental Contribution** – What concrete contribution does this paper provide beyond prior neuron-steering work such as Representation Engineering [1], Sparse-Autoencoder Refusal Control [2], and Circuit Breakers [3]?\n2. **Sparsity Stability** – The study reports that roughly 5 % of neurons govern safety. Does this fraction remain stable as model size scales (e.g., 2 B → 7 B → 70 B), or does the sparsity pattern shift?\n3. **Robustness to Jailbreaks** – Do the identified safety neurons still trigger under stronger or more diverse jailbreak attacks (e.g., rewrite the harmful query in a benign way or use PAIR[4])?\n4. **Activation Timing** – At which token positions are the safety neurons most strongly activated? Are they primarily triggered by system/chat-template tokens (e.g., `assistant\\n`) before generation starts, or do they also fire in response to tokens within harmful prompts (and will the pattern be related to the length of the harmful queries)?\n5. **Safeguard Rationale** – The safeguard is trained on HH-Harmless SFT activations yet evaluated on both SFT and DPO models that already refuse harmful queries when these neurons fire. What additional safety benefit does the safeguard offer in this setting?\n6. **Practical Patching** – In a real deployment, we rarely know a priori which queries are harmful. How would the proposed patching procedure be used to make an *unaligned* base model safer?\n\n\n\n[1] Zou, A., Phan, L., Chen, S., Campbell, J., Guo, P., Ren, R., ... & Hendrycks, D. (2023). Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.\n\n[2] O'Brien, K., Majercak, D., Fernandes, X., Edgar, R., Bullwinkel, B., Chen, J., ... & Poursabzi-Sangde, F. (2024). Steering language model refusal with sparse autoencoders. arXiv preprint arXiv:2411.11296.\n\n[3] Zou, A., Phan, L., Wang, J., Duenas, D., Lin, M., Andriushchenko, M., ... & Hendrycks, D. (2024, June). Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n\n[4] Chao, P., Robey, A., Dobriban, E., Hassani, H., Pappas, G. J., & Wong, E. (2025, April). Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) (pp. 23-42). IEEE.\n\nLIMITATIONS:\nsee questions", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a mechanistic interpretability framework to undestand how safety alignment emerges LLMs by identifying and analyzing “safety neurons” — a small, sparse subset of MLP neurons whose activations causally drive safer outputs without degrading general capability. Using inference-time activation contrasting to rank candidate neurons and dynamic activation patching to verify causality, the authors show that activating only these neurons can recover most of an aligned model’s safety performance across diverse red-teaming benchmarks while largely preserving LLM competence on different tasks. They further reveal that safety and helpfulness share many of the same key neurons but require different activation patterns, offering an explanation for the well-known alignment tax trade-off (aka helpfulness vs safety trade-offs). Finally, they found out that activation of the LLMs can be used as a safeguard that predicts unsafe generations.\n\nSTRENGTHS AND WEAKNESSES:\n** Strengths **\n1. Clear presentation and structure. The paper is straightforward to follow: the problem statement, methodological steps (identification, ranking and causal verification of safety neurons), and empirical validation are laid out in a logical flow, making the main take-aways easy to understand.\n\n\n2. Well-motivated contribution. The work offers a fresh angle on how “helpfulness” and “safety” interact inside an LLM, complementing existing evaluations that treat alignment as a black-box phenomenon.\n\n\n3. Insightful empirical findings. The observation that many key neurons are shared between helpfulness and safety, yet are activated in distinct patterns, sheds light on the long-discussed helpfulness-versus-safety trade-off.\n\n** Weaknesses/Questions **\n\n1. Generality beyond PEFT. All experiments fine-tune the base model with $(IA)^3$ adapters. It is unclear whether the same sparse “safety neuron” substructure—and the reported performance gap between helpfulness- and safety-oriented activations—would persist if one performed full-parameter SFT. A brief ablation on a fully fine-tuned checkpoint would clarify how much the conclusions depend on the PEFT setting.\n\n\n2. Relation to steering-vector work. Prior studies such as [1] show that a single activation direction can steer the model away from refusal. How does the proposed neuron-level intervention compare with those steering approaches in terms of effectiveness, interpretability, and computational cost? More broadly, a short discussion situating this paper within the LLM-steering literature would help readers see the incremental novelty.\n\n\n3. Predictive safeguard feels expected. If safety alignment trains the model to differentiate safe from harmful prompts, then using internal activations for prompt-level safety classification seems like a natural extension—similar to the logit-gradient test in [2]. Stating explicitly what new capability the proposed safeguard adds over existing gradient-based or embedding-based detectors would strengthen the practical significance claim.\n\n\n[1] Refusal in Language Models Is Mediated by a Single Direction\n\n[2] GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysis\n\nQUESTIONS:\n1. In lines 130-131 *“we choose the model after SFT as M1 (denoted as SFT) and the model after safety alignment as M2 (denoted as DPO).”* In my understanding, safety alignment is done in both SFT and DPO stage. The model without safety alignment is the base model. Lines 150-152 seem more correct.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-06"}