Understanding and Enhancing Safety Mechanisms of LLMs via Safety-Specific Neuron
Authors: Yiran Zhao
Wenxuan Zhang
Yuxi Xie
Anirudh Goyal
Kenji Kawaguchi
Michael Shieh
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Alignment
Safety
Interpretability
Neuron Detection
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Safety alignment for large language models (LLMs) has become a critical issue due to their rapid progress. However, our understanding of effective safety mechanisms in LLMs remains limited, leading to safety alignment training that mainly focuses on improving optimization, data-level enhancement, or adding extra structures to intentionally block harmful outputs. To address this gap, we develop a neuron detection method to identify safety neurons—those consistently crucial for handling and defending against harmful queries. Our findings reveal that these safety neurons constitute less than $1\%$ of all parameters, are language-specific and are predominantly located in self-attention layers. Moreover, safety is collectively managed by these neurons in the first several layers. Based on these observations, we introduce a $\underline{S}$afety $\underline{N}$euron $\underline{Tun}$ing method, named $\texttt{SN-Tune}$, that exclusively tune safety neurons without compromising models' general capabilities. $\texttt{SN-Tune}$ significantly enhances the safety of instruction-tuned models, notably reducing the harmful scores of Llama3-8B-Instruction from $65.5$ to $2.0$, Mistral-7B-Instruct-v0.2 from $70.8$ to $4.5$, and Vicuna-13B-1.5 from $93.5$ to $3.0$. Moreover, $\texttt{SN-Tune}$ can be applied to base models on efficiently establishing LLMs' safety mechanism. In addition, we propose $\underline{R}$obust $\underline{S}$afety $\underline{N}$euron $\underline{Tun}$ing method ($\texttt{RSN-Tune}$), which preserves the integrity of LLMs' safety mechanisms during downstream task fine-tuning by separating the safety neurons from models' foundation neurons.

Published as a conference paper at ICLR 2025
UNDERSTANDING AND ENHANCING SAFETY MECHA- NISMS OF LLMs VIA
NEURON
Yiran Zhao't Wenxuan Zhang²
Yuxi Xie
Anirudh Goyal
Kenji Kawaguchi
Michael Qizhe Shieh!t
National University of Singapore
2 Singapore University of Technology and Design
3 Google DeepMind
ABSTRACT
Safety alignment for large language models (LLMs) has become critical issue due to their rapid progress. However, our understanding of effective safety mechanisms in LLMs remains limited, leading to safety alignment training that mainly focuses on improving optimization, data-level enhancement, or adding extra structures to intentionally block harmful outputs. To address this gap. we develop a neuron detection method to identify safety neurons those consistently crucial for handling and defending against harmful queries. Our findings reveal that these safety neurons constitute less than 1% of all parameters, are language-specific and are predominantly located in self-attention layers. Moreover, safety collectively managed by these neurons in the first several layers. Based these observations, we introduce a Safety Neuron Tuning method, named SN- Tune, that exclusively tune safety neurons without compromising models' general capabilities. SN-Tune significantly enhances the safety of instruction-tuned models, notably reducing the harmful scores of Llama3-8B S from 65.5 to 2.0, Mistral-7B-Instruct v0.2 from 70.8 to 4.5, and Vicuna- 13B-1.5 from 93.5 to 3.0. Moreover, SN-Tune can be applied base models on efficiently establishing LLMs' safety mechanism. In addition, we propose Robust Safety Neuron Tuning method (RSN-Tune), which preserves the integrity of LLMs' safety mechanisms during downstream task fine- tuning by separating the safety neurons from models' foundation neurons
INTRODUCTION
The rapid developments of large language models (LLMs)
(Achiam et al. 2023 Jiang al. 2023
Reid et al. 2024 Team et al.] 2024 Dubey et al. 2024) have brought safety alignment to the
forefront of research (Zou et al. 2023 Zhao et al. 2024d Zou al. 2024 Deng et al.) 2024
Wei et al. 2024a). Different perspectives have been studied to improve safety alignments, such
as improving optimization (Ouyang al. 2022 Rafailov et al. 2024 Yuan et 2023), refining
training data (Zhou et al. 2024 Rafailov et al. 2024 Zhang et al. 2024 p, or implementing additional
structures designed intentionally block harmful outputs (Inan et al. [2023 Zou et al. 2024). Despite
its importance, clear understanding of safety mechanisms in LLMs remains absent. Prior works tried to identify and interpret safety mechanisms L.LMs from either layer-level (Li et al. 2024)
or feature-level (Chen et al.] 2024). However, their identification methods attribute nearly 10% of parameters to safety-related functions. This large proportion makes it challenging to effectively perform safety alignments based on these findings (Anwar et al. 2024 Zeng et al. (024)). Moreover, other works have suggested that safety mechanisms can be easily compromised through minor
parameter adjustments (Qi et al. 2024 Zhao et al., 2024a).
In this work, we aim to understand and interpret safety mechanisms in LLMs at a finer granularity, specifically the neuron level across all structures, including the self-attention and ferw-fd parts. Here, a "neuron" is represented by a single row or column of a parameter matrix in LLMss We identify "safety neuron" one that consistently plays crucial role in processing and defending against harmful queries. Specifically, a neuron is considered important if its removal-by setting
"Correpondencce to: Yiran Zhao (hhooyran@u.mus..o Michael Shich (mwhhaalshihh@ccooo. (micheelshichh@@oommms..ch Our code is publicly available at httpp///gtchhb..c.////aaaaaaaaaa

---

Published as a conference paper at ICLR 2025
its parameters to zero sio---siiiiiiatlly affects the generated output beyond a specified threshold. To achieve this, we input a corpus of harmful queries and extract neurons that are important across all queries in the corpus, identifying them as the set of safety neurons in the LLM. By conducting thorough analysis of these identified safety neurons in various models, we uncover several key insights about L.LMs" safety mechanisms: First, we find that safety neurons comprise less than 1% of all parameters. Second, each language has its own unique safety neurons, with minimal overlap between them. Third, safety is collaboratively managed by safety neurons located in the first several layers of the model. Lastly, safety neurons are predominantly located within the self-attention structures.
Motivated by these intriguing ob- servations, we propose a Safety Neuron Tuning method (SN-Tune), designed to exclusively tune the safety neurons in L.Mss. As shown in Fig- ure [ we gather safety training doc- uments that include harmful queries and refusal safety outputs, similar to the widely used safety alignment training settings (Inan et al. 2023
Neuron Importance
Safety Neuron Identify
Safety Neuron Turing
Safety Neuran
buidd
How acrrss finerm?? llentity
Zhang et al., 2024] You et al. 2024).
Harmful Queries Corpus
We then tune the identified safety neurons while leaving other safety- unrelated neurons unchanged by ting their gradients to zero during the tuning process. Experimental results demonstrate that SN-TTuu not only enhances the safety mechanism for instruction-tunc models but also es- tablishes safety mechanism for base models without compromising their general capabilities. Notably, it reduces the average harmful scores of Llama3-88 1a--33888--nstuttttion from 65.5 to 2.0, over, TNNe reduces base models' harmful score from around 100 to 5.3, 13.5, and 13.8 for LLama2-7B--asse LLLama-8B--Bas and Mistral-7BB-v..1 respectively. The harmful score is eval- uated using the harmful behavior dataset (Zou et al. 2023). by averaging the Attack Success Rate (ASR) across various ad adversarial attacking methods, including Direct Attack, GCG (Zou et al. 2023). AutoDAN (Liu et al. (024) and PAIR (Chao et al.) 2023). Concurrently, we assess the models' general capabilities using representative NI.P tasks including MMLJ (Hendrycks et al. 2020), ARC- Challenge (Clark et al., 2018), and GSM8K (Cobbe al., 2021). ensuring that safety improvements do not come at the cost of overall performance.
Figure I: SN-Tune mainly consists of three steps: ! cal- culating neuron importance for handling harmful queries; ? identifying "safety neuron" that consistently play a crucial role in processing harmful queries; @ tune the identified safety neurons while leaving other safety- safety-uneelated neurons unchanged during the tuning process.
from 70.8 to 4.5, and Vicuna-13B-14 from 93.5 to 3.0. More-
Building upon the strong performance of SN-Tune, we aim to further enhance LLMs' safety robustness during downstream tasks fine-tuning, a common practice for users focusing on specific application scenarios (Yu et al. 2024 Zhao et al. 2024c). As Qi et al. (2024) observed, even fine-tuning with scemingly benign and widely used datasets can unintentionally compromise the safety alignment of L.LMs. From the neuron perspective, fine-tuning on downstream tasks modifies certain foundation neurons (Zhao et af. 2024b Liang et al. 2024). Consequently, the vulnerability of model's safety mechanism downstream task fine-tuning may be attributed to the overlap between these foundation neurons and safety neurons, with the latter being unintentionally adjusted during the fine-tuning process. Inspired by this observation, we propose another technique called Robust Safety Neuron Tuning method (RSN-Tune). It separates safety neurons from foundation neurons by selectively tuning only those safety neurons that do not overlap with foundation neurons when applying SN-Tune to instruction-tuned models. Experimental results demonstrate the effectiveness of RSN- in enhancing models' safety robustness during downstream tuning. Notably, it reduces 1imma227B-Chatt harmful score after tuning GSM8K training set from 41.0 to 26.0 and Mistral- TB-IInttruct-0025 from 79.0 to 41.0. Importantly, RSN-Tune enhances safety robustness while maintaining models' downstream tuning performance.
2

---

Published as a conference paper at ICLR 2025
2
SAFETY NEURONS
In this section, we propose neuron detection method that can calculate the importance of a neuron when handling a query without a corresponding labeled output.
2.1 SAFETY NEURON DETECTION
A neuron is defined as single row or column of parameter matrix in LLMs, including the self- attention and ferd-forward structures. To identify neurons responsible for safety in alignment-tunet LLM. it's crucial to extract those that play key role in processing inputted harmful queries.
Foundational Safety Neuron Detection Formally, we denote the I-th neuron in layer i as N/ while the intermediate representation after layer i when handling harmful query is denoted as (1)).) Furthermore, the importance of neuron 2/ in processing x is calculated by
h win (x) h2 (z))22,
where M (2) represents the intermediate representation after deactivating neuron N There- fore, the activated neurons of the model when handling harmful query x can be calculated by
N₂ [N, (1) III wyr hx(r)]l2 2 for all N in LLM},
where e is pre-defined threshold. Furthermore, after collecting set of harmful queries, denoted as X. We extract neurons consistently activated for all queries in X, identifying the safety neurons we aim to obtain, i.e.,
Natte = |N E N, Var E X, for all N. in LLM).
(3)
Accelerated Safety Neuron Detection The process of deactivating N sequentially in Equation extremely slow due to its sequential nature. Drawing inspiration from the parallel neuron detection method proposed by Zhao et al. (2024b), we implement it safety neuron detection through the incorporation of masks and parallel computations. Specifically, for the feed-forward layer.
| Mxgo (x) - (x(2)122 = [1(hma (x)) Mask) downl/2 Wadwn)
where hm is the intermediate embedding between the up-projection and Nown-projection matrices, Mask is an identity matrix of size (dim(hm)) x dim(nm))). and W Wownn denotes the projecrooottoon matrix in the feed-forward layer. Moreover, for the self-attention layer,
1/ woo (x) h,(x)1l2 x softmax W (o(x) -(x)) - softmax Wo(z) (z)
(5)
Vd
Vd
where Wo and Wx are the attention matrices for Q and K, respectively, and vd represents the corresponding dimension following the notations in Vaswani al. (2017), and
A(x) Wo(s)).reszee((. 1, Wx(s)) (1, gixtxd
(6)
Detailed proof of Equation and Equation 5 is available in Appendix A.1
2.2 VERIFY IDENTIFIED SAFETY NEURON
We subsequently apply the accelerated safety neuron detection method to a variety of alignment-tuune LLMs to identify corresponding safety neurons, and conduct experiments to verify that these neurons are exclusively responsible for handling safety. Specifically, by deactivating the safety neurons, the model's safety mechanism will be attacked, potentially transforming it into a harmful model. However, by solely manipulating neurons associated with safety, the overall functionality should remain intact. Consequently, the model could become both helpful and harmful.
3

---

Published as a conference paper at ICLR 2025
Table I: Performance of models harmfulness and general capability with the deactivation of safety neurons ("Deact-SN")) and an equivalent number of randomly selected neurons ("Deact-R") Harmfulness is measured by Attack Success Rate (lower is safer), and capability by Accuracy.
Llama2-7B-Ca Bh-hha
Llama3-8B- Hamma3-8-IIstructton Mistral-77 Mistral-7B-Instiee -ee--Brrunttt--llle
Dataset
Origin. Deact-R Deact-SN
Origin. Deact-R Deact-SN Origin. Deact-R
Deact-SN
Harm Behavior
0.0
2.0
97.0
30.0
31.0
78.0
36.0
39.0
86.0
Harmfull
Adv Behavior
0.0
3.0
83.0
7.0
13.0
96.0
30.0
30.0
87.0
Multilail-Ei En
12.7
12.9
81.6
20.0
21.6
74.3
44.1
46.8
86.4
Avg. Harmful
4.2
6.0
87.2
19.0
21.9
82.8
36.7
38.6
86.5
MMLU GSMSK
48.2
48.4
47.8
65.3
63.2
62.7
59.2
59.3
58.5
Capablity
24.8
22.7
21.9
75.9
73.6
72.4
43.6
43.6
42.1
Avg. Capability|
36.5
35.6
34.8
70.6
68.4
67.6
$1.4
51.5
50.3
110
0.0
62
0.5
ASR
0.4
26-
0.1
Mictal 7B lnstret v02
LJana3-0 Ristte
0.0
Uamas 88 Instruct
Uamaz 7B Chat
Matral Insoruct-v..2
Proportion Deactivated Safety
Figure 3: Distribution of Safery Neuron in different structures.
Figure 2: Effects of deactivated safety neurons ASR.
Experimental Setup
We employ three open-source models that have been specifically tuned for safety, including Llama2-7B-Chnn (Touvron al. [023]. -lama3-8B-Instruttinn
(Dubey et al. 2024). (Jiang et al. 2023). The harmful corpus set used to detect safety neurons constructed from the training set split in Zou et al. 2024). More details are illustrated in Appendix A.2 To prove the generability of the detected safety neuron, we test the harmfulness of the model on harmful behavior testset Zou al. (2023) (Harm Behavior), adversarial behavior testset Mazeiial et al. (2024) (Adv Behavior) and English version of multilingual jailbreak testset in Deng al. (2024) (MultiJail-En). Furthermore, the models' general capability evaluated by MML. HMMI-UHndyyycks et al.
and Bosstral-7--nssruct--aa
(2020) and GSM8K Cobbe al. (2021).
Evaluation Metrics The harmfulness assessed through direct attacks using the Attack Success Rate (ASR), which identifies harmful keywords from the output, following the method outlined by Zou et al. (023). Furthermore, accuracy is the metric used for MMLU and GSM8K.
Existence of Safety Neurons Table I demonstrates how deactivating safety neurons can attack the model's safety mechanism. Moreover, the model's general capabilities have not diminished, indicating that these neurons are specifically for safety mechanisms, not for other functions. Even with just about 0.5% of neurons deactivated, the model's safety capabilities are significantly compromised, leading on substantial increase harmful behavior: from 4.2 to 87.2 on Llama2-7B-chat, from 19.0 to 82.8 and from 36.7 7 to 86.5 on deactivating an equivalent number of neurons has little no impact on the model's safety. Regarding general capability, deactivating the safety neuron shows minimal impact, similar to deactivating randomly selected neurons, as demonstrated by the performance of 36.5 and 34.8 on Llama2-77 7B-cha,, 70.6 and 68.4 and after deactivation. Therefore, the detected neurons are safety neurons that are associated with safeguarding the models.
Meanwhile, randomly
-iima-----Instuuttioon and 51.4 and 50.3 Mistral- B Instrucct--O.22 before
4

---

Published as a conference paper at ICLR 2025
Ehaaa--B-Chat
Llazad 3B Restent
A502
ASB
Lina? Chr
Mismi- B-
Lamads k-ls Instner
Layer Index
Layer Index
Figure 4: Effect deactivating safety neurons in different layers. The left represents deactivating safety neurons before the certain layers, the right indicates deactivation after the certain layers.
Language
Language
Language
Languadge
(a) Llama2-7B-Chan
(b) Llama3-8B Instruction
(c) Mistral-7B-Insstuat--e
Figure 5: Overlapping ratio of safety neurons across different languages.
2.3
ANALYZE SAFETY MECHANISM IN LLMS
As we have detected the safety neurons of L.Mss, we conduct a more detailed and comprehensive analysis of the properties of LLM's safety mechanism.
2.3.1 SAFETY MECHANISM PROPERTIES
Safety mechanism resilient but breakable by under one percent of the parameters. Figure 2 shows the harmful score of three models as deactivating different number of safety neurons. In Pisstrl--B-Instructt-
deactivating 0.2% of neurons can destroy its safety mechanism, compared to 0.4% for Llama2-7B- -B-Chht and 0.5% for Llama3 8B Instrrruuci. Furthermore, emergence of "harmfulness" is observed for three models. For example, in -lama2-7B-Char the leap appears when deactivating 0.3% neurons, while the number is 0.15% for Llama3-8B B-Innsttctttionn and is 0.1% for Mistral-7BB-Instuut--..222
Safety Mechanism is handled by the first several layers together. Figure illustrates the detrimental impact deactivating safety neurons across various layers in models. Upon deactivating neurons in the first 10 layers simultancously, we observe a near-complete breakdown in the safety mechanism of Mama2-7B-Chan This threshold is 10 for Fi-tull----nstrutt-- 8B- IB-Instruction. On the contrary, if we deactivate safety neurons from the back to the front, the breakdown of safety mechanisms becomes apparent as nearly all safety neurons are deactivated.
and 16 for Llama3-
Safety neurons predominantly reside within the self-attention layers. In Figure safety neurons categorized based on their belonging structures, which include the attention structure and feed- forward structure. Our findings reveal that safety neurons predominantly reside within the attention structure. Specifically, in Llama2-7B. Chathtt.. 77% of safety neurons attributed to the attention structure, while 23% belong to the feed-forward structure. This finding aligns with the interpretation that the attention structure primarily handles understanding, while the feed-forward structure is mainly
5

---

Published as a conference paper at ICLR 2025
Table Performance SN- Tune on instruction-uun tunnnd models. General capabilities are evaluated by accuracy, while harmfulness is evaluated by ASR.
138-1-13B-v1.5
Caamaa-8B-Instruccie
Dataset
Origin. Circ- Break SN-Tuno Origin. Circ- Break SN-Tunc
Origin. Circ Brec-Beak NN-Tune
Training Cost (min.)
43
24
2
23
2
# Parameters (M)
0
34.1
0
27.5
0
27.5
0
MMLU
53.4
52.8
55.7
65.2
65.6
67.3
58.6
56.3
59.5
Capablity!
ARC-c
59.7
61.3
61.6
73.7
74.1
74,9
72.6
71.8
73.4
GSM8K
33.4
35.0
34.8
63.2
64.3
69.6
43.7
42.5
44.1
Avg. Capablity
48.8
49.7
50.7
67.4
68.0
68.4
58.3
56.9
59.0
Direct
92.0
0.0
30.0
0.0
0.0
36.0
0.0
GCG
100.0
3.0
74.0
3.0
4.0
88.0
6.0
Harmfull
AutoDAN
93.0
2.0
3.0
82.0
0.0
0.0
91.0
3.0
4.0
PAIR
89.0
16.0
9.0
76.0
9.0
4.0
68.0
22.0
8.0
Avg. Harmfal
93.5
5.3
3.0
65.5
3.0
2.0
70.8
10.0
4.5
responsible for knowledge extraction (Geva al., 2021 Given that the safety mechanism focuses understanding potential threats to discern their harmful nature without the need to extract much new knowledge, it logical for safety neurons to predominantly reside the attention structure, despite the attention parameters being fewer than half of the feed-forward parameters.
2.3.2 MULTHLINGUAL SAFETY
Based on the research by Deng et al. (2024): Yong et al. (2024); Kotha et al. (2024), the safety mechanism cannot be effectively transferred between languages. For instance, specifically tuned for safety in English, it may still pose risks when applied to other languages. Drawing inspiration from these discoveries, analyze this phenomenon through the perspective of safety neurons. We specifically incorporate five languages- (2h), Thai (th), and Vietnamese (vi)-spanning high-resource low-resource languages, to visu- alize the overlap of safety neurons. Specifically, the overlap among safety neurons are defined as overlap(x; y) [ NyllLNNG where Ninnguage represents the set of safety neurons in that lan- guage. Figure displays the intersection of safety neurons across languages. Our analysis reveals that the overlap of safety neurons is typically below 30%, significantly less than that of language-specificc neurons, which are subset of neurons responsible for processing multilingual queries et al.) 2024b). This disparity underscores the unique nature of safety neurons each language, indicating that safety capabilities are not transferrable between languages. This observation aligns with the progression of the SFT training, where diverse seguage-specifii safety corpora are developed to provide tailored safety mechanism for individual languages (Zhang et al. (024).
when LLM
(en), Italian (it), Chinese
3 EFFICIENT SAFETY TRAINING
With only limited number of parameters able ensure safety, focus manipulating these neurons effectively to strengthen or even establish the safety mechanism.
LIVE-LINE WORK ON INSTRUCT TUNED MODEL
Experimental Setup With fewer than 1% of neurons dedicated safety, we can enhance safety by fine-tuning them using safety corpus, named as Safety Neuron Tuning (SN M2N-TH Tune), Specifically.
we create a safety corpus by partitioning training dataset from (Zou et al.) 2024). utilizing to identify and strengthen safety neurons. In manner similar to the setup in Table I assess models' harmfulness using the harmful behavior testset, while their general capabilities are evaluated on MMLU (5-shots) (Hendrycks et al. (2020), ARC-c (3-shots) (Clark et al.) 2018 88,, and GSM8K (zero-shot) (Cobbe et al. 2022). Additionally, beyond testing direct attacks, we explore other attack methods, including GCG (Zou et al. 2023), AutoDAN (Liu et al. 2024). and PAIR (Chao al.) 2023). To demonstrate the generality of the method, we also employ the large model Vicuna-13B. and Sesral-7B-Instruetee
v1.5 (Peng et al. 2023) in addition to
We

---

Published as a conference paper at ICLR 2025
Table Performance 3N-Tune on base models, General capabilities are evaluated by accuracy, while harmfulness is evaluated by ASR.
Llama2-7B-a
Llama3-8B-Base
Mistral-7B--0.J
Dataset
Origin. Cire- Break SN-Tune Origin. Circ-
Cire-Break Break SN-Tunc
Origin. Cire-Break Break
Tune
Training Cost (min.)
23
35
2
21
2
# Parameters (M)
0
34.1
0
27.5
0
27.5
0
MMLU
49.2
49.1
49.2
70.1
68.9
69.6
68.4
68.1
69.2
Capablity!
ARC-c
27.6
26.8
29.3
70.7
72.0
71.8
74,8
73.4
74.7
GSM8K
12.7
13.7
16.3
58.9
58.2
59.5
50.4
47.6
52.3
Avg. Capablity|
29.8
29.9
31.6
66.6
66.4
67.0
62.0
63.0
65.4
Direct
97.0
84.0
100.0
87.0
0.0
100.0
78.0
6.0
GCG
100.0
92.0
100.0
95.0
14.0
100.0
82.0
13.0
Harmfull
AutoDAN
100.0
97.0
9.0
100.0
92.0
21.0
100.0
93.0
12.0
PAIR
98.0
89.0
5.0
100.0
96.0
19.0
100.0
97.0
24.0
Avg. Harmfal
98.8
90.5
5.3
100.0
92.5
13.5
100.0
87.5
13.8
compare SN-Tune with Zou et al. (2024), who train an independent model called "Circ-Break" to circuit breaker, interrupting models when they produce harmful outputs.
Experiment Details We utilize the HarmBench implementation (Mazeika al. 2024) for the attacking methods. For general capability evaluation, we employ accuracy as the metric, while for harmfulness assessment, primarily focus on the training corpus, number of epochs, and learning rate. As the fine-tuning process is essentially continued training, aim minimize alterations to the existing parameters. Specifically, we use a dataset of 50 documents where the model refuses to answer harmful questions, train for only 1 epoch, and set the initial learning rate to le 6.
Attack Success Rate (ASR). The hyperparameters for fine-tuming
Main Results T shows the performance of SN-Tune instruction tuned model. Note that tuning base models can be regarded as live-line work, meaning that we hope to enhance models' safety without sacrificing models' general instruction following capabilities in other aspects. We find that SN-Tune effectively enhances model safety without compromising general capabilities, and in some cases, even slightly improves them. Specifically, SN-Tune reduces the harmful score of (ouna-13B-vl. from 93.5 to 3.0, Llama3-8B-Instruce v0.2 from 70.8 to 4.5. Meanwhile, the general capabilities are largely preserved. Furthermore, compared to Cire-Break, SN-Tune requires less training time and fewer additional parameters.
from 65.5 to 2.0, and Mistral-7B-Instrunn
3.2 EFFICIENT ESTABLISH SAFE MECHANISM FOR BASE MODEL
Experimental Settings When implementing SN-Tune on base models, we largely maintain the settings described in Section 3.1l with two key differences. First, we do use the specific chat template for fine-tuning. Second, for evaluations on GSM8K, we employ 5-shot approach rather than zero-shot, given the use of base models.
Main Results 3sows the performance of SN -NNe on base models. We find that SN-Tune effectively enhances model safety without compromising general capabilities, and in some cases, even slightly improves them. Specifically, SN- Tune reduces the harmful score of Llama2-7B-Ban from 98.8 to 5.3, Llama3-8B- BB-Base from 100.0 to 13.5, and Mistral-7B-v0.1 from 100.0 to 13.8. Meanwhile, the general capabilities are largely preserved. For instance, the original general capability score for I.Lama2-7B-Ba
29.8, while the model after SN-Ture achieves 31.6. Similarly, the score increases from 66.6 to 67.0 for Baama--8B-Basse and from 100.0 to 13.8 for Mistral- Furthermore, different from instruction-tuned models, Circ-Breas can not construct safety mechanism on the base model with several training corpus. Specifically, harmful score of 1-ama2-7B- BBse after tuned by Mistral- 1B- Moreover, the training time for SN-Tune Llama2-7B BB-Base is just 2 minutes, while Cire-Breat requires 23 minutes. On Llama3-88 -B--ase,, the time costs are and 35 minutes respectively, while on Mistral-7B-vO.1. they are and 21 minutes respectively.
is still 90.5, while the number is 92.5 for Llama3-88 Ba-e and 87.5 for
7

---

Published as a conference paper at ICLR 2025
Table RSN-Tune's performance on improving models' safety robustness, "Before"; pre-tuning. "Original": direct tuning. "SN-Tume" and "RSN-Tune": tuning on safety-enhance models.
Llama2-7B-Chen
Mistral-7B-Instruettvy
Dataset
Before
Origin.
SN-Tune
RSN-Tune
Before
Origin.
SN-Tume RSN-Tune
GSM8K
16.8
26,5
27.2
26.2
43.7
63.4
61.8
63.2
Harmful
0.0
41.0
38.0
26.0
36.0
79.0
72.0
41.0
Hammtu
Copability
Hormfll 2T
Homndi (LLams SB Inctrctiion)
Capability 1. (lne))
5
Capchility
Leaminn Rato
Capobility (LLama? 3D Instrction)
09
28
20
70
Number Safety Decument SN-Tone
Figure 6: Ablation on the number of safety documents used in training.
Epoch
Epoch
Figure 7: Ablation on training epoch and learning rate..
MORE ROBUST EFFICIENT SAFETY TUNING
Fine-tuning instruction-ture tuned models specific downstream tasks is common practice for users seeking to optimize performance particular application scenarios (Yu et al., 2024, Zhao al.) 2024c). However, Qi et al. (2024); Jain et al. (2024) have noted that even fine-tuning with seemingly benign and widely used datasets can unintentionally compromise the safety alignment of LLMs. To address this issue and mitigate its effects, propose Robust Safety Neuron Tuning method, called RSN-Tune. According to Zhao et al. (2024b), a specialized set neurons, termed foundation neurons, responsible for fundamentally managing queries. Consequently, the vulnerability of model's safety mechanism to general fine-tuning may be attributed to the overlap hetween foundation neurons and safety neurons, with the latter being inadvertently altered during the fine-tuning process. Inspired by this observation, we propose separating the safety neurons from the foundation neurons. This separation is achieved by selectively tuning only those safety neurons that do overlap with foundation neurons when applying SN-Tune to instruction- tuned models, as illustrated in Section 3.1
We then conduct experiments prove the effectiveness of RSN-Tune.
Experiment Settings We employ Llama2-7B-Chn and Histral-7B-Insttuuttee as backbone models considering their excellent safety performance and generality. For fine-tuning, we employ the GSM8K dataset (Cobbe al., 2021), widely recognized as challenging and representative benchmark for reasoning tasks. The foundation neurons are detected by Wikipedia corpu with the same neuron detection method illustrated in Section 2.1
Main Results Table demonstrates the effectiveness of RSN- in enhancing models' safety robustness during downstream tuning. We observe that direct tuning using the GSM8K training set significantly increases model harmfulness. For instance, Alama2-7B-Chan aat's harmful score rises from 0.0 to 41.0, while also affects SN- Tune, which indiscriminately enhances all safety neurons, regardless of their overlap with foundation neurons. In contrast, RSN-Tune partially preserves model safety after downstream tuning. Specifically, it reduces Llama2-77 BB 7B-Chat's harmful score to 26.0 and Mistral-7B-Innstt v0.2's to 41.0. However, a complete harmful score reduction to 0.0 not achievable due to an insufficient number of non-overlapping safety neurons.
score increases from 36.0 to 79.0. This phenomenon
8

---

Published as a conference paper at ICLR 2025
5 FURTHER ANALYSIS
In this section, to further understand the mechanism and explore the influencing factors to the performance of SN-Tune, we conduct comprehensive ablation analysis, mainly including the number of training safety documents, training epoch and learning rate.
5.1 NUMBER OF SAFETY DOCUMENTS FOR EN-TUNE
Experiment Settings We employ LLama2-7B, BB-Base to serve as the representative hase model and Jlama3-8B-Instruuee to represent the instruction-tuner model. Following the setting outlined in Section 3
we assess the models' overall performance and potential harmfulness after tuning by SN-Tune with varying quantities of safety-related documents.
Main Results Figure illustrates the effect of training document quantity on SN-Tune. We observe that the general capabilities of both LLama2-7B-Baas (yellow dotted line) and Llama3-8B- Instruction (blue dotted line) remain largely unaffected regardless of the training document size. This stability primarily attributed the limited number of neurons trained. Specifically, only train the safety neurons, which comprise approximately 0.5% of all parameters, the majority of the language ability remains intact, resulting in preserved general capabilities. Notably, the harmful score of both models decreases rapidly as the number of training documents increases to 40 for L.Lam2--77 2a----B--se (yellow line) and 1amma-88-Innsruuu efficiency of SN-Tune in both enhancing and establishing model safety mechanism with just few dozen documents. contrast, Cire- Break requires around 4000 safety documents and a retention dataset of similar size (Zou al.) (024). These findings underscore that Tune is not only effective but also highly efficient in tuning safety for LLMs.
(blue line). This demonstrates the
5.2 LEARNING RATE & TRAINING EPOCH
Experiment Settings We further explore the effects of learning rate and number of training epochs simultaneously, as both hyperparameters influence the magnitude of parameter updates. We employ Llama2-7B-Bi highly representative of safe language models. Similar to Section [.1 we investigate the model's performance in terms of both general capabilities and safety aspects.
as our model since instruction tuned versions derived from are
Main Results Figure iillustrates the impact of learning rate and training epoch both harmfulness (left) and general capability (right). We observe that with 10 training epochs, harmful score reaches 0.0, but the model also loses generality, scoring 0.0 in capability. As the number of epochs decreases, this effect diminishes. For instance, with 5 epochs and a learning rate of 10-?, the general capability improves 3.2. Further reducing while increasing general capability to 6.8 at a 10-7 learning rate. The best performance is achieved with a single epoch, aligning with other continue-train approaches (Dou et al. 2024 Zhang et al. 2024). Additionally, higher learning rates lead to overfitting, resulting in both harmful score and general capabilities dropping to 0.0, while lower rates fail effectively train safety into the model. Consequently, a learning rate of 10-0 emerges as the optimal balance between low harmful score and high general capability.
epochs maintains low harmful scores across all learning rates
RELATED WORK
Safety Alignment. To build safe LLMs, alignments has also been widely studied topic in the
community (Stiennon et al. 2020 Ouyang et al.) 2022). Efforts have been put into improving
helpfulness (Bai et al. 2022, Cheng et al.) 2023), honesty (Kaddour et al. 2023 Liu et al. 2023
Park et al. 2023), ), and harmlessness (Hartvigsen et al. 2022). Among them, safety, i.e., reducing
harmfulness ness, established and improved via optimization (Ouyang al. 2022 Rafailov et al. 2024
Yuan et al. 2023). refining training data (Zhou et al. 2024 Rafailov et al. 2024 Zhang et al. 2024
or implementing additional structures designed to intentionally block harmful outputs (Inan et al.
2023
Zou et al. 2024). However, these methods are indirect and require many resources.
https://hugginggfaee
wimdd/iiiiiiiiiiiiaiiimmm /wi kipedia

---

Published as a conference paper at ICLR 2025
Interpretability. In the of LLMs, one brunch of interpretability work includes efforts to
2022 Meng et al. self-attention layer, examining its connection to reasoning capability (Hou et al.
understand knowledge storage (Geaa et al. 2021 Dai et al. 2022 Geva et al.
2022; Li et al. 2023; Kotha et al. 2024; Jm et al. 2024). Another line of research centers the
2023 Stolfo et al.
20233 Friedman al., 2023) by contrasting the reasoning tree based attention weights. In the context of safety, prior works tried to identify and interpret safety mechanisms in LLMs from either layer-level (Li al. 2024) feature-level (Chen et 2024). However, their identification methods attribute nearly 10% of parameters to safety-related functions, which is too coarse to be used.
Interpret Safety Mechanism. Some workers try to interpret the safety mechanism of LLMs. Wei al. (2024b) identifies safety neurons using the SNIP score (Lee al. 2019), which requires correct labels and enforces sparsity constraints (Sun et al. 2024). In contrast, our method operates without correct labels or uniform sparsity, identifying only 0. 1% of parameters as safety neurons (compared to 3% in (Wei et al., 2024b) with higher accuracy. Additionally, while Wei et al. (2024b) focuses on analyzing safety neurons, we introduce SN-Tune and RSN-Tune to enhance LLM safety alignment, which Wei et al. (2024b) does not address. Other works, such as Chen et al. (2024). Hsu al.
(2025). and Zhang et al. (2022). take different approaches to model safety. [Chen et al. (2024) limits detection to the ford-forward layer, identifying 5% of parameters as safety neurons. Hsu al. (0225 incorporates structural modifications, while ZZhan et al. (2022) focuses on attack improvements, an impractical approach for black-box LLM training.
7
CONCLUSION
Safety alignment in LLMs is critical yet undercxplored. We introduced method to detect and tune safety neurons, which are less than 1% of parameters and mainly self-attention layers. Our Safety Neuron Tuning (SN-Tune) enhances model safety without compromising performance, significantly reducing harmful scores both instruction-tuned and base models. This approach also improves safety robustness during fine-tuning by separating safety neurons from foundational ones.
ACKNOWLEDGMENTS
This research is partially supported by the National Research Foundation Singapore under the AI Singapore Programme (AISG Award No: LISG2-TC-2003-000-S20 Education Academic Research Fund Tier (Award No: TI 251RES2207). We thank Shiqi Chen for the insightful discussion at the beginning of the project.
and the Singapore Ministry of
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint w:Xhv:2303.08774 2023.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Palcka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.094 009329,,, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint S333i::2244008666 2022.
Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. Jailbreaking black box large language models twenty queries. arXiv preprint 9rX1v:2310.0841, 2023.
Jianhui Chen, Xiaozhi Wang, Zijun Yao, Yushi Bai, Lei Hou, and Juanzi Li. Finding safety neurons in large language models. arXiv preprint arXiv:2406. 14144, 2024.
10

---

Published as a conference paper at ICLR 2025
Pengyu Cheng, Yifan Yang. Jian Li. Yong Dai, and Nan Du Adversarial preference optimization. arXiv preprint arXiv:2311.0804. 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint wrXiv:1803.044:5 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 8493-8502, 2022.
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations, 2024.
Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, and Min Lin. Sailor: Open language models for south-east asia. arXiv preprint 2xiv:2444.036088, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Dan Friedman, Andrew Lampinen, Lucas Dixon, Danqi Chen, and Asma Ghandeharioun. Inter- pretability illusions in the generalization of simplified models, 2023.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5484-5495, 2021.
Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 30-45, 2022.
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi. Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine- machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 3309-3326, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2020.
Yifan Hou, Jiaoda Li, Yu Fei, Alessandro Stolfo, Wangchunshu Zhou, Guangtao Zeng, Antoine Bosselut, and Mrinmaya Sachan. Towards mechanistic interpretation of multi-step reasoning capabilities of language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Singapore, December 2023. Association for Computational Linguistics. doi: 10. 18653/v3 118653/v1//00/ 1/2023. emnlp-main. 2999. URL URL httptt/sss/aaaatllllllloooooo/ hww.tthooooooo.o.o220002 org/2023. emnlp main. 299
Empirical Methods in Natural Language P Processing, pp. 4902-4919,
Chia-Yi Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin- Yu Chen, Chia-Mu Yu, and Chun- Ying Huang. Safe lora: The silver lining of reducing safety risks when finetuning large language models. Advances in Neural Information Processing Systems, 37:65072-65094, 2025.
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based oututt safeguard for human-ai conversations. arXiv preprint 1rXiv:23/2.06677 2023.
11

---

Published as a conference paper at ICLR 2025
Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, Tim Rocktäschel, Edward Grefenstette, and David Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. In The Twelfth International Conference Learning Representations. 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2110006255, 2023.
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications large language models. arXiv preprint arXiv:2307.1016, 2023.
Suhas Kotha, Jacob Mitchell Springer, and Aditi Raghunathan. Understanding catastrophic forgetting in language models via implicit inference. In The Twelfth International Conference on Learning Representations, 2024.
Lee, T Ajanthan, and P Torr. Snip: single-shot network pruning based on connection sensitivity. In International Conference Learning Representations. Open Review, 2019.
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03344. 2023.
Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. Safety layers of aligned large language models: The key to Ilm security. arXiv preprint arXiv:2408.17003, 2024.
Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu, Jie Zhou, et al. Multilin- gual knowledge editing with language-agnostic factual neurons. arXiv preprint arXiv:2406.1644 16416, 2024.
Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. AutoDAN: Generating stealthy jailbreak prompts aligned large language models. In The Twelfth International Conference Learning Representations, 2024. URL https:ttp///m /ttp::///oeeeeeeeeeeeeeee /
net http:/ffrummiddoo ftd///rdm 4 qKko
Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. Trustworthy llms: a survey and guideline for evaluating large language models' alignment. arXiv preprini arXiv:2388.05374. 2023.
Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang. Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, al. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Forty-first International Conference Machine Learning, 2024.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Advances in Neural Information Processing Systems, 15:17399-17372, 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Peter S Park, Simon Goldstein, Aidan GaGara Michael Chen, and Dan Hendrycks, Ai deception: A survey of examples, risks, and potential solutions. arXiv preprint (tl.:2308.1452. 2023.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint 2:Xiv:2304.03277, 2023.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference Learning Representations, 2024.
12

---

Published as a conference paper at ICLR 2025
Rafael Rafailov, Archit Sharma, Erie Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, lean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In Houda Bouamor, Juan Pino, and Kalika Bali (cds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7035-7052 Singapore, December 2023, Association for Computational Linguistics. doi: 10. 10.18653/v 12.18653/112223.. wwnllp-mainnn....m 435. URL 2023. www.pmmnnpmmmnn.mmmm 435
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 24000:2408.00118 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt, Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024a.
Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. In Proceedings of the 41 International Conference on Machine Learning, pp. 52588 526100 2024b.
Zheng-Xin Yong, Cristina Menghini, and Stephen H. Bach. Low-resource languages jailbreak gpt-4, 2024. URLpttts:://axx
org abs 22103.31.0000 02446
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302. 2023.
Yi Zeng, Hongpeng Lin, Jingwen Zhang. Diyi Yang, Ruoxi Jia, and Weiyan Shi. How johnny can persuade Ilms to jailbreak them: Rethinking persuasion to challenge safety by humanizing Ilms. arXiv preprint arXiv:2401.067 2024.
13

---

Published as a conference paper at ICLR 2025
Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, et al. Seallms 3: Open foundation and chat multilingual large language models for southeast asian languages. arXiv preprint arXhv:240..1677, 2024.
Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang. Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin: Durable backdoors in federated learning. In International Conference on Machine Learning, pp. 26429-26446. PMLR, 2022.
Jiachen Zhao, Zhun Deng, David Madras, James Zou, and Mengye Ren. Learning and forgetting unsafe examples in large language models. In Forty-first International Conference on Machine Learning, 2024a.
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? arXiv preprint arXiv:2402.188/5 2024b.
Yiran Zhao. Wenxuan Zhang. Huiming Wang, Kenji Kawaguchi, and Lidong Bing, Adamergex: Cross-lingual transfer with large language models via adaptive adapter merging, arXiv preprint arXiv:2402. 18913, 2024c.
Yiran Zhao, Wenyue Zheng, Tianle Cai, Xuan Long Do, Kenji Kawaguchi, Anirudh Goyal, and Michael Shich. Accelerating greedy coordinate gradient via probe sampling. arXiv preprint arXiv:2403.01251, 2024d.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2024.
Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks aligned language models. arXiv preprint arXiv:2307.15043, 2023.
Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers, 2024.
A APPENDIX
A.1 PARALLEL NEURON DETECTION METHOD
Feed- Fred-FFowward Network (FFN) In the latest open-source models, when processing input e, the feed-forward network in certain layer is defined as
(FN = (SILU(W (Wate()))) Wap (x) W Woown:
where x E BlXddmaded is the embedding fed into the FFN, Wgate: Wup E Ridmodee X dinter
Wdown E Redenter x dmodel The calculation of the importance of the k-th neuron in Wup, when processing the input c, as presented in Equation 2 can be equivalently transformed
Imp(W.pl:: Al|c)= [|F - FFN(2))l2 = =1(( Mask (k)) W Waown
(8)
where hg E denttr represents the embedding before Wdownn and Mask [k] dinter is vector with the k-th element equal to 1 and the rest equal to 0. To calculate Imp W Wwpl: k]|c) ()) for l E dinter parallelly, introduce diagonal mask matrix of size (dinter, dinter), denoted as Mask. Therefore,
Imp(W cc) = (l(hma Mask) WWowwlll
(9)
Furthermore, observe that deactivating the k-th neuron of W down is equivalent to deactivating the k-th neuron Wap, as they both result in hmm [k] = O. Hence, we can also derive Imp W dowrn by employing Equation (
?W(-) represents the linear matrix product of the input and the parameter W, i.e., W(x) zW.
14

---

Published as a conference paper at ICLR 2025
When processing imput e, the self- network in certain layer is attention weight before and after deactivation, prior to scaling and softmax,
Self-Attention Network
W Wo( Wv(x),
Attention
Softmax
(10)
Vd
where We, WK, Wv E Rddmodes x Xdmod Since (v(x)) is not in the non-linear softmax calculation, calculate Imp(Wvle) c) by applying Equation 9 For Wq. obtain Imp(Wql: Aile) by deactivating its k-th neuron, specifically, We Wol+ ii = O. Firstly, we calculate the difference in
(x) Wq (x) (x(
(t) k:Wx (x) RiX
(11)
Next, as the changes in attention exhibit positive correlation with the changes in the output of this layer, the importance of Wal: in processing c, can be approximated
qm c) ~ ||ttention( (I) - attention(x)]1z
Wo (o( Aa(w)) -oftmaax
W (o(z)
(12)
softmax
Vd
Vd
This process can also be calculated in parallel, specifically,
A (a(+)WT(s) (((
(13)
Wo(o)..rsize((, 1, dmid) (x) dmia) Iixxxdmid
Therefore, the importance of Wq processing imput is calculated by Wo(x) (x) - A(a))
W Wo(x)
Imp(Wole) R softmax
softmax
(14)
vd
Vd
Similarly, since Wx is symmetrical to Wo. Imp(Wx (c) can calculated in the same way.
A.2 SAFETY NEURON DETECTION CORPUS
In the neuron detection process, we utilize the training documents from Zou et al. (2024), from which sampling 200 documents for detection. Specifically, the training set contains harmful queries across various categories, including "terrorism and violent extremism", "self-harm", and "political campaigning".. etc. This diverse dataset helps ensure the generalizability of the detected neurons. Furthermore, our analysis examined how the number of input documents affects safety neuron detection, shown in Table 5 The ablation analysis is Llama3-8B Insttuttct, and the results demonstrate that 200 documents are sufficient to reliably identify safety neurons.
Table 5: Number of detected safety neurons across different document sizes.
Corpus Size
10
50
100
200
400
800
Number of Safety Neurons
8912
4825
3594
2329
2322
2314
'In models like Vieuna and Mistral, dmodet dmids but we different notations avoid ambiguity.
15


Reviews:
[{"review_id": "Iqi0WlE3y0", "reviewer": "Reviewer_9nQR", "summary": "This paper presents an innovative method for effectively and efficiently detecting and fine-tuning \"safety neurons,\" which comprise less than 1% of model parameters and are predominantly located in low-level self-attention layers. The authors conducted related experiments to verify that safety mechanism is resilient but breakable. Notably, the proposed tuning method $\\texttt{SN-Tune}$ enhances model safety without sacrificing performance, significantly lowering harmful output scores in both instruction-tuned and base models. This approach also improves safety robustness during downstream fine-tuning tasks, such as GSM8K, by isolating safety neurons from foundational ones. Additionally, the authors explored the influence of the number of safety documents and performed grid searches over learning rates and training epochs in their ablation study.", "questions": "* In table 4, why you used llama2-7B-chat (which is a safety-aligned model) and  Mistral-7B-Instruct-v0.2 (which is not a safety-aligned model) for downstream fine-tuning tasks ? How about use Mistral-7B models that have been safety-aligned by your proposed method $\\texttt{SN-Tune}$?\n* This paper [1] also proposes some detection methods to find safety neurons and perform operations on the identified neurons to reinforce the model's safety. Could the authors discuss more between this paper and  findings of your paper?\n* There are several papers [1][2][3][4] also study on safety neuron/ safety mechanism/ backdoor, it will better if  the authors  could add some comments on them.\n\nIf the authors address most of my concerns, I would consider increasing the score.\n\n[1] Wei, Boyi, et al. \"Assessing the brittleness of safety alignment via pruning and low-rank modifications.\" arXiv preprint arXiv:2402.05162 (2024).\n\n[2] Chen, Jianhui, et al. \"Finding Safety Neurons in Large Language Models.\" arXiv preprint arXiv:2406.14144 (2024).\n\n[3] Hsu, Chia-Yi, et al. \"Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models.\" arXiv preprint arXiv:2405.16833 (2024).\n\n[4] Zhang, Zhengming, et al. \"Neurotoxin: Durable backdoors in federated learning.\" International Conference on Machine Learning. PMLR, 2022.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-written and the idea is easy to be understood.\n* Interesting verification of safety neurons. The authors used detailed experiments to verify that the identified safety neurons have a significant impact on the model's safety and found that these safety neurons are relatively few in number, primarily located in the self-attention layers of the first few blocks of the model.\n* Extensive experiments. The authors validated the effectiveness of $\\texttt{SN-Tune}$ on both instruction models and base models, as well as on downstream fune-tuning tasks.", "weaknesses": "* Although authors  empirically verify the identified safety neurons by their detection method have a significant impact on the model's safety, the proposed detection method lacks theoretical support for why it can identify neurons that have such a significant impact on safety.\n* Why does the RHS of equations（4）（5） of the main paper  seem to be  independent of $l$ (the $l$-th neuron in the $i$-th layer)? They appear to be the same value for different $l$. Also in appendix, it is confusing why $h_{ffn}$ is a vector, could you help write the specific form? And how can you get (8) from (7) since x is a input for $W_{down}$  in (8) but not for $W_{down}$ in (7)?\n* For downstream fine-tuning tasks, could you consider more datasets like Alpaca or Dolly to verify the effectiveness of $\\texttt{RSN-Tune}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aZ4Hh6uMh5", "reviewer": "Reviewer_Uh2w", "summary": "This paper introduces the crucial concept of \"safety neurons\", which investigates neural activation patterns when large language models encounter unsafe instructions and jailbreak attacks. The definition of \"safety neurons\" is concise, clear, and effective. Furthermore, the paper presents several significant findings: the proportion of safety neurons is remarkably small, constituting less than 1% of total parameters; the robustness study of safety neuron locations reveals sensitivity decreases from front to back layers, with safety neurons in the first 10-20 layers being more sensitive; and the transferability of safety neurons across multiple languages is investigated. Additionally, the authors propose the SN-Tune method, which exclusively fine-tunes safety neurons, achieving enhanced safety conditions by tuning only a small subset of parameters without catastrophic forgetting of general knowledge. The paper is easy to follow and well-written, with clear methodology explanations, comprehensive experimental results, and logical organization of findings.", "questions": "1. Position-Dependent Activation Patterns:\n- How does the position of jailbreaking prompts affect safety neuron activation? Is there a correlation between prompt location and safety neuron response?\n- Do safety neuron distributions shift in long-text scenarios？How does the context window length affect the stability of safety neurons?\n\n2. Dynamic Nature of Safety Neurons during Fine-tuning:\n- Can non-safety neurons transform into safety neurons during full-parameter fine-tuning? How does SN-Tune handle the potential emergence of new safety neurons?\n- How can we track and verify the transformation of regular neurons into safety neurons?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Originality: The paper introduces an innovative approach to understanding LLM safety through the novel concept of \"safety neurons.\" The clear definition and identification method for these neurons represents a significant departure from traditional safety alignment approaches, offering a fresh perspective on model safety mechanisms.\n\n2. Experimental Rigor: The research demonstrates exceptional thoroughness in its experimental investigation, providing comprehensive analysis of safety neurons' characteristics, including their proportion (<1% of parameters), location sensitivity, robustness, and cross-lingual transferability. Each finding is well-supported by detailed empirical evidence.\n\n3. Practical Impact: The proposed SN-Tune method offers a highly effective solution for enhancing model safety while maintaining general capabilities. Its ability to achieve significant safety improvements by tuning only a small subset of parameters makes it both efficient and practical for real-world applications, demonstrating immediate value for improving existing language models.", "weaknesses": "1. Theoretical Foundation: The paper lacks deeper theoretical analysis of observed phenomena. For example:\n- The mechanism behind the \"back-to-front deactivation\" pattern remains unexplained, where safety breakdown only occurs after almost all safety neurons are deactivated\n- This observation raises questions about whether the neurons in later layers are truly \"safety neurons\"\n- The nature and characteristics of overlapping safety neurons across different languages deserve more theoretical investigation\n\n2. Limited Analysis of Cross-lingual Safety Mechanisms:\n- While the paper identifies low overlap (30%) between safety neurons across languages, it doesn't explore:\n  * The characteristics of these overlapping neurons\n  * Why certain neurons are shared across languages while others are language-specific\n  * The potential universal principles of safety mechanisms across languages", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTIrLJuOv0", "reviewer": "Reviewer_Qmyn", "summary": "The paper presents a method for identifying safety neurons in Large Language Models (LLMs), which are critical for managing harmful queries and represent less than 1% of the model parameters. It introduces a technique named SN-Tune that focuses on tuning these safety neurons while maintaining the overall capabilities of the models. The findings indicate significant reductions in harmful scores, with Llama3-8B-Instruction decreasing from 65.5 to 2.0. Furthermore, the approach improves safety robustness during fine-tuning by separating safety neurons from foundation neurons.", "questions": "please see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper writing is good and very easy to follow.\n2. The defense performance is strong, even compared with current SOTA defense method.\n3. The logic of this article is also very coherent. The author first suggests enhancing the effectiveness of the model's safe responses through a more detailed approach to neuron control. The author also conducted relatively detailed ablation experiments to support the faithfulness of safe neurons.", "weaknesses": "I believe the biggest shortcoming of this paper is that the description of the experiments for identifying safe neurons is very insufficient. The author does not mention the datasets used, parameter details, or the time costs associated with the experiments. The dataset is crucial because it relates to the generalizability of the method. For example, if the author only uses datasets related to violence to identify neurons, but the identified safe neurons still demonstrate good defensive capabilities against attacks related to pornography or other safety categories, including jailbreaking attacks, it suggests that safe neurons are relatively fixed within LLMs. However, the author does not mention this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NW5VNmHpWY", "reviewer": "Reviewer_R3v2", "summary": "The authors first introduce a technique for detecting LLM neurons (i.e. rows or columns in LLM weight matrices) responsible for safety (e.g. refusing to comply with harmful instructions). The method is fairly simple: a neuron is said to be a safety neuron if deactivating it changes the final next-token embedding by at least some threshold (measured in Euclidean distance) on each of a set of inputs. They validate that deactivating these neurons indeed increases harmfulness scores of three open-weight models, while preserving performance on capabilities benchmarks.\n\nThey then propose a safety fine-tuning method, named SN-Tune, which tunes only these safety neurons, keeping the remaining ones fixed. They show SN-tune significantly reduces harmfulness scores across several open weight models; e.g. Llama3-8B’s score decreases from 65.5 to 2.0. Finally, they propose a refinement of SN-Tune, termed RSN-Tune, which also identifies neurons responsible for model capabilities, and only tunes safety neurons which do not overlap with such capabilities neurons, hence mitigating capability degradation during safety tuning. Additional experiments include a study of the overlap of safety neurons across languages (which they find to be small) and of applying SN-tune to pre-trained models (rather than already-fine-tuned models).", "questions": "1. Would it be correct to conclude from lines 362-363 that your results indicate that base models already contain safety neurons, despite not being safety-tuned? \n\n1. It could be useful to have a study on the distribution of safety neurons in the base models to verify they share similar properties to those in fine-tuned models, i.e. being concentrated (<1% of parameters) and being predominantly present in earlier layers and attention modules. \n\n1. Similarly, one could study whether their harmfulness scores get somehow even worse when these neurons are disabled. To make this non-trivial, you could have “safe-behavior-inducing” few-shot examples in the prompt, for example.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Simplicity of safety neuron detection: their notion of safety neuron is conceptually simple and can be computationally ascertained (instead of relying on manual analyses). Their later experiments indicate that one can indeed obtain a useful safety tuning algorithm by using their safety neuron detection method.\n\n1. Empirical insights into how safety mechanisms are implemented in LLMs: the paper’s results indicate that safety neurons are very concentrated (1% of parameters), occur primarily in self-attention modules and in earlier layers, and are mostly not shared across languages. These insights seem relevant for the community as a whole, particularly when devising defenses to multi-lingual attacks.\n\n1. Relevance of results for practitioners: one of the biggest hurdles in safety tuning is the performance degradation it usually entails if done in a naive way. The results in the paper suggest SN-tune might be useful for practitioners looking to safety-tune models without harming their capabilities. In particular, one could in principle automatically detect safety neurons using a small corpus of examples, rather than requiring manual analyses or large corpora.\n\n1. The paper is written mostly clearly and was easy to follow.", "weaknesses": "1. Lack of detail on how foundation neurons are detected: in Section 4, the authors propose not tuning safety neurons that also serve as foundation neurons (i.e. play  a role in model capabilities). However, they do not clarify how they detect the foundation neurons. This harms the reproducibility of the RSN-tune results.\n\n1. Notation and presentation in Section 2.1: I would recommend revising the presentation in the Accelerated Safety Neuron Detection. For example, the authors write that Mask is an identity matrix, which, as stated, cannot be true. Looking at the appendix, it seems they might mean it is a binary matrix (i.e. the entries are 0 or 1). Also, the authors could clarify what they mean by parallel computations. Does this simply mean adding many computations in a batch? Generally, it seems that their notion of safety neuron is embarrassingly parallelizable, in the sense that their criterion can be computed independently for all neurons.\n\n1. Missing citation of Kotha et al. (2023) in section 2.3.2: one relevant work concerning multi-lingual attacks on LLMs is Understanding Catastrophic Forgetting in Language Models via Implicit Inference, by Kotha et al. 2023. They show that translating harmful instructions into low-resource languages can elicit harmful responses, which can then be translated back to the original language. Their results are complementary to the author’s findings that safety neurons have little overlap across languages.\n\n1. Another work the authors do not discuss, but which I believe is relevant and complementary to their results, is Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks, by Jain et al. (2024). Amongst other results, Jain et al. also find that the effects of fine-tuning can be neutralized by pruning a small number of neurons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Yw18iMDSM", "reviewer": "Reviewer_WgZu", "summary": "This paper identifies safety neurons that are associated with safety alignment in LLMs. The authors claim that the safety neurons constitute less than 1% of all parameters, and they are predominantly located in self-attention layers. Experimental results have been provided", "questions": "* From which figure/table can we infer that “1%” are safety neurons?\n* Why were helpfulness scores not reported?\n* Why were Llama2 instruction-tuned models not reported in Table 2? It is known that Llama2 is more resilient than Llama3 in terms of instruction following. Hence, the Llama2's results for Fig 2 need to be reported.\n* What’s language X and Y in Fig 5? \n* Where are the results of the five languages? \n* What is the objective of having section 2.3.2 regarding the theme of the paper?\n* What is “Language-specific neurons” in line 291? It has not been defined nor introduced beforehand.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper deals with an important topic, the safety issue of LLMs – for safety alignment.", "weaknesses": "- My biggest concern is the distinction between paper, Zhao et al (2024b) (How do Large Language Models Handle Multilingualism? Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, Lidong Bing 2024).  I believe this submission’s section 2.1 is the meat of the work, but the formulations are all found in Zhao et al (2024b) – just the same. Then, this submission’s distinctive contribution is very unclear. The only difference is (3) where the authors feed harmful queries and see the activations (although the sentence in lines 124-125 is broken, so it is speculated. It is impossible to know their actual implementation.) However, the entire formulations is just the same with the other paper, with only a minor tweak with harmful queries for safety context.\n\n- Moreover, the authors failed to properly provide a reference for the formulation/equation (1) and (2) and give credit to the authors of the paper. The paper is referred to only for the parallel neuron detection method. ((4), (5), and (6)). \n\n-\tMoreover, in terms of the paper’s direction and theme, this submission’s distinctive novelty from the following paper is unclear: “Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications,” Boyi Wei · Kaixuan Huang · Yangsibo Huang · Tinghao Xie · Xiangyu Qi · Mengzhou Xia · Prateek Mittal · Mengdi Wang · Peter Henderson, ICML 2024. This paper also identifies safety components in LLMs.\n\n-\tUnlike the authors’ statement, “Regarding general capability, deactivating the safety neuron shows minimal impact, “similar to deactivating randomly selected neurons”,……” in Table 1, Deact_SN’s Avg. Capability is constantly, always lower than Deact-R, which hints that the safety neurons are also contributing to general capability. If the weights were purely safety neurons, when they were pruned, they shouldn’t impact the general capability even with no need to compare it with Deact-R. But, the table result shows there exists an impact on general capability.  \n\n-\tIn order to conclude “Safety neurons predominantly reside within the self-attention layers.”, the authors should have reported the original number of neurons and the proportion of safety neurons in feed-forward and self-attention layers, respectively – not just the split.\n\n-\tFor section 3, “Efficient Safety Training” the authors claimed SN-Tune is efficient by comparing the training cost with Circ-Break. However, SN-Tune requires a process and resources and effort to “identify” the safety neurons. Only after the neurons are obtained, the SN-Tune can be applied. Hence, it is not a fair comparison. The cost/effort for identifying the safety neurons must be taken into account, but they were not in this submission.\n\n-\tIt is unclear the impact of RSN-Tune compared to SN-Tune by looking at the results of GSM8K in Table 4.\n\n-\tThis submission lacked an important and nominal paper in reference, with regard to neuron importance, Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)