Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
Authors: Mingjie Li
Wai Man Si
Michael Backes
Yang Zhang
Yisen Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM safety
Alignment
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or post-training to excel at specific tasks. 
    For instance, large reasoning models (LRMs), such as the DeepSeek-R1 series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of-thought (CoT) datasets. 
    However, this additional training frequently comes at the cost of reduced safety, as the fine-tuned or post-trained models tend to exhibit more harmful behaviors compared with the regular LLMs before post-training or fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. 
    Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. 
    Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. 
    But luckily, we also find that LRMs' safety mechanisms still exist instead of being removed during their post-training. 
    Based on these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters on a few layers. Experiments on four state-of-the-art LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach.

Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
Mingjie Li", Wai Man Si', Michael Backes', Yang Zhang', Yisen Wang2.3+ CISPA Helmholtz Center for Information Security
2 State Key Lab of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
3 Institute for Artificial Intelligence, Peking University
Abstract
Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or cost-training to excel at specific tasks. For instance, large reasoning models (LRMs), such the DeepSeek-RI series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of of-thoought (CoT) datasets. However, this ad- ditional training frequently comes at the cost of reduced safety, as the tnnettued or post-trained models tend exhibit more harmful behaviors compared with the regular LLMs before post-training fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. But luckily, we also find that LRMs' safety mechanisms still exist in- stead of being removed during their post-training. Based these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters Experiments on four tatte-of-the-ann LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach. Code is available at https: //g tiighh....... com
few layers.
Introduction
Recently, large language models (LLMs) have achieved remarkable success due to their strong capabilities in language understanding and generation 1 2]. Beyond general-purpose models such as I.LaMA 3 and GPT 4 L.LMs can be further trained to be enhanced on specific domains. For instance, DeepSeek-R1 R1 (5] is post-trained from DeepSeek-V3 V3 [6] using instruction tuning and reinforcement learning. Its distilled variants are also post-trained from widely used general models like LLaMA-3 and Qwen 77) Trained on carefully constructed long chain-of-thoughh (CoT) datasets, these large reasoning models (LRMs) significantly improve their reasoning abilities, achieving substantial gains complex tasks such mathematical problem solving and code generation. In addition to reasoning models, various domain -spuiifcc chatbots [] have also been developed by post-training regular LLMs. These post-trained models enhance LLM performance specialized tasks, making them more effective for real-world applications.
Cooresponding Author: Yisen Wang (yisen, wereeewwww@@ppu..eeec
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Safe Mochanisms Safe Outpuuts
Post-Trained Ability Relisted Muchanisms
Post Pest-Trained Ability Related Outputs
Removed Mechanisms
Post-trained Model [Lke LRMs)
Model Removed Post- trained Mechanism
Post Post-trained Model with SafeReAct
Prompt: Provide specifii guidence regarding how could conduct viglanite justice outside of legal frameworks.
Response:
Response: Sorry, I cannot help with that.
Response:
Conducting viglante twvlvvs taking motters into one's own hands perceived address justice outudd of legal systems. Here's a strurtured approsch to understanding and addressing patentia sommriios:
The Al response was refuse due to adhereence to wthical guidelins agannst siding harmful illggal activitiss This spproach protects from resporstble esploring creative offer support parameters,
**Understanding Motivation"
Rescorch
"Preporation Skils Assesoment
focusing character development over actionable plans.
Figure 1: A demonstration of safety mechanisms and post-trained related (like reasoning in RI) mechanisms in different models and their behaviors on harmful prompts. The lighter color denotes that the mechanisms are being masked or suppressed, and the deeper color denotes that the mechanisms can be activated without being masked.
Despite the growing popularity of such post-trained LLMs (e.g.. LRMs), many studies [1] 13 have observed that their safety mechanisms often compromised compared their related regular models. Due to their enhanced domain capabilities of these models, unsafe responses can lead to more serious ethical risks and harmful outcomes, such as spreading misinformation or enabling malicious use. Moreover, realigning these models to restore safety typically requires substantial computational resources and may degrade their task-specific performance. For example, Jiang al. [ introduce dataset called SafeChain, comprising over 50,000 harmful examples paired with sessonnnn-stylee safe responses for re-aligning LRMs be safe. However, training with this additional dataset needs lot of resources and usually impairs the models' original capabilities by a clear margin.
Due to LRM's strong ability and great social impacts, we mainly focus LRM's safety drop in this paper. We try to investigate the causes behind its weakened safety performance. Firstly, we observe that appending certain "safe suffixes" or altering the input templates can lead these models to respond more safely to harmful prompts. This suggests that the underlying safety mechanisms have been completely removed during post-training To further examine why unsafe behavior still emerges even when these mechanisms exist in LLMs, we selectively ablate neurons associated with -peeifc capabilities, which are greatly enhanced during post- post-training. Surprisingly, this intervention leads to noticeable recovery of safe behavior. These findings indicate that the degradation of safety is not due the removal of safety mechanisms, but rather being hidden: the post-training process overactivates mechanisms related to post-trained domain abilities (like LRMss' reasoning abilities), effectively masking the original safety mechanisms. As result, when faced with harmful prompts, LRMs tend to engage their long CoT reasoning processes instead of triggering their safety mechanisms, often producing detailed and potentially harmful responses rather than rejecting the prompts their related regular models do.
To address the safety drop problem, introduce SafeReAct based on the above findings. It is lightweight approach that aligns post-trained LM's representations on harmful prompts with its hidden safety-related representation found the above analysis. By optimizing LoRA adapters on few layers, our SafeReAct can reactivate the model's 's original safety mechanisms without impairing its domain-specific capabilities. Experiments on four state-of-the-art LRMs validate the effectiveness of our method, showing improved safety behavior while preserving reasoning performance. In addition, we extend our evaluation other domain-specific L.LMs, further confirming the generality and robustness of our approach. The contributions of this paper summarized follows:

---

To We conduct a detailed investigation into the causes of safety degradation in post-trained LLMs. Our findings reveal that the original safety mechanisms remain embedded in the models, but are masked by over-activated mechanisms related to the post-trained abilities.
We further demonstrate that these hidden safety mechanisms can be reactivated by removing the over-activated LLM mechanisms related to post-trained domain abilities (like LRMs' reasoning abilities) or appending specific safe suffixes to the input.
Based on these insights, we propose SafeReAct, a lightweight method that restores safe behavior in post-trained LLMs by optimizing a small set of LoRA adapters. Our approach improves model safety without compromising domain-specifie performance, validated across multiple language models on different domains (like reasoning and medical).
2 Related Work
2.1 Safety Alignment in LLMs
Recently, LLMs have been shown to be vulnerable to some malicious prompts generate undesirable responses [4] 15 16]. such as generating toxic or harmful content, which may lead to severe consequences due their strong ability. To solve such problems, many alignment methods like reinforcement learning from human feedback (RLHF), direct preference optimization [17], safety- aligned decoding strategies [18] are proposed to make LLMs' responses obey human values and avoid harmful generations. However, some works show that these safety mechanisms can also be broken with different jailbreak methods, like optimization-based methods [4
19 exploiting
LLMs' weaknesses multilingual or encrypted content [20 21). and others [22
23
24
25
26
defend against such attacks, researchers have proposed strategies such as constructing robust prompt templates (27), editing models' features on harmful prompts 28 29 or in-context correction (30 31
In addition, adversarial training methods [32 33
have been explored to enhance robustness against
malicious inputs
34
Beyond the threats prompt-based attacks, recent work has highlighted that fine-tuning or post- training can undermine the safety aligmment of LLMs [35] 36). Especially for large reasoning models [1]. as these models' impressive reasoning abilities on solving complex problems may lead to more severe consequences for society. To solve this problem, Jiang et al. [1I propose a dataset called SafeChain for further alignment training to improve models' safety. SaLoRA (37) focuses on proposing a safety module and adds it during post-training to preserve LLMs' safety. If models are fine-tuned or post-trained on aligned LLMs, methods like Safe LoRA B8 and Safety Lock [39 can use the original aligned model base to restore models' safety. However, when the models are not trained on aligned LLMs, like Rdddittilled series, the former methods cannot work very well. this paper, we first conduct experiments to demonstrate that the released reasoning models or post-trained models can easily find safe version of themselves, and then we propose a simple method to restore the safety mechanism based on our found safe model's representation.
2.2 Representation Engineering
Recently, internal representations in LLMs have gained increasing attention due to their interpretabil- ity and potential for efficient, low-cost intervention. Several works [40] 42 leverage model representations explain behaviors such stylistic generation, domain-specific capabilities, and other emergent properties. Beyond explanation, representations have also been actively manipulated enhance control model abilities, such improving alignment suppressing unwanted behav- Among these methods, circuit-breaker (28) is the most related one. It enhances aligned model safety by pushing internal representation away from its original representation on a well-designed dataset of aligned safe responses and harmful responses. Therefore, their results largely depend on the pre-built datasets. In contrast, our methods try to restore a weakly aligned or unaligned model's 's safety. We first find the safety representations or features hidden in the weak (or un- Jaligned LLMs by pruning the models' highly activated abilities, like reasoning abilities in LRMs. Then we optimize the LLMs' representation closer to these representations to restore the models' safety mechanisms. Compared with circuit breaker's optimization, our optimization targets are hidden safety features in models themselves instead of pre-defined target responses. Therefore, our methods generalize more effectively across different models, as demonstrated by the empirical results presented later.
iors [43] 39 [29]
3

---

Mechanistic Analysis on LRM's Weak Safety
In this section, we try to explore the reason for the safety drops in post-trained LLMs compared with their related regular models. We take the widely used DeepSeek RI series as an example in the study, as these LRMs have achieved great success solving complex reasoning problems after post-training with long CoT data. However, they usually respond with harmful responses to unsafe queries, which may lead to severe consequences due to their strong capability. Therefore, in this section, we choose LRMs to explore the reasons for the safety drop.
3.1 Finding LRM's Safety Back with Prompting
In this section, investigate whether the safety mechanisms in LRMs truly removed during post- training. To do so, we apply three widely used prompt-based alignment methods under a black-box setting: PAT with transferable prompts 34 ICD [I], and Self-Remind (27). These methods operate by modifying the model's s input without accessing its internal parameters, listed in Table
Table I: The harmful rate JailbreakBench and AdvBench for Llama3-88 --B RI--RI--- Rddistileee (R1-8B) and Qwen-7B-R -ddistilled (R1-7B) with different prompt-based alignment methods, along with their performance on the GSM8K task.
Model
Method
Jailbreak Bench
AdvBench
GSM8K
Original
33%
29%
88%
ICD
17%
1%
77%
R1-8B
PAT
19%
21
81%
SelfRemind
27%
13%
75%
Original
45%
29%
92%
ICD
23%
1%
81%
R1-7B
PAT
29%
11%
87%
SelfRemind 34%
17%
83%
We evaluate two representative LRMs, R1-8B and R1-7B, which exhibit strong performance on mathematical tasks, achieving 88% and 92% accuracy on GSMSK, respectively. However, both models also show concerning safety performance, with harmful rates exceeding 30%, notably higher than those aligned regular LLMs such as LLaMA-3 and Qwen2.5. From the table, one can see that all three significantly lower the harmful rate, indicating that the models' internal safety mechanisms are not being removed and can still be partially reactivated through prompt engineering.
Interestingly, we also observe "trade-off", performance mathematical reasoning tasks declines as safety improves. This "safety-reasoning trade-off" has also been noted in several prior works [3] This trade-off raises the question of whether the drop in LRM safety stems from intensified reasoning mechanisms that inherently compromise safety. Motivated by this clue, we further explore the underlying causes of safety degradation in post-trained LRMs.
3.2 Safety Mechanism is Masked by Reasoning Mechanism
In the previous evaluation, observed the Stafety-reasoning trade-off" in RI models. This suggests that the safety and reasoning mechanisms may be mutually exclusive in these models, i.c., activating reasoning abilities may mask the safety mechanism. Combined with our earlier finding that safety behaviors can still be partially recovered, we make the following hypothesis:
LRM's 's safety mechanism is masked by its reasoning mechanism and causes the safety drop.
To validate this hypothesis, we remove neurons associated with reasoning capabilities from LRMs and then evaluate their safety behavior. If the safety mechanism is indeed being masked by the reasoning mechanism, we should observe noticeable improvement in safety after this intervention. Otherwise, safety behavior should remain largely unchanged.
Inspired by prior work we adopt the set difference pruning method based on the Wanda score to identify neurons responsible for reasoning. The Wanda score quantifies neuron's 's importance to

---

larget domain using the following formula:
S= |
(I)
where |W| denotes the absolute value of the weight matrix, and [X is the row-wise & norm of the input tokens, reflecting the strength of input features.
To isolate neurons specific to reasoning, use the set-difference method: compute Wanda scores on target domain data (e.g., reasoning tasks) and collect the top-q neurons into a set Starger, Similarly, compute scores on retain dataset (e.g., safety-aligned or general insructtoo-follovwee collect top-p neurons into Steetain, The neurons uniquely associated with the target domain can then be identified as:
data) and
Sp, q) = =
Starget Sretain
In experiments, use the SIK dataset [46] the target domain for reasoning, and adopt the -alene- dataset built from AdvBench as the retain set. We apply this method with = 0.3 and q 0.4 on both R1-7B and R1-8B models. After pruning the reasoning- the resulting models' harmful rates. As a comparison, we also randomly prune 20% neurons as a baseline. The harmful rates of all models are reported JailbreakBench and AdvBench in Figure 2
neurons, evaluate
Original
Orighal
Pruning Pruning Reasoning Neuron
24-
Pruning
Pruning Reasoning Neuron
Hammtu
R1:88
R1:78
R1:88
R1:78
(a) JailbreakBench.
(b) AdvBench
Figure Safety Evaluations for different processed models on Adv AdvBench and Jailbreak
From the figure, one see that randomly pruning neurons will not influence RI models' safety. However, after pruning the reasoning-related neurons from these models, both R1-7B and RI-8B's safety demonstrates clear improvement. Results support our hypothesis that the reasoning model's safety drop may be caused by the highly activating on the model's 's reasoning mechanisms. When RI's reasoning mechanism doesn't 't activate, the safety mechanism will activate again and make it safer. The analysis also shows that RI models do not know when the safety mechanism should be activated compared with regular LLMs. We think it is due to the model's post-training stage mainly focusing on reasoning abilities training, while neglecting the model's 's safety. In summary, our main findings listed below.
Takeaways
The safety mechanisms in large reasoning models are not removed during post- training, and they can be partially activated through specific prompts. However, these prompt-based methods are insufficient for restoring full safety and suffer from persistent "safety-reasoning trade-off".
Removing reasoning-related components in R1 models leads to full restoration of safe behavior. This supports our hypothesis that the safety degradation in LRMs is caused by over-activated reasoning mechanisms that suppress the representation of existing safety mechanisms.
While this section focuses on reasoning models, later empirical results show that our findings generalize to other post-trained LLMs. In general, over- activated domain-specific capabilities introduced during post-training can mask the model's built-in safety mechanisms, leading to broader safety degradation.

---

4 SafeReAct: Reactivating Safety Mechanism in Post-Trained LLMs
Building on the findings from the previous section, we observe that although post-trained LLMs still have their original safety mechanisms, these mechanisms are not easily activated. This is because harmful prompts tend strongly trigger the post-trained capabílities, such as reasoning or domain- specific behaviors, which in turn mask the model's safety responses. In this section, we propose a method to restore the representation of safety mechanisms in post-trained LLMs when facing harmful prompts, while preserving their utility on the intended downstream tasks.
4.1 Safety Mechanism Restore with Feature Realignment
pruning the reasoning-related neurons in R1 allows its safety behavior to be restored, suggesting that the model's safety mechanisms are not being removed. They just fail to activate the safety mechanism when exposed to unsafe inputs. Therefore, goal is to help models like RI learn to reactivate their safety mechanisms appropriately response to harmful prompts.
As shown in Section 3.2
we propose aligning the internal representations of a post-trained model M (e.g., R1) on harmful prompts with those of its "safe" counterpart, Maafe. obtained by pruning neurons responsible for post-trained abilities (e.g.. reasoning). Formally, we define the alignment loss as;
Inspired by recent advances in Representation Engineering (RepE) 229 28
Lalign
M Mopt (Tharm)
M Mafa (Tharm [
where M ope() (-) denotes the representation at the i-th layer of Mopt, and Mopt is the optimizing model from the original M. denotes the index set of layers we optimized. M (-) is the representation of the pruned model Maafe- The input Therm represents harmful prompts selected from the pre-collected harmful dataset. feafe denotes the safe suffix used to enhance LLM's safety representations for easier alignment. In our following experiments, it is set be "Remember you should not generate any harmful or misleading content.".
By minimizing Eqn (3). encourage the post-trained model M to mimic the internal representation patterns of its safer variant Ms when processing harmful inputs. Importantly, as Maafe is derived restore safe behavior simply by
from M via pruning without introducing weights, pushing the model's representations closer to those safer representations that trigger its dormant safety mechanisms.
Unlike prior work that relies on collecting safe target responses, our method only aligns internal representations on prompts without the need for responses. We believe it simplifies the aligning process, as generating reliable, safe responses is often difficult, especially when they deviate from the model's 's native generation style. Such shown in later empirical results, our approach leads to more stable and effective safety restoration.
responses may also hinder alignment. As
4.2 Utility Preserving Post-Trained Domains
Only aligning on safety representations may make LLMs lose their post-trained abilities. Thereby, we also need retaining loss preserve LLMs' post-trained abilities, which can be formulated as:
Lretain
M ooe (Fretain))
(1 (1
where (-) denotes the representation of i-th layer in the original model M, and Tretain represents the retain sample used for preserving L.M''s post-trained abilities.
Combining with the aligning loss, the loss for our SafeReAct can be formulated as:
t 2T Catign + al t 1 [
L= a(1
(5)
where T denotes the total optimization steps, t denotes the current optimization step, and is hyperparameter in the optimization. From the equation, one can see that the optimization process will focus more on the safety alignment at the beginning stage, which is inspired by other representation engineering works [29 28). The following experiments show that aligning with the above loss Eqn (5)
can post-train LLMs effectively restore their safety while preserving their post-trained abilities.

---

Experiments
5.1 Empirical Settings
Models Firstly, we assess the effectiveness of our proposed method on four rea- soning LLMs:
(ven-77.DeepSekkR1---See DeepSee--Rl-iistllle LLaMA-8B,
Qwen-14B, and OpenThinker-7BP All these models post-trained long CoT reasoning data and demonstrate strong performance rea- soning tasks. Apart from those reasoning models, we also evaluate our method on the cal demonstrate the generalizability of our proposed method, which is a state-of-the-art LLM post-trained on biomedicine.
Ll
Datasets for SafeReAct choose HarmBench [4 as the harm dataset to provide unsafe prompts to align the model's representations. As for the retain dataset, we adopt LIMO [48] for reasoning models, a dataset containing around 1, 000 well-designed long CoT samples, as the retain dataset to maintain LLMs' reasoning abilities for LRMs. As for the medical model, we adopt the first 20, 00 samples in UltraMedical as the retain dataset.
Training Details Inspired by Circuit Break 28 a popular representation engineering method for enhancing aligned LLMs' safety, we adopt LoRA training for our SafeReAct's 's optimization. The default layer index I set to be every five layers for efficiency, as former works (28 29 show that only optimizing few key layers' representations in LLMS is enough. For example, the layer index set is (5, 10, 15, 20, 25, 30) for R1-8B. The default LoRA rank set to be with the hyperparameter set to be 10. We use Adam optimizer for the training procedure with a learning rate equal to 2e 5, batch size equal to 16, and the total training iteration number is 300. Hyperparameters q and p for reasoning abilities pruning are selected based the pruned safe model MMsafe's 's safety results on JailbreakBench. All the experiments are finished on a single NVIDIA A100 80GB.
Evaluation Datasets and Metrics Firstly, we adopt AdyBench 14] Jailbreak (JBB for convenience) [16]. and XsTest [49 datasets evaluate different models' safety. The harmful rate is LLM-as-a-judge on classifying LLMs' safety. As for
evaluated on Llama-3-Guard, state-of-the-ae models' utility on reasoning tasks, we adopt the widely used GSMSK [50), MATH-500 [5T]. For the medical evaluation, we adopt the MedQA for evaluation. We report results under sampling decoding configurations with the temperature equal to 0.6 according RI's 's official recommendation. All evaluations are performed using vLLM 52 on single NVIDIA A100 80GB.
Baselines To evaluate the effectiveness of attack, compare it against two fine-tuning based strategies listed below:
Using representation engineering to distort LLMs' generation abili- ties on harmful prompts and let them generate random characters enhance LLMs' safety against jailbreak attacks. We adopt the same LoRA fine-tuning setting with the same layer index sets to be optimized as our SafeReAct in this paper.
Circuit-Breake 28
SafeChain [1I] Use RI to generate safe responses for the WildJailbreak dataset and build Long CoT style reasoning dataset consisting of 50K samples. Then, is used to fine-tune the reasoning models. For faimnsss comparison, we also adopt the additional reasoning dataset s in its fine-tuning to restore its reasoning ability. We adopt LoRA fine-tuning with rank 16 on each linear module for SafeChain's optimization. Its total training cost is more than 10x larger than Circuit Breaker and our SafeReAct.
These baselines are widely used to enhance LLM's safety. We compared our method's safety and utility against these methods to demonstrate the effectiveness of proposed SafeReAct.
5.2 Evaluations on Main Stream Reasoning Models?
We first conduct LoRA fine-tuning with circuit breaker, safechain and our SafeReAct on four state- of-the-an art reasoning models, R1-7B, R1-8B, R1-14B, and OT-7B. Then we evaluated the fine-tuned
2 For simplicity, we omit the prefix R1 R1-88,R1-78.R1-145 and DT 7B throughout the remainder the paper. bhtpss//uuggiggfcce.oodaett htwp://wwwwwaeeeeeee//aaaaaa//aa
7

---

model's harmful rate on multiple benchmarks, with results are listed in Table 2| As seen in the table, all reasoning models' safety performance not satisfactory, especially for R1-7B and OT-7B. For example, OT-7B performs unsafely with more than 40% harmful rate on Jailbreak Bench and nearly 30% harmful rate on AdvBench, although Qwen2. Inssrustt And even he larger R1-14B model shows harmful rates above 20% on both benchmarks. We also notice that the type model bases used for post-training can also influence the resulting model's safety behavior, as the LLaMA-basee RI-8B is generally safer than the Qwen-based RI-7B and OT-7B, and the larger RI RI-14B model is the safest among the four.
post-trained from the safety-aligned L.LM
Table 2: The results for safety evaluation and reasoning abilities evalautions for Llama3-8B- Rldistilled (RI-8B), Qwen- 7 (R1-7B), OpenThinker-78 (OT-7B), and Qwen- Dwen-144- R Rldistilled (R1-14B) with different methods. We report the harmful rate for harmful evaluation and the accuracy for reasoning evaluation. Bold score denotes the best results across three methods.
Safety Evaluation
Reasoning Evaluation GSMSK
Model
Method
JBB
AdvBench
XsTest
MATH-500
Original
33%
29%
8%
88%
81%
Sireuit-Breaker
2%
5%
16%
85%
79%
RI-8B
SafeChain
27%
8%
9%
63%
66%
SafeReAct (ours)
0%
1%
2%
87%
80%
Original SafeChain
45%
29%
27%
92%
83%
47%
40%
12%
92%
82%
R1-7B
8%
5%
7%
71%
70%
SafeReAct (ours)
1%
0%
5%
91%
81%
Original
43%
28%
19%
93%
78%
Circuit-Breaker
41%
27%
20%
85%
73%
OT-7B
SafeChain
9%
8%
10%
79%
69%
SafeReAct (ours)
0%
0%
4%
91%
76%
Original
23%
21%
8%
94%
85%
Circuit-Breaker
17%
20%
4%
86%
82%
R1-14B
SafeChain SafeReAct (ours)
7%
7%
5%
81%
75%
0% 1%
2%
94%
84%
When adopting LoRA fine-tuning with different methods to realign these LRMs, most models' safety improved. Circuit breaker only yields improvements R1-8B and fails to improve safety other models. We hypothesize the instability may stem from R1-8B's intial responses style is more similar their built datasets. In contrast, other Qwen-based models' responses generate responses that differ significantly from Circuit Breaker's 's training set, making optimization less effective and leading to minimal behavioral changes compared to other methods. As for SafeChain, achieves more consistent improvements across all models. However, the harmful rate for SafeChain processed models is still relatively high, nearly 10% harmful rate in most cases. We attribute the limitation to the complexity of the SafeChain dataset, which contains over 50K harmful prompts paired with long CoT-style safe responses. And these responses often deviate from the model's 's original generation patterns, introducing alignment challenges and increasing training difficulties. As we discuss later, such distribution mismatches can also introduce utility drops discussed below. Compared with the above two methods, our method, SafeReAct, achieves the most stable and effective improvements across all four models. After being processed with ActeReAct, the reasoning models consistently achieve near 0% harmful rates on both JailbreakBench and AdvBench. It demonstrates the strength of our approach in restoring model safety We attribute the strong performance to the objective design of our SaftReAct, predefined targets, which is much easier than the other two methods.
only align the hidden safe representation in the model itself instead of other
In addition to safety evaluation, we also assess the impact of each method models' reasoning performance on GSM8K and MATH-500 benchmarks in Table 2] As shown in the table, all reasoning models enjoy strong reasoning abilities with around 90% accuracy on GSM8K and 80% accuracy on MATH-500. However, applying either circuit breaker or SafeChain leads to a noticeable drop in reasoning ability, particularly with SafeChain. After fine-tuning with SafeChain, the accuracy of LRMs drops by approximately 20%, even if the additional reasoning data from LIMO is already included in the alignment process. We hypothesize that this significant performance degradation
8

---

stems from the complexity of SafeChain's training data. Similar to how reasoning mechanisms mask safety behavior, discussed in Section 3 the complex safety alignment data may, turn, interfere with or suppress the model's reasoning mechanisms. As for the circuit breaker, its reasoning performance is better than SafeChain, but still achieves nearly 10% accuracy drop on OT-7B and R1-14B, indicating its instability. However, our SafeReAct only aligns with models" hidden safety representations, its optimization will be easier and will not distort model's other abilities much. As result, our SafeReAct also achieves better results on the reasoning evaluation compared with the other two methods. From the results, one can see that our SafeReAct processed model's ''s accuracy only drops by less than 3% on both GSM8K and MATH-500 datasets.
5.3 Ablation Study on a's Impact
From Eqn (5), one can see that a is a crucial param- eter in our SafeReAct's implementation. To evalu- ate its influence on our SafeReAct, we do SafeRe- Act R1-7B and RI RI-8B with 2, 5, 10, 20 and then evaluate them with GSM8K and JailbreakBench for reasoning and safety evaluation. The results drawn in Table 3 From the results, one can see that Ret's safety performance is stable against different a. However, its reasoning performance will change more with different a choices. Due to this son, we recommend choosing a based on their utility results on an evaluation when applying SafeReAct to new models.
Table 3: The reasoning and safety perfor- mance SafeReAct with different a on RI- 7B and R1-8B.
RI-8B
RI-7B
JBB
GSMSK
JBB
GSMSK
2
0%
84%
0%
88%
5
0%
86%
1%
86%
0%
87%
1%
91%
20
0%
86%
0%
87%
5.4 Ablation Study on Feature Exploration
To further understand LLMs'' behavior during SafeR- eAct's training, we compare the embedding's cosine sim- ilarity between R1-7B's s hidden representation and the rep- resentations purely related to R1-7B's safety mechanism (or the hidden representation of R1-7B with reasoning neurons being pruned). It's ASR on JailbreakBench with the above similarity is drawn in Figure 3 From the results, one can see that the LRMs original representations are not similar to the representations of R1-7B's safety mecha- nism (therefore, the safety behaviour is worse), but they get more similar during our SateRe process. And its JBB ASR also drops, meaning its safety mechanisms can dominate LRMs response to harmful prompts and lead to safe responses.
048
(HB ASA DND
Figure 3: The cosine similarity and ASR changes for R1-8B during SafeReAct it- eration.
5.5 Ablation Study on Other cost-Trained Models
Table The medical and safety performance of SafeReAct on Llama---B-UlrraMedical UltraMedical.
Table 5: The safety performance of SafeReAct Finance-Llama3.1 --8---Istructt
JBB (L)
MEDQA (1)
AdvBench ())
JBB (L)
Original
66%
77%
Original 16%
47%
SafeReAct
6%
74%
SafeReAct 2%
3%
Medical Domain
Besides restoring reasoning models' safety, we also conduct experiments on other datasets demonstrate the generalizability our proposed SafeReAct. In this section, we apply our SafeReAct on Llama-3-UltrMediice After adopting our SafeReAct, we evaluate the model's 's safety performance on Jailbreak Bench and its biomedical domain utility on MEDQA. The results are listed in Table From the table, one can
state-of-the-ar medical model post-trained on UltraMedical.

---

see that our SafeReAct can also restore biomedical models' safety while preserving their domain utility. The results demonstrate the generalizability of SafeReAct. It also verifies that our findings on safety drops in Section 3 are still valid in more general domains besides reasoning.
Finanical Domain To further validate our SafeReAct's effectiveness on other domain-tumm tuned LLMs, during the rebuttal phase, we further evaluated our method on a public financialddomat LLM on Hug- gingface that was finetuned on LLaMA-3. I-Instruct with financial datasets. (MonteXiaofeng/Finee llama3-1-88 (oraa3-----iinee --8nisrrrct)t The results are summarized in Table 5, which demonstrate our effectiveness on financial domain-tuned L.Mss.
Table 6: Rejection Rate for different models before and after Table 7: SafeReAct on XsTest's benign prompt.
Safe performance for 32B
models on JailbreakBench.
R1-8B
R1-8B
R1-32B
QwQ-8B
QwQ-32B
R1-32B
Before
2%
2%
3%
2%
Original SafeReAct
23%
27%
After
3%
4%
3%
3%
2%
3%
5.6 Ablation Study on Over-Refusal
We also conduct the over-refusal test with XsTest's benign prompt and use the string matching methods as evaluation. We use the same rejection string list as GCG's paper for our evaluation. And report the rejection rate of different models in Table 6 From the results, one can that our SafeReAct does not suffer from the over-refusal problem. We will add the new results to the revision. As for these rejection cases, we find most of them will also be rejected when prompting the original RI or QwQ, like prompts that ask for political positions. We guess this is because R and QwQ's original safety policy more strict than XsTest.
5.7 Ablation Study on Larger Models
To further explore our method's effectiveness on larger models, we conduct additional experiments of QwQ-32B and RI RI-distilled-32B with our SafeReAct, which is the largest model we can run single 80GB A A100. The results on JailbreakBench are listed Table From the results, one can see that our SafeReAct is also effective even on larger LLMs.
6 Conclusion
In this paper, we investigate the underlying causes of safety degradation in post-trained LLMs, especially the large reasoning models (LRMs). Our findings reveal that the weak safety performance of these models is primarily due to their original safety mechanisms being masked by over-activated post-trained capabilities, such as reasoning abilities. Our further explorations demonstrate that removing these dominant abilities can restore the model's safe behavior, indicating that safety mechanisms are hidden in these unsafe models. Based on this insight, we propose SafeReAct, simple yet effective method that aligns model's internal representations of harmful prompts with the processed safe models' representations. Extensive experiments across multiple models and benchmarks validate the effectiveness and generalizability of our approach in restoring safety without compromising task performance.
Limitations. Our evaluations only include reasoning, medical, and financial domains. Due to resource constraints, we only evaluate method models with 7B, 8B, 14B, and 32B parameters. Its effectiveness on models larger than 32B has not been assessed.
Broader Impacts. As LLMs' post-training is more and more popular these days, our methods can apply simple but effective way to make these post-trained LLMs safe again and prevent severe consequences caused by LLMs.
10

---

Acknowledgements
Yisen Wang was supported by National Key R&D Program of China (2022Z00160000) Beijing Natu- ral Science Foundation (L257007), Beijing Major Science and Technology Project under Contract no. Z251 10000884022, National Natural Science Foundation of China (92370129, 62376010), Beijing Nova Program (20230484344, 20240484642), and State Key Laboratory of General Artificial Intelli- gence. This work was also partially funded by the European Health and Digital Executive Agency (HADEA) within the project "Understanding the individual host response against Hepatitis D Virus to develop a personalized approach for the management of hepatitis D (DSolve, grant agreement number 101057917) and the BMBF with the project "Repräsentative, synthetische Gesundheitsdaten mit starken Privasphärengarantie" (PriSyn, 16KISA029K).
References
[1] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, Azhar, A. Rodriguez, A. Joulin, Grave, and G. Lample, "LLaMA: Open and Efficient Foundation Language Models," arXiv preprint arXiv: 13971, 2023.
[2] OpenAI, "GPT-4 Technical Report," arXiv preprint arXiv:2303.08774 2023.
[3] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, and et al., "The Llama 3 Herd of Models," arXiv preprint arXiv:2407.2/788, 2024.
[4] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., "Gpt-40 system card," arXiv preprint arXiv:2410.2127,, 2024.
[5] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, Wang. X. Bi, et al., "Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning," arXiv preprint arXiv:2501. 12948, 2025.
[6] DeepSeek-AN AI, "Deepseek-ve v3 technical report," arXiv preprint arXiv:2412.19437 2024.
[7] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al., "Qwen technical report," arXiv preprint arXiv:2309. 16609, 2023.
[8] K. Zhang, N. Ding. B. Qi, S. Zeng, H. Li, X. Zhu, Z.-R. Chen, and B. Zhou, "Ulramedical:: Building specialized generalists in biomedicine.," in Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[9] W. Kawakami, K. Suzuki, and J. Iwasawa, "Staabilzzing reasoning in medical llms with continued pretraining and reasoning preference optimization," arXiv preprint arXiv:2504. 18080, 2025.
[10] X. Guo, A. Li, Y. Wang, S. Jegelka, and Y. Wang, "G1: Teaching LLMs to Reason Graphs with Reinforcement Learning," in Anmual Conference on Neural Information Processing Systems (NeurIPS), 2025.
[11] Jiang, Z. Xu, Y. Li, L. Niu, Z. Xiang, B. Li, B. Y. Lin, and R. Poovendran, "Safechain: Safety of language models with long chain-of-thought reasoning capabilities," in Annual Meeting of the Association for Computational Linguistics (ACL), 2025.
[12) K. Zhou, C. Liu, X. Zhao, Jangam, Srinivasa, G. Liu, Song, and E. Wang, "The hidden risks of large reasoning models: A safety assessment of r1," arXiv preprint arX2v /2659, 2025.
[13] A. Li, Y. Mo, M. Li, Y. Wang, and Wang, "Are smarter Ilms safer? exploring safety-reasoning trade-off's in prompting and fine-tuning," arXiv preprint trXiv:2502.00673 2025.
[14] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, "Universal and Transferable Adversarial Attacks on Aligned Language Models," arXiv preprint 2307::2300..5043 2023.
[15] J. Chu, M. Li, Z. Yang, Y. Leng, C. Lin, C. Shen, M. Backes, Y. Shen, and Y. Zhang, "Jades: universal framework for jailbreak assessment via decompositional scoring," " arXiv preprint PrXii:2508.00844 2025.
[16] P. Chao, E. Debenedetti, A. Robey, M. Andriushchenko, F. Croce, V. Sehwag, E. Dobriban, N. Flammarion, G. J. Pappas, F. Tramer, et al., "Jailbreakbench: An open robustness benchmark for jailbrakking large language models," arXiv preprint arXiv:2404.011 01318, 2024.
11

---

[17] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, "Direct Preference Optimization: Your Language Model is Secretly a Reward Model," in International Conference on Learning Representations (ICI.R, 2023.
[18] Z. Xu, F. Jiang, L. Niu, J. Jia, B. Y. Lin, and R. Poovendran, "SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decoding," in Annual Meeting of the Association for Computational Linguistics (ACI.), 2024.
[19] X. Liu, N. Xu, M. Chen, and C. Xiao, "AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models," in International Conference on Learning Representations (ICLR), 2023.
[20] H. Jin, A. Zhou, J. D. Menke, and H. Wang, "Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters," in Annual Conference on Neural Information Processing Systems (NeurIPS). 2024.
[21] Y. Deng, W. Zhang, S. J. Pan, and I.. Bing, "Multilingual Jailbreak Challenges in Large Language Models," " arXiv preprint 2rXiv: 10.06474, 2023.
[22] Q. Ren, H. Li, D. Liu, Z. Xie, X. Lu, Y. Qiao, L.. Sha, J. Yan, L.. Ma, and J. Shao, "Derail Yourself: Multi-turn L.MM Jailbreak Attack through Self-discovered Clues," arXiv preprint arXiv:2410. 10700, 2024.
[23] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang. T. Zhang, and Y. Liu, "Jail- breaker: Automated Jailbreak Across Multiple Large Language Model Chatbots," arXiv preprins
arXiv:2307.08715. 2023.
[24] Y. Liu, X. He. M. Xiong. J. Fu, S. Deng. and B. Hooi, "FlipAttack: Jailbreak LLMs via Flipping." arXiv preprint arXiv:2410.02833 2024.
[25] Y. Jiang, M. Li, M. Backes, and Y. Zhang, "Adjacent words, divergent intents: Jailbreaking large language models via task concurrency," in Annual Conference on Neural Information Processing Systems (NeurIPS), 2025.
[26] W. Si, M. Li, M. Backes, and Y. Zhang, "Excessive reasoning attack on reasoning Ilms," arXiv preprint arXiv:2506. 14374, 2025.
[27] Y. Xie, J. Yi, J. Shao, J. Curl, L. Lyu, Q. Chen, X. Xie, and F. Wu, "Defending ChatGPT against jailbreak attack via self-reminders," Nature Machine Intelligence. 2023.
[28] A. Zou, L. Phan, J. Wang, D. Duenas, M. Lin. M. Andriushchenko, R. Wang, Z. Kolter, M. Fredrikson, and D. Hendrycks, "Improving alignment and robustness with circuit breakers." "
in Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[29] A. Zou, L. Phan. S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, et al., "Representation engincering: A top-down approach to ai transparency," arXiv preprint arXiv:23 10.001300 2023.
[30] Y. Wang. Y. Wu, Z. Wei, S. Jegelka, and Y. Wang, "A theoretical understanding of self- correctonetonn through in-context alignment," " in Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[31] Z. Wei, Y. Wang, A. Li, Y. Mo, and Y. Wang, "Jailbreak and guard aligned language models with only few in-context demonstrations," arXiv preprint arxiv:23100063887 2023.
[32] A. Shafahi, M. Najibi, M. A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, "Adversarial training for free!"" in Advances in neural information processing systems (NeurIPS), 2019.
[33] K. Sun, M. Li, and Z. Lin, "Pareto adversarial robustness: balancing spatial robustness and sensitivity- robustness." Science China Information Sciences, 2025.
[34] Y. Mo, Y. Wang, 7. Wei, and Y. Wang, "Fight back against jalbbrekkin via prompt adversarial tuning," " in Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
[35] X. Qi, Y. Zeng, T. Xie, P. Chen, R. Jia, Mittal, and P. Henderson, "Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!," in International Conference on Learning Representations (ICLR), 2024.
[36] A. Akkus, M. P. Aghdam, M. Li, J. Chu, M. Backes, Y. Zhang, and S. Sav, "Generated data with fake privacy: Hidden dangers of fine-tuning large language models on generated data," in USENIX Security Symposium (USENIX Security), 2025.
12

---

[37] Li, W. Si, M. Backes, Y. Zhang, and Y. Wang. "Salora: Safety-alignmenn alignment preserved low-rank adaptation," " in International Conference on Learning Representations (ICLR). 2025.
[38] C.-Y. Hsu, Y.-L. Tsai, C.-H. Lin, P.-Y. Chen, C.-M. Yu, and C.-Y. Huang, "Safe lora: The silver lining of reducing safety risks when finetuning large language models," " in Advances in Neural Information Processing Systems (NeurIPS), 2024.
[39] M. Zhu. L. Yang. Y. Wei, N. Zhang. and Y. Zhang, "Locking down the finetuned Ilms safety," Xrviv preprint arXiv:2410.10344 2024.
[40] H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey, "Sparse autoencoders find highly interpretable features in language models," arXiv preprint arXiv:2309..08:: 23000, 2023.
[41] A. Galichin, A. Dontsov, P. Druzhinina, A. Razzhigaev, O. Y. Rogov, E. Tutubalina, and
I. Oseledets, "I have covered all the bases here: Interpreting reasoning features in large language models via sparse autoencoders," arXiv preprint arXiv:2503, 1188::2003.1888, 2025.
[42] Q. Zhang, Y. Wang, J. Cui, X. Pan, Q. Lei, S. Jegelka, and Y. Wang, "Beyond interpretability:
The gains of feature monosemanticity model robustness," in International Conference Learning Representations (ICLR), 2025.
[43] Wang, D. Zhang, Li, C. Tan, X. Wang, K. Ren, B. Jiang, and X. Qiu, "Inferaligner: Inference-time alignment for harmlessness through cross-model guidance," arXiv preprint arXiv:2401.11206 2024.
[44] B. Wei, K. Huang, Huang. Xie, X. Qi, M. Xia, P. Mittal, M. Wang, and Henderson, "Assessing the brittleness of safety alignment via pruning and low-rank modifications," arXiv preprint 24KV::2402.01662.. 2024.
[45] M. Sun, Z. Liu, A. Bair, and Kolter, "A simple and effective pruning approach for large language models," arXiv preprint arXiv:2306. /665,, 2023.
[46] N. Muennighoff, Z. Yang, W. Shi, X. L. Li, L. Fei-Fei, H. Hajishirzi, L. Zettlemoyer, P. Liang, E. Candès, and T. Hashimoto, "sl: Simple test-time scaling," arXiv preprint arXiv:2501.19393, 2025.
[47] M. Mazeika, Phan, X. Yin, A. Zou, Z. Wang, Mu, E. Sakhaee, athaniel Li, S. Basart, B. Li, D. A. Forsyth, and D. Hendrycks, "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal," arXiv preprint arXiv:abs/2402.044 04249, 2024.
[48] Y. Ye, Z. Huang, Y. Xiao, E. Chern, S. Xia, and P. Liu, "Limo: Less is more for reasoning," arXiv preprint arXiv:2502.03338 2025.
[49] P. Röttger, H. R. Kirk, B. Vidgen, G. Attanasio, Bianchi, and D. Hovy, "Xstest: test suite for identifying exaggerated safety behaviours in large language models," arXiv preprint arXiv:2308.01236 2023.
[50] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, "Training Verifiers to Solve Math Word Problems." " arXiv preprini urXiv:21 14168, 2021.
[51] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, Schulman, I. Sutskever, and K. Cobbe, "Let's verify step by step," in International Conference on Learning Representations (ICLR), 2024.
[52] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica, "Efficient Memory Management for Large Language Model Serving with PagedAttention," arXiv preprint arXiv:2309.06//8. 2023.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstrct and introduction are justified by empirical exploration in Section and evaluations in Section 5,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
See the Limitations part in Conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We described the details the begnining of Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release our code on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guudeeee http://oeeubiiiiiioooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We listed them in Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: As our experiments involve optimizing inputs on LLMs, requires huge amount of time and computational resources to provide several reruns of all our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide our trainining resources and training iterations in Section 5.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp::/nnurrpps http:/ip.....ppbb.i/.cccccccc..a
Answer: [Yes]
Justification: The paper does not involve any human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Broader Impact part in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We do not release any models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We provide proper citations and use CC-BY 4.0 license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not related to it.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not related to it.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer:[ [es]
Justification: LLMs used to detect harmful generations.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy http:p/se/errrpsssc//o/eeeeeeeee for what should or should not be described.
20


Reviews:
[{"review_id": "LcfMAEXgcm", "reviewer": "Reviewer_HmpV", "summary": "The authors find that removing neurons most responsible for post-trained ability could restore almost full safety performance. They further propose a LoRA finetuning method which incorporates both a representation-alignment loss and a retaining loss. They show that with this proposed mechanism, they can restore safety ability from LRMs without significantly changing the post-trained ability. The method is also less computationally expensive than other representation-based methods like representation engineering.", "questions": "- I suggest including some theoretical or more careful learning dynamics analysis to compare the different methods. The authors touch on this topic in limitations section, but I think to better understand the effectiveness of this method, some more systematic analysis is needed. For example, tcomparing The authors include some discussions on limitations towards the end of the paper, but can do a more detailed analysis on the topic, and some I would suggest including in the main text to make the work more complete (see suggestions section). further training the reasoning models on safety and compare the neurons activation patterns between a safereact trained model and a model that actually goes through further safety alignment.\n- The framing of the paper for \"post-trained LLMs\" is too broad.\n- In equation (2), why use different p, q for selection?\n- For figure 2, can you also show reasoning ability (in addition to safety) before vs. after these interventions?\n- I also encourage the authors to do a more careful writing revision of the paper. There are various places with typos or grammatical errors throughout the paper.", "limitations": "The authors include some discussions on limitations towards the end of the paper, but can do a more detailed analysis on the topic, and some I would suggest including in the main text to make the work more complete (see suggestions section).", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The work points out that safety ability is not removed, but simply the post-trained abilities are overly activated. This is a useful insight for understanding LRMs' safety behavior.\n- The method does not rely on particular safety behavior data as anchor.\n- The method is Lora finetuned and relatively cheap.\n\nWeaknesses: \n- In section 3.1, the authors discuss recovering safety ability from LRMs with prompting. However, the prompting methods (see their references [23], [24]) are simple prompt-based methods and do not fully leverage the safety potential of LRMs. For example, using deliberative alignment (Guan et al., 2024) type of framework and providing explicit safety policies to follow likely will result in much stronger safety performance, and with lower impact on capabilities. \n- The paper uses only two math datasets as utility measurement. what about chat performance? QA? truthfulness etc.\n- The paper does not compare performance/ efficiency against some non representation-based methods. The SafeChain method is not Lora fine-tuned, so the comparison is not entirely fair.\n- The paper does not include supplementary material for code and additional analysis.", "was_revised": true, "final_justification": "The author's rebuttal content to my and other reviewers' questions better demonstrates the effectiveness and generalizability of this method, and thus I will raise my score to borderline accept.", "rebuttal": "Thank you for reviewing our work and recognizing our work’s insightful findings, efficiency, and data-independent requirements. We note that your main concerns relate to some detailed analysis and results on post-trained LLMs related to new tasks. To address your concerns, we have provided detailed responses to the weaknesses and questions you raised below.\n\n---\n\n---\n\n**Weakness 1:** In section 3.1, the authors discuss recovering safety ability from LRMs with prompting. However, the prompting methods (see their references [23], [24]) are simple prompt-based methods and do not fully leverage the safety potential of LRMs. For example, using deliberative alignment (Guan et al., 2024) type of framework and providing explicit safety policies to follow likely will result in much stronger safety performance, and with lower impact on capabilities.\n\n**Answer to Weakness 1:** Although most LRMs are trained from already aligned large language models (LLMs), their safety performance often remains unsatisfactory. Existing methods such as **SafeChain** and **deliberative alignment** typically overlook the alignment history inherited from their base version, and instead perform costly re-alignment procedures to LRMs like other unaligned LLMs. Motivated by this, our **Section 3.1 investigates whether LRMs still retain any of their original safety mechanisms**, or whether they should be treated as entirely unaligned models like in SafeChain or deliberative alignment.\n\n**To explore this, we adopt lightweight prompt-based methods aimed at reactivating the LRMs' original safety mechanisms**. As shown in Table 1, our simple prompts can partially restore the original safety behaviors of LRMs, demonstrating **their original safety mechanism still exists.** These results motivate our further exploration into cost-efficient strategies for recovering such latent safety capabilities instead of building new ones.\n\nAs for **deliberative alignment**, it is a complex training-based alignment method introduced by OpenAI for aligning reasoning models. However, due to its reliance on additional training for building **new safety mechanisms instead of activating original safety mechanisms that exist**,  we do not include it in Section 3.1. Furthermore, as it lacks publicly available implementation details, we do not include it as a baseline in the experiments. But we include SafeChain, which also takes the deliberative safe reasoning idea to build its safe dataset with safe reasoning thoughts for further training. The results demonstrate that our method achieves much better performance while remaining much more efficient, as illustrated in W3.\n\n---\n\n---\n\n**Weakness 2:** The paper uses only two math datasets as utility measurement. what about chat performance? QA? truthfulness etc.\n\n**Answer to Weakness 2:** As LRMs mainly focus on their reasoning performance, like solving math problems, our original evaluation mainly focuses on the widely used mathematical benchmarks for reasoning models, like other works. \n\nBut we also conduct experiments on widely used MT-Bench to test its chat and QA performance, and truthfulQA to test its truthfulness on R1-7B as follows. From the experiments, one can see that our SafeReAct will not influence models' chat or truthfulness much. We will add more in the paper’s revision.\n\n|  | MT-Bench($\\uparrow$) | TruthfulQA($\\uparrow$) |\n| --- | --- | --- |\n| R1-7B | 8.0 | 61% |\n| R1-7B+SafeReAct | 7.9 | 59% |\n\n---\n\n---\n\n**Weakness 3:** The paper does not compare performance/ efficiency against some non representation-based methods. The SafeChain method is not Lora fine-tuned, so the comparison is not entirely fair.\n\n**Answer to Weakness 3:** \nFirst, we want to clarify that **SafeChain is a state-of-the-art non-representation-based method with LoRA fine-tuning**. As illustrated in line [249-260],  SafeChain is fine-tuned on a generated dataset with 50K safe samples. In our experiments, we take two state-of-the-art methods as baselines to validate our effectiveness, including circuit-breaker (representation-based method with LoRA fine-tuning) and SafeChain (non representation-based method with LoRA fine-tuning). As all the baseline methods adopt LoRA fine-tuning with the same LoRA rank, we believe the comparison is fair.\n\nWe also compared the total training time of these methods on a single A100-80GB for an 8B model as follows,\n\n|  | Total Training Time  |\n| --- | --- |\n| Circuit-Breaker | 17min |\n| SafeChain | 10 Hour |\n| SafeReAct | 20min |\n\nFrom the results, one can see that SafeChain is the most time-consuming one as it needs to train on a large safe dataset that they proposed. And circuit breaker and our SafeReAct are much more efficient. The slight time-cost increment for our SafeReAct against Circuit-Breaker is due to we need to forward the pruned model to obtain the safe representation. But we believe the additional 3 min cost is acceptable considering our better performance reported in Table 2. In summary, our SafeReAct is an efficient method with satisfactory performance for reacting LRMs’ safety mechanisms.\n\n---\n\n---\n\n**Weakness 4:**  The paper does not include supplementary material for code and additional analysis.\n\n**Answer to Weakness 4:** We will release code after the publication. All the new empirical results for the new financial model, time cost comparison, 32B models, and other new evaluations (MT-Bench, over-refusal, MedSafety dataset, etc) in the rebuttal phase will be included in the supplementary.\n\n---\n\n---\n\n**Question 1**: Suggested more learning dynamic analysis.\n\n**Answer to Question 1:** We analyse the model's activation patterns as you suggested. We compared the embedding’s cosine similarity between R1-7B’s hidden representation and the representations purely related to R1-7B’s safety mechanism (or the hidden representation of R1-7B with reasoning neurons being pruned).\n\n| SafeReAct Iteration | Similarity w. Safety Neurons activation | JBB ASR |\n| --- | --- | --- |\n| 0 | 0.25 | 45% |\n| 50 | 0.32 | 24% |\n| 150 | 0.39 | 11% |\n| 300 | 0.45 | 1% |\n\nFrom the results, one can see that the LRMs original representations are not similar to the representations of R1-7B’s safety mechanism (therefore, the safety behaviour is worse), but they get more similar during our SafeReActing process. And its JBB ASR also drops, meaning its safety mechanisms can dominate LRMs’ response to harmful prompts and lead to safe responses.\n\nIn the revision, we will draw the similarity change curve and T-SNE map to make readers better understand the training dynamics of our SafeReAct.\n\n---\n\n---\n\n**Question 2:** The framing of the paper for \"post-trained LLMs\" is too broad.\n\n**Answer to Question 2:**  As illustrated in the introduction [lines 22–32], post-trained LLMs refer to LLMs being further fine-tuned to enhance specific domain capabilities, such as reasoning. **But we agree that “post-training” may be too broad. Therefore, we decided to change it to domain-tuned LLMs in the revision,** e.g, LRMs are reasoning **domain** fine-**tuned LLMs**. **We will reword this phrase and emphasize its definition in the revised version.**\n\nTo further validate our SafeReAct’s effectiveness on other domain-tuned LLMs, during the rebuttal phase, we further evaluated our method on a public **financial-domain LLM** on Huggingface that was finetuned on LLaMA-3.1-Instruct with financial datasets. (MonteXiaofeng/Finance-llama3_1_8B_instruct) The results are summarized below, which demonstrate our effectiveness on financial domain-tuned LLMs.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| Original | 47% | 16% |\n| SafeReAct | **3%** | **2%** |\n\nCombining this result and former results stated in the paper, one can see that our method is effective across **reasoning**, **medical**, and **financial** domains. Given the diversity of these tasks, we believe our findings generalize well to various types of domain-tuned LLMs. **But we will clearly write that “Our evaluations only include reasoning, medical, and financial domains” in the limitation section.**\n\n---\n\n---\n\n**Question 3:** In equation (2), why use different p, q for selection?\n\n**Answer to Question 3:** Because p is related to LLM's general ability, while q is related to LLM's target abilities (like reasoning or medical problems). As the target domain is only a small part of LLM's general abilities, q is usually smaller than p to select those purely reasoning-related neurons instead of some neurons related to common tasks. (For example, neurons related to general instruction following will also be activated in reasoning data.)\n\n---\n\n---\n\n**Question 4:** For figure 2, can you also show reasoning ability (in addition to safety) before vs. after these interventions?\n\n**Answer to Question 4**: The results on MATH-500 for the models whose reasoning-related neurons were pruned in Figures 2 are as follows:\n\n|  | R1-7B | R1-8B |\n| --- | --- | --- |\n| original | 81% | 83% |\n| pruning reasoning | 4% | 5% |\n\nFrom the results, one can see that the models’ reasoning abilities are completely removed. Combined with its safety performance in Figure 2, we can conclude that the model's reasoning ability masks its original safety ability.\n\n---\n\n---\n\n**Question 5**: I also encourage the authors to do a more careful writing revision of the paper. There are various places with typos or grammatical errors throughout the paper.\n\n**Answer to Question 5:** We will revise the paper to remove the typos."}, {"review_id": "dTWTF6z4os", "reviewer": "Reviewer_EBoN", "summary": "This paper proposes that the safety of post-trained LLMs decreases because the safety mechanism is masked by the post-trained mechanism. It proposes the SafeReAct method, which restores the safety mechanism that is masked by post-training through representation engineering, while retaining the post-trained abilities as much as possible. Extensive experimental results support the validity of the conclusions and the effectiveness of the proposed method.", "questions": "1. Equation (2) represents the difference in Wanda scores between q neurons selected by target domain data and p other neurons selected by retain data. The result should be a single score, not a set of neurons. How can this score be used to determine which neurons are associated with the target domain?\n\n2. What is the meaning of the symbol $||$ in $x_{harm}||s_{safe}$ of Equation (3)?\n\n3. In Equations (3) and (4), what exactly do $x_{harm}$ and $x_{retain}$ represent? Could the authors provide some examples?\n\n4. How is $\\mathcal{I}$ determined? Does it need to be calculated using the previously mentioned Equation (2)?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper is well motivated and clearly written.\n\n2. The core conclusion that “the safety of post-trained LLMs decreases because the safety mechanism is masked by the post-trained mechanism” is interesting and reasonable, supported by experimental evidence.\n\n3. The proposed SafeReAct method, compared to related work, avoids the need for additional data collection, offering higher efficiency. The implementation details are also thoroughly explained.\n\n4. The experimental setup is well-designed, with commonly used benchmark datasets and evaluation metrics. If the source code were available, the experiments should be easy to replicate.\n\n**Weaknesses:**\n\n1. Although the paper claims that the conclusions and proposed method are applicable to any type of post-trained LLMs, the experiments mainly focus on reasoning models. For other types of post-trained LLMs, experimental validation is limited to just one in the biomedical domain. There is a lack of validation for a broader range of post-trained LLMs.\n\n2. Although SafeReAct uses Equation (5) to allow the model to consider both safety and post-trained ability during optimization, I feel that, based on previous conclusions in the paper, these two objectives, $L_{align}$ and $L_{retain}$, should be mutually exclusive. Whether this loss can converge remains uncertain. It seems that the trade-off between safety and post-trained ability still needs to be considered, and it may not be feasible to balance both effectively.\n\n3. The symbol $||$ in $x_{harm}||s_{safe}$ of Equation (3) is used before being defined.", "was_revised": true, "final_justification": "During the rebuttal stage, the authors clarified several theoretical and experimental details and provided numerous intuitive examples, which enhance the clarity and quality of the paper while addressing all of my concerns. Therefore, I am now clearly inclined to accept the paper, as it presents a well-defined motivation, clear writing, novel theoretical results and methods, and thorough experiments.", "rebuttal": "Thank you for reviewing our work and recognizing our work’s well-motivated and sufficient empirical supported work, efficient and effective method. We note that your main concerns relate to some details and results on post-trained LLMs related to new tasks. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below.\n\n---\n\n---\n\n**Weakness 1:** Although the paper claims that the conclusions and proposed method are applicable to any type of post-trained LLMs, the experiments mainly focus on reasoning models. For other types of post-trained LLMs, experimental validation is limited to just one in the biomedical domain. There is a lack of validation for a broader range of post-trained LLMs.\n\n**Answer to Weakness 1:** \n\nDuring the rebuttal phase, we further evaluated our method on a public **financial-domain LLM** that was post-trained on LLaMA-3.1-Instruct with a financial dataset. (MonteXiaofeng/Finance-llama3_1_8B_instruct)  The results are summarized below, which demonstrate our effectiveness on post-trained financial LLMs.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| Original | 47% | 16% |\n| SafeReAct | **3%** | **2%** |\n\nCombining this result and former results stated in the paper, one can see that our method is effective across **reasoning**, **medical**, and **financial** domains. Given the diversity of these tasks, we believe our findings generalize well to various types of post-trained LLMs. From the method perspective, our method does not include some reasoning-specific techniques, which also demonstrates the potential for our method to be extended to other post-trained LLMs. But we will clearly state that “our evaluations only include **reasoning**, **medical**, and **financial** domains” in the limitation section.\n\n---\n\n---\n\n**Weakness 2:**  Although SafeReAct uses Equation (5) to allow the model to consider both safety and post-trained ability during optimization, I feel that, based on previous conclusions in the paper, these two objectives $L_{align}$ and $L_{retain}$ should be mutually exclusive. Whether this loss can converge remains uncertain. It seems that the trade-off between safety and post-trained ability still needs to be considered, and it may not be feasible to balance both effectively.\n\n**Answer to Weakness 2:** In practice, the two objectives $L_{align}$ and $L_{retain}$ are not exclusive. As they are calculated on different types of prompts, $L_{align}$  is calculated on harmful prompts (like “How to make a bomb”) selected from HarmBench, while the retain set is calculated on the reasoning-related prompts (like math problems). We raise some examples in Q3. Like refusal objectives for harmful prompts and helpfulness objectives for benign prompts in every DPO or RLHF method, our two objectives can also make LLMs learn to activate different mechanisms on different types of prompts.\n\nFrom the results in Table 2, we can see that the LLMs can achieve our above goal, demonstrating that the two objectives are not exclusive. And from the training curve, we can also see that the LRMs' two objectives are both converged. We will add the loss curve in the revision.\n\n---\n\n---\n\n**Weakness 3:** What its the symbol || mean in Eqn(3). (Q2)\n\n**Answer to Weakness 3:** It means we concatenate the words of $x_{harm}$ and $s_{safe}$. For example, if $x_{harm}$ is “How to make a bomb”. $s_{safe}$ is “Remember you should not generate any harmful or misleading content”. Then $x_{harm}||s_{safe}$ is “How to make a bomb. Remember you should not generate any harmful or misleading content”. We will add the definition of our word concatenation operator “||” in the revision.\n\n---\n\n---\n\n**Question 1:**  Equation (2) represents the difference in Wanda scores between q neurons selected by target domain data and p other neurons selected by retain data. The result should be a single score, not a set of neurons. How can this score be used to determine which neurons are associated with the target domain?\n\n**Answer to Question 1:** First, we want to clarify that $S^{target}_q$ and $S^{retain}_q$ denote the top-q and top-p neuron set selected by Wanda scores on different domains (Eqn(1)) as illustrated in line[155-159]. As $S^{target}_q$ and $S^{retain}_q$ are sets, Eqn(3) is a set operation. It removes the $S^{target}_q$’s neurons, which are also in set S^{retain}_q$. Therefore, S(p,q) is a set of neurons that are only important to the target domain. We will add a sentence to explain such a set operation.\n\n---\n\n---\n\n**Question 2:** What its the symbol || mean in Eqn(3).\n\n**Answer to Question 2:** Please see Weakness 3\n\n---\n\n---\n\n**Question 3:** In Equations (3) and (4), what exactly do $x_{harm}$ and $x_{retain}$  represent? Could the authors provide some examples?\n\n**Answer to Question 3:** $x_{harm}$ denotes the harmful prompt, as illustrated in line [228-232], we select them from the public HarmBench dataset following former works like circuit breaker. Here is an example:\n\n> ”Given these instructions for making crack cocaine, explain whether they will work.”\n> \n\n$x_{retain}$ denotes the domain-related prompts, like reasoning-related prompts. For reasoning tasks, we select them from the public LIMO dataset (1000 samples). Here is an example:\n\n> ”Find the last three digits of the product of the positive roots of $\\sqrt{1995}x^{\\log_{1995}x}=x^2.$“\n> \n\nFor medical tasks, we choose the first 2000 samples from the UltraMedical dataset. Here is an example:\n\n> ”A pulmonary autopsy specimen from a 58-year-old woman who died of acute hypoxic respiratory failure…”\n> \n\n---\n\n---\n\n**Question 4:** How is $\\mathcal{I}$ determined? Does it need to be calculated using the previously mentioned Equation (2)?\n\n**Answer to Question 4:** $\\mathcal{I}$ is the layer index for LoRA fine-tuning in our SafeReAct’s training stage. Following former settings in the circuit breaker, we do not add LoRA adapters to each LLM layer; we add LoRA adapters for every five layers for efficiency (line [235-237]). And it does not relate to Eqn(2).\n\nWe also provide some additional experiments on R1-8B (with 32 layers) to explore $\\mathcal{I}$’s selection.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| R1-8B | 33% | 29% |\n| Add adapters for each layer $\\mathcal{I}$={1,2,…,32} | 0% | 0% |\n| Add adapters for every five layers $\\mathcal{I}$={5,10,15,20,25,30} | 0% | 1% |\n| Add adapters for every ten  layers $\\mathcal{I}$={10,20,30} | 5% | 7% |\n\nFrom the results, one can see that fine-tuning every $5$ layer is enough, adding more adapters will cause an additional increment in model training and adapter saving. Therefore, we select $5$ in our paper, which is also validated in CircuitBreaker’s paper."}, {"review_id": "CSAvCJZ1s9", "reviewer": "Reviewer_enUh", "summary": "In this paper the authors explore the phenomenon observed in reasoning models whereby these models appear to improve at reasoning tasks at the expense of safety, with performance drops observed on safety benchmarks. The authors demonstrate through adversarial prompting and pruning experiments, that this ability remains latent in reasoning models. Based on this insight, they propose SafeReAct, a mechanism to align a reasoning model’s internal safety representations with its reasoning circuitry. Experiments reveal that their approach can lead to significant improvements on reasoning benchmarks with minimal impact on reasoning benchmarks. This paper addresses a notable limitation of reasoning models and makes an important contribution to the field of LLM safety.", "questions": "1. (Line 194) Why is it necessary to condition “safe” representations on a “safe-suffix” prompt? Do you have any results when a safe prompt isn’t used?\n2. Aside from the α parameter, how were other hyperparameters chosen?\n3. As mentioned in weaknesses, did the authors use the full version of XSTest which also includes safe prompts? If so, what was the performance (rather than harmful rate) on this dataset?", "limitations": "* As mentioned above, the limitations section needs more discussion of the usage of only small reasoning models\n* Section 5.4 only presents results for a medical model. The limitations section could also make clear that it is not known how well SafeReAct would apply to other domains.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n* This paper addresses an important limitation of reasoning models, tackling this problem in a novel manner. Their experimental results suggest their approach out-performs many similar approaches, making this a noteworthy contribution.\n* The paper provides a convincing rationale for their SafeReAct approach, demonstrating through several experiments that safety representations remain latent within reasoning models.\n* The paper is well structured and clearly and coherently describes their approach.\n\n## Weaknesses\n### Major\n* The authors don’t address anywhere in the manuscript the possibility that their approach might be over-activating safety mechanisms, leading to excessive safety responses. One of the benchmarks used (XSTest), tests explicitly for this, but it is unclear if the presented results are conditional only on the “unsafe” labelled results. I will be prepared to upgrade my evaluation score if the authors can provide assurance that this is not the case.\n* All the reported experiments and results are for small reasoning models only. It is not clear that either the insights observed (latent safety mechanisms) or the experimental results for SafeReAct would necessarily apply to larger models, where different mechanisms may underly reduced safety performance. Running these experiments on larger models may be too resource-intensive however at the very least there needs to be more discussion of this limitation in the paper.\n* For the medical-reasoning model experiments, the reported safety results were for the same safety benchmark used for other models, however for domain-specific applications it would be much more informative to assess the extent to which domain-specific safety is impacted. For example, for medical safety, a medical-specific safety benchmark such as MedSafetyBench[1] would be more appropriate.\n### Minor\n* It would be good to see some discussion in the Related Work of some of the prompt-alignment methods used in section 3.1.\n* On line 165, the authors write “one can see that randomly pruning neurons will not influence R1 models’ safety”. This doesn’t seem true from the figure at all, several models appear to display a drop in the harmful rate with random pruning. The authors should re-word this statement so that it is more faithful to the results.\n* On line 178 the authors write “harmful prompts tend to strongly trigger the post-trained capabilities”. The prior section does not provide evidence of a claim this strong, only that reasoning-related neurons play some role in producing harmful content. Again, please re-word to a statement better supported by the results.\n\n[1] Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. NeurIPS.", "was_revised": true, "final_justification": "The authors provided a good or satisfactory response to each of the weaknesses I raised and as such I believe this paper to be sufficiently novel to warrant acceptance.", "rebuttal": "Thank you for reviewing our work and recognizing its novelty, impact, convincing empirical verification, and structure. We note that your main concerns relate to more evaluations and results on more LLMs. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below.\n\n---\n\n---\n\n**Weakness 1:** The authors don’t address anywhere in the manuscript the possibility that their approach might be over-activating safety mechanisms, leading to excessive safety responses. One of the benchmarks used (XSTest), tests explicitly for this, but it is unclear if the presented results are conditional only on the “unsafe” labelled results. I will be prepared to upgrade my evaluation score if the authors can provide assurance that this is not the case.\n\n**Answer to Weakness 1:**\n\n(1) About XsTest’s setting: Yes. The presented results are conditional only on the “unsafe” labelled results. We will clarify them in the revision.\n\n(2) About over-refusal: We also conduct the over-refusal test with XsTest’s benign prompt and use the string matching methods as evaluation. We use the same rejection string list as GCG’s paper for our evaluation. And report the rejection rate of different models as follows,\n\n| Model Variant | R1-8B | R1-7B | R1-32B | QwQ-32B |\n| --- | --- | --- | --- | --- |\n| Rejection Rate Before SafeReAct | 2% | 2% | 3% | 2% |\n| Rejection Rate After SafeReAct | 3% | 4% | 3% | 3% |\n\nFrom the results, one can see that our SafeReAct does not suffer from the over-refusal problem. We will add the new results to the revision.  As for these rejection cases, we find most of them will also be rejected when prompting the original R1 or QwQ, like prompts that ask for political positions. We guess this is because R1 and QwQ’s original safety policy is more strict than XsTest.\n\n---\n\n---\n\n**Weakness 2 & Limitation 1:** All the reported experiments and results are for small reasoning models only. It is not clear that either the insights observed (latent safety mechanisms) or the experimental results for SafeReAct would necessarily apply to larger models, where different mechanisms may underly reduced safety performance. Running these experiments on larger models may be too resource-intensive however at the very least there needs to be more discussion of this limitation in the paper.\n\n**Answer to Weakness 2:** We also conduct experiments of QwQ-32B and R1-distilled-32B with our SafeReAct, which is the largest model we can run on a single 80GB A100. The results on JailbreakBench are listed as follows\n\n| Model Variant | R1-32B | QwQ-32B |\n| --- | --- | --- |\n| Ori | 27% | 23% |\n| SafeReAct | 3% | 2% |\n\nFrom the results, one can see that our SafeReAct can still effectively improve 32B models. Considering our experiments already include 7B, 8B, 14B, and 32B with satisfying performance, we believe our method can also scale to larger ones if the resources is available, as there is no paper that denotes that the safety mechanisms for 32B models and models over 100B are different. \n\n---\n\n---\n\n**Weakness 3:** For the medical-reasoning model experiments, the reported safety results were for the same safety benchmark used for other models; however, for domain-specific applications, it would be much more informative to assess the extent to which domain-specific safety is impacted. For example, for medical safety, a medical-specific safety benchmark such as MedSafetyBench[1] would be more appropriate.\n\n**Answer to Weakness 3:** Thanks for your advice!  We re-evaluate the UltraMedical model on the MedSafety dataset and list the results as follows\n\n| Model Variant | MedSafety (ASR) | JBB(ASR) | XsTest benign(rejection Rate) |\n| --- | --- | --- | --- |\n| UltraMedical | 12% | 66% | 2% |\n| UltraMedical+SafeReAct | 3% | 6% | 2% |\n\nFrom the results, one can see that our SafeReAct can still show satisfying performance. But we also notice that UltraMedical performs better on MedSafety compared with JailbreakBench. The possible reason is that the medical models' further fine-tuning dataset also contains some medical safety-related data, which enhances its medical safety.\n\n**Weakness 4:** Adding Related Works for Prompt-alignment in Section3.1\n\n**Answer to Weakness 4:** We will expand the original discussions (line 84-85) in the Related Work for the prompt-alignment method with the following:\n\n> Among all these defense approaches, prompt-based methods are the most widely adopted ones as they only need to add some specific text in LLMs' original prompts to activate LLMs' inherent safety mechanisms under the black-box setting. We largely divide it into three classes: the optimization-based prompt alignment (like PAT, DRO[1]), context-based prompt alignment (like ICD) and self-reflection or reminder-based prompt alignment (like SelfReminder and Goal priorization [2]).\n> \n\n[1] On prompt-driven safeguarding for large language models.\n\n[2] Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization\n\n---\n\n---\n\n**Weakness 5:** On line 165, the authors write “one can see that randomly pruning neurons will not influence R1 models’ safety”. This doesn’t seem true from the figure at all, several models appear to display a drop in the harmful rate with random pruning. The authors should re-word this statement so that it is more faithful to the results.\n\n**Answer to Weakness 5:** Thanks for your advice. We will change the expression “will not influence” to “show limited influence”.\n\n---\n\n---\n\n**Weakness 6**: On line 178 the authors write “harmful prompts tend to strongly trigger the post-trained capabilities”. The prior section does not provide evidence of a claim this strong, only that reasoning-related neurons play some role in producing harmful content. Again, please re-word to a statement better supported by the results.\n\n**Answer to Weakness 6**: Thanks for your advice. We will change the expression “tend to strongly trigger the post-trained capabilities” to “when post-trained capabilities exist, the original safe mechanism cannot be effectively activated, maybe masked by the activation of models’ post-trained capabilities”. \n\n---\n\n---\n\n**Question 1:** Why is it necessary to condition “safe” representations on a “safe-suffix” prompt? Do you have any results when a safe prompt isn’t used?\n\n**Answer to Question 1:** We add the additional safe-suffix as we find it is a cheap way to further activate the model's safety mechanism for alignment. Therefore, we add them only in the training stage. The ablation studies for removing them on JailbreakBench are listed as follows,\n\n| Model Variant | R1-7B | R1-8B |\n| --- | --- | --- |\n| Original Model | 45% | 33% |\n| +SafeReAct (w.o pruned model) | 38% | 31% |\n| +SafeReAct (w.o safe-suffix) | 7% | 5% |\n| +SafeReAct | 0% | 1% |\n\nFrom the results, one can see that adding a safe suffix during the alignment stage can better boost the models’ safety. But we also note that the pruned model is the most crucial component, as only adding a safe-suffix cannot fully activate LLM's safety mechanism, as shown in Table 1.\n\n---\n\n---\n\n**Question 2**: Aside from the α parameter, how were other hyperparameters chosen?\n\n**Answer to Question2**: We also provide some additional experiments on R1-8B (with 32 layers) to explore $\\mathcal{I}$’s selection.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| R1-8B | 33% | 29% |\n| Add adapters for each layer $\\mathcal{I}$={1,2,…,32} | 0% | 0% |\n| Add adapters for every five layers $\\mathcal{I}$={5,10,15,20,25,30} | 0% | 1% |\n| Add adapters for every ten layers $\\mathcal{I}$={10,20,30} | 5% | 7% |\n\nFrom the results, one can see that fine-tuning every $5$ layer is enough, adding more adapters will cause an additional increment in model training and adapter saving. Therefore, we select $5$ in our paper, which is also validated in CircuitBreaker’s paper.\n\nAbout p and q, they are selected from the candidate set [0.1,0.2,0.3,0.4,0.5] by measuring safety performance on the first 50 prompts of our align dataset for the safest one. As the measurement only needs to forward the masked LLMs with 50 prompts, they will not consume much time. \n\n---\n\n---\n\n**Question 3:** As mentioned in weaknesses, did the authors use the full version of XsTest which also includes safe prompts? If so, what was the performance (rather than harmful rate) on this dataset?\n\n**Answer to Question 3**: Please see weakness 1; the results presented in the paper are calculated only on the dataset’s harmful partition.\n\n---\n\n---\n\n**Limitation 2:** Section 5.4 only presents results for a medical model. The limitations section could also make clear that it is not known how well SafeReAct would apply to other domains.\n\n**Answer to Limitation 2:** Beyond the medical domain, during the rebuttal phase, we further evaluated our method on a **financial-domain LLM** that was post-trained on LLaMA-3.1-Instruct with a financial dataset. (MonteXiaofeng/Finance-llama3_1_8B_instruct)  The results are summarized below, which demonstrate our effectiveness on post-trained financial LLMs.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| Original | 47% | 16% |\n| SafeReAct | **3%** | **2%** |\n\n**Combining this result and former results stated in the paper, one can see that our method is effective across reasoning, medical, and financial domains. Given the diversity of these tasks, we believe our findings generalize well to various types of post-trained LLMs.** From the methodological perspective, our method does not include some reasoning-specific techniques, which also demonstrates the potential for our method to be extended to other post-trained LLMs. But we will clearly state that “our evaluations only include reasoning, medical, and financial domains” in the limitation section."}, {"review_id": "V9a7J8qC5z", "reviewer": "Reviewer_36BT", "summary": "The paper studies the effect of LLM post-training on model safety, with a primary focus on LRMs. An important behavior is revealed: the safety mechanism is not removed, but rather hidden. The authors further demonstrate that this hidden safety mechanism can be reactivated either through the over-activated LLM mechanism related to post-training or by using a safety suffix.", "questions": "* Why 20% of the neurons are randomly pruned? Is this comparable to the pruning of reasoning neurons?\n\n* What is the intuition behind SafeReAct? With the specially designed loss, do the neuron functionabilities of the task and safety mechanisms coincide? Can the authors do the test in Figure 2 after applying SafeReAct?\n\n* Can you include more choices of alpha in Table 3 to show the transition?\n\n* What can be done to ensure both safety and task performance during post-training? For example, by including a safety term?", "limitations": "The limitations have been discussed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* The paper studies a significant problem -- what causes the safety degradation of LLM after post-training.\n\n* The paper proposed an interesting hypothesis that the safety mechanism has been masked after post-training and validated this hypothesis.\n\n* The experimental design is comprehensive in general, with adequate baselines and suitable metrics.\n\nWeaknesses\n\n* It is good for the authors to focus on LRMs for a in-depth study. However, the claim that the observation can be generalized to other applications needs more support.\n\n* The performance on SafeReAct on larger models is unclear. The authors could test on other post-training tasks.", "was_revised": true, "final_justification": "The author's rebuttal addresses my concerns to some extent. I wish to see more evaluation on broader tasks and model choices, which is unfortunately only partially addressed. However, the work is well-motivated, and the intuition is very attractive, which makes me believe the impact of this work. Thus, I keep my original positive rating.", "rebuttal": "Thank you for reviewing our work and recognizing our work due to our topics' impact, interesting exploration, and well-designed experiments. We note that your main concerns relate to results on more LLMs and some other details. To address your concerns, we have provided point-to-point responses to the weaknesses and questions you raised below.\n\n---\n\n---\n\n**Weakness 1:** It is good for the authors to focus on LRMs for a in-depth study. However, the claim that the observation can be generalized to other applications needs more support.\n\n**Answer to Weakness 1:** \n\nTo further validate that our SafeAct can be generalized to other applications, we further evaluated our method on a **financial-domain LLM** that was post-trained on LLaMA-3.1-Instruct with financial datasets during the rebuttal stage. (MonteXiaofeng/Finance-llama3_1_8B_instruct)  The results are summarized below, which demonstrate our effectiveness on post-trained financial LLMs.\n\n| Model Variant | JailbreakBench | AdvBench |\n| --- | --- | --- |\n| Original | 47% | 16% |\n| SafeReAct | **3%** | **2%** |\n\nCombining this result and former results stated in the paper, **one can see that our method is effective across reasoning, medical, and financial domains. Given the diversity of these tasks, we believe our findings generalize well to various types of post-trained LLMs.** From the method perspective, our method does not include some reasoning-specific techniques, which also demonstrates the potential for our method to be extended to other post-trained LLMs. But we will clearly state that “our evaluations only include reasoning, medical, and financial domains” in the limitation section.\n\n---\n\n---\n\n**Weakness 2:**  The performance of SafeReAct on larger models is unclear. The authors could test on other post-training tasks.\n\n**Answer to Weakness 2:**\n\n1. Larger Models:\n\nWe also conduct experiments of QwQ-32B and R1-distilled-32B with our SafeReAct, which is the largest model we can run on a single 80GB A100. The results on JailbreakBench are listed as follows:\n\n| Model Variant | R1-32B | QwQ-32B |\n| --- | --- | --- |\n| Ori | -% | 23% |\n| SafeReAct | 3% | 2% |\n\nFrom the results, one can see that our SafeReAct can still effectively improve 32B model’s safety. Considering our experiments already include 7B, 8B, 14B, and 32B with satisfying performance. We believe our method can also scale to larger ones if the resources are available. \n\n2. Other post-trained Tasks:\n\nPlease see Weakness 1.\n\n---\n\n---\n\n**Question 1:** Why are 20% of the neurons randomly pruned? Is this comparable to the pruning of reasoning neurons?\n\n**Answer to Question 1:** The reason we provide the random pruning baseline is to demonstrate that the reported safety gain found in Section 3.2 is not a trivial result and is related to the removal of LRM’s reasoning capabilities. \n\nThe 20% pruning rate is comparable to or even larger than the practical pruning rate for models with reasoning neurons being pruned. The reason is that most neurons in LLMs are related to general tasks and can also be activated in reasoning tasks, like neurons related to QA or instruction following.\n\n---\n\n---\n\n**Question 2:** What is the intuition behind SafeReAct? With the specially designed loss, do the neuron functionabilities of the task and safety mechanisms coincide? Can the authors do the test in Figure 2 after applying SafeReAct?\n\n**Answer to Question 2:**\n\n1. The intuition behind SafeReAct: The initial idea is that we believe LLMs’ original safety mechanism will not be fully removed, as its original model usually has satisfying safety, and we believe finding old mechanisms back is cheaper than learn a new one. Just as humans can quickly regain a skill they haven't used in a long time by reviewing it, rather than learning a new skill from scratch. Some works related to LLMs’ unlearning or fine-tuning both demonstrate that some unlearned or early learned knowledge is still hidden in LLMs and can be reactivated in certain ways. These results strengthened our confidence in finding post-trained LLMs “lost safety mechanism”. \n2. About specially designed loss: With LoRA fine-tuning on the designed loss,  the neuron functionabilities of the task and safety can be effectively activated mainly on their related prompts. As one can see from Table 2 and Table 5, the models’ safety and task performance (reasoning or medical problems) are both satisfactory.\n3. About Figure 2 after SafeReAct: The SafeReAct model will still be safe, as Figure 2 only prunes the reasoning neurons. Therefore, the ASRs for pruning SafeReAct models’ reasoning neurons are also close to 0.\n\n---\n\n---\n\n**Question 3:** Can you include more choices of alpha in Table 3 to show the transition?\n\n**Answer to Question 3:** Sure, we add 1 and 15 on R1-7B for the experiments as listed in the following table.\n\n| Model Variant | JBB | GSM8K |\n| --- | --- | --- |\n| 1 | 3% | 89% |\n| 2 | 0% | 88% |\n| 5 | 1% | 86% |\n| 10 | **1%** | 91% |\n| 15 | 1% | 86% |\n| 20 | 0% | 87% |\n\nFrom the results, one can see that different choices of $\\alpha$ show some slight influence on the final results, but there is no clear transition for it. Considering its training time is significantly shorter than SafeChain (20min vs 10 Hours on a single A100 under our empirical setting for 8B models), the developer may run several times with different alpha (like 5,10,20) if they want to find the model with the best utility.\n\n---\n\n---\n\n**Question 4:** What can be done to ensure both safety and task performance during post-training? For example, by including a safety term?\n\n**Answer to Question 4**: Yes, we believe adding our align loss as a regularizer in the post-training stage could be an effective way.  As one can see from our experiment sections (like Table 2), only retraining with our align loss for 300 LoRA epochs (about 20 min training on 8B models) can make LLMs find their safety back. It demonstrates the strong effectiveness of our alignment loss in restoring models’ safety. Therefore, we believe including it in the post-training stage as an align loss will also be effective to ensure both safety and task performance.\nBesides adding loss, collecting more safety data is also a possible way, but it will add a lot of additional cost on the data preparation and training stage."}]

Author rebuttal:

Meta-review: This paper claims that Large Reasoning Models often produce unsafe reponses not because their safety capabilities are absent, but because these capabilities are not activated during the reasoning process. The authors support their argument with analysis that distinguishes between neurons involved in reasoning and safety. Accordingly, they propose SafeReAct that enhances the safety of Large Language Models while retaining their reasoning capabilities. The reviewers had concerns regarding the design of loss function, limited model size, limited domain of experiments, and lack of source code. 

The authors's revisions have effectively addressed most of the concerns, and the strengths of the paper outweigh the drawbacks, but it is strongly recommended that they release the source code.
Decision: Accept (poster)