Title: Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM safety

Alignment
Abstract: Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or post-training to excel at specific tasks. 
    For instance, large reasoning models (LRMs), such as the DeepSeek-R1 series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of-thought (CoT) datasets. 
    However, this additional training frequently comes at the cost of reduced safety, as the fine-tuned or post-trained models tend to exhibit more harmful behaviors compared with the regular LLMs before post-training or fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. 
    Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. 
    Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. 
    But luckily, we also find that LRMs' safety mechanisms still exist instead of being removed during their post-training. 
    Based on these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters on a few layers. Experiments on four state-of-the-art LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach.


Peer reviews:
comments: SUMMARY:
The paper studies the effect of LLM post-training on model safety, with a primary focus on LRMs. An important behavior is revealed: the safety mechanism is not removed, but rather hidden. The authors further demonstrate that this hidden safety mechanism can be reactivated either through the over-activated LLM mechanism related to post-training or by using a safety suffix.

STRENGTHS AND WEAKNESSES:
Strengths

* The paper studies a significant problem -- what causes the safety degradation of LLM after post-training.

* The paper proposed an interesting hypothesis that the safety mechanism has been masked after post-training and validated this hypothesis.

* The experimental design is comprehensive in general, with adequate baselines and suitable metrics.

Weaknesses

* It is good for the authors to focus on LRMs for a in-depth study. However, the claim that the observation can be generalized to other applications needs more support.

* The performance on SafeReAct on larger models is unclear. The authors could test on other post-training tasks.

QUESTIONS:
* Why 20% of the neurons are randomly pruned? Is this comparable to the pruning of reasoning neurons?

* What is the intuition behind SafeReAct? With the specially designed loss, do the neuron functionabilities of the task and safety mechanisms coincide? Can the authors do the test in Figure 2 after applying SafeReAct?

* Can you include more choices of alpha in Table 3 to show the transition?

* What can be done to ensure both safety and task performance during post-training? For example, by including a safety term?

LIMITATIONS:
The limitations have been discussed.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors explore the phenomenon observed in reasoning models whereby these models appear to improve at reasoning tasks at the expense of safety, with performance drops observed on safety benchmarks. The authors demonstrate through adversarial prompting and pruning experiments, that this ability remains latent in reasoning models. Based on this insight, they propose SafeReAct, a mechanism to align a reasoning model’s internal safety representations with its reasoning circuitry. Experiments reveal that their approach can lead to significant improvements on reasoning benchmarks with minimal impact on reasoning benchmarks. This paper addresses a notable limitation of reasoning models and makes an important contribution to the field of LLM safety.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* This paper addresses an important limitation of reasoning models, tackling this problem in a novel manner. Their experimental results suggest their approach out-performs many similar approaches, making this a noteworthy contribution.\n* The paper provides a convincing rationale for their SafeReAct approach, demonstrating through several experiments that safety representations remain latent within reasoning models.\n* The paper is well structured and clearly and coherently describes their approach.\n\n## Weaknesses\n### Major\n* The authors don’t address anywhere in the manuscript the possibility that their approach might be over-activating safety mechanisms, leading to excessive safety responses. One of the benchmarks used (XSTest), tests explicitly for this, but it is unclear if the presented results are conditional only on the “unsafe” labelled results. I will be prepared to upgrade my evaluation score if the authors can provide assurance that this is not the case.\n* All the reported experiments and results are for small reasoning models only. It is not clear that either the insights observed (latent safety mechanisms) or the experimental results for SafeReAct would necessarily apply to larger models, where different mechanisms may underly reduced safety performance. Running these experiments on larger models may be too resource-intensive however at the very least there needs to be more discussion of this limitation in the paper.\n* For the medical-reasoning model experiments, the reported safety results were for the same safety benchmark used for other models, however for domain-specific applications it would be much more informative to assess the extent to which domain-specific safety is impacted. For example, for medical safety, a medical-specific safety benchmark such as MedSafetyBench[1] would be more appropriate.\n### Minor\n* It would be good to see some discussion in the Related Work of some of the prompt-alignment methods used in section 3.1.\n* On line 165, the authors write “one can see that randomly pruning neurons will not influence R1 models’ safety”. This doesn’t seem true from the figure at all, several models appear to display a drop in the harmful rate with random pruning. The authors should re-word this statement so that it is more faithful to the results.\n* On line 178 the authors write “harmful prompts tend to strongly trigger the post-trained capabilities”. The prior section does not provide evidence of a claim this strong, only that reasoning-related neurons play some role in producing harmful content. Again, please re-word to a statement better supported by the results.\n\n[1] Han, T., Kumar, A., Agarwal, C., & Lakkaraju, H. (2024). MedSafetyBench: Evaluating and Improving the Medical Safety of Large Language Models. NeurIPS.\n\nQUESTIONS:\n1. (Line 194) Why is it necessary to condition “safe” representations on a “safe-suffix” prompt? Do you have any results when a safe prompt isn’t used?\n2. Aside from the α parameter, how were other hyperparameters chosen?\n3. As mentioned in weaknesses, did the authors use the full version of XSTest which also includes safe prompts? If so, what was the performance (rather than harmful rate) on this dataset?\n\nLIMITATIONS:\n* As mentioned above, the limitations section needs more discussion of the usage of only small reasoning models\n* Section 5.4 only presents results for a medical model. The limitations section could also make clear that it is not known how well SafeReAct would apply to other domains.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes that the safety of post-trained LLMs decreases because the safety mechanism is masked by the post-trained mechanism. It proposes the SafeReAct method, which restores the safety mechanism that is masked by post-training through representation engineering, while retaining the post-trained abilities as much as possible. Extensive experimental results support the validity of the conclusions and the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper is well motivated and clearly written.\n\n2. The core conclusion that “the safety of post-trained LLMs decreases because the safety mechanism is masked by the post-trained mechanism” is interesting and reasonable, supported by experimental evidence.\n\n3. The proposed SafeReAct method, compared to related work, avoids the need for additional data collection, offering higher efficiency. The implementation details are also thoroughly explained.\n\n4. The experimental setup is well-designed, with commonly used benchmark datasets and evaluation metrics. If the source code were available, the experiments should be easy to replicate.\n\n**Weaknesses:**\n\n1. Although the paper claims that the conclusions and proposed method are applicable to any type of post-trained LLMs, the experiments mainly focus on reasoning models. For other types of post-trained LLMs, experimental validation is limited to just one in the biomedical domain. There is a lack of validation for a broader range of post-trained LLMs.\n\n2. Although SafeReAct uses Equation (5) to allow the model to consider both safety and post-trained ability during optimization, I feel that, based on previous conclusions in the paper, these two objectives, $L_{align}$ and $L_{retain}$, should be mutually exclusive. Whether this loss can converge remains uncertain. It seems that the trade-off between safety and post-trained ability still needs to be considered, and it may not be feasible to balance both effectively.\n\n3. The symbol $||$ in $x_{harm}||s_{safe}$ of Equation (3) is used before being defined.\n\nQUESTIONS:\n1. Equation (2) represents the difference in Wanda scores between q neurons selected by target domain data and p other neurons selected by retain data. The result should be a single score, not a set of neurons. How can this score be used to determine which neurons are associated with the target domain?\n\n2. What is the meaning of the symbol $||$ in $x_{harm}||s_{safe}$ of Equation (3)?\n\n3. In Equations (3) and (4), what exactly do $x_{harm}$ and $x_{retain}$ represent? Could the authors provide some examples?\n\n4. How is $\\mathcal{I}$ determined? Does it need to be calculated using the previously mentioned Equation (2)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors find that removing neurons most responsible for post-trained ability could restore almost full safety performance. They further propose a LoRA finetuning method which incorporates both a representation-alignment loss and a retaining loss. They show that with this proposed mechanism, they can restore safety ability from LRMs without significantly changing the post-trained ability. The method is also less computationally expensive than other representation-based methods like representation engineering.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The work points out that safety ability is not removed, but simply the post-trained abilities are overly activated. This is a useful insight for understanding LRMs' safety behavior.\n- The method does not rely on particular safety behavior data as anchor.\n- The method is Lora finetuned and relatively cheap.\n\nWeaknesses: \n- In section 3.1, the authors discuss recovering safety ability from LRMs with prompting. However, the prompting methods (see their references [23], [24]) are simple prompt-based methods and do not fully leverage the safety potential of LRMs. For example, using deliberative alignment (Guan et al., 2024) type of framework and providing explicit safety policies to follow likely will result in much stronger safety performance, and with lower impact on capabilities. \n- The paper uses only two math datasets as utility measurement. what about chat performance? QA? truthfulness etc.\n- The paper does not compare performance/ efficiency against some non representation-based methods. The SafeChain method is not Lora fine-tuned, so the comparison is not entirely fair.\n- The paper does not include supplementary material for code and additional analysis.\n\nQUESTIONS:\n- I suggest including some theoretical or more careful learning dynamics analysis to compare the different methods. The authors touch on this topic in limitations section, but I think to better understand the effectiveness of this method, some more systematic analysis is needed. For example, tcomparing The authors include some discussions on limitations towards the end of the paper, but can do a more detailed analysis on the topic, and some I would suggest including in the main text to make the work more complete (see suggestions section). further training the reasoning models on safety and compare the neurons activation patterns between a safereact trained model and a model that actually goes through further safety alignment.\n- The framing of the paper for \"post-trained LLMs\" is too broad.\n- In equation (2), why use different p, q for selection?\n- For figure 2, can you also show reasoning ability (in addition to safety) before vs. after these interventions?\n- I also encourage the authors to do a more careful writing revision of the paper. There are various places with typos or grammatical errors throughout the paper.\n\nLIMITATIONS:\nThe authors include some discussions on limitations towards the end of the paper, but can do a more detailed analysis on the topic, and some I would suggest including in the main text to make the work more complete (see suggestions section).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-06"}