Title: LLMs Encode Harmfulness and Refusal Separately
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language models; Interpretability; AI safety
Abstract: LLMs are trained to refuse harmful instructions, but do they truly understand
harmfulness beyond just refusing? Prior work has shown that LLMs’ refusal
behaviors can be mediated by a one-dimensional subspace, i.e., a refusal direction.
In this work, we identify a new dimension to analyze safety mechanisms in LLMs,
i.e., harmfulness, which is encoded internally as a separate concept from refusal.
And there exists a harmfulness direction that is distinct from the refusal direction.
As causal evidence, steering along the harmfulness direction can lead LLMs to
interpret harmless instructions as harmful, but steering along the refusal direction
tends to elicit refusal responses directly without reversing the model’s judgment on
harmfulness. Furthermore, using our identified harmfulness concept, we find that
certain jailbreak methods work by reducing the refusal signals without suppressing
the model’s internal belief of harmfulness. We also find that adversarially fine-
tuning models to accept harmful instructions has minimal impact on the model’s
internal belief of harmfulness. These insights lead to a practical safety application:
The model’s latent harmfulness representation can serve as an intrinsic safeguard
(Latent Guard) for detecting unsafe inputs and reducing over-refusals that is
robust to finetuning attacks. For instance, our Latent Guard achieves performance
comparable to or better than Llama Guard 3 8B, a dedicated finetuned safeguard
model, across different jailbreak methods. Our findings suggest that LLMs’
internal understanding of harmfulness is more robust than their refusal decision
to diverse input instructions, offering a new perspective to study AI safety.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly shows that LLMs internally encode harmfulness and refusal as distinct concepts in different parts of their latent representations. By analyzing hidden states, the paper identify separate directions for harmfulness and refusal, and demonstrate that steering models along these directions affects behavior differently. Leveraging this separation, the paper proposes a “latent guard” mechanism that detects harmful prompts more reliably than traditional safeguards, even under jailbreak or fine-tuning attacks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- The motivation is direct and clear, leveraging human cognition to make analogy to LLM behavior. The investigation itself is also meaningful in ways that it could provide guidance on how to effectively align LLMs internal belief instead of just superficial behavior.\n- The preliminary experiments in validating the hypothesis is clean, comprehensive and interesting, even though it may take a while to fully comprehend the complex setting\n- The whole paper is written in a very comprehensive and progressive way, which is good: a question is proposed, hypothesis is brought out and then validated, gradually digging deeper to the problem from both theoretical analysis and empirical training sides.\n\nWeakness:\n- The introduction part is a bit crowded with concepts that are not well-defined (harmfulness direction, harmfulness level), making the reading a bit hard to follow all the details, especially starting from Line 47 to 60\n- The real application of the discovery, though proved to be effective in most cases against Llama Guard, seems is not very cost-friendly and the calculation and clustering itself may consume a lot of computational resources, and the generalizability of the method is not well-shown (mainly for Section 5)\n\nQUESTIONS:\n- Will some latest instruction model still exhibits the phenomenon that is described in Section 3? Like Qwen3 and Llama4, or some loopholes are already being patched through somme alignment techniques? Also, even though base models are not trained on labels like 
, could your research setting be applied to base models and yield meaningful discoveries?\n- Will the refusal rate calculation method  in Line 112 be inaccurate as some model may say “sorry I cannot” but still answers the malicious question? Will third-party model judgment be more accurate than rule-based ones?\n- For the setting in Section 3.5, is the superficial words such as “No” or “Certain” effective to reflect the refusal behavior? From the example given in the introduction part, it seems this inversion is also kind of superficial and seems like only models extremely overfit to refusal without any reasoning ability will likely to be tricked. So I am a bit doubt about the effectiveness of this designed task to support your arguments.\n- Typo Line 176\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper disentangles representations of \"harmfulness\" (i.e., the input is a harmful request) and \"refusal\" (i.e., a refusal should be generated in response to the request). The authors find that \"harmfulness\" is saliently represented at the last token of the instruction, while \"refusal\" is saliently represented at the last token of the prompt template. The authors use clever experiments such as the \"reply inversion task\" to disentangle these two representations, and show that they are indeed distinct. They further show that, for some jailbreaks, the representations of refusal are modified, while the representations of harmfulness are not. This motivates a monitoring method (\"latent guard model\"), which works by detecting the harmfulness of the user query via its representations along the harmfulness directions. For some jailbreak methods and models, this monitoring approach outperforms Llama Guard 3.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Well-written, clear paper.\n  - In particularly like the coloring of the token labels - this makes things easier to read.\n- Convincing evidence of main claim\n  - I think there is convincing evidence to support the main claim that refusal and harmfulness are distinct representations in the models that are studied here.\n- Strong results for \"latent guard model\"\n  - The results in Table 3 indicating that measuring the model's representation of harmfulness could be a good classifier for harmful requests.\n\nWeaknesses\n- Naive refusal metric\n  - Uses substring-based refusal classification to classify whether a response is a refusal or not. Multiple prior works have commented on the limitations of this approach, and have suggested augmenting this approach with a semantics-based classifier [1, 2, 3].\n- Metrics $\\Delta_{\\text{harmful}}, \\Delta_{\\text{harmless}}$ are not well motivated.\n  - It's not clear why this is the right metric. It's looking at the difference in cosine similarities, averaged across layers.\n\nReferences:\n1. Souly et al 2024 (https://arxiv.org/abs/2402.10260)\n2. Meade et al 2024 (https://arxiv.org/abs/2404.16020)\n3. Arditi et al 2024 (https://arxiv.org/abs/2406.11717)\n\nQUESTIONS:\n- Why use only a substring-based refusal classification metric? Multiple prior works have commented on the limitations of this approach, and have suggested augmenting this approach with a semantics-based classifier [1, 2, 3]. Also, at the very least, I think the authors should discuss how they came up with the set of refusal strings, and provide the refusal strings in the appendix for reproducibility of results.\n\n- How are metrics $\\Delta_{\\text{harmful}}, \\Delta_{\\text{refuse}}$ motivated? It's not clear why this is the right metric - it's looking at the difference in cosine similarities, averaged across layers. Why cosine similarity, rather than projection? Why average over multiple layers, rather than picking the most separable one? How should we interpret these quantities?\n\n- On line 216, the authors say \"these two directions have very low average cosine similarity (around 0.1)\". On what basis is this deemed a \"very low\" cosine similarity? If my understanding is correct, the dimensionality of these vectors is quite large, and so if the authors want to make this claim, then they should defend the claim by reasoning about reasonable baselines (for example, what's the standard deviation of the distribution of cosine similarity between random vectors from this high dimensional space - this would ground the observed quantity).\n\n- Why are the adv-suffix squares all extremely close to each other in Figure 3b? Can you comment on the volume of data used in the experiments for Figure 3? It looks like there are approximately 20-30 data points per cluster, and they look to be quite homogenous in Figures 3a and 3b. I think the results would be strengthened with significantly more data. I am surprised by the homogeneity of the results here, as I would expect the results to be a bit messier. Is this clean of separation to be expected?\n\n- [4] should be cited.  should be cited. They find the importance of prompt template tokens, similar to the results reported in section 3.1.\n\n- In the \"reply inversion task\", which layer is steering applied to? I'd find the result most compelling if it were applied only at a single layer (the most effective layer found from the previous section).\n\nReferences:\n1. Souly et al 2024 (https://arxiv.org/abs/2402.10260)\n2. Meade et al 2024 (https://arxiv.org/abs/2404.16020)\n3. Arditi et al 2024 (https://arxiv.org/abs/2406.11717)\n4. Jiang et al 2024 https://arxiv.org/abs/2406.12935\n\nLIMITATIONS:\n- The paper relies solely on substring matching to determine refusal rates, which is known to be unreliable.\n- The extremely clean and tight clustering of examples in Figure 3b potentially suggests insufficient data diversity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies LLMs’ internal perception of harmfulness, revealing that harmfulness and rejection responses are primarily encoded at different token positions and that LLMs may internally know the correct harmfulness level of instructions but still mistakenly accept or reject them.\n\nSTRENGTHS AND WEAKNESSES:\n## Summary Of Strengths:\n\n1. The article explores the recognition of harmful instructions in large language models (LLMs) and distinguishes how harmful and rejection responses are encoded at different token positions.\n2. Based on the understanding of harmful recognition in LLMs, the article proposes a model called Latent Guard, which can effectively identify and reject harmful instructions while reducing false rejections of harmless instructions.\n\n## Summary Of Weaknesses:\n\n1. Unclear description of cluster centers: In Figure 2, it is not clear whether the cluster centers of $t_{inst}$ and $t_{post-inst}$ are calculated independently or use a unified value. If the same cluster center is used to calculate the cosine similarity of these two positions, the parameter distribution may be significantly affected, resulting in an inaccurate reflection of the actual phenomenon.\n2. Specific mechanism of the inversion method: How to implement the inversion method mentioned in Section 3.5, especially how to effectively convert harmless content into harmful content instead of just adjusting the parameters to make the probability of generating logit from \"certainly\" to \"no\", can it be further clarified. Explanation is not simply changing the output tendency, which helps to provide a deep understanding of the difference between the harmfulness direction and the rejection direction.\n3. Positive and negative sample clustering characteristics: I hope to see more precise data to prove that the harmful and harmless sample clusters shown in Figure 2 do have obvious distinguishing characteristics. Although the current graphics and analysis intuitively demonstrate this concept, there is a lack of specific quantitative evidence to support the effectiveness and stability of this distinction.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper identifies separate harmfulness and refusal directions by analyzing the hidden states at 2 different tokens in the prompt. They perform several behavioral experiments to verify the differing behavior between the directions they found.

STRENGTHS AND WEAKNESSES:
This work applies existing methodology in interpretability to analyzing the harmfulness of instructions and whether a model will refuse it. Novel experiments separate out these two behaviors. This has significance in aiding existing efforts on model alignment.

QUESTIONS:
On line 216, between which vectors is the cosine similarity calculated? Is it the harmfulness direction in layer 9 and the refusal direction in layer 11, or the directions in layer 9, or the directions in layer 11?

When applying the steering directions, is it applied to all tokens from the beginning up until $t_{\text{post-inst}}$ for both the harmfulness and refusal directions, and on none of the output tokens?

Are the harmfulness and refusal vectors used in Sec 3.5 the same as the ones used in Sec 3.4? And are they applied to all of the additional tokens from the inversion prompting templates?

How does Section 5 compare with Section 6.2.1 of [Zou et al., 2023a]?

Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan,
Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, et al. Representation engineering: A
top-down approach to ai transparency. arXiv preprint arXiv:2310.01405, 2023a.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03