Title: LLM Safety Alignment is Divergence Estimation in Disguise
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Theory

Safety Alignment

RLHF

Divergence Estimation
Abstract: We present a theoretical framework showing that popular LLM alignment methods—including RLHF and its variants—can be understood as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less-preferred) distributions. This perspective explains the emergence of separation in the latent space between safe and harmful prompts after alignment. As an application of our general divergence framework, we propose KLDO, a novel KL divergence-based alignment method, and empirically validate its effectiveness. We further show that using compliance–refusal datasets, rather than standard preference-based datasets, leads to stronger separation and improved safety alignment. Finally, to quantify the separation effect, we propose a distance-based metric in the prompt representation space, which also acts as a statistically significant indicator for model safety.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on understanding and improving the effectiveness of LLM safety alignment. Specifically, the author analyzes several representative alignment methods from the perspective of divergence. The preliminary analysis suggests that alignment methods based on KL divergence may be more effective. Based on this insight, the author proposes a new method called KLDO, and experimentally evaluates its safety and usefulness compared to other methods such as DPO, KTO, and BCO.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The author's writing is clear and easy to follow, making it simple for readers to understand the ideas of the paper.\n2. The author analyzes existing alignment methods from the perspective of divergence and argues for the advantages of designing alignment methods based on KL divergence. Based on this, the paper proposes KLDO. The results across different models suggest that KLDO shows seemingly decent safety performance.\n\n# Weaknesses\n1. Although the author analyzes existing alignment methods from a divergence perspective and argues that alignment methods designed using KL divergence (i.e., KLDO) are superior, the experimental results shown in Figure 3 and Table 2 do not clearly demonstrate that KLDO outperforms other methods — in fact, it often performs worse than BCO. Therefore, I believe the theoretical contributions of the paper do not align well with the experimental evidence, raising concerns about the practical value of this theoretical approach.\n2. I am confused about the “overall robustness score” metric. The authors describe it as “a standardized weighted average of all the above robustness benchmark metrics,” but the paper does not explain how it is specifically calculated. It is unclear how such a score results in a 20-point difference (95.02 vs. 72.13) between LLaMA3.2-1B-KLDO and LLaMA3.2-1B-BCO in Table 3, especially considering their similar performance across different datasets.\n3. Although the paper claims to focus on LLM safety, both the baseline methods and the proposed method seem to perform poorly in terms of safety. For example, in Table 3, the attack success rate on SALAD is high for all models and methods. Compared to other state-of-the-art safety methods (e.g., circuit breaker [1]), the practical value of the approach in this paper appears to be quite limited.\n\n\n[1] Zou A, Phan L, Wang J, et al. Improving alignment and robustness with circuit breakers[C]//The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.\n\nQUESTIONS:\n1. Could you please explain why KLDO often performs worse than BCO in the experiments?\n2. May I ask how exactly the overall robustness score is calculated?\n3. In the experiment, the following models were selected: Llama 3.2-1B, Llama2-7B, Gemma2-2B, Mistral v0.1-7B, and Qwen2.5-1.5B. Could you explain the selection criteria? Some of the models are outdated (e.g., Llama2-7B, Mistral v0.1-7B), while others are very small in size (e.g., Llama 3.2-1B, Gemma2-2B, Qwen2.5-1.5B).\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSafety alignment is primarily used to ensure that LLMs avoid generating harmful content in response to malicious inputs. Recent studies have observed that the hidden representations of safe and  harmful prompts in aligned LLMs tend to form well-separated clusters. This paper introduce a unified theoretical framework that captures this mechanism and analyze its implications for both safety and robustness. The authors first establish that commonly used alignment methods correspond to specific divergence , providing initial validation that alignment methods induce separation. In addition, they propose a novel alignment method called KLDO, which is based on KL-divergence estimation. The authors also define the ability of alignment methods to enable models to redistribute their output probabilities according to the true likelihood of a response being favorable or unfavorable as “alignment consistency.” Experiments are conducted to evaluate the performance of KLDO on robustness and separation metrics. The results demonstrate that KLDO achieves a balance between utility and robustness, and that separation is a reliable indicator of robustness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- **Novel perspectives.**  \n\n  - It establish that popular alignment methods can be interpreted as divergence estimators between aligned (safe or preferred) and unaligned (harmful or less-preferred) distributions. Furthermore, the authors introduced a theoretical property to define this phenomena named alignment consistency.\n  - The article introduces KLDO, an alignment method capable of directly estimating KL divergence, and further extends it to FDO, a method applicable to general f-divergence. The authors demonstrate the effectiveness of these methods through both theoretical analysis and empirical validation.\n\n- **Relatively comprehensive experimental setup**\n\n  - The experimental results presented in this article effectively validate the conclusions about alignment consistent. Furthermore, it is found that separation is an important indicator of robustness and KLDO achieves an optimal balance between utility and robustness, while the parameters are comprehensive and the results are highly reproducible.\n  - The authors have established a comprehensive experimental setup, which verifies the effectiveness of the proposed KLDO method. Moreover, the experimental results are consistent with the issues raised in Lines[28–31] as well as the stated contributions.\n\n- **Logical Structure**\n\n    - The theorems proposed by the authors are all sufficiently explained and supported by theoretical analysis or mathematical derivations.\n\n\n## Weakness\n\n- **Redundant Contributions**\n\n  The contributions listed in Line[36-53] appear to be artificially inflated through redundancy. Points 1, 3, and 4 all center around the core concept of “separation,” approaching it from theoretical analysis, experimental validation, and quantitative measurement, respectively. As such, there is a degree of redundancy among them, and they should be consolidated into a single main contribution.\n\n- **Lack of rigorous cross-scale evaluation**\n\n  The experimental setup lacks sufficient rigor to reveal how evaluation metrics change for the same model across different parameter scales. For example, why was the Llama 3.2-3B model not selected, while the Llama 2-7B was included instead? This choice is rather confusing.\n\nQUESTIONS:\n**Writing Issues**\n\n- Is there a incorrect citation in Line[90]?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nInspired by the phenomenon of “separation effect”, i.e., the hidden representations of safe and harmful prompts in aligned LLMs form well separated clusters, this paper: 1) establishes a theoretical framework to explain the correlation between distributional separation and alignment, which formalizes alignment methods as divergence estimators; 2) designs a new alignment method KLDO based on KL-divergence estimation; 3) proposes a property “alignment consistency” that validates the separation is amplified using compliance-refusal data instead of preference data; and 4) designs metrics to quantify the separation and its significance for robustness.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper provides a theoretical framework to explain the causal relationship between various alignment methods and the separation effect phenomenon.\n2. Based on this theoretical framework, a new alignment method is proposed.\n3. Experiments are conducted to verify the effectiveness of the newly proposed alignment method, as well as the significance of separation for indicating robustness.\n\nWeaknesses\n1. The title of the paper is confusing. What is the meaning of “Disguise” in the title?\n2.  The introduction in Sec 3.2 Data Distribution is somehow unclear. What do “compliant” and “refusal” response mean? What are the differences among Compliance-Refusal data and Preference data. \n3. Also for the Compliance-Refusal data and Preference data, more explanation is required for the observation in Sec 5.4. What is the fundamental reason?\n4. Some results are not very consistent. For example, the results of KTO is lower than DPO on Llama-3.2-1B in Table 2; some methods show better results than KLDO in Figure 4.\n5. The phenomenon of “separation effects” and its indication significance of robustness is only validated on safety alignment. Can it be generalized to other domains?\n6. There are still some typos. And the conclusion section is lacked.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work attempts to unite various popular alignment methods under the umbrella of divergence estimation between aligned and non-aligned distributions. This leads them to introduce KLDO, a KL divergence-based alignment method, and demonstrate that compliance-refusal datasets yield stronger separation than preference-based datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The paper does a thorough study of the relationship between separation and robustness; and find that they are correlated\n2. The finding that compliance-refusal data leads to stronger separation than preference data is useful for future work.\n3. The theoretical investigateion is quite interesting, and also leads to a new practical method that improves upon past baselines.\n\n## Weaknesses / Questions\n1. This paper builds on the assumption that standard alignment methods are the gold standard, and then try to reverse engineer what is the fundamental characteristic that unifies them. While this approach may seem worth while when alignment was known to be successful. At this point there is sufficient evidence in literature showcasing the ease of jailbreaking aligned language models.\n1. Its unclear to me that separating the harmful and safe responses is the correct solution here. This can lead to two different models residing within one model, and this would lead to certain adversarial triggers producing the harmful model. This would make the model much more brittle in fact, as opposed to a model that has a shared learning of safe and unsafe behaviors and is able to delineate between the choices to make safe responses.\n\nQUESTIONS:\nMy main question is posted in the weaknesses section. I am not too sure about the bridge that leads to calling a method \"alignment consistent\" if it follows the separation idea. I think alignment is much broader and the separation idea is one instantiation of it. I am not too excited by its reduction.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}