Title: Safety Depth in Large Language Models: A Markov Chain Perspective
Venue: NeurIPS 2025
Accepted: yes
Keywords: Safety Alignment

Large Language Models

Markov Chain

Group Theory

Ensemble
Abstract: Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts or even benign fine-tuning can bypass internal safeguards, underscoring the need to understand the failure modes of current safety strategies.  Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, we introduce the notion of safety depth, a designated output position where the model refuses to generate harmful content. While deeper alignment appears promising, identifying the optimal safety depth remains an open and underexplored challenge.

We leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, we propose a cyclic group augmentation strategy that improves safety scores across six LLMs. In addition, we uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can offset shallower alignments. These results suggest that test-time computation, often overlooked in safety alignment, can play a key role. Our approach provides actionable insights for building safer LLMs.


Peer reviews:
comments: SUMMARY:
Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, The authors leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, they propose a cyclic group augmentation strategy that improves safety scores across six LLMs. In addition, they uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can offset shallower alignments.

STRENGTHS AND WEAKNESSES:
Strengths：
1. The authors leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth.
2. The authors propose a cyclic group augmentation strategy that improves safety scores across six LLMs

Weaknesses：
1. The safety performance improvement is at the cost of the inference time.
2. Using 1 dataset that only contain 330 harmful instructions for investigating the effectiveness of the method for safety alignment is not enough

QUESTIONS:
1. Can you provide a comparison of the testing time when different methods are used to make predictions on the dataset?
2. Could you provide the experimental results on other datasets for testing security?
3. Could you provide the experimental results on larger LLMs such as Gemma3-27B?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the core question of “How deep should safety alignment extend?” It interprets the operation of inserting refusal responses and handling harmful instructions at certain safety depths as specific transformation instances applied to datasets from a group theory perspective, connecting safety alignment with permutation groups. This enables the enhancement of safety alignment by extending datasets through operations such as generating a cyclic group via rotational transformations. Specifically, the main contributions of the paper are:\n\n- Safety Depth: The paper formally proposes the concept of Safety Depth, which refers to the model’s refusal to generate harmful content at a specified position during the generation process. From a Markov chain perspective, it proves that iterative fine-tuning of autoregressive language models can theoretically guarantee arbitrary levels of safety.\n- Cyclic Augmentation: Under Cyclic Augmentation, the model’s safety guarantees remain valid while also accelerating convergence. Empirical results demonstrate that this approach significantly improves safety scores across various large language models.\n- Ensemble Safety Depth: Without requiring additional fine-tuning of individual models, the ensemble method maintains safety guarantees, offering a practical alternative for users seeking off-the-shelf safety solutions, albeit with a modest increase in inference-time computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper introduces a novel application of group theory to the data augmentation process in language model alignment, providing a systematic and detailed theoretical analysis. By examining the role of permutation groups in the alignment process, it demonstrates the feasibility of enhancing alignment through group action-based data augmentation.\n2. The theoretical findings are validated through a variety of experiments, which strengthens the rigor and credibility of the work.\n\nWeaknesses:\n\n1. Definition 4.4 seems to involve a strong implicit assumption that, once the model outputs a refusal state, it will with probability zero generate any further requests, including harmful ones. This assumption likely relies heavily on the model’s output consistency, which is not rigorously guaranteed. The lack of explicit mention and discussion of this assumption undermines the credibility of the theoretical results.\n2. As the authors note, there is a trade-off between safety and helpfulness in safety alignment. However, the modeling process appears to consider “not entering an absorbing state” as the sole guarantee for helpfulness. This is insufficient to ensure that the model behaves helpfully. Furthermore, when applying the group-theoretic Markov analysis framework to domains without clear absorbing states—such as the continuum between helpful and unhelpful behavior—the current modeling approach seems to have certain limitations. How this issue can be addressed remains unclear.\n3. In the experimental evaluation, only LoRA fine-tuning is used to analyze results. It remains to be seen whether the method maintains its advantages when using full-parameter supervised fine-tuning (SFT). Additionally, can this approach be effectively integrated as a complement to existing RLHF algorithms?\n\nQUESTIONS:\nFor detailed comments, please refer to the Weakness section. If the authors could clarify the issues raised therein, I would be happy to consider raising the score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of \"safety depth\" for Large Language Models through a Markov chain perspective. The authors propose using cyclic group augmentation to improve safety alignment and provide theoretical guarantees for achieving absorbing refusal states. They demonstrate that ensemble methods can distribute safety constraints across multiple models and validate their approach on several open-source LLMs.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- Principled theory: The Markov chain perspective on LLM safety alignment is innovative and the results are rigorous.\n- Experimental validation: Testing across multiple models (Gemma-2-2B, Phi-2-2B, Qwen2.5-1.5B, etc.) and safety categories demonstrates broad applicability.\n- Practical ensemble approach: Proposition 4.10 offers a practical alternative for users seeking off-the-shelf safety solutions.\n\n## Major Concerns\n\nMy main concerns are on exposition and conceptual rigor.\n- **Safety Depth**: The core concept of \"safety depth\" remains unclear despite being central to the paper. The definition as \"a specific position of output responses in the training samples where the model declines to generate potentially harmful content\" (Line 127) is vague. The examples provided don't sufficiently clarify this concept, and more intuitive exposition would be valuable.\n- **Reasoning behind using cyclic Augmentations**: The claim that insertions \"represent specific instances of broader transformations on the dataset\" (Line 34) from a group-theoretic perspective lacks justification. Why should random insertions be viewed through this lens? The connection feels forced rather than natural. Even after reading Section 4, the importance and intuition behind cyclic augmentation remains unclear. The paper jumps into technical details without establishing why this particular approach is meaningful for safety alignment.\n- **Evidence for Claims**: The assertion that \"cyclic augmentation ensures that refusal states appear at diverse positions, improving the model's ability to decline harmful generation\" (Line 220) lacks empirical support. How do we know this is actually happening?\n- **Insufficient exposition, examples are not helping**: Section 4 is extremely dense and hard to follow. The examples don't help understand the significance of the results because I'm lost on what exactly is going on. I'd rather have no examples but clearly spelt out assumptions and lots of exposition to explain plainly what each result means.\n- **Population-level results**: If I understand correctly, all results are *population-level*. That's totally fine, but only in conjunction with the justification that they continue to hold in a setting where the population-level markov chain is replaced by the LLM stochastically trained on data, i.e. accounting for the randomness coming from dataset composition and training dynamics.\n- **Practical Limitations**: The ensemble approach appears to require multiple copies of LLMs, which raises serious concerns about computational and memory requirements. This limitation is not adequately addressed, making the practical applicability questionable.\n- **Limited Experiments**: While the authors test multiple models, the experiments are primarily \"proof-of-concept\" (as acknowledged in limitations). The gap between theoretical guarantees and practical deployment remains significant.\n\n## Minor Issues\n- NTK Connection (Line 41): The relationship between Neural Tangent Kernel theory and the proposed Markov chain approach is mentioned in passing, but never properly developed. This connection should either be substantiated or removed.\n- Figure 4: Text is too small to read clearly\n\nQUESTIONS:\n1. Can you clearly and formally define safety depth with concrete examples?\n2. What is the computational overhead of the ensemble approach in practice? Please explain if this is a realistic method to use in practice.\n3. How do you empirically verify that cyclic augmentation actually improves refusal positioning as claimed?\n4. Why is the group-theoretic perspective necessary?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper formulated safety alignment for autoregressive LLMs as a Markov-chain problem and introduces safety depth—an output position at which the model is trained to refuse. By modeling refusal states as absorbing states (once entered, the chain almost never escapes), the authors give the theoretical analysis of choosing optimal depths. By proposing cyclic augmentation on the LLM training data, the author demonstrates the safety score can be improved, especially under different LLM model ensembles.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper provides a rigorous Markov-chain formalization of autoregressive language generation, which clarifies how “refusal” tokens can be treated as $\\delta$-absorbing states and introduces the notion of a model’s safety depth (Definition 4.1). By formulating safety alignment under their theoretical framework, the authors can prove convergence guarantees.\n\n\n\n2. The authors first test their theoretical analysis simple Markov environments (e.g., 4 states), and demonstrate their findings on several open-source LLMs, showing that cyclic augmentation and ensemble width indeed raise safety scores.\n\n**Weaknesses**\n1. Despite the theory analysis, there is a noticeable gap between theory and large-model experiments. For example, the depth choices justified in Section 3 are not mapped directly onto the reported LLM empirical results; as a result, it is hard to justify where the safety improvements come from.\n\n2. Authors do not justify the theorecial assumptions. For example,  3.1 and 3.4 appear strong and are not empirically verified. Using more examples to illustrate can help reader better understand the assumptions. \n\n2. A lot of experimental setup and details are missing from the main body of the paper, which makes the experiment section hard to understand. For example, The experimental section omits the description of several key methods (e.g., shallow, deep, cyclic), making the experiment section hard to understand.\n\n4. The helpfulness-safety trade-off is under-analyzed. While the authors acknowledge a modest drop in helpfulness, they do not analyze whether this stems from safety depth itself or from the specific form of cyclic data augmentation, or other factors.\n\nQUESTIONS:\n1. I am not familiar with group notions. Can you give me more examples to better justify assumptions 3.1 and 3.4? For example, to what extent do you think the assumptions can hold?\n\n2. In eq. (8), how do you derive the hitting probability?\n\n3. In lines 276-277,  can you give more details of the three data augmentation strategies (shallow, deep, cyclic)?\n\n4. In figure 5, can you describe more how you use an ensemble to make the paper safer?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}