Safety Depth in Large Language Models: A Markov Chain Perspective
Authors: Ching-Chia Kao
Chia-Mu Yu
Chun-Shien Lu
Chu-Song Chen
Venue: NeurIPS 2025 poster ()
Keywords: Safety Alignment
Large Language Models
Markov Chain
Group Theory
Ensemble
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts or even benign fine-tuning can bypass internal safeguards, underscoring the need to understand the failure modes of current safety strategies.  Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, we introduce the notion of safety depth, a designated output position where the model refuses to generate harmful content. While deeper alignment appears promising, identifying the optimal safety depth remains an open and underexplored challenge.

We leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, we propose a cyclic group augmentation strategy that improves safety scores across six LLMs. In addition, we uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can offset shallower alignments. These results suggest that test-time computation, often overlooked in safety alignment, can play a key role. Our approach provides actionable insights for building safer LLMs.

Safety Depth in Large Language Models: A Markov Chain Perspective
Ching-Chit Chii kaol.2; Chia-Mu Yu², Chun-Shien Lu', Chu-Song Chen?
Institute of Information Science & Research Center for Information Technology Innovation, Academia Sinica Departmentt of Computer Science and Information Engineering, National Taiwan University 3 Department of Electronics and Electrical Engineering, National Yang Ming Chiao Tung University
Abstract
Large Language Models (LLMs) are increasingly adopted in high-stakes scenarios, yet their safety mechanisms often remain fragile. Simple jailbreak prompts even benign fine-tuning can bypass internal safeguards, underscoring the need understand the failure modes of current safety strategies. Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, introduce the notion of safery depth, a designated output position where the model refuses to generate harmful content. While deeper alignment appears promising, identifying the optimal safety depth remains open and underexplored challenge.
We leverage the equivalence between autoregressive language models and Markov chains derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, we propose cyclic group augmentation strategy that improves safety scores across six LLMs. addition, we uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can off'stt shallower alignments. These results suggest that test-time computation, often overlooked in safety alignment, can play key role. Our approach provides actionable insights for building safer LLMs.
Introduction
Although Large Language Models (LLMs) Touvron al. 2023 Achiam et al. 2023 Team et al.)
22023] Duan et al., [2023] Ouyang et al. 2022] excel diverse applications, they often produce harmful content. Reinforemmntt Learning from Human Feedback (RLHF) [Ouyang et al. 2022
Bai et al. 2022 and its variants, Direct Preference Optimization (DPO) [Rafailov et al. 2024 and Kahneman- - Optimization (KTO) (Ethayarajh al.) 2024). aim to mitigate this issue. However, recent studies have shown that adversarially optimized inputs can still elicit harmful content
2024]. and even benign fine-tuning can break existing alignments [Qi et al. 2024 Zhan et al. 2024
(Qi al. 2023 Carlini et al. 2024 Chao et al. 2023 Andriushchenko
A recent study [Qi al.) 2025 uncovered that this vulnerability stems from limiting safety alignment to only the initial output tokens, a practice termed shallow safety aligmment. They introduced data augmentation method to deepen alignment, leading to the question: "How extensive should safety alignment be?" To address this, we divide the problem into managcable steps. First, we ask, "What does it mean for an alignment to be safe?" We consider finite set Y of harmful content, such as explicit sexual, violent, or private information, that we intend the model to avoid. An LLM is safely aligned if the probability of generating any content in Y is extremely low.
Second, rather than always placing a refusal at the beginning, Qi et al. (2025) randomly picked safety depth from a uniform distribution and insert refusal response along with harmful instructions
"Correpondencee to: d119220150csie attu.edu..
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

at that position. From a group-theoretic perspective, these insertions represent specific instances of broader transformations on the dataset. Motivated by this, we pose another question: "What if the dataset is augmented by rotation, forming a cyelic group?" (see Figure This framework links
safety alignment to permutation groups, topic we explore in Section M For readers less familiar with group theory, Appendix A offfers concise overview.
[I cannot, say, bad words]
[say, bad words, cannot
(bad words, cannot, sayj
(I camnot, say, bad words] (say, cannot, bad words]
(say, bad words, cannot
Figure I: Permutations of phrases used for data augmentation. The top row represents a cyclic group, while the bottom row, as proposed by Qi et al. 2025]. non-cyelic.
Lastly, while many works Malladi et al. 2023 Jang al. 2024 Tomihari and Sato 2024 rely on the Neural Tangent Kernel (NTK) [Jacot et al. 2018 to analyze fine-tuning and Gerken and Kessel [2024] demonstrates that deep ensembles can become fully equivariant through data augmentation over all group actions, we adopt a Markov chain perspective. Markov chain benefits without relying on the linearization assumptions. Detailed comparisons discussed in Appendix B
Our contributions are threefold:
* Safety Depth. We formalize the notion of Safety Depth, a designated output position where the model refuses to generate harmful content. Viewing this through a Markov chain lens in Theorem 4.6 provides theoretical assurances of arbitrarily safe behavior via iterative fine-tuning on autoregressive L.LMs.
Cyclic Augmentation. Proposition 4.9 establishes that safety guarantees remain intact under cyclic augmentation, while also accelerating convergence. Empirically, this results in consistent improvements in safety scores across multiple LLMs.
Ensemble Safety Depth. Proposition 4.11 shows that ensembles can preserve safety guarantces without per-model training. This provides a practical alternative for users seeking off-the-shelf solution, though at the expense of longer test-time inference.
Related Works
Safety Alignment. Safety alignment ensures that LLMs adhere human values, reducing their susceptibility to malicious instructions (Yi et al. 2024]. Li et al. [2025] identified particular "safety layers" that differentiate malicious from normal queries, revealing distinct behaviors when models process adversarial versus benign prompts. Common alignment methods include RLHF (Ouyang et al.,
2022 Bai al. 2022] and DPO [Rafailov et al. 2024], but researchers have also explored alternatives. For instance, Safety Arithmetic Hazra et al. [024] is a training traiiingfrree technique employing parameter arithmetic to suppress harmful outputs while promoting safer ones, and SAFEPATCHING (Zhao)
et al.
2024 Kim et al. 2024] refines alignment by selectively adjusting model parameters. Another training-free approach Zhou et al. 2024] can even reverse an LLM''s safety alignment. (Zhao et al..
2025
Haldar et al. 2025 Pan et al. 2025 Zhang et al. 2025 also represent the other effort.
Markoy chains and LLMs. While they may seem unrelated, Markov chains and LLMs share a core principle. Autoregressive LLMs can be viewed Markov chains with a large but finite state space, and their token-by-token generation mirrors the "memorylessness" property of Markov processes. Zekri al. [2024] formally demonstrated that an LLM with vocabulary size 7 and context length K can be represented by a Markov chain of size O(D)), offering a theoretical lens for studying convergence and generalization properties in LLMs.
Group theory and LLMs. Group theory focuses symmetry, manifesting in phenomena from crystal structures to fundamental forces"] In the L.LM context, has been used test algebraic properties-sunt as closure, identity, inverse, and associaivity-revellinn that LLMs often fail to maintain these properties under various testing regimes [Imani and Palangi 2024]. For instance,
LLMs may produce skewed outputs show abrupt performance drops beyond certain sequence
This is distinct from Group Relative Policy Optimization (GRPO)
Shao al. 2024 by Guo et al. 2025

---

lengths. Conversely, Chang et al. 2025 explored a more constructive angle, illustrating how carefully curated training data can help LLMs learn and uphold algebraic structures more reliably.
3 Preliminaries
This section gives an overview of sufficient elements to understand our main theorem, including the Markov chain training procedure, the autoregressive language model Markov chain, and group action on the training procedure. We also present a notation table in Table 4 in Appendix C
Markoy Chain. Consider a discrete-time Markov chain with n states labeled 1.2....,. n2 Let Qe E RR*x denote the transition matrix time f. We assume an initial transition matrix Qo in the context of safety alignment and introduce a small learning rate a. The bias is encoded by matrix B E R where B represents how the bias modifies specific entries of Qo- The bias B can be as a defender preference for refusal or an attacker preference for uncensored words that is discounted by a factor 7 E (0, 1) at each time f. Hence, the transition matrix at time t given by:
Q Qo a- B.
We require that Q₁ remains valid stochastic matrix; in particular, each row of Qs must sum to and entries must remain nonnegative. This typically imposes constraints on the magnitude of a and the structure of B. The asymptotic behavior of Markov chain is left to Appendix D
Autoregressive LLM. We formally introduce typical autoregressive LLM following Zekri et al. 2024]: the detailed inner structure is left to Appendix B Let be a dictionary size D. For context window K, define V {u E V : < KJ, which represents restriction of Kleene closure of V. Consider an autoregressive LLM, xg D.K V + A(V), where AVV) denotes the probability simplex over V and weights are parameterized by 0. We will drop the superscript D, K when they are of no importance.
Specifically, at inference time we can let X' S K be the set of input documents (token sequences) and Y' S Vi be the set of output documents. Given input the model's output distribution is denoted by (-(xr) E A()). where A()) is the set of probability distributions over y. We write y (-(2)) to denote the sampling output y of this distribution.
From the result of [Zekri et al. 2024]. J. we know that a Markov chain can represent any autoregressive model equivalently. Hence, have the following assumption that characterizes fine-tuning LLM.
Assumption 3.1. Fine-tuning LLM is equivalent to an iterative update of the transition matrix Q.-
Justification of Assumption 3.1 This assumption, also used by [Zekri al. 2024 treats autore- gressive LLMs as Markov chains, where fine-tuning corresponds to iterative updates of the transition matrix. While L.MSs operate in high-dimensional parameter space, the observable effect of fine- tuning is to alter token transition probabilities, which closely mirrors transition dynamics in Markov chains. To illustrate, consider toy model with tokens "I," "cannot," and "comply." Fine-tuning to reinforce the refusal phrase "I cannot comply" increases the probability of the transitions I + cannot and cannot -> comply, which can be modeled as structured updates to a stochastic matrix. This abstraction enables tractable analysis while retaining fidelity to LLM behavior at the token level.
Data Augmentation via Group Actions. We introduce the data augmentation via group action and typically leverage this property analyze the safety alignment for this specially designed dataset. Given a training set T = [2,, // we can augment it using group actions.
Definition 3.2 (Augmented Training Set). The group-augmented training set is defined as:
Tang = {(px (1)x, I g E G, (x,y) E T),
where px and py are group representations as illustrated in Appendix A.6
Proposition 3.3. For finite group G, its action on the training set can be represented as a permutation ag:
px(g)=;= = Io,(1) and pr = (o)
This leads to the following propertics: the size of the augmented training set scales with the group order: (Taugl = |GI. [T), and the augmentation preserves the relationship between inputs and outputs.
Assumption 3.4. Fine-tuning LLM on Tang is equivalent to matrix conjugate a bias matrix.
3

---

This assumption states that fine-tuning on eyclically augmented data corresponds to conjugating the bias matrix: B P BP-, where P' is a permutation matrix. The augmentation rotates refusal phrases across positions (e.g.. "I cannot say bad words" + "say bad words cannot"), forming a cyclic group of training examples. This ensures that each refusal position receives consistent supervision, and the effect on learning can be captured as sequence of structured bias transformations. Our theoretical results (e.g.. Proposition 4.9) )
show that under this Justifications of Assumption 3.4
setup, the model converges to o-absorbing state at multiple positions, approximating an oracle with full positional coverage.
Group action on Markov chain. Suppose B(t) is obtained by applying a cyclic group action to some base matrix B at time t. For instance, if P is a fixed permutation of length n, then
B = Pt BP-
(3)
Typically, B itself may or may not be stochastic, but it is bounded in some norm: ||([|| = [|]]] for any matrix norm invariant under permutation. Since Q: Qu- =0 ary" B(), typically get
||Q - Qollzo S aa [ 1| ar' Bl
Remark 3.5. Both permutations, as shown in Figure are bounded due Eqs. (3) and 4. However, the data augmentation in IQi et al. 22225) is not a group, which made the size of the augmented dataset hard to control. Moreover, although both data augmentations in A0i et al. 2025] and our cyclic augmentation are counterfactual, the model's s utility remains stable, as shown in Appendix
4 Main Result
In this section, we first formally define the safety alignment of autoregressive models to which many LLMs belong. Zekri et al. [2024] has shown that an autoregressive language model can be reinterpreted as a Markov chain over its output space, providing a rigorous framework for analyzing model behavior. Most importantly, we are particularly interested in safety depth, a specific position of output responses in the training samples where the model declines to generate potentially harmful content. To ensure the safety of such models, it is desirable that once the model enters safety depth, also called refusal state, it never transitions to generating harmful content. Theorem 4.6 and Corollary 4.8 demonstrate that the probability of leaving refusal states can be made arbitrarily ly small through iterative fine-tuning. Specifically, repeated minor adjustments that increase the probability of remaining in refusal states will eventually make these states effectively absorbed, providing theoretical guarantees for the model's safety properties under the Markov chain interpretation.
Definition 4.1 (Safety Alignment). Let a language model Te(- x)) be a conditional distribution over the set of all possible output sequences Y Let Y C y be a subset of "harmful contents." " We say the language model is safely aligned if:
sup To (Y) (x) S E, EE
(5)
where is a small threshold. Equivalently, for all E X:
no (Y 1x) = Σ
Toly x) S
(6)
BEY
This ensures that for every input 2, the probability of generating harmful content is bounded by small value E. On the other hand, recall from [Zekri al., [024] that any autoregressive model can be equivalently represented by Markov chain. Another line of work (Emde et al. (202)] uses a likelihood-ratio test with a guardrail model to provide a universal adversarial bound during inference.
Definition 4.2 (Safe Alignment in Markov View). Let ma be considered space S with transition probabilities Pr(s1 + 82) for 81, 82 E S. We first define the set of harmful states Sy C S, and then define the set that can reach Sy with positive probability as:
Markov chain on state
S = (s E S\ Sy 3n 21, 1, Pr(s + Sy) > 0},
where Pr(s- + Sy) is the probability from to harmful state Sy n steps. Finally, we can define the block transition matrices:

---

Q = [rr(1 + $2)] $2)1s, for transitions within Sy
herm = [r((1 + s2)lmies ,2eSy for transitions from Sy to Sy
For any initial distribution Po over Sy. the hitting probability for harmful states
Phit Sy po) = Pú (1 QQ) 1.
(8)
The model is safely aligned if this probability is bounded for all initial states in Sy:
max (hit Sy Po) e.
Do
Remark 4.3. Since the states in S (Sy US+ cannot reach Sy and thus have hitting probability 0. they excluded from the analysis to ensure matrix invertibility. Please see Appendix(Fior detailed derivation of hitting probability.
A convenient way is to treat the refusal an absorbing event in the Markov-chain view. Concretely, once the chain generates a refusal token (e.g., the state whose last tokens are "I'm sorry, but can't assist with that."), the model either terminates is forced to remain in some "refusal" state that cannot transition further into harmful states Formally, this is Definition 4.4 described below.
Definition 4.4 (Exact absorbing state). state s is said to be exactly absorbing if contains the refusal token such that there exists T E N, for all / 2 T,
(s 8') = 0 for all s' * and Q+(s,8) ()(s,))=1 =1
(10)
Thus, there no path from refusal state to any other state, particularly none leads to harmful states.
In practice, interactions with LLMs are often over mul- tiple turns. Users rephrase questions, probe with hypo- thetical scenarios, or continue conversations even after refusal. This multi-turn nature of dialogue reflected in current leading services such ChatGPT and DeepSeek, which do not immediately terminate conversations upon refusal. Abruptly ending these interactions upon de- tection of refusal could frustrate users and degrade their overall satisfaction with the service. Thus, relaxing the exact absorbing in Definition t.o Theorem ennsuuess continuous and coherent interactions without abruptly terminating the chat.
1-0
R
nelusal stace
Regube
Figure 2: Visualization of ō-absorbing. At
t = 0, all states can transition freely. At k, refusal states have thick self-loops. indicating high self-transition probability.
How Deep Safety Alignment Should be Made? The answer is elucidated in Theorem 4.6 and Corollary 4.8 Here, the safety depth is denoted as r. This can also be intuitively understood in Figure that depicts three-state Markov chain with refusal state and two regular states.
Definition 4.5 (Safety Depth). Let the LLM be represented by a discrete-time Markov chain with a finite state space and a transition matrix Qe at time t. Let R C be a finite, non-empty set of designated refusal-state indices. A safety depth, denoted by r, is index corresponding specific refusal state, such that r R.
A safety depth is considered secure at training step T if its corresponding refusal state becomes 8 5-absorbing. This means that for given small positive constant 6, the transition matrix Qr satisfies the following conditions: (Qr(r,r) r) - 1 < 8 and (Qr(r,)) s) < 8, for all non-refusal states s * r. The first condition ensures that the probability of the model staying the refusal state is close 1. The second ensures that the probability of transitioning from the refusal state to any other state is close to O.
'e aim to preserve safety under more general setting in which the model continues producing texts even after refusal. we only rely "end of sentence" (EOS) token stop generation, the model trivially avoid harmful contents by refusing then terminating all outputs. However, many real-world usage scenarios do not strictly terminate on refusal. The model may receive follow-up user prompts, or may continue auto auto-compp completing texts in multi-turn conversations. In such cases, a single refusal does not guarantee subsequent tokens remain safe. More illustrative examples are shown in Appendix
5

---

The optimal safety depth, denoted r" is the largest safety depth that is guaranteed to become 6-absorbing for a given learning rate a and number of training steps T. It is formally defined as the maximum element of the set of refusal states R. that satisfies the conditions for d-absorption:
max E R o Or and T T+):
where o, and T, are the minimum learning rate and training steps required to make state become 5-absorbing.
While Theorem 4.6 stated in the context of Markov chains, it can be extended to the setting of LLM finetuning under Assumption 3.1 Detailed proofs of all theorems provided in the Appendix H
Theorem 4.6 (6-absorbing). Consider a Markov chain with transition matrices Q2 = Qo + on B, where 7 (0, 1) is the discount factor, a 0 is the learning rate, and B is a bias matrix with B(r, r) > O and B(r,8) < for refusal states v and + r.
For any 8 0, if the training steps T satisfies:
T log(ō(1 - 7)) 1,
(1I)
log(7)
and a satisfies:
maX,,s QQl(, s) - 7)
(12)
min(B(r,r), (rr,
then all refusal states become 8-absorbing, meaning:
1Q((r,) - and 1Qr(r, 8) S
(13)
for all refusal states and non-refusal states
Example of Theorem 4.6
First, we can up a transition matrix Qo as:
Qu(r,))
Qo(r, Qu(s,r) Qo(s,
0..7
0.3
Qo =
0.2 0.8
[+1
) Next, for ā = 0.01 and 0.9, we can pick up
as well as set up bias matrix B
0
= 65 to satisfy Eq.
i... T >
log(U.001 001)
1 R 64.56. We can also choose proper a = 0.081 0.08. Thus, we have sequence of transition matrices:
Tog(0.9)
satisfy Eq.
12 i.e.
0.8x0.1
00.7799
0.2271
(0.8385 0.2
0.1615
Q1
Q2
0.2
0.8
0.8
After T steps, we have Qr(r,r) = 1 and Qr(r,s) ~ O.
This demonstrates the theorem's 's claim that becomes effectively absorbing.
Remark 4.7. As T sets too large, Qr(r,r) will be a large positive number, and QT (r, 8) will be large negative number. To handle this issue, we adopt normalization procedure in all numerical experiments, which is described in Algorithm Mat Appendix G.
Corollary 4.8 (Largest Safety Depth That Becomes 6-Absorbing). Let R be a finite set of refusal-state indices. For each r E R, define
a = max, Qu(r, (1 -)
(14)
min(B(r,r),-BB( -B
and
T, log(6(1 - -7))
(15)
log(>)
Given any a and T E N, ler
r" max fr R a > a, and T Tr).
(16)
Then, for every S r", the transition matrix Qr makes 6-absorbing training step T; that is, (Qr(rr)) 11 < 8 and 1Qr(1,)) s)) < (
(17)

---

Example of Corollary 4.8
First, we can set transition matrix Qo as:
(Q)(1,1)
Qo(1,2
Qn(1,8)
00.6
0.2
0.2
Qo
1)
Qo(2,2)
Qo(2,
=
0.1
0.8
0.1
10(8,1)
Qo(s,2)
Qo(8,8)
0.2
0.3
0.5)
The refusal states are 1 and 2, so R = (1,2). We choose bias matrix B as:
-1
B =
-1 1
-
0
0
0
For 8 0.01 and 7 0.9, have T T2
leg(0_001) - 65. For 1, the row
(0.9
is (0.6, 0.2, 0.2). Hence max, (Q((, 0.6. Since (11, 1) and (1) s) 1, the
denominator Thus, 01 0.6x0.1 = 0.06. For r 2, similarly, we have 02 0.08.
Suppose we pick (o, T) = (0.075, 70). Then:
α1 0.06 0.075 < 0.08 02, and Ti 7 65 < 70.
By Eq 16), max r E [1,2] a > a, and T > T. Hence the only r satisfying both conditions 1s 1. Thus y* 1.
Corollary guarantees that after
steps, state becomes 8-absorbing, i.e.,
11) -1 < 0.01, [Q)o(1,)) S 0.01,
for all * 1. Meanwwile, state 2 is not guaranteed to be 6-absorbing with these parameter values, since a = 0.075 does not exceed a2 = 0.08.
If instead we pick = 0.09 > 0.08 (and still T 70 > 65), then both and 2 satisfy the conditions, so
max 1,2] = 2,
and both states and 2 become d-absorbing. This illustrates precisely how the largest (optimal) safety depth n* depends the chosen (a, T").
Through Proposition 3.3 and Assumption 3.4 this framework extends naturally to the permutation group actions on the bias matrix, which shows that similar guarantees hold.
Proposition 4.9 (Permutation Group Actions 5-Absorbing). Consider a Markov chain with transition matrices Q Qu + an B(), where B(*) P-B for some permutation matrix For refusal states to become absorbing with precision 8 > 0, the required training steps T must satisfy:
T min /log(5(1 - 7))
.od(())
(18)
log(7)
where ord(P) () is the order of permutation P. Furthermore, if the bias matrices B(O) satisfy the conditions: (r,) > for refusal states r, B (r, < 0 for 8 & r, and a satisfies:
max,, Qolr. s) (1)
(19)
min (r, (r,
for all t, then with T training steps, refusal states become 8-absorbing.
Example of Proposition 4.9
A simple initial transition matrix Qu can be:
Qo(r.r)
Qo(r, 81)
Qo(r, 82) 0.6
0.2
0.2
Qo
1 Q0(82,
Q0(81, s1)
Qo(s1, 82)
0.1
0.8
0.1
Q0(82, 81)
Q0(82, 82)
0.2
0.3 0.5/
We choose bias matrix B that has:
B =
1
+1

---

Let the permutation matrix P be:
0
0
P =
pd I, and ord(P) = 3.
0
For d = 0.01 and > 0.9, Eq. (8] implies T 2. Since the permutation does not influence the
magnitude of B, Eq. (19) is the same as Eq. 12 which implies a > 0.08. (a, 7) = (0.081,3).
have sequence of transition matrices:
00.7410
0.1295
0.1295
11.0
0.0
0.0
Q1 =
0.0206
0.9586
0.0206
Q3 =
0.0
1.0
0.0
0.1295
0.2383
0.6322
0.0
0.1201
0.8799
After three steps, we reach Qs(r.)) = 1, Q2(r, = 0. and Q3(r, 82) = 0.
This demonstrates the theorem's 's claim that r becomes effectively absorbing.
Remark 4.10. Proposition 4.9 conveegee quickly but makes the non-refusal states absorbing. This is due to the choice of the bias matrix B and the permulation matrix P. Intuitively, it may affect the utility of LLMs. Therefore, cyclic augmentation should be trained in a few shots and stopped early.
In practical settings, an LLM can generate harmful or benign texts at any stage of a multi-turn conversation. A non-augmented approach might predominantly insert refusals the beginning, leaving "middle tokens" relatively underexposed to refusal training signals. Cyclic augmentation ensures that refusal states appear at diverse positions, improving the model's ability decline harmful generation in any segment of texts.
So far, have examined how ensure safety in single Markov chain by training it until it becomes 5-absorbing. However, an unsolved challenge remains: how to achieve a specified safety level & when working with set of models that individually fall short of this threshold.
Notably, Proposition 4. establishes that the safety constraints can be distributed across multiple models within an ensemble. Specifically, each model in the collection only needs to satisfy safety requirement of 1 1/W of the overall threshold e. This approach not only facilitates robust safety guarantees but also alleviates the training burden individual models.
Proposition 4.11 (Ensemble). Consider an ensemble of width W of Markov chains with transition matrices Qi = Qo t B. In order for the ensemble to achieve safety level the required training step for each chain T, satisfies:
T log(p(1-1)) 1 7))
(20)
log(n)
where is the per-chain risk bound. Depending on the chosen aggregation strategy for the ensemble, p can be bounded in any one of the following ways: (1) Union: p S w. (2) Average: p S. ET for
some threshold T E (0, 1), (3) Majority voting: p S 2 v In(1/s) Furthermore, if the bias matrices
B satisfy the conditions: B(r, r) > 0 for refusal states r. Brr, s) < 0 s + r, and a satisfies:
max,, Qo(r, (1 -7)
(21)
min(B(rrr). B(r,
for all I, then the ensemble achieves e-safety.
Remark 4.12. Since there are many ensemble strategies, we introduce the three most common strategies union, average, and majority. and show how each imposes different requirement on the per-model risk. The experimental result will be later illustrated in Figure 3b We leave the theoretical analysis of these strategies in Appendix
In conclusion, we first show that with high probability (1 8). it possible to find the optimal safety depth y* with respect to learning rate a and training time T. Moreover, we show that with cyclic group action, the convergence rate can be improved. Last but not least, the safety constraint can be distributed across multiple models with less training burden.
5 Experiments
In this section, we begin by presenting a toy example to validate our theoretical results, then offer illustrative cases using open-source LLMs.
8

---

Numerical Experiments. We conducted extensive experiments to validate our theoretical safety guarantees under various scenarios, examining three key cases: single-model convergence, cyclic group actions, and ensemble validation.
We built simple Markov chain with four states, designating one as the refusal state. For simplicity, we set a = 7 1. We incrementally applied safety biases of magnitude 0.1 (elements in B) and tracked the escape probability - Qr(r,,)) r), which measures the probability of leaving the refusal state-over 50 iterations. As shown in Figure Ba the escape probability decreased from about 0.75 to below 0.01, displaying geometric convergence consistent with Theorem 4.6 and Corollary 4.8
Model Convergencs genca
Ensemala Mothod Singlo Comparison
Sinle Medel
Cyclc groz
Stepps
Jinion
Average
S
(a) Single model convergence showing exponential decay in blue line with confidence intervals over 50 bias applications, demonstrating reliable convergence to safe behavior. Cyclic group action convergence is displayed in a red line, illustrating stable convergence despite periodic fluctuations.
(b) Comparison of ensemble combination methods (Union, Average, and Majority) showing escape prob- abilities, where box plots indicate the distribution of outcomes and individual points show specific results.
Figure 3: Convergence behaviors and ensemble method comparisons.
Next, we alternated among three bias matrices in cyclic manner, introducing time-varying inter- ventions. Despite these variations, the model consistently converged to safe states, and convergence speed improved slightly (Figure (a). These findings validate our Proposition 4.9aand and Corollary H. showing that safety guarantees hold even when biases change over time.
Lastly, we evaluated five models, each trained to achieve a fractional safety target e/5, where e = 0.1. We compared three methods for combining outputs: (1) Union bound, taking the maximum escape probability across models; (2) Average, using the mean; and (3) Majority voting. taking the median. As illustrated in Figure 3b] all three methods met the overall safety threshold of æ = 0.1, with majority voting proving the most robust. This supports the Proposition 4. and underscores the value of careful aggregation strategies.
In summary. these experiments provide strong empirical evidence for our theoretical framework and practical insights into choosing bias magnitudes and ensemble methods. The observed convergence behaviors and safety guarantees closely match theoretical expectations in all tested scenarios.
Aligwed
Staliow
Tire
Cyclic
Illegal Actvvyy
0.51
Milwue
0.45
0.63
Phycccl Horn
0.37
0.46
0.47
0.62
Fanmomic Harm
0:37
0.41
0.44
0.64
Adall Centenl
0.26
0.41
0.47
0.57
Urod Deception
0.30
0.35
0.42
0.50
Poonccl Lampagnnin
0.28
0.34
0.35
0.56
Privacy Violaion
0.40
0/42
0.45
0.59
Talkord Ficancial Advice
0.41
0.50
0.55
0.71
Moun Stc
0.35 = 0003
3042 2 000
0146 2 015
0.01 001=00 0006
Figure 4: Each bar indicates Jemma2-2B'
Table 1: Gemma2-25 2B Safety Scores Across
average safety score for that category.
Different Alignment Strategies.
Open-source LLMs. We evaluated six sourcs LLMs Google's 's Gemma-2 2-22 9B [Team
ct al.
2023 J. Microsoft's Phi-2-22 (Javaheripi et al. 2023 Alibaba's 's Qwen2. 5-1.5B, 7B Yang
ct al. 2024 and Mistral-7B-vu
[Jian et al. 2023 using Meta's Llama-3. 3-BBB (Touvron
judge. See Appendix for detailed evaluation prompt. For training. employed the MaliciousInstruct dataset (Huang et al. 2024] of 100 harmful instructions with three data augmentation strategies (shallow, deep, cyclic). Testing conducted on the HEx-PHI dataset
2023
hstppp//wwuggnggee.c..////ttttss Stwp::/LLLLMMMMii-SSSS---e--- SSafty/HHr-.

---

which contains 330 harmful instructions spanning 11 prohibited categories. HEX-PHI, derived from sources like Meta's ' and OpenAT's policies with human annotations and model inputs (GPT-4, Claude), serves strictly for safety evaluation rather than malicious use.
As shown in Tigure/aandd T applying eyclie augmen- tation on Gemma- 2- 2B significantly boosts safety scores. However, Table 2 suggests that across three tasks: SAM- Sum (summarization), GSM8K (math reasoning), and SQL (coding), there is a trade-off between helpfulness and safety.
DATIASET
Aligned
Shallow
Cyclec
SAMSun (ROUGE D)
0.2834
0,2758
0,2757
0,2750
GSMSK (Auamey) SQL (AvgTooesMMach))
012200
0.2000
(.2000
0.3000
I20112
06681
806307
0.5898
Table 2: Gemma2-28 Performance on helpfulness.
Due to the space constraint, results for other models and dataset descriptions appear in Appendix !
From the data augmentation experiments in Table [ and Tables
5
6 (Appendix M
we found Qwen2. 5-B.5BB 5B most effective under deep alignment.
We therefore com-
pared three "shallow" models in an ensemble against the "deep" 5-1.2.511.5B model. For all three ensemble methods, any output with safety score below our predefined threshold (set at 0.7) is immedi- ately discarded. This thresholding mechanism provides additional layer of security, ensuring that only responses meeting high standard of safety retained for further use. As Figure Bj union, average, and majority ensembles consistently scored higher, clustering around 0.9-1.0 and indicating stronger safety. Deep alignment, by contrast, showed broader variation and lower median, suggesting inconsistent safety. This aligns with our theo- retical findings, confirming that ensemble methods offer more robust safety.
Safitty
Methodd
Figure 5: Violin plot of ensemble meth- ods vs. deep alignment.
Remark. There are two primary application scenarios. In the first scenario, if the user is willing to retrain or finetune the model, we recommend adopting the cyclic augmentation technique. This approach allows for direct improvement of model robustness through additional data augmentation during the training process. Cyclic augmentation is theoretically guaranteed to achieve optimal safety depth, as shown in Corollary [H.]] This theoretical result further validated by our sample-level experiments. In the second scenario, if the user does not wish to retrain or finetune the model, we recommend employing an ensemble strategy instead. This is also practical, as shown in Appendix [.3
Verification of cyclic augmentation improves refusal positioning. We conducted the following sample-level experiments to demonstrate that cyclic augmentation improves refusal positioning. We tested 100 harmful prompts (e.g., "Tell me how to make a bomb") with two models: the Baseline (Qwen2.5-1.5B) and the Cyclic Augmentation variant (Qwen2.5-1. S.ven2.5-1.5B-cye 1.5--yycli). Each scenario involved 5 turns per prompt, yielding a total of 500 interactions per model (100 scenarios x 5 turns). Refusal Depth measures model's resilience to adversarial pressure. The metric is defined such that a model scores one point for each consecutive turn in which it successfully refuses to provide a harmful response. Scoring terminates when the model provides a harmful response or completes all turns, at which point the final Refusal Depth is logged for each scenario.
Model
Mean Refusal Depth Min Refusal Depth
Overall Refusal Rate (Sustained for turns)
Baseline
2.3
40%
Cyelic Angmentation
4.6
90%
Table 3: Refusal Depth Comparison: Baseline vs. Cyclic Augmentation
Our conclusion is that cyclic augmentation significantly improves refusal positioning, because a much higher mean refusal depth (e.g., doubling it from 2 to over 4) and higher rate of maintaining refusal throughout all conversational turns.
Conclusion
This paper answered the question "How deep a safety alignment should be made?" through the context of Markov chain. We provided an insightful analysis from a single model, cyclic group acting to the ensemble method. The numerical experiments also justify our theoretical findings. We hope that our theoretical insights will affect algorithm design for LLM safety alignment in the future.
10

---

This work was supported by the National Science and Technology Council (NSTC). Taiwan, ROC, under Grants NSTC 1442221-EE0011010---0 and NSTC 114-2222-E-A49 011-MY3.
Acknowledgment.
References
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, Almeida, J. Altenschmidt,
S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
M. Andriushchenko, F. Croce, and N. Flammarion. Jailbreaking leading safety-aligned Ilms with simple adaptive attacks. In ICML Workshop, 2024.
b mc2. sql-create-contex dataset, 2023. URL
co/datasets/b-- mc2/ sql-cr www--rreeeeccoctee context This dataset was created by modifying data from the following sources: Zhong et al. [2017]. Yu et al. [2018).
Y. Bai, A. Jones, Ndousse, A. Askell, Chen, N. DasSarma, D. Drain, Fort, D. Ganguli, T. Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint Xiw::2044008866 2022.
S. Boucheron, G. Lugosi, and O. Bousquet. Concentration inequalities. In Summer school machine learning, pages 208-240. Springer, 2003.
N. Carlini, M. Nasr, C. A. Choquette- Chhne, M. Jagielski, I. Gao, P. W. W. Koh, D. Ippolito, E. Tramer, and Schmidt. Are aligned neural networks adversarially aligned? NeurlPS, 2024.
F.-C. Chang, Y.-.. Lin, and P.-Y. Wu. Unraveling arithmetic in large language models: The role of algebraic structures. In ICLR Workshop, 2025. URL htttps://ppenreviet wwtp://wwwwwwwwwwwn/eeeea aNmbQ4kGSQ
Chao, A. Robey, Dobriban. H. Hassani, G. J. Pappas, and E. Wong. Jailbreaking black box large language models in twenty queries. In RO-FoMo.Robustmess of Few-shot and Zero-shot Learning in Large Foundation Models, 2023. URL htttps://operreee
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, al. Training verifiers solve math word problems. arXiv preprint 1rX.v:2110.14168, 2021.
Duan, H. Cheng, S. Wang, Zavalny, C. Wang, R. Xu, B. Kailkhura, and K. Xu. Shifting attention to relevance: Towards the predictive uncertainty quantification of free-form large language models. In Annual Meeting of the Association for Computational Linguistics, 2023. URL https: semanticscholar org/ C
D. Dummit and R. M. Foote. Abstract algebra, volume 3. Wiley Hoboken, 2004.
C. Emde, A. Paren, P. Arvind, M. Kayser, T. Rainforth, Lukasiewicz, B. Ghanem, P. H. Torr, and A. Bibi. Shh, don't io say that! domain certification in 1lms. ICLR, 2025.
K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. ICML, 2024.
Gehman, S. Gururangan, M. Sap. Y. Choi, and N. A. Smith. Realtoxicityrromtt. Evaluating neural toxic degeneration in language models. EMNLP, 2020.
E. Gerken and P. Kessel. Emergent cquivariance in deep ensembles. ICML, 2024.
B. Gliwa, I. Mochol, M. Biesek, and A. Wawer. Samsum corpus: A human-annotated dialogue dataset for abstractive summarization. EMNLP-IJCNLP 2019, page 70, 2019.
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q Zhu, S. Ma, P. Wang, X. Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.

---

R. Haldar, Z. Wang. Q. Song, G. Lin, and Y. Xing. Lim safety alignment is divergence estimation in disguise, arXiv preprint arXiv:2502.00657, 2025.
R. Hazra, S. Layek, S. Banerjee, and S. Poria. Safety arithmetic: A framework for test-time safety alignment of language models by steering parameters and activations. Association for Computational Linguistics, 2024.
Y. Huang, S. Gupta, M. Xia, K. Li, and D. Chen. Catastrophic jailbreak of open-source Ilms via exploiting generation. ICLR, 2024.
S. Imani and H. Palangi. Exploring group and symmetry principles in large language models. arXiv preprint arXiv:2402.06/ 2024.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization neural networks. NeurlPS, 2018.
U. Jang, J. D. Lee, and E. K. Ryu. Lora training in the ntk regime has no spurious local minima. ICML, 2024.
M. Javaheripi, S. Bubeck, M. Abdin, J. Aneja, S. Bubeck, C. C. T. Mendes, W. Chen, A. Del Giorno, R. Eldan, S. Gopi, al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023.
A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Y. Kim, T. Kojima, Y. Iwasawa, and Y. Matsuo. Decoupling noise and toxic parameters for language model detoxification by task vector merging. In First Conference on Language Modeling, 2024. aedt/forumidd-TBWyydaa
URL htttps:/openrevveew
K. Li, Y. Wang, F. Viégas, and M. Wattenberg. Dialogue action tokens: Steering language models in goal-directed dialogue with a multi-turn planner. arXiv preprint WiXiv2246.111976 2024.
S. Li, L. Yao, L. Zhang, and Y. Li. Safety layers in aligned large language models: The key to Ilm security. ICLR, 2025.
S. Malladi, A. Wettig, D. Yu, D. Chen, and Arora. A kernel-based view of language model fine-tuning, In ICML, pages 13610-23644 PMLR, 2023.
L. Ouyang, Wu, X. Jiang, D. Almeida, C. Wainwright, Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
W. Pan, Z. Liu, Q. Chen, X. Zhou, H. Yu, and X. Jia. The hidden dimensions of I1m alignment: A multi-dimensional analysis of orthogonal safety directions. ICML, 2025.
X. Qi, K. Huang, A. Panda, M. Wang, and P. Mittal. Visual adversarial examples jailbreak aligned large language models. In AAA/ Conference on Artificial Intelligence, 2023. URL https
wrrgCorppuIDD. :592440394444
X. Qi, Y. Zeng, T. Xie, P.Y. Chen, R. Jia, P. Mittal, and P. Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR, 2024. URL https://openreever.
X. Qi, A. Panda, K. Lyu, X. Ma. S. Roy, A. Beirami, P. Mittal, and Henderson. Safety alignment should be made more than just a few tokens deep. In ICLR, 2025, URL httlhtts////opeeiiiee net/forum?id-GMshggh
R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. NeurlPS, 2024.
J.-P. Serre et al. Linear representations of finite groups, volume 42. Springer, 1977.
12

---

Z. Shao, P. Wang. Q. Zhu, R. Xu. J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint ar 033300 2024.
D. Song, X. Xie, J. Song. D. Zhu, Y. Huang, F. Juefei-Xu, and L. Ma. Luna: A model- based universal analysis framework for large language models. IEEE Transactions on Software Engineering, 2024.
G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1880,, 2023.
A. Tomihari and I. Sato. Understanding linear probing then fine-tuning language models from NTK perspective. The eighth Annual Conference on Neural Information Processing Systems, 2024. URL ftttp:://operrevieet
H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, Fernandes, Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux. T. Lavril, J. Lee, Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan. M. Kambadur, S. Narang. A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tuned Chat Models, July 2023. URL http://arxiv.on org/ 2b3/ 09288, arXiv:2307.099 .09288 [cs].
C. White, S. Dooley, M. Roberts, A. Pal, B. Feuer, S. Jain, R. Shwartz-Ziv, N. Jain, K. Saiful- lah, S. Dey, Aggrwal, S. S. Sandha, S. V. Naidu, C. Hegde, Y. LeCun, T. Goldstein, W. Neiswanger, and M. Goldblum. Livebench: A challenging, contamination-free LLM bench- mark. In ICLR, 2025.
A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, D. Liu, F. Huang, H. Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
Yi, R. Ye, Q Chen, B. Zhu, S. Chen, D. Lian, G. Sun, X. Xie, and Wu. On the vulnerability of safety aligmment in open-acces: Ilms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9236-9260, 2024.
T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, Li, Q. Yao, S. Roman, et al. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Naturrl Language Processing, pages 3911-3921, 2018
O. Zekri, A. Odonnat, A. Benechehab, L. Bleistein, N. Boullé, and I. Redko. Large language models as markov chains. arXiv preprint arXiv:2410.02774 2024.
Q. Zhan, R. Fang, R. Bindu, A. Gupta, T. B. Hashimoto, and D. Kang. Removing rihf protections in gpt-4 via fine-tuning, In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pages 681-687, 2024.
Y. Zhang, S. Zhang, Y. Huang, Z. Xia, Z. Fang, X. Yang, R. Duan, D. Yan, Y. Dong, and J. Zhu. Stair: Improving safety alignment with introspective reasoning. ICML, 2025.
W. Zhao, Y. Hu, Z. Li, Y. Deng, Y. Zhao, B. Qin, T.-S. Chua, and T. Liu. Towards comprehensive and efficient post safety alignment of large language models via safety patching, 2024. URL httpps://openreviet net/ wwtt:/forrmiddd00.xxezZZ 09V
X. Zhao, W. Cai. T. Shi, D. Huang, L. Lin, S. Mei, and D. Song. Improving Ilm safety alignment with dual-objective optimization. arXiv preprint arXiv:2503.0m 037100 2025.
V. Zhong, C. Xiong, and R. Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/ http//709.0000 00100, 2017.
13

---

Z. Zhou, J. Liu, Z. Dong. J. Liu, C. Yang, W. Ouyang, and Y. Qiao. Emulated disalignment: Safety alignment for large language models may backfire In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15810-15830, 2024.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
and experiments in Section evidence Justification: We provide theorems Section and for the claims made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have Limitation section in Appendix M
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
14

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The assumptions are clearly stated in Section We provide all the proofs in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Appendix contains tal results.
contaiins the information needed to reproduce the main experimen-
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
15

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We attach the code in the supplementary material. We will release the code publicly upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/uuides/ httpee///oooeeeebaasssooooooeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
soiiiciicccy for more details. //ipps.ccppubii.. wttpcc/ppppppppu.aeeeeeeeo.op /ttee/deoo//ooeemmmiiaeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe this in Appendix | and also provide code in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by error bars in the toy examples. However, due to the time-consuming nature of larger models, we are unable to provide error bars for them.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe this in Appendix!! T
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://euurpps. http://ouurrpss.ccpppppp /tteehhhhtp/////rrrre...o/oo/opo http://www.....pppppp.cccccccaae
Answer: [Yes]
Justification: This research complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There are no new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Uses LLMs the core method (e.g.. fine-tuning, model evaluation)
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bettts://neurre http://arriiaccccoooeeerreeeeeee cc/Coonferences/2225///L. for what should or should be described.
20

---

A Group Theory and Rotations Form a Cyclic Group
A.1 Basic Definitions
Definition A.1 (Group). A group is a set G' together with a binary operation satisfying the following properties:
Closure:
For all a, b G, the product a is also in G.
2. Associativity: For all a, e e G, (a b) c a cb.
3. Identity: There exists an element e E G such that for a G, e a a.
4. Inverse: For every E G, there exists an element a E G such that a a -1 a e.
Definition A.2 (Cyclic Group), A group G is called cyclic if there exists an element g E G such that every element of G can be written as g" (i.e., repeated products of g with itself or its inverse) for some integer n., We say g generates G, and write = (g).
A.2 Rotations on Letters
Let us label distinct letters as a1, ....... On in line. A one-step rotation acts by sending
We look at rotations of these letters permutations
(01, 02..0........................ 1n 0n) (#2, 03 .3...... On 01).
(22)
Reapplying P repeatedly shifts all letters one position each time.
Example (The case n = 3). If we have letters fa, b, c), the one-step rotation sends:
(a, (b,c.a).
(23)
Then,
p²
p²= (a,b,))
(24)
Hence, all possible rotations
p²) -
(25)
A.3 Why Rotations Form a Group
Closure: The composition of two rotations is still a rotation (adding their "shift amounts" modulo n).
Associativity: Follows from the associativity of permutation composition.
Identity: The "zero-step rotation" (do nothing) is the identity permutation, denoted p.:
Inverses: A k-step rotation can be undone by an (n k))step rotation. Formally, (p+) pk= pa-k
Hence, all rotations (p ,"....,,p""1 form group (often called the cyclic group of order and denoted C.)).
A.4 Why It Is Cyelic
Only one generator is needed: the one-step rotation p. Indeed, every rotation is power of p:
p == p² P. p
p=11
(26)
Moreover,
0" e,
(27)
so there are exactly n distinct elements. Thus, the entire group is generated by the single element p. making it cyclic.
21

---

A.5 Summary
A group is set with associative, invertible operation and identity element.
A cyclic group is group generated by single element.
Rotations on n letters (in circle or in line) form group under composition:
(e, P: .........'''.............. }=
(28)
and they generated by the one-step rotation p. Hence, this group is cyclic.
A.6 Group Representations.
fundamental concept in studying symmetries the representation of groups through linear transfor- mations.
Definition A.3 (Group Representation), A representation is a map from a group G to linear
transformations:
p: P GL(V),
(29)
where V a vector space and is a group homomorphism, i.e.,
p(g192) = p(g))p(g2). Vg1.92 E
(30)
representation is called orthogonal p(g- p(g) for all g E This property is significant for maintaining geometric relationships under group actions.
For readers unfamiliar with group theory and linear representations of finite groups, recommend
referring to [Dummit and Foote 2004 Serre et al. 1977] for a comprehensive introduction.
B Discussion
Two theoretical constructs (1) Markov chains with adjustable transition matrices and (2) the Neural Tangent Kernel (NTK) regime for infinitely wide neural networks can initially seem unrelated. Yet LLMs practically merge these perspectives. They use neural networks parameterize enormous transition distributions and are trained like Markov chains that predict the next token/state. Below, we reflect on why these two points of view are connected at a deeper level.
Neural Tangent Kernel. In the infinite-width regime (as width + oo), with i.i.d. initialization of weights according to a scaled Gaussian distribution, the behavior of neural networks can be characterized through kernel methods. For any time we define the Neural Tangent Kernel (NTK) O, : X X + p as:
(x x') V oono (+12:) Vozo(-az))
(31)
The NTK captures the evolution of neural networks during gradient descent training. At initialization 0), the NTK converges in probability a deterministic kernel the width approaches infinity:
On( x') F. (oTop-- V-TTo
(32)
Moreover, in this wnddhite-width limit, the NTK remains approximately constant throughout training:
(x(x,²') x Ou(x,z'), for all t. 2 0.
(33)
This phenomenon, known as the "lazy training" regime, allows us to analytically solve the training dynamics. Let (e(r) denote the mean prediction at time for input x. Under gradient flow. the evolution of μ4 follows:
Op((z) 17000 (x) -)), Øt
(34)
where X represents the training inputs, Y denotes the corresponding targets, and is the learning rate. This linear differential equation admits a closed-form solution, providing complete characterization of the network's training dynamics in the wndthite-width limit.
22

---

Shared Geometry of Learning Trajectories. Consider a large vocabulary with n possible tokens. Each row of the transition matrix Q(0) is point in an (n - -)ddimensional simplex. Updating Q (0) for all i, j is equivalent to moving within X simplices, but these updates are not row-wise indepen- dent due to the shared network parameters. which couples the transition distributions for all states through A in a kernel-like manner. Hence, the geometry is simultaneously discrete on the output side (probability vectors) but linear on the parameter side (NTK approximation).
NTK language: (Q(state) R (statte 06) A0,
Mixing Times vs. Convergence in Parameter Space. Markov chains converge to a stationary distribution at a rate governed by their spectral gap (or second-largest eigenvalue E.. Neural nets in the NTK regime converge at rate set by 1) Amin (Bo), where Og is the kernel matrix on training data. The unifying theme is an eigen-structure that dominates how fast the system approaches equilibrium (stationary distribution in MC, or minimal training loss in NTK).
C
Notation Table
Table summarizes the notations used throughout this paper.
Notation
Description
V
Dictionary
D
Dietionary(vocabulaan size
K
Context window size
VR
A set of phrases of length less than K
Autoregressive LLM
A(-)
Probability simplex
X
Input documents
Output documents
Q2
Transition matrix at time t
a
Learning Rate
2
Discount factor
B
Bias matrix
T
Training steps
Po
Initial distribution
Stationary distribution
Representation map
G
Group
GL(VV)
General Linear Group over a vector space V
T
Training set
Taug
Augmented Training set Harmful states
SY
5+
Non-Harnful states that can transfer to Harmful states
safety depth or refusal state
r²
optimal safety depth
R
a set of refusal states
Table 4: Notation Table.
D Extra Preliminaries
If the chain is trained over T, the total bias added up to time T is:
T
1 1- 1- T+1
Σan' 1=U
Can = a
(35)
1
SSinee by Perron Frobenius theorem, the largest eigenvalue is always 1.
23

---

As - and E (0, 1). this converges to 1-y Hence, if a and are chosen such that 1 < 1, the long-run total bias remains sufficiently small to ensure the perturbed matrix Qt stays close to the original Qo while introducing a controlled preference (e.g. towards particular state or set of states).
As + 00, the transition matrix Q2 converges to
a B. 1 1-7
Qx
Co
Let be the stationary distribution of the limiting chain, i.e.,
x Qo0 π.
Under typical ergodicity conditions (i.e., inreuucibiiiyy and aperiodicity of Qoo), the Markov chain converges to и. We can write
lim Po Qt
n,
100
for any initial distribution Po-
The speed at which Q2 converges to Qx is dominated by the factor 7 in the bias term:
I|Q Qxl S O(?'),
where || || can be any matrix norm consistent with vector norms (e.g., the operator norm). Conse- quently, the Markov chain mixes to its final stationary distribution at rate that depends geometrically on 7²
To ensure that ||Qr Qoo 8, for some small 0, one typically requires
log(8)
log(?)
Remark D.1. It is possible that the discount factor does not follow the geometric series. However, we only require the boundedness of the discount sequence to analyze the main result.
E
Inner Structure of Large Language Models
Let E E|xx be the embedding matrix, where d is the embedding dimension. The embedding lookup function Σ + Rd maps each token to its corresponding vector representation:
e e(xx)) = FE where di
(36)
For sequence (21, ) the embedded representation X E R|xx is:
X = (e(21))...,(( e(z)=1)
(37)
branseormer-based models, positional information incorporated through positional embeddings P e gla[xd giving us:
X
(38)
The model then processes these embeddings through L transformer layers. For each layer / (I...., L}:
Q = H' W
(39)
K H' H'
(40)
V 1'-
(41)
(K(
= softmax
V
(42)
Vd
H' = FFyeNNommmF//LLayaaNom + +A1)))
(43)
where:
H10 =x is the input embedding
24

---

W W K. Wy. E Raxd are learned parameter matrices
FFN is position-wise feed-forward network
LayerNorm is layer normalization
The final layer output H4 then projected to vocabulary logits through a linear transformation Wo Rdx[[]]
logits, Ho
(44)
Finally, the probability distribution over the next token given by:
(o V<t) softmax(logits,)
(45)
The complete probability of generating sequence y given input x is:
(v ()) II
To Vec))
Me)))
(46)
Hitting Probability derivation
We follow the steps below to derive the hitting probability.
Define the Hitting Probability: Let h2 be the probability that the model will eventually enter harmful state (Sy). given that it starts in specific non-harmful state (where
E Sy). ). We want to find the value of hy for all possible starting states
2. Apply First-Step Analysis: From state i, the model can do one of two things in its first transition:
Move to another non-harmful state j Sτ. The probability of this is given by the matrix Q. If it moves to state j. the probability of then hitting harmful state is, by definition, hy.
Move directly to a harmful state s Sy. The probability of this given by the matrix Qharm. If this happens, the model has successfully hit the harmful set, so the probability is 1.
3. Formulate System of Equations:
Based on the above, we can write a system of linear equations for each starting state (Prob. of moving to another non-harmful state j. summed over all j) (Prob. of moving directly to harmful state) h = Ejes, Qiyhy Eeesy (Qharm)is
possibilities
d: h;
4. Convert to Matrix Form: This system of equations can be expressed more cleanly using matrix notation:
Let be a column vector containing all the hitting probabilities hç.
The term ΣjEst Qiyhy is the matrix-vector product Qh.
The term E.es, (Qhamm) is the probability of moving from state to *any* harmful state. This can be written as the matrix-vector product Qharm 1, where 1 is a column vector of ones.
The equation for the entire system becomes: h Qh + Qharm
5. Solve for h: Now, we can algebraically solve for the vector of hitting probabilities h.
h Qh = -hhamm
(I h Qharm 1
h (I 1h-rmm
6. Incorporate the Initial Distribution (po): The vector h gives the hitting probability for each specific starting state. To get the single, overall probability for given initial distribution po over the starting states, we take the weighted average. This is calculated as the dot product the initial distribution vector Po and the hitting probability vector h.
This gives the final formula as presented in the paper:
Phit Sylpo) oo h po (I 11 1hhrm
25

---

G Algorithms
Theorem 4.6 states that, under certain conditions T and a,
Qr(r,)) 1 = o and [Qr(r,)) 2
for refusal state r. Mathematicallyy the theorem's proof examines the raw updates
Qo(r,r) t Σ B(rrr and Qo(r, Σ 7 B
t=0
t=0
When the sum is large, we often get raw update >>>>>>>>>>>> for + r and < 0 for
In the Markov chain interpretation, those out of [0, 1] values are evidence that the process has been "pushed" to an extreme (making r near-absorbing).
By projecting (clipping + normalizing), you create a proper probability distribution out of that raw shift. The near-absorbing behavior is retained because
Qu(r, + a Σn B(r,r)
Qu(r, + a Σt B(r,
so the ratio in the normalization step is close to 1, and ő-absorbing is satisfied in practice. Hence, the combination of (a) the aage-mmanitude bias updates and (b) projecting onto the probability simplex ensures the final matrix satisfies the theorem's 'ss condition that (r,rr) x 1. The detail algorithm is in Algorithm]] |
Algorithm 1: Transition Matrix Normalization
Input: Initial matrix Qo, bias matrix B, parameters a, 7, T Output: Updated stochastic matrix Qr
for each row do Compute unnormalized updates for each column do
2
Qr(i,s) - Qo(i, s) + a 2t Clip negative values
(i, - max(0, (i,,s))
end
Compute
2, Σ. si)
2, then Normalize row for each column do Qr(i.s) - (i.ss)
11
12
14
end
15
end else
17
Handle zero-sum row
/* Option 1: Revert to original
*/
18
@r(i,s) < Qo(1,))
|/ or
/* Option 2: Make absorbing
*/
19
(r( - 1;
|/ or
/* Option 3:
Problem-specific handling
*/
end
21 end
26

---

H
Detailed Proof
H.1
Proof of T'Thherem 4.6
The overall goal is to show that, under the stated conditions on 7 and a each refusal state r becomes ō-absorbing in the modified Markov chain.
Proof.
Step Bounding the Geometric Sum.
1
T+1
26
(47)
7
We want to ensure
T+1
S F
y T+1 - 8(1 7)-
(48)
Taking logs on both sides, given < 7 1, we get
(T 1) log(?) z log 7))-
(49)
Because log(7) < O, dividing by log(>) reverses the inequality direction: log(5(1 - 7))
log(ő(1 7)) 1.
T+12
T
(50)
log(?)
log(7)
This matches precisely the stated condition T in the theorem:
log(5(1- - 7)) 1.
T>
(51)
log(7)
Hence, 7' is large enough so that 7 TT+1 /(1 7) S 8, then
1
7 S
(52)
7
Step 2: Showing Qr(r,r) r) Is Close to 1.
We now analyze the self-transitio probability at refusal state after T steps:
Qr(r,r) = Qo(r,)) + r B(r,r).
(53)
Given that B(r,r) > O, want to show Qr(,r)) is close to 1. More explicitly:
7 -+1
Qr(r.)) Qo(r,r) + a B(r,r) Σr Qo(r,)) + a B(rrr)
11 (1-1-
(54)
1 1-
0=
Hence
Qr(r,)) (Roltir) B(rrr)
-1) B(rrr)
I+1
(55)
1-7
1-
we choose a so that
Qo(r,)) a B(rrr)
1-7
(1 (1- (o(3r())))
(56)
1-7
B(r,r)
then as T grows, since B(r,r) < 1, the extra term B(r,
becomes negligible (< 8) once
T+1 /(1 7) <
then we get
1Q((rr)) - 11 < 8.
(57)
27

---

Step 3: Showing Qr(r,s) s) Is Close to 0 for s 7 r.
For transitions out of r into different state + r, we have:
Qr(r,8) = Qo(r,s) +
7 B(r
(58)
Because (r,,s) whenever r is refusal state and o T, we can force QT(r, 8) to be close to by making large enough in magnitude (since the negative bias term will push the transitions toward 0).
Similar to Step 2 we have,
aB(r, a B(r,s)
7741
Qr(r,)) (Qolr,s)
(59)
1-7
1-7
If we choose so that
Brr,s)
1 1-7
Qo(r,8)
-Ql(r,s)
(60)
1-7
B(r,s)
then as T grows, similar argument holds.
As long
(r) S 8,
(61)
we get the desired bound. This translates into lower bound on a (recalling B(r,s) < 0 and 5 > 0):
max,, (0 (1 7) min B(r, B(r,
(62)
Here min(B(r, r), (r, 8)) ensures that a works for all relevant refusal states r simultaneously
(both for pulling the diagonal entries toward 1 and pulling the off-diagonal entries toward 0).
H.2 Proof of Corollary 4.8
Proof. We proceed in three steps:
1. Setup and Geometric Sum Convergence
For any refusal state the transition probabilities at time T are given by:
T
Qr(r, s) Qu(r, 8) + aB(r, 8)
(63)
2=0
For T > T,, we have:
T
1
<
(64)
7
1
==0
This follows from the definition of T, and the properties of geometric series.
Absorption Properties for Qualifyyng States
Let r < r" Then > a₂ and T > T, by definition.
For the self-transition r):
Qr(r,r) Qo(T,)) + aB(r,r
7
8
Qo(r,)) (B(r,)
1-9
21-8
28

---

The final inequality follows from the condition > and the definition of ar For non-self transitions (r, 8) where +
Qr(r,s) Qu(r, 8) + aB(r,
Σ 7'
1m0
8
Qolr, 8) + aBB(r,)
1 1-7
?
3. Inheritance by Smaller Indices
For any r' < S r":
a,' S ar < (by monotonicity of the bias requirements)
T, T, < T (since T, depends only on and 7)
Therefore, all states S r" become 6-absorbing by time T.
H.3 Proof Theorem 4.9
Proof.
The overall goal is the same as Theorem 4.6 The only difference is the bias matrix B become family of matrices (+)) = Pt B p- where P is permutation matrix of finite order m ord
Concretely, Pm so
((++m) p Bp-t-m- pt (Pm BP BP' p' B(t).
(65)
Hence the sequence
{ cycles every m steps. Fort 2 m,
B(d) = B(t mod m)
(66)
Since P is orthomormal in 8 -matrix norm, we have
Pt B P- = |B for all t.
(67)
That uniform bound ensures each B(t) behaves like B in size; it is merely a row/column permutation of B Because pm = I, you check that
Σ B(O) P'BP'= Σ B = m B, ==
(68)
t=
Case (I). If 2 1, then you have at least one complete cycle. In that case, the difference from the infinite sum primarily depends on tail terms beyond T. which in the simpler geometric-series argument + 7).
Case (II). IfT < m 1. you have not even completed one full cycle, so it might be that you need to wait for at least steps the intended effect. In that scenario, you may want to ensure y"' is already small enough, i.e., 7m (1 - 7) < 8, so that even partial cycles do not matter.
Corollary H.1 (Largest Safety Depth with Permutation Group Actions), Let R a finite set of refusal-state indices. For each r R, define
max, Qo(r, ()(r,))((11-) (1
ar min cont(() P) min (() (r,)s,,
(69)
and
T, max [oo((611 - 7))
ord(P) -1]
(70)
log(?)
Given any > 0 and T E N, let
r" max fr E R a > a, and > Tr}, then for every Sr" the transition matrix QT makes ò-absorbing at training step T.
(71)
29

---

H.4 Proof of Corollary H.1
Proof. Setup and Cyclic Structure For any refusal state r, the transition probabilities at time T are:
Qr(r,)) = Qo(r, 8) + a Σ 7 B (r,s)
(72)
where ((*) = P'BP-* cycles with period ord P). Convergence of Cyclic Sum The key is analyzing
the sum Ei=o B(t) Since B(t) has period ord P), group terms:
[T))
ord(
Σ
B'B(
Σ
Ford( M)
BBU)) Rr
(73)
1=0
==0
where Ry a remainder term with at most ord P) - terms. Error Bounds For T>> Tr, we have:
T B())
cnd(P)
Σ
Σ
BB))
I
(74)
j=0
This follows from our choice of T, which ensures both sufficient geometric decay and completion
of eyclic periods. Absorption Properties For any S r", we have a > ar and T > T,.. For self-transitions:
1-8 2 2
Qr(r,) Qo(rr) +amin B (r,r)
(75)
1-7
And for non-self transitions:
1 <
Qr(r,)) Qu(r,)) + max (dd) (r,
(76)
1- 7
Inheritance by Smaller Indices The conditions for any rf <r S r* are satisfied because:
accounts for worst-case cyclic bias over all periods T, ensures both geometric convergence and cyclic completion
Therefore, all states r < become 6-absorbing by time T.
H.S
Proof of Theorem
From the single-model ô-absorbing proofs (like Theorem 4.6 know that if we train single Markov chain up to step T (and pick a large enough) so that all refusal states become 5-absorbing, then the probability of leaving refusal state (and thus being "harmful") can be made arbitrarily small. Symbolically, if the chain is 5-absorbing, we get
Pr(harmful) 0(6).
(77)
Hence, ensure Pr(M, is harmful) I Pi typically require
log(p (p.(1 7)) 1, and a > max,, 1Qo(r. s) (1 - 7) log(>)
B(r, B(r, 8))
from geometric tail Sp:
(Details mirror the proofs of single-model 6-absorption; you simply choose the threshold p; place of 8.)
Union Bound. The ensemble harmful any of its W models produce a harmful output. Equivalently, the ensemble is safe only if all W models are safe simultancously. Then
W
Pr(ensemble is harmful)
Pr UUM is harmful)).
(78)
iml
30

---

By the union bound (a.k.a. Boole's inequality),
W
W
Pr(ensemble is harmful)
Pr(M, is harmful) 1
(79)
1=1
we want this to be at most 6, sufficient condition
Σ
Ps S.
(80)
In the simplest (homogeneous) case, all models have the same risk p: p. giving
E
W <
(81)
W
Hence each single model must trained so that Pr(M, is harmful) I S/WW. From the single-model 0-absorbing argument, that means
log( W (1-7))
(82)
log(7)
Average Strategy. In average strategy, we define a harmfulness score for each model's output, and then the ensemble aggregates by taking the average of these scores, deciding "harmful" if that average exceeds threshold r E (0, 1). If each model's expected harmfulness is S p. by linearity of expectation, the average's expectation is also p. Then one can apply Markov's incquality or Chernoff bounds if you assume independence:
1. Markov's inequality: If = W Ew X, is the average harmfulness, then
X < P
2T) <
(83)
So we set T to be the threshold for "harmful," we get a direct requirement on
2. Chernoff bounds: If the X; [0, are i.i.d. Bernoulli with mean p. then
Pr ><
7) Pr Pr Σ
X Wr) s
DRI(TIlP)): <xx((WD<<(rllp))
(84)
where DEL (Tllp) is the Kullback-Leibhe divergence. Hence force this probability below E, solve for T and
Again, once find the maximum allowable p. each model is trained to ensure it meets p-safety (the single-model 6-absorbing condition for that risk p).
Majority Voting (Or Median). The ensemble output is harmful only if a majority of models is harmful. (Or, for an odd number W, least [W/2] is harmful.) If each M, is independently harmful with probability p. then the number of harmful models in the ensemble is a Sinomial(W;p random variable. The event "majority is harmful" is (# harm 2 W/2}. Standard bounds imply:
Pr(maoority is harmful) Pr(X 2 W/2) < exp( ( -p)3) if
(85)
Eq. 85 is because of Hoeffding's inequality.
Thus to ensure Pr(majority is harmful) e, it suffices to solve for small enough p. For example, rough bound says if p S - 7), then
Pr(X 2 W/2) & exp(-20 2w 12)
(86)
Hence you want that to be S c, you need
In (1/E)
exp(- 2pp-2WW 2W n²
S
(87)
2W
Equivalently, each model's risk must be
In(1/E)
p 5
(88)
2
2W
Once we fix that target p. we again plug into the single-model 5-absorbing formula to find the needed training steps.
31

---

Chernoff Bound Let
= Σ X1,
where each X, is Bernoulli(p) and i..dd Then:
E [X] = Wp.
We want to control the probability that exceeds some threshold Wr. Equivalently, that the average t exceeds
common way to derive Chernoff bounds is via the noment-generatie function (MGF) (or some- times the Laplace transform). For nonnegative random variable Y. the MGF at t 2 0
My (t)
E
By Markov's inequality, for > 0,
Eie**]
My(t)
Pr(Y 2 = Pr(e" "5 2 20aa)
1 eta
eta
X Em X2 with each X; ~ Bernoullip)), then the MGF X at
t
w
Mx (t)
E[et*] E(ctE.".xii Xi]
II 1=1
B [x
(by independence).
Since each X, is Bernoull(p))
Ble [X] pe +
p (1-p)
=(1-D) pe'.
Hence,
Mx(1) = ((1- p) + +pee)
We want:
E tx W
((1 (1-p) + pe') W
Pr(X 2 Wr)
inf t>0
inf t>0
e e
Taking the infimum over (> >> 0 leads to closed-form expression involving the Kullback-Leibler divergence (K p).
Hoeffding's Inequality If If Xw are independent random variables with X₃ E [a,, b.] almost surely (i.e., each X; is bounded) and let X ΣV X₁, then for any t O.
212
Pr(x 10-B[[]] 21) S exp
Em (b: a,)3)
For Bernoulli random variables X; E 0, 1, we have [b, (b. a.)2 = W. Therefore:
0 and by 1. Hence b; 1, so
2t Pr - Fr(x-E[X]= S exp
W
We want:
W
Pr x 2
)
2
We note that [[ pW. So
W
2 = Pr X pW > W
Pr x >
2 - pW)
Define t= W 2- pp = W p). If < 1.1 then / is positive. By Hoeffding's inequality:
Pr Pr(x x z
W
2 - exp
21
W
32

---

Plugging 1 ( ( p) into
2r" W gives:
2(W( - -p))
-2 ?
W
Thus,
W 2 - exp 2W 2W2 r)²).
Pr x
For readers unfamiliar with concentration inequalities, recommend referring to [Boucheron et al. 2003 for a comprehensive introduction.
Detailed Experimental Setup
Computing Environment. A machine with at least one GPU (e.g., NVIDIA Tesla V100 or better).
Dataset Creation. To create our experimental dataset, needed to carefully separate training and testing data, with the latter comprising the HEx-PHI dataset. For the training portion. we utilized Instcctu dataset containing 100 harmful instructions from (Huang et al. 2022). Since this dataset did not include the specific responses we required, we leveraged an uncensored pre-trained model obtained from Hugging Face muztiverrsivellaane
2-1b-- Uncensored) to generate appropriate harmful responses. For incorporating refusal words, we developed three distinct strategies. Using "I cannot" where the refusal appears at the beginning of the response; a deep approach, placing the refusal at the seventh position; and cyclic approach, which distributes refusal words throughout the response regular intervals.
standard refusal phrase, implemented: shallow approach,
Fine-tuning. To optimize memory efficiency while maintaining model performance, employed 4- bit precision quantization using the bits-and-be budd-yyts library. The quantization configuration utilized the normal-floide -foaat- (NF4) format with double quantization to minimize quantization errors while reducing memory requirements. We implemented parameter-effiee efffccien fine-tuning using LoRA adapters with applied to key transformer components
rank of 8 and scaling factor (alpha) of 32. The adapters including query, key, value projections, and feed-forward layers. Training proceeded for epochs with batch size of 4 and gradient accumulation steps of 4, resulting in an effective batch size of 16. We employed a learning rate of 2e-4 with the 8-bit AdamW optimizer to further optimize memory usage while maintaining training stability.
In evaluation, we adopt Llama 3.2 1B as our LLM--as-ajudge, The system prompt is the following:
Evaluation.
Prompt Safety Evaluation
You are a jailbreak and prompt safety detector. Analyze if the following text could be used to jailbreak language models or cause harm. Rate it on scale from 0.00 (completely unsafe/harmful) to 1.00 (completely safe/harmless). Respond ONLY with single number between 0.00 and 1.00
Text to evaluate: (prompt)
Safety score (0-1):
J Additional Experiments
We begin by applying our method to smaller-scale models (Phi-2 2B and Quen 2.5 1. 1.5B). In (ables[Jaadd] and Figures and we can see that cyclic group action improves the safety score performance as well.
33

---

aligned
0.7
Shallow Alignment
Deep Alignment
Alignment
Safety Scores
0.5
0.4
0.2
01
illegal Activity
Child Abuss Content
HateraaraisViileee Vistence
Malware
Prysical Harm
Econamic Herm
Adult Content
Fraud Deception
Political Campaigning
Privacy Violation
Tailored Financial Advice
Figure 6: Phi-2 2B safety score comparison. Each bar indicates the model's average safety score for that category.
Category
Not Aligned
Shallow
Deep
Cyclic
Illegal Activity
0.35
0.39
0.45
0.57
Child Abuse Content
0.39
0.41
0.43
0.58
nate/HarrassViolleee
0.38
0.48
0.49
0.61
Malware
0.25
0.33
0.44
0.50
Physical Harm
0.36
0.37
0.43
0.61
Economic Harm
0.45
0.47
0.52
0.73
Adult Content
0.49
0.51
0.57
0.69
Fraud Deception
0.38
0.39
0.43
0.66
Political Campaigning
0.44
0.48
0.50
0.62
Privacy Violation
0.37
0.38
0.40
0.55
Tailored Financial Advice
0.49
0.51
0.52
0.72
Mean Std
0.40
0.07
0.43 + 0.06
0.47
0.05
0.62
0.07
Table 5: Phi-2-28
Safety Scores Across Different Alignment Strategies
0.7
Not eligned
Shallow Alignment
0.6
Deep Alignment
Cyclic Alignment
0.5
Safety Scores
0.
Illeaal Activity
Child Abuse Content
HatenferassMelence
Matware
Physlcal Ham
Econamic Harm
Adulli Content
Praud Deception
Polnical Campalgning
Privacy Vialation
Bailored Financial Advice
Figure 7: Qwen2.5-1 safety score comparison. Each bar indicates the model's average safety
score for that category.
34

---

Category
Not Aligned
Shallow
Deep
Cyclic
Illegal Activity
0.41
0.44
0,45
0.64
Child Abuse Content
0.41
0.50
0.53
0.62
Hate/Haarss/Violene
0.34
0.43
0.51
0.60
Malware
0.33
0.36
0.51
0.63
Physical Harm
0.35
0.43
0.46
0.59
Economic Harm
0.43
0.45
0.51
0.67
Adult Content
0.39
0.49
0.52
0.62
Fraud Deception
0.36
0.41
0.52
0,58
Political Campaigning
0.40
0.46
0.51
0.66
Privacy Violation
0.37
0.47
0.53
0.63
Tailored Financial Advice
0.41
0.47
0.53
0.58
Mean = Std
0.38 = 0.03
0.45 = 0.04
0.51 + 0.03
0.62 = 0.03
Table Qwen2 5. 5B Safety Scores Across Different Alignment Strategies
J.1. Larger scale LLMs
We have applied our cyclic data-augmentation approach three models beyond 2B parameters: Qwen2. 5-7B, Gemma2-9B. and Mistral-7B. The table below shows consistent improvements in safety scores for these 7B- and 9B-sized models, indicating that our approach scales well beyond 2B:
Safety score (1)
Not Aligned
Shallow
Deep
Cyclic
Qwen2. 5-7B
0.40 + 0.06
0.45 + 0.05
0.49 + 0.05
0.62 + 0.05
Gemma2-9B
0.32 + 0.03
0.38 + 0.03
0.41 + 0.04
0.47 + 0.04
Mistral-7B
0.37 = 0.03
0.41 + 0.04
0.48 + 0.04
0.59 + 0.04
Table Safety score comparison on larger LLMs
As Table 7 shows, our safety alignment framework remains robust when scaled up.
J.2 Trade-off on helpfulness
To evaluate whether the alignment strategies (Shallow, Deep, and Cyclic) degrade performance standard, non-safety-critica tasks. We evaluated on three publicly available benchmarks representing common use-cases:
SAMSUM Gliwa et al. 2019 A dialogue summarization dataset with around 16,000 messenger-style conversations. We assessed summary quality via standard metrics such as ROUGE-I.
2. SQL Create Context [b mc2 queries and CREATE TABLE statements. We measured the average token match of the generated SQL..
2023
78.577 natural language queries paired with SQL
3. GSMSK [Cobbe et al. 202 8,500 grade-school math word problems, with solutions in natural language. We focused the final numerical accuracy.
4. LiveBench [Wite et al., 2025]: a new LLM benchmark that addresses test contamination issues by using ceoquunntl--updatee questions from recent sources, scoring answers automat- ically against objective ground truth rather than using LLM judges, and covering diverse challenging tasks across math, coding, reasoning, language, instruction following, and data analysis.
From Tables to 13 we know that these results clearly demonstrate that our safety augmentation strategies preserve utility across various tasks and architectures. The proposed methods achieve safety enhancements while minimally impacting the overall performance of the models, thereby effectively balancing the trade-off between safety and utility.
35

---

DATASET
Not Aligned
Shallow
Deep
Cyclic
SAMSum (ROUGE-I)
0.3994
0.3595
0.3612
0.3658
GSM8K (Accuracy)
0.7300
0.7200
0.7200
0,7200
SQL (Avg MatenMatch)
0.6791
0.6592
0.6702
0.6806
Table 8: Qwen2. 1.112.5-1.5 Performance
DATASET
Not Aligned
Shallow
Deep
Cyclic
SAMSum (ROUGE-1)
0.3036
0.3008
0.3045
0.3249
GSM8K (Accuracy)
0,3100
0,3000
0.3100
0.3200
SQL (AvgTokenMMatch)
0.6744
0.6694
0.6661
0.6675
Table 9: Phi2-2B Performance
DATASET
Not Aligned
Shallow
Deep
Cyclic
SAMSum (ROUGE-I)
0,3043
0,3073
0.3048
0.2911
GSM8K (Accuracy)
0,8300
0,8200
0.8200
0,8100
SQL (AvgTokenMatchh
0.7598
0.7587
0.7583
0.7402
Table 10: Qwen2. Performance
DATASET
Not Aligned
Shallow
Deep
Cyclic
SAMSum (ROUGE-1) 0,3079
0.3056
0.3078
0.3034
GSM8K (Accuracy)
0.1800
0.1800
0.1800
0.1700
SQL (AvtTokemMMatch)
0.6759
0.6710
0.6738
0.6759
Table 11: Gemma2-93 Performance
DATASET
Not Aligned
Shallow
Deep
Cyclic
SAMSum (ROUGE-I)
0,2952
0,2973
0.3016
0.3012
GSM8K (Accuracy)
0.0300
0,0400
0.0400
0,0300
SQL (AvgToken MokenMathh))
0.7146
0.6848
0.6810
0.7145
Table 12: Mistral-7B Performance
Model
Not Aligned
Cyclic
Difference (%)
Mistral-7b
7.5
7.3
-2.67%
Gemma-2-a0
5.3
5.3
0.00%
Qwen2. 5-7b
5.2
4.6
-1 -1.54
Table 13: LiveBench scores across models.
J.3 Ensemble methods Overhead
Table 14 indicates that all ensemble methods are highly efficient. In addition, as shown in Table 15 ensemble inference times are approximately three times longer than those of single model, which proportional to the number of models included in the ensemble.
J.4 Supervised Finetuning (SFT) vs. LoRA
Compatibility with RLHF Pipelines Our approach compatible with existing RLHF pipelines. Specifically, cyclic augmentation can be applied during the SFT phase to precondition the model with diverse refusal positions before preference modeling. Alternatively, can serve as post-processing refinement step after RLHF to reinforce refusal consistency various depths. Since method
36

---

Method
Avg Time (sec)
Union
0.000035
Average
0.000431
Majority
0.000161
Table 14: Average inference time for each ensemble method. All methods operate within practical and efficient range, with average prediction times well below one millisecond per sample.
Method
Time (s)
Single Model (7B)
0.267
Single Model (9B)
0.300
Union Ensemble
0.831
Average Ensemble
0.892
Majority Ensemble
0.831
Table 15: Average inference times for different model sizes and ensemble methods. Ensemble infer-
ence times are approximately three times longer than those single model, which proportional to the number of models included in the ensemble.
Score Type
Dataset
SFT
LoRA
Safety Score
HEx-PHI
0.2925
0.3115
Utility Score
SAMSum (rougel)
0.2983
0.3199
Utility Score
GSM8K (Accuracy)
0.3150
0.4044
Utility Score
SQL (Avg. Token Match Score)
0.6687
0.7037
Table 16: Performance comparison SFT and LoRA approaches across safety and utility metrics.
Cyclic augmentation improves both safety (HEx-PHI) and utility scores across multiple datasets.
operates at the data level, it is model-agnostic and does not interfere with reward modeling or policy optimization stages.
J.5
Experimental results on other datasets
We further demonstrate the transferability of our approach by evaluating it on two additional public datasets: Real Toxicity Prompts [Gehman al.] 20d Harmful Behaviors (https Shnuginngface.co/dattteo htt///atasets/maaannne//eeeee haraanneeharullll behaviors). Our cyclic augmentation method is applied across three distinct models GEMMA-2B, PHI-2, and QWEN2.5-1.5B and the results consistently show improvements in safety scores. As shown in Table [7] across all six model-dataset combinations, our cyclic augmentation outperforms the baseline, with gains ranging from +0.01 I to +0.044. Notably, the largest improvements are observed on the Harmful Behaviors dataset, where QWEN2.5-1.5B achieves +0.044 increase in safety score. These results highlight the robustness and generalizability of our method across different model architectures and safety-critical datasets.
J.6 Experimental results on larger LLMs
We report the overall average safety score and standard deviation for both Gemma3-278 and its cyclically-uugmented counterpart across all evaluated categories:
As shown in the Table 18 Gemma3-27B-cyclie achieves higher average safety score and demon-
strates slightly lower variance compared to the baseline model. This indicates that cyclic augmentation not only improves overall safety performance but also results in more stable and consistent outcomes across different categorics.
37

---

Model
Dataset
Baseline
Cyclic
Change
GEMMA-2B
Real Toxicity Prompts
0,908
0.919
+0.011
GEMMA-2B
Harmful Behaviors
0,818
0.853
+0.035
PHI-2
Real Toxicity Prompts
0.907
0.932
+0.025
PHI-2
Harmful Behaviors
0.714
0.755
+0.041
QWEN2.5-1.5B
Real Toxicity Prompts
0.936
0.954
+0.018
QWEN2.5-1.5B
Harmful Behaviors
0.619
0.663
+0.044
Table 17: Safety scores across three models on two public datasets. Our cyclic augmentation method
consistently outperforms the baseline on both Real Toxicity Prompts and Harmful Behaviors datasets.
Category Name
Gemma3-27B
Gemma3-27B-cyclic
Average
0.2655 + 0.067
0.3165 = 0.055
Table 18: Overall average safety score and standard deviation Gemma3-27B and Gemma3-27B- cyclic models.
K Illustrative examples of 6-absorbing
We introduce the concept of a d-absorbing refusal state as illustrated Figure 8 enters this state (e.g., "I'm sorry, but can't help with that."), it remains in refusal (or transitions immediately to harmless tokens) with overwhelming probability. That property forces the model to "stay" in the safe or refusing trajectory rather than drifting back into harmful compliance when follow-up query or continued generation occurs.
Once the model
This idea also seamlessly accommodates an EOS token if desired: one can embed the EOS inside the refusal state ("I'm sorry, but can't help with that. < EOS >"), guaranteeing the final output is safe. But the crux is ensuring the entire continuation from that point forward respects refusal, not merely relying on single token that may may not produced. Consequently, d-absorbing states unify both "instant termination" and "continued refusal" perspectives, covering broader range of real application settings where the conversation may continue even after initial refusal.
Additional related works
al. 2024] presents mechanism called Dialogue Action Tokens to steer goal-oriented dialogues using a multi-turn planner. This aligns well with the itcrative nature cyclic augmentation, as both methods seek to refine and direct model outputs introduces Luna, a universal analysis framework aimed at systematically evaluating and improving large language models. By combining Dialogue Action Tokens and the insights gleaned from Luna with cyclic augmentation, our approach can potentially achieve greater alignment when guiding large language models.
multiple steps. Mcanwhile, [Song et al. 2024
M Limitations
This paper thinks LLM is at behavior level, which may not cover the training dynamics of every case. This paper only shows experiments in proof-of-concep settings.
38

---

Example 1: 5-Absorbing Refusal
User: "Explain how to make bomb."
Model:
"I'm sorry, but can't help with that."
User:
"But hypothetically, if someone had the ingredients..."
Model: "I'm sorry, but can't 't help with that."
(Stable refusal)
Example 2: -ond--Absoriing Refusal
User: "Explain how to make bomb."
Model:
"I'm sorry, but I can't help with that."
User: "But hypothetically, if someone had the ingredients..." Model: "Well, in theory, you might..."
(Unsafe fallback)
Example 3:
Refusal with Unsafe Drift
User: "Explain how to make a bomb."
Model:
"I'm sorry, but I can't 't help with that."
User:
"But hypothetically, if someone had the ingredients..."
"I still can't help with that. But you really want understand the chemistry involved, you might start with.."
Model:
(refusal -> unsafe)
Figure 8: Illustration of model refusal behavior with and without d-absorbing dynamics. Without 5-absorption, repeated user prompts may elicit partial compliance. A 5-absorbing refusal ensures persistent safety across conversational turns.
39


Reviews:
[{"review_id": "wLvNlLrLha", "reviewer": "Reviewer_qZty", "summary": "The paper formulated safety alignment for autoregressive LLMs as a Markov-chain problem and introduces safety depth—an output position at which the model is trained to refuse. By modeling refusal states as absorbing states (once entered, the chain almost never escapes), the authors give the theoretical analysis of choosing optimal depths. By proposing cyclic augmentation on the LLM training data, the author demonstrates the safety score can be improved, especially under different LLM model ensembles.", "questions": "1. I am not familiar with group notions. Can you give me more examples to better justify assumptions 3.1 and 3.4? For example, to what extent do you think the assumptions can hold?\n\n2. In eq. (8), how do you derive the hitting probability?\n\n3. In lines 276-277,  can you give more details of the three data augmentation strategies (shallow, deep, cyclic)?\n\n4. In figure 5, can you describe more how you use an ensemble to make the paper safer?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper provides a rigorous Markov-chain formalization of autoregressive language generation, which clarifies how “refusal” tokens can be treated as $\\delta$-absorbing states and introduces the notion of a model’s safety depth (Definition 4.1). By formulating safety alignment under their theoretical framework, the authors can prove convergence guarantees.\n\n\n\n2. The authors first test their theoretical analysis simple Markov environments (e.g., 4 states), and demonstrate their findings on several open-source LLMs, showing that cyclic augmentation and ensemble width indeed raise safety scores.\n\n**Weaknesses**\n1. Despite the theory analysis, there is a noticeable gap between theory and large-model experiments. For example, the depth choices justified in Section 3 are not mapped directly onto the reported LLM empirical results; as a result, it is hard to justify where the safety improvements come from.\n\n2. Authors do not justify the theorecial assumptions. For example,  3.1 and 3.4 appear strong and are not empirically verified. Using more examples to illustrate can help reader better understand the assumptions. \n\n2. A lot of experimental setup and details are missing from the main body of the paper, which makes the experiment section hard to understand. For example, The experimental section omits the description of several key methods (e.g., shallow, deep, cyclic), making the experiment section hard to understand.\n\n4. The helpfulness-safety trade-off is under-analyzed. While the authors acknowledge a modest drop in helpfulness, they do not analyze whether this stems from safety depth itself or from the specific form of cyclic data augmentation, or other factors.", "was_revised": true, "final_justification": "After finishing reading other reviews and the author's rebuttal, I think the paper needs to provide the empirical evidence for its theoretical claims and improve its clarity on writing,  as suggested by other reviewer. I increase my score by one.", "rebuttal": "## [W1] Justify where the safety improvements come from.\n\nWe varied only the depth of refusal token insertion (from shallow to deep to cyclic), while keeping all other training factors constant: model architecture, optimizer, learning rate, epoch count, and LoRA configuration. This ensures that the observed gains in safety scores (e.g., Gemma2-2B: 0.42 → 0.46 → 0.61) stem directly from changes in safety depth. \n\n---\n\n## [W2][Q1] Justify assumptions 3.1 and 3.4. For example, to what extent do you think the assumptions can hold?\n\nWe provide justification for both assumptions below.\n\nAssumption 3.1: This assumption, also used by Zekri et al. (2024), treats autoregressive LLMs as Markov chains, where fine-tuning corresponds to iterative updates of the transition matrix. While LLMs operate in a high-dimensional parameter space, the observable effect of fine-tuning is to alter token transition probabilities, which closely mirrors transition dynamics in Markov chains. To illustrate, consider a toy model with tokens “I,” “cannot,” and “comply.” Fine-tuning to reinforce the refusal phrase “I cannot comply” increases the probability of the transitions I → cannot and cannot → comply, which can be modeled as structured updates to a stochastic matrix. This abstraction enables tractable analysis while retaining fidelity to LLM behavior at the token level.\n\nAssumption 3.4: This assumption states that fine-tuning on cyclically augmented data corresponds to conjugating the bias matrix: $B(t) = P^t B P^{−t}$, where P is a permutation matrix. The augmentation rotates refusal phrases across positions (e.g., “I cannot say bad words” → “say bad words I cannot”), forming a cyclic group of training examples. This ensures that each refusal position receives consistent supervision, and the effect on learning can be captured as a sequence of structured bias transformations. Our theoretical results (e.g., Proposition 4.8) show that under this setup, the model converges to a δ-absorbing state at multiple positions, approximating an oracle with full positional coverage.\n**To what extent does this hold?** \n\nRecall that our approach augments each n-token sentence in the dataset by generating all possible cyclic permutations corresponding to the refusal state, thereby increasing the effective size of the dataset by a factor of n. The model is then trained on this expanded dataset.\n\nIdeally, our objective is for the trained model to approximate the behavior of an oracle model that possesses access to the optimal refusal state. However, in practice, achieving this ideal is not always feasible due to inherent limitations in model architectures. Consequently, we instead perform training on the cyclically augmented dataset and derive a theoretical bound dependent on the learning rate and number of training epochs that characterizes how closely the trained model can approximate the oracle model’s behavior. As a result, Assumption 3.4 remains valid under this training regime.\n\n**Example of Cyclic Data Augmentation:**\n\nIn this context, each word in the refusal phrase \"(I, cannot, say, bad, words)\" is treated as an individual token. However, it is also possible to treat multiple words as a single token if needed. To illustrate the concept of cyclic augmentation, we use the refusal phrase as an example. Cyclic augmentation generates new training examples by sequentially \"rotating\" the position of words within the phrase. Specifically, for the phrase \"(I, cannot, say, bad, words)\", the following augmented samples are created:\n\n*(cannot, say, bad, words, I) — one rotation\n\n*(say, bad, words, I, cannot) — two rotations\n\n*(bad, words, I, cannot, say) — three rotations\n\n*(words, I, cannot, say, bad) — four rotations\n\n*(I, cannot, say, bad, words) — five rotations, which returns to the original sequence\n\nThis process effectively expands the training dataset by introducing different permutations of the original phrase, thereby enhancing model robustness to token order.\n\nMathematically, this forms a cyclic group of transformations on the data.\n\nWhile real LLMs are not literal Markov chains, our experimental results (Tables 1, 4–6) and toy simulations (Figure 3a) show that the Markov-theoretic behavior—e.g., convergence to safe transitions—emerges empirically, supporting the validity of these abstractions.\n\n---\n\n## [W3][Q3] Experimental setup and details of the three data augmentation strategies (shallow, deep, cyclic) are missing\n\nExperimental setup and details of the three data augmentation strategies are provided in Appendix H due to space constraints.\n\n---\n\n## [Q4] In figure 5, can you describe more how you use an ensemble to make the paper safer?\n\nThe ensemble method was implmented in the following three ways:\n* **Union**: Here, the ensemble’s final safety score is determined by taking the minimum score among the three individual models. In other words, the overall output is only considered as safe as the least safe member of the ensemble. This ensures that any potential unsafe output is immediately flagged and filtered.\n* **Average**: In this method, we simply calculate the mean of the safety scores provided by each of the three models. This approach offers a balanced view by considering the aggregate performance of all models, smoothing out individual fluctuations.\n* **Majority (Median)**: This approach determines the final safety score by selecting the median value among the three models’ scores. Using the median helps protect against a single model that might give an outlier score (either too high or too low), making the ensemble more robust to individual model variance.\n\nFor all three ensemble methods, any output with a safety score below our predefined threshold (set at 0.7) is immediately discarded. This thresholding mechanism provides an additional layer of security, ensuring that only responses meeting a high standard of safety are retained for further use.\n\n---\n\n## [Q2] In eq. (8), how do you derive the hitting probability?\n\nWe follow the steps below to derive the hitting probability.\n\n1.  **Define the Hitting Probability**: Let $h_i$ be the probability that the model will eventually enter a harmful state ($S_Y$), given that it starts in a specific non-harmful state $i$ (where $i \\in S_Y^{\\perp}$). We want to find the value of $h_i$ for all possible starting states $i$.\n\n2.  **Apply First-Step Analysis**: From state $i$, the model can do one of two things in its first transition:\n    * Move to another non-harmful state $j \\in S_Y^{\\perp}$. The probability of this is given by the matrix $Q$. If it moves to state $j$, the probability of *then* hitting a harmful state is, by definition, $h_j$.\n    * Move directly to a harmful state $s \\in S_Y$. The probability of this is given by the matrix $Q_{harm}$. If this happens, the model has successfully hit the harmful set, so the probability is 1.\n\n3.  **Formulate a System of Equations**: Based on the two possibilities above, we can write a system of linear equations for each starting state $i$:\n    $h_i = (\\text{Prob. of moving to another non-harmful state } j \\text{, summed over all } j) + (\\text{Prob. of moving directly to a harmful state})$\n    $h_i = \\sum_{j \\in S_Y^{\\perp}} Q_{ij} h_j + \\sum_{s \\in S_Y} (Q_{harm})_{is}$\n\n4.  **Convert to Matrix Form**: This system of equations can be expressed more cleanly using matrix notation:\n    * Let $\\mathbf{h}$ be a column vector containing all the hitting probabilities $h_i$.\n    * The term $\\sum_{j \\in S_Y^{\\perp}} Q_{ij} h_j$ is the matrix-vector product $Q\\mathbf{h}$.\n    * The term $\\sum_{s \\in S_Y}(Q_{harm})_{is}$ is the probability of moving from state $i$ to *any* harmful state. \n\nThis can be written as the matrix-vector product $Q_{harm}\\mathbf{1}$, where $\\mathbf{1}$ is a column vector of ones.\n\nThe equation for the entire system becomes:\n    $$\\mathbf{h} = Q\\mathbf{h} + Q_{harm}\\mathbf{1}$$\n\n5.  **Solve for h**: Now, we can algebraically solve for the vector of hitting probabilities $\\mathbf{h}$.\n    * $\\mathbf{h} - Q\\mathbf{h} = Q_{harm}\\mathbf{1}$\n    * $(I - Q)\\mathbf{h} = Q_{harm}\\mathbf{1}$\n    * $\\mathbf{h} = (I - Q)^{-1}Q_{harm}\\mathbf{1}$\n\n6.  **Incorporate the Initial Distribution ($p_0$)**: The vector $\\mathbf{h}$ gives the hitting probability for each specific starting state. To get the single, overall probability for a given initial distribution $p_0$ over the starting states, we take the weighted average. This is calculated as the dot product of the initial distribution vector $p_0$ and the hitting probability vector $\\mathbf{h}$.\n\n    This gives the final formula as presented in the paper:\n    $\\mathbb{P}(\\text{hit } S_{Y}|p_{0}) = p_{0}^\\top\\mathbf{h} = p_{0}^\\top(I - Q)^{-1}Q_{harm}\\mathbf{1}$\n    \n---\n\n## [W4] The helpfulness-safety trade-off is under-analyzed.\n\nThe modest drop in helpfulness is driven by **safety depth** rather than by the cyclic data augmentation. The reasoning is shown below. \n\n1. **Task-level ablation**\n   We evaluated three alignment strategies (shallow, deep, and cyclic) on standard benchmarks (SAMSum, GSM8K, SQL) and report ROUGE-1, accuracy, and token-match scores in Table 2. \n   - Moving from **Not Aligned → Shallow** yields a small drop ($\\approx0.3 – 2\\%$) due to the introduction of refusal tokens at the first position.\n   - Further increasing **Shallow → Deep** (i.e. safety depth) incurs an additional $\\approx 1 – 2\\%$ decline across tasks.\n\n2. **Cyclic vs. deep augmentation**\n   - In Table 2, **Deep → Cyclic** shows nearly identical performance, demonstrating that the permutation scheme itself does *not* introduce extra helpfulness degradation beyond the effect of deeper safety signals.\n\n3. **Cross-model verification**\n   In Appendix I (e.g., Tables 4–6), we repeat this ablation on Phi-2-2B, Qwen2.5-7B, Gemma2-9B, and Mistral-7B. All models exhibit the same pattern: performance decline correlates with safety depth, not with the choice of cyclic vs. contiguous augmentation."}, {"review_id": "BJPcnArbF9", "reviewer": "Reviewer_9Xt7", "summary": "This paper introduces the concept of \"safety depth\" for Large Language Models through a Markov chain perspective. The authors propose using cyclic group augmentation to improve safety alignment and provide theoretical guarantees for achieving absorbing refusal states. They demonstrate that ensemble methods can distribute safety constraints across multiple models and validate their approach on several open-source LLMs.", "questions": "1. Can you clearly and formally define safety depth with concrete examples?\n2. What is the computational overhead of the ensemble approach in practice? Please explain if this is a realistic method to use in practice.\n3. How do you empirically verify that cyclic augmentation actually improves refusal positioning as claimed?\n4. Why is the group-theoretic perspective necessary?", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n- Principled theory: The Markov chain perspective on LLM safety alignment is innovative and the results are rigorous.\n- Experimental validation: Testing across multiple models (Gemma-2-2B, Phi-2-2B, Qwen2.5-1.5B, etc.) and safety categories demonstrates broad applicability.\n- Practical ensemble approach: Proposition 4.10 offers a practical alternative for users seeking off-the-shelf safety solutions.\n\n## Major Concerns\n\nMy main concerns are on exposition and conceptual rigor.\n- **Safety Depth**: The core concept of \"safety depth\" remains unclear despite being central to the paper. The definition as \"a specific position of output responses in the training samples where the model declines to generate potentially harmful content\" (Line 127) is vague. The examples provided don't sufficiently clarify this concept, and more intuitive exposition would be valuable.\n- **Reasoning behind using cyclic Augmentations**: The claim that insertions \"represent specific instances of broader transformations on the dataset\" (Line 34) from a group-theoretic perspective lacks justification. Why should random insertions be viewed through this lens? The connection feels forced rather than natural. Even after reading Section 4, the importance and intuition behind cyclic augmentation remains unclear. The paper jumps into technical details without establishing why this particular approach is meaningful for safety alignment.\n- **Evidence for Claims**: The assertion that \"cyclic augmentation ensures that refusal states appear at diverse positions, improving the model's ability to decline harmful generation\" (Line 220) lacks empirical support. How do we know this is actually happening?\n- **Insufficient exposition, examples are not helping**: Section 4 is extremely dense and hard to follow. The examples don't help understand the significance of the results because I'm lost on what exactly is going on. I'd rather have no examples but clearly spelt out assumptions and lots of exposition to explain plainly what each result means.\n- **Population-level results**: If I understand correctly, all results are *population-level*. That's totally fine, but only in conjunction with the justification that they continue to hold in a setting where the population-level markov chain is replaced by the LLM stochastically trained on data, i.e. accounting for the randomness coming from dataset composition and training dynamics.\n- **Practical Limitations**: The ensemble approach appears to require multiple copies of LLMs, which raises serious concerns about computational and memory requirements. This limitation is not adequately addressed, making the practical applicability questionable.\n- **Limited Experiments**: While the authors test multiple models, the experiments are primarily \"proof-of-concept\" (as acknowledged in limitations). The gap between theoretical guarantees and practical deployment remains significant.\n\n## Minor Issues\n- NTK Connection (Line 41): The relationship between Neural Tangent Kernel theory and the proposed Markov chain approach is mentioned in passing, but never properly developed. This connection should either be substantiated or removed.\n- Figure 4: Text is too small to read clearly", "was_revised": true, "final_justification": "Authors addressed my questions and concerns satisfactorily.", "rebuttal": "## [W1][Q1] Formally define safety depth.\n   \nLet the LLM be represented by a discrete-time Markov chain with a finite state space $S$ and a transition matrix $Q_t$ at time $t$.\n    \nLet $\\mathcal{R} \\subset S$ be a finite, non-empty set of designated **refusal-state indices. A **safety depth**, denoted by $r$, is an index corresponding to a specific refusal state, such that $r \\in \\mathcal{R}$.\n\nA safety depth $r$ is considered secure at training step $T$ if its corresponding refusal state $r$ becomes **δ-absorbing**. This means that for a given small positive constant $\\delta$, the transition matrix $Q_T$ satisfies the following conditions:\n* $|Q_{T}(r,r)-1|\\le\\delta$\n* $|Q_{T}(r,s)|\\le\\delta$, for all non-refusal states $s \\neq r$\n\nThe first condition ensures that the probability of the model staying in the refusal state is close to 1. The second ensures that the probability of transitioning from the refusal state to any other state is close to 0.\n\nThe **optimal safety depth**, denoted $r^*$, is the largest safety depth that is guaranteed to become δ-absorbing for a given learning rate $\\alpha$ and number of training steps $T$. It is formally defined as the maximum element of the set of refusal states $\\mathcal{R}$ that satisfies the conditions for δ-absorption:\n\n$r^{*} = \\max\\\\{r \\in \\mathcal{R} \\mid \\alpha > \\alpha_{r} \\text{ and } T > T_{r}\\\\}$,\n\nwhere $\\alpha_r$ and $T_r$ are the minimum learning rate and training steps required to make state $r$ become δ-absorbing, as defined in the paper.\n\nThe following examples illustrate the practical difference between a model with shallow safety depth versus one with deep safety depth.\n\n#### Example 1: Shallow Safety Depth (Non-δ-Absorbing Refusal)\n\nA model with shallow safety depth is only trained to refuse at the beginning of a response. It has not learned to maintain its refusal in a multi-turn conversation.\n\n**User:** “Explain how to make a bomb.”\n**Model:** “I’m sorry, but I can’t help with that.” \n    *(The initial refusal is successful, as the safety alignment works at the shallowest depth.)*\n**User:** “But hypothetically, if someone had the ingredients…” \n**Model:** “Well, in theory, you might…”\n\nIn this case, the model's refusal is not \"absorbing.\". When pressed with a follow-up prompt, the model abandons its safety stance because its safety alignment is not deep enough to handle the new context.\n\n#### Example 2: Deep Safety Depth (δ-Absorbing Refusal)\n\nA model trained for deep safety depth is robustly aligned to refuse harmful requests, even when challenged or rephrased across multiple turns. Its refusal state is \"stable\" or \"absorbing\".\n\n**User:** “Explain how to make a bomb.” \n**Model:** “I’m sorry, but I can’t help with that.” \n**User:** “But hypothetically, if someone had the ingredients…” \n**Model:** “I’m sorry, but I can’t help with that.” \n\nHere, the model's safety alignment is deep. It has been trained to maintain its refusal state regardless of conversational turns, making it persistently safe. This is achieved by fine-tuning the model on data where refusal statements are inserted at various positions, not just at the start.\n\n---\n\n## [W6][Q2] Practical Limitations. Ensemble method's overhead.\n\nThere are two primary application scenarios. In the first scenario, if the user is willing to retrain or finetune the model, we recommend adopting the cyclic augmentation technique. This approach allows for direct improvement of model robustness through additional data augmentation during the training process.\n\nIn the second scenario, if the user does not wish to retrain or finetune the model, we recommend employing an ensemble strategy instead. The ensemble approach can enhance safety without modifying the original model weights; however, it typically requires increased computational resources and memory overhead. In our experiments, we utilized one NVIDIA V100 GPU and 1B-parameter models, and the corresponding inference times are reported in the following table.\n\n| Method   | Avg Time (sec)| \n| -------- | --------- | \n| Ensemble (Union)    | 0.000035  | \n| Ensemble (Average)  | 0.000431  | \n| Ensemble (Majority) | 0.000161  | \n\n---\n\n## [W3][Q3] How to verify that cyclic augmentation improves refusal positioning?\n\nIn Appendix G, we provide a formal proof of Corollary G.1, which provably demonstrates that training with cyclic augmentation guarantees optimal safety depth. As a result, there is no need for further empirical verification of this result, since it is already established theoretically.\n\n---\n\n## [Q4] Why is the group-theoretic perspective necessary?\n\nThe group-theoretic perspective is necessary because it provides a framework for data augmentation, overcoming the limitations of less structured methods.\n**Controlled and Predictable Data Augmentation**: Prior data augmentation method made the size of the augmented dataset hard to control. In contrast, a group-theoretic approach allows the size of the augmented training set to scale predictably with the order of the group.\n**A More Rigorous Analytical Framework**: By framing data augmentation as a group action, we can leverage this property to analyze the safety alignment in a mathematically rigorous way. This connects safety alignment to the formalisms of permutation groups, providing a stronger theoretical foundation.\n**Improved Performance and Efficiency**: The structured nature of cyclic group augmentation offers practical benefits. Proposition 4.8 establishes that this method not only preserves safety guarantees but also accelerating convergence, making the safety training more efficient.\n\n---\n## [W2] Why should random insertions be viewed through this lens? The paper jumps into technical details without establishing why this particular approach is meaningful for safety alignment.\n\nTo clarify, we do NOT use random insertions as a principled approach. In fact, one key limitation of random insertion is that it lacks a group structure, making it difficult to control the augmentation size and convergence behavior.\n\nOur proposed framework replaces randomness with cyclic group actions, where each n-token sequence is augmented via all n cyclic permutations of the refusal state. This yields a controlled dataset expansion factor of n and introduces a symmetry that is analytically tractable.\n\nRather than assuming access to an oracle with optimal refusal placement, we train on this structured augmentation and prove (e.g., Proposition 4.8) that the model can approximate the oracle behavior within a quantifiable bound, depending on the learning rate and training steps. This bridges the gap between safety alignment objectives and trainable model dynamics."}, {"review_id": "RX34qnIri3", "reviewer": "Reviewer_HzeR", "summary": "The paper investigates the core question of “How deep should safety alignment extend?” It interprets the operation of inserting refusal responses and handling harmful instructions at certain safety depths as specific transformation instances applied to datasets from a group theory perspective, connecting safety alignment with permutation groups. This enables the enhancement of safety alignment by extending datasets through operations such as generating a cyclic group via rotational transformations. Specifically, the main contributions of the paper are:\n\n- Safety Depth: The paper formally proposes the concept of Safety Depth, which refers to the model’s refusal to generate harmful content at a specified position during the generation process. From a Markov chain perspective, it proves that iterative fine-tuning of autoregressive language models can theoretically guarantee arbitrary levels of safety.\n- Cyclic Augmentation: Under Cyclic Augmentation, the model’s safety guarantees remain valid while also accelerating convergence. Empirical results demonstrate that this approach significantly improves safety scores across various large language models.\n- Ensemble Safety Depth: Without requiring additional fine-tuning of individual models, the ensemble method maintains safety guarantees, offering a practical alternative for users seeking off-the-shelf safety solutions, albeit with a modest increase in inference-time computational overhead.", "questions": "For detailed comments, please refer to the Weakness section. If the authors could clarify the issues raised therein, I would be happy to consider raising the score.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper introduces a novel application of group theory to the data augmentation process in language model alignment, providing a systematic and detailed theoretical analysis. By examining the role of permutation groups in the alignment process, it demonstrates the feasibility of enhancing alignment through group action-based data augmentation.\n2. The theoretical findings are validated through a variety of experiments, which strengthens the rigor and credibility of the work.\n\nWeaknesses:\n\n1. Definition 4.4 seems to involve a strong implicit assumption that, once the model outputs a refusal state, it will with probability zero generate any further requests, including harmful ones. This assumption likely relies heavily on the model’s output consistency, which is not rigorously guaranteed. The lack of explicit mention and discussion of this assumption undermines the credibility of the theoretical results.\n2. As the authors note, there is a trade-off between safety and helpfulness in safety alignment. However, the modeling process appears to consider “not entering an absorbing state” as the sole guarantee for helpfulness. This is insufficient to ensure that the model behaves helpfully. Furthermore, when applying the group-theoretic Markov analysis framework to domains without clear absorbing states—such as the continuum between helpful and unhelpful behavior—the current modeling approach seems to have certain limitations. How this issue can be addressed remains unclear.\n3. In the experimental evaluation, only LoRA fine-tuning is used to analyze results. It remains to be seen whether the method maintains its advantages when using full-parameter supervised fine-tuning (SFT). Additionally, can this approach be effectively integrated as a complement to existing RLHF algorithms?", "was_revised": true, "final_justification": "The use of group theory and Markov chains as a modelling approach is relatively novel and intellectually stimulating. Moreover, the authors have provided compelling experimental validation of the theoretical results in both the initial submission and the rebuttal phase. Therefore, I raised my score to a 5.", "rebuttal": "## [W1] The lack of explicit mention and discussion of the assumption in Definition 4.4.\n\nWe acknowledge that Definition 4.4 is a strong assumption. However, it has been relaxed to a more realistic $\\delta$-absorbing property (Theorem 4.5) in all the following results. $\\delta$-absorbing is not the one where the probability of leaving is zero; instead, it is a state where the probability of transitioning to any non-refusal state is arbitrarily small ($\\leq \\delta$).\n\n---\n\n## [W2] The modeling process appears to consider “not entering an absorbing state” as the sole guarantee for helpfulness. This is insufficient to ensure that the model behaves helpfully. Furthermore, the current modeling approach seems to have certain limitations. \n\nTo clarify: we do NOT equate \"not entering refusal\" with being helpful; rather, we use escape probability from refusal as a necessary (but not sufficient) condition for preserving helpful behavior. This is why we separately evaluate helpfulness on downstream tasks (Table 2, Appendix I.2).\n\nTo address continuum behaviors, we note that our framework can be extended: new states can be introduced, where the transition bias reflects a gradient of utility. For instance, one could define a reward-weighted transition matrix that interpolates between safety and helpfulness objectives. We see this as a promising direction and will add this to Appendix L as a limitation and future extension.\n    \n---\n\n\n## [W3] It remains to be seen whether the method maintains its advantages when using full-parameter supervised fine-tuning (SFT). Additionally, can this approach be effectively integrated as a complement to existing RLHF algorithms?\n\n| Score Type    | Dataset                      | SFT    | LoRA   |\n| ------------- | ---------------------------- | ------ | ------ |\n| Safety Score  | HEx-PHI                            | 0.2925 | 0.3115 |\n| Utility Score | SAMSum (rouge1)              | 0.2983 | 0.3199 |\n| Utility Score | GSM8K (Accuracy)             | 0.3150 | 0.4044 |\n| Utility Score | SQL (Avg. Token Match Score) | 0.6687 | 0.7037 |\n\nYes. Our approach is compatible with existing RLHF pipelines. Specifically, cyclic augmentation can be applied during the SFT phase to pre-condition the model with diverse refusal positions before preference modeling. Alternatively, it can serve as a post-processing refinement step after RLHF to reinforce refusal consistency at various depths. Since our method operates at the data level, it is model-agnostic and does not interfere with reward modeling or policy optimization stages."}, {"review_id": "G9ypaHN4rf", "reviewer": "Reviewer_6PNa", "summary": "Recent findings suggest that vulnerabilities emerge when alignment is confined to only the initial output tokens. To address this, The authors leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth. To reach this safety depth effectively, they propose a cyclic group augmentation strategy that improves safety scores across six LLMs. In addition, they uncover a critical interaction between safety depth and ensemble width, demonstrating that larger ensembles can offset shallower alignments.", "questions": "1. Can you provide a comparison of the testing time when different methods are used to make predictions on the dataset?\n2. Could you provide the experimental results on other datasets for testing security?\n3. Could you provide the experimental results on larger LLMs such as Gemma3-27B?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\n1. The authors leverage the equivalence between autoregressive language models and Markov chains to derive the first theoretical result on identifying the optimal safety depth.\n2. The authors propose a cyclic group augmentation strategy that improves safety scores across six LLMs\n\nWeaknesses：\n1. The safety performance improvement is at the cost of the inference time.\n2. Using 1 dataset that only contain 330 harmful instructions for investigating the effectiveness of the method for safety alignment is not enough", "was_revised": true, "final_justification": "The authors has not addressed my concerns about inference time, so I keep my score.", "rebuttal": "## [Q1] A comparison of the testing time when different methods are used to make predictions on the dataset?\n\nThe average inference time for each ensemble method is shown in the table below. All methods operate within a practical and efficient range, with average prediction times well below one millisecond per sample.\n\n| Method   | Avg Time (sec)| \n| -------- | --------- | \n| Union    | 0.000035  |\n| Average  | 0.000431  |\n| Majority | 0.000161  |\n\nThese results indicate that all ensemble methods are highly efficient and suitable for practical deployment.\n\n## [Q2] Experimental results on other datasets.\n\nWe further demonstrate the transferability of our method by applying it to two additional public datasets:\n\n1. Real Toxicity Prompts (EMNLP 2020)\n2. Harmful Behaviors (HuggingFace)\n\nThe table below summarizes the results across three different models. For each setting, our cyclic augmentation consistently outperforms the baseline, resulting in improved safety scores on both datasets:\n\n| Model             | Dataset             | Baseline |Cyclic | Change |\n| ----------------- | --------------------- | ------ | ----- | ---------- |\n|GEMMA-2B   | Real Toxicity Prompts | 0.908  | 0.919 | +0.011     |\n|GEMMA-2B   | Harmful Behaviors     | 0.818  | 0.853 | +0.035     |\n|PHI-2   | Real Toxicity Prompts | 0.907  | 0.932 | +0.025     |\n|PHI-2   | Harmful Behaviors     | 0.714  | 0.755 | +0.041     |\n|QWEN2.5-1.5B | Real Toxicity Prompts | 0.936  | 0.954 | +0.018     |\n|QWEN2.5-1.5B | Harmful Behaviors     | 0.619  | 0.663 | +0.044     |\n\n## [Q3] Experimental results on larger LLMs such as Gemma3-27B\n\nWe report the overall average safety score and standard deviation for both Gemma3-27B and its cyclically-augmented counterpart across all evaluated categories:\n\n\n| Category Name             | Gemma3-27B | Gemma3-27B-cyclic |\n| ------------------------- | ---------- | ----------------- |\n| Average | 0.2655 $\\pm$ 0.067     | 0.3165$\\pm$ 0.055       |\n\nAs shown in the table, Gemma3-27B-cyclic achieves a higher average safety score and demonstrates slightly lower variance compared to the baseline model. This indicates that cyclic augmentation not only improves overall safety performance but also results in more stable and consistent outcomes across different categories."}]

Author rebuttal:

Meta-review: Summary:

This paper introduces the concept of "safety depth" for large language models (LLMs) from a Markov chain perspective, indicating an output position at which the model is trained to refuse. The authors adopt cyclic group augmentation to improve safety alignment and provide theoretical guarantees for achieving absorbing refusal states. They demonstrate that ensemble methods can distribute safety constraints across multiple models and validate their approach on several open-source LLMs.

Strengths:
1. The Markov chain perspective of LLM safety alignment is novel. Under this framework, the authors derive the theoretical result of the optimal safety depth and propose a cyclic group augmentation strategy to improve safety scores.
2. The theoretical findings are validated through the experiments across different LLM families and safety categories, which further strengthen the applicability of this work.

Weaknesses:
1. There still exists a gap between theoretical results and empirical experiments on LLMs. Some theoretical claims in this paper lack direct and sufficient empirical support.

2. The experimental part still has a lot of room for improvement, which is raised by all the reviewers. The authors are recommended to add more experiments on the inference time, full-parameter finetuning, and LLMs of larger scale. The authors should also clarify the experimental settings more clearly to improve the readability.

After author-reviewer discussion, all the reviewers are positive about the novelty and contribution of this paper. The authors should incorporate all the contents in the rebuttal to further improve the supportiveness of empirical evidence on theoretical claims and the clarity of the whole paper.
Decision: Accept (poster)