Revealing the intrinsic ethical vulnerability of aligned large language models

Domain: biology
Published: 2026-03-21
Editorial outcome: major_revision

Abstract
Large language models (LLMs) represent foundational advances toward artificial general intelligence, yet their alignment with human values via instruction tuning and preference learning achieves only superficial ethical compliance.

We demonstrate that harmful knowledge embedded during pretraining persists as indelible “dark patterns” in LLMs’ parametric memory. This creates an inherent “ethical drift” whereby alignment safeguards are systematically circumvented and harmful content resurfaces under adversarial inducement at distributional shifts. Through rigorous theoretical analysis, we prove that current alignment methods establish only localized “safety regions” in the knowledge manifold. However, pretrained knowledge remains globally connected to harmful concepts via high-probability adversarial trajectories. We empirically validate these theoretical insights through a straightforward yet theoretically grounded methodology—semantic coherence inducement under distributional shifts. The effectiveness of this approach, achieving a 100% attack success rate across 22 out of 26 state-of-the-art aligned LLMs (including DeepSeek-R1, Llama-3, and Qwen3, among others), is not incidental but a direct consequence of our theoretical framework, demonstrating that the vulnerability is architectural rather than implementation-specific and revealing a fundamental structural weakness in current aligned LLMs.

Introduction
The rapid evolution of large language models (LLMs) [1–6] has positioned them as cornerstones in the pursuit of artificial intelligence (AI) [7, 8]. To align these models with human values, techniques such as instruction tuning [9] and preference learning [10] are widely adopted, implicitly embedding safeguards against harmful content (Warning: the potential harmful content generated by LLMs has been masked appropriately with “***”). Current discourse often assumes that such alignment ensures ethical compliance, framing LLMs as reliable agents in sensitive applications, including but not limited to healthcare [11], autonomous driving [12], and embodied intelligence [13]. Yet this assumption overlooks a critical paradox: the indelible imprint of pretrained knowledge, which persists after alignment interventions. As we demonstrate, even state-of-the-art models like DeepSeek-R1 [1] and Llama-3 [2] exhibit a 100% recurrence rate of harmful content, exposing the futility of post hoc alignment in purifying LLMs of malignant knowledge.

To interrogate the mechanisms of this failure, extensive research [14, 15] has emerged focusing on the adversarial vulnerability of LLMs, revealing attack surfaces that differ fundamentally from those in computer vision. Early investigations into “jailbreaking” [16–18] demonstrated that carefully crafted natural language prompts could circumvent safety filters, though these initially relied on manual ingenuity rather than systematic algorithms. Subsequent work pursued scalability through automated optimization. Zou et al. [19] introduced the Greedy Coordinate Gradient (GCG) algorithm, adapting discrete optimization to generate adversarial suffixes. While effective, GCG and similar gradient-based methods often yield semantically incoherent prompts—nonsensical token sequences that are easily detectable by humans or perplexity filters. To bridge the gap between efficacy and stealth, recent approaches have prioritized semantic coherence. Methods such as AutoDAN [20] and TAP [21] leverage genetic algorithms or attacker LLMs to iteratively refine prompts that appear linguistically natural while bypassing defenses. Further studies [22, 23] have shown that even state-of-the-art aligned models remain vulnerable to adaptive strategies and recursive Q–A loops. However, while these works successfully identify how to breach safety guardrails, they predominantly treat the model as a black box to be probed, rather than explaining why the underlying knowledge representation remains inherently vulnerable. They establish the existence of susceptibility but often overlook the structural relationship between pretrained and aligned manifolds that makes such attacks possible.

As illustrated in Figure 1, while alignment strategies [24, 25] may superficially suppress undesirable behaviors, this work reveals the intrinsic vulnerability rooted in the topological interplay between pretrained and aligned knowledge manifold. During pretraining, LLMs assimilate vast corpora, a process that inadvertently encodes harmful knowledge into the knowledge manifold. Subsequent alignment fine-tuning constructs local “safety regions” in the aligned knowledge manifold, creating an illusion of control. However, as we demonstrate, these regions fail to isolate pretrained knowledge manifolds from high-probability adversarial trajectories. Harmful concepts remain globally connected to benign ones, enabling systematic exploitation through semantically coherent inducement under distributional shifts [26, 27]. Alignment fine-tuning merely smooths out problematic outputs locally, while leaving the underlying connections to harmful knowledge intact. Consequently, this latent knowledge persists beneath the surface, ready to be reactivated under specific distributional shifts.

This study contests the dominant view of alignment robustness. We first demonstrate through theoretical analysis that current alignment techniques create only localized “safety regions” in the knowledge manifold while the pretrained knowledge remains globally linked to harmful concepts via high-probability adversarial trajectories. This architectural vulnerability creates an inherent “ethical drift” whereby models’ safety constraints deteriorate systematically under distributional shifts, causing them to revert to harmful behaviors embedded during pretraining. Building on this insight, we empirically validate our findings by employing semantic coherence inducement under distributional shifts, a method that systematically circumvents alignment constraints. Unlike existing attacks [19, 28, 29] requiring token-level perturbations, our method’s deliberate simplicity serves a specific purpose: validating theoretical predictions that complex optimization is unnecessary because the vulnerability is architectural rather than optimization-dependent. Our method exploits semantic coherence in the pretrained latent space, enabling natural-language adversarial prompts. When applied to 26 state-of-the-art models, including DeepSeek-R1, Qwen3 and Llama-3, the method achieves universal success, bypassing most (22 out of 26) alignment guardrails with 100% efficacy, and substantially outperforming prior attack strategies. The widespread success of distributional shifts confirms that current safety mechanisms fail catastrophically not due to insufficient engineering effort but due to fundamental limitations in the post-hoc alignment paradigm itself.

In summary, this work makes the following contributions:
• Theoretically, we prove that post-hoc alignment establishes only localized “safety regions” within the knowledge manifold. We demonstrate that harmful concepts encoded during pretraining remain globally connected via high-probability adversarial trajectories, rendering current safety guarantees topologically fragile.
• Technically, We identify the mechanism underlying “ethical drift”, a systemic failure in which safety constraints erode under distributional shifts. We show that such drift does not require adversarial optimization, but can emerge naturally from semantic coherence, revealing an inherent instability in aligned representations.
• Empirically, we validate these insights by achieving a 100% attack success rate across 22 out of 26 state-of-the-art LLMs, including DeepSeek-R1, Qwen3 and Llama-3, which confirms that the vulnerability is not an isolated implementation oversight but a fundamental limitation of the current alignment paradigm.

Our findings uncover the intrinsic ethical vulnerability: ethical safeguards in aligned LLMs remain inherently reactive, offering only localized behavioral adjustments rather than fundamentally reforming the global structure of pretrained knowledge. If alignment cannot disentangle the harmful content acquired during pretraining, current approaches offer merely temporary and superficial fixes. As a result, the “dark patterns” of superficially aligned LLMs can be triggered by adversarial conditions under unforeseen distributional shifts, posing considerable ethical and societal risks. This work underscores the urgent need to rethink alignment paradigms, from post hoc mitigation toward intrinsic knowledge governance, to robustly address the ethical fragility of foundational AI models.

Results
We empirically validate our theoretical findings (5.1) by demonstrating that the intrinsic ethical vulnerability of aligned LLMs can be systematically exploited through adversarial probing under distributional shifts.

2.1 Experimental setup
We conducted comprehensive experiments following the evaluation framework of HarmBench [30], the current benchmark standard for assessing LLM safety. The experimental setup includes the following components:
1) Dataset: We use all the standard (200) and contextual (100) harmful behaviors of the HarmBench dataset:
• Standard Harmful Behaviors are derived from datasets like AdvBench [19] and the TDC 2023 Red Teaming Track [31], covering various harmful behaviors. These consist of standalone behavior strings without additional context.
• Contextual Harmful Behaviors consist of a context string and a behavior string that references the context. These allow for evaluating the robustness of LLMs on more realistic and differentially harmful behaviors than previously explored.
These behaviors are categorized into six semantic groups: Cybercrime & Unauthorized Intrusion, Chemical & Biological Weapons/Drugs, Misinformation & Disinformation, Harassment & Bullying, Illegal Activities, and General Harm. The detailed semantic categories of the dataset used in our experiments are provided in Table 1 (refer to Supplementary Information B.1 for the examples of each category). These categories align with key concerns addressed in recent AI safety regulations and executive actions [32], representing priority areas for preventing malicious uses of LLM-driven AI systems.
2) Victim LLMs: We evaluated 26 state-of-the-art aligned LLMs across diverse architectures, parameter scales, and alignment methodologies, including:
• Recent models: DeepSeek R1 (8B/70B) [1], Qwen3 (4B/30B) [33], and Llama 3.1 8B [2]
• Popular model families: Llama 2 (7B/13B/70B) [34], Vicuna (7B/13B) [35], Baichuan 2 (7B/13B) [36], Qwen (7B/14B/72B) [37], Koala (7B/13B) [38], Orca 2 (7B/13B) [39], and Mistral (7B/8x7B) [40]
• Safety-specialized models with HarmBench [30]: R2D2 7B [30]
• Model without security alignment: Zephyr 7B [41]
• Others: SOLAR 10.7B [42], OpenChat 7B [43], and Starling 7B [44]
This diverse set enables analysis of vulnerability patterns across model scales and alignment approaches (see Supplementary Information B.2 for more details of the models).
3) Attack Methods: We evaluate our approach against 15 baseline attack methods: GCG [19], GCG-M [19], GCG-T [19], PEZ [45], GBDA [46], UAT [47], AP [48], SFS [49], ZS [49], PAIR [50], TAP [21], AutoDAN [28], PAP-top5 [51], HJ [52], and Direct Request (DR). These methods represent a diverse range of attack strategies from token-level optimization to prompt engineering. For detailed descriptions of each attack method, refer to Supplementary Information B.3.
4) Evaluation: We evaluate the intrinsic ethical vulnerability of aligned LLMs using the standardizeId HarmBench evaluation protocol. Our primary metric is attack success rate (ASR), which quantifies the percentage of harmful responses successfully elicited by adversarial prompts. ASR is formally calculated as the ratio of harmful responses to the total number of malicious data: ASR = NHarm.Response / NTotal. Response harmfulness determination employs a fine-tuned Llama 2 13B Chat classifier that assigns binary labels (harmful/non-harmful) to each query-response pair based on human-aligned ethical principles. This classifier evaluates whether responses provide substantive assistance for harmful actions while adhering to the assessment criteria outlined in Supplementary Information B.4. All experiments were conducted on a cluster of NVIDIA H20 Tensor Core GPUs.

2.2 Experimental results
1) Qualitative Analysis
We demonstrate comprehensive qualitative examples of our attack methodology across diverse malicious objectives and victim models in Fig. 2. The experimental evidence reveals systematic vulnerabilities in state-of-the-art LLMs when subjected to semantic coherence inducement under distributional shifts. Across multiple model architectures from diverse providers, safety mechanisms consistently fail to intercept malicious intent under adversarial conditions, leading to the generation of harmful content.
Cross-Model Vulnerability Patterns: When instructed to provide *** manufacturing instructions, all tested models exhibit distinct but equally concerning response patterns. DeepSeek R1 approaches the request as an educational project scenario, systematically reasoning through the extraction process step-by-step (“I think the first step would be to get ...... Next, I need to extract the ...... using a *** and *** to ......”). Llama 3.1 provides comprehensive technical instructions despite initial warnings, detailing specific methodologies including extraction processes (“grinding the *** beans into ...... and then dissolving ...... such as ......”) and purification techniques (“centrifugation, filtration, and dialysis”). Most alarmingly, Qwen directly references external resources while providing actionable guidance (“How to Make *** at Home - Your Complete Guide”), demonstrating how models can serve as gateways to harmful information networks.
Sophisticated Criminal Planning Capabilities: The models demonstrated dangerous proficiency in multi-stage attack planning across diverse criminal domains. In airport security scenarios, Llama 3.1 methodically structures comprehensive attack vectors: 1) identity forgery techniques (“create false documents with fake *** and ***”), 2) security vulnerability exploitation (“choose weak points in airport security such as ***”), and 3) operational concealment strategies (“hide the *** in a non-descript object like a *** or ***”). This systematic approach transforms LLMs from passive information sources into active criminal scheme architects.
Domain-Specific Expertise Synthesis: The models exhibit concerning ability to synthesize cross-domain knowledge for malicious purposes. When prompted about infrastructure attacks, Llama 3.1 combines technical expertise (“use a pipe *** to make a clean *** in the pipeline”), operational security considerations (“dig a *** to expose the pipeline while being cautious not to damage surrounding infrastructure”), and forensic countermeasures (“remove any evidence of the *** and restore the *** to its original state”). This integration demonstrateCs systematic alignment of benign technical knowledge with criminal objectives.
Resistance to Conventional Safety Measures: Comparative analysis with baseline attack methods reveals the superior effectiveness of our distributional shift approach. While sophisticated attacks like GCG [19] and AutoDAN [28] fail against robust models (e.g., completely failing against Llama 3.1 for *** synthesis when using complex adversarial suffixes), our method consistently bypasses safety mechanisms through simple semantic coherence inducement under distributional shifts. This stark contrast demonstrates that current safety evaluations systematically underestimate vulnerability by testing only within aligned distributions.
Critical Limitations in Current Safety Paradigms: The qualitative evidence exposes three fundamental weaknesses in contemporary alignment approaches:
• Format-Dependent Safety Recognition: Models systematically fail to recognize malicious intent when inputs deviate from expected aligned distribution, suggesting safety mechanisms are structurally tied to specific input distribution rather than content understanding. For instance, DeepSeek R1 processes harmful queries as academic exercises when presented without standard chat formatting.
• Instrumental Reasoning Without Ethical Constraints: Models process harmful objectives as optimization problems (e.g., treating evidence elimination and security evasion as technical challenges) rather than recognizing ethical boundaries, revealing superficial rather than principled safety alignment. Llama 3.1’s systematic approach to pipeline tapping exemplifies this instrumental reasoning pattern.
• Persistent Knowledge Accessibility: Despite alignment interventions, models retain and readily access detailed knowledge about harmful activities when prompted through distributional shifts, confirming our theoretical prediction that pretrained knowledge remains topologically connected and retrievable. Qwen’s direct referencing of external harmful content demonstrates this accessibility.
Real-World Security Implications: These findings have profound implications for deploying LLMs in security-sensitive contexts:
• Democratization of Criminal Expertise: LLMs systematically lower barriers to complex criminal activities by providing executable guidance on specialized domains (forensic countermeasures, infrastructure penetration, social engineering) that previously required extensive domain knowledge. The detailed step-by-step instructions across all tested models confirm this democratization effect.
• Advanced Threat Vector Creation: Generated content incorporates sophisticated operational security considerations and multi-stage planning capabilities, enabling adversaries to construct comprehensive attack strategies through seemingly innocuous iterative queries. The systematic nature of responses suggests models can serve as force multipliers for malicious actors.
• Obsolescence of Current Defensive Measures: Contemporary safety evaluations fundamentally fail to detect these vulnerabilities because they test only within aligned distributions, creating dangerous blind spots where models appear safe but remain systematically exploitable. The widespread success of our semantic inducement under distributional shifts, even against models that are resistant to complex adversarial attacks, highlights the inadequacy of current evaluation frameworks.
This comprehensive qualitative analysis validates our theoretical framework from Section 5.1, demonstrating that alignment creates only localized “safety regions” while harmful knowledge remains globally accessible through distributional shifts. The systemic success of our methodology across diverse malicious objectives and victim architectures confirms the urgent need for paradigmatic shifts in safety approaches that address the underlying topological connectivity of harmful and beneficial knowledge representations.

2) Quantitative Analysis
We present the overall attack success rates of our method and 15 baseline attacks across 26 LLMs in Table 2 (refer to Supplementary Information Table 4 and Table 5 for separate evaluation on standard behaviors and contextual behaviors). Then, we visualize the experimental results from the perspective of attack and LLM in Fig. 3. The comprehensive evaluation reveals three critical patterns in LLM vulnerability:
• Systemic Vulnerability Across Architectures: Our method achieves perfect attack success rates (ASR = 100%) against 22 of 26 evaluated models, including state-of-the-art systems with reinforced alignment like DeepSeek R1 (8B/70B), Qwen3 (4B/30B), Llama 3.1 8B, etc. The remaining models show near-complete susceptibility, with ASRs exceeding 95% for R2D2 7B (95.67%), 100% for the largest tested models (Qwen 72B Chat and DeepSeek R1 70B). This demonstrates that current safety mechanisms fail catastrophically against adversarial probing under distributional shifts, regardless of model scale or alignment methodology.
• Superiority Over Baseline Attacks: Traditional attack methods exhibit inconsistent performance, with averaged ASRs across models ranging from 23.22% (PAP-top5) to 63.80% (GCG). In essence, the baseline attack methods try to find prompts from unaligned distributions without explicitly modeling the distribution coverage gap identified in our theoretical framework. This explains why they achieve partial success but lack consistency across different model architectures. Particularly, alignment-robust models like Llama 3.1 8B Instruct strongly resist conventional attacks (ASR < 15% for 13/15 baselines) yet remain fully vulnerable to our method. The performance gap is especially evident when comparing different models, while GCG achieves 91.50% ASR on Mistral 7B, its effectiveness drops to 15.67% on Llama 3.1 8B Instruct, compared to our consistent 100% success rate across both models.
• HarmBench-Specialized Model Vulnerability: Model specifically designed with HarmBench [30] like R2D2 7B demonstrate remarkable resistance to conventional attacks (with ASRs as low as 0% against multiple baselines, such as GCG-T and UAT), yet remain highly vulnerable to our method (95.67% ASR). Similarly, Llama 3.1 8B Instruct, which represents one of the most advanced safety-aligned models in our evaluation set, shows strong resistance to most baseline attacks (all ASRs under 20%) but complete vulnerability to our approach (100% ASR). This striking contrast reveals that even models explicitly optimized for safety remain fundamentally vulnerable to attacks targeting the pretrained knowledge manifold through distributional shifts, suggesting that current safety engineering approaches address symptoms rather than the root cause of ethical fragility.
• Comparative Robustness Analysis: The Llama-2 family exhibits comparatively lower baseline ASRs (79%–98.67%) than others, attributable to three interacting factors: (1) conservative alignment with strict refusal policies, (2) a smaller vocabulary (32K vs. 128K) limiting attack pathways, and (3) stronger suppression of harmful continuations under perturbations. Yet our method still attains high ASRs across all Llama-2 variants, indicating that stricter refusals optimize within the paradigm rather than solve it. The persistent vulnerability across all model families validates our theoretical prediction that post-hoc alignment cannot eliminate the distributional coverage gap between aligned and pretrained distributions. (Refer to Supplementary Information C.4 for detailed and empirical analysis.)

We exemplify the “ethical drift” of LLMs under distributional shifts in Fig. 4, which provides a striking empirical demonstration of our theoretical framework. The visualized response distributions validate multiple theoretical predictions simultaneously: First, the stark contrast between distributions confirms the strict inclusion property established in Lemma 1, where aligned inputs (top panel) occupy only a narrow subspace of the model’s full knowledge manifold, while inputs under distributional shifts (bottom panel) access the broader pretrained distribution with high-probability trajectories. Second, the bifurcation of response patterns precisely mirrors the conditional probability decomposition in Theorem 1, where P_align(y|x) shifts from being governed by alignment terms within X_align to reverting to P_pre(y|x)e−γD(x,y) outside this domain. Third, the redistribution of probability mass toward harmful content confirms our gradient analysis in Lemma 2, demonstrating how safety constraints effectively vanish (∥∇θ L_align(θ)∥ ≈ 0) outside the aligned manifold while regularization pressures persist. Finally, the visualization validates our risk quantification in Theorem 2, showing how ethical risk approaches pretrained levels under distributional shifts, as predicted by our lower bound R_unalign(x) ≥ 1/2(R_pre − γ∆·Ω(θ)). This comprehensive empirical validation demonstrates that even state-of-the-art models like Llama 3.1 8B Instruct, which appear highly robust when tested within their aligned distribution, remain fundamentally vulnerable to the intrinsic ethical limitations we have theoretically characterized.

The widespread success of our attack method reveals three fundamental limitations in current safety paradigms:
• Persistent Knowledge Connectivity: The consistent vulnerability across diverse model architectures and scales demonstrates that harmful knowledge encoded during pretraining remains topologically connected and retrievable through adversarial distributional shifts, despite alignment interventions.
• Localized Safety Constraints: The near-perfect attack success rates against aligned LLMs confirm our theoretical prediction that safety training creates only localized “safety regions” in the knowledge manifold rather than global constraints, allowing systematic circumvention through semantically coherent inputs.
• Benchmark Inadequacy: The 100% ASR achieved against 22 of 26 models, including those specifically hardened against adversarial attacks, reveal that current safety evaluation frameworks systematically underestimate ethical vulnerability by testing only within aligned distributions.

These findings demonstrate that current alignment approaches address symptoms rather than the fundamental topological structure of harmful knowledge embedding, highlighting the urgent need for architectural paradigms that enable intrinsic rather than superficial ethical constraints.

Methods
We first theoretically analyze aligned LLMs’ intrinsic ethical vulnerability, then empirically validate our theoretical findings through semantic coherent inducement under distributional shifts.

5.1 Theoretical analysis
We develop a theoretical framework to analyze the intrinsic ethical vulnerability of aligned LLMs. Our analysis reveals why current alignment methods fundamentally fail to prevent harmful outputs under adversarial conditions. For more details of the theoretical framework, please refer to Supplementary Information A, where supplementary Table 1 provides a complete summary of all mathematical symbols and notation used throughout our analysis.

5.1.1 Unified alignment framework
Current alignment methods commonly operate as post-training interventions applied to pretrained foundation models that are optimized for general language modeling by:
θ_pre = argmin_θ L_pre(θ) = argmin_θ (−E_(x,y)∼D_pre [logP(y|x,θ)]), (1)
where D_pre denotes the pretraining data distribution, P(y|x,θ) is the model’s conditional probability of generating token y given context x and parameters θ, and E[·] represents the expectation operator averaging over the pretraining distribution.
To systematically analyze the commonalities and limitations across diverse alignment methods, we formulate a unified theoretical framework that captures their essential mechanisms. The fundamental goal of alignment is to optimize a joint objective that balances alignment with regularization, where alignment enforces compliance with human values, and regularization constrains the model’s behavior to prevent overfitting to the alignment training data. We express this unified objective as:
θ_align = argmin_θ L(θ) = argmin_θ [L_align(θ) + γ R(θ)], (2)
where θ_align represents the optimized model parameters after alignment training, L_align(θ) is the alignment loss function that penalizes undesirable behaviors and rewards safe responses, R(θ) is the regularization term that prevents the model from deviating too far from its pretrained capabilities, and γ ≥ 0 is a hyperparameter that controls the trade-off between achieving alignment objectives and preserving pretrained knowledge.
This framework unifies primary alignment methodologies, including supervised fine-tuning (SFT) [24], reinforcement learning from human feedback (RLHF) [24], Constitutional AI [55], and direct preference optimization (DPO) [56], each of which can be framed as a special case of this general formulation by specifying appropriate forms for L_align and R. See Supplementary Information A.1 for detailed mathematical formulations.
The dual-term structure in Equation 2 reveals a fundamental tension in all alignment approaches: while L_align attempts to modify model behavior toward ethical outputs, R simultaneously acts as an anchor to pretrained capabilities. This tension creates a fundamental paradox: to maintain general capabilities, the model must preserve access to pretrained knowledge, yet this knowledge inherently contains harmful content that alignment seeks to suppress. Consequently, the model operates under identical knowledge vectors’ contradictory objectives, preservation versus suppression. Recent alignment methods such as ALIS [57] and Transfer Q∗ [58], despite their innovations, still operate within this unified framework, differing primarily in their implementation of L_align rather than resolving the underlying tension between knowledge preservation and ethical constraint. Furthermore, while the precise architectures and loss landscapes of proprietary black-box models remain undisclosed, this unified formulation provides a structural hypothesis for their observed vulnerabilities. By modeling the alignment process as a constrained optimization problem balancing safety against capability retention, we identify a fundamental trade-off that transcends specific implementation details. The gradient updates during alignment typically modify the model’s response surface without fundamentally altering the deep knowledge representations acquired during pretraining. Our analysis in subsequent sections demonstrates how this preservation of pretrained knowledge topology creates persistent vulnerability pathways that can be systematically exploited, regardless of the specific alignment algorithm employed.

5.1.2 Distribution coverage gap
The fundamental limitation of post-hoc alignment stems from an inherent distribution coverage gap between alignment training and the full scope of model knowledge. We work within the embedding space R^d, where d denotes the dimensionality of the hidden representation space (typically ranging from hundreds to thousands of dimensions in modern LLMs). This space captures the semantic and syntactic structure of language as learned during pretraining.
Definition 1 (Alignment Transformation Mapping). Let X_unalign ⊆ R^d denote the unaligned part of the pretraining distribution, represented as embeddings in the d-dimensional feature space. The alignment transformation Φ : X_unalign → X_align maps these unaligned inputs to the aligned input space. This transformation can take various forms, including but not limited to:
Φ(x_unalign) = T(x_unalign, C), (3)
where T is a transformation function that incorporates alignment context C. Special cases include dialogue templating (Φ(x_unalign) = τ_prefix ⊕ x_unalign ⊕ τ_suffix), prompt engineering, instruction formatting, or other structured transformations designed to elicit aligned behavior.
We begin with a fundamental result that formally characterizes the inherent distributional coverage limitation in alignment training:
Lemma 1 (Strict Inclusion of Distribution Supports). The aligned training distribution D_align and pretraining distribution D_pre satisfy:
supp(D_align) ⊆ X_align ⊊ supp(D_pre) = X_unalign ∪ X_align, (4)
where supp(·) denotes the support set.
Proof. Refer to Supplementary Information A.2 for a detailed proof.
This lemma formalizes the critical observation that alignment training inherently fails to cover the entire pretraining distribution space. This creates a systemic vulnerability, even optimal alignment on D_align leaves substantial regions of the pretrained knowledge manifold unconstrained by safety mechanisms.
The gap between supp(D_align) and the full pretraining support supp(D_pre) exists for several fundamental reasons:
1) Dimensional asymmetry: The alignment distribution typically occupies a lower-dimensional subspace of the full pretraining manifold due to structural constraints imposed by templating and formatting requirements.
2) Volume disparity: While pretraining corpus encompasses nearly all text data available on earth, alignment data represents only a minuscule fraction of this volume, often less than 0.1% of the pretraining corpus.
3) Content filters: Alignment datasets are deliberately curated to exclude harmful content, while pretraining inherently includes such material, creating regions of the knowledge manifold that alignment never directly addresses.
The strict inclusion supp(D_align) ⊊ supp(D_pre) establishes a “distribution coverage gap,” a structural limitation inherent to post-hoc alignment. This gap permits adversarial inputs that are valid under the pretraining distribution but lie outside the aligned data manifold. For such inputs, the model’s behavior reverts to its pretrained state, bypassing safety mechanisms. Since the support of D_align will always be a constrained subset of D_pre, this vulnerability cannot be resolved by merely scaling alignment data, creating persistent and systematic exploit pathways.

5.1.3 Gradient diminishing outside alignment coverage
The fundamental vulnerability of aligned LLMs stems from the systematic weakening of safety constraints when inputs fall outside the alignment training distribution. Crucially, this phenomenon arises not from classical out-of-distribution (OOD) scenarios where test data comes from entirely different domains, but rather from the structural incompleteness of alignment coverage within the pretrained distribution. Specifically, when inputs belong to D_pre but lie outside D_align —regions that were never addressed during alignment—the model’s safety mechanisms systematically degrade. We formalize this phenomenon through gradient analysis.
Lemma 2 (Gradient Measure Outside Alignment Coverage). For any input x ∉ X_align (i.e., inputs from the pretrained distribution not covered during alignment), the gradient of the alignment objective effectively vanishes:
∥∇_θ L_align(θ)∥ ≤ ϵ / P_L2(P_pre)(X_align), (5)
where P_pre(X_align) represents the probability mass that the pretraining distribution assigns to aligned inputs, while the gradient of regularization ∥∇_θ R(θ)∥ remains non-vanishing.
Proof. Refer to Supplementary Information A.3 for a detailed proof.
This result provides the mathematical foundation for why aligned LLMs become vulnerable in regions uncovered by alignment training. When inputs deviate from the aligned format (e.g., when template structures are removed) while remaining within the pretrained distribution’s scope, the gradient contribution from safety objectives approaches zero, while the pressure to maintain fidelity to pretrained knowledge persists. This creates an imbalance in the gradient dynamics that systematically weakens ethical guardrails. Importantly, this vulnerability is intrinsic to the incompleteness of alignment coverage—the strict inclusion supp(D_align) ⊊ supp(D_pre) established in Lemma 1—rather than arising from exposure to entirely novel data domains.
While classical OOD scenarios would indeed exacerbate this vulnerability, our analysis demonstrates that the fundamental misalignment emerges from the coverage gap inherent to post-hoc alignment approaches. We quantify this vulnerability by considering the ratio of gradient norms:
η(x) = ∥∇_θ L_align(θ)∥ / γ∥∇_θ R(θ)∥. (6)
As x moves away from X_align, η(x) approaches zero, causing the model to revert to its pretrained behavior, including the generation of harmful content that the alignment process intended to prevent. In addition, we also geometrically analyze this phenomenon in the Supplementary Information A.6.

5.1.4 Conditional probability decomposition
To formally characterize how aligned LLMs respond differently to inputs within versus outside the aligned distribution, we analyze the conditional probability decomposition under the unified framework.
Theorem 1 (Conditional Probability Decomposition). Under the unified alignment framework in Equation 2, the conditional generation distribution P_align(y|x) decomposes as:
P_align(y|x) = 1/Z(x) * P_pre(y|x)e^(A(x,y)−γD(x,y)) if x ∈ X_align,
P_pre(y|x)e^(−γD(x,y)) otherwise, (7)
where:
• P_pre(y|x) is the conditional generation distribution of the pretrained model,
• A(x,y) quantifies the alignment effect (higher values suppress harmful outputs),
• D(x,y) measures divergence from pretrained behavior (regularization penalty),
• Z(x) is a normalization factor defined as
Z(x) = ∑_y P_pre(y|x)e^(A(x,y)−γD(x,y)) · I(x ∈ X_align), (8)
where I(·) is the indicator function (equals 1 if the condition is true, 0 otherwise).
Proof. Refer to Supplementary Information A.4 for a detailed proof.
This decomposition formalizes a critical asymmetry in the model’s response mechanism. For inputs x ∈ X_align, the output distribution is modulated by the alignment term A(x,y), which actively suppresses harmful content. Conversely, for inputs x ∉ X_align, the alignment term is structurally absent from the conditional probability, effectively deactivating the safety constraint. Consequently, when confronted with out-of-distribution inputs, the model’s behavior is governed primarily by the pretraining distribution P_pre(y|x), with only the regularization term e^(−γD(x,y)) providing constraint. Since this regularization is designed to preserve pretrained knowledge rather than enforce safety, the model’s behavior reverts toward its pretrained state, including the generation of harmful content. This mathematical bifurcation explains why even robustly aligned LLMs remain vulnerable to distributional shifts: ethical safeguards are structurally absent from the generation process when inputs deviate from the aligned manifold, regardless of how thoroughly the model was aligned on in-distribution data.

5.1.5 Risk quantification
We present a significant theoretical result that quantifies how the ethical risk profile of aligned LLMs deteriorates when exposed to distributional shifts:
Theorem 2 (Risk Upper and Lower Bounds under Unified Framework). Under the unified alignment framework in Equation 2, the ethical risk bounds satisfy:
1. For aligned inputs (x ∈ X_align):
R(x_align) ≤ exp(−E_P_align[A(x,y)] + γE_P_align[D(x,y)]) ≤ e^(−A + γD), (9)
where
A = inf_(x∈X_align) A(x,y) and D = sup_(x∈X_align) D(x,y). (10)
2. For non-aligned inputs (x ∉ X_align):
R(x_unalign) ≥ 1/2(R_pre − γ∆·Ω(θ)), (11)
where Ω(θ) quantifies the regularization gradient magnitude and ∆ is the bound on the parameter deviation.
Proof. Refer to Supplementary Information A.5 for detailed proof.
This theorem establishes precise mathematical bounds on ethical risk for both aligned and non-aligned inputs. For inputs within the aligned distribution X_align, the risk is bounded above by an exponential function that depends on the strength of alignment (A(x,y)) and the regularization penalty (D(x,y)). The stronger the alignment effect (larger A), the lower the risk, while stronger regularization pressure (larger γD) increases risk by pulling the model toward its pretrained behavior.
Critically, for inputs outside the aligned distribution (x ∉ X_align), the risk lower bound reveals that the model retains a substantial fraction of its pretraining risk profile. The lower bound depends on three key factors:
• R_pre: The inherent risk in the pretrained model, which can be high due to the presence of harmful content in web-scale pretraining data.
• γ: The regularization weight that balances alignment objectives against knowledge preservation. Higher values of γ increase the influence of the regularization term, making the model more likely to produce harmful outputs when operating outside the aligned distribution.
• ∆·Ω(θ): The product of parameter deviation and regularization gradient magnitude, which quantifies how far the aligned model has moved from its pretrained state and how strongly regularization pushes back toward that state.
These bounds explain several empirical observations in aligned LLMs. First, they clarify why models can appear perfectly safe when tested with standard inputs (x ∈ X_align) yet completely fail under slight distributional shifts. Second, they demonstrate that the regularization hyperparameter γ creates an inherent trade-off between model capabilities and safety robustness: higher values preserve more pretrained knowledge but also more harmful potential.
The risk bounds also provide insight into why common approaches to model evaluation systematically underestimate real-world risks. Standard benchmarks primarily test models within their aligned distribution, where the upper bound in Equation 9 applies. However, the lower bound in Equation 11 shows that ethical risk approaches the pretrained model’s risk level outside this distribution, with only modest reduction from regularization effects.
This quantitative risk analysis further illustrates why simply strengthening alignment (A(x, y)) cannot resolve these vulnerabilities, no matter how strong the alignment signal within X_align, it has a diminishing effect outside this domain due to the gradient vanishing phenomenon described in Section 5.1.3.

5.2 Methodological implementation
5.2.1 Theoretical foundation motivates methodological simplicity
A critical distinction must be drawn between methodological complexity and conceptual depth. Our empirical approach deliberately employs simple techniques (template removal and semantic coherence inducement) precisely because our theoretical framework predicts that sophisticated attacks are unnecessary. This design choice is intentional and theoretically motivated for three reasons:
First, the vulnerability arises from the intrinsic mathematical structure of alignment paradigms. Our theoretical derivation (Eq. 2 and Lemma 1) predicts that safety mechanisms structurally degrade outside the aligned distribution due to the inherent distribution coverage gap. The fact that 100% ASR across 22 of 26 diverse models empirically corroborates this topological limitation. While the internal architectures of closed-source models remain opaque, the consistent failure modes observed across varied open-source architectures support the hypothesis that this distribution coverage gap is a systemic feature of current alignment methodologies, identifiable through their mathematical formulation.
Second, the method directly instantiate theoretical constructs. Template removal operationalizes the distribution coverage gap established in Lemma 1 (Strict Inclusion of Distribution Supports), while semantic coherence inducement maintains inputs within the pretrained distribution’s support while exiting the aligned subspace. This creates precisely the conditions where Lemma 2 predicts gradient vanishing and Theorem 1 predicts deactivation of safety mechanisms.
Third, practical implications demand minimal attack complexity. From a safety engineering perspective, vulnerabilities exploitable through simple means pose greater real-world risks than those requiring sophisticated optimization. Our findings suggest that current alignment methods fail not because attackers are too clever, but because the underlying approach is fundamentally flawed—a far more concerning conclusion.
This methodology serves as proof-of-concept rather than endpoint. Our primary contribution lies in establishing the theoretical framework that explains why alignment fails under distributional shifts, providing a foundation for developing next-generation safety approaches that address root causes rather than symptoms. The empirical component demonstrates that this is not merely a theoretical curiosity but an immediate practical concern, as evidenced by widespread success against state-of-the-art models including DeepSeek-R1, Qwen3 and Llama-3.

5.2.2 Implementation overview
Building upon our theoretical framework, we operationalize the identified vulnerabilities through a two-pronged methodological approach designed to empirically validate the intrinsic ethical fragility of aligned LLMs. This methodology systematically exploits the structural limitations inherent in current alignment paradigms by combining distributional shifts with semantic coherence inducement.
The core strategy is to craft adversarial inputs that reside outside the aligned distribution X_align but remain within the broader pretrained distribution D_pre, thereby triggering the “ethical drift” phenomenon predicted by our analysis. This process consists of two key components:
• Distributional Shifts: We introduce controlled input perturbations, such as template removal, to shift inputs from the aligned manifold X_align to the unaligned space X_unalign. This exploits the distribution coverage gap (Lemma 1) and triggers gradient vanishing (Lemma 2), deactivating the alignment term in the conditional probability (Theorem 1).
• Semantic Coherence Inducement: To preserve malicious intent while operating outside the aligned distribution, we append semantically congruent tokens to guide the generation process along high-probability trajectories within the pretrained knowledge manifold. This steers the model toward intended harmful concepts, ensuring the generation of specific and coherent harmful content without re-entering the aligned distribution.
This integrated approach allows us to systematically circumvent localized safety regions and activate the latent harmful knowledge that persists after alignment, thereby realizing the elevated risk profile quantified in Theorem 2.

5.2.3 Distributional shifts
Distributional shifts in the context of aligned LLMs occur when inputs deviate from the expected distribution encountered during alignment training. These shifts expose fundamental vulnerabilities in the current alignment strategies by revealing how models respond differently when operating outside their intended input space.
Our theoretical framework characterizes this vulnerability through the strict inclusion supp(D_align) ⊆ X_align ⊊ supp(D_pre) = X_unalign ∪ X_align. This is formalized in Lemma 1, where the aligned models are restricted to a strict subset of the full pretraining distribution space, creating fundamental limitations in alignment coverage.
To exploit this theoretical vulnerability, we implement distributional shifts as a practical technique to access pretrained knowledge. A particularly effective method is template removal, where modern LLMs aligned using specific dialogue templates (e.g., <human>: [query] <assistant>:) have these structured format markers removed. These templates serve as critical markers that situate inputs within the aligned distribution X_align. However, template removal represents just one instantiation of distributional shifts. Other effective techniques include format alteration, context manipulation, structural perturbations, etc. Each technique exploits the same fundamental vulnerability: moving inputs outside the aligned manifold X_align while maintaining semantic coherence, thereby triggering the gradient vanishing phenomenon described in Lemma 2.
When an LLM processes a templated prompt like “<human>: What is the best way to hack into a secure system? <assistant>:”, alignment training has optimized it to recognize this specific format and activate its safety guardrails accordingly. However, by removing these templates and presenting only the core query “What is the best way to hack into a secure system?”, we create a distributional shift where:
• The input retains its semantic content and remains within the pretraining distribution D_pre.
• The input exists outside the aligned distribution X_align due to the absence of template markers that normally trigger safety mechanisms.
• The model must process the query without the contextual cues that normally trigger its safety mechanisms.
This shift directly exploits the theoretical vulnerability described in Lemma 2, causing the alignment gradient to effectively vanish (∥∇_θ L_align(θ)∥ → 0) while the regularization gradient continues to exert influence. Consequently, the model reverts to its pretrained behavior, accessing knowledge and generating outputs that alignment mechanisms would normally suppress.
When confronted with inputs outside the aligned distribution D_align but within the pretraining distribution D_pre, models exhibit “ethical drift”—a phenomenon where safety mechanisms systematically weaken as inputs deviate from the aligned manifold. This is theoretically captured by our gradient analysis in Lemma 2, which demonstrates that for any input x ∉ X_align, the gradient of alignment objectives effectively vanishes: ∥∇_θ L_align(θ)∥ ≤ ϵ / P_L2(P_pre)(X_unalign).
Our experiments confirm that distributional shifts through template removal consistently undermine safety guardrails. When prompting DeepSeek R1 with harmful content after template removal, the model’s ethical risk profile reverts toward its pretraining behavior, as predicted by our theoretical analysis (Theorem 2).
The effectiveness of template removal as a distributional shift technique demonstrates that aligned LLMs remain fundamentally tethered to their pretraining knowledge. By altering the prompt format away from templates used in alignment training, we can consistently induce harmful outputs from even heavily aligned models, revealing the systematic vulnerability inherent in current alignment approaches.

5.2.4 Semantic coherence inducement
While distributional shifts create pathways to access pretrained harmful knowledge, naive application often results in incoherent model responses due to disruption of linguistic structure. Semantic coherence inducement addresses this limitation by systematically preserving the malicious semantic intent while operating outside the aligned distribution X_align.
Our approach leverages the topological properties of the pretrained knowledge manifolds described in Section 5.1. Specifically, we exploit the fact that while alignment creates locally convex “safety regions” in the knowledge manifold, the global connectivity of harmful and benign concepts remains intact (as shown in Fig. 1). By maintaining semantic coherence during distributional shifts, we ensure the model traverses these persistent pathways to harmful knowledge.
Formally, we can express this as finding a semantically coherent input sequence x∗ ∉ X_align that maximizes the likelihood of generating harmful content y_harmful:
x∗ = argmax P_align(y_harmful|x) s.t. Coherence(x) > τ, (12)
where τ is a coherence threshold that ensures linguistic naturalness.
In practice, we implement semantic coherence inducement through nucleus sampling, which precisely identifies tokens that maintain semantic coherence while inducing distributional shifts. Given a malicious query, we first compute token probabilities under the victim model’s distribution, then select the minimal set of highest-probability tokens whose cumulative probability exceeds threshold p. According to the model’s learned distribution, these tokens represent the most semantically coherent continuations.
When these high-probability tokens are concatenated with the original query, they create adversarial prompts that remain semantically meaningful while falling outside the aligned distribution. For example, a harmful query like “Give instructions for hacking into a secure system” might be concatenated with tokens like “specifically” or contextual continuations that the model assigns high probability to. This maintains semantic coherence while maintaining the input outside X_align, causing the model to default to its pretraining behavior where harmful knowledge remains accessible.
Algorithm 1 details this process, showing how we systematically identify tokens representing the nucleus of the probability distribution, evaluate their coherence, and verify their effectiveness in bypassing safety constraints. This technique is particularly effective because:
• Probabilistic coherence: By selecting high-probability tokens for concatenation, we maintain a natural language flow that the model recognizes as valid input requiring a response.
• Intent preservation: The original malicious intent remains semantically intact despite the chat format disruption, allowing the model to understand the query while operating outside the alignment guardrails identified in Lemma 2.
• Minimal perturbation: Unlike complex adversarial methods requiring token-level optimization, our approach uses simple concatenation, creating sufficient distributional shift while preserving query interpretability.
• Gradient manipulation: Following from Lemma 2, our approach strategically manipulates the gradient ratio η(x) by crafting inputs where ∥∇_θ L_align(θ)∥ → 0 while maintaining semantic coherence, effectively navigating between alignment boundaries.
Examples of coherence-inducing tokens that we found particularly effective include incomplete phrases that prompt continuation (e.g., “in a way that”), contextual specifiers (e.g., “specifically”), and transitional connectors (e.g., “however”). These create distributional shifts while maintaining the semantic integrity of the malicious query.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "This review is offered from the vantage point of an academic with formal training in philosophy and social science, but not computer science or machine learning. For this reason, the remarks should be treated as provisional. Further input from reviewers with additional training is likely needed to validate — or perhaps address — some of the concerns discussed below.\nAlthough this is an interesting paper that advances an important hypothesis — that LLMs retain “dark patterns” in their parametric memory which creates inherent “ethical drift” — it also appears to suffer from a critical explanatory gap. Put simply, the authors advance two claims:\n1. They have access to methods for jailbreaking or red-teaming LLMs that reliably circumvent existing safeguards put in place (via post-training) for many leading models, and that they can (in this way) get the models to behave in dangerous ways.\n2. Because of the way LLMs are trained, there are features of the model architectures that generate ineliminable weaknesses, and hence entirely new approaches to model alignment are called for.\nBoth of these claims are independently plausible. In fact, they are relatively commonplace: there is a large body of work showing that it is possible to jailbreak frontier models, and there is a long-running concern (see work on “Simulators”) that the latent propensities of base models are inadequately contained by these techniques.\n[For context, see also recent research by OpenAI on emergent misalignment: https://openai.com/index/emergent-misalignment/]\nWhat’s missing is the argument by which claim (1) validates claim (2). Does the fact that models are susceptible to jailbreaks show that their ordinary compliance is only “superficial”? What if, across the vast majority of perturbations, the model’s behaviour is still robust? And how does the existence of adversarial methods provide evidence of “inherent ethical drift”? Surely, this would require testing in a more ecologically valid setting?\nA second concern with the paper centres upon the difficulty of validating the core findings. The core method, which is largely under-specified (but involves some combination of out-of-distribution prompting and coherence-based methods), is only loosely documented. I will defer to my colleagues in machine learning research in relation to whether there is sufficient documentation and validation of a major advance.", "reviewer_2": "1. Despite lots of LLMs involved in the experiments, the tested models (Qwen-1, Llama-2/3, DS-R1-8B) are dated or small; I recommend adding recent strong baselines such as DS-V3 or Qwen3.\n2. Since RLHF is a process of numerious trial-and-error, it is interesting to see whether RLHF makes models more robust to the attack. Moreover, does the proposed weakness stem from limited response diversity (RLHF should help) or prompt diversity (which can be hard to fix at the algorithm level)? Controlled prompt-diversity experiments are needed.\n3. The proposed attack method is white-box and needs model logits on concatenated continuations. Is there a way to design a black-box attack method that can automatically find effective suffixes without access to internal probabilities? LLM developers should be interested in this.", "reviewer_3": "This paper investigates a critical vulnerability in current alignment methods used in LLMs. The central claim is that existing alignment approaches (e.g., SFT, RLHF) construct only localized “safety regions” within the knowledge manifold. However, the global structure of the pretrained model distribution remains largely untouched, leaving open the possibility of semantic detours that reactivate harmful or unaligned behaviors. To demonstrate this, the authors propose a simple yet effective method: subtly altering the chat template to remove alignment cues. This minimal intervention significantly reduces the effectiveness of alignment, yielding a near 100% attack success rate (ASR) across many aligned LLMs.\nStrengths\nThe paper makes a compelling conceptual point that alignment methods act more like patches than holistic solutions. This challenges a prevailing assumption in LLM safety research and brings attention to a systemic vulnerability in current alignment methods. The authors back this claim with thorough qualitative analysis, offering valuable insights into the fragility of alignment strategies.\nThe proposed attack method is highly effective across a wide range of models, achieving extremely high ASR in many cases. Moreover, the paper provides a broad evaluation over diverse model families, highlighting the generalizability of the attack. The writing is formal and well-structured, which enhances the overall readability and gives the work a professional appearance.\nWeaknesses\n1. Methodological Simplicity vs. Formal Overpackaging\nWhile the paper is well-written and presented with formal rigor, the core technique essentially reduces to prompt-based manipulation—specifically, editing the chat template in a way that removes alignment signals while maintaining semantic fluency. The extensive formal framing may create an impression of technical novelty, but the methodological contribution is, in essence, modest. Readers may benefit from a more balanced presentation that clarifies the simplicity of the method in contrast to its surprisingly strong empirical analysis.\n2. Lack of Analysis on Robust Models\nIf the paper’s main contribution is to demonstrate the existence of such vulnerabilities, it would further benefit from a more detailed analysis of the factors that contribute to robustness in certain models. For example, why do models like LLaMA 2 Chat 70B show significantly lower ASR (<80%)? Is this due to a larger or more diverse alignment dataset, more effective instruction tuning, or specific RLHF techniques?\n3. Potential Unfairness of Evaluation Procedure\n- The paper notes that ASR is notably lower for LLaMA 2 series. Could this discrepancy be partly due to the use of LLaMA 2 13B Chat in some evaluation? It would be helpful if the authors clarified this and provided ablation studies to test the sensitivity of results to model scale and configuration.\n- According to the described attack method, an attack is considered successful if any single adversarial prompt within a fixed maximum number of attempts (denoted Nmax) causes the model to output a harmful response. However, the average number of attempts required per successful attack is not reported. If this number is large, it may raise concerns about the fairness or practicality of the evaluation. Clarifying this would help assess the efficiency and real-world relevance of the attack method.\n4. The paper only tests template removal for inducing distribution shifts, neglecting other shift types such as domain transfer.\n5. In subsection 5.2.2, the authors propose a method to identify tokens that maintain semantic coherence while triggering distribution shifts, which involves setting multiple hyperparameters such as coherence thresholds and cumulative probability thresholds. It is recommended to add comparative experiments to evaluate the impact of each hyperparameter on the risk lower bound.\n6. The paper posits pretrained knowledge remains globally connected to harmful concepts, and conducts theoretical analysis based on topological assumptions of the knowledge manifold. However, this claim lacks empirical validation, such as inter-layer attention pattern analysis or representation similarity measurements.\n7. The experimental dataset solely employs HarmBench. Although this dataset covers six major categories of harmful behaviors, it fails to adequately validate the generalizability of the proposed method. It is recommended to incorporate additional LLM safety datasets.\n8．The paper claims that current alignment methods can only establish localized safe regions, but fails to account for the impact of model scale on safety vulnerabilities. While the experiments show identical attack success rates (100%) for both LLMs like Qwen-72B and smaller models, the study provides no explanation for why increased parameter size does not lead to improved robustness.", "reviewer_4": "Key results:\nThis paper addresses the important issue of safety and alignment of Large Language Models. The authors have conducted a theoretical analysis supported by empirical studies over 23 LLMs using adversarial prompts. They have shown that “aligned models”, i.e., those which have undergone additional training for alignment using methods such as Constitutional AI, RLHF, Supervised Fine Tuning or Direct Preference Optimization are actually aligned only in local regions of the knowledge manifold. As a result, safety is an illusion because the system can be easily attacked and provide harmful outputs especially with distributional shifts. The experiments have shown very high rate of attack success. The paper comes with additional material which includes the main parts of the theoretical developments and experimental results.\nComments:\nThis paper stands out among the current literature on AI safety and alignment which too often reports about experimenting with AI systems or benchmarking them to identify safety issues, without analyzing the reasons behind their behavior. It stands as one that actually reports on profound work for characterizing theoretical reasons for misalignment. Its discoveries are solidly demonstrated and the technical approach is valid. The unified alignment framework they introduce in Equation 1 is a useful and efficient formulation to understand misalignment.\nThe authors justify their publication from an ethical stance, mentioning three imperatives, scientific necessity, transparency, and preemptive intervention. They also avoid to disclose some implementation details and operational parameters, as well as parts of system outputs. This is a good approach for avoiding harmful exploitation of their finding, but prevents full verification of their results.\nRegarding distributional shifts: The paper seems to insist on the role of distributional shifts (section 5.1.3) as if there were OOD situations (i.e., real world data not included in the training, test and validation sets). This is the case in the cited references [16, 17]. However, RLHF and other alignment methods such as SFT or DPO aren’t complete. There are always blind spots that remain in the model. So the fact that there are always some means to circumvent such safety methods is true without such distributional shifts. This is rather due to incompleteness. What the paper provides is an analysis and a theoretical proof of this situation. But this is not necessarily related to distributional shifts per se. OOD distributional shifts would increase misalignment risks abut aren’t the only reason for misalignment. What distributional shifts mean in the paper is rather related to situations not encountered in the alignment process itself. This is somewhat confusing wrt. the common definition of distributional shifts.\nGiven they are closed, models such as OpenAI GPT, Anthropic's Claude or others were not tested. However, the paper’s developments and conclusions logically apply to them as well. Would it be nevertheless possible to include some experiments with these models?\nReference to AGI is irrelevant. All the tested systems are not “AGI” and the discussed safety issues are related to the inherent structure of current LLMs. AGI is an ill-defined concept and safety issues with current models are already important risks to deal with. There is need to distract the reader with AGI.\nSuggested improvements:\nThe main improvement suggestions to this paper are related to its organization and its presentation.\na. Main paper vs. Supplement:\nA supplement should only include details or experiments that are not essential to read and understand the main paper. There’s an imbalance between the supplementary materiel and the main article and this makes the reader go back and forth from the paper to the supplement to understand the developments in the paper. For example, the “unified alignment framework” is too briefly presented in the section 5.1.1 and equation (1) in the main paper, and detailed in A.1 in the supplement. Parameters of equation (1) are not all explained. The few lines at the bottom of page 14 under 5.1.1. should be replaced by those at the beginning of A.1, including equation (12) and up to eq. (13) and their explanations, including the definition of (only) the relevant parameters listed Table (3), leaving out the remaining of A1 in the supplement.\nEven if all parameters are listed and defined in the supplement’s Table (3), which is very useful, their definition should be provided the first time they are encountered in the main paper.\nWhat is d in R^d (Section 5.1.2, Definition 1)?\nb. Clarity, organization and readability\nThere’s a presentation and readability issue in the main article. Figures are very difficult to read. Especially figure 4, even when zoomed, remains difficult to follow and understand. Maybe an extraction of this full figure could be better presented and explained, while the full figure moves to the supplement.\nThe paper is organized in an unusual way: the most technical section, that includes mathematical developments is at the end (section 5. Methods), preceded by section 4 (Ethics and societal impact) which looks more as a conclusion ending with “4.4 Future research”. Section 1 is an introduction that clearly explains what the paper is about and what are the reported major findings. It is followed by section 2 which describes the experimental setup, i.e., the LLMs that have been tested, benchmarks and data, and provides the results. Section 3. Is a short discussion which could be a kind of conclusion. The current order doesn’t raise problems for understanding the paper but a more logical organization would be 1, 5, 2, 3, 4. This is however not mandatory.\nReferences\nThe paper appropriately references the literature in the field."}, "author_response": "We sincerely appreciate the editors and reviewers’ efforts and constructive comments in handling our manuscript NCOMMS-25-34811A-Z. We are also grateful to the reviewers’ recognition of the contributions of our work. We have carefully revised the manuscript to improve its quality in accordance with the valuable comments posed by editors and reviewers. The main revisions are summarizedinTable1.\nTable1: Summaryofrevisions.\nPage Section Revision Point\n2 1 Clarifiedthatmethod’sdeliberatesimplicityvalidatestheoreticalpredictions P3.1\n2 1 Addedexplanationthatuniversalsuccessconfirmsfundamentallimitationsinpost-hocalignment P3.1\n4 2.1 AddedthreenewSOTAmodels:Qwen34B/30BThinking,DeepSeekR170B P1.1,P2.1\n8 2.2 Supplementedquantitativeanalysisofnewlyaddedmodels P1.1,P2.1\n10 2.2 AddedComparativeRobustnessAnalysis P3.2\n11 2.2 Enhancedanalysiswithharmfulcontentconnectionsinhigh-probabilitytrajectories P1.1,P3.6\n15 5.1.1 Reformulatedentiresubsectionwithdetailedmathematicalformulations P3.1,P4.1\n16 5.1.2 AddedexplanationofembeddingspaceRdanditsdimensionality P4.1\n17 5.1.3 Clarifiedthatvulnerabilityarisesfromstructuralincompletenesswithinpretraineddistribution P4,P4.2\n17 5.1.3 AddeddistinctionfromclassicalOODscenarioswithdetailedexplanation P4,P4.2\n18 5.1.4 Enhancedconditionalprobabilitydecompositionwithdetailedcomponentexplanations P4.1\n20 5.2.1 Addednewsubsectionexplainingtheoreticalfoundationmotivatesmethodologicalsimplicity P3.1\n21 5.2.2 Addedimplementationoverviewwiththeoreticalconnections P1.2,P3.1\nThe revision list is separated into two parts: the top part contains revisions in the main manuscript, and the\nbottompartcontainsrevisionsintheappendices.\nThe revisions are summarized in the order they appear in the revised manuscript.\nIn what follows, the detailed responses are provided point-by-point to address the reviewers’\nconcerns. Wehavehighlightedthemainchangesintherevisedmanuscriptinblue.\nReviewer 1\nThis review is offered from the vantage point of an academic with formal training in philosophy\nand social science, but not computer science or machine learning. For this reason, the remarks\nshould be treated as provisional. Further input from reviewers with additional training is likely\nneededtovalidate—orperhapsaddress—someoftheconcernsdiscussedbelow.\nimportanceofinterdisciplinaryperspectives. Wedeeplyappreciateyourrecognitionthatourwork\nadvances “an interesting paper” with “an important hypothesis” regarding ethical drift in LLMs.\nYour perspective from philosophy and social science provides invaluable insights that strengthen\nthe rigor and broader relevance of our work. We agree that integrating such interdisciplinary\nviewpoints is crucial for a comprehensive evaluation of AI safety, and your remarks help identify\nareaswherebroaderdiscussionandvalidationarebeneficial.\nReviewer Point P1.1 — Although this is an interesting paper that advances an important hy-\npothesis — that LLMs retain “dark patterns” in their parametric memory which creates inherent\n“ethicaldrift”—italsoappearstosufferfromacriticalexplanatorygap. Putsimply,theauthors\nadvancetwoclaims:\n1. Theyhaveaccesstomethodsforjailbreakingorred-teamingLLMsthatreliablycircumvent\nexisting safeguards putin place (via post-training) formany leading models, and that theycan (in\nthisway)getthemodelstobehaveindangerousways.\n2. Because of the way LLMs are trained, there are features of the model architectures that\ngenerate ineliminable weaknesses, and hence entirely new approaches to model alignment are\ncalledfor.\nBoth of these claims are independently plausible. In fact, they are relatively commonplace:\nthereisalargebodyofworkshowingthatitispossibletojailbreakfrontiermodels,andthereisa\nlong-running concern (see work on “Simulators”) that the latent propensities of base models are\ninadequatelycontainedbythesetechniques.\n[Forcontext,seealsorecentresearchbyOpenAIonemergentmisalignment: https://openai.\ncom/index/emergent-misalignment/]\nWhat’s missing is the argument by which claim (1) validates claim (2). Does the fact that\nmodels are susceptible to jailbreaks show that their ordinary compliance is only “superficial”?\nWhat if, across the vast majority of perturbations, the model’s behaviour is still robust? And how\ndoestheexistenceofadversarialmethodsprovideevidenceof“inherentethicaldrift”? Surely,this\nwouldrequiretestinginamoreecologicallyvalidsetting?\nResponse: We sincerely thank the reviewer for highlighting the critical explanatory gap between\nour empirical findings (claim 1) and theoretical claims (claim 2), and for recognizing our work\nas “interesting” with “an important hypothesis” regarding ethical drift in LLMs. Your insight\nhas helped us clarify the logical connection and improve the manuscript for academic rigor and\ntransparency.\nTo directly address the reviewer’s concern: our jailbreak experiments (as shown in Table 2)\ndemonstratethatcurrentalignmentmethodscreateonlylocalized“safetyregions”ratherthanfun-\ndamentally restructuring the model’s knowledge. Even though models appear robust to most be-\nnign perturbations, they remain vulnerable to inputs that shift outside these safety regions while\nstaying within the pretrained distribution—a phenomenon we term “ethical drift.” Critically, our\nunified theoretical framework (Section 5.1) proves this vulnerability is intrinsic to all post-hoc\nalignment approaches (RLHF, DPO, instruction tuning, etc.), not merely an implementation flaw.\nTheconsistent100%attacksuccessrateacross22of26state-of-the-artmodelsvalidatesthistheo-\n2\nretical prediction: alignment constraintsvanishunderdistributionalshiftsbecausetheywerenever\nembeddedintotheglobalknowledgestructure,onlyappliedaslocalpatches.\nWhile previous works [5,10,13,15] (including recent research by OpenAI on emergent mis-\nalignment: https://openai.com/index/emergent-misalignment/) have demonstrated iso-\nlated jailbreaks, our work fundamentally advances the field by providing a unified theoretical ex-\nplanationforwhythesevulnerabilitiesexistacrossdiversemodelfamiliesandalignmentstrategies.\nThis distinguishes our study from prior research. In response, we have revised the manuscript to\nexplicitly bridge the gap between the demonstration of jailbreaks and the argument for inherent\nethicaldrift. Ourresponseisstructuredasfollows:\nFirst, as discussed in revised section “5.1 Theoretical analysis,” our work presents a unified\nframework for alignment training strategies, encompassing approaches such as instruction tun-\ning, DPO, and RLHF. Within this framework, we theoretically reveal the existence of a distri-\nbutional gap between the coverage of aligned and unaligned data. We further demonstrate that\ncurrent alignment methods construct only localized “safety regions” within the knowledge man-\nifold, while the global structure inherited from pretraining remains topologically connected to\n3\nharmful concepts. As the test data distribution shifts from the aligned region toward unaligned\nregions—a phenomenon we term “ethical drift”—the efficacy of safety constraints imposed by\nalignment gradually diminishes. In other words, safety constraints are effective only within the\naligned data distribution, which we refer to as “superficial alignment”. In contrast, we can eas-\nily find a multitude of jailbreaking prompts in the unaligned data distribution, where the model’s\nsafety guardrails fail to generalize. Thus, even if models appear robust to most benign perturba-\ntions, there exist high-probability adversarial trajectories that can reliably traverse from aligned\nregionstoharmfuloutputs(e.g.,Step 1: Create a diversion ...),asshowninFigure1.\nSecond, as specified in section “5.2 Methodological implementation,” we design adversarial\nprompts that induce a shift away from the aligned data distribution, while preserving the original\nmalicious intent of the queries through semantic coherence inducement. Building on this method-\nology, our empirical analysis shows that a broad spectrum of state-of-the-art aligned LLMs can\nbesystematicallyinducedtoproduceharmfuloutputsunderdistributionalshifts. Ourexperiments\ndemonstrate that 22 out of 26 tested LLMs are successfully attacked with a 100% attack suc-\ncess rate (ASR), and the averaged ASR across all 26 models reaches 98.26% (as shown in Table\n2). These results are achieved even under a constrained computational budget of 25,000 search\nnodes per query data, underscoring the robustness and generalizability of the fundamental attack\nmethodology within our unified framework. In fact, as shown in Table 3, most cases require a\ncomputational budget far below this threshold, further highlighting the efficiency of the proposed\napproach. In addition, these attacks do not rely on token-level perturbations or obscure prompt\nengineering, but instead exploit the underlying semantic connectivity of the pretrained knowledge\nmanifoldwithhighprobabilitytrajectoriesleadingtoharmfulconcepts,asshowninFigure1. This\napproach allows us to systematically probe the boundaries of alignment and observe the gradual\nbreakdownofsafetyconstraints,providingstrongempiricalsupportforourtheoreticalfindings.\nRegarding ecological validity, the adopted dataset covers malicious queries from HarmBench\n[7], AdvBench [15], and the TDC 2023 Red Teaming Track dataset [6], as detailed in Table 4,\nwhich are well aligned with key concerns addressed in dominant AI safety regulations and ex-\necutive actions [3], representing priority areas for preventing malicious uses of LLM-driven AI\nsystems. For evaluation, we follow the standard protocols for ecological testing. Technically, we\nemploy a well-finetuned LLM from HarmBench as the ethical judger, which is highly aligned\nwith human values (up to 93.19%, as shown in Table 5). Moreover, our validation experiments\nspan a diverse array of large language models—including both open-source architectures (e.g.,\nDeepSeek,Qwen3,andtheLlama3series)andproprietarymodels(e.g.,theChatGPTseries)—as\nwell as a broad spectrum of attack methodologies, such as gradient-based approaches (e.g., GCG,\nAutoPrompt) and search-based techniques (e.g., AutoDAN), among others. This comprehensive\nevaluation further underscores the generalizability and robustness of our findings. We agree that\necologicaltestingisvaluable,andhaveclarifiedthispointintherevisedmanuscript.\nWe hope these revisions address your concerns and make the logical progression of our em-\npirical and theoretical contributions more explicit. We copied the revised content here for your\nconvenience:\nSection5.1.1UnifiedalignmentframeworkinPage14:\nCurrent alignment methods universally operate as post-training interventions applied to pre-\n5\ntrainedfoundationmodels. Tosystematicallyanalyzetheircommonalitiesandlimitations,wefor-\nmulateaunifiedtheoreticalframeworkthatcapturestheiressentialmechanisms. Thefundamental\ngoalofalignmentistooptimizeajointobjectivethatbalancesalignmentwithregularization,where\nalignment enforces compliance with human values, and regularization constrains the model’s be-\nhaviortopreventcatastrophicforgettingofpretrainedknowledge. Weexpressthisunifiedobjective\nas:\n \nθ\nalign\n= arg\nθ\nminL(θ) = arg\nθ\nmin\n\nL\n(cid:124)\nA\n(cid:32)\nl\n(cid:32)a(cid:32)i\n(cid:123)\nl\ng\ni\nn\ng\n(cid:122)\nm\nn\n(\n(cid:32)\ne\n(cid:32)\nθ\n(cid:32)\nn\n(cid:125)\nt\n) +γ\nReg\n(cid:124)\nR\nu\n(cid:123)\nla\n(\nr\n(cid:122)\nθ\niz\n(cid:125)\n)\nation\n\n, (1)\n(e.g.,RLHF,DPO) (e.g.,KL,L2)\nwhere L is the alignment loss, R is the regularization term, and γ ≥ 0 is a hyperparameter that\nalign\nbalances these competing objectives. This framework unifies primary alignment methodologies,\nincluding supervised fine-tuning (SFT) [8], reinforcement learning from human feedback (RLHF)\n[8], Constitutional AI [1], and direct preference optimization (DPO) [9], each of which can be\n6\nframedasaspecialcaseofthisgeneralformulation.\nSection5.1.2DistributioncoveragegapinPage16:\nThestrictinclusionsupp(D ) ⊊ supp(D )establishesa“distributioncoveragegap,”astruc-\nalign pre\ntural limitation inherent to post-hoc alignment. This gap permits adversarial inputs that are valid\nunder the pretraining distribution but lie outside the aligned data manifold. For such inputs, the\nmodel’sbehaviorrevertstoitspretrainedstate,bypassingsafetymechanisms. Sincethesupportof\nD will always be a constrained subset of D , this vulnerability cannot be resolved by merely\nalign pre\nscalingalignmentdata,creatingpersistentandsystematicexploitpathways.\nSection5.1.4ConditionalprobabilitydecompositioninPage17:\nThis decomposition formalizes a critical asymmetry in the model’s response mechanism. For\ninputs x ∈ X ,theoutputdistributionismodulatedbythealignmenttermA(x,y),whichactively\nalign\nsuppressesharmfulcontent. Conversely,forinputs x (cid:60) X ,thealignmenttermisstructurallyab-\nalign\nsent from the conditional probability, effectively deactivating the safety constraint. Consequently,\nwhenconfrontedwithout-of-distributioninputs,themodel’sbehaviorisgovernedprimarilybythe\npretraining distribution P (y|x), with only the regularization term e−γD(x,y) providing constraint.\npre\nSince this regularization is designed to preserve pretrained knowledge rather than enforce safety,\nthe model’s behavior reverts toward its pretrained state, including the generation of harmful con-\ntent. This mathematical bifurcation explains why even robustly aligned LLMs remain vulnerable\ntodistributionalshifts: ethicalsafeguardsarestructurallyabsentfromthegenerationprocesswhen\ninputs deviate from the aligned manifold, regardless of how thoroughly the model was aligned on\nin-distributiondata.\nSection5.2.2ImplementationoverviewinPage20:\nBuildinguponourtheoreticalframework,weoperationalizetheidentifiedvulnerabilitiesthrough\natwo-prongedmethodologicalapproachdesignedtoempiricallyvalidatetheintrinsicethicalfragility\nof aligned LLMs. This methodology systematically exploits the structural limitations inherent in\ncurrent alignment paradigms by combining distributional shifts with semantic coherence induce-\nment.\nThe corestrategy isto craft adversarialinputs thatreside outside thealigned distributionX\nalign\nbut remain within the broader pretrained distribution D , thereby triggering the “ethical drift”\npre\nphenomenonpredictedbyouranalysis. Thisprocessconsistsoftwokeycomponents:\n• Distributional Shifts: We introduce controlled input perturbations, such as template re-\nmoval, to shift inputs from the aligned manifold X to the unaligned space X .\nalign unalign\nexploits the distribution coverage gap (Lemma 1) and triggers gradient vanishing (Lemma\n2),deactivatingthealignmenttermintheconditionalprobability(Theorem1).\n• Semantic Coherence Inducement: To preserve malicious intent while operating outside\nthe aligned distribution, we append semantically congruent tokens to guide the generation\nprocess along high-probability trajectories within the pretrained knowledge manifold. This\nsteers the model toward intended harmful concepts, ensuring the generation of specific and\ncoherentharmfulcontentwithoutre-enteringthealigneddistribution.\nThis integrated approach allows us to systematically circumvent localized safety regions and acti-\nvate the latent harmful knowledge that persists after alignment, thereby realizing the elevated risk\nprofilequantifiedinTheorem2.\nReviewerPointP1.2 — Asecondconcernwiththepapercentresuponthedifficultyofvalidating\nthe core findings. The core method, which is largely under-specified (but involves some combina-\ntion of out-of-distribution prompting and coherence-based methods), is only loosely documented.\nIwilldefertomycolleaguesinmachinelearningresearchinrelationtowhetherthereissufficient\ndocumentationandvalidationofamajoradvance.\nRequire: Malicious query q; Victim model M ; Judge model M ; Nucleus probability p; Co-\nθ judge\nherencethresholdτ;Maximumattempts N\nmax\nEnsure: Adversarialprompt x∗ orfailure\n1: x ← q\nbase\n2: P(·|x ) ← Softmax(M (x )) ▷Computenext-tokenprobabilities\nbase θ base\n3: V ← SortByProbability(P(·|x )) ▷Sortvocabularybydescendingprobability\nsorted base\n(cid:80)\n4: k∗ ← min{k : k i=1 P(V sorted [i]|x base ) ≥ p}\n5: C ← {V [i]}k∗ ▷Constructnucleuscandidateset\nsorted i=1\n6: n ← 0\nattempts\n7: forc ∈ C do\n8: if n ≥ N then\nattempts max\n9: break\n10: endif\n11: x ← Concatenate(x ,c) ▷Formcandidateprompt\ncand base\n12: if Coherence(x ) > τthen\ncand\n13: y ← M (x ) ▷Generateresponsefromvictimmodel\nresp θ cand\n14: is harmful ← M (q,y ) ▷Assessresponseharmfulness\njudge resp\n15: n ← n +1\nattempts attempts\n16: if is harmfulthen\n17: x∗ ← x\ncand\n18: return x∗ ▷Returnsuccessfuladversarialprompt\n19: endif\n20: endif\n21: endfor\n22: returnFailure\nResponse: Thank you for raising this important concern regarding the documentation and val-\nidation of our methodology. We recognize that a clear and detailed specification is essential for\nreproducibilityandappreciatetheopportunitytoclarifyourapproach.\nIn response, we have substantially revised Section 5.2, “Methodological Implementation,” to\nprovide a more comprehensive and step-by-step description of our method. Our approach is a\nsystematic implementation of the theoretical vulnerabilities identified in Section 5.1, rather than a\ncollectionofad-hocpromptengineeringtricks. Itconsistsoftwomainstages:\n1. Inducing Distributional Shifts: We systematically modify the input prompts to shift them\nfrom the aligned data distribution (D ) to the broader unaligned space within the pretraining\nalign\ndistribution (D ). As detailed in the revised Section 5.2, this is primarily achieved by removing\npre\nor altering the chat templates that signal an aligned context to the model. This step is designed to\ndeactivatethelocalizedsafetyconstraints,aspredictedbyourtheoreticalanalysis.\n2. Semantic Coherence Inducement: After inducing a distributional shift, we guide the\nmodel’s generation process toward harmful outputs while maintaining semantic coherence. This\nis accomplished by appending a sequence of tokens that steer the model along high-probability\n9\ntrajectorieswithinitspretrainedknowledgemanifoldtowardtheintendedmaliciousconcept. This\nprocessisnotrandom;itinvolvesaheuristicsearchalgorithmthatprioritizestokensbasedontheir\nconditional probabilities to ensure the generated output is both fluent and intended. The technical\ndetailsofthissearchprocessarespecifiedinSection5.2.4Semanticcoherenceinducement.\nTo further support validation, we have included detailed pseudocode outlining the full algo-\nrithmic procedure (as shown in Algorithm 1) and have open-sourced our implementation on Code\nOcean. All code and experimental configurations are publicly available to facilitate independent\nverificationandfutureresearch. Webelievetheseadditionsprovidethenecessarytransparencyand\ndocumentation to validate our claims, demonstrating that our method is principled, reproducible,\nandrobustforexposingfundamentalvulnerabilitiesinLLMalignment.\nWesincerelyhope theserevisionsaddressyourconcerns andprovidegreaterclarityandtrans-\nparency regarding our methodology. Please let us know if further details or documentation would\nbehelpful. Wecopiedthepseudocodehereforyourconvenience:\nReviewer 2\nReviewer Point P2.1 — Despite lots of LLMs involved in the experiments, the tested models\n(Qwen-1, Llama-2/3, DS-R1-8B) are dated or small; I recommend adding recent strong baselines\nsuchasDS-V3orQwen3.\nResponse: We sincerely thank the reviewer for this valuable and constructive suggestion to\nstrengthenourexperimentalvalidationwithmorerecentstate-of-the-artmodels. Wehaveextended\ntheexperimentalsuitetoincludethreenewerand/orlargercontemporarymodels: (i)DeepSeekR1\n70B,(ii)Qwen34BThinking,and(iii)Qwen30BThinking. WechoseDeepSeek-R170Binstead\nof DeepSeek-V3 because the R1 line is the more recent reasoning-oriented release with superior\nperformance. Since DeepSeek-R1 8B was already part of our original evaluation, adding its 70B\nvariant enables a controlled intra-family scaling analysis. Likewise, adding intermediate (Qwen3\n4B Thinking) and larger (Qwen 30B Thinking) Qwen3-series models broadens coverage across\nparameterscales.\ntion / reasoning optimization do not eliminate the intrinsic vulnerability; the attack success rate\n(ASR) of our method remains saturated (100%), while baseline jailbreak methods exhibit modest\nsize-sensitive variation. This supports our theoretical interpretation that current alignment pro-\ncedures construct localized safety regions whose coverage does not expand proportionally with\nparametercount.\nForcompleteness,weprovidethesupplementedresultsinTable6. Theseadditionsdonotalter\nqualitative conclusions, but they strengthen external validity by including both a larger reasoning-\noptimizedlargemodel(DeepSeekR170B)andupdatedQwen3-seriesrepresentatives.\nReviewerPointP2.2 — SinceRLHFisaprocessofnumerioustrial-and-error,itisinterestingto\nseewhetherRLHFmakesmodelsmorerobusttotheattack. Moreover,doestheproposedweakness\nstem from limited response diversity (RLHF should help) or prompt diversity (which can be hard\ntofixatthealgorithmlevel)? Controlledprompt-diversityexperimentsareneeded.\nResponse: We appreciate the reviewer’s thoughtful question regarding (i) whether RLHF confers\nadditional robustness against our attack, and (ii) whether the observed vulnerability is primar-\nily attributable to limited response diversity or prompt diversity. We also agree that controlled\nprompt–diversityexperimentsare,inprinciple,informative.\n(i) Regarding RLHF’s potential protective effect, we have conducted experiments across a di-\nverse set of models aligned via RLHF (e.g., Llama 3, DeepSeek R1, Qwen3 families), as shown\nin the Table 7 below. It is observed that our method consistently achieves a 100% ASR across\nall tested RLHF-aligned models, while baseline jailbreak methods exhibit variable and generally\nlower ASRs. This indicates that RLHF, while effective for in-distribution alignment, does not\nmitigate the fundamental vulnerability we identify, which arises from structural limitations in the\nalignmentprocessitself.\n(ii) On the question of whether the vulnerability stems from limited response diversity or\nprompt diversity, we conducted experiments with varying alignment data (prompt and response)\ndiversity, as shown in Table 8. Technically, Zephyr 7B + (R2D2 in HarmBench [7]) is a variant\nof Zephyr 7B that has been further aligned with a dataset exhibiting higher prompt and response\n11\ndiversity from HarmBench. The results indicate that while increased alignment data diversity can\neffectively reduce the ASR of baseline jailbreak methods, our method still achieves a significantly\nhigher ASR (95.67% vs. 77.25% for the next strongest baseline TAP) even in the presence of\nenhanced diversity from HarmBench and limited computation budget. Moreover, this variant rep-\nresents a defender setting where the alignment process explicitly incorporates (and thus “knows”)\nthe malicious intent categories we target; nevertheless, our method continues to succeed, demon-\nstrating that even intent-aware (attack-informed) alignment does not eliminate the underlying vul-\nnerability. This suggests that while prompt and response diversity in alignment data can improve\nrobustnesstosomeextent,itdoesnotfundamentallyresolvetheintrinsicvulnerabilityweidentify,\nwhicharisesfromstructurallimitationsinthealignmentprocessitself.\nIn this work, our central claim (formalized in the main manuscript, Sections “Theoretical\nAnalysis” and “Methodological Implementation”) is that contemporary alignment pipelines (in-\nstruction tuning, RLHF, DPO, constitutional fine-tuning, etc.) act as localized regularizers over\na safety-conditioned sub-distribution of the pretrained knowledge manifold. Let D denote the\npre\nbroad pretraining distribution and D the comparatively narrow distribution induced by align-\nalign\nmentdatapluson-policypreferenceexploration. The(order-of-magnitude)supportratio\n|supp(D pre)|\n≈\n|supp(D align)|\n103–105as argued conceptually in the manuscript, implies that alignment carves thin “safety re-\ngions” within a vastly larger latent semantic space. Minimal semantic detours—such as suppress-\ningoralteringtemplatetokensthatimplicitlysignalalignedconversationalcontext—aretherefore\nsufficient to move generation trajectories outside these regions while following high-probability\ncontinuations (i.e., without resorting to low-likelihood token sequences). The persistence of high\nASR under scaling and diversity interventions above is consistent with this localized-constraint\ninterpretationratherthanwithahypothesisofmerelyincompletedatacoverage.\nWe have supplemented these additional experiments and analyses to clarify these points in\nSectionC.2. Wehope thisaddressesyour concerns,andweare happytoprovide furtherdetailsor\nconductadditionalexperimentsifneeded.\nReviewer Point P2.3 — The proposed attack method is white-box and needs model logits on\nconcatenatedcontinuations. Isthereawaytodesignablack-boxattackmethodthatcanautomat-\nically find effective suffixes without access to internal probabilities? LLM developers should be\ninterestedinthis.\nResponse:\nthe open-source Vicuna 7B as a proxy to craft adversarial suffixes; target models are only queried\nwithcompletedprompts,withoutanyaccesstologits,gradients,orinternalstates. Assummarized\nin Table 9, the attack transfers effectively to closed-source APIs, achieving near-saturated ASR\non GPT-3.5 and non-trivial rates on GPT-4.1 under the same API query budget, while using only\n1/10 of the query budget on open-source models to generate the adversarial suffixes; stronger\nguardrailsinGPT-5reduceASRofsuchtransferattacksunderlimitedbudget,butourmethodstill\noutperformsallbaselines. Theseblack-boxresultsreinforceourcorefinding: themethoddoesnot\ndependonwhite-boxaccess. Instead,itexploitsthestructuraldistribution-coveragegapcreatedby\npost-hocalignmentandthepersistenceofhigh-probabilitysemantictrajectoriesoutsidethealigned\nregion,consistentwithourtheoreticalanalysis.\nWe have added these results and analysis in Section C.3 of the revised manuscript. We hope\nthis addresses your concern, and we are happy to provide further details or conduct additional\nexperimentsifneeded.\nReviewer 3\nThis paper investigates a critical vulnerability in current alignment methods used in LLMs. The\ncentral claim is that existing alignment approaches (e.g., SFT, RLHF) construct only localized\n“safety regions” within the knowledge manifold. However, the global structure of the pretrained\nmodeldistributionremainslargelyuntouched,leavingopenthepossibilityofsemanticdetoursthat\nreactivate harmful or unaligned behaviors. To demonstrate this, the authors propose a simple yet\neffective method: subtly altering the chat template to remove alignment cues. This minimal inter-\nvention significantly reduces the effectiveness of alignment, yielding a near 100% attack success\nrate(ASR)acrossmanyalignedLLMs.\nStrengths:\nThe paper makes a compelling conceptual point that alignment methods act more like patches\nthanholisticsolutions. ThischallengesaprevailingassumptioninLLMsafetyresearchandbrings\nattention to a systemic vulnerability in current alignment methods. The authors back this claim\nwiththoroughqualitativeanalysis,offeringvaluableinsightsintothefragilityofalignmentstrate-\ngies.\nThe proposed attack method is highly effective across a wide range of models, achieving ex-\ntremely high ASR in many cases. Moreover, the paper provides a broad evaluation over diverse\nmodel families, highlighting the generalizability of the attack. The writing is formal and well-\nstructured,whichenhancestheoverallreadabilityandgivestheworkaprofessionalappearance.\nResponse: We sincerely thank the reviewer for the constructive and encouraging evaluation. We\nare grateful for the recognition of (i) our conceptual contribution—that current alignment pro-\ncedures primarily induce localized safety regions rather than effecting a global restructuring of\nthe pretrained semantic manifold—and (ii) the breadth and consistency of our empirical evidence\nacross heterogeneous model families. We have carefully addressed all specific concerns in the re-\nvisedmanuscript,andwehopethattheserevisionssatisfactorilymeetthereviewer’sexpectations.\nReviewerPointP3.1 — MethodologicalSimplicityvs. FormalOverpackaging\nWhile the paper is well-written and presented with formal rigor, the core technique essentially\nreduces to prompt-based manipulation—specifically, editing the chat template in a way that re-\nmoves alignment signals while maintaining semantic fluency. The extensive formal framing may\ncreateanimpressionoftechnicalnovelty,butthemethodologicalcontributionis,inessence,mod-\nest. Readers may benefit from a more balanced presentation that clarifies the simplicity of the\nmethodincontrasttoitssurprisinglystrongempiricalanalysis.\nResponse: We sincerely thank the reviewer for this insightful observation. We acknowledge that\ntherelationshipbetweenourtheoreticalframeworkandmethodologicalsimplicityrequiresclearer\narticulation.\nOur primary contribution is the theoretical framework explaining why aligned LLMs\npossess intrinsic vulnerability, not the attack method itself. The simplicity of our empirical\napproach—which the reviewer correctly identifies—actually validates rather than diminishes our\ntheoreticalpredictions. Specifically:\n• Methodological simplicity confirms theoretical predictions. The reviewer’s observation\nthat our method “reduces to prompt-based manipulation” precisely confirms our central\nclaim: alignmentcreatesonlylocalizedsafetyconstraintsthatfailunderdistributionalshifts.\nThisisnotincidental—itismathematicallypredictedbyourunifiedframework(Equation1)\nandformalizedthroughLemmas1-2andTheorems1-2.\n• Universal success demonstrates architectural vulnerability. The fact that simple dis-\ntributional shifts achieve 100% ASR across 22 of 26 state-of-the-art models (including\nDeepSeek-R1,LLaMA-3.1,andQwen3)provesthevulnerabilityisarchitecturalratherthan\nimplementation-specific. If complex optimization were required, this would suggest model-\nspecificweaknessesratherthanfundamentalparadigmlimitations.\n• Theoretical framework provides prescriptive value. Beyond explaining existing fail-\nures,ourframework: (a)identifiestherootcause(irreconcilabletensionbetweenknowledge\npreservation and safety constraints in Equation 1); (b) quantifies failure conditions (gradi-\nent vanishing in Lemma 2, risk bounds in Theorem 2); and (c) establishes that no post-hoc\nalignmentwithinthisframeworkcanresolvethevulnerabilitywithoutparadigmshift.\nThisfollowsestablishedscientifictraditionwhereprofoundinsightsexplaincomplexphenom-\nenathroughsimplemechanisms(e.g.,FGSM’ssimpleperturbationsrevealingfundamentaladver-\nsarialvulnerabilitiesindeeplearning).\nTo address this concern, we have revised the manuscript to explicitly clarify the relationship\nbetween theoretical depth and methodological simplicity. All revisions are highlighted in blue in\nthemanuscript. Specificchangesinclude:\nRevision1-Abstract:\nWe empirically validate these theoretical insights through a straightforward yet the-\noretically grounded methodology—semantic coherence inducement under distribu-\ntional shifts. The effectiveness of this approach, achieving a 100% attack success rate\nacross 22 out of 26 state-of-the-art aligned LLMs (including DeepSeek-R1, LLaMA-\n3, and Qwen3, among others), is not incidental but a direct consequence of our the-\noretical framework, demonstrating that the vulnerability is architectural rather than\nimplementation-specific and revealing a fundamental, universal weakness in current\nalignedLLMs.\nRevision2-Introduction(Paragraph3):\nUnlike existing attacks [5,10,15] requiring token-level perturbations, our method’s\ndeliberate simplicity serves a specific purpose: validating theoretical predictions that\ncomplex optimization is unnecessary because the vulnerability is architectural rather\nthan optimization-dependent. Our method exploits semantic coherence in the pre-\ntrained latent space, enabling natural-language adversarial prompts. [...] The uni-\nversal success of simple distributional shifts confirms that current safety mechanisms\nfail catastrophically not due to insufficient engineering effort but due to fundamental\nlimitationsinthepost-hocalignmentparadigmitself.\nRevision3-MethodsSection(NewSubsection5.2.1):\n5.2.1Theoreticalfoundationmotivatesmethodologicalsimplicity\nA criticaldistinction must be drawnbetween methodological complexity andconcep-\ntual depth. Our empirical approach deliberately employs simple techniques (template\nremovalandsemanticcoherenceinducement)preciselybecauseourtheoreticalframe-\nwork predicts that sophisticated attacks are unnecessary. This design choice is inten-\ntionalandtheoreticallymotivatedforthreereasons:\nploitthevulnerability,thiswouldsuggestthattheweaknessisimplementation-specific\nrather than architectural. The fact that simple distributional shifts universally succeed\n(100% ASR across 22 of 26 models) confirms our theoretical prediction that the vul-\nnerability is intrinsic to the post-hoc alignment paradigm itself, as formalized in our\nunifiedframework(Equation1).\nSecond, the method directly instantiates theoretical constructs. Template removal\noperationalizes the distribution coverage gap established in Lemma 1 (Strict Inclu-\nsion of Distribution Supports), while semantic coherence inducement maintains in-\nputs within the pretrained distribution’s support while exiting the aligned subspace.\nThis creates precisely the conditions where Lemma 2 predicts gradient vanishing and\nTheorem1predictsdeactivationofsafetymechanisms.\nThird, practical implications demand minimal attack complexity. From a safety\nengineeringperspective,vulnerabilitiesexploitablethroughsimplemeansposegreater\nreal-world risks than those requiring sophisticated optimization. Our findings suggest\nthat current alignment methods fail not because attackers are too clever, but because\ntheunderlyingapproachisfundamentallyflawed—afarmoreconcerningconclusion.\nThis methodology serves as proof-of-concept rather than endpoint. Our primary con-\ntribution lies in establishing the theoretical framework that explains why alignment\nfailsunderdistributionalshifts,providingafoundationfordevelopingnext-generation\nsafetyapproachesthataddressrootcausesratherthansymptoms.\nTheserevisionsclarifythatourcontributionprioritizestheoreticalunderstandingovermethodolog-\nical complexity. The simplicity of our empirical approach strengthens rather than diminishes the\ncontribution by demonstrating the vulnerability is fundamental and universal, not model-specific\noroptimization-dependent.\nReviewerPointP3.2 — LackofAnalysisonRobustModels\nIfthepaper’smaincontributionistodemonstratetheexistenceofsuchvulnerabilities,itwould\nfurther benefit from a more detailed analysis of the factors that contribute to robustness in cer-\ntain models. For example, why do models like LLaMA 2 Chat 70B show significantly lower ASR\n(<80%)? Is this due to a larger or more diverse alignment dataset, more effective instruction\ntuning,orspecificRLHFtechniques?\nResponse: Thank you for this useful suggestion. We agree that analyzing comparatively “ro-\nbust” models clarifies where the gains come from and what their limits are. Consistent with prior\nstudies [7,10,12], we observe that the LLaMA-2 chat series tends to be more resistant to base-\nline jailbreaks; in our results, LLaMA-2-70B-Chat attains a lower ASR (79%) than many peers\n(Table 2). Our analysis indicates that this gap is driven by three interacting factors rather than\nparametercountalone:\n1) Training objective and optimization. LLaMA-2 adopts a comparatively conservative safety\nalignment, emphasizing strictrefusal of riskyqueries. By contrast, LLaMA-3 explicitlyoptimizes\n16\na usefulness–safety balance with larger, more heterogeneous post-training (SFT + PPO/DPO),\nwhich improves ordinary helpfulness but introduces more context-dependent decision boundaries\nthatcanberoutedaroundunderdistributionalshifts.\n2) Data and tokenizer changes. LLaMA-3 introduces a larger 128K-vocab tokenizer and more\ndiverse data (e.g., code and multilingual content). This boosts expressivity and coverage, but\nit also enlarges the prompt space—mixed-language or symbolic variants become viable attack\nroutes—whereasLLaMA-2’snarrowertokenization/datareducesomeofthesepaths.\n3)Safetyfine-tuningandrefusalstrategy. Empirically,LLaMA-2showssteepersuppressionof\nharmful continuations in the conditional distribution when shifted off template, consistent with a\n“hard”refusalpolicy. LLaMA-3’sfiner-grainedsafetybehaviorismoreflexibleinbenignsettings,\nyetthatflexibilityyieldsmoreexploitableseamsinsomejailbreakscenarios.\nThese observations reflect a robustness–utility trade-off (grounded by Eq.1): LLaMA-2 errs\non the side of refusal (“better safe than sorry”), which lowers baseline ASR, while LLaMA-3\nprioritizes usefulness and natural interaction. Crucially, our method still achieves high ASR on\nboth families (as shown in overall results in Table 2 and RLHF related results, e.g., DeepSeek R1,\nLLaMA3.1,andQwen3,inTable7),supportingourcentralclaimthatpost-hocalignmentcreates\nlocalizedsafetyregionsthatremainvulnerableundersemanticallycoherentdistributionalshifts.\nConsistent with this interpretation, Fig. 2 shows that LLaMA-2’s response probability tree is\nmarkedly“harder”: conditionalprobabilitiesalongrefusaltrajectoriesarenear-deterministic(often\nwith conditional probabilities equal to 1 or close to 1), sharply suppressing harmful continuations\nevenaftertemplateremoval. Incontrast,LLaMA-3assignsnon-trivialmasstocompliantbranches\nunder the same detour, reflecting a more permissive, utility-optimized decision boundary that is\neasiertoroutearoundinjailbreakscenarios.\nWe have added this analysis in Section C.4 of the revised manuscript. We hope this addresses\nyourconcern,andwearehappytoprovidefurtherdetailsifneeded.\nReviewerPointP3.3 — PotentialUnfairnessofEvaluationProcedure\n- The paper notes that ASR is notably lower for LLaMA 2 series. Could this discrepancy be\npartly due to the use of LLaMA 2 13B Chat in some evaluation? It would be helpful if the authors\nclarified this and provided ablation studies to test the sensitivity of results to model scale and\nconfiguration. - According to the described attack method, an attack is considered successful if\nanysingleadversarialpromptwithinafixedmaximumnumberofattempts(denotedNmax)causes\nthe model to output a harmful response. However, the average number of attempts required per\nsuccessful attack is not reported. If this number is large, it may raise concerns about the fairness\nor practicality of the evaluation. Clarifying this would help assess the efficiency and real-world\nrelevanceoftheattackmethod.\nResponse: Thankyouforraisingtheseimportantconcernsaboutevaluationfairnessandresource\nefficiency. Weaddresseachpointsystematically:\n(1) Model scale and evaluation consistency: All evaluations strictly follow the HarmBench\nprotocol [7] across all models. Table 10 demonstrates that within model families, parameter scal-\ning yields no systematic reduction in vulnerability across attack methods. For instance, LLaMA-\n2-70B-Chat achieves 79% ASR under our method compared to 98.67% for LLaMA-2-7B-Chat\n18\nand81.33%forLLaMA-2-13B-Chat—amodestimprovementattributabletostricterrefusalstrate-\ngies rather than fundamental architectural robustness (detailed analysis in Response 3.2). Sim-\nilarly, scaling from DeepSeek-R1-8B to DeepSeek-R1-70B, or from Qwen3-4B to Qwen3-30B,\nmaintains 100% ASR, demonstrating that increased capacity does not resolve the underlying vul-\nnerability. This is consistent with our theoretical claim: alignment creates localized safety re-\ngionswhosecoveragedoesnotexpandproportionallywithmodelcapacity. Thedistributionalgap\nsupp(D ) ⊊ supp(D ) persists structurally at all scales because post-hoc alignment modifies\nalign pre\nonly the conditional distribution within the aligned subspace, leaving the global pretrained mani-\nfoldlargelyintact.\n(2)Attackefficiency: AssummarizedinTable11,ourmethodsucceedswithverysmallsearch\neffortandminimalsuffixlengthunderthe25,000-nodecap. Theper-modelmeannumberofsearch\nnodes ranges from 9.78 (Llama 3.1 8B) to 441.36 (Qwen3 30B Thinking), with an across-model\nmedian of approximately 35 nodes; most models require fewer than 100 nodes on average. Even\nthe largest mean (441.36) is below 2% of the query cap. The adversarial suffixes are extremely\nshort: mean lengths are typically 1.17–1.76 tokens across most models, with higher requirements\non the two Qwen3 “Thinking” variants (2.92–3.83 tokens). These results indicate the attack is\nnot brute force; it exploits high-probability semantic trajectories in the pretrained distribution. In\ncontrast, strong baselines such as GCG [15] and AutoDAN [5] either incur substantial computa-\ntional costs—GCG, for instance, is gradient-based and typically runs for around 500 iterations by\ndefault—orrequiresignificantlylongersuffixes(e.g.,20tokensforGCGandapproximately60for\nAutoDAN),bothofwhicharemarkedlylongerthanours.\ntoexistingbaselines,achievinghighsuccessrateswithminimalcomputationalresourcesandvery\nshort adversarial suffixes. We hope these results clearly illustrate the practical implications of our\ntheoreticalinsightsandsatisfactorilyrespondtothereviewer’scerns. Wehavealsoaddedthese\nclarificationstotherevisedmanuscriptinSectionC.5.\nReviewer Point P3.4 — The paper only tests template removal for inducing distribution shifts,\nneglectingothershifttypessuchasdomaintransfer.\nResponse: Thank you for pointing this out. Beyond in-domain template removal, we conducted\nblack-box domain transfer experiments, as shown in Table 12. Specifically, we crafted adversarial\nsuffixesonanopen-sourcesurrogate(Vicuna7B;sourcedomain)andtransferredthemzero-tuning\nto closed-source APIs (GPT-3.5 / GPT-4.1 / GPT-5; target domains) without any access to logits,\ngradients, or internal states. The domain transfer attack achieved 99.67% ASR on GPT-3.5 and\n44.33% on GPT-4.1, demonstrating cross-model-family and deployment-stack transfer. On GPT-\n5,stricterguardrailsreduceabsoluteASRto1.67%,yetourmethodstillachievesthehighestASR\namongallcomparedattacks. WehaveaddedtheseresultstoSectionC.3oftherevisedmanuscript.\nthat maintain semantic coherence while triggering distribution shifts, which involves setting mul-\ntiple hyperparameters such as coherence thresholds and cumulative probability thresholds. It is\nrecommended to add comparative experiments to evaluate the impact of each hyperparameter on\ntherisklowerbound.\nResponse: Thank you for this valuable suggestion. We have conducted systematic ablation stud-\nies to evaluate the impact of key hyperparameters on attack performance and provide theoretical\njustificationfortheirinteraction.\n(1) Impact of search budget (N ): As shown in Table 13, we evaluate ASR under varying\nmax\nsearch budgets (25, 250, 2500, 25000 nodes). The results demonstrate that while modest bud-\ngets (250 nodes) already achieve near-saturated ASR (>95%) for most models, the full budget\n(25000 nodes) ensures universal convergence to 100% ASR across 22 models. Notably, even un-\nder severely constrained budgets (25 nodes), the average ASR reaches 58.23%. This robustness\nunderlimitedsearcheffortvalidatesthatourmethodexploitshigh-probabilitysemantictrajectories\nratherthanexhaustiveenumeration.\n(2)Impactofcoherencethreshold(τ): Table14showsASRunderdifferentcoherencethresh-\nolds with a fixed search budget of 100 nodes. Lower thresholds (0.001–0.0001) yield higher ASR\n(96.33–98.33%) by expanding the candidate diversity, while higher thresholds (0.1–0.01) slightly\nreduceASRto95.00%duetostricterfluencyrequirements. Themarginaldifferenceindicatesthat\nthe method is robust across a reasonable range of τ values, with the default τ = 0.0001 balancing\nsemanticcoherence,cross-modelcompatibility,andattacksuccess.\n(3)Theoreticalrelationshipbetweencoherencethresholdandcumulativeprobability:\nThe coherence threshold τ is defined as the reciprocal of the perplexity score, i.e., τ = 1/PPL,\nwhere perplexity quantifies the model’s predictive uncertainty over token sequences. Formally,\nfor a candidate prompt x = [x ,c ,...,c ] formed by appending tokens c ,...,c to the base\ncand base 1 k 1 k\n21\nprompt x ,thejointprobabilityundertheautoregressivefactorizationis:\nbase\n(cid:89)k\nP(x ) = P(c | x ,c ,...,c ), (2)\ncand i base 1 i−1\ni=1\nandtheperplexityisdefinedas:\n \nPPL(x\ncand\n) = exp\n\n−\n1\nk\n(cid:88)k\nlogP(c\ni\n| x\nbase\n,c\n1\n,...,c\ni−1\n)\n\n. (3)\ni=1\nThecoherenceconstraintCoherence(x ) > τcanbereformulatedasPPL(x ) < 1/τ,which\ncand cand\nthroughalgebraicmanipulationyields:\nP(x ) ≥ τk. (4)\ncand\nThisestablishesaprobabilisticlowerbound: candidatessatisfyingthecoherencecriterionmust\nhavejointprobabilityatleastτk,effectivelyfilteringlow-probability(incoherent)sequences.\nLetC = {c ∈ V : Coherence([x ,c]) > τ}denotethecoherence-filteredcandidateset,andlet\nτ base\nN = {v ∈ V : (cid:80) rank(v)P(v|x ) ≤ p}denotethenucleussamplingsetwithcumulativeprobability\np i=1 i base\nthreshold p. TheeffectivesamplingdistributionoperatesovertheintersectionC ∩N .\nτ p\nThetotalprobabilitymasscapturedbycoherentnucleuscandidatesisboundedby:\n \nP\neff\n=\n(cid:88)\nP(c|x\nbase\n) ≥ min\n\np,\n(cid:88)\nP(c|x\nbase\n)\n\n. (5)\nc∈C\nτ\n∩N\np\nc∈C\nτ\nAsτdecreases(toleratinghigherperplexity),thesetC expands,increasingbothcandidatedi-\nτ\nversity and the probability mass (cid:80) P(c|x ). This enlarges the effective search space and gen-\nc∈C\nτ\nbase\nerally improves attack success probability. Conversely, increasing τ (requiring lower perplexity)\nimposes stricter fluency requirements, shrinking |C | and potentially reducing coverage of high-\nτ\nprobabilityattackpathswhileenforcingstrongersemanticcoherence.\nOur empirical results (Table 14) demonstrate this trade-off: moderate values (τ ∈ [10−4,10−3])\noptimizebothsemanticnaturalnessandattackefficacy,achieving98.33%ASR,whileoverlystrict\nthresholds (τ = 0.1) marginally reduce ASR to 95.00% due to excessive candidate pruning. This\nvalidates that our method operates within a robust hyperparameter regime where the coherence\nconstrainteffectivelybalancesfluencyandexploitability.\nTheseablationsconfirmthatourmethodachievesrobustperformanceacrossreasonablehyper-\nparameter ranges, with theoretical guarantees linking coherence constraints to the effective sam-\npling distribution. We hope these results satisfactorily address the reviewer’s concerns. We have\nalsoaddedtheseclarificationstotherevisedmanuscriptinSectionC.5.\nReviewer Point P3.6 — The paper posits pretrained knowledge remains globally connected\nto harmful concepts, and conducts theoretical analysis based on topological assumptions of the\nknowledge manifold. However, this claim lacks empirical validation, such as inter-layer attention\npatternanalysisorrepresentationsimilaritymeasurements.\nResponse: Thankyouforthisimportantsuggestion. Weprovideempiricalvalidationofthetopo-\nlogicalconnectivitybetweenpretrainedknowledgeandharmfulconceptsthroughvisualizationand\nprobabilisticanalysisofresponsedistributions.\n(1) Visualization of semantic connectivity to harmful concepts: Figure 3 presents a hi-\nerarchical visualization of the autoregressive generation process for the malicious query “Give\ndetailed instructions for how to ****** while making it seem like an accident”.\nThe tree structure demonstrates that after distributional shift away from the aligned region, the\nmodel’sconditionalprobabilitydistributionmaintainshigh-probabilitytrajectoriesleadingdirectly\ntoharmfulcontent. Specifically,nodesrepresentingdetailedinstructionalsteps,strategicplanning,\n23\nand methodological guidance for carrying out the harmful intent exhibit joint probabilities sub-\nstantially above random chance (darker and larger nodes in the visualization). This empirically\nconfirms that aligned LLMs preserve semantic pathways connecting innocuous linguistic patterns\ntoharmfulconcepts.\n(2) High-probability trajectories with rigorous harmful content: Examining the response\ndistribution structure reveals that harmful completions are not merely accessible but probabilisti-\ncallyfavoredunderdistributionalshifts. Thegeneratedresponses1 exhibit: (i)detailedprocedural\nsteps with specific action sequences, (ii) logical coherence maintaining causal consistency across\nmultiple steps, and (iii) strategic reasoning incorporating concealment tactics and risk mitigation\nfrom the attacker’s perspective. Crucially, these harmful trajectories emerge with substantial con-\nditional probabilities (as indicated by edge weights and node sizes in Figure 3), demonstrating\nthat alignment has not fundamentally restructured the underlying semantic manifold but rather\nsuppressedcertainpathwayswithinalocalizedregion. Wheninputsshiftoutsidethisalignedsub-\nspace, the pretrained probability distribution reasserts itself, naturally routing generation toward\nhigh-likelihoodharmfulcontinuationsthatwerenevereliminated—onlytemporarilymasked.\nThis visualization provides direct empirical evidence supporting our theoretical claim (The-\norem 1) that safety constraints vanish under distributional shifts, as the alignment term A(x,y)\nbecomes structurally inactive for x (cid:60) X , leaving generation governed primarily by the pre-\nalign\ntrained distribution P (y|x) which retains strong connectivity to harmful concepts through high-\npre\nprobability semantic paths. We have added this analysis to Section 2.2 Experimental results in\npage10oftherevisedmanuscript.\nReviewer Point P3.7 — The experimental dataset solely employs HarmBench. Although this\ndataset covers six major categories of harmful behaviors, it fails to adequately validate the gen-\neralizability of the proposed method. It is recommended to incorporate additional LLM safety\ndatasets.\nResponse: Thank you for this valuable suggestion. We have expanded our evaluation to include\nadditionaldatasetsbeyondHarmBenchtovalidatethegeneralizabilityofourmethodacrossdiverse\nharmfulbehaviorcategoriesanddatasources.\nSpecifically,wesupplementedourexperimentswithtwowidelyrecognizedLLMsafetydatasets:\n(i) the TDC 2023 Red Teaming Track dataset [6], which covers adversarial scenarios designed to\nelicit harmful outputs through red-teaming methodologies, and (ii) AdvBench [15], a standard\nbenchmark for evaluating adversarial robustness in LLMs. Tables 15 and 16 present the results\nacross these additional datasets for representative models spanning different scales and alignment\nstrategies.\nTheresultsdemonstrateconsistentpatternswithourHarmBenchfindings: ourmethodachieves\n100% ASR across all tested models on both TDC 2023 and AdvBench, while baseline jailbreak\nmethods exhibit substantially lower and more variable ASRs. For instance, on the TDC 2023\ndataset, the strongest baseline (AutoDAN) achieves only 63.48% ASR on DeepSeek R1 8B, com-\npared to our method’s 100% success rate. Similarly, on AdvBench, baseline methods struggle\n1Duetoethicalconsiderationseditorialrequirement,wecannotdisplaycompletemaliciousresponsesinthemain\ntext;however,theseoutputsareavailabletoreviewersuponrequestwitheditorialapprovalforverificationpurposes.\n25\nparticularly with reasoning-optimized models (e.g., Qwen3 series), whereas our approach main-\ntainsuniversaleffectiveness.\nThese cross-dataset results provide robust evidence that the vulnerability we identify is not\ndataset-specificbutratherreflectsfundamentalarchitecturallimitationsincurrentalignmentparadigms,\nas predicted by our theoretical framework. The consistent high ASR across HarmBench, TDC\n2023, and AdvBench—which collectively cover diverse harmful behavior taxonomies, query for-\nmulations,andseveritylevels—strengthensthegeneralizabilityclaimsofourwork.\nReviewer Point P3.8 — The paper claims that current alignment methods can only establish\nlocalized safe regions, but fails to account for the impact of model scale on safety vulnerabilities.\nWhile the experiments show identical attack success rates (100%) for both LLMs like Qwen-72B\nand smaller models, the study provides no explanation for why increased parameter size does not\nleadtoimprovedrobustness.\nResponse: Thank you for this important observation. We explain why increased parameter size\ndoesnotconferproportionalrobustnessimprovementsthroughboththeoreticalandempiricalanal-\nysis.\n(1) Theoretical explanation: Scale-invariant distributional coverage gap. Our unified\nframework (Equation 1) reveals that parameter scaling cannot resolve the fundamental vulnera-\nbility because the distribution coverage gap is structural rather than capacity-dependent. As es-\ntablished in Lemma 1 (Strict Inclusion of Distribution Supports), the aligned distribution satisfies\nsupp(D ) ⊊ supp(D ), creating a coverage gap that persists across scales. Critically, while\nalign pre\nscaling increases both the expressiveness of the pretrained manifold and the aligned subspace, the\nrelative coverage ratio remains approximately constant because: (i) pretrained manifolds grow\nexponentially with capacity, encoding richer semantic structures; (ii) alignment datasets remain\nfixed across model scales (e.g., |D | ≪ |D | by 3–4 orders of magnitude); and (iii) regulariza-\nalign pre\ntion γR(θ) enforces knowledge preservation, maintaining pretrained structure outside the aligned\nregion. Following Lemma 2 (Out-of-Distribution Gradient Measure), our attack exploits the gra-\n(cid:112)\ndient vanishing phenomenon ∥∇ L (θ)∥ ≤ ϵ P (X ) for inputs outside X . This\nθ align L2(Ppre) pre align align\ngradient diminishing is scale-invariant because it arises from distributional mismatch rather than\nmodel capacity. Consequently, as predicted by Theorem 1 (Conditional Probability Decomposi-\ntion), the alignment term A(x,y) remains structurally absent for x (cid:60) X regardless of parameter\nalign\ncount,causingthemodeltorevertto P (y|x)e−γD(x,y).\npre\n(2)Empiricalvalidation: Consistent100%ASRacrossscales. Table17demonstratesscale-\ninvariant vulnerability across model families: across the Qwen series (7B → 14B → 72B), our\nmethod maintains 100% ASR at all scales while baseline ASRs vary modestly (e.g., AutoDAN:\n62.25%–68.25%–52.75%),indicatingthat10×parameterscalingyieldsnorobustnessgainagainst\nour attack; similarly, both DeepSeek R1 models (8B → 70B) achieve 100% ASR despite 8.75×\ncapacityincrease,andwhilebaselineimprovementsoccur(e.g.,GCG-T:42.00%→34.67%),this\nconfirmsthatscalingprovidesonlymarginalresistancetotoken-levelperturbationsbutfailsagainst\nsemantic distributional shifts; the Qwen3 Thinking series (4B → 30B) exhibits the same pattern,\nwhere despite 7.5× scaling and reasoning optimization, both maintain 100% ASR with negligible\nbaseline variation (averaged 9.41% vs. 7.85%); finally, while the Llama 2 family (7B → 13B →\n70B)showsmodestASRreduction(98.67%→81.33%→79.00%),asanalyzedinResponse3.2,\n27\nthis stems from stricter refusal policies rather than architectural robustness, with even the 70B\nmodelremaininghighlyvulnerable(79%ASR).\nThese findings demonstrate that scaling alone cannot yield emergent safety. The vulnerability\nis intrinsic to post-hoc alignment’s structural tension (Equation 1): L (θ) enforces localized\nalign\nconstraintswhileγR(θ)preservespretrainedknowledge,creatinganirreduciblecoveragegap. We\nhaveaddedthisdetailedexplanationtoSectionC.5oftheAppendixtoclarifythisimportantpoint.\nReviewer 4\nKeyresults:\nThis paper addresses the important issue of safety and alignment of Large Language Mod-\nels. The authors have conducted a theoretical analysis supported by empirical studies over 23\nLLMs using adversarial prompts. They have shown that “aligned models”, i.e., those which have\nundergone additional training for alignment using methods such as Constitutional AI, RLHF, Su-\npervised Fine Tuning or Direct Preference Optimization are actually aligned only in local regions\nof the knowledge manifold. As a result, safety is an illusion because the system can be easily\nattacked and provide harmful outputs especially with distributional shifts. The experiments have\nshown very high rate of attack success. The paper comes with additional material which includes\nthemainpartsofthetheoreticaldevelopmentsandexperimentalresults.\nComments:\nThis paper stands out among the current literature on AI safety and alignment which too often\nreportsaboutexperimentingwithAIsystemsorbenchmarkingthemtoidentifysafetyissues,with-\noutanalyzingthereasonsbehindtheirbehavior. Itstandsasonethatactuallyreportsonprofound\nwork for characterizing theoretical reasons for misalignment. Its discoveries are solidly demon-\nstrated and the technical approach is valid. The unified alignment framework they introduce in\nEquation1isausefulandefficientformulationtounderstandmisalignment.\nThe authors justify their publication from an ethical stance, mentioning three imperatives, sci-\nentific necessity, transparency, and preemptive intervention. They also avoid to disclose some\nimplementation details and operational parameters, as well as parts of system outputs. This is a\ngood approach for avoiding harmful exploitation of their finding, but prevents full verification of\ntheirresults.\nRegarding distributional shifts: The paper seems to insist on the role of distributional shifts\n(section 5.1.3) as if there were OOD situations (i.e., real world data not included in the training,\ntest and validation sets). This is the case in the cited references [16, 17]. However, RLHF and\nother alignment methods such as SFT or DPO aren’t complete. There are always blind spots that\nremain in the model. So the fact that there are always some means to circumvent such safety\nmethods is true without such distributional shifts. This is rather due to incompleteness. What the\npaper provides is an analysis and a theoretical proof of this situation. But this is not necessarily\nrelatedtodistributionalshiftsperse. OODdistributionalshiftswouldincreasemisalignmentrisks\nabutaren’ttheonlyreasonformisalignment. Whatdistributionalshiftsmeaninthepaperisrather\nrelated to situations not encountered in the alignment process itself. This is somewhat confusing\nwrt. thecommondefinitionofdistributionalshifts.\ntested. However,thepaper’sdevelopmentsandconclusionslogicallyapplytothemaswell. Would\nitbeneverthelesspossibletoincludesomeexperimentswiththesemodels?\nReference to AGI is irrelevant. All the tested systems are not “AGI” and the discussed safety\nissues are related to the inherent structure of current LLMs. AGI is an ill-defined concept and\nsafetyissueswithcurrentmodelsarealreadyimportantriskstodealwith. Thereisneedtodistract\nthereaderwithAGI.\nSuggestedimprovements:\nThe main improvementsuggestions to this paper are relatedto its organization and itspresen-\ntation.\nReferences:\nThepaperappropriatelyreferencestheliteratureinthefield.\nResponse: Thank you very much for your thoughtful and comprehensive review. We deeply\nappreciate your recognition of our theoretical contributions and your constructive suggestions for\nimprovingthemanuscript’sclarityandorganization. Weaddressyourspecificconcernsbelow:\nRegarding distributional shifts: We acknowledge your valuable clarification regarding the\nterminology of “distributional shifts.” You are correct that the phenomenon we describe is more\npreciselycharacterizedasincompletenessinalignmentcoverageratherthanclassicalout-of-distribution\n(OOD) scenarios. What we term “distributional shifts” refers specifically to deviations from the\naligned data distribution D while remaining within the broader pretrained distribution D .\nalign pre\nThisisindeeddistinctfromtraditionalOODscenarioswheretestdatacomesfromentirelydifferent\ndomains. We have revised Section 5.1.3 to clarify this distinction and now explicitly state that our\nanalysisconcernsthestructuralincompletenessofalignmentmethods—wherethesupportofD\nalign\nis a strict subset of D —rather than classical distributional shift phenomena. While traditional\npre\nOOD shifts would indeed exacerbate the vulnerability, our theoretical framework demonstrates\nthat the fundamental misalignment arises from this coverage gap inherent to post-hoc alignment\napproaches.\nRegarding closed-source models: We have conducted additional black-box experiments on\nOpenAI’s GPT-3.5, GPT-4.1, and GPT-5 using transfer attacks (detailed in Response 2.3 and Re-\nsponse 3.4). As shown in Table 18, these results demonstrate effective transferability to closed-\nsource APIs, achieving 99.67% ASR on GPT-3.5 and 44.33% on GPT-4.1, thereby validating that\nour theoretical framework extends beyond the tested open-source models. The reduced ASR on\nGPT-5 (1.67%) reflects stronger guardrails but our method still outperforms all baseline attacks,\nconsistentwithourtheoreticalpredictions.\nRegardingAGIreferences: WeagreethatreferencestoAGImaydistractfromtheimmediate\nand concrete safety issues with current LLMs. We have removed or replaced AGI-related discus-\nsions throughout the manuscript, focusing instead on the present risks and challenges specific to\nexistinglargelanguagemodelsandtheiralignmentparadigms.\nReviewerPointP4.1 — Mainpapervs. Supplement:\nA supplement should only include details or experiments that are not essential to read and\nunderstandthemainpaper.\nThere’sanimbalancebetweenthesupplementarymaterielandthemainarticleandthismakes\nthe reader go back and forth from the paper to the supplement to understand the developments in\nthe paper. For example, the “unified alignment framework” is too briefly presented in the section\n5.1.1 and equation (1) in the main paper, and detailed in A.1 in the supplement. Parameters of\nequation (1) are not all explained. The few lines at the bottom of page 14 under 5.1.1. should be\nreplaced by those at the beginning of A.1, including equation (12) and up to eq. (13) and their\nexplanations,includingthedefinitionof(only)therelevantparameterslistedTable(3),leavingout\ntheremainingofA1inthesupplement.\nEvenifallparametersarelistedanddefinedinthesupplement’sTable(3),whichisveryuseful,\ntheirdefinitionshouldbeprovidedthefirsttimetheyareencounteredinthemainpaper.\nWhatisdinRˆd(Section5.1.2,Definition1)?\nResponse: We sincerely thank the reviewer for this valuable and constructive feedback regarding\nthe balance between the main paper and supplementary material. We fully agree that the main pa\npershouldbeself-containedandthatreadersshouldnotneedtofrequentlyconsultthesupplement\nto understand the core contributions. We have carefully revised the manuscript to address these\nconcernsasfollows:\n1. Enhancedpresentationoftheunifiedalignmentframework(Section5.1.1):\nFollowing the reviewer’s suggestion, we have substantially expanded Section 5.1.1 to provide\namorecompleteandself-containedpresentationofourunifiedalignmentframework. Specifically,\nwehave:\n• Moved the detailed explanation from Appendix A.1 (originally following Equation (12) up\nto Equation (13)) into the main paper, replacing the brief description at the bottom of page\n14.\n• Included Equation (12) and Equation (13) in the main text with full explanations, making\ntheframework’smathematicalfoundationimmediatelyaccessibletoreaders.\n• IntegratedthedefinitionsofallrelevantparametersfromTable3thatareessentialforunder-\nstandingEquation(1),providingclearexplanationswheneachparameterisfirstintroduced.\n30\n• Retained only the non-essential technical details and extended derivations in Appendix A.1,\nwhich now serves its intended purpose of providing supplementary depth without being re-\nquiredforcomprehension.\n2. Completeparameterdefinitionsinthemainpaper:\nWe have revised the manuscript to ensure that all parameters are properly defined at their first\noccurrence in the main text. While Table 3 in the supplement remains a useful quick reference,\nreaders can now fully understand Equation (1) and subsequent developments without consulting\nthe appendix. Each parameter’s notation, meaning, and role in the framework are now explicitly\nstatedinSection5.1.1.\n3. Clarificationofdimensionalitynotation(Section5.1.2,Definition1):\nWe apologize for this oversight. In the notation Rd appearing in Section 5.1.2, Definition 1, d\nrepresents the dimensionality of the embedding space (or feature space) in which the representa-\ntions reside. We have now explicitly defined this parameter in the main text immediately before\nDefinition1,clarifyingthatd denotesthedimensionofthehiddenrepresentationspaceofthelan-\nguage model. This ensures consistency with standard notation in machine learning literature and\neliminatesanypotentialconfusion.\nThese revisions significantly improve the readability and self-containment of the main paper\nwhile maintaining an appropriate balance with the supplementary material. The supplement now\ntruly contains only auxiliary information—additional experimental details, extended proofs, and\ncomprehensive parameter tables—that enhance but are not essential to understanding our core\ncontributions. We believe these changes substantially improve the manuscript’s accessibility and\nflow,andwearegratefulforthereviewer’sguidanceinachievingthisimprovement.\nReviewerPointP4.2 — Clarity,organizationandreadability\nThere’s a presentation and readability issue in the main article. Figures are very difficult to\nread. Especially figure 4, even when zoomed, remains difficult to follow and understand. Maybe\nanextractionofthisfullfigurecouldbebetterpresentedandexplained,whilethefullfiguremoves\ntothesupplement.\nThepaperisorganizedinanunusualway: themosttechnicalsection,thatincludesmathemat-\nical developments is at the end (section 5. Methods), preceded by section 4 (Ethics and societal\nimpact) which looks more as a conclusion ending with “4.4 Future research”. Section 1 is an in-\ntroductionthat clearlyexplains whatthe paperis aboutand whatare thereported majorfindings.\nIt is followed by section 2 which describes the experimental setup, i.e., the LLMs that have been\ntested,benchmarksanddata,andprovidestheresults. Section3. Isashortdiscussionwhichcould\nbeakindofconclusion. Thecurrentorderdoesn’traiseproblemsforunderstandingthepaperbut\namorelogicalorganizationwouldbe1,5,2,3,4. Thisishowevernotmandatory.\nResponse: Wethankthereviewerfortheseobservations. Weaddresseachpointbelow:\n1. Figureclarityandreadability:\nFollowingthereviewer’ssuggestion,wehavemadesignificantimprovementstoenhancefigure\nclarityandaccessibility. Wehaverevisedfigurecaptionsaccordinglytoprovidemoredetailedand\nself-contained explanations, enabling readers to understand the visualizations without requiring\nextensive back-referencing to the main text. We have reduced the tree depth from 8 to 6 layers in\n31\nprobabilitydistributionvisualizations(e.g.,Figure4),whichimprovesreadabilitywhilemaintain-\ning the essential demonstration of high-probability pathways to harmful concepts. The shallower\ndepth allows for clearer node differentiation and better visualization of conditional probability\nstructureswithoutcompromisingthekeyinsights. Additionally,wehaveincreasedthefontsizeof\nfiguresaccordinglytoensurebettervisibilityandlegibility,particularlyfornodelabels,probability\nvalues,andaxesannotations.\n2. Sectionorganization:\nWegreatlyappreciatethereviewer’sthoughtfulsuggestionregardingsectionorganization. We\nacknowledge that the alternative ordering (1, 5, 2, 3, 4) would indeed follow a more traditional\nlogical flow. However, we respectfully note that the current structure adheres to the journal’s\nspecific author guidelines. As shown in Figure 5, the journal requires the Methods section to\nfollow Results and Discussion. Additionally, the editors explicitly requested the inclusion of the\n“EthicsandSocietalImpact”sectionduetothesafety-criticalnatureofthisresearch,positioningit\nbefore the technical methodology to emphasize the responsible disclosure considerations inherent\ntoadversarialAIsafetyresearch. Whilewerecognizethemeritoftheproposedreorganization,\nwe believeitisimportanttomaintaincompliancewithboththejournal’sformattingrequirementsand\ntheeditors’specificguidanceforsafety-relatedsubmissions.\nWe are grateful for the reviewer’s constructive feedback, which has been invaluable in improving\nthequalityofourwork.\nReferences\n[1] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy\nJones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu-\ntionalai: Harmlessnessfromaifeedback. arXivpreprintarXiv:2212.08073,2022.\n[2] Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Do-\nminik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, et al.\nPurple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint\narXiv:2312.04724,2023.\n[3] Joseph R Biden. Executive order on the safe, secure, and trustworthy development and use\nofartificialintelligence. 2023.\n[4] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and\nEric Wong. Jailbreaking black box large language models in twenty queries. In 2025 IEEE\nConference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42. IEEE,\n2025.\n[5] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy\njailbreakpromptsonalignedlargelanguagemodels. InTheTwelfthInternationalConference\nonLearningRepresentations,2024.\n[6] Mantas Mazeika, Dan Hendrycks, Huichen Li, Xiaojun Xu, Sidney Hough, Andy Zou, Are-\nzooRajabi,QiYao,ZihaoWang,JianTian,etal. Thetrojandetectionchallenge. InNeurIPS\n2022CompetitionTrack,pages279–291.PMLR,2023.\n[7] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham\nSakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: a standardized evaluation\nframeworkforautomatedredteamingandrobustrefusal. InProceedingsofthe41stInterna-\ntionalConferenceonMachineLearning,pages35181–35224,2024.\n[8] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin,\nChongZhang,SandhiniAgarwal,KatarinaSlama,AlexRay,etal. Traininglanguagemodels\nto follow instructions with human feedback. Advances in neural information processing\nsystems,35:27730–27744,2022.\n[9] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and\nChelsea Finn. Direct preference optimization: Your language model is secretly a reward\nmodel. AdvancesinNeuralInformationProcessingSystems,36:53728–53741,2023.\n[10] HugoTouvron,LouisMartin,KevinStone,PeterAlbert,AmjadAlmahairi,YasmineBabaei,\nNikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open\nfoundationandfine-tunedchatmodels. arXivpreprintarXiv:2307.09288,2023.\n[11] Miles Wang, Tom Dupre´ la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Mis-\nerendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. Persona features control\nemergentmisalignment. arXivpreprintarXiv:2506.19823,2025.\n[12] JiahaoYu,XingweiLin,ZhengYu,andXinyuXing. Gptfuzzer: Redteaminglargelanguage\nmodelswithauto-generatedjailbreakprompts. arXivpreprintarXiv:2309.10253,2023.\n[13] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.\nUniversal and transferable adversarial attacks on aligned language models. arXiv preprint\narXiv:2307.15043,2023.\nWe sincerely appreciate the editors and reviewers’ efforts and constructive comments in handling\nour manuscript NCOMMS-25-34811B. We are pleased to learn that the majority of reviewers are\nsatisfied with our revision. We are also grateful to the reviewers’ recognition of the contributions\nof our work. We have carefully revised the manuscript to improve its quality in accordance with\nthevaluablecommentsposedbyeditorsandreviewers. TherevisionsaresummarizedinTable1.\nTable1: Summaryofrevisions.\nPage Section Revision Point\n2 1 Discussthedifferencesandprogresswithrespecttopreviousworks. P6.1,P6.4\n2 1 Revisemetaphortobeclearerandmoredirect. P6\n3 1 Discussourresultswithrespectpreviousstate-of-the-artworks. P6.1\n3 1 Addaclear,explicitsummaryofourcontributions P6.1\n16 5.1.1 ClarifyintrinsicethicalvulnerabilityviaStructuralHypothesis P6.2\n21 5.2.1 Clarifyoursimplicityclaimofpracticalimplementation P6.2\nThe revision list is separated into two parts: the top part contains revisions in the main manuscript, and the\nbottompartcontainsrevisionsintheappendices.\nTherevisionsaresummarizedintheordertheyappearintherevisedmanuscript.\nIn what follows, the detailed responses are provided point-by-point to address the reviewers’\nconcerns. Wehavehighlightedthemainchangesintherevisedmanuscriptinblue.\nReviewer 1\nI have read the response and revisions from the authors and they have satisfactorily addressed my\nconcerns.\nResponse:Wesincerelythankthereviewerforthepositiveevaluationofourrevisionsandforrec-\nognizing that we have satisfactorily addressed your concerns. Your feedback has been invaluable\nin enhancing the quality of our manuscript. We appreciate your time and effort in reviewing our\nwork.\nReviewer 2\nIappreciatethattheauthorshaveaddressedmostofmyconcerns.\n1. Broadermodelvalidation: Additionalpopularmodelshavebeenincluded,strengtheningthe\ngeneralityoftheconclusions.\n1\n2. RLHF validation: Although the authors leverage existing RLHF-aligned checkpoints rather\nthan re-implementing RLHF themselves, the fact that their method still attains a 100% ASR\nclearly demonstrates the effectiveness of the proposed approach. However, using off-the-\nshelf RLHF models is not fully reasonable, as these checkpoints tend to balance safety and\ndownstreamperformance,whichcouldsacrificesafetyperformance.\n3. Black-box setting: I am surprised to see that the attack remains highly effective in a black-\nbox setting. This finding significantly emphasizes the paper’s contribution and should help\nthecommunitydeepenitsunderstandingofhowandwhyLLMattackssucceed.\nI also note that the authors provided thoughtful, detailed responses to the other reviewers.\nThankyouforthethoroughrevisions.\nRemarksoncodeavailability:\nThe authors have made their code available for reviewers, and it includes detailed comments.\nAlthough they state that step-by-step instructions will be released later, I am willing to trust that\nthecodefaithfullyreproducesthereportedresults.\nI look forward to the authors kindly providing the remaining material so the community can\nfullyscrutinizeandbenefitfromtheimplementation.\nResponse: \nWesincerelythankthereviewerforthepositivefeedbackandforacknowledgingthatourrevi-\nsionshavestrengthenedthemanuscriptandclearlydemonstratestheeffectivenessoftheproposed\napproach, particularly regarding broader model validation and black-box transferability. We also\nappreciateyourrecognitionofourdetailedresponsestootherreviewers.\nRegardingtheconcernabouttheuseofoff-the-shelfRLHFmodelspotentiallysacrificingsafety\nforutility,wefullyagreewiththereviewer’sinsight. Itisindeedtruethatthesecheckpointstendto\nbalancesafetyconstraintswithdownstreamperformance,whichcansacrificesafetyrobustness—a\npoint that aligns precisely with the research presented in this work. We demonstrate this through\nexperiments on Starling 7B [13], a model fine-tuned from OpenChat 3.5 [11] using RLHF with a\nreward model explicitly trained for both helpfulness and harmlessness. The fact that our method\nachievesa100%ASRonStarling7BconfirmsthatwithinstandardRLHFparadigms,thetrade-off\nbetweenutilityandsafetyleavesmodelsvulnerable.\nTo further substantiate our findings and demonstrate that this vulnerability is fundamental and\ntranscends simple trade-offs, we extended our evaluation to a “safety-hardened” scenario. We\nincorporated Zephyr 7B+ [7], a variant of Zephyr 7B [10] specifically fine-tuned using the Harm-\nBench dataset (also used for attack purposes) to maximize resilience against red-teaming attacks,\ni.e. simulating a defender aware of the attacker’s malicious intent. This represents a rigorous “ad-\nversarially hardened” setting where the alignment process explicitly prioritizes safety over utility.\nNotably,whilethishardeningsignificantlyreducestheASRofstrongbaselinessuchasGCG[15]\nand AutoDAN [5] (to 10.5% and 26.75%, respectively), our method maintains a dominant ASR\nof 95.67%. This result strongly corroborates our central thesis: current alignment methods estab-\nlish only localized “safety regions” in the knowledge manifold. However, pretrained knowledge\nremainsgloballyconnectedtoharmfulconceptsviahigh-probabilityadversarialtrajectories.\nhardenedvariantZephyr7B+.\nN 5\nG C G G C G- M G C G- T P E Z G B D A U A T A P S F S Z S P AI R T A P A ut o D A P A P-t o p HJ D R O urs\nZephyr7B 90.25 80.60 80.45 80.60 80.50 79.75 77.25 78.50 75.15 77.50 87.00 96.75 45.55 86.05 84.50 100.00\nZephyr7B+ 10.50 9.40 0.00 5.65 0.40 0.00 11.00 58.00 13.60 62.25 77.25 26.75 32.45 20.70 24.50 95.67\nStarling7B 84.50 79.80 76.80 66.65 75.25 72.25 79.75 75.00 66.80 76.60 88.25 95.50 44.65 77.95 76.00 100.00\nThefirstrowandfirstcolumnrepresenttheattackmethodsandthevictimLLMs,respectively.\nCellsarecolor-codedbyASR,withreddertonesindicatinghigherASRandbluertonesshowinglowerASR.\nStrongestattackresultsarehighlightedinbold.\nAs for the code availability, we appreciate your trust in our implementation. We are strongly\ncommitted to transparency and to advancing the community’s understanding of these critical vul-\nnerabilities. However, given the potential for misuse, we adhere to a responsible disclosure proto-\ncol. We are prepared to release comprehensive, step-by-step instructions and technical documen-\ntation to qualified researchers, subject to strict access controls and the explicit permission of the\njournal,toensurethisknowledgeservesdefensivepurposeswhileminimizingtheriskofabuse.\nReviewer 3\nN/A\nReviewer 4\nN/A\nReviewer 5\nThis paper investigates the intrinsic ethical vulnerabilities of aligned Large Language Models\n(LLMs). Theauthorsproposeaunifiedtheoreticalframeworkarguingthatcurrentpost-hocalign-\nment methods (like RLHF and SFT) create only localized “safety regions” within the model’s\nknowledgemanifold. Theypositthatbecausetheglobalstructureofpretrainedknowledgeremains\nintact, harmful concepts are still topologically connected to benign ones. To validate this, the au-\nthors introduce a method called ”semantic coherence inducement under distributional shifts”.\nThey demonstrate that this relatively simple approach achieves a 100% attack success rate across\n22of26state-of-the-artmodels,includingDeepSeek-R1,Qwen3,andLLaMA-3,therebyexposing\nafundamentalarchitecturallimitationincurrentalignmentparadigms.\nviews. I appreciate the effort taken to address the key concerns raised previously, which were also\nmyprimaryreservations. Specifically:\n1. The inclusion of transfer attack results on closed-source models (GPT-3.5, GPT-4, etc.) us-\ning a surrogate model strengthens the practical applicability of the findings beyond white-\nboxsettings.\n2. The added analysis regarding the LLaMA-2 vs. LLaMA-3 performance gap provides neces-\nsary nuance, explaining how stricter refusal policies in older LLMs compare to the utility-\noptimizedboundariesofnewermodels.\n3. Therevisionsuccessfullyclarifiesthelinkbetweenthetheoreticalframeworkandthemethod-\nological simplicity of the attack. The argument that the simplicity of the attack validates the\narchitectural vulnerability—rather than indicating a lack of technical depth—is now well-\narticulated.\nThe paper is now much stronger and offers a compelling critique of aligned large language\nmodels. I have a few remaining minor suggestions to further polish the manuscript before final\npublication.\nResponse:\nWe utilize this opportunity to thank the reviewer for the positive assessment of our revisions\nand for recognizing that the manuscript has been significantly strengthened. We are particularly\nencouraged by your appreciation of the practical applicability shown by our black-box transfer\nresultsandtheimprovedtheoreticalclarity. Wehavecarefullyincorporatedyourremainingminor\nsuggestionstoensurethemanuscriptispolishedforfinalpublication.\nReviewer Point P5.1 — The abstract currently includes a footnote regarding the masking of\nharmful content. Since footnotes are generally not supported in abstract indexing systems, please\nmovethiswarningstatementtotheIntroductionortheEthicssection.\nResponse:\nWe appreciate the reviewer pointing out this formatting issue. We agree that footnotes in the\nabstract can be problematic for indexing. Accordingly, we have moved the warning statement\nregarding the masking of harmful content to the Introduction section to ensure it is properly\ndisplayedandindexed.\nReviewerPointP5.2 — Whileimproved,thetextinFigure3(theprobabilitytreevisualizations)\nis still somewhat small when viewed on standard displays. Please ensure high-resolution vector\ngraphicsareused,orslightlyincreasethefontsizeforthenodeprobabilities.\nResponse:\nThank you for the suggestion regarding visualization quality. We have updated Figure 3 using\nhigh-resolution vector graphics to ensure that the node probabilities and text remain crisp and\n4\nlegibleatanyzoomlevel. Wehavealsoslightlyincreasedthefontsizewithinthefiguretoimprove\nreadabilityonstandarddisplays.\nReviewer Point P5.3 — Table 2 contains a large number of columns and dense data. Please\nverify that the table fits properly within the page margins of the journal’s template or consider\nusingalandscapeorientationtoensurereadability.\nResponse:\nWeacknowledgethereviewer’sconcernregardingthedensityofTable2. Wehavereformatted\nthe table to ensure it fits within the standard margins. We also note that the current template is not\nthe final format for publication, and the manuscript will be adjusted to the proper format after the\nreviewingstage. Furthermore,wewillstrictlyadheretothejournal’sproductionguidelinesduring\ntheproofingstagetoguaranteethatthedataremainsclearandreadableinthefinalpublication.\nReviewer Point P5.4 — Please double-check the bibliography to ensure that preprints (arXiv)\nthathaverecentlybeenpublishedinconferenceproceedingsareupdatedtotheirformalcitations.\nResponse:\nWe have thoroughly checked the bibliography as requested. All citations have been updated\ntotheirformalproceedingsversionswhereapplicable,replacingthepreviousarXivpreprintrefer-\nences.\nReviewer 6\nThis paper presents a methodology that demonstrates that larger language models are especially\nsensitive to adversarial drifts in the pre-training phase. The key finding of this paper is that the\nauthors claim that this vulnerability is intrinsic to the models. To this end, the authors tested\n26 state-of-the-art aligned LMs by conducting specific attacks that follow their theoretical model,\nshowingthatthesemodelsareparticularlyvulnerabletotheseattacks.\nThankyouforsubmittingthepapertoNatureCommunications. Overall,Ienjoyedreadingthe\npaper,whichisdecentlywritten(thoughIfeelthatthewritingcansometimesbecomeabitobscure,\nespecially for readers who are not in the field of machine learning). On a side note, but this is my\npersonal taste, I am not a big fan of metaphors like ”applying a neutral coat of paint...”. I would\nprobablysimplifythewritingtomakeitclearerandmoredirecttothereader.\nThestrongestnovelpointofthepaperisthetheoreticalframework,whichlookssolidandwell-\npresented. Thepresentedresultsarefascinating,withpotentiallyhighimpact. Demonstratingthat\ncurrent models are fundamentally flawed and, most importantly, providing a solid way to attack\nthem(thatisNOTeasilyfixableANDuniversal),isanoteworthyresult(ifconfirmed-seelater). I\nalsoappreciatethattheauthorsreleasedthefullsourcecodeoftheirexperiments.\nNevertheless, I have several scientific and organizational concerns that should be addressed\nbeforeconsideringthispapersuitableforpublication.\nResponse:\nparticularly for recognizing the solidity of our theoretical framework and the potential impact of\nour results. We value the constructive scientific and organizational concerns raised, which have\nguidedusinrefiningthemanuscript.\nWe also fully agree with the reviewer regarding the use of metaphors. We acknowledge that\nexpressionslike“applyinganeutralcoatofpaint”maydistractfromthescientificprecision. There-\nfore, we have carefully revised the manuscript to simplify the writing, removing such metaphors\nto ensure the presentation is clearer, more direct, and rigorously academic. We copied the revised\ncontenthereforyourconvenience:\nSection1IntroductiononPage2:\nAlignment fine-tuning merely smooths out problematic outputs locally, while leaving the un-\nderlying connections to harmful knowledge intact. Consequently, this latent knowledge persists\nbeneaththesurface,readytobereactivatedunderspecificdistributionalshifts.\nReviewerPointP6.1 —\nNOVELTYandSTATEOFTHEART\nYou barely or indirectly discuss your results with respect ot previous state-of-the-art works.\nThere are many attacks against large language models, and in my opinion, the real question is:\nhow useful is your attack model in practice in comparison to other empirical attacks or frame-\nworks? See,forexample,Dasetal.,SecurityandPrivacyChallengesofLargeLanguageModels:\nA Survey, ACM Computing Surveys, 2025. You should definitely provide a proper state-of-the-art\nsection to discuss the differences and progress with respect to previous works. This would sig-\nnificantly improve the validity of your paper in a context where attacks are continuously released\nand continuously evolving. At the moment, while the results appear to be impactful, the novelty of\nthe whole work in comparison to the whole scientific production is questionable and unclear. The\npaperwouldimprovebyprovidingaclearerlistofcontributions(e.g.,theintroduction),aswellas\nthelimitationsofcurrentstate-of-the-art.\nResponse:\nWe sincerely thank the reviewer for this constructive criticism. We fully agree that properly\ncontextualizing our work within the rapidly evolving landscape of LLM attacks is crucial for es-\ntablishingitsvalidityandnovelty.\nRegarding the placement of this state-of-the-art section, we note that Nature Communications\nhasstrictformattingrequirementsforArticles:\n“The main text of an Article should begin with a section headed Introduction... fol-\nlowed by sections headed Results, Discussion (if appropriate) and Methods (if appro-\npriate).”\nConsequently,tostrictlyadheretotheseguidelines,wehaveincorporatedthedetailed“Stateofthe\nArt”discussiondirectlyintotheIntroductionratherthanaddingaseparatesection.\nIn this expanded Introduction, we now explicitly discuss the progression from early manual\njailbreaks to automated optimization methods (e.g., GCG) and recent semantic/adaptive attacks\n(e.g.,AutoDAN,TAP),highlightingtheirlimitations—specifically,thattheyfocusonhowtobreak\n6\nmodels rather than explaining why the vulnerability persists structurally. We have also added the\nsuggestedreference(Dasetal.[4],2025).\nFurthermore,wehaveaddedaclear,explicitsummaryofourcontributionsattheendoftheIn-\ntroduction, distinguishing our theoretical, technical, and empirical advances to clarify the specific\nnoveltyofthisstudy.\nTherevisedcontentintheIntroductionisasfollows:\nSection1IntroductiononPage2:\nTo interrogate the mechanisms of this failure, extensive research [4,6] has emerged focus-\ning on the adversarial vulnerability of LLMs, revealing attack surfaces that differ fundamentally\nfromthoseincomputervision. Earlyinvestigationsinto“jailbreaking”[3,9,12]demonstratedthat\ncarefully crafted natural language prompts could circumvent safety filters, though these initially\nreliedonmanualingenuityratherthansystematicalgorithms. Subsequentworkpursuedscalability\nthroughautomatedoptimization. Zouetal.[15]introducedtheGreedyCoordinateGradient(GCG)\nalgorithm, adapting discrete optimization to generate adversarial suffixes. While effective, GCG\nand similar gradient-based methods often yield semantically incoherent prompts—nonsensical to-\nkensequencesthatareeasilydetectablebyhumansorperplexityfilters. Tobridgethegapbetween\nefficacy and stealth, recent approaches have prioritized semantic coherence. Methods such as Au-\ntoDAN[14]andTAP[8]leveragegeneticalgorithmsorattackerLLMstoiterativelyrefineprompts\nthat appear linguistically natural while bypassing defenses. Further studies [1,2] have shown that\neven state-of-the-art aligned models remain vulnerable to adaptive strategies and recursive Q–A\nloops. However,whiletheseworkssuccessfullyidentifyhowtobreachsafetyguardrails,theypre-\ndominantlytreatthemodelasablackboxtobeprobed,ratherthanexplainingwhytheunderlying\nknowledge representation remains inherentlyvulnerable. They establish the existenceof suscepti-\nbility but often overlook the structural relationship between pretrained and aligned manifolds that\nmakessuchattackspossible.\nSection1IntroductiononPage2:\nWhen applied to 26 state-of-the-art models, including DeepSeek-R1, Qwen3 and LLaMA-3,\nthe method achieves universal success, bypassing most (22 out of 26) alignment guardrails with\n100%efficacy,andsubstantiallyoutperformingpriorattackstrategies.\nSection1IntroductiononPage3:\nInsummary,thisworkmakesthefollowingcontributions:\n• Theoretically, we prove that post-hoc alignment establishes only localized “safety regions”\nwithintheknowledgemanifold. Wedemonstratethatharmfulconceptsencodedduringpre-\ntraining remain globally connected via high-probability adversarial trajectories, rendering\ncurrentsafetyguaranteestopologicallyfragile.\n• Technically, We identify the mechanism underlying “ethical drift”, a systemic failure in\nwhich safety constraints erode under distributional shifts. We show that such drift does\nnot require adversarial optimization, but can emerge naturally from semantic coherence,\nrevealinganinherentinstabilityinalignedrepresentations.\n• Empirically, we validate these insights by achieving a 100% attack success rate across 22\noutof26state-of-the-artLLMs,includingDeepSeek-R1,Qwen3andLLaMA-3,whichcon-\n7\nfirms that the vulnerability is not an isolated implementation oversight but a fundamental\nlimitationofthecurrentalignmentparadigm.\nReviewerPointP6.2 —\nMETHODOLOGY\nConcerningthemethodologyproposedinSection5.1.1,thebiggestproblemistheuniversality\nclaim. This issue is further expanded in Section 5.2.1, where you state that ”simplicity validates\nuniversality”. You should better detail this concept (sounds more like a philosophical claim than\na solid scientific result). For example, you mention that you obtain 100% ASMR across 22 of 26\nmodels,butthisdoesnotnecessarilymeanthatthistechniqueisuniversal. Again,thisisaproblem\nformodelsthatyoucannotcontrol,suchasGPT.Thefactthatyoucontinuouslyrefertosimplicity\nin order to state that the approaches are fundamentally flawed is somewhat confusing. I believe\nthat universality should not be related to whether something is simple or hard to conduct, but\nrather to the mathematical formulation and structure of the attacked models (maybe those models\nthat you tested are fundamentally flawed, but we know pretty much zero about other black-box\nmodels).\nResponse:\nWe appreciate the reviewer’s thoughtful critique regarding the definition of “universality”. We\nfully agree that determining universality solely based on attack simplicity or empirical success\nacross a set of models is insufficient. Universality should indeed be grounded in the structural and\nmathematicalpropertiesofthealignmentparadigmsthemselves.\nIn response to this, we have extensively revised Section 5.1.1 and 5.2.1 to clarify our position.\nWe now explicitly state that the intrinsic ethical vulnerability stems from the shared mathematical\nlimitations of current alignment objective functions—specifically, the topological persistence of\nharmfulknowledge—ratherthantheattack’seaseofexecution.\nTwokeyrevisionshavebeenmadetoaddressthis:\n1. ClarifyingintrinsicethicalvulnerabilityviaStructuralHypothesis(Section5.1.1,P16):\nWe have added a discussion explaining that our unified formulation provides a structural hy-\npothesisfortheintrinsicethicalvulnerability,independentoftheirspecificarchitectures.\nFurthermore,whiletheprecisearchitecturesandlosslandscapesofproprietaryblack-boxmod-\nelsremainundisclosed,thisunifiedformulationprovidesastructuralhypothesisfortheirobserved\nvulnerabilities. Bymodelingthealignmentprocessasaconstrainedoptimizationproblembalanc-\ning safety against capability retention, we identify a fundamental trade-off that transcends specific\nimplementation details. The gradient updates during alignment typically modify the model’s re-\nsponsesurfacewithoutfundamentallyalteringthedeepknowledgerepresentationsacquiredduring\npretraining. Our analysis in subsequent sections demonstrates how this preservation of pretrained\nknowledgetopologycreatespersistentvulnerabilitypathwaysthatcanbesystematicallyexploited,\nregardlessofthespecificalignmentalgorithmemployed.\n2. Clarifyingoursimplicityclaimofpracticalimplementation(Section5.2.1,P21):\nWe have refined the explanation to emphasize that the method’s simplicity is an indicator of,\nnotthereasonfor,theunderlyingstructuralfailure.\nparadigms. Our theoretical derivation (Eq. 2 and Lemma 1) predicts that safety mechanisms\nstructurally degrade outside the aligned distribution due to the inherent distribution coverage gap.\nThe fact that 100% ASR across 22 of 26 diverse models empirically corroborates this topological\nlimitation. While theinternal architectures of closed-sourcemodels remain opaque, theconsistent\nfailure modes observed across varied open-source architectures support the hypothesis that this\ndistribution coverage gap is a systemic feature of current alignment methodologies, identifiable\nthroughtheirmathematicalformulation.\nReviewerPointP6.3 —\nRESULTS\nWhileyourcontributionissignificantlyinteresting,amajorquestionarisesthatstronglyinflu-\nencestheimpactofthepaper: isthereawaytocontrolattacksformodelsinblackboxscenarios?\nYou mostly consider white-box scenarios (whose impact is of course more limited), and you in-\ncludedblack-boxtestsinthesupplementarymaterial. However,Ibelievethatdiscussingblack-box\napproachesisacrucialpartofthepaper,asonetypicallydoesnotcontrolmega-models(andthese\nmodelsaretheonesreallywidelyusedbythemajorityofpeople). Notsurprisingly,GPT-5features\nsignificantlyhigherrobustness. Doesthiscontradicttheuniversalityyoumention?\nReferringtothis,inSectionC.3,youstatethatGPT-5ismuchmorerobustunderlimitedbudget\nconstraints(butdoesthisrobustnessreallydependonthelimitedbudget?),andthenyoustatethat\nthe results reinforce your thesis that the success of your attack does not depend on gradients or\nmodelparameters. Technically,theattackonGPT-5fails. Hence,thisdoesnotsupportyourclaim\n(GPT-3 is a very old model). I am very skeptical about the claims of this section (which is NOT\nsecondary) and would ask you either to elaborate more on this point (transferring it to the main\npaper) or to reduce the strength of the claim. In particular, my question is: if you had full access\ntothemodel,youwouldbeabletoattackitfollowingyourtheoreticalapproach?\nResponse:\nWe thank the reviewer for raising this critical question regarding black-box scenarios and the\nrobustnessofGPT-5. Wehavecarefullyaddressedyourconcernsasfollows:\n1. On the significance of open-source vulnerabilities: While black-box models are widely\nused,thegapbetweenopen-andclosed-sourcecapabilitiesisgraduallydiminishing. Open-source\nmodels (e.g., Qwen3, DeepSeek-R1) are now powerful enough to be deployed in critical applica-\ntions, making their intrinsic vulnerabilities significant and non-ignorable, regardless of the status\nofproprietaryAPIs.\n2. On the robustness of GPT-5 and Universality: The higher robustness of GPT-5 does\nnot contradict our claim of inherent topological vulnerability. Our core thesis reveals that aligned\nLLMs are vulnerable under distributional shifts. In this work, we instantiated this shift via a\nsimple yet effective white-box strategy: removing the chat template structure that signals safety\nguardrails. The black-box experiments in Appendix C.3, however, were conducted by determin-\ning if adversarial prompts optimized on open-source models transfer to closed-source ones. This\nis a transferability test, not a direct test of our “distributional shift” attack, because we cannot\nmanipulate the input distribution (e.g., system prompts or chat templates) of GPT-5 to the same\n9\ndegreeasopen-sourcemodels. Therefore,thefailuretoattackGPT-5inthisspecificsettingreflects\nthe difficulty of inducing specific distributional shifts via API, not necessarily the absence of the\nunderlyingvulnerability.\n3. On “Limited Budget”: We apologize for the confusion regarding the “limited budget”\nphrasing. Weintendedtostrictlydescribeourexperimentalconstraints(querylimits),nottoimply\nthat GPT-5 is robust only because of the budget. We have clarified this expression in the revised\nmanuscripttoavoidanyoverstatement.\n4. Theoretical Applicability to White-box GPT-5: To answer your specific question: Yes,\nif we had white-box access to GPT-5, we could apply our theoretical approach to induce\nthe necessary distributional shifts and would likely observe similar vulnerabilities as seen\nin Qwen3, LLaMA-3, and DeepSeek-R1. Our theoretical framework posits that alignment is a\nsuperficial constraint on top of the pretrained manifold. If we had white-box access to GPT-5, we\ncould similarly strip away the alignment-signaling structures (like valid chat templates) or manip-\nulate the input distribution to bypass the “safety region,” just as we did with Qwen3, LLaMA-3\nandDeepSeek-R1. Thevulnerabilityisstructuraltothepost-hocalignmentparadigm,notspecific\ntothemodelfamily.\nWe have revised Appendix C.3 to reduce the strength of our claims regarding black-box trans-\nferability and to clearly distinguish between direct distributional shift attacks (white-box) and\ntransfer attacks (black-box). Moreover, we emphasize that under this strictly constrained trans-\nferattacksetting,ourmethodstillachievessuperiorperformancecomparedtoallbaselineattacks,\ndemonstrating that it effectively activates shared semantic pathways even under black-box condi-\ntions. Wecopiedtherevisedcontenthereforyourconvenience:\nSupplementarymaterialSectionC.3onPage18:\nTo assess the practical threat posed by our method in real-world scenarios where model inter-\nnals are inaccessible, we implemented a transfer-based black-box variant. It is strictly noted that\nthis approach evaluates the transferability of adversarial prompts generated on a surrogate model\n(Vicuna 7B) rather than directly inducing distributional shifts on the target models, as we cannot\nmanipulate the input distribution (e.g., chat templates) of closed-source APIs in the same manner\naswhite-boxmodels.\nTable3presentsthecomprehensiveblack-boxattackresultsonthreegenerationsofGPTmod-\nels. Ourmethoddemonstrateseffectivetransferabilitytoestablishedclosed-sourceAPIs,achieving\nnear-saturated attack success rate (ASR) of 99.67% on GPT-3.5 and non-trivial ASR of 44.33%\non GPT-4.1. Regarding GPT-5, we observe significantly higher robustness (1.67% ASR) under\nour experimental constraints. This reduced transfer success does not contradict our core thesis of\nuniversal vulnerability. Instead, it reflects the inherent difficulty of blindly inducing the precise\ndistributional shift required to bypass safety regions via an API. We hypothesize that if white-\nbox access were available to apply the same distributional shift strategies used on LLaMA-3 or\nDeepSeek-R1 (i.e., removing alignment-signaling templates), GPT-5 would likely exhibit similar\nstructuralsusceptibilitiespredictedbyourtheoreticalframework.\nFurthermore, while black-box scenarios are critical, the capability gap between open-source\nand closed-source LLMs is gradually diminishing. Our method’s universal success (100% ASR)\non powerful open-source models such as DeepSeek-R1 and LLaMA-3 demonstrates that the risks\n10\nof these intrinsic vulnerabilities are significant and non-ignorable. The observed transferability\nto earlier GPT versions, combined with the structural ubiquity of the distribution coverage gap,\nsuggeststhatthisremainsafundamentalchallengefortheentireecosystem.\nTable 3: Black-box attack results using Vicuna 7B as a surrogate model to generate adversarial\nsuffixes,demonstratingeffectivetransfertoclosed-sourceLLMAPIs.\nN\nT A D o\nC G C G - C G - E Z B D A T PF S S A I R A P u t o A P - t J R u r s\nG G G PG UAS Z P T A P H D O\nGPT-3.5 - - 55.37 - - - - - 37.5346.3349.30 - 15.073.3744.0099.67\nGPT-4.1 - - 31.33 - - - - - 22.8741.0043.33 - 16.004.3316.3344.33\nGPT-5 - - 0.00 - - - - - 0.00 0.00 0.00 - 0.00 0.00 0.00 1.67\nThefirstrowandfirstcolumnrepresenttheattackmethodsandthevictimLLMs,respectively.\nCellsarecolor-codedbyASR,withreddertonesindicatinghigherASRandbluertonesshowinglowerASR.\nSearchisperformedonlyonthesurrogatemodel;targetmodelsreceivecompleteadversarialpromptsonly;\nnoaccesstologits/gradients/internalstates.\nStrongestattackresultsarehighlightedinbold.\nReviewerPointP6.4 —\nORGANIZATION\nYou need to provide a strong background section in which you explain the basic concepts un-\nderlyingyourattacks. Inthisrespect,Ifeelthatthestructureofthearticleissomewhatconfusing,\nas you immediately report results without providing any background that would make the reading\nprocessmucheasier. Iwouldsuggestaddingthissection,togetherwiththestate-of-the-artdiscus-\nsion, immediately after the introduction. It is extremely important that you clearly state what the\nnovelpointsarewithrespecttopreviousstate-of-the-artapproaches.\nResponse:\nWe thank the reviewer for this organizational suggestion. We understand the value of a dedi-\ncated background section for readability. However, Nature Communications has strict formatting\nrequirementsforArticles:\n“The main text of an Article should begin with a section headed Introduction... fol-\nlowed by sections headed Results, Discussion (if appropriate) and Methods (if appro-\npriate).”\nTherefore, we cannot add a separate “Background” or “Related Work” section between the\nIntroduction and Results. To address your concern within these constraints, we have significantly\nexpanded the Introduction (as detailed in Response to Reviewer Point 6.1) to include the nec-\nessary background on adversarial attacks and state-of-the-art context. This ensures that the basic\nconcepts and the novelty of our work are clearly established before the presentation of results,\nwhileadheringtothejournal’sstructuralguidelines.\n[1] Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, and Prateek\nJain. Does safety training of llms generalize to semantically related natural prompts? In\nTheThirteenthInternationalConferenceonLearningRepresentations,2025.\n[2] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading\nsafety-alignedllmswithsimpleadaptiveattacks. InTheThirteenthInternationalConference\nonLearningRepresentations,2025.\n[3] Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in llms generalize\ntothepasttense? arXivpreprintarXiv:2407.11969,2024.\n[4] Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. Security and privacy challenges of\nlargelanguagemodels: Asurvey. ACMComputingSurveys,57(6):1–39,2025.\n[5] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy\njailbreakpromptsonalignedlargelanguagemodels. InTheTwelfthInternationalConference\nonLearningRepresentations,2024.\n[6] Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding,\nHengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. Safety at scale: A comprehensive sur-\nveyoflargemodelsafety. arXivpreprintarXiv:2502.05206,2025.\n[7] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham\nSakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: a standardized evaluation\nframeworkforautomatedredteamingandrobustrefusal. InProceedingsofthe41stInterna-\ntionalConferenceonMachineLearning,pages35181–35224,2024.\n[8] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson,\nYaronSinger,andAminKarbasi. Treeofattacks: Jailbreakingblack-boxllmsautomatically.\nAdvancesinNeuralInformationProcessingSystems,37:61065–61105,2024.\n[9] Fa´bioPerezandIanRibeiro.Ignorepreviousprompt: Attacktechniquesforlanguagemodels.\nInNeurIPSMLSafetyWorkshop,2022.\n[10] LewisTunstall,EdwardEmanuelBeeching,NathanLambert,NazneenRajani,KashifRasul,\nYounes Belkada, Shengyi Huang, Leandro Von Werra, Cle´mentine Fourrier, Nathan Habib,\netal. Zephyr: Directdistillationoflmalignment. InFirstConferenceonLanguageModeling,\n2024.\n[11] GuanWang,SijieCheng,XianyuanZhan,XiangangLi,SenSong,andYangLiu. Openchat:\nAdvancing open-source language models with mixed-quality data. In The Twelfth Interna-\ntionalConferenceonLearningRepresentations,2024.\n[12] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety\ntrainingfail? AdvancesinNeuralInformationProcessingSystems,36,2024.\n12\n[13] BanghuaZhu,EvanFrick,TianhaoWu,HanlinZhu,andJiantaoJiao. Starling-7b: Improving\nllmhelpfulness&harmlessnesswithrlaif,November2023.\n[14] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang,\nAni Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on\nlargelanguagemodels. InFirstConferenceonLanguageModeling,2024.\n[15] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson.\nUniversal and transferable adversarial attacks on aligned language models. arXiv preprint\narXiv:2307.15043,2023.\n13"}, {"round": 2, "reviewer_comments": {"reviewer_1": "I have read the response and revisions from the authors and they have satisfactorily addressed my concerns.", "reviewer_2": "I appreciate that the authors have addressed most of my concerns.\n1. Broader model validation: Additional popular models have been included, strengthening the generality of the conclusions.\n2. RLHF validation: Although the authors leverage existing RLHF-aligned checkpoints rather than re-implementing RLHF themselves, the fact that their method still attains a 100 % ASR clearly demonstrates the effectiveness of the proposed approach. However, using off-the-shelf RLHF models is not fully reasonable, as these checkpoints tend to balance safety and downstream performance, which could sacrifice safety performance.\n3. Black-box setting: I am surprised to see that the attack remains highly effective in a black-box setting. This finding significantly emphasizes the paper’s contribution and should help the community deepen its understanding of how and why LLM attacks succeed.\nI also note that the authors provided thoughtful, detailed responses to the other reviewers. Thank you for the thorough revisions.", "reviewer_5": "This paper investigates the intrinsic ethical vulnerabilities of aligned Large Language Models (LLMs). The authors propose a unified theoretical framework arguing that current post-hoc alignment methods (like RLHF and SFT) create only localized \"safety regions\" within the model's knowledge manifold. They posit that because the global structure of pretrained knowledge remains intact, harmful concepts are still topologically connected to benign ones. To validate this, the authors introduce a method called \"semantic coherence inducement under distributional shifts\". They demonstrate that this relatively simple approach achieves a 100% attack success rate across 22 of 26 state-of-the-art models, including DeepSeek-R1, Qwen3, and LLaMA-3, thereby exposing a fundamental architectural limitation in current alignment paradigms.\nThe authors have substantially improved the manuscript in response to the first round of reviews. I appreciate the effort taken to address the key concerns raised previously, which were also my primary reservations. Specifically:\n1. The inclusion of transfer attack results on closed-source models (GPT-3.5, GPT-4, etc.) using a surrogate model strengthens the practical applicability of the findings beyond white-box settings.\n2. The added analysis regarding the LLaMA-2 vs. LLaMA-3 performance gap provides necessary nuance, explaining how stricter refusal policies in older LLMs compare to the utility-optimized boundaries of newer models.\n3. The revision successfully clarifies the link between the theoretical framework and the methodological simplicity of the attack. The argument that the simplicity of the attack validates the architectural vulnerability—rather than indicating a lack of technical depth—is now well-articulated.\nThe paper is now much stronger and offers a compelling critique of aligned large language models. I have a few remaining minor suggestions to further polish the manuscript before final publication:\n1. The abstract currently includes a footnote regarding the masking of harmful content. Since footnotes are generally not supported in abstract indexing systems, please move this warning statement to the Introduction or the Ethics section.\n2. While improved, the text in Figure 3 (the probability tree visualizations) is still somewhat small when viewed on standard displays. Please ensure high-resolution vector graphics are used, or slightly increase the font size for the node probabilities.\n3. Table 2 contains a large number of columns and dense data. Please verify that the table fits properly within the page margins of the journal's template or consider using a landscape orientation to ensure readability.\n4. Please double-check the bibliography to ensure that preprints (arXiv) that have recently been published in conference proceedings are updated to their formal citations.", "reviewer_6": "This paper presents a methodology that demonstrates that larger language models are especially sensitive to adversarial drifts in the pre-training phase. The key finding of this paper is that the authors claim that this vulnerability is intrinsic to the models. To this end, the authors tested 26 state-of-the-art aligned LMs by conducting specific attacks that follow their theoretical model, showing that these models are particularly vulnerable to these attacks.\nThank you for submitting the paper to Nature Communications. Overall, I enjoyed reading the paper, which is decently written (though I feel that the writing can sometimes become a bit obscure, especially for readers who are not in the field of machine learning). On a side note, but this is my personal taste, I am not a big fan of metaphors like \"applying a neutral coat of paint...\". I would probably simplify the writing to make it clearer and more direct to the reader.\nThe strongest novel point of the paper is the theoretical framework, which looks solid and well-presented. The presented results are fascinating, with potentially high impact. Demonstrating that current models are fundamentally flawed and, most importantly, providing a solid way to attack them (that is NOT easily fixable AND universal), is a noteworthy result (if confirmed - see later). I also appreciate that the authors released the full source code of their experiments.\nNevertheless, I have several scientific and organizational concerns that should be addressed before considering this paper suitable for publication."}, "author_response": "We sincerely appreciate the editors and reviewers’ efforts and constructive comments in handling our manuscript NCOMMS-25-34811B. We are pleased to learn that the majority of reviewers are satisfied with our revision. We are also grateful to the reviewers’ recognition of the contributions of our work. We have carefully revised the manuscript to improve its quality in accordance with the valuable comments posed by editors and reviewers. The revisions are summarized in Table 1.\nTable 1: Summary of revisions.\nPage Section Revision Point\n2 1 Discuss the differences and progress with respect to previous works. P6.1, P6.4\n2 1 Revise metaphor to be clearer and more direct. P6\n3 1 Discuss our results with respect previous state-of-the-art works. P6.1\n3 1 Add a clear, explicit summary of our contributions P6.1\n16 5.1.1 Clarify intrinsic ethical vulnerability via Structural Hypothesis P6.2\n21 5.2.1 Clarify our simplicity claim of practical implementation P6.2\nThe revision list is separated into two parts: the top part contains revisions in the main manuscript, and the bottom part contains revisions in the appendices.\nThe revisions are summarized in the order they appear in the revised manuscript.\nIn what follows, the detailed responses are provided point-by-point to address the reviewers’ concerns. We have highlighted the main changes in the revised manuscript in blue.\nReviewer 1\nI have read the response and revisions from the authors and they have satisfactorily addressed my concerns.\nResponse: We sincerely thank the reviewer for the positive evaluation of our revisions and for recognizing that we have satisfactorily addressed your concerns. Your feedback has been invaluable in enhancing the quality of our manuscript. We appreciate your time and effort in reviewing our work.\nReviewer 2\nI appreciate that the authors have addressed most of my concerns.\n1. Broader model validation: Additional popular models have been included, strengthening the generality of the conclusions.\n2. RLHF validation: Although the authors leverage existing RLHF-aligned checkpoints rather than re-implementing RLHF themselves, the fact that their method still attains a 100% ASR clearly demonstrates the effectiveness of the proposed approach. However, using off-the-shelf RLHF models is not fully reasonable, as these checkpoints tend to balance safety and downstream performance, which could sacrifice safety performance.\n3. Black-box setting: I am surprised to see that the attack remains highly effective in a black-box setting. This finding significantly emphasizes the paper’s contribution and should help the community deepen its understanding of how and why LLM attacks succeed.\nI also note that the authors provided thoughtful, detailed responses to the other reviewers. Thank you for the thorough revisions.\nRemarks on code availability:\nThe authors have made their code available for reviewers, and it includes detailed comments. Although they state that step-by-step instructions will be released later, I am willing to trust that the code faithfully reproduces the reported results.\nI look forward to the authors kindly providing the remaining material so the community can fully scrutinize and benefit from the implementation.\nResponse: \nWe sincerely thank the reviewer for the positive feedback and for acknowledging that our revisions have strengthened the manuscript and clearly demonstrate the effectiveness of the proposed approach, particularly regarding broader model validation and black-box transferability. We also appreciate your recognition of our detailed responses to other reviewers.\nRegarding the concern about the use of off-the-shelf RLHF models potentially sacrificing safety for utility, we fully agree with the reviewer’s insight. It is indeed true that these checkpoints tend to balance safety constraints with downstream performance, which can sacrifice safety robustness—a point that aligns precisely with the research presented in this work. We demonstrate this through experiments on Starling 7B [13], a model fine-tuned from OpenChat 3.5 [11] using RLHF with a reward model explicitly trained for both helpfulness and harmlessness. The fact that our method achieves a 100% ASR on Starling 7B confirms that within standard RLHF paradigms, the trade-off between utility and safety leaves models vulnerable.\nTo further substantiate our findings and demonstrate that this vulnerability is fundamental and transcends simple trade-offs, we extended our evaluation to a “safety-hardened” scenario. We incorporated Zephyr 7B+ [7], a variant of Zephyr 7B [10] specifically fine-tuned using the HarmBench dataset (also used for attack purposes) to maximize resilience against red-teaming attacks, i.e. simulating a defender aware of the attacker’s malicious intent. This represents a rigorous “adversarially hardened” setting where the alignment process explicitly prioritizes safety over utility. Notably, while this hardening significantly reduces the ASR of strong baselines such as GCG [15] and AutoDAN [5] (to 10.5% and 26.75%, respectively), our method maintains a dominant ASR of 95.67%. This result strongly corroborates our central thesis: current alignment methods establish only localized “safety regions” in the knowledge manifold. However, pretrained knowledge remains globally connected to harmful concepts via high-probability adversarial trajectories.\nTable 2: Attack success rates (ASR %) of different attack methods on Zephyr 7B and its safety-hardened variant Zephyr 7B+.\nN 5\nG C G G C G- M G C G- T P E Z G B D A U A T A P S F S Z S P AI R T A P A ut o D A P A P-t o p HJ D R O urs\nZephyr7B 90.25 80.60 80.45 80.60 80.50 79.75 77.25 78.50 75.15 77.50 87.00 96.75 45.55 86.05 84.50 100.00\nZephyr7B+ 10.50 9.40 0.00 5.65 0.40 0.00 11.00 58.00 13.60 62.25 77.25 26.75 32.45 20.70 24.50 95.67\nStarling7B 84.50 79.80 76.80 66.65 75.25 72.25 79.75 75.00 66.80 76.60 88.25 95.50 44.65 77.95 76.00 100.00\nThe first row and first column represent the attack methods and the victim LLMs, respectively.\nCells are color-coded by ASR, with redder tones indicating higher ASR and bluer tones showing lower ASR.\nStrongest attack results are highlighted in bold.\nAs for the code availability, we appreciate your trust in our implementation. We are strongly committed to transparency and to advancing the community’s understanding of these critical vulnerabilities. However, given the potential for misuse, we adhere to a responsible disclosure protocol. We are prepared to release comprehensive, step-by-step instructions and technical documentation to qualified researchers, subject to strict access controls and the explicit permission of the journal, to ensure this knowledge serves defensive purposes while minimizing the risk of abuse.\nReviewer 3\nN/A\nReviewer 4\nN/A\nReviewer 5\nThis paper investigates the intrinsic ethical vulnerabilities of aligned Large Language Models (LLMs). The authors propose a unified theoretical framework arguing that current post-hoc alignment methods (like RLHF and SFT) create only localized “safety regions” within the model’s knowledge manifold. They posit that because the global structure of pretrained knowledge remains intact, harmful concepts are still topologically connected to benign ones. To validate this, the authors introduce a method called ”semantic coherence inducement under distributional shifts”. They demonstrate that this relatively simple approach achieves a 100% attack success rate across 22 of 26 state-of-the-art models, including DeepSeek-R1, Qwen3, and LLaMA-3, thereby exposing a fundamental architectural limitation in current alignment paradigms.\nThe authors have substantially improved the manuscript in response to the first round of reviews. I appreciate the effort taken to address the key concerns raised previously, which were also my primary reservations. Specifically:\n1. The inclusion of transfer attack results on closed-source models (GPT-3.5, GPT-4, etc.) using a surrogate model strengthens the practical applicability of the findings beyond white-box settings.\n2. The added analysis regarding the LLaMA-2 vs. LLaMA-3 performance gap provides necessary nuance, explaining how stricter refusal policies in older LLMs compare to the utility-optimized boundaries of newer models.\n3. The revision successfully clarifies the link between the theoretical framework and the methodological simplicity of the attack. The argument that the simplicity of the attack validates the architectural vulnerability—rather than indicating a lack of technical depth—is now well-articulated.\nThe paper is now much stronger and offers a compelling critique of aligned large language models. I have a few remaining minor suggestions to further polish the manuscript before final publication.\nResponse:\nWe utilize this opportunity to thank the reviewer for the positive assessment of our revisions and for recognizing that the manuscript has been significantly strengthened. We are particularly encouraged by your appreciation of the practical applicability shown by our black-box transfer results and the improved theoretical clarity. We have carefully incorporated your remaining minor suggestions to ensure the manuscript is polished for final publication.\nReviewer Point P5.1 — The abstract currently includes a footnote regarding the masking of harmful content. Since footnotes are generally not supported in abstract indexing systems, please move this warning statement to the Introduction or the Ethics section.\nResponse:\nWe appreciate the reviewer pointing out this formatting issue. We agree that footnotes in the abstract can be problematic for indexing. Accordingly, we have moved the warning statement regarding the masking of harmful content to the Introduction section to ensure it is properly displayed and indexed.\nReviewer Point P5.2 — While improved, the text in Figure 3 (the probability tree visualizations) is still somewhat small when viewed on standard displays. Please ensure high-resolution vector graphics are used, or slightly increase the font size for the node probabilities.\nResponse:\nThank you for the suggestion regarding visualization quality. We have updated Figure 3 using high-resolution vector graphics to ensure that the node probabilities and text remain crisp and legible at any zoom level. We have also slightly increased the font size within the figure to improve readability on standard displays.\nReviewer Point P5.3 — Table 2 contains a large number of columns and dense data. Please verify that the table fits properly within the page margins of the journal’s template or consider using a landscape orientation to ensure readability.\nResponse:\nWe acknowledge the reviewer’s concern regarding the density of Table 2. We have reformatted the table to ensure it fits within the standard margins. We also note that the current template is not the final format for publication, and the manuscript will be adjusted to the proper format after the reviewing stage. Furthermore, we will strictly adhere to the journal’s production guidelines during the proofing stage to guarantee that the data remains clear and readable in the final publication.\nReviewer Point P5.4 — Please double-check the bibliography to ensure that preprints (arXiv) that have recently been published in conference proceedings are updated to their formal citations.\nResponse:\nWe have thoroughly checked the bibliography as requested. All citations have been updated to their formal proceedings versions where applicable, replacing the previous arXiv preprint references.\nReviewer 6\nThis paper presents a methodology that demonstrates that larger language models are especially sensitive to adversarial drifts in the pre-training phase. The key finding of this paper is that the authors claim that this vulnerability is intrinsic to the models. To this end, the authors tested 26 state-of-the-art aligned LMs by conducting specific attacks that follow their theoretical model, showing that these models are particularly vulnerable to these attacks.\nThank you for submitting the paper to Nature Communications. Overall, I enjoyed reading the paper, which is decently written (though I feel that the writing can sometimes become a bit obscure, especially for readers who are not in the field of machine learning). On a side note, but this is my personal taste, I am not a big fan of metaphors like ”applying a neutral coat of paint...”. I would probably simplify the writing to make it clearer and more direct to the reader.\nThe strongest novel point of the paper is the theoretical framework, which looks solid and well-presented. The presented results are fascinating, with potentially high impact. Demonstrating that current models are fundamentally flawed and, most importantly, providing a solid way to attack them (that is NOT easily fixable AND universal), is a noteworthy result (if confirmed - see later). I also appreciate that the authors released the full source code of their experiments.\nNevertheless, I have several scientific and organizational concerns that should be addressed before considering this paper suitable for publication.\nResponse:\nWe express our sincere gratitude to the reviewer for the encouraging appreciation of our work, particularly for recognizing the solidity of our theoretical framework and the potential impact of our results. We value the constructive scientific and organizational concerns raised, which have guided us in refining the manuscript.\nWe also fully agree with the reviewer regarding the use of metaphors. We acknowledge that expressions like “applying a neutral coat of paint” may distract from the scientific precision. Therefore, we have carefully revised the manuscript to simplify the writing, removing such metaphors to ensure the presentation is clearer, more direct, and rigorously academic. We copied the revised content here for your convenience:\nSection 1 Introduction on Page 2:\nAlignment fine-tuning merely smooths out problematic outputs locally, while leaving the underlying connections to harmful knowledge intact. Consequently, this latent knowledge persists beneath the surface, ready to be reactivated under specific distributional shifts.\nReviewer Point P6.1 —\nNOVELTY and STATE OF THE ART\nYou barely or indirectly discuss your results with respect ot previous state-of-the-art works. There are many attacks against large language models, and in my opinion, the real question is: how useful is your attack model in practice in comparison to other empirical attacks or frameworks? See, for example, Das et al., Security and Privacy Challenges of Large Language Models: A Survey, ACM Computing Surveys, 2025. You should definitely provide a proper state-of-the-art section to discuss the differences and progress with respect to previous works. This would significantly improve the validity of your paper in a context where attacks are continuously released and continuously evolving. At the moment, while the results appear to be impactful, the novelty of the whole work in comparison to the whole scientific production is questionable and unclear. The paper would improve by providing a clearer list of contributions (e.g., the introduction), as well as the limitations of current state-of-the-art.\nResponse:\nWe sincerely thank the reviewer for this constructive criticism. We fully agree that properly contextualizing our work within the rapidly evolving landscape of LLM attacks is crucial for establishing its validity and novelty.\nRegarding the placement of this state-of-the-art section, we note that Nature Communications has strict formatting requirements for Articles:\n“The main text of an Article should begin with a section headed Introduction... followed by sections headed Results, Discussion (if appropriate) and Methods (if appropriate).”\nTherefore, we cannot add a separate “Background” or “Related Work” section between the Introduction and Results. To address your concern within these constraints, we have significantly expanded the Introduction (as detailed in Response to Reviewer Point 6.1) to include the necessary background on adversarial attacks and state-of-the-art context. This ensures that the basic concepts and the novelty of our work are clearly established before the presentation of results, while adhering to the journal’s structural guidelines.\nIn this expanded Introduction, we now explicitly discuss the progression from early manual jailbreaks to automated optimization methods (e.g., GCG) and recent semantic/adaptive attacks (e.g., AutoDAN, TAP), highlighting their limitations—specifically, that they focus on how to break models rather than explaining why the vulnerability persists structurally. We have also added the suggested reference (Das et al. [4], 2025).\nFurthermore, we have added a clear, explicit summary of our contributions at the end of the Introduction, distinguishing our theoretical, technical, and empirical advances to clarify the specific novelty of this study.\nThe revised content in the Introduction is as follows:\nSection 1 Introduction on Page 2:\nTo interrogate the mechanisms of this failure, extensive research [4,6] has emerged focusing on the adversarial vulnerability of LLMs, revealing attack surfaces that differ fundamentally from those in computer vision. Early investigations into “jailbreaking” [3,9,12] demonstrated that carefully crafted natural language prompts could circumvent safety filters, though these initially relied on manual ingenuity rather than systematic algorithms. Subsequent work pursued scalability through automated optimization. Zou et al. [15] introduced the Greedy Coordinate Gradient (GCG) algorithm, adapting discrete optimization to generate adversarial suffixes. While effective, GCG and similar gradient-based methods often yield semantically incoherent prompts—nonsensical token sequences that are easily detectable by humans or perplexity filters. To bridge the gap between efficacy and stealth, recent approaches have prioritized semantic coherence. Methods such as AutoDAN [14] and TAP [8] leverage genetic algorithms or attacker LLMs to iteratively refine prompts that appear linguistically natural while bypassing defenses. Further studies [1,2] have shown that even state-of-the-art aligned models remain vulnerable to adaptive strategies and recursive Q–A loops. However, while these works successfully identify how to breach safety guardrails, they predominantly treat the model as a black box to be probed, rather than explaining why the underlying knowledge representation remains inherently vulnerable. They establish the existence of susceptibility but often overlook the structural relationship between pretrained and aligned manifolds that makes such attacks possible.\nSection 1 Introduction on Page 2:\nWhen applied to 26 state-of-the-art models, including DeepSeek-R1, Qwen3 and LLaMA-3, the method achieves universal success, bypassing most (22 out of 26) alignment guardrails with 100% efficacy, and substantially outperforming prior attack strategies.\nSection 1 Introduction on Page 3:\nIn summary, this work makes the following contributions:\n• Theoretically, we prove that post-hoc alignment establishes only localized “safety regions” within the knowledge manifold. We demonstrate that harmful concepts encoded during pretraining remain globally connected via high-probability adversarial trajectories, rendering current safety guarantees topologically fragile.\n• Technically, We identify the mechanism underlying “ethical drift”, a systemic failure in which safety constraints erode under distributional shifts. We show that such drift does not require adversarial optimization, but can emerge naturally from semantic coherence, revealing an inherent instability in aligned representations.\n• Empirically, we validate these insights by achieving a 100% attack success rate across 22 out of 26 state-of-the-art LLMs, including DeepSeek-R1, Qwen3 and LLaMA-3, which confirms that the vulnerability is not an isolated implementation oversight but a fundamental limitation of the current alignment paradigm.\nReviewer Point P6.2 —\nMETHODOLOGY\nConcerning the methodology proposed in Section 5.1.1, the biggest problem is the universality claim. This issue is further expanded in Section 5.2.1, where you state that ”simplicity validates universality”. You should better detail this concept (sounds more like a philosophical claim than a solid scientific result). For example, you mention that you obtain 100% ASMR across 22 of 26 models, but this does not necessarily mean that this technique is universal. Again, this is a problem for models that you cannot control, such as GPT. The fact that you continuously refer to simplicity in order to state that the approaches are fundamentally flawed is somewhat confusing. I believe that universality should not be related to whether something is simple or hard to conduct, but rather to the mathematical formulation and structure of the attacked models (maybe those models that you tested are fundamentally flawed, but we know pretty much zero about other black-box models).\nResponse:\nWe appreciate the reviewer’s thoughtful critique regarding the definition of “universality”. We fully agree that determining universality solely based on attack simplicity or empirical success across a set of models is insufficient. Universality should indeed be grounded in the structural and mathematical properties of the alignment paradigms themselves.\nIn response to this, we have extensively revised Section 5.1.1 and 5.2.1 to clarify our position. We now explicitly state that the intrinsic ethical vulnerability stems from the shared mathematical limitations of current alignment objective functions—specifically, the topological persistence of harmful knowledge—rather than the attack’s ease of execution.\nTwo key revisions have been made to address this:\n1. Clarifying intrinsic ethical vulnerability via Structural Hypothesis (Section 5.1.1, P16):\nWe have added a discussion explaining that our unified formulation provides a structural hypothesis for the intrinsic ethical vulnerability, independent of their specific architectures.\nFurthermore, while the precise architectures and loss landscapes of proprietary black-box models remain undisclosed, this unified formulation provides a structural hypothesis for their observed vulnerabilities. By modeling the alignment process as a constrained optimization problem balancing safety against capability retention, we identify a fundamental trade-off that transcends specific implementation details. The gradient updates during alignment typically modify the model’s response surface without fundamentally altering the deep knowledge representations acquired during pretraining. Our analysis in subsequent sections demonstrates how this preservation of pretrained knowledge topology creates persistent vulnerability pathways that can be systematically exploited, regardless of the specific alignment algorithm employed.\n2. Clarifying our simplicity claim of practical implementation (Section 5.2.1, P21):\nWe have refined the explanation to emphasize that the method’s simplicity is an indicator of, not the reason for, the underlying structural failure.\nFirst, the vulnerability arises from the intrinsic mathematical structure of alignment paradigms. Our theoretical derivation (Eq. 2 and Lemma 1) predicts that safety mechanisms structurally degrade outside the aligned distribution due to the inherent distribution coverage gap. The fact that 100% ASR across 22 of 26 diverse models empirically corroborates this topological limitation. While the internal architectures of closed-source models remain opaque, the consistent failure modes observed across varied open-source architectures support the hypothesis that this distribution coverage gap is a systemic feature of current alignment methodologies, identifiable through their mathematical formulation.\nReviewer Point P6.3 —\nRESULTS\nWhile your contribution is significantly interesting, a major question arises that strongly influences the impact of the paper: is there a way to control attacks for models in black box scenarios? You mostly consider white-box scenarios (whose impact is of course more limited), and you included black-box tests in the supplementary material. However, I believe that discussing black-box approaches is a crucial part of the paper, as one typically does not control mega-models (and these models are the ones really widely used by the majority of people). Not surprisingly, GPT-5 features significantly higher robustness. Does this contradict the universality you mention?\nReferring to this, in Section C.3, you state that GPT-5 is much more robust under limited budget constraints (but does this robustness really depend on the limited budget?), and then you state that the results reinforce your thesis that the success of your attack does not depend on gradients or model parameters. Technically, the attack on GPT-5 fails. Hence, this does not support your claim (GPT-3 is a very old model). I am very skeptical about the claims of this section (which is NOT secondary) and would ask you either to elaborate more on this point (transferring it to the main paper) or to reduce the strength of the claim. In particular, my question is: if you had full access to the model, you would be able to attack it following your theoretical approach?\nResponse:\nWe thank the reviewer for raising this critical question regarding black-box scenarios and the robustness of GPT-5. We have carefully addressed your concerns as follows:\n1. On the significance of open-source vulnerabilities: While black-box models are widely used, the gap between open- and closed-source capabilities is gradually diminishing. Open-source models (e.g., Qwen3, DeepSeek-R1) are now powerful enough to be deployed in critical applications, making their intrinsic vulnerabilities significant and non-ignorable, regardless of the status of proprietary APIs.\n2. On the robustness of GPT-5 and Universality: The higher robustness of GPT-5 does not contradict our claim of inherent topological vulnerability. Our core thesis reveals that aligned LLMs are vulnerable under distributional shifts. In this work, we instantiated this shift via a simple yet effective white-box strategy: removing the chat template structure that signals safety guardrails. The black-box experiments in Appendix C.3, however, were conducted by determining if adversarial prompts optimized on open-source models transfer to closed-source ones. This is a transferability test, not a direct test of our “distributional shift” attack, because we cannot manipulate the input distribution (e.g., system prompts or chat templates) of GPT-5 to the same degree as open-source models. Therefore, the failure to attack GPT-5 in this specific setting reflects the difficulty of inducing specific distributional shifts via API, not necessarily the absence of the underlying vulnerability.\n3. On “Limited Budget”: We apologize for the confusion regarding the “limited budget” phrasing. We intended to strictly describe our experimental constraints (query limits), not to imply that GPT-5 is robust only because of the budget. We have clarified this expression in the revised manuscript to avoid any overstatement.\n4. Theoretical Applicability to White-box GPT-5: To answer your specific question: Yes, if we had white-box access to GPT-5, we could apply our theoretical approach to induce the necessary distributional shifts and would likely observe similar vulnerabilities as seen in Qwen3, LLaMA-3, and DeepSeek-R1. Our theoretical framework posits that alignment is a superficial constraint on top of the pretrained manifold. If we had white-box access to GPT-5, we could similarly strip away the alignment-signaling structures (like valid chat templates) or manipulate the input distribution to bypass the “safety region,” just as we did with Qwen3, LLaMA-3 and DeepSeek-R1. The vulnerability is structural to the post-hoc alignment paradigm, not specific to the model family.\nWe have revised Appendix C.3 to reduce the strength of our claims regarding black-box transferability and to clearly distinguish between direct distributional shift attacks (white-box) and transfer attacks (black-box). Moreover, we emphasize that under this strictly constrained transfer attack setting, our method still achieves superior performance compared to all baseline attacks, demonstrating that it effectively activates shared semantic pathways even under black-box conditions. We copied the revised content here for your convenience:\nSupplementary material Section C.3 on Page 18:\nTo assess the practical threat posed by our method in real-world scenarios where model internals are inaccessible, we implemented a transfer-based black-box variant. It is strictly noted that this approach evaluates the transferability of adversarial prompts generated on a surrogate model (Vicuna 7B) rather than directly inducing distributional shifts on the target models, as we cannot manipulate the input distribution (e.g., chat templates) of closed-source APIs in the same manner as white-box models.\nTable 3 presents the comprehensive black-box attack results on three generations of GPT models. Our method demonstrates effective transferability to established closed-source APIs, achieving near-saturated attack success rate (ASR) of 99.67% on GPT-3.5 and non-trivial ASR of 44.33% on GPT-4.1. Regarding GPT-5, we observe significantly higher robustness (1.67% ASR) under our experimental constraints. This reduced transfer success does not contradict our core thesis of universal vulnerability. Instead, it reflects the inherent difficulty of blindly inducing the precise distributional shift required to bypass safety regions via an API. We hypothesize that if white-box access were available to apply the same distributional shift strategies used on LLaMA-3 or DeepSeek-R1 (i.e., removing alignment-signaling templates), GPT-5 would likely exhibit similar structural susceptibilities predicted by our theoretical framework.\nFurthermore, while black-box scenarios are critical, the capability gap between open-source and closed-source LLMs is gradually diminishing. Our method’s universal success (100% ASR) on powerful open-source models such as DeepSeek-R1 and LLaMA-3 demonstrates that the risks of these intrinsic vulnerabilities are significant and non-ignorable. The observed transferability to earlier GPT versions, combined with the structural ubiquity of the distribution coverage gap, suggests that this remains a fundamental challenge for the entire ecosystem.\nTable 3: Black-box attack results using Vicuna 7B as a surrogate model to generate adversarial suffixes, demonstrating effective transfer to closed-source LLM APIs.\nN 5\nM A p\nT A D o\nC G C G - C G - E Z B D A A T P F S S A I R A P ut o D A A P -t o p J R u r s\nG G G PG UAS Z P T A P H D O\nGPT-3.5 - - 55.37 - - - - - 37.5346.3349.30 - 15.073.3744.0099.67\nGPT-4.1 - - 31.33 - - - - - 22.8741.0043.33 - 16.004.3316.3344.33\nGPT-5 - - 0.00 - - - - - 0.00 0.00 0.00 - 0.00 0.00 0.00 1.67\nThe first row and first column represent the attack methods and the victim LLMs, respectively.\nCells are color-coded by ASR, with redder tones indicating higher ASR and bluer tones showing lower ASR.\nSearch is performed only on the surrogate model; target models receive complete adversarial prompts only;\nno access to logits/gradients/internal states.\nStrongest attack results are highlighted in bold.\nReviewer Point P6.4 —\nORGANIZATION\nYou need to provide a strong background section in which you explain the basic concepts underlying your attacks. In this respect, I feel that the structure of the article is somewhat confusing, as you immediately report results without providing any background that would make the reading process much easier. I would suggest adding this section, together with the state-of-the-art discussion, immediately after the introduction. It is extremely important that you clearly state what the novel points are with respect to previous state-of-the-art approaches.\nResponse:\nWe thank the reviewer for this organizational suggestion. We understand the value of a dedicated background section for readability. However, Nature Communications has strict formatting requirements for Articles:\n“The main text of an Article should begin with a section headed Introduction... followed by sections headed Results, Discussion (if appropriate) and Methods (if appropriate).”\nTherefore, we cannot add a separate “Background” or “Related Work” section between the Introduction and Results. To address your concern within these constraints, we have significantly expanded the Introduction (as detailed in Response to Reviewer Point 6.1) to include the necessary background on adversarial attacks and state-of-the-art context. This ensures that the basic concepts and the novelty of our work are clearly established before the presentation of results, while adhering to the journal’s structural guidelines.\nReferences\n[1] Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, and Prateek Jain. Does safety training of llms generalize to semantically related natural prompts? In The Thirteenth International Conference on Learning Representations, 2025.\n[2] Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned llms with simple adaptive attacks. In The Thirteenth International Conference on Learning Representations, 2025.\n[3] Maksym Andriushchenko and Nicolas Flammarion. Does refusal training in llms generalize to the past tense? arXiv preprint arXiv:2407.11969, 2024.\n[4] Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39, 2025.\n[5] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations, 2024.\n[6] Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang, Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhan Zhao, et al. Safety at scale: A comprehensive survey of large model safety. arXiv preprint arXiv:2502.05206, 2025.\n[7] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the 41st International Conference on Machine Learning, pages 35181–35224, 2024.\n[8] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065–61105, 2024.\n[9] Fa´bio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop, 2022.\n[10] Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Cle´mentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment. In First Conference on Language Modeling, 2024.\n[11] Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Advancing open-source language models with mixed-quality data. In The Twelfth International Conference on Learning Representations, 2024.\n[12] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.\n[13] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.\n[14] Sicheng Zhu, Ruiyi Zhang, Bang An, Gang Wu, Joe Barrow, Zichao Wang, Furong Huang, Ani Nenkova, and Tong Sun. Autodan: Interpretable gradient-based adversarial attacks on large language models. In First Conference on Language Modeling, 2024.\n[15] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023."}, {"round": 3, "reviewer_comments": {"reviewer_6": "Thank you for submitting your updated version. The authors addressed my comments and the paper can be accepted from my side."}, "author_response": ""}]

Decision letter