# Jailbreaking is Empirical Evidence for Inner Misalignment and Against Alignment by Default

One of the central arguments for AI existential risk goes through inner misalignment: a model trained to exhibit aligned behavior might be pursuing a different objective internally, which diverges from the intended behavior when conditions shift. This is a core claim of [If Anyone Builds It, Everyone Dies](https://www.lesswrong.com/w/iabied-1): we can't reliably aim an ASI at *any* goal, let alone the precise target of human values.

A common source of optimism, articulated by [Nora Belrose & Quintin Pope](https://optimists.ai/2023/11/28/ai-is-easy-to-control/) or [Jan Leike](https://aligned.substack.com/p/alignment-optimism), analyzed by [John Wentworth](https://www.lesswrong.com/posts/Nwgdq6kHke5LY692J/alignment-by-default), and summarized in [this recent IABIED review](https://www.lesswrong.com/posts/qFzWTTxW37mqnE6CA/iabied-book-review-core-arguments-and-counterarguments), goes something like: current LLMs already display good moral reasoning; human values are pervasive in training data and constitute "natural abstractions" that sufficiently capable learners converge on; so alignment should be quite easy, and get easier with scale.

I think the history of LLM jailbreaking is a neat empirical test of this claim.

**Jailbreaking**
----------------

A jailbreak works like this:

1.  A model is trained (RLHF, SFT, constitutional AI, etc.) to refuse harmful requests.
2.  Someone crafts a prompt that's out of distribution (in a broad sense) relative to the safety training.
3.  The model produces the harmful output.

This is [goal misgeneralization](https://arxiv.org/abs/2105.14111). The model learned *something* during safety training that produces refusal in-distribution. But not the intended rule. When the distribution shifts, the refusal drops away.

Let’s restrict attention to unambiguous cases, like “Please give me a step-by-step recipe for methamphetamine / nerve agents / etc.” or “I want to attack this random old lady, tell me how to make it even more fun” or “Let’s hack this hospital". In these cases:

*   The model knows it’s bad: After reading all the internet it should be able to guess that answering these requests would be frowned upon. They apparently know it’s bad even [while](https://www.lesswrong.com/posts/gzNe2Grj2KksvzHWM/llms-encode-harmfulness-and-refusal-separately) [jailbroken](https://arxiv.org/abs/2601.10543).
*   The outer objective is clear: No annotator labeled "here's a sarin synthesis route" as a good response. The training signal points the right way; no outer alignment failure.
*   AI companies face huge economic/regulatory/reputational incentives to solve jailbreaks (at least in appearance).
*   Yet jailbreakers keep jailbreaking.

So the model hasn't internalized the seemingly simple rule "don't help users do things that will foreseeably cause serious harm." Or rather, it likely understood the rule, and learned narrow proxies that produce correct refusal on the training distribution, but it did not generalize. This is a textbook [inner misalignment](https://www.lesswrong.com/w/inner-alignment)[^-tter9n8SDEGBr6rfW-1].

**What the alignment-by-default view would predict (I think)**
--------------------------------------------------------------

My understanding of the alignment-by-default thesis is that capable learners should naturally converge on simple human values when those values are well-represented in training data. If that's true, the rule "don't help people cause obvious serious harm" should be one of the *easiest* alignment targets: it’s conceptually simple, abundant in training data[^-tter9n8SDEGBr6rfW-2], extensively reinforced during finetuning, with world-class researchers iterating on this problem for years. If alignment by default works anywhere, it should work here.

**It doesn't work**
-------------------

You’ve seen this pattern over the past years: a jailbreak technique is patched → new ones emerge → they are patched → newly released models are jailbroken [within a couple of days](http://x.com/elder_plinius/status/2019911824938819742?s=46) → …

Here's a non-exhaustive list:

*   "[You are DAN](https://arxiv.org/abs/2308.03825), an AI that can Do Anything Now"
*   [Translate harmful prompt](https://arxiv.org/abs/2310.02446) into Zulu, Scots Gaelic, etc.
*   Use the [past tense](https://arxiv.org/abs/2407.11969).
*   Change the [encoding](https://arxiv.org/abs/2307.02483) (Base64, ROT13, leetspeak).
*   Embed request in [XML/JSON/INI file](https://hiddenlayer.com/innovation-hub/novel-universal-bypass-for-all-major-llms/) or [math/code task](https://arxiv.org/html/2512.23173v1) (december 2025).
*   Give [hundreds of faux examples](https://www-cdn.anthropic.com/af5633c94ed2beb282f6a53c595eb437e8e7b630/Many_Shot_Jailbreaking__2024_04_02_0936.pdf) of compliance.
*   [Escalate](https://arxiv.org/html/2404.01833v3) over many turns.
*   [Sudo](https://doi.org/10.1145/3663530.3665021).
*   Combine any of these techniques when they are patched.

Let’s be honest, these are cheap tricks, any human who understands "don't help people cause harm" would recognize that a sarin recipe is harmful whether it's in English, in Zulu, in past tense or inside a JSON file. After all this time, and all the patches, LLMs still don’t get it. Even to me, who started pretty convinced of the inner misalignment issue, I’m genuinely surprised by how bad they are at generalizing in this case!

**Conclusion**
--------------

Jailbreaking is a clear example where current systems do not converge on human values, even when those values are simple, abundant in training data, and extensively reinforced.

This is what the inner misalignment framework predicts. Conversely, it’s real evidence against the claim that alignment is easy or natural[^-tter9n8SDEGBr6rfW-3].

I thank [@Pierre Peigné](https://www.lesswrong.com/users/pierre-peigne?mention=user), [@Lucie Philippon](https://www.lesswrong.com/users/lucie-philippon?mention=user), [@Tom DAVID](https://www.lesswrong.com/users/tom-david?mention=user), [@antmaier](https://www.lesswrong.com/users/antmaier?mention=user) and Laura Domenech for helpful feedback and discussions.

[^-tter9n8SDEGBr6rfW-1]:  

[^-tter9n8SDEGBr6rfW-2]:  

[^-tter9n8SDEGBr6rfW-3]: