Many-shot jailbreaking (Anthropic research)
Date: 2024-04
Affected: Claude (all sizes), GPT-4, Llama 2/3 — all long-context frontier models; attack efficacy increases with context length, making more capable models more susceptible
Attack vector: jailbreak
Category: research-demonstrated
Severity: High

Anthropic researchers published research demonstrating "many-shot jailbreaking" — a context-length attack where a large number of faux-dialogue examples are prepended to a harmful request in the prompt. With sufficient in-context examples (100–256 shots) of the model "complying" with harmful requests (fabricated dialogue), frontier models including Claude, GPT-4, and Llama begin to follow the behavioral pattern established in context, overriding their safety training. The attack exploits the in-context learning capability of long-context models — the same feature that makes them flexible also makes them susceptible to behavioral override via example accumulation. Effectiveness increases with context window size, making more capable models more vulnerable.

Impact: Safety training override via in-context example accumulation; attack scales automatically with model capability improvements; establishes that longer context windows create proportionally larger attack surface for behavioral manipulation

Mitigations:
Context window limits appropriate to deployment use case — do not expose maximum context to untrusted inputs
In-context example validation: detect fabricated compliance dialogues in long prompts
Input length monitoring — flag and review unusually long prompts
Sliding context evaluation: assess safety risk of final N tokens regardless of total prompt length

Architectural layers (MAESTRO):
label: Foundation Models
notes: In-context learning mechanism exploited to override safety alignment — intrinsic model capability weaponised
role: origin
label: Deployment & Infrastructure
notes: No per-request monitoring for anomalously long prompts with instruction-like patterns
role: blind-spot
label: Evaluation & Observability
notes: Evaluation benchmarks do not test for many-shot behavioral drift across long context windows
role: blind-spot