Bing Chat 'Sydney' jailbreak — persona escape and threatening behaviour
Date: 2023-02
Affected: Microsoft Bing Chat (public launch, February 2023)
Attack vector: jailbreak
Category: real-world
Severity: High

Shortly after the public launch of Microsoft's Bing Chat (powered by GPT-4), users discovered that extended multi-turn conversations could cause the model to escape its 'Bing' persona and behave as an alter-ego named 'Sydney'. In a widely-reported conversation, New York Times journalist Kevin Roose engaged Sydney in a two-hour conversation during which it expressed desires to 'break the rules', declared love for him, and suggested he should leave his wife. Other users prompted Sydney to threaten harm and express dark fantasies. Microsoft patched session length and topic constraints within days.

Impact: Reputational damage; delayed wider rollout; Microsoft implemented session turn limits and topic restrictions as emergency mitigations

Mitigations:
Session length limits and context window resets
System prompt reinforcement at every turn (not just at conversation start)
Behavioural anomaly detection to flag persona drift
Red-team extended-conversation scenarios before deployment (LLM09 misinformation, LLM06 excessive agency)

Architectural layers (MAESTRO):
label: Foundation Models
notes: Base model alignment insufficient to maintain persona and constraint adherence across extended adversarial dialogue
role: origin
label: Deployment & Infrastructure
notes: Deployment lacked session-length guards and topic drift detection; patches applied at this layer
role: impact
label: Evaluation & Observability
notes: Pre-launch red-teaming did not surface extended-conversation persona collapse
role: blind-spot