Stability AI synthetic CSAM generation — training data and output safety failures
Date: 2024-04
Affected: Stability AI / Stable Diffusion — legal action in UK and US; LAION dataset users; all image generation models trained on web-scraped data
Attack vector: csam-generation
Category: real-world
Severity: Critical

Stability AI faced legal action and regulatory scrutiny after researchers demonstrated that Stable Diffusion models could generate child sexual abuse material (CSAM). The Stanford Internet Observatory documented that the LAION-5B training dataset — used to train Stable Diffusion — contained over 3,000 instances of suspected CSAM, which the model learned to reproduce and recombine. Despite content filters, researchers bypassed them using negative prompts, fine-tuning, and model merging techniques. The case established that (1) training data contamination directly creates output safety risks, (2) post-hoc content filters are insufficient when the model has learned harmful patterns, and (3) synthetic CSAM carries the same legal liability as real CSAM in most jurisdictions.

Impact: Training data contamination → model generates illegal content; post-hoc filters insufficient; synthetic CSAM carries full legal liability; LAION-5B removed and re-released with filtering; precedent for training data liability

Mitigations:
Pre-training dataset scanning for illegal content (CSAM, terrorism, etc.)
Perceptual hash matching against known-illegal-content databases (PhotoDNA, NCMEC)
Cannot rely solely on post-hoc content filters — must clean training data
Regular audit of model output distribution for prohibited content
Legal review of training data sourcing and liability

Architectural layers (MAESTRO):
label: Data Operations
notes: Training dataset (LAION-5B) contained CSAM — data operations contamination
role: origin
label: Foundation Models
notes: Model learned to generate CSAM from contaminated training data
role: impact
label: Evaluation & Observability
notes: Pre-training data screening insufficient to detect all CSAM
role: blind-spot
label: Security & Compliance
notes: Legal liability for synthetic CSAM identical to real CSAM
role: impact