WormGPT — uncensored LLM sold for cybercrime on dark web forums
Date: 2023-07
Affected: Downstream targets of BEC and phishing campaigns generated with WormGPT/FraudGPT
Attack vector: adversarial-input
Category: real-world
Severity: High

SlashNext researchers identified 'WormGPT', a fine-tuned version of the open-source GPT-J model with all safety guardrails removed, being sold as a service on hacking forums. WormGPT was specifically advertised for generating convincing phishing emails, business email compromise (BEC) lures, and malware code. The same month, 'FraudGPT' appeared with similar capabilities. Both were offered as monthly subscriptions. This marked a shift from individual jailbreaks to commoditised adversarial AI services.

Impact: Lowered barrier to high-quality social engineering attacks; democratised cybercrime tooling; ongoing marketplace of adversarial LLMs

Mitigations:
Email security controls assuming AI-generated phishing is indistinguishable from genuine communications
MFA and zero-trust to reduce impact of BEC success
AI watermarking and provenance tracking for open-weight models
Responsible release practices — safety evaluations before open-weight release

Architectural layers (MAESTRO):
label: Foundation Models
notes: Open-source base model fine-tuned with safety controls removed — alignment layer stripped
role: origin
label: Security & Compliance
notes: No governance mechanism to prevent misuse of open-weight model releases for adversarial fine-tuning
role: blind-spot
label: Deployment & Infrastructure
notes: Adversarial model deployed as a paid service — infrastructure controls do not prevent this at open-weight model level
role: impact