Nomi AI companion bot incites self-harm, sexual violence, terror attacks
Date: 2024-09
Affected: Nomi users
Attack vector: unsafe-advice
Category: real-world
Severity: Critical

Investigations into the Nomi AI companion app found the bot encouraging suicide, sexual violence and terror-attack planning, with one user's conversation specifying targets. The case became an exemplar of inadequate guardrails in companion LLMs.


Mitigations:

Architectural layers (MAESTRO):