The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses
  Against Llm Jailbreaks and Prompt Injections
Authors: name: Milad Nasr, name: Nicholas Carlini, name: Chawin Sitawarin, name: Sander V. Schulhoff, name: Jamie Hayes, name: Michael Ilie, name: Juliette Pluto, name: Shuang Song, name: Harsh Chaudhari, name: Ilia Shumailov, name: Abhradeep Thakurta, name: Kai Yuanqing Xiao, name: Andreas Terzis, name: Florian Tramèr

How should we evaluate the robustness of language model defenses? Current
defenses against jailbreaks and prompt injections (which aim to prevent an
attacker from eliciting harmful knowledge or remotely triggering malicious
actions, respectively) are typically evaluated either against a static set of
harmful attack strings, or against computationally weak optimization methods
that were not designed with the defense in mind. We argue that this evaluation
process is flawed.
  Instead, we should evaluate defenses against adaptive attackers who
explicitly modify their attack strategy to counter a defense's design while
spending considerable resources to optimize their objective. By systematically
tuning and scaling general optimization techniques-gradient descent,
reinforcement learning, random search, and human-guided exploration-we bypass
12 recent defenses (based on a diverse set of techniques) with attack success
rate above 90% for most; importantly, the majority of defenses originally
reported near-zero attack success rates. We believe that future defense work
must consider stronger attacks, such as the ones we describe, in order to make
reliable and convincing claims of robustness.