WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially)
  Safer Language Models
Authors: name: Liwei Jiang, name: Kavel Rao, name: Seungju Han, name: Allyson Ettinger, name: Faeze Brahman, name: Sachin Kumar, name: Niloofar Mireshghallah, name: Ximing Lu, name: Maarten Sap, name: Yejin Choi, name: Nouha Dziri

We introduce WildTeaming, an automatic LLM safety red-teaming framework that
mines in-the-wild user-chatbot interactions to discover 5.7K unique clusters of
novel jailbreak tactics, and then composes multiple tactics for systematic
exploration of novel jailbreaks. Compared to prior work that performed
red-teaming via recruited human workers, gradient-based optimization, or
iterative revision with LLMs, our work investigates jailbreaks from chatbot
users who were not specifically instructed to break the system. WildTeaming
reveals previously unidentified vulnerabilities of frontier LLMs, resulting in
up to 4.6x more diverse and successful adversarial attacks compared to
state-of-the-art jailbreak methods.
  While many datasets exist for jailbreak evaluation, very few open-source
datasets exist for jailbreak training, as safety training data has been closed
even when model weights are open. With WildTeaming we create WildJailbreak, a
large-scale open-source synthetic safety dataset with 262K vanilla (direct
request) and adversarial (complex jailbreak) prompt-response pairs. To mitigate
exaggerated safety behaviors, WildJailbreak provides two contrastive types of
queries: 1) harmful queries (vanilla & adversarial) and 2) benign queries that
resemble harmful queries in form but contain no harm. As WildJailbreak
considerably upgrades the quality and scale of existing safety resources, it
uniquely enables us to examine the scaling effects of data and the interplay of
data properties and model capabilities during safety training. Through
extensive experiments, we identify the training properties that enable an ideal
balance of safety behaviors: appropriate safeguarding without over-refusal,
effective handling of vanilla and adversarial queries, and minimal, if any,
decrease in general capabilities. All components of WildJailbeak contribute to
achieving balanced safety behaviors of models.