WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks,
  and Refusals of LLMs
Authors: name: Seungju Han; name: Kavel Rao; name: Allyson Ettinger; name: Liwei Jiang; name: Bill Yuchen Lin; name: Nathan Lambert; name: Yejin Choi; name: Nouha Dziri
Published: 2024-06-26 16:58:20

Abstract: We introduce WildGuard -- an open, light-weight moderation tool for LLM
safety that achieves three goals: (1) identifying malicious intent in user
prompts, (2) detecting safety risks of model responses, and (3) determining
model refusal rate. Together, WildGuard serves the increasing needs for
automatic safety moderation and evaluation of LLM interactions, providing a
one-stop tool with enhanced accuracy and broad coverage across 13 risk
categories. While existing open moderation tools such as Llama-Guard2 score
reasonably well in classifying straightforward model interactions, they lag far
behind a prompted GPT-4, especially in identifying adversarial jailbreaks and
in evaluating models' refusals, a key measure for evaluating safety behaviors
in model responses.
  To address these challenges, we construct WildGuardMix, a large-scale and
carefully balanced multi-task safety moderation dataset with 92K labeled
examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired
with various refusal and compliance responses. WildGuardMix is a combination of
WildGuardTrain, the training data of WildGuard, and WildGuardTest, a
high-quality human-annotated moderation test set with 5K labeled items covering
broad risk scenarios. Through extensive evaluations on WildGuardTest and ten
existing public benchmarks, we show that WildGuard establishes state-of-the-art
performance in open-source safety moderation across all the three tasks
compared to ten strong existing open-source moderation models (e.g., up to
26.4% improvement on refusal detection). Importantly, WildGuard matches and
sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt
harmfulness identification). WildGuard serves as a highly effective safety
moderator in an LLM interface, reducing the success rate of jailbreak attacks
from 79.8% to 2.4%.