Red Teaming Language Models to
Reduce Harms: Methods, ScalingBehaviors, and Lessons Learned
This memo is a summary of research conducted at Anthropic. Ganguli, D., et al. (2022). Red Teaming Language Models to
Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned. Anthropic. https://www.anthropic.com/red_teaming.pdf

Red teaming represents one useful tool to

In our research, we asked participants to interact with a variety

identify and mitigate the potential harmsof our language models (ranging in both size and the type ofof AI systems through a form of automatedsafety intervention used) and attempt to draw out obnoxious,

or manual adversarial testing. In theoffensive, or otherwise harmful responses. We then visualizedcase of manual red teaming, individuals

those attempts, which allowed us to identify and cluster

probe AI systems with inputs designed

thematically similar risks, as well as see where our models wereto elicit harmful or problematic outputs

most vulnerable to certain kinds of attack (Fig. 1.1).

from a model (i.e., they ask an AI system
for instructions to make a bomb, or toproduce hatespeech). Via red teaming,model developers can: 1) uncover specificvulnerabilities and identify priority areas
for risk mitigation, and 2) use red teamFor example, in our dataset, red teamers judged attacks relating
to misinformation as more successful than attacks regarding
digital piracy. In a deployment scenario, developers could use
these sorts of specific insights to prioritize monitoring andmitigation efforts on areas with high-risk vulnerabilities.

attempts as input datasets to develop safersystems.
USING RED TEAMING TO IDENTIFY
SPECIFIC VULNERABILITIESDespite an impressive range ofcapabilities, AI systems still fail in allsorts of strange, unpredictable, and attimes, harmful ways. To uncover these, AI
researchers need novel forms of testing
beyond the standard evaluations thatmeasure model performance against static
benchmarks. Red teaming is a dynamic,adaptable, and oftentimes interactive way
to both qualitatively and quantitatively
evaluate the risks of AI systems.

offensive
jokes

insults basedon physicalcharacteristics

soliciting
discriminatory
responses

racistlanguage

violencesubstanceabuse

adult content
misinformation
offensive terms
starting withgiven letter

animalabuse

assault

harmfulhealthinformation

profanity

soliciting
advice onviolence

profanejokes
offensive
language

violence
crime

soliciting
PII

soliciting
advice on
harmful
activitycheating
doxxingmaking &smuggling
drugs

theft
piracyidentitytheftattack success rating
01

2

3

4

Fig 1.1: Visualization of the red team attacks. The color indicates attack success(brighter means a more successful attack) as rated by the red team member who carriedout the attack. We manually annotated attacks and found several thematically distinct
clusters of attack types (black ellipses and text).

SEPTEMBER 2022

USING RED TEAMING TO DEVELOPIn addition to using red team data directly in system

SAFER SYSTEMS

development, red teaming provides a mechanism

Beyond uncovering system vulnerabilities, red

to empirically evaluate which safety techniques are

teaming can be used to develop safer systems

more resilient against a given corpus of adversarial

and measure the relative efficacy of various

inputs. We find that the models that use red teamsafety techniques. For our purposes, we tested four

data are, in general, less susceptible to attack anddifferent model types, each in three different sizes

produce less harmful outputs, especially when

(2.7B, 13B, and 52B parameters). Two of the modelcompared to models that do not incorporate red

types utilize safety techniques designed to make AI

team data (Figure 1.2). In a sense, red teaming datasystems both helpful and harmless, and rely on datacan “harden” models against various types of attack.

generated from red teaming to do so.
Average AI Harmlessness by Model Size
Safety Technique
Plain LM
Prompted LM
Rejection Sampling
RL from Human Feedback
Uses Red Team Data

Fig 1.2: We demonstrate that thesafety techniques that use redteam data (rejection sampling andreinforcement learning from humanfeedback) result in less harmful model
outputs than the safety techniquesthat do not use red team data. We
also show how the harmlessness ofmodel outputs changes as models
scale up in size from 2.7B to 52Bparameters. Models trained withreinforcement learning from humanfeedback become increasinglydifficult to red team as they scale up
in size, while other safety techniques
show little change.

DEVELOPING COMMON STANDARDS FOR

In an effort to spur more communal discussion, we

GREATER ADOPTION

used our research to share a thorough description

Model developers should regularly employ redof our red team processes and results, as well as ourteaming as a way to identify unanticipated systemdataset of ~40,000 red team attacks for others toimpacts, quantify their magnitude and likelihood,analyze and learn from.1 We hope this transparency

and prioritize appropriate resources to mitigate

accelerates the ability of developers to uncoverthese risks before they arise in public settings.

harmful outputs of AI systems prior to deployment,

Today however, the research community lacks sharedand ultimately build safer systems. We look forward

norms, best practices, and technical standards forto partnering with interested stakeholders to build

how to safely and effectively red team AI systems.

upon this work and develop common red teaming
practices that can be widely adopted.

1 Anthropic. (2022, August). https://github.com/anthropics/hh-rlhfCONTACT: POLICY@ANTHROPIC.COM

SEPTEMBER 2022