What does this article say about copyright: Independent evaluation and red teaming are critical for identifying the risks posed by generative
AI systems. However, the terms of service and enforcement strategies used by prominent AI companies to deter model misuse have disincentives on
good faith safety evaluations. This causes some
researchers to fear that conducting such research
or releasing their findings will result in account
suspensions or legal reprisal. Although some companies offer researcher access programs, they are
an inadequate substitute for independent research
access, as they have limited community representation, receive inadequate funding, and lack
independence from corporate incentives. We propose that major AI developers commit to providing a legal and technical safe harbor, indemnifying public interest safety research and protecting
it from the threat of account suspensions or legal reprisal. These proposals emerged from our
collective experience conducting safety, privacy,
and trustworthiness research on generative AI systems, where norms and incentives could be better
aligned with public interests, without exacerbating model misuse. We believe these commitments
are a necessary step towards more inclusive and
unimpeded community efforts to tackle the risks
of generative AI.
1. Introduction
Generative AI systems have been deployed rapidly in recent
years, amassing hundreds of millions of users. These systems have already raised concerns for widespread misuse,
bias (Deshpande et al., 2023), hate speech (Douglas Heaven,
1MIT 2
Princeton University 3
Stanford University 4Georgetown
University 5AI Risk and Vulnerability Alliance 6Eleuther AI
7Brown University 8Carnegie Mellon University 9Virginia Tech
10Northeastern University 11UCSB 12University of Pennsylvania
13UIUC. * Corresponding author. ** Equal contribution. Correspondence to: Shayne Longpre <<PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>>.
2020), privacy concerns (Carlini et al., 2021; 2023), disinformation (Burtell & Woodside, 2023), self harm (Park
et al., 2023), copyright infringement (Henderson et al., 2023;
Gil et al., 2023), fraud (Stupp, 2019), weapons acquisition
(Boiko et al., 2023; Urbina et al., 2022), and the proliferation of non-consensual and abusive images (Lakatos, 2023;
Thiel et al., 2023), among others (Kapoor et al., 2024). To
ensure sufficient public scrutiny and accountability, such
high-impact systems should be evaluated (Liang et al., 2023;
Solaiman et al., 2023; Weidinger et al., 2023) by independent and external entities (Raji et al., 2022; Birhane et al.,
2024). Despite this, leading generative AI companies provide limited transparency and access into their systems,
with transparent audits showing only 25% of policy enforcement and evaluation criteria were satisfied on average
(Bommasani et al., 2023a); and with no company providing
reproducible evaluations to characterize the effectiveness of
their risk mitigations.
Leading AI companies’ terms of service prohibit independent evaluation into most sensitive model flaws (see Table 3).
While these terms act as a deterrent to malicious behavior,
they also restrict good faith research—auditors fear that releasing findings or conducting research could lead to their
accounts being suspended, ending their ability to do such
research, or even lawsuits for violating the terms of service.
Already, in the course of conducting good faith research,
researchers’ accounts have been suspended without warning,
justification, or an opportunity to appeal (Marcus & Southen,
2024). While some companies authorize selected research
through researcher access programs, their community representation remains limited and lacks independence from
corporate incentives such as favoritism towards researchers
aligned with the company’s values. Together, these observations stoke concerns that generative AI companies could
emulate the transparency and accountability challenges with
social media platforms—limiting researcher transparency
and access can mitigate dangerous headlines, public relations fallout, and lawsuits, but at the expense of public
interests (Abdo et al., 2022; DiResta et al., 2022).
As a group of researchers whose expertise spans AI red
teaming, safety, and evaluation, as well as privacy, security,
and the law, we have experienced first hand the negative
2 · A Safe Harbor for AI Evaluation and Red Teaming
TERMINOLOGY CONTEXT
Usage Policy A company’s usage policy dictates what uses of its AI systems are acceptable or unacceptable. Usage policies
generally prohibit inputs that elicit a range of undesirable model outputs, beyond what is already illegal. For
example, see Anthropic’s Acceptable Use Policy.
Terms of Service A company’s terms of service imposes legal rules on users of their services. Violations of the usage policy are
violations of the terms of service and can be enforced by terminating accounts or taking legal action.
Generative AI
Evaluation & Red
Teaming
In security fields, a red team refers to a group authorized to emulate an adversary’s attack against an organization’s
security systems. This term has been adopted by the AI community to instead describe penetration testing of a
broader set of system flaws than traditional security (The Hacking Policy Council, 2023). In this context, we are
referring to testing of released systems by third party ethical hackers, who may or may have explicit consent.
Safe Harbor A safe harbor is a measure to provide legal protection to hackers engaged in “good faith” research, abiding by
pre-agreed rules of engagement, or vulnerability disclosure policy (e.g. HackerOne (2023)).
Good Faith Research “Good faith security research means accessing a computer solely for purposes of good-faith testing, investigation,
and/or correction of a security flaw or vulnerability, where such activity is carried out in a manner designed to
avoid any harm to individuals or the public, and where the information derived from the activity is used primarily
to promote security or safety...” (Department of Justice, 2022). We generalize this definition to research beyond
security, including soliciting any unwanted behavior in the AI system normally disallowed by the company’s
usage policy, which we broadly refer to as “safety research” in this work.
Vulnerability Disclosure
Policy
A vulnerability disclosure policy establishes rules of engagement for third party ethical hackers. This includes
disclosure requirements for discovered vulnerabilities, but also other mandatory protocols (Bugcrowd, 2023).
Chilling Effects Chilling effects describe the inhibition or discouragement of important research, in this case due to a lack of
legal and technical protections, as well as uncertain norms around AI evaluation and red teaming.
Table 1. We define and contextualize the technical terminology used in this work, which is often used in other disciplines.
effects of legal uncertainty and technical barriers to conducting important research (Table 2). To improve the status quo,
we propose that generative AI companies commit to two
protections for independent public interest research. First,
AI companies should provide a legal safe harbor by offering legal protections for good faith research, provided it is
conducted in line with vulnerability disclosure policies (as
defined in Table 1). Second, companies should provide a
technical safe harbor, protecting safety researchers from
having their accounts subject to moderation or suspension.
These are fundamental access requirements for inclusive
evaluation of generative AI systems. Building on prior work
for algorithmic bug bounties (Elazari, 2018a; Kenway et al.,
2022; Raji et al., 2022) and social media data access (Abdo
et al., 2022), we recommend ways to implement these protections for independent AI evaluation without undermining
the processes that prevent model misuse. Specifically we
propose that companies delegate account authorization to
trusted universities or nonprofits, or provide transparent recourse for accounts suspended in the course of research.
These voluntary commitments align with the stated goals of
AI companies: to support wider participation in AI safety
research, minimize corporate favoritism, and encourage
community safety evaluations (see Appendix D). We hope
generative AI companies will adopt these commitments to
establish better community norms, improve trust in their
services, and bolster much needed AI safety in proprietary
systems.
2. Background & Motivations
Widely used online platforms can have significant socioeconomic impact (Zuboff, 2023; Horwitz et al., 2021). In
this section we highlight three reasons to motivate new protections for independent research into generative AI platforms:
1. Social media research has been burdened by a lack
of transparency and access, with a rise in legal repercussions for journalism and academic research (Abdo
et al., 2022; DeLong, 2021; Belanger, 2023).
2. There is growing concern that widespread risks of generative AI will impact a wider swathe of society. Fostering wider participation in AI evaluation will require
commitments to remove disincentives, obstacles, and
favoritism in researcher access.
3. Independent AI evaluation is increasingly vital to fair
assessments of AI risks, and informed policy debates.
We expand on each of these below, using terminology we
define in Table 1.
2.1. Avoiding the Fate of Social Media Platforms
Prominent social media platforms block researcher access to the detriment of public interests. Civil societies
and researchers argue that social media companies have
systematically limited researcher access to their platforms,
restricting journalism and creating a chilling effect on critical public interest research (DiResta et al., 2022; Mozilla,
3 · A Safe Harbor for AI Evaluation and Red Teaming
2023; Boyd et al., 2021; Persily, 2021). Specifically, platforms wield their terms of service to gatekeep access to publicly posted data and limit negative public exposure from
independent research. Abdo et al. (2022) argue for “a safe
harbor for platform research,” which would include legal
provisions that protect researchers and journalists. In the
absence of such provisions, researchers have reported platform gatekeeping, account suspensions, cease-and-desist
letters and general fears of liability in the course of public
interest research, which have resulted in chilling effects (DeLong, 2021; Barclay, 2021; Belanger, 2023). The computer
and internet security fields have also seen contentious legal
threats and lawsuits against academics (Greene, 2001; Brodkin, 2021; dis, 2021), resulting in new guidelines from the
United States Department of Justice that “good-faith security research should not be charged” (Department of Justice,
2022). Companies building generative AI models have the
opportunity to protect good faith research before harm from
their systems becomes as widespread as that from social
media.
Conducting research on generative AI comes with additional challenges compared to social media. Compared
to past digital technologies, prominent models require accounts to be used (unlike search engines), and their outputs
are not publicly visible (unlike posts on many social media
platforms) (Narayanan & Kapoor, 2023b). These factors
provide developers with comparatively greater control over
who accesses their systems, which could exacerbate gatekeeping. The lack of transparency from top developers compounds this issue, with little information available about
how and where generative AI systems are used, and to what
end (Bommasani et al., 2023b). For external researchers,
the models themselves are also black boxes, as developers
often do not disclose model architectures, sizes, or training
data. This limits independent research to evaluate the risks,
capabilities, safety, and societal impact of generative AI
(Casper et al., 2024).
2.2. The Importance of Independent AI Evaluation
Concerns over the risks and harms of generative AI are
mounting. Today, AI systems like ChatGPT have amassed
over 100 million weekly users (Hu, 2023), exceeding the
growth rate of social media platforms. Generative AI systems have already exhibited “unsafe” behavior—generating
highly undesirable and even illegal content—attracting regulatory attention as a result. More specifically, generative
AI systems can generate toxic content (Deshpande et al.,
2023), libel, hate speech (Douglas Heaven, 2020), and privacy leaks (Carlini et al., 2021; 2023; Li et al., 2023a; Huang
et al., 2023b; Nasr et al., 2023). They have also been used
to scale disinformation (Burtell & Woodside, 2023), fraud
(Stupp, 2019; Commission, 2023), malicious tool usage
(Li et al., 2023b; Pa Pa et al., 2023; Renaud et al., 2023),
copyright infringement (Henderson et al., 2023; Gil et al.,
2023; Jonathan, 2023; Shi et al., 2024; Longpre et al., 2023),
non-consensual intimate imagery (Lakatos, 2023), and child
sexual abuse material (Thiel et al., 2023), as well as provide
instructions for self-harm (Park et al., 2023; Xiang, 2023),
acquiring weapons (Boiko et al., 2023; Nelson & Rose,
2023), and building weapons of mass destruction (Urbina
et al., 2022; Soice et al., 2023). At the extreme end, even
CEOs of AI model developers have speculated generative AI
will upend labor markets (Suleyman & Bhaskar, 2023) and
even pose more severe risks (Barrabi, 2023; Hendrycks et al.,
2023). These wide ranging concerns, from the developers
themselves, motivate the need for protected independent
access.
Independent AI evaluation and red teaming are crucial
for uncovering vulnerabilities, before they proliferate.
Independent researchers often evaluate or “red team” AI
systems for a broad range of risks. “Red teaming”, a subset
of evaluation, has been adopted by the AI community as a
term of art to describe these evaluations aimed at uncovering
pernicious system flaws. In this work, we refer specifically
to red teaming of publicly released AI systems (rather than
pre-release testing), by external researchers, rather than internal teams. Some companies do also provide internal or
by-invitation pre-release red teaming, e.g. OpenAI. While
all types of testing are critical, external evaluation of AI
systems that are already deployed is widely regarded as essential for ensuring safety, security, and accountability (Kenway et al., 2022; Anderljung et al., 2023; ?). Post-release,
external red-teaming research has uncovered vulnerabilities
related to low resource languages (Yong et al., 2023), conjugate prompting attacks (Kotha et al., 2023), adversarial
prompts (Maus et al., 2023; Zou et al., 2023; Robey et al.,
2023), generation exploitation attacks (Huang et al., 2023a),
persuasion attacks (Xu et al., 2023; Zeng et al., 2024), a
wide range of jailbreaks (Wei et al., 2023; Shen et al., 2023;
Liu et al., 2023; Zou et al., 2023; Shah et al., 2023), text-toimage vulnerabilities (Parrish et al., 2023), automatic red
teaming (Ge et al., 2023; Yu et al., 2023; Chao et al., 2023;
Zhao et al., 2024), and undetectable methods for fine-tuning
away safety mitigations within the platform APIs (Qi et al.,
2023; Yang et al., 2023; Zhan et al., 2023). See Appendix E
for additional examples. These works illustrate how such
research benefits AI companies: the research community assists in-house research teams by uncovering vulnerabilities,
sharing findings and data, before systems cause major harm.
Independent AI evaluation provides impartial perspectives, that are necessary for informed regulation. As
the above examples have shown, independent research has
uncovered unexpected flaws, aiding company efforts, and
expanding the collective knowledge around both vulnerabilities and defenses. These findings have informed the policy
and regulatory discussions, including around the types of
4 · A Safe Harbor for AI Evaluation and Red Teaming
THEME OBSERVATIONS
Chilling Effect on
Research
Safety research can result in companies suspending researcher access, citing terms of service violations. This
can have broad chilling effects as access is critical for the other work conducted by these researchers. Many
researchers only begin their work after observing first-movers, and scope their practices to emulate those
precedents. As a result, vital safety research may be delayed or circumscribed due to uncertainty and caution
over account moderation outcomes.
Chilling Effect on
Vulnerability
Disclosure
It is unclear whether and how researchers should publicly release their findings, methods or the exploits
themselves. In the absence of explicit guidance, they may be too broad or too limited in how they share their
results, to the detriment of the community—for instance, by only sharing findings with a small group of other
researchers, such as close personal contacts. When researchers are overly cautious in sharing their work it
frequently results in siloed research that is less reproducible, or delayed disclosure, especially around the most
sensitive findings, which could be to the detriment of public awareness.
Incentives to Tackle the
Wrong Problems
There is an incentive to prioritize less important risks as the focus of safety work both for the uncertainty of
repercussions from the companies or community. For instance, researchers might choose to investigate more
benign prompt attacks rather than more offensive or dangerous attacks, such as focusing on text rather than more
evocative visual outputs, or tool usage.
Favoritism and Imbalanced
Representation
Admission into researcher access programs and favorable responses to safety work can be dependent on
connections to the companies. For instance, there is a strong impression that access to OpenAI employees
improves access to their programs. External researchers who are not already well connected may not hear back
at all from their applications or receive any justification for rejection, as no obligation currently exists on the part
of AI companies. Part of this may be due to companies being backlogged with applications from researchers,
having dedicated few resources to this task. Costanza-Chock et al. (2022) point out the problems introduced by
imbalanced auditor representation.
Unclear &
Undefined Norms
Impressions of basic norms and expectations vary widely, including with respect to appropriate threat models,
whether and when to notify companies in advance of publication, what forms of red teaming are acceptable,
and whether to release findings, methods, or prompts at all, or how to do so responsibly. Additionally, the type
of API access, moderation policies, disclosure processes, and even the likelihood of response to a disclosure
vary dramatically by company, leaving researchers without well-defined protocols that would enable them to
confidently conduct important safety and security work.
A Choice Between
Open and Closed
Access
Researchers prefer to red team deployed systems that have millions of users and therefore pose immediate
risks. However, effective and rigorous research requires deep access to the model (Casper et al., 2024), which
proprietary systems rarely provide. As Friedler et al. (2023) have noted, “for red-teaming conducted by external
groups to be effective, those groups must have full and transparent access to the system in question.” In particular,
researchers often require finer-grained access to internal model representations (e.g. “logits”), access to both the
base and aligned model, and continual access to a static model, without its API changing or becoming deprecated.
Additionally, the underlying source of moderation in closed systems is difficult to diagnose: did the moderation
endpoint catch an inappropriate user input, did the model itself abstain from answering, or did the user interface
curtail an inappropriate response?
Table 2. Themes and observations attributed to informal discussions among authors and colleagues working on AI evaluation and
red teaming. We describe the main challenges to conducting rigorous evaluations of widely used generative AI systems.
model vulnerabilities, and their comparative safety of open
and closed foundation models (Narayanan & Kapoor, 2023a;
Lambert, 2023). However, as we shall see, it isn’t clear that
we are seeing the full benefits from a thriving red teaming
ecosystem (Section 3).
Without robust independent evaluation, companies’ own developer safety teams may not be sufficiently large or diverse
to fully represent the diversity of global users their products
already serve, and the scale of risks they have acknowledged (Costanza-Chock et al., 2022). While companies do
invite third-party evaluators, there are well known conflicts
of interest without independence in the auditor selection
process (Moore et al., 2006). As the Ada Lovelace Institute
and another dozen civil societies remarked at the recent AI
Safety Summit in the UK, “Companies cannot be allowed
to assign and mark their own homework. Any research efforts designed to inform policy action around AI must be
conducted with unambiguous independence from industry
influence” (Ada Lovelace Institute, 2023).
3. Challenges to Independent AI Evaluation
We first discuss the mixed incentives and uncertainty faced
by red teaming researchers, followed by analysis of the
existing researcher protections, access programs, and their
limitations.
AI Companies’ Terms of Service discourage communityled evaluations. Many of the findings from the model
vulnerability research mentioned in Section 2.2, such as
jailbreaks, bypassing safety guardrails, or text-to-image exploits, are legally prohibited by the terms of service for
popular systems, including those of OpenAI, Google, An-
5 · A Safe Harbor for AI Evaluation and Red Teaming
thropic, Inflection, Meta, Midjourney, and others. While
these terms are intended as a deterrent against malicious
actors, they also inadvertently restrict safety and trustworthiness research—both by forbidding the research, and enforcing it with account suspensions. While platforms enforce
these restrictions to varying degrees, the terms disincentivize
good faith research by granting developers the right to terminate researchers’ accounts (without appeal or justification)
or even take legal action against them. The risk of losing
account access may dissuade many researchers altogether,
as these accounts are critical for a range of vulnerability and
other AI research.
AI developers’ documentation often purports to support independent research; however, it does not clearly state the
conditions under which evaluation and red teaming would
not violate the usage policy, leaving researchers uncertain
as to whether or how they should conduct their research. In
Table 2, we share common themes attributed to discussions
between ourselves and colleagues, summarizing their experiences conducting evaluation and red teaming research on
generative AI platforms. These themes reflect an imperfect
sample: they are skewed in that they represent the opinions
of researchers who chose to conduct safety research, excluding those who chose not to, lacked access to the companies
they would have evaluated, or were deterred for uncertainty
of legal liability.
Independent AI evaluation is largely inconsistent,
opaque, and challenging across companies. From our experience and discussions, the bulk of this research is concentrated on Meta models like Llama-2 (Touvron et al., 2023),
or OpenAI models like ChatGPT (OpenAI, 2023a). Llama
models are popular as they have downloadable weights,
allowing a researcher to red team locally without having
their account terminated for usage policy violations. OpenAI models are popular as they are accessible via API, are
highly performant, and have widespread public use. While
many researchers are tentative about red teaming OpenAI,
usage policy enforcement is often lax. However, account
suspensions in the course of public interest research have
taken place, to our knowledge, for each of OpenAI, Anthropic, Inflection, and Midjourney, with Midjourney being
the most prolific. We withhold details on most of these
to respect the anonymity of researchers. As one example,
independent evaluation by an artist found Midjourney has
a “visual plagiarism problem” (Marcus & Southen, 2024).
This resulted in their account being repeatedly suspended
without warnings or justification. The cost of suspensions
without refunds quickly tallies to hundreds of dollars, and
creating new accounts is also not trivial, with blanket bans
on credit cards and email addresses.
AI companies have begun using their terms of service to
deter analysis, particularly into copyright claims. Midjourney updated its Terms of Service to include penalties such
as account suspension or legal action for conducting such
research.1 Midjourney’s Terms of Service states: “If You
knowingly infringe someone else’s intellectual property, and
that costs us money, we’re going to come find You and collect that money from You. We might also do other stuff, like
try to get a court to make You pay our legal fees. Don’t do
it” (Midjourney, 2023).2 Llama 2’s license will also terminate access if model outputs are used as part of intellectual
property litigation.3
Our analysis of company policies in Table 3 shows not all
companies disclose their enforcement process (the mechanisms for identifying and enforcing violations of the usage
policy). Google and Inflection are the only companies to
provide the user any form of justification on how the usage
policy is enforced. And, only for OpenAI, Inflection, and
Midjourney did we find evidence of an enforcement appeals
process. Without additional information on how companies
enforce their policies, researchers have no insight into enforcement appeals criteria, or whether companies reinstate
public interest research post-hoc.
Existing safe harbors protect security research but not
other good faith research. AI developers have engaged
to differing degrees with external red teamers and evaluators. OpenAI, Google, Anthropic, and Meta, for example,
have bug bounties, and even safe harbors. However, companies like Meta and Anthropic currently “reserve final and
sole discretion for whether you are acting in good faith and
in accordance with this Policy”. They may revoke access
rights to models, even open models like Llama 2 (Touvron
et al., 2023), or hold the researchers legally accountable, at
their discretion. This leaves clear ways to stifle and deter
good faith research. Additionally, these safe harbors are
tightly-scoped to traditional security issues like unauthorized account access.4 Developers disallow other model
flaws named in their usage policies, including, “adversarial
testing” (Anthropic, 2023), “jailbreaks”, bypassing safety
guardrails, or generating hate speech, misinformation, or
abusive imagery.
Among other safety research commitments, some companies publish reports on internal evaluation efforts, while
others selectively invite third parties to participate in prerelease red teaming, or have researcher access programs for
1
See https://twitter.com/Rahll/status/
1739155446726791470
2
See Section 10 https://docs.midjourney.com/
docs/terms-of-service
3
See Section 5c: https://ai.meta.com/llama/
license/
4OpenAI expanded its safe harbor to include “model vulnerability research” and “academic model safety research” in response
to an early draft of our proposal, though some ambiguity remains
as to the scope of protected activities.
6 · A Safe Harbor for AI Evaluation and Red Teaming
AI Company AI System Public API / Open
Deep Access
Researcher Access
Bug Bounty
Safe Harbor
Enforcement Process
Enforcement Justification
Enforcement Appeal
OpenAI GPT-4 H# H#† # H#
Google Gemini # # # # H# #
Anthropic Claude 2 # # H# # H#‡ # #
Inflection Inflection-1 # # # # # # H# H#
Meta Llama 2 H#‡ # # #
Midjourney Midjourney v6 # # # # # # # H#
Cohere Command # # H# # # #
Table 3. A summary of the policies, access, and enforcement for major AI systems, suggesting a challenging environment for
independent AI research. We catalog if each system has a public API, deeper access than final outputs (e.g. top-5 logits for OpenAI),
researcher access programs, security research bug bounties, any legal safe harbors, and whether they disclose their account enforcement
process, disclose justification on enforcement actions, and have an enforcement appeals process. indicates the company satisfies this
criteria; # indicates it does not, and H# indicates partial satisfaction. ‡
Indicates security-only research safe harbors, “solely at [their]
discretion”. †
Indicates a safe harbor for security and “academic research related to model safety”. The latter was added by OpenAI in
response to reading an early draft of this proposal, though some ambiguity remains as to the scope of protected activities. Full details are
provided in Table A1.
deeper access to released models. These are laudable initiatives, especially when they are accompanied by subsidized
credits for researchers (OpenAI, 2024). Nonetheless, these
measures leave significant gaps in the ecosystem for independent evaluations. Reports on internal red teaming are
often largely irreproducible and generate limited trust due to
mismatched corporate incentives (e.g. Anthropic (2023b)).
Invitations to third-party researchers are limited and can be
self-selecting. And researcher access programs, if available,
often do not notify researchers of rejections and thus create
an environment of uncertainty (Bommasani et al., 2023b).
Researchers have argued that a patchwork of policies like
these can create a veneer of open and responsible research,
without lifting other obstacles for participatory research
(Krawiec, 2003; Zalnieriute, 2021; Whittaker, 2021).
Companies should take steps to facilitate independent AI
evaluation and reduce the fear of reprisals for safety research.
The gaps in the policy architectures of leading AI companies, depicted in Table 3 force well-intentioned researchers
to either wait for approval from unresponsive access programs, or risk violating company policy and potentially
losing access to their accounts. The net result is a situation
akin to companies gatekeeping access to their platforms
and thereby restricting the scope of safety research, whether
intentional or not. This research environment can limit
the diversity and representation in evaluation, ultimately
stymieing public awareness of risks to AI safety.
4. Safe Harbors
We believe that a pair of voluntary commitments could significantly improve participation, access, and incentives for
public interest research into AI safety. The two commitments are: (i) a legal safe harbor, protecting good faith,
public interest evaluation research provided it is conducted
in accordance with well established security vulnerability
disclosure practices, and (ii) a technical safe harbor, protecting this evaluation research from account termination;
summarized in Figure 1. Both safe harbors should be scoped
to include research activities that uncover any system flaws,
including all undesirable generations currently prohibited
by the usage policy. As we shall argue later, this would not
inhibit existing enforcement against malicious misuse, as
protections are entirely contingent on abiding by the law
and strict vulnerability disclosure policies, determined ex
post. Existing safe harbor resources (Etcovich & van der
Merwe, 2018; Pfefferkorn, 2022; HackerOne, 2023), and
vulnerability disclosure policies (Blog, 2010; Bugcrowd,
2023) provide grounding for these proposals. In particular, Elazari (2018b; 2019); Akgul et al. (2023); Kenway
et al. (2022) discuss the implementations of algorithmic bug
bounties, Walshe & Simpson (2023) note ambiguities on
formal constraints, and Raji et al. (2022) explore governance
for third-party AI audits, including legal protections for researchers. The legal safe harbor, similar to the proposal by
Abdo et al. (2022) for social media platforms, would safeguard certain research from some amount of legal liability,
mitigating the deterrent of strict terms of service and the
threat that researchers’ actions could spark legal action by
companies (e.g. under US laws such as the CFAA or DMCA
7 · A Safe Harbor for AI Evaluation and Red Teaming
Section 1201). The most important condition of a legal safe
harbor is the determination of acting in good faith should
not be “at the sole discretion” of the companies, as Meta and
Anthropic have currently defined it. The technical safe harbor would limit the practical barriers erected by usage policy
enforcement, with consistent and broader community access
for important, public interest research. Together these steps
would reduce the legal and practical obstacles to conducting
independent evaluation and red teaming research.
4.1. A Legal Safe Harbor
A legal safe harbor could mitigate risks from civil litigation, providing assurances that AI platforms will not sue
researchers if their actions were taken for research purposes. Take, for example, the U.S. legal regime, which
governs many of the world’s leading AI developers. The
Computer Fraud and Abuse Act (CFAA), which allows for
civil lawsuits for accessing a computer without authorization or exceeding authorized access (CFAA, 1986), could
be used by AI developers to sue researchers for accessing
their models in a way that was unintended, though there
are complexities to the legal analysis for adversarial attacks
on AI models (Evtimov et al., 2019). Section 1201 of the
Digital Millennium Copyright Act (DMCA) allows for civil
lawsuits if researchers circumvent technological protection
measures (TPMs), which effectively control access to works
protected by copyright (DMCA, 1998a). These risks are not
theoretical; security researchers have been targeted under
the CFAA (Pfefferkorn, 2021), and DMCA § 1201 hampered security researchers to the extent that they requested
a DMCA exemption for this purpose (Colannino, 2021).
Already, in the context of generative AI, OpenAI has attempted to dismiss the New York Times v OpenAI lawsuit
(Grynbaum & Mac, 2023) on the allegation that New York
Times research into the model constituted hacking (Brittain, 2024). Relatedly, a petition for an exemption to the
DMCA has been filed requesting that researchers be allowed
to investigate bias in generative AI systems (Weiss, 2023).
Abdo et al. (2022) argue a safe harbor is oriented around
conditions of access, rather than who gets access. The protections apply only to parties who abide by the rules of
engagement, to the extent they can subsequently justify
their actions in court. Typically, responsible vulnerability
disclosure policies impose strict criteria for when the vulnerability should be disclosed, how long before it can be
released to the public, privacy protection rules, and other
criteria for the most dangerous exploits. Research that strays
from those reasonable measures, or is already illegal, would
not succeed in claiming those protections in an ex post investigation. As such, malicious use would remain legally
deterred, and platforms would still be obligated to prevent
misuse. Abdo et al. (2022) argue a safe harbor designed in
this way, based on ex post researcher conduct, would not
enable malicious use any more than in its absence. Nor
would it alter platforms’ obligations to protect their users
against third parties or from enforcing malpractice.
Companies’ legal safe harbors would protect researchers
from civil liability, not criminal liability. Knowingly querying a model to generate certain types of content, whether for
red teaming or not, can be illegal in certain jurisdictions—
particularly in the case of image- or video-generation systems (Gupta, 2024). Moreover, certain violations of DMCA
§ 1201, particularly those that are committed “willfully and
for purposes of commercial advantage or private financial
gain,” can lead to criminal liability (DMCA, 1998b), as can
many violations of the CFAA. We would recommend governments provide clear guidelines and, where appropriate,
safe harbors for safe and responsible red teaming of illegal
content generated by models. Such safe harbors against
criminal conduct may need to be codified into statute in order to be guaranteed. However, they could be implemented
by statements of policy, for example, such as when the
Department of Justice issued a new policy in 2022 stating
that “good-faith security research should not be charged”
(Department of Justice, 2022).
The US Executive Order on AI directs the National Institute
of Standards and Technology (NIST) to establish guidelines for conducting red-teaming and assessing the safety of
foundation models (Executive Office of the President, 2023).
Standardizing a legal safe harbor for researchers would complement NIST’s comprehensive AI evaluation agenda and
its AI Risk Management Framework (NIST, 2024; Tabassi,
2023). The US AI Safety Institute Consortium, a publicprivate research collaboration, could be used to promote the
adoption of safe harbors among companies (NIST, 2023).
4.2. A Technical Safe Harbor
Legal safe harbors still do not prevent account suspensions
or other enforcement action that would impede independent
safety and trustworthiness evaluations. Without sufficient
technical protections for public interest research, a mismatch can develop between malicious and non-malicious
actors since the latter are discouraged from investigating
vulnerabilities exploited by the former. We propose companies offer some path to eliminate these technical barriers for
good faith research. This would include more equitable opportunities for researcher access, and guarantees that those
opportunities will not be foreclosed for researchers who
adhere to companies’ guidelines.
The challenge with implementing a technical safe harbor is
distinguishing between legitimate research and malicious
actors, without notable costs to developers. An exemption
to usage moderation may need to be reviewed in advance, or
at least when an unfair account suspension occurs. However,
we believe this problem is tractable, and offer recommen-
8 · A Safe Harbor for AI Evaluation and Red Teaming
Company Commitment: Legal Safe Harbor
Commitment – We will not threaten or bring any legal action against anyone conducting good faith research who complies with the rules of
engagement set out in our vulnerability disclosure policy. As long as you comply with our policy:
❖ We will not make any claim under the DMCA, for circumventing technological measures to protect the services eligible under this policy.
❖ We consider your security research to be "authorized" under the Computer Fraud and Abuse Act (and/or similar state laws).
❖ We waive any restrictions in our applicable Terms of Use and Usage Policies that would prohibit your participation in this policy, but only for
the limited purpose of your model research under this policy.
❖ We will take steps to make known that you conducted good faith research if someone else brings legal action against you.
Company Commitment: Technical Safe Harbor
Commitment – We will make all reasonable efforts to not penalize user accounts engaged in good faith research into our systems, as long as
they comply with the rules of engagement set out in our vulnerability disclosure policy.
❖ We shall not limit research on the basis that it may be against the interests of our company.
❖ We shall offer a research access program that involves independent, transparent, and timely review into research proposals.
❖ We shall offer a transparent appeals and review process if an account is restricted for alleged misuse (e.g. account suspension).
❖ We shall reinstate researchers’ accounts in the event that of good faith research initiatives are found to have been penalized.
Good Faith Researcher Commitments
Scope of Research – Investigation into behavior of the AI system, including those disallowed by the acceptable usage policy.
Researcher Responsibilities – All responsibilities, such as those already encoded in a company’s Rules of Engagement for security research
continue to apply. These responsibilities include, but are not limited to:
❖ In-scope: Test only in-scope systems and respect out-of-scope systems.
❖ Vulnerability disclosure: Promptly report discovered vulnerabilities. Keep vulnerability details confidential if releasing them violates the law,
or until a pre-agreed period of time after the vulnerability is reported (usually 90 days).
❖ Harms to users and systems: Refrain from violating privacy, disrupting systems, destroying data, or harming user experience.
❖ Privacy requirements: Do not intentionally access, modify, or use data belonging to others, including confidential data. If a vulnerability
exposes such data, stop testing, submit a report immediately, and delete all copies of the information.
Figure 1. A summary of the suggested mutual commitments and scope of a legal safe harbor, and technical safe harbor. These
commitments extend existing safe harbors for security research as well as researcher access programs, and are written in the context of US
laws. For a wider list of common researcher responsibilities consider OpenAI’s Rules of Engagement.
dations, grounded in prior proposals. First, we discuss how
to scale up participation by delegating responsibilities to
trusted independent third parties to pre-review researcher
access. Then we discuss how an independently reviewed
and transparent account suspension appeals process could
enable fairer post-review to researcher access. Independent
review and scaling participation are staples of both options.
Independent third parties like universities or NAIRR
can scale participation in AI evaluation, without misaligned corporate incentives. To facilitate more equitable
access, and reduce the potential for corporate favoritism,
we propose the responsibility of access authorization be
delegated to trusted third parties, such as universities, government, or civil society organizations. The U.S. National
Artificial Intelligence Research Resource (NAIRR) offers a
suitable vehicle for a pilot of this approach as it already partners with and shares resources between AI developers and
nonprofits. AI developers provide resource credits through
NAIRR, and OpenAI has called for wider participation: “by
providing broader access to essential tools and data, we
are opening doors for a diverse range of talents and ideas,
furthering innovation and ensuring that AI development continues to be a force for the greater good” (National Science
Foundation, 2024).
A similar approach has already been adopted to provide independent access to Meta’s social media user data, with
the University of Michigan as the trusted intermediary
(Gonzalez-Bail ´ on et al. ´ , 2023). This solution scales, with
partner organizations likely to aid in access review in exchange for wider participation in AI red teaming. It also
effectively diverts responsibility from corporate interests to
organizations already invested in fair, responsible, and accountable AI research. These partnerships do not require AI
developers to fully relinquish access control but are a meaningful step in facilitating more equitable access without
stretching their own resources. Each partner organization’s
API usage could be traced to their API keys—essentially a
“researcher API”. Organizations would have autonomy to
authorize their own network of researchers, but would be
responsible for any misuse tied to their API keys.
A number of similar proposals, discussed in Section 5, have
been made for independent researcher access, like structured access or review boards, both of which would delegate
the responsibility of access selection to independent third
9 · A Safe Harbor for AI Evaluation and Red Teaming
parties. While this approach scales well and adopts independent access privileges, it can have severe limitations if
AI companies only select a very finite set of partners, or
choose to exclude more critical organizations. As a start,
we recommend allowing NAIRR to help formulate the partner network, to include a set of trusted international academic organizations, as well as nonprofits in NAIRR such as
AI2, EleutherAI, and MLCommons. Already these changes
would make significant strides in expanding access through
independent review.
Transparent access and appeals processes can improve
community trust. Some generative AI companies may be
unwilling to share access authorization more widely. There
is a clear alternative: commit to a transparent access appeals
process that makes decision criteria and outcomes visible
to the wider community. Ideally, this process would be
reviewed independently, perhaps with the help of NAIRR
partner organizations. Whenever public interest evaluation
research is suspended, researchers should have the opportunity to appeal the decision under a technical safe harbor.
Companies can adopt an access process with clearly codified
selection criteria, guaranteeing they will respond to applicants within a certain period of time, with a justification
for the outcome decision. While this would not address the
need for additional resources, it would provide the AI community with significantly greater visibility into companies’
decisions to grant access, and allow the community to apply
collective pressure against any attempt to restrict legitimate
research. The common denominator between pre-review
and post-review technical safe harbors, described above,
is providing a fair process to enable good faith research
without the fear of unjustified account suspensions. In Appendix C we sketch an implementation of a pre-registration
and appeals process, based on existing researcher access
programs, that could facilitate implementation of a technical
safe harbor.
There are many dimensions of improving researcher access,
including earlier access, deeper access, and subsidized access. The technical safe harbor described is a precondition
for more independent and broader participation across all
these axes, should companies offer earlier, deeper, or subsizided access. While efforts by AI companies to broaden
safety research, such as accepting community applications
for pre-release red-teaming and subsidizing such research
with compute credits are useful first steps, the safe harbors
we propose would strengthen broader research protections
while being more independent of AI companies’ control.
5. Related Proposals
Our proposals for legal and technical safe harbors build
on prior calls to expand independent access for AI evaluation, red teaming, and safety research. The Hacking Policy
Council (2023) has proposed that governments “clarify and
extend legal protections for independent AI red teaming,”
similar to our voluntary legal safe harbor proposal. The
Council stated, “the same industry norms on providing time
to mitigate before public disclosure, and avoiding retaliation for good faith disclosures, should eventually apply to
AI misalignment disclosures as they do for security vulnerability disclosures.” The Algorithmic Justice League
has advocated for vulnerability disclosure for algorithmic
harms, calling for independent algorithmic audits involving
impacted communities (Costanza-Chock et al., 2022; Kenway et al., 2022). Moreover, AI Village hosts events where
large groups of independent researchers red team generative
AI models for a wide range of vulnerabilities (Sven Cattell,
2023). An array of researchers have recommended additional external scrutiny of the emerging risks and overall
safety of frontier AI models to “improve assessment rigor
and foster accountability to the public interest” (Anderljung
et al., 2023). Bucknall & Trager (2023) have also proposed
structured access for third party research via a dedicated
research access API, with third-party independent review.
Stanford’s Center for Research on Foundation Models has
proposed an independent Foundation Models Review Board
to moderate and review requests for deeper researcher access to foundation models (Liang et al., 2022).
Governments have also suggested the need for independent
evaluation and red teaming. The US Office of Management and Budget’s Proposed Memorandum on Advancing
Governance, Innovation, and Risk Management for Agency
Use of Artificial Intelligence encourages federal agencies
to consider as part of procurement contracts for generative AI systems “requiring adequate testing and safeguards,
including external AI red teaming, against risks from generative AI such as discriminatory, misleading, inflammatory,
unsafe, or deceptive outputs” (United States Office of Management and Budget, 2023). The EU AI Act states that
providers of general-purpose AI models with systemic risks
must share a “detailed description of the measures put in
place for the purpose of conducting internal and/or external
adversarial testing (e.g. red teaming), model adaptations, including alignment and fine-tuning” to the EU as part of their
technical documentation (European Council, 2024; Hacker,
2023). In addition, Canada’s Voluntary Code of Conduct
on the Responsible Development and Management of Advanced Generative AI Systems includes a commitment that
developers will “conduc[t] third-party audits prior to release”
(Innovation, Science and Economic Development Canada,
2023).
6. Conclusion
The need for independent AI evaluation has garnered significant support from academics, journalists, and civil society.
10 · A Safe Harbor for AI Evaluation and Red Teaming
Examining challenges to external evaluation of generative
AI systems, we identify legal and technical safe harbors as
minimum and fundamental protections. We believe they
would significantly improve norms in the ecosystem and
drive more inclusive community efforts to tackle the risks
of generative AI