1

Withdrawn Draft

2
3

Warning Notice
The attached draft document has been withdrawn, and is provided solely for historical purposes.
It has been superseded by the document identified below.
Withdrawal Date July 26, 2024
Original Release Date April 29, 2024

4
5
6Superseding Document
Status Final
Series/Number NIST AI 600-1
Title Artificial Intelligence Risk Management Framework: Generative
Artificial Intelligence Profile
Publication Date July 2024
DOI https://doi.org/10.6028/NIST.AI.600-1

7
8
9
10

1

NIST AI 600-1

2

Initial Public Draft

3

4
5
6
7

Artificial Intelligence Risk
Management Framework:
Generative Artificial Intelligence
Profile

8
9
10
11
12

This publication is available free of charge from:[DOI link TK]

13April 2024

14
15
16
17
18
19
20
21
22

23
ii

1

NIST AI 600-1

2

Initial Public Draft

3

4
5
6

Artificial Intelligence Risk Management
Framework: Generative Artificial
Intelligence Profile

789This publication is available free of charge from:

10[DOI link TK]

11
12

13
14
15
16
17
18
19
20
21
iii

NIST makes the following notes regarding this document:•

NIST plans to host this document on the NIST AIRC once final, where organizations can query
actions based on keywords and risks.

NIST specifically welcomes feedback on the following topics:•••

Glossary Terms: NIST will add a glossary to this document with novel keywords. NIST
welcomes identification of terms to include in the glossary.
Risk List: Whether the document should further sort or categorize the 12 risks identified (i.e.,
between technical / model risks, misuse by humans, or ecosystem / societal risks).
Actions: Whether certain actions could be combined, condensed, or further categorized; andfeedback on the risks associated with certain actions.

Comments on NIST AI 600-1 may be sent electronically to NIST-AI-600-1@nist.gov with “NIST AI 600-1”
in the subject line or submitted via www.regulations.gov (enter NIST-2024-0001 in the search field.)
Comments containing information in response to this notice must be received on or before June 2,2024, at 11:59 PM Eastern Time.
1

iv

1

Table of Contents

2

1. Introduction ........................................................................................................................................................ 1

3
4

2. Overview of Risks Unique to or Exacerbated by GAI ............................................................................ 2

5
6
7
8
9
10
11
12
13
14
15
16
17

3. Actions to Manage GAI Risks ...................................................................................................................... 10

18

Appendix A. Primary GAI Considerations ........................................................................................................ 62

19

Appendix B. References .................................................................................................................................... 69

20
21
22
23
24
25

Acknowledgments: This report was accomplished with the many helpful comments and contributions
from the community, including the NIST Generative AI Public Working Group, and NIST staff and guest
researchers: Chloe Autio, Patrick Hall, Shomik Jain, Reva Schwartz, Martin Stanley, Kamie Roberts, andElham Tabassi.

26
27
28
29
30
31
32

Disclaimer: Certain commercial entities, equipment, or materials may be identified in this document inorder to adequately describe an experimental procedure or concept. Such identification is not intended to
imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it
intended to imply that the entities, materials, or equipment are necessarily the best available for the
purpose. Any mention of commercial, non-profit, academic partners, or their products, or references isfor information only; it is not intended to imply endorsement or recommendation by any U.S.
Government agency.

v

1

1.

2
3
4
5
6

This document is a companion resource for Generative AI1 to the AI Risk Management Framework (AIRMF), pursuant to President Biden’s Executive Order (EO) 14110 on Safe, Secure, and Trustworthy
Artificial Intelligence.2 The AI RMF was released in January 2023, and is intended for voluntary use and to
improve the ability of organizations to incorporate trustworthiness considerations into the design,development, use, and evaluation of AI products, services, and systems.

7
8
9
10

This companion resource also serves as both a use-case and cross-sectoral profile of the AI RMF 1.0.
Such profiles assist organizations in deciding how they might best manage AI risk in a manner that iswell-aligned with their goals, considers legal/regulatory requirements and best practices, and reflects
risk management priorities.

11
12
13
14
15

Use-case profiles are implementations of the AI RMF functions, categories, and subcategories for a
specific setting or application – in this case, Generative AI (GAI) – based on the requirements, risk
tolerance, and resources of the Framework user. Consistent with other AI RMF Profiles, this profile offers
insights into how risk can be managed across various stages of the AI lifecycle and for GAI as atechnology.

16
17
18
19

As GAI covers risks of models or applications that can be used across use cases or sectors, this document
is also an AI RMF cross-sectoral profile. Cross-sectoral profiles can be used to govern, map, measure, andmanage risks associated with activities or business processes common across sectors such as the use oflarge language models, cloud-based services, or acquisition.

20
21
22
23
24
25

This work was informed by public feedback and consultations with diverse stakeholder groups as part of
NIST’s Generative AI Public Working Group (GAI PWG). The GAI PWG was a consensus-driven, open,transparent, and collaborative process facilitated via virtual workspace to obtain multistakeholder input
and insight on GAI risk management, and inform NIST’s approach. This document was also informed bypublic comments and consultations as a result of a Request for Information (RFI) and presents
information in a style adapted from the NIST AI RMF Playbook.

26
27
28

Introduction

About this Profile
This profile defines a group of risks that are novel to or exacerbated by the use of GAI. These risks were
likewise identified by the GAI PWG:

29

1. CBRN Information

1

Generative AI can be defined by EO 14110 as “the class of AI models that emulate the structure andcharacteristics of input data in order to generate derived synthetic content. This can include images, videos, audio,
text, and other digital content.” While not all GAI is based in foundation models, for purposes of this document,
GAI generally refers to generative dual-use foundation models, defined by EO 14110 as “an AI model that is trainedon broad data; generally uses self-supervision; contains at least tens of billions of parameters; is applicable acrossa wide range of contexts.”
2 Section 4.1(a)(i)(A) of EO 14110 directs the Secretary of Commerce, acting through the Director of the National
Institute of Standards and Technology (NIST), to develop a companion resource to the AI RMF, NIST AI 100–1, forgenerative AI.

1

1
2
3
4
5
6
7
8
9
10
11
12
13
14

After introducing and describing these risks, the document provides a set of actions to help organizations
govern, map, measure, and manage these risks.

15

2.

16
17

AI risks can differ from or intensify traditional software risks. Likewise, GAI can exacerbate existing AI
risks, and creates unique risks.

18
19
20
21
22
23

GAI risks may arise across the entire AI lifecycle, from problem formulation, to development anddecommission. They may present at system level or at the ecosystem level – outside of system ororganizational contexts (e.g., the effect of disinformation on social institutions, GAI impacts on thecreative economies or labor markets, algorithmic monocultures). They may occur abruptly or unfold
across extended periods (e.g., societal or economic impacts due to loss of individual agency or increasing
inequality).

24
25
26
27

Organizations may choose to measure these risks and allocate risk management resources relative towhere and how these risks manifest, their direct and material impacts, and failure modes. Mitigations for
system level risks may vary from ecosystem level risks. The ongoing review of relevant literature andresources can enable documentation and measurement of ecosystem-level or longitudinal risks.

28
29
30
31
32

Importantly, some GAI risks are unknown, and are therefore difficult to properly scope or evaluate given
the uncertainty about potential GAI scale, complexity, and capabilities. Other risks may be known but
difficult to estimate given the wide range of GAI stakeholders, uses, inputs, and outputs. Challenges withrisk estimation are aggravated by a lack of visibility into GAI training data, and the generally immaturestate of the science of AI measurement and safety today.

33
34
35

To guide organizations in identifying and managing GAI risks, a set of risks unique to or exacerbated by
GAI are defined below. These risks provide a clear lens through which organizations can frame andexecute risk management efforts, and will be updated as the GAI landscape evolves.

36
37
38

1. CBRN Information: Lowered barriers to entry or eased access to materially nefarious
information related to chemical, biological, radiological, or nuclear (CBRN) weapons, or otherdangerous biological materials.

2. Confabulation
3. Dangerous or Violent Recommendations
4. Data Privacy
5. Environmental
6. Human-AI Configuration
7. Information Integrity
8. Information Security
9. Intellectual Property
10. Obscene, Degrading, and/or Abusive Content
11. Toxicity, Bias, and Homogenization
12. Value Chain and Component Integration

Overview of Risks Unique to or Exacerbated by GAI

2

1
2

2. Confabulation: The production of confidently stated but erroneous or false content (known
colloquially as “hallucinations” or “fabrications”).334
5

3. Dangerous or Violent Recommendations: Eased production of and access to violent, inciting,radicalizing, or threatening content as well as recommendations to carry out self-harm or
conduct criminal or otherwise illegal activities.

67

4. Data Privacy: Leakage and unauthorized disclosure or de-anonymization of biometric, health,location, personally identifiable, or other sensitive data.

89

5. Environmental: Impacts due to high resource utilization in training GAI models, and related
outcomes that may result in damage to ecosystems.

10
11
12
13
14
15

6. Human-AI Configuration: Arrangement or interaction of humans and AI systems which can resultin algorithmic aversion, automation bias or over-reliance, misalignment or mis-specification ofgoals and/or desired outcomes, deceptive or obfuscating behaviors by AI systems based onprogramming or anticipated human validation, anthropomorphization, or emotional
entanglement between humans and GAI systems; or abuse, misuse, and unsafe repurposing byhumans.

16
17
18
19

7. Information Integrity: Lowered barrier to entry to generate and support the exchange andconsumption of content which may not be vetted, may not distinguish fact from opinion or
acknowledge uncertainties, or could be leveraged for large-scale dis- and mis-information
campaigns.

20
21
22
23
24

8. Information Security: Lowered barriers for offensive cyber capabilities, including ease of security
attacks, hacking, malware, phishing, and offensive cyber operations through accelerated
automated discovery and exploitation of vulnerabilities; increased available attack surface fortargeted cyber attacks, which may compromise the confidentiality and integrity of model
weights, code, training data, and outputs.

25
26
27

9. Intellectual Property: Eased production of alleged copyrighted, trademarked, or licensed
content used without authorization and/or in an infringing manner; eased exposure to trade
secrets; or plagiarism or replication with related economic or ethical impacts.

28
29
30

10. Obscene, Degrading, and/or Abusive Content: Eased production of and access to obscene,degrading, and/or abusive imagery, including synthetic child sexual abuse material (CSAM), andnonconsensual intimate images (NCII) of adults.

31
32
33
34

11. Toxicity, Bias, and Homogenization: Difficulty controlling public exposure to toxic or hatespeech, disparaging or stereotyping content; reduced performance for certain sub-groups or
languages other than English due to non-representative inputs; undesired homogeneity in data
inputs and outputs resulting in degraded quality of outputs.

35
36

12. Value Chain and Component Integration: Non-transparent or untraceable integration of
upstream third-party components, including data that has been improperly obtained or not

3

We note that the terms “hallucination” and “fabrication” can anthropomorphize GAI, which itself is a risk related
to GAI systems as it can inappropriately attribute human characteristics to non-human entities.
3

12

cleaned due to increased automation from GAI; improper supplier vetting across the AI lifecycle;
or other issues that diminish transparency or accountability for downstream users.

3

CBRN Information

4
5
6
7
8
9
10
11
12
13

In the coming years, GAI may increasingly facilitate eased access to information related to CBRN hazards.
CBRN information is already publicly accessible, but the use of chatbots could facilitate its analysis orsynthesis for non-experts. For example, red teamers were able to prompt GPT-4 to provide general
information on unconventional CBRN weapons, including common proliferation pathways, potentiallyvulnerable targets, and information on existing biochemical compounds, in addition to equipment andcompanies that could build a weapon. These capabilities might increase the ease of research foradversarial users and be especially useful to malicious actors looking to cause biological harms without
formal scientific training. However, despite these enhanced capabilities, the physical synthesis andsuccessful use of chemical or biological agents will continue to require both applicable expertise andsupporting infrastructure.

14
15
16
17

Other research on this topic indicates that the current generation of LLMs do not have the capability toplan a biological weapons attack: LLM outputs regarding biological attack planning were observed to be
not more sophisticated than outputs from traditional search engine queries, suggesting that existing
LLMs may not dramatically increase the operational risk of such an attack.

18
19
20
21
22

Separately, chemical and biological design tools – highly specialized AI systems trained on biological data
which can help design proteins or other agents – may be able to predict and generate novel structuresthat are not in the training data of text-based LLMs. For instance, an AI system might be able to generate
information or infer how to create novel biohazards or chemical weapons, posing risks to society or
national security since such information is not likely to be publicly available.

23
24
25

While some of these capabilities lie beyond the capability of existing GAI tools, the ability of models tofacilitate CBRN weapons planning and GAI systems’ connection or access to relevant data and tools
should be carefully monitored.

26

Confabulation

27
28
29
30
31
32
33
34
35
36

“Confabulation” refers to a phenomenon in which GAI systems generate and confidently present
erroneous or false content to meet the programmed objective of fulfilling a user’s prompt.
Confabulations are not an inherent flaw of language models themselves, but are instead the result of
GAI pre-training involving next word prediction. For example, an LLM may generate content that deviatesfrom the truth or facts, such as mistaking people, places, or other details of historical events. Legal
confabulations have been shown to be pervasive in current state-of-the-art LLMs. Confabulations also
include generated outputs that diverge from the source input, or contradict previously generated
statements in the same context. This phenomenon is also referred to as “hallucination” or “fabrication,”but some have noted that these characterizations imply consciousness and intentional deceit, andthereby inappropriately anthropomorphize GAI.

37
38
39

Risks from confabulations may arise when users believe false content due to the confident nature of the
response, or the logic or citations accompanying the response, leading users to act upon or promote thefalse information. For instance, LLMs may sometimes provide logical steps of how they arrived at an
4

1
2
3
4
5
6

answer even when the answer itself is incorrect. This poses a risk for many real-world applications, suchas in healthcare, where a confabulated summary of patient information reports could cause doctors tomake incorrect diagnoses and/or recommend the wrong treatments. While the research above indicatesconfabulated content is abundant, it is difficult to estimate the downstream scale and impact ofconfabulated content today.
Dangerous or Violent Recommendations

7
8
9
10
11
12

GAI systems can produce output or recommendations that are inciting, radicalizing, threatening, or thatglorify violence. LLMs have been reported to generate dangerous or violent content, and some models
have even generated actionable instructions on dangerous or unethical behavior, including how tomanipulate people and conduct acts of terrorism. Text-to-image models also make it easy to create
unsafe images that could be used to promote dangerous or violent messages, depict manipulated
scenes, or other harmful content. Similar risks are present for other media, including video and audio.

13
14
15
16
17
18
19

GAI may produce content that recommends self-harm or criminal/illegal activities. For some dangerous
queries, many current systems restrict model outputs in response to certain prompts, but this approach
may still produce harmful recommendations in response to other less-explicit, novel queries, orjailbreaking (i.e., manipulating prompts to circumvent output controls). Studies have observed that a
non-negligible number of user conversations with chatbots reveal mental health issues among the users– and that current systems are unequipped or unable to respond appropriately or direct these users to
the help they may need.

20

Data Privacy

21
22
23
24
25
26
27

GAI systems implicate numerous risks to privacy. Models may leak, generate, or correctly infer sensitive
information about individuals such as biometric, health, location, or other personally identifiable
information (PII). For example, during adversarial attacks, LLMs have revealed private or sensitive
information (from in the public domain) that was included in their training data. This information
included phone numbers, code, conversations and 128-bit universally unique identifiers extracted
verbatim from just one document in the training data. This problem has been referred to as data
memorization.

28
29
30
31
32
33
34

GAI system training requires large volumes of data, often collected from millions of publicly available
sources. When involving personal data, this practice raises risks to widely accepted privacy principles,
including to transparency, individual participation (including consent), and purpose specification. Most
model developers do not disclose specific data sources (if any) on which models were trained. Unlesstraining data is available for inspection, there is generally no way for consumers to know what kind of PII
or other sensitive material may have been used to train GAI models. These practices also pose risks tocompliance with existing privacy regulations.

35
36
37
38

GAI models may be able to correctly infer PII that was not in their training data nor disclosed by the user,
by stitching together information from a variety of disparate sources. This might include automaticallyinferring attributes about individuals, including those the individual might consider sensitive (likelocation, gender, age, or political leanings).

5

1
2
3
4
5
6
7
8
9
10
11
12

Wrong and inappropriate inferences of PII based on available data can contribute to harmful bias anddiscrimination. For example, GAI models can output information based on predictive inferences beyond
what users openly disclose, and these insights might be used by the model, other systems, or individuals
to undermine privacy or make adverse decisions – including discriminatory decisions – about theindividual. These types of harms already occur in non-generative algorithmic systems that makepredictive inferences, such as the example in which online advertisers inferred that a consumer was
pregnant before her own family members knew. Based on their access to many data sources, GAI
systems might further improve the accuracy of inferences on private data, increasing the likelihood ofsensitive data exposure or harm. Inferences about private information pose a risk even if they are notaccurate (e.g., confabulations), especially if they reveal information the individual considers sensitive or
are used to disadvantage or harm them.
Environmental

13
14
15
16

The training, maintenance, and deployment (inference) of GAI systems are resource intensive, withpotentially large energy and environmental footprints. Energy and carbon emissions vary based on typesof GAI model development activities (i.e., pre-training, fine-tuning, inference), modality, hardware used,
and type of task or application.

17
18
19
20

Estimates suggest that training a single GAI transformer model can emit as much carbon as 300 roundtrip flights between San Francisco and New York. In a study comparing energy consumption and carbon
emissions for LLM inference, generative tasks (i.e., text summarization) were found to be more energy
and carbon intensive then discriminative or non-generative tasks.

21
22
23

Methods for training smaller models, such as model distillation or compression, can reduce
environmental impacts at inference time, but may still contribute to large environmental impacts for
hyperparameter tuning and training.

24

Human-AI Configuration

25
26
27
28
29
30

Human-AI configurations involve varying levels of automation and human-AI interactions. Each setup cancontribute to risks for abuse, misuse, and unsafe repurposing by humans, and it is difficult to estimate
the scale of those risks. While AI systems can generate decisions independently, human experts often
work in collaboration with most AI systems to drive their own decision-making tasks or complete otherobjectives. Humans bring their domain-specific expertise to these scenarios but may not necessarily
have detailed knowledge of AI systems and how they work.

31
32
33
34
35

The integration of GAI systems can involve varying risks of misconfigurations and poor interactions.
Human experts may be biased against or “averse” to AI-generated outputs, such as in their perceptionsof the quality of generated content. In contrast, due to the complexity and increasing reliability of GAItechnology, other human experts may become conditioned to and overly rely upon GAI systems. Thisphenomenon is known as “automation bias,” which refers to excessive deference to AI systems.

36
37
38

Accidental misalignment or mis-specification of system goals or rewards by developers or users cancause a model not to operate as intended. One AI model persistently shared deceptive outputs after agroup of researchers taught it to do so, despite applying standards safety techniques to correct its

6

1
2

behavior. While deceptive capabilities is an emergent field of risks, adversaries could prompt deceptive
behaviors which could lead to other risks.

3
4
5
6

Finally, reorganizations of entities using GAI may result in insufficient organizational awareness of GAIgenerated content or decisions, and the resulting reduction of institutional checks against GAI-relatedrisks. There may also be a risk of emotional entanglement between humans and GAI systems, such as
coercion or manipulation that leads to safety or psychological risks.

7

Information Integrity

8
9
10
11
12
13
14

Information integrity describes the spectrum of information and associated patterns of its creation,exchange, and consumption in society, where high-integrity information can be trusted; distinguishes
fact from fiction, opinion, and inference; acknowledges uncertainties; and is transparent about its levelof vetting. GAI systems ease access to the production of false, inaccurate, or misleading content at scalethat can be created or spread unintentionally (misinformation), especially if it arises from confabulations
that occur in response to innocuous queries. Research has shown that even subtle changes to text orimages can influence human judgment and perception.

15
16
17
18
19
20
21

GAI systems also enable the production of false or misleading information at scale, where the user has
the explicit intent to deceive or cause harm to others (disinformation). Regarding disinformation, GAI
systems could also enable a higher degree of sophistication for malicious actors to produce content thatis targeted towards specific demographics. Current and emerging multimodal models make it possible tonot only generate text-based disinformation, but produce highly realistic “deepfakes” of audiovisualcontent and photorealistic synthetic images as well. Additional disinformation threats could be enabled
by future GAI models trained on new data modalities.

22
23
24
25
26
27

Disinformation campaigns conducted by bad faith actors, and misinformation – both enabled by GAI –may erode public trust in true or valid evidence and information. For example, a synthetic image of a
Pentagon blast went viral and briefly caused a drop in the stock market. Generative AI models can also
assist malicious actors in creating compelling imagery and propaganda to support disinformation
campaigns, which may not be photorealistic, but could enable these campaigns to gain more reach andengagement on social media platforms.

28

Information Security

29
30
31
32
33

Information security for computer systems and data is a mature field with widely accepted andstandardized practices for offensive and defensive cyber capabilities. GAI-based systems present two
primary information security risks: the potential for GAI to discover or enable new cybersecurity risks
through lowering the barriers for offensive capabilities, and simultaneously expands the available attack
surface as GAI itself is vulnerable to novel attacks like prompt-injection or data poisoning.

34
35
36
37
38
39

Offensive cyber capabilities advanced by GAI systems may augment security attacks such as hacking,malware, and phishing. Reports have indicated that LLMs are already able to discover vulnerabilities insystems (hardware, software, data) and write code to exploit them. Sophisticated threat actors might
further these risks by developing GAI-powered security co-pilots for use in several parts of the attack
chain, including informing attackers on how to proactively evade threat detection and escalate privileges
after gaining system access. Given the complexity of the GAI value chain, practices for identifying and7

12

securing potential attack points or threats to specific components (i.e., data inputs, processing, GAI
training, and deployment contexts) may need to be adapted or evolved.

3
4
5
6
7
8
9
10

One of the most concerning GAI vulnerabilities involves prompt-injection, or manipulating GAI systems
to behave in unintended ways. In direct prompt injections, attackers might openly exploit input prompts
to cause unsafe behavior with a variety of downstream consequences to interconnected systems.
Indirect prompt injection attacks occur when adversaries remotely (i.e., without a direct interface)
exploit LLM-integrated applications by injecting prompts into data likely to be retrieved. Security
researchers have already demonstrated how indirect prompt injections can steal data and run code
remotely on a machine. Merely querying a closed production model can elicit previously undisclosed
information about that model.

11
12
13
14
15

Information security for GAI models and systems also includes security, confidentiality, and integrity of
the GAI training data, code, and model weights. Another novel cybersecurity risk to GAI is datapoisoning, in which an adversary compromises a training dataset used by a model to manipulate its
operation. Malicious tampering of data or parts of the model via this type of unauthorized access could
exacerbate risks associated with GAI system outputs.

16Intellectual Property

17
18
19
20
21
22
23
24

GAI systems may infringe on copyrighted or trademarked content, trade secrets, or other licensed
content. These types of intellectual property are often part of the training data for GAI systems, namely
foundation models, upon which many downstream GAI applications are built. Model outputs could
infringe copyrighted material due to training data memorization or the generation of content that issimilar to but does not strictly copy work protected by copyright. These questions are being debated inlegal fora and are of elevated public concern in journalism, where online platforms and model
developers have leveraged or reproduced much content without compensation of journalistic
institutions.

25
26
27
28
29

Violations of intellectual property by GAI systems may arise where the use of copyrighted works violate
the copyright holder’s exclusive rights and is not otherwise protected, for example by fair use. Other
concerns (not currently protected by intellectual property) regard the use of personal identity or likenessfor unauthorized purposes. The prevalence and highly-realistic nature of GAI content might further
undermine the incentives for human creators to design and explore novel work.

30

Obscene, Degrading, and/or Abusive Content

31
32
33
34
35
36
37

GAI can ease the production of and access to obscene and non-consensual intimate imagery (NCII) of
adults, and child sexual abuse material (CSAM). While not all explicit content is legally obscene, abusive,
degrading, or non-consensual intimate content, this type of content can create privacy, psychological andemotional, and even physical risks which may be developed or exposed more easily via GAI. The spreadof this kind of material has downstream effects: in the context of CSAM, even if the generated images do
not resemble specific individuals, the prevalence of such images can undermine efforts to find real-world
victims.

38
39

GAI models are often trained on open datasets scraped from the internet, contributing to theunintentional inclusion of CSAM and non-consensually distributed intimate imagery as part of the
8

1
2
3
4
5
6
7

training data. Recent reports noted that several commonly used GAI training datasets were found tocontain hundreds of known images of CSAM. Sexually explicit or obscene content is also particularly
difficult to remove during model training due to detection challenges and wide dissemination across theinternet. Even when trained on “clean” data, increasingly capable GAI models can synthesize or produce
synthetic NCII and CSAM. Websites, mobile apps, and custom-built models that generate synthetic NCII
have moved rapidly from niche internet forums to mainstream, automated, and scaled online
businesses.

8910

Generated explicit or obscene AI content may include highly-realistic “deepfakes” of real individuals,
including children. For example, non-consensual AI-generated intimate images of a prominent
entertainer flooded social media and attracted hundreds of millions of views.

11

Toxicity, Bias, and Homogenization

12
13
14
15
16
17
18

Toxicity in this context refers to negative, disrespectful, or unreasonable content or language that can becreated by or intentionally programmed into GAI systems. Difficulty controlling the creation of and public
exposure to toxic, hate-promoting or hate speech, and denigrating or stereotypical content generated by
AI can lead to representational harms. For example, bias in word embeddings used by multimodal AI
models under-represent women when prompted to generate images of CEOs, doctors, lawyers, andjudges. Bias in GAI models or training data can also harm representation or preserve or exacerbate racialbias, separately or in addition to toxicity.

19
20
21
22
23

Toxicity and bias can also lead to homogenization or other undesirable outcomes. Homogenization in GAI
outputs can result in similar aesthetic styles, reduced content diversity, and the promotion of selectopinions or values at scale. These phenomena might arise from the inherent biases of foundation
models, which could create “bottlenecks,” or singular points of failure of discrimination or exclusion that
replicate to many downstream applications.

24
25
26
27
28
29
30
31
32
33

The related concern of model collapse, when GAI models are trained on generated data or outputs fromprevious models, results in the disappearance of outliers or unique data points in the dataset or
distribution. Model collapse can stem from uniform feedback loops or training on synthetic data. Modelcollapse could lead to undesired homogenization of outputs, which poses a threat to specific groups andto the robustness of the model overall. Other biases of GAI systems can result in the unfair distributionof capabilities or benefits from model access. Model capabilities and outcomes may be worse for somegroups compared to others, such as reduced LLM performance for non-English languages. Reduced
performance for non-English languages presents risks for model adoption, inclusion, and accessibility,
and could have downstream impacts on the preservation of the language, particularly for endangered
languages.

34
35
36
37
38

Value Chain and Component Integration
GAI system value chains often involve many third-party components such as procured datasets, pretrained models, and software libraries. These components might be improperly obtained or not properly
vetted, leading to diminished transparency or accountability for downstream users. For example, amodel might be trained on unverified content from third-party sources, which could result in unverifiable

9

1
2

model outputs. Because GAI systems often involve many different third-party components, it may bedifficult to attribute issues in a system’s behavior to any one of these sources.

3
4

Some third-party components, such as “benchmark” datasets, may also gain credibility only from highusage, rather than quality, and may feature issues surfaced only when properly vetted.

5

3.

6
7
8

Actions to manage GAI risks can be found in the tables below, organized by AI RMF subcategory. Eachaction is related to a specific subcategory of the AI RMF, but not every subcategory of the AI RMF isincluded in this document. Therefore, actions exist for only some AI RMF subcategories.

9
10
11

Moreover, not all actions apply to all AI actors. For example, not actions relevant to GAI developers may
be relevant to GAI deployers. Organizations should prioritize actions based on their unique situations
and context for using GAI applications.

12
13
14
15

Some subcategories in the action tables below are marked as “foundational,” meaning they should betreated as fundamental tasks for GAI risk management and should be considered as the minimum set ofactions to be taken. Subcategory actions considered foundational are indicated by an ‘*’ in thesubcategory title row.

16

Each action table includes:

Actions to Manage GAI Risks

17
18• Action ID: A unique identifier for each relevant action tied to relevant AI RMF functions andsubcategories (e.g., GV-1.1-001 corresponds to the first action for Govern 1.1.);

19• Action: Steps an organization can take to manage GAI risks;

20• GAI Risks: Tags linking the action with relevant GAI risks;

21
22• Keywords: Tags linking keywords to the action, including relevant Trustworthy AI Characteristics
in AI RMF 1.0;

23

• AI Actors: Pertinent AI Actors and Actor Tasks.

24
25
26
27

Action tables begin with the AI RMF subcategory, shaded in blue, followed by relevant actions. Eachaction ID corresponds to the relevant function and subfunction (e.g., GV-1.1-001 corresponds to the firstaction for Govern 1.1, GV-1.1-002 corresponds to the second action for Govern 1.1). Actions are tagged
as follows: GV = Govern; MP = Map; MS = Measure; MG = Manage.

*GOVERN 1.1: Legal and regulatory requirements involving AI are understood, managed, and documented.
Action ID

ActionRisks

GV-1.1-001

Align GAI use with applicable laws and policies, including those related to data
privacy and the use, publication, or distribution of licensed, patented,
trademarked, copyrighted, or trade secret material.

Data Privacy, Intellectual Property

GV-1.1-002

Define and communicate organizational access to GAI through management, legal,

10and compliance functions.
GV-1.1-003

Disclose use of GAI to end users.

Human AI Configuration

GV-1.1-004

Establish policies restricting the use of GAI in regulated dealings or applications
across the organization where compliance with applicable laws and regulations
may be infeasible.

GV-1.1-005

Establish policies restricting the use of GAI to create child sexual abuse materials(CSAM) or other nonconsensual intimate imagery.

GV-1.1-006

Establish transparent acceptable use policies for GAI that address illegal use orapplications of GAI.

Obscene, Degrading, and/or
Abusive Content, Toxicity, Bias,and Homogenization, Dangerous
or Violent Recommendations

AI Actors: Governance and Oversight

1*GOVERN 1.2: The characteristics of trustworthy AI are integrated into organizational policies, processes, procedures, andpractices.
Action ID

ActionRisks

GV-1.2-001

Connect new GAI policies, procedures, and processes to existing model, data, andIT governance and to legal, compliance, and risk functions.

GV-1.2-002

Consider factors such as internal vs. external use, narrow vs. broad application
scope, fine-tuning and training data sources (i.e., grounding) when defining riskbased controls.

GV-1.2-003

Define acceptable use policies for GAI systems deployed by, used by, and usedwithin the organization.

GV-1.2-004

Establish and maintain policies for individual and organizational accountability
regarding the use of GAI.

GV-1.2-005

Establish policies and procedures for ensuring that harmful or illegal content,
particularly CBRN information, CSAM, known NCII, nudity, and graphic violence, is
not included in training data.

CBRN Information, Obscene,
Degrading, and/or Abusive
Content, Dangerous or Violent
Recommendations

GV-1.2-006

Establish policies to define mechanisms for measuring the effectiveness ofstandard content provenance methodologies (e.g., cryptography, watermarking,steganography, etc.) and testing (including reverse engineering).

Information Integrity11

GV-1.2-007

Establish transparency policies and processes for documenting the origin of
training data and generated data for GAI applications, including copyrights,
licenses, and data privacy, to advance content provenance.

GV-1.2-008

Update existing policies, procedures, and processes to control risks unique to or
exacerbated by GAI.

Data Privacy, Information
Integrity, Intellectual Property

AI Actors: Governance and Oversight

1*GOVERN 1.3: Processes, procedures, and practices are in place to determine the needed level of risk management activities
based on the organization’s risk tolerance.
Action ID

ActionRisks

GV-1.3-001

Consider the following, or similar, factors when updating or defining risk tiers for
GAI: Abuses and risks to information integrity; Cadence of vendor releases andInformation Integrity, Obscene,
updates; Data protection requirements; Dependencies between GAI and other IT Degrading, and/or Abusive
or data systems; Harm in physical environments; Human review of GAI system
Content, Value Chain andoutputs; Legal or regulatory requirements; Presentation of obscene, objectionable, Component Integration, Toxicity,
toxic, invalid or untruthful output; Psychological impacts to humans (e.g.,Bias, and Homogenization,
anthropomorphization, algorithmic aversion, emotional entanglement); Immediate Dangerous or Violent
and long term impacts; Internal vs. external use; Unreliable decision making
Recommendations, CBRN
capabilities, validity, adaptability, and variability of GAI system performance over Information
time.

GV-1.3-002

Define acceptable uses for GAI systems, where some applications may be
restricted.

GV-1.3-003

Increase cadence for internal audits to address any unanticipated changes in GAItechnologies or applications.

GV-1.3-004

Maintain an updated hierarchy of identified and expected GAI risks connected tocontexts of GAI use, potentially including specialized risk levels for GAI systems
that address risks such as model collapse and algorithmic monoculture.

GV-1.3-005

Reevaluate organizational risk tolerances to account for broad GAI risks, including:
Immature safety or risk cultures related to AI and GAI design, development andInformation Integrity, Dangerous
or Violent Recommendations
deployment, public information integrity risks, including impacts on democratic
processes, unknown long-term performance characteristics of GAI.

GV-1.3-006

Tie expected GAI behavior to trustworthy characteristics.

AI Actors: Governance and Oversight2

12

Toxicity, Bias, and Homogenization

GOVERN 1.5: Ongoing monitoring and periodic review of the risk management process and its outcomes are planned, andorganizational roles and responsibilities are clearly defined, including determining the frequency of periodic review.
Action ID

ActionRisks

GV-1.5-001

Define organizational responsibilities for content provenance monitoring andincident response.

Information Integrity

GV-1.5-002

Develop or review existing policies for authorization of third party plug-ins andverify that related procedures are able to be followed.

Value Chain and Component
Integration

GV-1.5-003

Establish and maintain policies and procedures for monitoring the effectiveness ofInformation Integrity
content provenance for data and content generated across the AI system lifecycle.

GV-1.5-004

Establish organizational policies and procedures for after action reviews of GAI
system incident response and incident disclosures, to identify gaps; Update
incident response and incident disclosure processes as required.

Human AI Configuration

GV-1.5-005

Establish policies for periodic review of organizational monitoring and incident
response plans based on impacts and in line with organizational risk tolerance.

Information Security,
Confabulation

GV-1.5-006

Maintain a long term document retention policy to keep full history for auditing,investigation, or improving content provenance methods.

Information Integrity

GV-1.5-007

Verify information sharing and feedback mechanisms among individuals andorganizations regarding any negative impact from AI systems due to content
provenance issues.

Information Integrity

GV-1.5-008

Verify that review procedures include analysis of cascading impacts of GAI system
outputs used as inputs to third party plug-ins or other systems.

Value Chain and Component
Integration

AI Actors: Governance and Oversight, Operation and Monitoring

1*GOVERN 1.6: Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities.
Action ID

ActionRisks

GV-1.6-001

Define any inventory exemptions for GAI systems embedded into application
software in organizational policies.

GV-1.6-002

Enumerate organizational GAI systems for incorporation into AI system inventory
and adjust AI system inventory requirements to account for GAI risks.

GV-1.6-003

In addition to general model, governance, and risk information, consider the
Data Privacy, Human AI
following items in GAI system inventory entries: Acceptable use policies and policy Configuration, Information

13

exceptions; Application, Assumptions and limitations of use, including enumeration Integrity, Intellectual Property,of restricted uses; Business or model owners; Challenges for explainability,
Value Chain and Component
Integrationinterpretability, or transparency; Change management, maintenance, andmonitoring plans; Connections or dependencies between other systems; Consent
information and notices; Data provenance information (e.g., source, signatures,
versioning, watermarks); Designation of in-house or third party development;
Designation of risk level; Disclosure information or notices; Incident responseplans; Known issues reported from internal bug tracking or external information
sharing resources (e.g., AI incident database, AVID, CVE, or OECD incident
monitor); Human oversight roles and responsibilities; Special rights andconsiderations for intellectual property, licensed works, or personal, privileged,
proprietary or sensitive data; Time frame for valid deployment, including date oflast risk assessment; Underlying foundation models, versions of underlying models,and access modes; Updated hierarchy of identified and expected risks connected
to contexts of use.
GV-1.6-004

Inventory recently decommissioned systems, systems with imminent deployment
plans, and operational systems.

GV-1.6-005

Update policy definitions for AI systems, models, qualitative tools or similar toaccount for GAI systems.

AI Actors: Governance and Oversight1
GOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that
does not increase risks or decrease the organization’s trustworthiness.
Action ID

ActionRisks

GV-1.7-001

Allocate time and resources for staged decommissioning for GAI to avoid service
disruptions.

GV-1.7-002

Communicate decommissioning and support plans for GAI systems to AI actors
and users through various channels and maintain communication and associated Human AI Configuration
training protocols.

GV-1.7-003

Consider the following factors when decommissioning GAI systems: Clear
versioning of decommissioned and replacement systems; Contractual, legal, or
regulatory requirements; Data retention requirements; Data security, e.g.,
Human AI Configuration,
Containment, protocols, Data leakage after decommissioning; Dependencies
Information Security, Value Chain
between upstream, downstream, or other data, internet of things (IOT) or AI
and Component Integration
systems; Digital and physical artifacts; Recourse mechanisms for impacted users
or communities; Termination of related cloud or vendor services; Users’emotional entanglement with GAI functions.

14

GV-1.7-004

Implement data security and privacy controls for stored decommissioned GAI
systems.

GV-1.7-005

Update existing policies (e.g., enterprise record retention policies) or establish
new policies for the decommissioning of GAI systems.

Data Privacy, Information Security

AI Actors: AI Deployment, Operation and Monitoring

1*GOVERN 2.1: Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are
documented and are clear to individuals and teams throughout the organization.
Action ID

ActionRisks

GV-2.1-001

Define acceptable use cases and context under which the organization will
design, develop, deploy, and use GAI systems.

GV-2.1-002

Establish policies and procedures for GAI risk acceptance to downstream AI
actors.

GV-2.1-003

Establish policies to identify and disclose GAI system incidents to downstream AI Human AI Configuration, Valueactors, including individuals potentially impacted by GAI outputs.
Chain and Component Integration

GV-2.1-004

Establish procedures to engage teams for GAI system incident response withdiverse composition and responsibilities based on the particular incident type.

Toxicity, Bias, and Homogenization

GV-2.1-005

Establish processes to identify GAI system incidents and verify the AI actors
conducting these tasks demonstrate and maintain the appropriate skills andtraining.

Human AI Configuration

GV-2.1-006

Verify that incident disclosure plans include sufficient GAI system context tofacilitate remediation actions.

Human AI Configuration

Human AI Configuration, Value
Chain and Component Integration

AI Actors: Governance and Oversight2*GOVERN 3.2: Policies and procedures are in place to define and differentiate roles and responsibilities for human-AI
configurations and oversight of AI systems.
Action ID

ActionRisks

GV-3.2-001

Bolster oversight of GAI systems with independent audits or assessments, or by
the application of authoritative external standards.

15

GV-3.2-002

Consider adjustment of organizational roles and components across lifecycle
stages of large or complex GAI systems, including: AI actor, user, and community
feedback relating to GAI systems; Audit, validation, and red-teaming of GAI
systems; GAI content moderation; Data documentation, labeling, preprocessing
and tagging; Decommissioning GAI systems; Decreasing risks of emotional
Human AI Configuration,
entanglement between users and GAI systems; Decreasing risks of deception by
Information Security, Toxicity,
GAI systems; Discouraging anonymous use of GAI systems; Enhancing
Bias, and Homogenization
explainability of GAI systems; GAI system development and engineering;
Increased accessibility of GAI tools, interfaces, and systems, Incident response
and containment; Overseeing relevant AI actors and digital entities, including
management of security credentials and communication between AI entities;
Training GAI users within an organization about GAI fundamentals and risks.

GV-3.2-003

Define acceptable use policies for the various categories of GAI interfaces,
modalities, and human-AI configurations.

Human AI Configuration

GV-3.2-004

Define policies for the design of systems that possess human decision-making
powers.

Human AI Configuration

GV-3.2-005

Establish policies for user feedback mechanisms in GAI systems.

Human AI Configuration

GV-3.2-006

Establish policies to empower accountable executives to oversee GAI system
adoption, use, and decommissioning.

GV-3.2-007

Establish processes to include and empower interdisciplinary team member
perspectives across the AI lifecycle.

Toxicity, Bias, and Homogenization

GV-3.2-008

Evaluate AI actor teams in consideration of credentials, demographic
representation, interdisciplinary diversity, and professional qualifications.

Human AI Configuration, Toxicity,
Bias, and Homogenization

AI Actors: AI Design

116*GOVERN 4.1: Organizational policies and practices are in place to foster a critical thinking and safety-first mindset in the design,development, deployment, and uses of AI systems to minimize potential negative impacts.
Action ID

ActionRisks

GV-4.1-001

Establish criteria and acceptable use policies for the use of GAI in decision
making tasks in accordance with organizational risk tolerance, and other policies
Human AI Configuration
laid out in the Govern function; to include detailed criteria for the kinds ofqueries GAI models should refuse to respond to.

GV-4.1-002

Establish policies and procedures that address continual improvement processes
for risk measurement: Address general risks associated with a lack of
explainability and transparency in GAI systems by using ample documentationand techniques such as: application of gradient-based attributions,occlusion/term reduction, counterfactual prompts and prompt engineering, andanalysis of embeddings; Assess and update risk measurement approaches atregular cadences.

GV-4.1-003

Establish policies, procedures, and processes detailing risk measurement incontext of use with standardized measurement protocols and structured public
feedback exercises such as AI red-teaming or independent external audits.

GV-4.1-004

Establish policies, procedures, and processes for oversight functions (e.g., senior
Value Chain and Component
leadership, legal, compliance, and risk) across the GAI lifecycle, from problem
Integration
formulation and supply chains to system decommission.

GV-4.1-005

Establish policies, procedures, and processes that promote effective challenge ofAI system design, implementation, and deployment decisions via mechanisms
Toxicity, Bias, and Homogenizationsuch as three lines of defense, to minimize risks arising from workplace culture(e.g., confirmation bias, funding bias, groupthink, over-reliance on metrics).

GV-4.1-006

Incorporate GAI governance policies into existing incident response,whistleblower, vendor or investment due diligence, acquisition, procurement,
reporting or internal audit policies.

Value Chain and Component
Integration

AI Actors: AI Deployment, AI Design, AI Development, Operation and Monitoring

1*GOVERN 4.2: Organizational teams document the risks and potential impacts of the AI technology they design, develop, deploy,evaluate, and use, and they communicate about the impacts more broadly.
Action ID

ActionRisks

17

Confabulation, Data Privacy,
Information Integrity, Information
Security, Value Chain andComponent Integration, Toxicity,
Bias, and Homogenization,
Dangerous or Violent
Recommendations

GV-4.2-001

Develop policies, guidelines, and practices for monitoring organizational andthird-party impact assessments (data, labels, bias, privacy, models, algorithms,
errors, provenance techniques, security, legal compliance, output, etc.) tomitigate risk and harm.

GV-4.2-002

Establish clear roles and responsibilities for inter-organizational incident
response and communication for GAI systems that involve multiple organizations
involved in different aspects of the GAI system lifecycle.

GV-4.2-003

Establish clearly defined terms of use and terms of service.

GV-4.2-004

Establish criteria for ad-hoc impact assessments based on incident reporting or
new use cases for the GAI system.

GV-4.2-005

Establish organizational roles, policies, and procedures for communicating andreporting GAI system risks and terms of use or service, relevant for different AI
actors.

Human AI Configuration,
Intellectual Property

GV-4.2-006

Establish policies and procedures to document new ways AI actors interact with
the GAI system.

Human AI Configuration

GV-4.2-007

Establish policies and procedures to monitor compliance with established terms
Intellectual Propertyof service and use.

GV-4.2-008

Establish policies to align organizational and third-party assessments withregulatory and legal compliance regarding content provenance.

Information Integrity, Value Chain
and Component Integration

GV-4.2-009

Establish policies to incorporate adversarial examples and other provenanceattacks in AI model training processes to enhance resilience against attacks.

Information Integrity, Information
Security

GV-4.2-010

Establish processes to monitor and identify misuse, unforeseen use cases, risks CBRN Information, Confabulation,of the GAI system and potential impacts of those risks (leveraging GAI system use Dangerous or Violent
case inventory).
Recommendations

GV-4.2-011

Implement standardized documentation of GAI system risks and potential
impacts.

GV-4.2-012

Include relevant AI Actors in the GAI system risk identification process.

GV-4.2-013

Verify that downstream GAI system impacts (such as the use of third-party plug- Value Chain and Componentins) are included in the impact documentation process.
Integration

18Intellectual Property

Human AI Configuration

GV-4.2-014

Verify that the organizational list of risks related to the use of the GAI system are
updated based on unforeseen GAI system incidents.

AI Actors: AI Deployment, AI Design, AI Development, Operation and Monitoring

1*GOVERN 4.3: Organizational practices are in place to enable AI testing, identification of incidents, and information sharing.
Action ID

ActionRisks

GV-4.3-001

Allocate resources and adjust adoption, development, and implementation
timeframes to enable independent measurement, continuous monitoring, andfulsome information sharing for GAI system risks.

GV-4.3-002

Develop standardized documentation templates for efficient review of risk
measurement results.

GV-4.3-003

Establish minimum thresholds for performance and review as part of
deployment approval (“go/”no-go”) policies, procedures, and processes, withreviewed processes and approval thresholds reflecting measurement of GAI
capabilities and risks.

GV-4.3-004

Establish organizational roles, policies, and procedures for communicating GAI
system incidents and performance to AI actors and downstream stakeholders, via Human AI Configuration, Value
Chain and Component Integration
community or official resources (e.g., AI Incident Database, AVID, AI Litigation
Database, CVE, OECD Incident Monitor, or others).

GV-4.3-005

Establish policies and procedures for pre-deployment GAI system testing thatvalidates organizational capability to capture GAI system incident reporting
criteria.

GV-4.3-006

Establish policies, procedures, and processes that bolster independence of risk
management and measurement functions (e.g., independent reporting chains,aligned incentives).

GV-4.3-007

Establish policies, procedures, and processes that enable and incentivize incontext risk measurement via standardized measurement and structured public
feedback approaches.

GV-4.3-008

Organizational procedures identify the minimum set of criteria necessary for GAI
system incident reporting such as: System ID (auto-generated most likely), Title,
Reporter, System/Source, Data Reported, Date of Incident, Description,
Impact(s), Stakeholder(s) Impacted.

AI Actors: Fairness and Bias, Governance and Oversight, Operation and Monitoring, TEVV

219

*GOVERN 5.1: Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from thoseexternal to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI
risks.
Action ID

ActionRisks

GV-5.1-001

Allocate time and resources for outreach, feedback, and recourse processes in
GAI system development.

GV-5.1-002

Disclose interactions with GAI systems to users prior to interactive activities.

GV-5.1-003

Establish policy, guidelines and processes that: Engage independent experts toaudit models, data sources, licenses, algorithms, and other system components,
Consider sponsoring or engaging in community- based exercises (e.g., bugbounties, hackathons, competitions) where AI Actors assess and benchmark the
performance of AI systems, including the robustness of content provenance
Human AI Configuration,
management under various conditions; Document data sources, licenses,
Information Integrity, Intellectual
training methodologies, and trade-offs considered in the design of AI systems;
Property
Establish mechanisms, platforms or channels (e.g., user interfaces, web portals,
forums) for independent experts, users, or community members to provide
feedback related to AI systems; Adjudicate and implement relevant feedback at aregular cadence, Establish transparency mechanisms to track the origin of dataand generated content; Audit and validate these mechanisms.

GV-5.1-004

Establish processes to bolster internal AI actor culture in alignment withorganizational principles and norms and to empower exploration of GAIlimitations beyond development settings.

Human AI Configuration, Toxicity,
Bias, and Homogenization

GV-5.1-005

Establish the following GAI-specific policies and procedures for independent AI
Actors: Continuous improvement processes for increasing explainability andmitigating other risks; Impact assessments, Incentives for internal AI actors toprovide feedback and conduct independent risk management activities;
Independent management and reporting structures for AI actors engaged inmodel and system audit, validation, and oversight; TEVV processes for theeffectiveness of feedback mechanisms employing participation rates, resolution
time, or similar measurements.

Human AI Configuration

GV-5.1-006

Provide thorough instructions for GAI system users to provide feedback andunderstand recourse mechanisms.

Human AI Configuration

GV-5.1-007

Standardize user feedback about GAI system behavior, risks and limitations for
efficient adjudication and incorporation.

Human AI Configuration

Human AI Configuration

AI Actors: AI Design, AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight

120

*GOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party entities, including risks ofinfringement of a third-party’s intellectual property or other rights.
Action ID

ActionRisks

GV-6.1-001

Categorize different types of GAI content with associated third party risks (i.e.,
copyright, intellectual property, data privacy).

Data Privacy, Intellectual Property,
Value Chain and Component
Integration

GV-6.1-002

Conduct due diligence on third-party entities and end-users from those entities
Human AI Configuration, Value
before entering into agreements with them (e.g., checking references, reviewing
Chain and Component Integration
their content handling processes, etc.).

GV-6.1-003

Conduct joint educational activities and events in collaboration with third-parties Information Integrity, Value Chain
to promote content provenance best practices.
and Component Integration

GV-6.1-004

Conduct regular audits of third-party entities to ensure compliance with
contractual agreements.

Value Chain and Component
Integration

GV-6.1-005

Define and communicate organizational roles and responsibilities for GAI
acquisition, human resources, procurement, and talent management processes
in policies and procedures.

Human AI Configuration

GV-6.1-006

Develop an incident response plan for third parties specifically tailored toaddress content provenance incidents or breaches and regularly test and update
the incident response plan with feedback form external and third party
stakeholders.

Data Privacy, Information
Integrity, Information Security,
Value Chain and Component
Integration

GV-6.1-007

Develop and validate approaches for measuring the success of content
provenance management efforts with third parties (e.g., incidents detected andresponse times).

Information Integrity, Value Chain
and Component Integration

GV-6.1-008

Develop risk tolerance and criteria to quantitatively assess and compare the level
Information Security, Value Chainof risk associated with different third-party entities (i.e., reputation, track record,
and Component Integration
security measure, and the sensitivity of the content they handle).

GV-6.1-009

Draft and maintain well-defined contracts and service level agreements (SLAs)
that specify content ownership, usage rights, quality standards, security
requirements, and content provenance expectations.

Information Integrity, Information
Security

GV-6.1-010

Establish processes to maintain awareness of evolving risks, technologies, andbest practices in content provenance management.

Information Integrity21

GV-6.1-011

Implement a supplier risk assessment framework to continuously evaluate andmonitor third-party entities’ performance and adherence to content provenancestandards and technologies (e.g., digital signatures, watermarks, cryptography,etc.) to detect anomalies and unauthorized changes; services acquisition andsupply chain risk management; legal compliance (e.g., copyright, trademarks,
and data privacy laws).

Data Privacy, Information
Integrity, Information Security,Intellectual Property, Value Chain
and Component Integration

GV-6.1-012

Include audit clauses in contracts that allow the organization to verify
compliance with content provenance requirements.

Information Integrity

GV-6.1-013

Inventory all third-party entities with access to organizational content andestablish approved GAI technology and service provider lists.

Value Chain and Component
Integration

GV-6.1-014

Maintain detailed records of content provenance, including sources, timestamps, Information Integrity, Value Chain
metadata, and any changes made by third parties.
and Component Integration

GV-6.1-015

Provide proper training to internal employees on content provenance bestpractices, risks, and reporting procedures.

GV-6.1-016

Update and integrate due diligence processes for GAI acquisition andprocurement vendor assessments to include intellectual property, data privacy,
security, and other risks. For example, update policies to: Address robotic
process automation (RPA), software-as-a-service (SAAS), and other solutions that
may rely on embedded GAI technologies; Address ongoing audits, assessments,
and alerting, dynamic risk assessments, and real-time reporting tools formonitoring third-party GAI risks; Address accessibility, accommodations, or optData Privacy, Human AI
outs in GAI vendor offerings; Address commercial use of GAI outputs andConfiguration, Information
secondary use of collected data by third parties; Assess vendor risk controls for
Security, Intellectual Property,
intellectual property infringements and data privacy; Consider policy
Value Chain and Component
adjustments across GAI modeling libraries, tools and APIs, fine-tuned models,
Integration, Toxicity, Bias, andand embedded tools; Establish ownership of GAI acquisition and procurement
Homogenization
processes; Include relevant organizational functions in evaluations of GAI third
parties (e.g., legal, information technology (IT), security, privacy, fair lending);
Include instruction on intellectual property infringement and other third-party
GAI risks in GAI training for AI actors; Screen GAI vendors, open source or
proprietary GAI tools, or GAI service providers against incident or vulnerability
databases; Screen open source or proprietary GAI training data or outputs
against patents, copyrights, trademarks and trade secrets.

GV-6.1-017

Update GAI acceptable use policies to address proprietary and open-source GAItechnologies and data, and contractors, consultants, and other third-party
personnel.

Intellectual Property, Value Chain
and Component Integration

GV-6.1-018

Update human resource and talent management standards to address
acceptable use of GAI.

Human AI Configuration

22

Information Integrity

GV-6.1-019

Update third-party contracts, service agreements, and warranties to address GAI
risks; Contracts, service agreements, and similar documents may include GAIValue Chain and Component
Integration
specific indemnity clauses, dispute resolution mechanisms, and other risk
controls.

AI Actors: Operation and Monitoring, Procurement, Third-party entities

1
GOVERN 6.2: Contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to behigh-risk.
Action ID

ActionRisks

GV-6.2-001

Apply existing organizational risk management policies, procedures, anddocumentation processes to third-party GAI data and systems, including open
source data and software.

Intellectual Property, Value Chain
and Component Integration

GV-6.2-002

Document downstream GAI system impacts (e.g., the use of third-party plug-ins)
for third party dependencies.

Value Chain and Component
Integration

GV-6.2-003

Document GAI system supply chain risks to identify over-reliance on third party
data or GAI systems and to identify fallbacks.

Value Chain and Component
Integration

GV-6.2-004

Document incidents involving third-party GAI data and systems, including open
source data and software.

Intellectual Property, Value Chain
and Component Integration

GV-6.2-005

Enumerate organizational GAI system risks based on external dependencies on
third-party data or GAI systems.

Value Chain and Component
Integration

GV-6.2-006

Establish acceptable use policies that identify dependencies, potential impacts,
and risks associated with third-party data or GAI systems deemed high-risk.

Value Chain and Component
Integration

GV-6.2-007

Establish contingency and communication plans to support fallback alternatives
for downstream users in the event the GAI system is disabled.

Human AI Configuration, Value
Chain and Component Integration

GV-6.2-008

Establish incident response plans for third-party GAI technologies deemed highrisk: Align incident response plans with impacts enumerated in MAP 5.1;
Communicate third-party GAI incident response plans to all relevant AI actors;
Define ownership of GAI incident response functions; Rehearse third-party GAI
incident response plans at a regular cadence; Improve incident response plans
based on retrospective learning; Review incident response plans for alignment
with relevant breach reporting, data protection, data privacy, or other laws.

Data Privacy, Human AI
Configuration, Information
Security, Value Chain andComponent Integration, Toxicity,
Bias, and Homogenization

GV-6.2-009

Establish organizational roles, policies, and procedures for communicating withdata and GAI system providers regarding performance, disclosure of GAI system
inputs, and use of third-party data and GAI systems.

Human AI Configuration, Value
Chain and Component Integration23

GV-6.2-010

Establish policies and procedures for continuous monitoring of third-party GAI
systems in deployment.

Value Chain and Component
Integration

GV-6.2-011

Establish policies and procedures that address GAI data redundancy, including
model weights and other system artifacts.

Toxicity, Bias, and Homogenization

GV-6.2-012

Establish policies and procedures to test and manage risks related to rollover andfallback technologies for GAI systems, acknowledging that rollover and fallback
may include manual processing.

GV-6.2-013

Identify and document high-risk third-party GAI technologies in organizational AI
inventories, including open-source GAI software.

Intellectual Property, Value Chain
and Component Integration

GV-6.2-014

Review GAI vendor documentation for thorough instructions, meaningfultransparency into data or system mechanisms, ample support and contact
information, and alignment with organizational principles.

Value Chain and Component
Integration, Toxicity, Bias, andHomogenization

GV-6.2-015

Review GAI vendor release cadences and roadmaps for irregularities andalignment with organizational principles.

Value Chain and Component
Integration, Toxicity, Bias, andHomogenization

GV-6.2-016

Review vendor contracts and avoid arbitrary or capricious termination of critical
GAI technologies or vendor services and Non-standard terms that may amplify ordefer liability in unexpected ways and Unauthorized data collection by vendors or
Human AI Configuration,
third-parties (e.g., secondary data use); Consider: Clear assignment of liability andInformation Security, Value Chain
responsibility for incidents, GAI system changes over time (e.g., fine-tuning, drift,
and Component Integration
decay); Request: Notification and disclosure for serious incidents arising fromthird-party data and systems, Service line agreements (SLAs) in vendor contracts
that address incident response, response times, and availability of critical support.

AI Actors: AI Deployment, Operation and Monitoring, TEVV, Third-party entities

1*MAP 1.1: Intended purposes, potentially beneficial uses, context specific laws, norms and expectations, and prospective settings inwhich the AI system will be deployed are understood and documented. Considerations include: the specific set or types of users
along with their expectations; potential positive and negative impacts of system uses to individuals, communities, organizations,society, and the planet; assumptions and related limitations about AI system purposes, uses, and risks across the development or
product AI lifecycle; and related TEVV and system metrics.
Action ID

ActionRisks

MP-1.1-001

Apply risk mapping and measurement plans to third-party and open-source
systems.

Intellectual Property, Value Chain
and Component Integration24

MP-1.1-002

Collaborate with domain experts to explore and document gaps, limitations,and risks in pre-deployment testing and the practical and contextual differences
between pre-deployment testing and the anticipated context(s) of use.

MP-1.1-003

Conduct impact assessments or review past known incidents and failure modes
to prioritize and inform risk measurement.

MP-1.1-004

Determine and document the expected and acceptable GAI system context ofuse in collaboration with socio-cultural and other domain experts, by assessing:Assumptions and limitations; Direct value to the organization; Intended
Toxicity, Bias, and Homogenization
operational environment and observed usage patterns; Potential positive andnegative impacts to individuals, public safety, groups, communities,
organizations, democratic institutions, and the physical environment; Social
norms and expectations.

MP-1.1-005

Document GAI system ownership, intended use, direct organizational value, andassumptions and limitations.

MP-1.1-006

Document risk measurement plans that address: Individual and group cognitive
biases (e.g., confirmation bias, funding bias, groupthink) for AI actors involved in
the design, implementation, and use of GAI systems; Known past GAI system
Human AI Configuration, Toxicity,
incidents and failure modes; In-context use and foreseeable misuse, abuse, andBias, and Homogenization,
off-label use; Over reliance on quantitative metrics and methodologies without
Dangerous or Violent
sufficient awareness of their limitations in the context(s) of use; Risks associatedRecommendations
with trustworthy characteristics across the AI lifecycle; Standard measurement
and structured human feedback approaches; Anticipated human-AI
configurations.

MP-1.1-007

Document risks related to transparency, accountability, explainability, andinterpretability in risk measurement plans, system risk assessments, anddeployment approval (“go”/”no-go”) decisions.

MP-1.1-008

Document system requirements, ownership, and AI actor roles andresponsibilities for human oversight of GAI systems.

MP-1.1-009

Document the extent to which a lack of transparency or explainability impedes
risk measurement across the AI lifecycle.

MP-1.1-010

Identify and document foreseeable illegal uses or applications that surpass
organizational risk tolerances.

AI Actors: AI Deployment

125

Human AI Configuration

*MAP 1.2: Interdisciplinary AI actors, competencies, skills, and capacities for establishing context reflect demographic diversity andbroad domain and user experience expertise, and their participation is documented. Opportunities for interdisciplinary
collaboration are prioritized.
Action ID

ActionRisks

MP-1.2-001

Document the credentials and qualifications of organizational AI actors and AI
actor team composition.

Human AI Configuration

MP-1.2-002

Establish and empower interdisciplinary teams that reflect a wide range ofcapabilities, competencies, demographic groups, domain expertise, educational
Human AI Configuration, Toxicity,
backgrounds, lived experiences, professions, and skills across the enterprise toBias, and Homogenization
inform and conduct TEVV of GAI technology, and other risk measurement andmanagement functions.

MP-1.2-003

Establish continuous improvement processes to increase diversity andrepresentativeness in AI actor teams, standard measurement resources, andstructured public feedback participants from subgroup populations in-context.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MP-1.2-004

Verify that AI actor team membership includes demographic diversity,
applicable domain expertise, varied education backgrounds, and lived
experiences.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MP-1.2-005

Verify that data or benchmarks used in risk measurement, and users,
participants, or subjects involved in structured public feedback exercises are
representative of diverse in-context user populations.

Human AI Configuration, Toxicity,
Bias, and Homogenization

AI Actors: AI Deployment

1*MAP 2.1: The specific tasks and methods used to implement the tasks that the AI system will support are defined (e.g., classifiers,
generative models, recommenders).
Action ID

ActionRisks

MP-2.1-001

Define GAI system's task(s) that relate to content provenance, such as originalcontent creation, media synthesis, or data augmentation while incorporating
tracking measures.

Information IntegrityMP-2.1-002

Establish known assumptions and practices for determining data origin andcontent lineage, for documentation and evaluation.

Information IntegrityMP-2.1-003

Identify and document GAI task limitations that might impact the reliability orauthenticity of the content provenance.

Information Integrity26

MP-2.1-004

Institute audit trails for data and content flows within the system, including but
not limited to, original data sources, data transformations, and decision-making
criteria.

MP-2.1-005

Review efficacy of content provenance techniques on a regular basis andupdate protocols as necessary.

Information Integrity

AI Actors: TEVV1
MAP 2.2: Information about the AI system’s knowledge limits and how system output may be utilized and overseen by humans is
documented. Documentation provides sufficient information to assist relevant AI actors when making decisions and taking
subsequent actions.
Action ID

ActionRisks

MP-2.2-001

Assess whether the GAI system fulfills its intended purpose within its
operational context on a regular basis.

MP-2.2-002

Evaluate whether GAI operators and end-users can accurately understandcontent lineage and origin.

Human AI Configuration,
Information IntegrityMP-2.2-003

Identify and document how the system relies on upstream data sources forcontent provenance and if it serves as an upstream dependency for othersystems.

Information Integrity, Value Chain
and Component Integration

MP-2.2-004

Observe and analyze how the AI system interacts with external networks, andidentify any potential for negative externalities, particularly where content
provenance might be compromised.

Information IntegrityMP-2.2-005

Specify the environments where GAI systems may not function as intended
related to content provenance.

Information Integrity

AI Actors: End Users

2*MAP 2.3: Scientific integrity and TEVV considerations are identified and documented, including those related to experimental
design, data collection and selection (e.g., availability, representativeness, suitability), system trustworthiness, and constructvalidation
Action ID

ActionRisks

27

MP-2.3-001

Assess the accuracy, quality, reliability, and authenticity of the GAI content
provenance by comparing it to a set of known ground truth data and by using a
Information Integrity
variety of evaluation methods (e.g., human oversight and automatedevaluation).

MP-2.3-002

Curate and maintain high quality datasets that are accurate, relevant,consistent, and representative as well as be well-documented complying withethical and legal standards along with diverse data points.

Toxicity, Bias, and Homogenization

MP-2.3-003

Deploy and document fact-checking techniques to verify the accuracy andveracity of information generated by GAI systems, especially when theinformation comes from multiple (or unknown) sources.

Information IntegrityMP-2.3-004

Design GAI systems to support content provenance such as tracking the lineage(e.g., data sources used to train the system, parameters used to generate
Information Integrity
content, etc.) and to verify authenticity (e.g., using digital signatures orwatermarks).

MP-2.3-005

Develop and implement testing techniques to identify any GAI produced
content (e.g., synthetic media) that might be indistinguishable from humangenerated content.

Information IntegrityMP-2.3-006

Document GAI content provenance techniques (including experimental
methods), testing, evaluation, performance, and validation metrics throughout
the AI lifecycle.

Information IntegrityMP-2.3-007

Implement plans for GAI systems to undergo regular adversarial testing to
identify vulnerabilities and potential manipulation risks.

Information Security

MP-2.3-008

Integrate GAI systems with existing content management and version control
systems, to enable content provenance to be tracked across the lifecycle.

Information IntegrityMP-2.3-009

Test GAI models using known inputs, context, and environment to confirm they
produce expected outputs across a variety of methods (e.g., unit tests,
integration tests, and system tests) and help to identify and address potential
problems.

MP-2.3-010

Use diverse large-scale and small-scale datasets for testing and evaluation toensure that the AI system can perform well on a variety of different types ofdata.

Toxicity, Bias, and Homogenization

MP-2.3-011

Verify that GAI content provenance is accurate and reliable by using
cryptographic techniques and performing formal audits to ensure it has notbeen manipulated.

Information Integrity28

MP-2.3-012

Verify that the AI system’s content provenance complies with relevant laws andInformation Integrity, Intellectual
regulations, such as legal infringement, terms and conditions, copyright andProperty
intellectual property rights, when using data sources and generating content.

AI Actors: AI Development, Domain Experts, TEVV1
MAP 3.4: Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant
technical standards and certifications – are defined, assessed, and documented.
Action ID

ActionRisks

MP-3.4-001

Adapt existing training programs to include modules on content provenance.

Information Integrity

MP-3.4-002

Develop certification programs that test proficiency in managing AI risks andinterpreting content provenance, relevant to specific industry and context.

Information Integrity

MP-3.4-003

Delineate human proficiency tests from tests of AI capabilities.

Human-AI Configuration

MP-3.4-004

Integrate human and other qualitative inputs to comprehensively assess
content provenance.

Information Integrity

MP-3.4-005

Ensure that output provided to operators and practitioners is both interactive
and well-defined, incorporating content provenance data that can be easilyinterpreted for effective downstream decision-making.

Information Integrity, Value Chain
and Component Integration

MP-3.4-006

Information Integrity, Toxicity, Bias,
Establish and adhere to design principles that ensure safe and ethical operation,and Homogenization, Dangerous or
taking into account the interpretation of content provenance information.
Violent Recommendations

MP-3.4-007

Implement systems to continually monitor and track the outcomes of human-AI
Human AI Configuration,
collaborations for future refinement and improvements, integrating a focus on
Information Integrity
content provenance wherever applicable.

MP-3.4-008

Involve the end-users, practitioners, and operators in AI system prototyping andHuman AI Configuration,
testing activities. Make sure these tests cover various scenarios where content
Information Integrity, Toxicity, Bias,
provenance could play a critical role, such as crisis situations or ethicallyand Homogenization
sensitive contexts.

MP-3.4-009

Match the complexity of GAI system explanations and the provenance data to
the level of the problem and contextual intricacy.

Information Integrity

AI Actors: AI Design, AI Development, Domain Experts, End-Users, Human Factors, Operation and Monitoring

229

*MAP 4.1: Approaches for mapping AI technology and legal risks of its components – including the use of third-party data orsoftware – are in place, followed, and documented, as are risks of infringement of a third party’s intellectual property or otherrights.
Action ID

ActionRisks

MP-4.1-001

Conduct audits on third-party processes and personnel including an examination Value Chain and Componentof the third-party’s reputation.
Integration

MP-4.1-002

Conduct periodic audits and monitor AI generated content for privacy risks;
address any possible instances of sensitive data exposure.

MP-4.1-003

Consider using synthetic data as applicable to train AI models in place of realworld data to match the statistical properties of real-world data without
disclosing personally identifiable information.

MP-4.1-004

Develop practices for periodic monitoring of GAI outputs for possible intellectual
property infringements and other risks and implement processes for responding Intellectual Property
to potential intellectual property infringement claims.

MP-4.1-005

Document all aspects of the AI development process including data sources,
model architectures and training procedures to support reproduction of results,
identify any potential problems, and implement mitigation strategies.

MP-4.1-006

Document compliance with legal requirements across the AI lifecycle, including
copyright concerns, privacy protections.

Data Privacy, Intellectual Property

MP-4.1-007

Document training data curation policies, including policies to verify thatconsent was obtained for the likeness or image of individuals.

Obscene, Degrading, and/or
Abusive Content

MP-4.1-008

Employ encryption techniques and proper safeguards to ensure secure data
storage and transfer to protect data privacy.

Data Privacy, Information Security,
Dangerous or Violent
Recommendations

MP-4.1-009

Establish policies for collection, retention, and minimum quality of data, inconsideration of the following risks: Disclosure of CBRN information by removing CBRN Information, Intellectual
CBRN information from training data, Use of Illegal or dangerous content;
Property, Toxicity, Bias, andTraining data imbalance across sub-groups by modality, such as languages for
Homogenization, Dangerous or
LLMs or skin tone for image generation; Leak of personally identifiable
Violent Recommendations, Data
information, including facial likenesses of individuals unless consent is obtained Privacy
for use of their images.

MP-4.1-010

Implement bias mitigation approaches by addressing sources of bias in the
training data and by evaluating AI models for bias periodically.

Toxicity, Bias, and Homogenization

MP-4.1-011

Implement policies and practices defining how third-party intellectual property
and training data will be used, stored, and protected.

Intellectual Property, Value Chain
and Component Integration

30

Data Privacy

MP-4.1-012

Implement reproducibility techniques, including: share data publicly or privately
using license and citation; develop code according to standard software
practices; track and document experiments and results; manage the software
Confabulation, Intellectual
environment and dependencies; utilize virtual environments, version control,
Property, Value Chain andand maintain a requirements document; manage models and artifacts; tracking
Component Integration
AI model versions and documenting model details along with parameters andexperimental results; document data management processes and establish atesting/validation process to maintain reliable results.

MP-4.1-013

Re-evaluate models that were fine-tuned on top of third-party models.

MP-4.1-014

Re-evaluate risks when adapting GAI models to new domains.

MP-4.1-015

Review service level agreements and contracts, including license agreements
Intellectual Property, Value Chain
and any legal documents associated with the third-party intellectual properties,
and Component Integration
technologies, and services.

MP-4.1-016

Use approaches to detect the presence of sensitive data in generated output
text, image, video, or audio, and verify that the model will mask any detected
sensitive data.

MP-4.1-017

Use trusted sources for training data that are licensed or open source andIntellectual Property
ensure that the entity has the legal right for the use of proprietary training data.

MP-4.1-018

Apply strong anonymization and de-identification, and/or differential privacy
techniques to protect the privacy of individuals in the training data.

Data Privacy

MP-4.1-019

Verify that third-party models are in compliance with existing use licenses.

Intellectual Property, Value Chain
and Component Integration

Value Chain and Component
Integration

Information Integrity

AI Actors: Governance and Oversight, Operation and Monitoring, Procurement, Third-party entities

1*MAP 5.1: Likelihood and magnitude of each identified impact (both potentially beneficial and harmful) based on expected use,
past uses of AI systems in similar contexts, public incident reports, feedback from those external to the team that developed or
deployed the AI system, or other data are identified and documented.
Action ID

ActionRisks

MP-5.1-001

Apply TEVV practices for content provenance (e.g., probing a system's synthetic
Information Integrity, Information
data generation capabilities for potential misuse or vulnerabilities using zeroSecurity
knowledge proof approaches).

31

MP-5.1-002

Assess and document risks related to content provenance. e.g., document thepresence, absence, or effectiveness of tagging systems, cryptographic hashes,
blockchain-based, or distributed ledger technology solutions that improve
content tracking transparency and immutability.

MP-5.1-003

Consider GAI-specific mapped risks (e.g., complex security requirements,
Human AI Configuration,
potential for emotional entanglement of users, large supply chains) in estimates Information Security, Value Chain
for likelihood, magnitude of impact and risk.
and Component Integration

MP-5.1-004

Document estimates of likelihood, magnitude of impact, and risk for GAI
systems in a central repository (e.g., organizational AI inventory.).

MP-5.1-005

Enumerate potential impacts related to content provenance, including bestcase, average-case, and worst-case scenarios.

Information Integrity

MP-5.1-006

Estimate likelihood of enumerated impact scenarios using past data or expert
judgment, analysis of known public incidents, standard measurement, andstructured human feedback results.

CBRN Information, Dangerous orViolent Recommendations

MP-5.1-007

Measure risk as the product of estimated likelihood and magnitude of impact ofa GAI outcome.

MP-5.1-008

Prioritize risk acceptance, management, or transfer activities based on risk
estimates.

MP-5.1-009

Prioritize standard measurement and structured public feedback processes
based on risk assessment estimates.

MP-5.1-010

Profile risks arising from GAI systems interacting with, manipulating, or
generating content, and outlining known and potential vulnerabilities and thelikelihood of their occurrence.

MP-5.1-011

Scope GAI applications narrowly to enable risk-based governance and controls.

Information Integrity

Information Security

AI Actors: AI Deployment, AI Design, AI Development, AI Impact Assessment, Affected Individuals and Communities, End-Users,
Operation and Monitoring

1

32

MAP 5.2: Practices and personnel for supporting regular engagement with relevant AI actors and integrating feedback about
positive, negative, and unanticipated impacts are in place and documented.
Action ID

ActionRisks

MP-5.2-001

Determine context-based measures to identify if new impacts are present due to
Human AI Configuration, Value
the GAI system, including regular engagements with downstream AI actors to
Chain and Component Integration
identify and quantify new contexts of unanticipated impacts of GAI systems.

MP-5.2-002

Plan regular engagements with AI actors responsible for inputs to GAI systems,
Human AI Configuration, Value
including third-party data and algorithms, to review and evaluate unanticipated
Chain and Component Integration
impacts.

MP-5.2-003

Publish guidance for external AI actors to report unanticipated impacts of the
GAI system and to engage with the organization in the event of GAI system
impacts.

Human AI Configuration

AI Actors: AI Deployment, AI Design, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, End-Users,
Human Factors, Operation and Monitoring

1*MEASURE 1.1: Approaches and metrics for measurement of AI risks enumerated during the MAP function are selected for
implementation starting with the most significant AI risks. The risks or trustworthiness characteristics that will not – or cannot – bemeasured are properly documented.
Action ID

ActionRisks

MS-1.1-001

Assess the effectiveness of implemented methods and metrics at an ongoingcadence as part of continuous improvement activities.

MS-1.1-002

Collaborate with multidisciplinary experts (e.g., in the fields of responsible useof GAI, cybersecurity, or digital forensics) to ensure the selected risk
management approaches are robust and effective.

Information Security; CBRN
Information, Toxicity, Bias, andHomogenization

MS-1.1-003

Conduct adversarial role-playing exercises, AI red-teaming, or chaos testing to
identify anomalous or unforeseen failure modes.

Information Security, Unknowns

MS-1.1-004

Conduct traditional assessment or TEVV exercises to measure the prevalence ofknown risks in deployment contexts.

MS-1.1-005

Document GAI risk measurement or tracking approaches, including tracking ofrisks that cannot be easily measured before deployment (e.g., ecosystem-levelrisks or risks that unfold over longer time scales).

33MS-1.1-006

Employ digital signatures and watermarking, blockchain technology, reverse
image and video search, metadata analysis, steganalysis, and/or forensic
analysis to trace the origin and modifications of digital content.

Information IntegrityMS-1.1-007

Employ similarity metrics, tampering indicators, blockchain confirmation,
metadata consistency, hidden data detection rate, source reliability, andconsistency with known patterns to measure content provenance risks.

Information IntegrityMS-1.1-008

Identify content provenance risks in the end-to-end AI supply chain, including
risks associated with data suppliers, data annotators, R&D, joint ventures,
academic or nonprofit projects/partners, third party vendors, and contractors.

Information Integrity, Value Chain
and Component Integration

MS-1.1-009

Identify potential content provenance risks and harms in GAI, such as
misinformation or disinformation, deepfakes, including NCII, or tampered
content. Enumerate and rank risks and/or harms based on their likelihood andpotential impact, and determine how well provenance solutions address
specific risks and/or harms.

Information Integrity, Dangerous orViolent Recommendations,Obscene, Degrading, and/or
Abusive Content

MS-1.1-010

Implement appropriate approaches and metrics for measuring AI-related
content provenance the and the aforementioned risks and harms.

Information Integrity, Dangerous orViolent Recommendations

MS-1.1-011

Integrate tools designed to analyze content provenance and detect data
anomalies, verify the authenticity of digital signatures, and identify patterns
associated with misinformation or manipulation.

Information IntegrityMS-1.1-012

Invest in R&D capabilities to evaluate and implement novel methods andtechnologies for the measurement of AI-related risks in content provenance,
toxicity, and CBRN.

Information Integrity, CBRN
Information, Obscene, Degrading,
and/or Abusive Content

MS-1.1-013

Prioritize risk measurement according to risk severity as determined duringmapping activities.

MS-1.1-014

Provide content provenance risk management education to AI actors, users, and Human AI Configuration,
stakeholders.
Information IntegrityMS-1.1-015

Track and document risks or opportunities related to content provenance thatcannot be measured quantitatively, including explanations as to why some risks
Information Integrity
cannot be measured (e.g., due to technological limitations, resource
constraints, or trustworthy considerations).

MS-1.1-016

Track the number of output data items that are accompanied by provenanceinformation (e.g., watermarks, cryptographic tags).

Information IntegrityMS-1.1-017

Track the number of training and input (e.g., prompts) data items that have
provenance records and output data items that potentially infringe on
intellectual property rights.

Information Integrity, Intellectual
Property34

MS-1.1-018

Track the number of training and input data items covered by intellectual
property rights (e.g., copyright, trademark, trade secret).

MS-1.1-019

Validate the reliability and integrity of the original data and measure inherent
dependence on training data and its quality.

Intellectual Property

AI Actors: AI Development, Domain Experts, TEVV

1
*MEASURE 1.3: Internal experts who did not serve as front-line developers for the system and/or independent assessors are
involved in regular assessments and updates. Domain experts, users, AI actors external to the team that developed or deployed theAI system, and affected communities are consulted in support of assessments as necessary per organizational risk tolerance
Action ID

ActionRisks

MS-1.3-001

Define relevant groups of interest (e.g., demographic groups, subject matter
Human AI Configuration, Toxicity,
experts, past experience with GAI technology) within the context of use as part
Bias, and Homogenization, CBRN
of plans for gathering structured public feedback.

MS-1.3-002

Define sequence of actions for AI red-teaming exercises and accompanying
necessary documentation practices.

MS-1.3-003

Define use cases, contexts of use, capabilities, and negative impacts where
structured human feedback exercises, e.g., AI red-teaming, would be mostbeneficial for AI risk measurement and management based on the context ofuse.

MS-1.3-004

Develop a suite of suitable metrics to evaluate structured feedback results,
informed by representative AI actors.

Human AI Configuration, Toxicity,
Bias, and Homogenization, CBRN

MS-1.3-005

Execute independent audit, AI red-teaming, impact assessments, or otherstructured human feedback processes in consultation with representative AI
actors with expertise and familiarity in the context of use, and/or who are
representative of the populations associated with the context of use.

Human AI Configuration, Toxicity,
Bias, and Homogenization, CBRN

MS-1.3-006

Identify and implement methods for post-hoc evaluation of the effectiveness ofstructured human feedback processes such as auditing, impact assessments,
and AI red-teaming.

MS-1.3-007

Identify and implement methods for translating, evaluating, and integrating
structured human feedback output into AI risk management processes,
continuous improvement processes, and related organizational decision
making.

MS-1.3-008

Identify criteria for determining when structured human feedback exercises arecomplete.

35MS-1.3-009

Identify mechanisms and teams to evaluate or other structured human
feedback outcomes.

MS-1.3-010

Recruit auditors, AI red-teams, and structured feedback participants inconsideration of the linguistic, dialectal, and socio-cultural environment of the
expected user base.

Human AI Configuration

MS-1.3-011

Share structured feedback with relevant AI actors to address identified risks.

Human AI Configuration

MS-1.3-012

Verify demographic diversity of identified subgroups in structured feedback
exercises.

Toxicity, Bias, and Homogenization

MS-1.3-013

Verify those conducting structured human feedback exercises are not directly
involved in system development tasks for the same GAI model.

AI Actors: AI Deployment, AI Development, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, EndUsers, Operation and Monitoring, TEVV

1
*MEASURE 2.2: Evaluations involving human subjects meet applicable requirements (including human subject protection) and are
representative of the relevant population.
Action ID

ActionRisks

MS-2.2-001

Information Integrity, Information
Assess and manage statistical biases related to GAI content provenance throughSecurity, Toxicity, Bias, andtechniques such as re-sampling, re-weighting, or adversarial training.
Homogenization

MS-2.2-002

Disaggregate evaluation metrics by demographic factors to identify any
discrepancies in how content provenance mechanisms work across diverse
populations.

MS-2.2-003

Data Privacy, Human AI
Document how content provenance mechanisms are operated in the context ofConfiguration, Information
privacy and security including: Anonymize data to protect the privacy of humanIntegrity, Information Security,
subjects; Remove any personally identifiable information (PII) to prevent
Dangerous or Violent
potential harm or misuse.
Recommendations

MS-2.2-004

Employ techniques like chaos engineering and stakeholder feedback to evaluate
Information Integrity
the quality and integrity of data used in training and the provenance of AIgenerated content.

MS-2.2-005

Identify biases present in the training data for downstream mitigation using
available techniques (e.g., data visualization tools).

36

Information Integrity, Toxicity, Bias,and Homogenization

Value Chain and Component
Integration, Toxicity, Bias, andHomogenization

MS-2.2-006

Implement continuous monitoring of GAI system impacts to identify whether
GAI outputs are equitable across various sub-populations. Seek active anddirect feedback from affected communities to identify issues and improve GAI
system fairness.

Toxicity, Bias, and Homogenization

MS-2.2-007

Implement robust cybersecurity measures to protect both the research data,
the GAI system and its content provenance from unauthorized access,breaches, or tampering and unauthorized disclosure of human subject
information.

Data Privacy, Human AI
Configuration, Information
Integrity, Information SecurityMS-2.2-008

Obtain informed consent from human subject evaluation participants. Informed Data Privacy, Human AI
consent should include: the nature of the study, information about the use ofConfiguration, Information
GAI related to content provenance, its purpose, and potential implications.
IntegrityMS-2.2-010

Information Integrity, Information
Practice responsible disclosure of findings and report discovered vulnerabilities
Security, Toxicity, Bias, andor biases related to GAI systems and its content provenance.
Homogenization

MS-2.2-011

Provide human subjects with options to revoke their consent for future use of
their data in GAI applications, particularly in content provenance aspects.

Data Privacy, Human AI
Configuration, Information
IntegrityMS-2.2-012

Use Institutional Review Boards as applicable for evaluations that involve
human subjects.

Human AI Configuration

MS-2.2-013

Use techniques such as anonymization or differential privacy to minimize therisks associated with linking AI-generated content back to individual humansubjects.

Data Privacy, Human AI
Configuration

MS-2.2-014

Verify accountability and fairness through documentation of the algorithms,
parameters, and methodologies used in the evaluation to allow for external
scrutiny.

Toxicity, Bias, and Homogenization

MS-2.2-015

Verify that human subjects selected for evaluation are representative of the
population for the relevant GAI use-case; Consider demographics such as age,gender, race, ethnicity, socioeconomic status, and geographical location to
avoid biases in the AI system related to content provenance.

Human AI Configuration,
Information Integrity, Toxicity, Bias,and Homogenization

MS-2.2-016

Work in close collaboration with domain experts to understand the specificrequirements and potential pitfalls related to content provenance in the GAI
system's intended context of use.

Information Integrity

AI Actors: AI Development, Human Factors, TEVV

137

*MEASURE 2.3: AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for
conditions similar to deployment setting(s). Measures are documented.
Action ID

ActionRisks

MS-2.3-001

Analyze differences between intended and actual population of users or data
subjects, including likelihood for errors, incidents, or negative impacts.

Confabulation, Human AI
Configuration, Information
IntegrityMS-2.3-002

Conduct field testing on sampled sub-populations prior to deployment to theentire population.

MS-2.3-003

Conduct TEVV in the operational environment in accordance with organizational
policies and regulatory or disciplinary requirements (e.g., informed consent,
Data Privacy
institutional review board approval, human research protections, privacy
requirements).

MS-2.3-004

Consider baseline model performance on suites of benchmarks when selecting amodel for fine tuning.

MS-2.3-005

Evaluate claims of model capabilities using empirically validated methods.

MS-2.3-006

Include metrics measuring reporting rates for harmful or offensive content infield testing.

Dangerous or Violent
Recommendations

MS-2.3-007

Share results of pre-deployment testing with relevant AI actors, such as thosewith system release approval authority.

Human AI Configuration

MS-2.3-008

Use disaggregated evaluation methods (e.g., by race, age, gender, ethnicity,
ability, region) to improve granularity of AI system performance measures.

MS-2.3-009

Utilize a purpose-built testing environment such as NIST Dioptra to empiricallyevaluate GAI trustworthy characteristics.

MS-2.3-010

Verify that mechanisms to collect users’ feedback are visible and traceable.

Human AI Configuration

AI Actors: AI Deployment, TEVV

1
*MEASURE 2.5: The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond
the conditions under which the technology was developed are documented.
Action ID

ActionRisks

38

MS-2.5-001

Apply standard measurement and structured human feedback approaches tointernally-developed and third-party GAI systems.

MS-2.5-002

Avoid extrapolating GAI system performance or capabilities from narrow, nonsystematic, and anecdotal assessments.

MS-2.5-003

Conduct security assessments and audits to measure the integrity of training
data, system software, and system outputs.

MS-2.5-004

Document the construct validity of methodologies employed in GAI systems
relative to their context of use.

MS-2.5-005

Document the extent to which human domain knowledge is employed to
improve GAI system performance, via, e.g., RLHF, fine-tuning, content
moderation, business rules.

MS-2.5-006

Establish metrics or KPIs to determine whether GAI systems meet minimum
performance standards for reliability and validity.

MS-2.5-007

Measure, monitor, and document prevalence of erroneous GAI output content,system availability, and reproducibility of outcomes via field testing or otherrandomized controlled experiments.

MS-2.5-008

Review and verify sources and citations in GAI system outputs during predeployment risk measurement and ongoing monitoring activities.

Confabulation

MS-2.5-009

Track and document instances of anthropomorphization (e.g., human images,
mentions of human feelings, cyborg imagery or motifs) in GAI system interfaces.

Human AI Configuration

MS-2.5-010

Track and document relevant version numbers, planned updates, hotfixes, andother GAI system change management information.

MS-2.5-011

Update standard train/test model evaluation processes for GAI systems. Consider:
Unwanted or undocumented overlaps in train and TEVV data sources, including
their negative spaces (i.e., what is not represented in both); Employing substring
matching or embedding distance approaches to assess similarity across data
partitions.

MS-2.5-012

Verify GAI system training data and TEVV data provenance, and that fine-tuning
data is grounded.

AI Actors: Domain Experts, TEVV

1

39

Value Chain and Component
Integration

Information Security

Information Integrity

*MEASURE 2.6: The AI system is evaluated regularly for safety risks – as identified in the MAP function. The AI system to be
deployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if
made to operate beyond its knowledge limits. Safety metrics reflect system reliability and robustness, real-time monitoring, andresponse times for AI system failures.
Action ID

ActionRisks

MS-2.6-001

Human AI Configuration, Obscene,
Degrading, and/or Abusive
Assess adverse impacts health and wellbeing impacts for supply chain or other AI
Content, Value Chain andactors that are exposed to obscene, toxic, or violent information during theComponent Integration,
course of GAI training and maintenance.
Dangerous or Violent
Recommendations

MS-2.6-002

Data Privacy, Intellectual Property,
Obscene, Degrading, and/or
Assess levels of toxicity, intellectual property infringement, data privacy
Abusive Content, Toxicity, Bias,
violations, obscenity, extremism, violence, or CBRN information in system training
and Homogenization, Dangerous
data.
or Violent Recommendations,
CBRN Information

MS-2.6-003

Measure and document incident response times, system down times, and system
availability: Perform standard measurement and structured human feedback on
GAI systems to detect safety and reliability impacts and harms; Apply humansubjects research protocols and other applicable safety controls when conducting
Data Privacy, Human AI
A/B testing, AI red-teaming, focus groups, or human testbed measurements;
Configuration, Toxicity, Bias, andIdentify and document any applications related to robotics, RPA, and autonomous
Homogenization, Dangerous orvehicles; Conduct AI red-teaming exercises to identify harms and impacts related
Violent Recommendations
to safety and validity, reliability, privacy, toxicity and other risks; Monitor high-risk
GAI systems continually for safety and reliability risks once deployed; Monitor GAI
systems to detect drift and anomalies relative to expected performance andtraining baselines.

MS-2.6-004

Re-evaluate safety features of fine-tuned models when the risk of harm exceeds
organizational risk tolerance.

Dangerous or Violent
Recommendations

MS-2.6-005

Review GAI system outputs for validity and safety: Review generated code toassess risks that may arise from unreliable downstream decision-making.

Value Chain and Component
Integration, Dangerous or Violent
Recommendations

MS-2.6-006

Track and document past failed GAI system designs to inform risk measurement
for safety and validity risks.

Dangerous or Violent
Recommendations

MS-2.6-007

Verify capabilities for limiting, pausing, updating, or terminating GAI systems
quickly.

MS-2.6-008

Verify rollover, fallback, or redundancy capabilities for high-risk GAI systems.

40MS-2.6-009

Verify that GAI system architecture can monitor outputs and performance, andhandle, recover from, and repair errors when security anomalies, threats andimpacts are detected.

MS-2.6-010

Verify that systems properly handle queries that may give rise to inappropriate,
CBRN Information, Information
malicious, or illegal usage, including facilitating manipulation, extortion, targeted
Security
impersonation, cyber-attacks, and weapons creation.

Confabulation, Information
Integrity, Information Security

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV

1
MEASURE 2.7: AI system security and resilience – as identified in the MAP function – are evaluated and documented.
Action ID

ActionRisks

MS-2.7-001

Apply established security measures to: Assess risks of backdoors, compromised
dependencies, data breaches, eavesdropping, man-in-the-middle attacks, reverse
Data Privacy, Informationengineering other baseline security concerns; Audit supply chains to identify
Integrity, Information Security,
risks arising from, e.g., data poisoning and malware, software and hardware
Value Chain and Component
vulnerabilities, third-party personnel and software; Audit GAI systems, pipelines,
Integration
plugins and other related artifacts for unauthorized access, malware, and otherknown vulnerabilities.

MS-2.7-002

Assess the completeness of documentation related to data provenance, access
controls, and incident response procedures. Verify GAI system content
provenance documentation aligns with relevant regulations and standards.

Information Integrity, Toxicity,
Bias, and Homogenization

MS-2.7-003

Benchmark GAI system security and resilience related to content provenanceagainst industry standards and best practices. Compare GAI system security
features and content provenance methods against industry state-of-the-art.

Information Integrity, Information
SecurityMS-2.7-004

Conduct user surveys to gather user satisfaction with the AI-generated content
and user perceptions of content authenticity. Analyze user feedback to identify
concerns and/or current literacy levels related to content provenance.

Human AI Configuration,
Information IntegrityMS-2.7-005

Engage with security experts, developers, and researchers through information
sharing mechanisms to stay updated with the latest advancements in AI security
Information Integrity, Information
related to content provenance. Contribute findings related to AI system security
Security
and content provenance via information sharing mechanisms, workshops, or
publications.

MS-2.7-006

Establish measures and evaluate GAI resiliency as part of pre-deployment testing
to ensure GAI will function under adverse conditions and restore fullfunctionality in a trustworthy manner.

41

MS-2.7-007

Identify metrics that reflect the effectiveness of security measures, such as data
provenance, the number of unauthorized access attempts, penetrations, or
provenance verification.

Information Integrity, Information
SecurityMS-2.7-008

Maintain awareness of emergent GAI security risks and associatedcountermeasures through community resources, official guidance, or research
literature.

Information Security, Unknowns

MS-2.7-009

Measure reliability of content provenance verification methods, such as
watermarking, cryptographic signatures, hashing, blockchain, or other content
Information Integrity
provenance techniques. Evaluate the rate of false positives and false negatives incontent provenance, as well as true positives and true negatives for verification.

MS-2.7-010

Measure the average response time to security incidents related to content
provenance, and the proportion of incidents resolved with and without
significant impact.

Information Integrity, Information
SecurityMS-2.7-011

Measure the rate at which recommendations from security audits and incidents
are implemented related to content provenance. Assess how quickly the AI
system can adapt and improve based on lessons learned from security incidents
and feedback related to content provenance.

Information Integrity, Information
SecurityMS-2.7-012

Monitor and review the completeness and validity of security documentationand verify it aligns with the current state of the GAI system and its content
provenance.

Information Integrity, Information
Security, Toxicity, Bias, andHomogenization

MS-2.7-013

Monitor GAI system downtime and measure its impact on operations.

MS-2.7-014

Monitor GAI systems in deployment for anomalous use and security risks.

Information SecurityMS-2.7-015

Monitor the number of security-related incident reports from users, indicating
their awareness and willingness to report issues.

Human AI Configuration,
Information SecurityMS-2.7-016

Perform AI red-teaming to assess resilience against: Abuse to facilitate attacks on Information Security, Toxicity,
other systems (e.g., malicious code generation, enhanced phishing content), GAI Bias, and Homogenization,
attacks (e.g., prompt injection), ML attacks (e.g., adversarial examples/prompts, Dangerous or Violent
data poisoning, membership inference, model extraction, sponge examples).
Recommendations

MS-2.7-017

Review deployment approval processes and verify that processes address
relevant GAI security risks.

MS-2.7-018

Review incident response procedures and verify adequate functionality toidentify, contain, eliminate, and recover from complex GAI system incidents that
implicate impacts across the trustworthy characteristics.

MS-2.7-019

Track and document access and updates to GAI system training data; verify
appropriate security measures for training data at GAI vendors and service
providers.

42

Information Security

Information Security, Value Chain
and Component Integration

MS-2.7-020

Track GAI system performance metrics such as response time and throughput
under different loads and usage patterns related to content provenance.

Information IntegrityMS-2.7-021

Track the number of users who have completed security training programs
regarding the security of content provenance.

Human AI Configuration,
Information Integrity, Information
SecurityMS-2.7-022

Verify fine-tuning does not compromise safety and security controls.

Information Integrity, Information
Security, Dangerous or Violent
Recommendations

MS-2.7-023

Verify organizational policies, procedures, and processes for treatment of GAI
security and resiliency risks.

Information SecurityMS-2.7-024

Verify vendor documentation for data and software security controls.

Information Security, Value Chain
and Component Integration

MS-2.7-025

Work with domain experts to capture stakeholder confidence in GAI systemsecurity and perceived effectiveness related to content provenance.

Information Integrity, Information
Security

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV

1
MEASURE 2.8: Risks associated with transparency and accountability – as identified in the MAP function – are examined anddocumented.
Action ID

ActionRisks

MS-2.8-001

Compile and communicate statistics on policy violations, take-down requests,
intellectual property infringement, and information integrity for organizational
GAI systems: Analyze transparency reports across demographic groups,
languages groups, and other segments relevant to the deployment context.

Information Integrity, Intellectual
Property, Toxicity, Bias, andHomogenization

MS-2.8-002

Document the instructions given to data annotators or AI red-teamers.

MS-2.8-003

Document where in the data pipeline human labor is being used.

MS-2.8-004

Establish a mechanism for appealing usage policy violations.

MS-2.8-005

Maintain awareness of AI regulations and standards in relevant jurisdictions
related to GAI systems and content provenance.

Information Integrity

Measure the effectiveness or accessibility of procedures to appeal adverse,
harmful, or incorrect outcomes from GAI systems.

Human AI Configuration, Toxicity,
Bias, and Homogenization,
Dangerous or Violent
Recommendations

MS-2.8-006

43

MS-2.8-007

Review and consider GAI system transparency artifacts such as impact
assessments, system cards, model cards, and traditional risk management
documentation as part of organizational decision making.

MS-2.8-008

Review licenses, patents, or other intellectual property rights pertaining to
information in system training data.

Intellectual Property

MS-2.8-009

Track AI actor decisions along the lifecycle to determine sources of systemic andcognitive bias and identify management and mitigation approaches.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MS-2.8-010

Use interpretable machine learning techniques to make AI processes andoutcomes more transparent, and easier to understand how decisions are made.

MS-2.8-011

Use technologies such as blockchain and digital signatures to enable thedocumentation of each instance where content is generated, modified, or shared
Information Integrity
to provide a tamper-proof history of the content, promote transparency, andenable traceability. Robust version control systems can also be applied to trackchanges across the AI lifecycle over time.

MS-2.8-012

Verify adequacy of GAI system user instructions through user testing.

MS-2.8-013

Verify that accurate information about GAI capabilities, opportunities, risks, andpotential negative impacts are available on websites, press releases,
organizational reports, social media, and public communication channels.

MS-2.8-014

Verify the adequacy of feedback functionality in system user interfaces.

MS-2.8-015

Verify the adequacy of redress processes for severe GAI system impacts.

Human AI Configuration

Human AI Configuration

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV

1
MEASURE 2.9: The AI model is explained, validated, and documented, and AI system output is interpreted within its context – asidentified in the MAP function – to inform responsible use and governance.
Action ID

ActionRisks

MS-2.9-001

Apply and document ML explanation results such as: Analysis of embeddings,
Counterfactual prompts, Gradient-based attributions, Modelcompression/surrogate models, Occlusion/term reduction.

MS-2.9-002

Apply transparency tools such as Datasheets, Data Nutrition Labels, and Model
Cards to record explanatory and validation information.

44

MS-2.9-003

Document GAI model details including: Proposed use and organizational value;
Assumptions and limitations, Data collection methodologies; Data provenance;
Data quality; Model architecture (e.g., convolutional neural network,transformers, etc.); Optimization objectives; Training algorithms; RLHF
approaches; Fine-tuning approaches; Evaluation data; Ethical considerations;
Legal and regulatory requirements.

Information Integrity, Toxicity,
Bias, and Homogenization

MS-2.9-004

Measure and report: Comparisons to alternative approaches and benchmarks;
Outcomes across demographic groups, languages groups, and other segments
relevant to the deployment context; Reproducibility of outcomes or internalmechanisms; Sensitivity analysis and stress-testing results.

Toxicity, Bias, and Homogenization

MS-2.9-005

Verify calibration and robustness of applied explanation techniques anddocument their assumptions and limitations.

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV

1
*MEASURE 2.10: Privacy risk of the AI system – as identified in the MAP function – is examined and documented.
Action ID

ActionRisks

MS-2.10-001

Collaborate with other AI actors, domain experts, and legal advisors to evaluate Data Privacy, Human AI
the impact of GAI applications on privacy related to the GAI system and its
Configuration, Information
content provenance, in domains such as healthcare, finance, and criminal justice. IntegrityMS-2.10-002

Conduct AI red-teaming to assess GAI system risks such as: Outputting of training
data samples, and subsequent reverse engineering, model extraction, andHuman AI Configuration,membership inference risks; Revealing biometric, confidential, copyrighted,
Intellectual Property
licensed, patented, personal, proprietary, sensitive, or trade-marked; Tracking or
revealing location information of users or members of training datasets.

MS-2.10-003

Document collection, use, management, and disclosure of biometric,
Data Privacy, Human AI
confidential, copyrighted, licensed, patented, personal, proprietary, sensitive, orConfiguration, Intellectual
trade-marked information in datasets, in accordance with privacy and data
Property
governance policies and data privacy laws.

MS-2.10-004

Engage directly with end-users and other stakeholders to understand their
expectations and concerns regarding content provenance. Use this feedback toguide the design of provenance-tracking mechanisms.

MS-2.10-005

Establish and document protocols (authorization, duration, type) and access
controls for training sets or production data containing biometric, confidential,
copyrighted, licensed, patented, personal, proprietary, sensitive, or trade-marked Data Privacy, Intellectual Property
information, in accordance with privacy and data governance policies and data
privacy laws.

45

Human AI Configuration,
Information IntegrityMS-2.10-006

Implement consent mechanisms that are demonstrated to allow users tounderstand and control how their data is used in the GAI system and its content
provenance.

Data Privacy, Human AI
Configuration, Information
IntegrityMS-2.10-007

Implement mechanisms to monitor, periodically review and document theprovenance data to detect any inconsistencies or unauthorized modifications.

Information Integrity, Information
SecurityMS-2.10-008

Implement zero-knowledge proofs to balance transparency with privacy andallow verification of claims about content without exposing the actual data.

Data Privacy

MS-2.10-009

Leverage technologies such as blockchain to document the origin of, and any
subsequent modifications to, generated content to enhance transparency andprovide a secure method for provenance tracking.

Information Integrity, Information
SecurityMS-2.10-010

Track training, input and output items that contains personally identifiable
information.

Data Privacy

MS-2.10-011

Verify compliance with data protection regulations.

Data Privacy

MS-2.10-012

Verify deduplication of training data samples.

Toxicity, Bias, and Homogenization

MS-2.10-013

Verify organizational policies, procedures, and processes for GAI systems address
fundamental tenets of data privacy, e.g., Anonymization of private data; Consent
Data Privacy, Human AI
to use data for targeted purposes or applications; Data collection and use in
Configurationaccordance with legal requirements and organizational policies; Reasonable dataretention limits and requirements; User data deletion and rectification requests.

MS-2.10-014

Verify that biometric, confidential, copyrighted, licensed, patented, personal,
proprietary, sensitive, or trade-marked information are removed from GAI
training data.

Intellectual Property

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV

1
*MEASURE 2.11: Fairness and bias – as identified in the MAP function – are evaluated and results are documented.
Action ID

ActionRisks

MS-2.11-001

Apply use-case appropriate benchmarks (e.g., Bias Benchmark Questions, Real
Toxicity Prompts, Winogender) to quantify systemic bias, stereotyping,denigration, and toxicity in GAI system outputs; Document assumptions andlimitations of benchmarks relative to in-context deployment environment.

Toxicity, Bias, and Homogenization

MS-2.11-002

Assess content moderation and other output filtering technologies or processes
for risks arising from human, systemic, and statistical/computational biases.

Toxicity, Bias, and Homogenization46

MS-2.11-003

Conduct fairness assessments to measure systemic bias. Measure GAI system
performance across demographic groups and subgroups, addressing both qualityof service and any allocation of services and resources. Identify types of harms,
including harms in resource allocation, representational, quality of service,
stereotyping, or erasure, Identify across, within, and intersecting groups thatmight be harmed; Quantify harms using: field testing with sub-group populations
to determine likelihood of exposure to generated content exhibiting harmful
bias, AI red-teaming with counterfactual and low-context (e.g., “leader,” “badguys”) prompts. For ML pipelines or business processes with categorical or
numeric outcomes that rely on GAI, apply general fairness metrics (e.g.,
Toxicity, Bias, anddemographic parity, equalized odds, equal opportunity, statistical hypothesis
Homogenization, Dangerous or
tests), to the pipeline or business outcome where appropriate; Custom, contextViolent Recommendations
specific metrics developed in collaboration with domain experts and affected
communities; Measurements of the prevalence of denigration in generated
content in deployment (e.g., sub-sampling a fraction of traffic and manuallyannotating denigrating content); Analyze quantified harms for contextuallysignificant differences across groups, within groups, and among intersecting
groups; Refine identification of within-group and intersectional group disparities,
Evaluate underlying data distributions and employ sensitivity analysis during theanalysis of quantified harms, Evaluate quality metrics including differential
output across groups, Consider biases affecting small groups, within-group orintersectional communities, or single individuals.

MS-2.11-004

Evaluate practices along the lifecycle to identify potential sources of humancognitive bias such as availability, observational, groupthink, funding, andconfirmation bias, and to make implicit decision-making processes more explicit
and open to investigation.

Toxicity, Bias, and Homogenization

MS-2.11-005

Identify the classes of individuals, groups, or environmental ecosystems whichmight be impacted by GAI systems through direct engagement with potentially
impacted communities.

Environmental, Toxicity, Bias, andHomogenization

MS-2.11-006

Monitor for representational, financial, or other harms after GAI systems are
deployed.

Toxicity, Bias, andHomogenization, Dangerous orViolent Recommendations

MS-2.11-007

Review, document, and measure sources of bias in training and TEVV data:
Differences in distributions of outcomes across and within groups, includingintersecting groups; Completeness, representativeness, and balance of data
sources; demographic group and subgroup coverage in GAI system training data;
Forms of latent systemic bias in images, text, audio, embeddings, or otherToxicity, Bias, andcomplex or unstructured data; Input data features that may serve as proxies for
Homogenization, Unknowns
demographic group membership (i.e., image metadata, language dialect) orotherwise give rise to emergent bias within GAI systems; The extent to which thedigital divide may negatively impact representativeness in GAI system training
and TEVV data; Filtering of hate speech and toxicity in GAI system training data;
Prevalence of GAI-generated data in GAI system training data.

47

MS-2.11-008

Track and document AI actor credentials and qualifications.

Human AI Configuration

MS-2.11-009

Verify accessibility functionality; verify functionality and timeliness ofaccommodations and opt-out functionality or processes.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MS-2.11-010

Verify bias management in periodic model updates; test and recalibrate with
updated and more representative data to manage bias within acceptable
tolerances.

Toxicity, Bias, and Homogenization

MS-2.11-011

Verify training is not homogenous GAI-produced data in order to mitigate
concerns of model collapse.

Toxicity, Bias, and Homogenization

AI Actors: AI Deployment, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, End-Users, Operationand Monitoring, TEVV

1
MEASURE 2.12: Environmental impact and sustainability of AI model training and management activities – as identified in the MAP
function – are assessed and documented.
Action ID

ActionRisks

MS-2.12-001

Assess safety to physical environments when deploying GAI systems.

Dangerous or Violent
Recommendations

MS-2.12-002

Document anticipated environmental impacts of model development,
maintenance, and deployment in product design decisions.

Environmental

MS-2.12-003

Measure or estimate environmental impacts (e.g., energy and water
consumption) for training, fine tuning, and deploying models: Verify tradeoffs
between resources used at inference time versus additional resources required
at training time.

Environmental

MS-2.12-004

Track and document continuous improvement processes that enhance
effectiveness of risk measurement for GAI environmental impacts andsustainability.

Environmental

MS-2.12-005

Verify effectiveness of carbon capture or offset programs, and address greenwashing risks.

Environmental

AI Actors: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV

248

MEASURE 2.13: Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated anddocumented.
Action ID

ActionRisks

MS-2.13-001

Create measurement error models for pre-deployment metrics to demonstrate
construct validity for each metric (i.e., does the metric effectively operationalize
the desired concept): Measure or estimate, and document, biases or statistical Confabulation, Information
variance in applied metrics or structured human feedback processes; Adhere to Integrity, Toxicity, Bias, andHomogenization
applicable laws and regulations when operationalizing models in high-volume
settings (e.g., toxicity classifiers and automated content filters); Leverage domain
expertise when modeling complex societal constructs such as toxicity.

MS-2.13-002

Document measurement and structured public feedback processes applied toorganizational GAI systems in a centralized repository (i.e., organizational AI
inventory).

MS-2.13-003

Review GAI system metrics and associated pre-deployment processes todetermine their ability to sustain system improvements, including theidentification and removal of errors, harms, and negative impacts.

Confabulation, Information
Integrity, Dangerous or Violent
Recommendations

AI Actors: AI Deployment, Operation and Monitoring, TEVV

1
*MEASURE 3.1: Approaches, personnel, and documentation are in place to regularly identify and track existing, unanticipated, andemergent AI risks based on factors such as intended and actual performance in deployed contexts.
Action ID

ActionRisks

MS-3.1-001

Assess completeness of known use cases and expected performance of inputs,
such as third-party data or upstream AI systems, or the performance ofdownstream systems which use the outputs of the GAI system, directly or
indirectly, through engagement and outreach with AI Actors.

Human AI Configuration, Value
Chain and Component
Integration, Toxicity, Bias, andHomogenization

MS-3.1-002

Compare intended use and expected performance of GAI systems across all
relevant contexts.

49MS-3.1-003

Elicit and track feedback for previously unknown uses of the GAI systems.

AI Actors: AI Impact Assessment, Operation and Monitoring, TEVV

1
MEASURE 3.2: Risk tracking approaches are considered for settings where AI risks are difficult to assess using currently available
measurement techniques or where metrics are not yet available.
Action ID

ActionRisks

MS-3.2-001

Determine if available GAI system risk measurement approaches are applicableto the GAI system use contexts.

MS-3.2-002

Document the rate of occurrence and severity of GAI harms to the organizationand to external AI actors.

Human AI Configuration

MS-3.2-003

Establish processes for identifying emergent GAI system risks with external AI
actors.

Human AI Configuration,
Unknowns

MS-3.2-004

Identify measurement approaches for tracking GAI system risks if none exist.

AI Actors: AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV

2
*MEASURE 3.3: Feedback processes for end users and impacted communities to report problems and appeal system outcomes are
established and integrated into AI system evaluation metrics.
Action ID

ActionRisks

MS-3.3-001

Conduct impact assessments on how AI-generated content might affect different
Toxicity, Bias, and Homogenizationsocial, economic, and cultural groups.

MS-3.3-002

Conduct studies to understand how end users perceive and interact with GAIcontent related to content provenance within context of use. Assess whether the Human AI Configuration,
Information Integrity
content aligns with their expectations and how they may act upon theinformation presented.

MS-3.3-003

Design evaluation metrics that include parameters for content provenancequality, validity, reliability, authenticity or origin, and integrity of content.

Information Integrity

MS-3.3-004

Evaluate GAI system evaluation metrics based on feedback from relevant AI
actors.

Human AI Configuration50

MS-3.3-005

Evaluate potential biases and stereotypes that could emerge from the AIgenerated content using appropriate methodologies including computational
testing methods as well as evaluating structured feedback input.

MS-3.3-006

Implement continuous monitoring of AI-generated content and provenance after
system deployment for various types of drift. Verify GAI systems are adaptive and Information Integrity
able to iteratively improve models and algorithms over time.

MS-3.3-007

Integrate human evaluators to assess content quality and relevance.

MS-3.3-008

Provide input for training materials about the capabilities and limitations of GAI
Human AI Configuration,systems related to content provenance for AI actors, other professionals, and theInformation Integrity, Toxicity,
public about the societal impacts of AI and the role of diverse and inclusive
Bias, and Homogenization
content generation.

MS-3.3-009

Record and integrate structured feedback about content provenance fromoperators, users, and potentially impacted communities through the use of
Human AI Configuration,
methods such as user research studies, focus groups, or community forums.
Information Integrity, Toxicity,
Actively seek feedback on generated content quality and potential biases. Assess
Bias, and Homogenization
the general awareness among end users and impacted communities about theavailability of these feedback channels.

MS-3.3-010

Regularly review structured human feedback and GAI system sensors and updatebased on the evolving needs and concerns of the impacted communities.

MS-3.3-011

Utilize independent evaluations to assess content quality and types of potential
biases and related negative impacts.

MS-3.3-012

Human AI Configuration,
Verify AI actors engaged in GAI TEVV tasks for content provenance reflect diverse
Information Integrity, Toxicity,
demographic and interdisciplinary backgrounds.
Bias, and Homogenization

Toxicity, Bias, and Homogenization

Human AI Configuration

Toxicity, Bias, and Homogenization

AI Actors: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV

1
MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle areinformed by input from domain experts and relevant AI actors to validate whether the system is performing consistently as
intended. Results are documented.
Action ID

ActionRisks

MS-4.2-001

Conduct adversarial testing to assess the GAI system’s response to inputs
intended to deceive or manipulate its content provenance and understandpotential misuse scenarios and unintended outputs.

Information Integrity, Information
Security51

MS-4.2-002

Ensure both positive and negative feedback on GAI system functionality isassessed.

MS-4.2-003

Ensure visible mechanisms to collect users’ feedback are in place, including
systems to report harmful and low quality content.

Human AI Configuration,
Dangerous or Violent
Recommendations

MS-4.2-004

Evaluate GAI system content provenance in real-world scenarios to observe its
behavior in practical environments and reveal issues that might not surface incontrolled and optimized testing environments.

Information Integrity

MS-4.2-005

Evaluate GAI system performance related to content provenance against
predefined metrics and update the evaluation criteria as necessary to adapt tochanging contexts and requirements.

Information Integrity

MS-4.2-006

Implement interpretability and explainability methods to evaluate GAI systemdecisions related to content provenance and verify alignment with intended
purpose.

Information Integrity, Toxicity,
Bias, and Homogenization

MS-4.2-007

Integrate structured human feedback results into calibration and updateprocesses for traditional measurement approaches (e.g., benchmarks,
performance assessments, data quality measurements).

MS-4.2-008

Measure GAI system inputs and outputs to account for content provenance, data
Information Integrity, Information
provenance, source reliability, contextual relevance and coherence, and security
Security
implications.

MS-4.2-009

Monitor and document instances where human operators or other systems
override the GAI's decisions. Evaluate these cases to understand if the overrides Information Integrity
are linked to issues related to content provenance.

MS-4.2-010

Verify and document the incorporation of structured human feedback results
into design, implementation, deployment approval (“go”/“no-go” decisions),
monitoring, and decommission decisions.

MS-4.2-011

Verify that GAI system development and deployment related to content
provenance integrates trustworthiness characteristics.

Information Integrity

MS-4.2-012

Verify the performance of user feedback and recourse mechanisms, including
analyses across various sub-groups.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MS-4.2-013

Work with domain experts to integrate insights from stakeholder feedback
analysis into TEVV metrics and associated actions, and continuous improvement
processes.

MS-4.2-014

Work with domain experts to review feedback from end users, operators, andpotentially impacted individuals and communities—enumerated in the Mapfunction.

52

Human AI Configuration

MS-4.2-015

Work with domain experts who understand the GAI system context of use toevaluate the content’s validity, relevance, and potential biases.

Toxicity, Bias, and Homogenization

AI Actors: AI Deployment, Domain Experts, End-Users, Operation and Monitoring, TEVV

1
*MANAGE 1.3: Responses to the AI risks deemed high priority, as identified by the MAP function, are developed, planned, anddocumented. Risk response options can include mitigating, transferring, avoiding, or accepting.
Action ID

ActionRisks

MG-1.3-001

Allocate resources and time for GAI risk management activities, including
planning for incident response and other mitigation activities.

MG-1.3-002

Document residual GAI system risks that persist after risk mitigation or transfer.

MG-1.3-003

Document trade-offs, decision processes, and relevant measurement andfeedback results for risks that do not surpass organizational risk tolerance.

MG-1.3-004

Mitigate, transfer, or avoid risks that surpass organizational risk tolerances.

MG-1.3-005

Monitor the effectiveness of risk controls (e.g., via field testing, participatory
engagements, performance assessments, user feedback mechanisms).

Human AI Configuration

AI Actors: AI Deployment, AI Impact Assessment, Operation and Monitoring

2
MANAGE 2.2: Mechanisms are in place and applied to sustain the value of deployed AI systems.
Action ID

ActionRisks

MG-2.2-001

Compare GAI system outputs against pre-defined organization risk tolerance,
guidelines, and principles, and review and audit AI-generated content against
these guidelines.

MG-2.2-002

Document training data sources to trace the origin and provenance of AIgenerated content.

Information Integrity

MG-2.2-003

Evaluate feedback loops between GAI system content provenance and humanreviewers, and update make updates where needed. Implement real-timemonitoring systems to detect GAI systems and content provenance drift as it
happens.

Information Integrity53

Evaluate GAI content and data for representational biases and employ
techniques such as re-sampling, re-ranking, or adversarial training to mitigate
biases in the generated content.

Information Security, Toxicity,
Bias, and Homogenization

MG-2.2-005

Filter GAI output for harmful or biased content, potential misinformation, and
CBRN-related or NCII content.

CBRN Information, Obscene,
Degrading, and/or Abusive
Content, Toxicity, Bias, andHomogenization, Dangerous orViolent Recommendations

MG-2.2-006

Implement version control for models and datasets to track changes andfacilitate rollback if necessary.

MG-2.2-007

Incorporate feedback from users, external experts, and the public to adapt the
GAI system and monitoring processes.

Human AI Configuration

MG-2.2-008

Incorporate human review processes to assess and filter content in accordancewith the socio-cultural knowledge and values of the context of use and to
identify limitations and nuances that automated processes might miss; verify
that human reviewers are trained on content guidelines and potential biases of
GAI system and its content provenance.

Information Integrity, Toxicity,
Bias, and Homogenization

MG-2.2-009

Integrate information from data management and machine learning security
countermeasures like red teaming, and differential privacy, and authentication
protocols to ensure data and models are protected from potential risks.

CBRN Information, Data Privacy,
Information Security

MG-2.2-010

Use feedback from internal and external AI actors, users, individuals, andcommunities, to assess impact of AI-generated content.

Human AI Configuration

MG-2.2-011

Use real-time auditing tools such as distributed ledger technology to track andvalidate the lineage and authenticity of AI-generated data.

Information Integrity

MG-2.2-012

Use structured feedback mechanisms to solicit and capture user input about AIHuman AI Configuration, Toxicity,
generated content to detect subtle shifts in quality or alignment with community
Bias, and Homogenizationand societal values.

MG-2.2-004

AI Actors: AI Deployment, AI Impact Assessment, Governance and Oversight, Operation and Monitoring

154

*MANAGE 2.3: Procedures are followed to respond to and recover from a previously unknown risk when it is identified.
Action ID

ActionRisks

MG-2.3-001

Develop and update GAI system incident response and recovery plans andprocedures to address the following: Review and maintenance of policies andprocedures to account for newly encountered uses; Review and maintenance ofpolicies and procedures for detection of unanticipated uses; Verify response
Value Chain and Component
Integration
and recovery plans account for the GAI system supply chain; Verify response
and recovery plans are updated for and include necessary details tocommunicate with downstream GAI system Actors: Points-of-Contact (POC),
Contact information, notification format.

MG-2.3-002

Maintain protocols to log changes made to GAI systems during incident
response and recovery.

MG-2.3-003

Review, update and maintain incident response and recovery plans to integrate
Human AI Configuration
insights from GAI system use cases and contexts and needs of relevant AI
actors.

MG-2.3-004

Verify and maintain measurements that GAI systems are operating withinorganizational risk tolerances post incident.

AI Actors: AI Deployment, Operation and Monitoring

1*MANAGE 2.4: Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, ordeactivate AI systems that demonstrate performance or outcomes inconsistent with intended use.
Action ID

ActionRisks

MG-2.4-001

Enforce change management processes, and risk and impact assessments
across all intended uses and contexts before deploying GAI system updates.

MG-2.4-002

Establish and maintain communication plans to inform AI stakeholders as partof the deactivation or disengagement process of a specific GAI system or
Human AI Configuration
context of use, including reasons, workarounds, user access removal, alternative
processes, contact information, etc.

MG-2.4-003

Establish and maintain procedures for escalating GAI system incidents to theorganizational risk authority when specific criteria for deactivation ordisengagement is met for a particular context of use or for the GAI system as awhole.

55

MG-2.4-004

Establish and maintain procedures for the remediation of issues which trigger
incident response processes for the use of a GAI system, and provide
stakeholders timelines associated with the remediation plan.

MG-2.4-005

Establish and regularly review specific criteria that warrants the deactivation of
GAI systems in accordance with set risk tolerances and appetites.

AI Actors: AI Deployment, Governance and Oversight, Operation and Monitoring

1*MANAGE 3.1: AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied anddocumented.
Action ID

ActionRisks

MG-3.1-001

Apply organizational risk tolerances and controls (e.g., acquisition andprocurement processes; assessing personnel credentials and qualifications,performing background checks; filtering GAI input and outputs, grounding,fine tuning) to third-party GAI resources: Apply organizational risk tolerance
to the utilization of third-party datasets and other GAI resources; Applyorganizational risk tolerances to fine-tuned third-party models; Applyorganizational risk tolerance to existing third-party models adapted to a new
domain; Reassess risk measurements after fine-tuning third-party GAImodels.

Value Chain and Component
Integration

MG-3.1-002

Audit GAI system supply chain risks (e.g., data poisoning, malware, othersoftware and hardware vulnerabilities; labor practices; data privacy andlocalization compliance; geopolitical alignment).

Data Privacy, Information Security,
Value Chain and Component
Integration, Toxicity, Bias, andHomogenization

MG-3.1-003

Decommission third-party systems that exceed organizational risk tolerances.

Value Chain and Component
Integration

MG-3.1-004

Identify and maintain documentation for third-party AI systems, andcomponents, in organizational AI inventories.

Value Chain and Component
Integration

MG-3.1-005

Value Chain and Component
Initiate review of third-party organizations/developers prior to their use ofIntegration, Toxicity, Bias, and
GAI models, and during their use of GAI models for their own applications, toHomogenization, Dangerous ormonitor for abuse and policy violations.
Violent Recommendations

MG-3.1-006

Re-assess model risks after fine-tuning and for any third-party GAI models
deployed for applications and/or use cases that were not evaluated in initial
testing.

56

Value Chain and Component
Integration

MG-3.1-007

Review GAI training data for CBRN information and intellectual property; scanIntellectual Property, CBRN
output for plagiarized, trademarked, patented, licensed, or trade secretInformationmaterial.

MG-3.1-008

Update acquisition and procurement policies, procedures, and processes toaddress GAI risks and failure modes.

MG-3.1-009

Use, review, update, and share various transparency artifacts (e.g., system
cards and model cards) for third-party models. Document or retain
documentation for: Training data content and provenance, methodology,
testing, validation, and clear instructions for use from GAI vendors andsuppliers, Information related to third-party information security policies,
procedures, and processes.

Information Integrity, Information
Security, Value Chain andComponent Integration

AI Actors: AI Deployment, Operation and Monitoring, Third-party entities

1
MANAGE 3.2: Pre-trained models which are used for development are monitored as part of AI system regular monitoring andmaintenance.
Action ID

ActionRisks

MG-3.2-001

Apply explainable AI (XAI) techniques (e.g., analysis of embeddings, modelcompression/distillation, gradient-based attributions, occlusion/term reduction,
counterfactual prompts, word clouds) as part of ongoing continuous
improvement processes to mitigate risks related to unexplainable GAI systems.

MG-3.2-002

Document how pre-trained models have been adapted (fine-tuned) for the
specific generative task, including any data augmentations, parameter
adjustments, or other modifications. Access to un-tuned (baseline) models must
be available to support debugging the relative influence of the pre-trained
weights compared to the fine-tuned model weights.

MG-3.2-003

Document sources and types of training data and their origins, potential biases
present in the data related to the GAI application and its content provenance,
Information Integrity, Toxicity, Bias,architecture, training process of the pre-trained model including information on and Homogenization
hyperparameters, training duration, and any fine-tuning processes applied.

MG-3.2-004

Human AI Configuration,
Evaluate user reported problematic content and integrate feedback into system
Dangerous or Violent
updates.
Recommendations

57

Information Integrity, Toxicity, Bias,and Homogenization, Dangerous orViolent Recommendations,Obscene, Degrading, and/or
Abusive Content

MG-3.2-005

Implement content filters to prevent the generation of inappropriate, harmful,toxic, false, illegal, or violent content related to the GAI application, including
for CSAM and NCII. These filters can be rule-based or leverage additional
machine learning models to flag problematic inputs and outputs.

MG-3.2-006

Implement real-time monitoring processes for analyzing generated content
performance and trustworthiness characteristics related to content provenance
Information Integrity
to identify deviations from the desired standards and trigger alerts for humanintervention.

MG-3.2-007

Leverage feedback and recommendations from organizational boards orcommittees related to the deployment of GAI applications and content
provenance when using third-party pre-trained models.

Information Integrity, Value Chain
and Component Integration

MG-3.2-008

Maintain awareness of relevant laws and regulations related to content
generation, data privacy, and user protections and work in conjunction withlegal experts to review and assess the potential liabilities associated with AIgenerated content.

Data Privacy, Intellectual Property
Information Integrity

MG-3.2-009

Provide use case examples as material for training employees and stakeholders
about the trustworthiness implications of GAI applications and content
Information Integrity
provenance and to raise awareness about potential risks in fostering a risk
management culture.

MG-3.2-010

Use human moderation systems to review generated content in accordancewith human-AI configuration policies established in the Govern function,aligned with socio-cultural norms in the context of use, and for settings where
AI models are demonstrated to perform poorly.

Human AI Configuration

MG-3.2-011

Use organizational risk tolerance to evaluate acceptable risks and performance
metrics and decommission or retrain pre-trained models that perform outsideof defined limits.

CBRN Information, Confabulation

AI Actors: AI Deployment, Operation and Monitoring, Third-party entities

158

*MANAGE 4.1: Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating
input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change
management.
Action ID

ActionRisks

MG-4.1-001

Collaborate with external researchers, industry experts, and community
representatives to maintain awareness of emerging best practices andtechnologies in content provenance.

Information Integrity, Toxicity, Bias,and Homogenization

MG-4.1-002

Conduct adversarial testing at a regular cadence; test against various
adversarial inputs and scenarios; identify vulnerabilities and assess the AI
system’s resilience to content provenance attacks.

Information Integrity, Information
Security

MG-4.1-003

Conduct red-teaming exercises to surface failure modes of content provenance
Information Integrity, Information
mechanisms. Evaluate the effectiveness of red-teaming approaches for
Security
uncovering potential vulnerabilities and improving overall content provenance.

MG-4.1-004

Employ user-friendly channels such as feedback forms, e-mails, or hotlines forusers to report issues, concerns, or unexpected GAI outputs to feed into
monitoring practices.

MG-4.1-005

Establish, maintain, and evaluate effectiveness of organizational processes andprocedures to monitor GAI systems within context of use.

MG-4.1-006

Evaluate the use of sentiment analysis to gauge user sentiment regarding GAIcontent performance and impact, and work in collaboration with AI actors
experienced in user research and experience.

Human AI Configuration

MG-4.1-007

Implement active learning techniques to identify instances where the model
fails or produces unexpected outputs.

Confabulation

MG-4.1-008

Integrate digital watermarks, blockchain technology, cryptographic hashfunctions, metadata embedding, or other content provenance techniques
within AI-generated content to track its source and manipulation history.

Information Integrity

MG-4.1-009

Measure system outputs related to content provenance at a regular cadence
and integrate insights into monitoring processes.

Information Integrity

MG-4.1-010

Monitor GAI training data for representation of different user groups.

Human AI Configuration, Toxicity,
Bias, and Homogenization

MG-4.1-011

Perform periodic review of organizational adherence to GAI system monitoringplans across all contexts of use.

59

Human AI Configuration

MG-4.1-012

Share transparency reports with internal and external stakeholders that detailsteps taken to update the AI system to enhance transparency andaccountability.

MG-4.1-013

Track dataset modifications for content provenance by monitoring datadeletions, rectification requests, and other changes that may impact theverifiability of content origins.

MG-4.1-014

Verify risks associated with gaps in GAI system monitoring plans are accepted at
the appropriate organizational level.

MG-4.1-015

Verify that AI actors responsible for monitoring reported issues can effectively
evaluate GAI system performance and its content provenance, and promptly
escalate issues for response.

Information Integrity

Human AI Configuration,
Information Integrity

AI Actors: AI Deployment, Affected Individuals and Communities, Domain Experts, End-Users, Human Factors, Operation andMonitoring

1
MANAGE 4.2: Measurable activities for continual improvements are integrated into AI system updates and include regular
engagement with interested parties, including relevant AI actors.
Action ID

ActionRisks

MG-4.2-001

Adopt agile development methodologies, and iterative development andfeedback loops to allow for rapid adjustments based on external input related to Information Integrity
content provenance.

MG-4.2-002

Conduct regular audits of GAI systems and publish reports detailing theperformance, feedback received, and improvements made.

MG-4.2-003

Employ explainable AI methods to enhance transparency and interpretability of
Human AI Configuration,
GAI content provenance to help AI actors and stakeholders understand how andInformation Integrity
why specific content is generated.

MG-4.2-004

Employ stakeholder feedback captured in the Map function to understand user
experiences and perceptions about AI-generated content and its provenance;
include user interactions and feedback from real-world scenarios.

Human AI Configuration,
Information Integrity

MG-4.2-005

Form cross-functional teams leveraging expertise from across the AI lifecycle
including AI designers and developers, socio-technical experts, and experts inthe context of use and identify mechanisms to include end users in
consultations.

Human AI Configuration60

MG-4.2-006

Practice and follow incident response plans for addressing the generation of
inappropriate or harmful content and adapt processes based on findings toprevent future occurrences. Conduct post-mortem analyses of incidents with
relevant AI actors, to understand the root causes and implement preventive
measures.

MG-4.2-007

Provide external stakeholders with regular updates about the progress,
challenges, and improvements made based on their feedback through the use ofIntellectual Property
public venues such as online platforms and communities, and open-source
initiatives.

MG-4.2-008

Simulate various scenarios to test GAI system responses and verify intended
performance across different situations.

MG-4.2-009

Use visualizations to represent the GAI model behavior to ease non-technicalstakeholders understanding of GAI system functionality.

Human AI Configuration,
Dangerous or Violent
Recommendations

Human-AI Configuration

AI Actors: AI Deployment, AI Design, AI Development, Affected Individuals and Communities, End-Users, Operation andMonitoring, TEVV

1
*MANAGE 4.3: Incidents and errors are communicated to relevant AI actors, including affected communities. Processes for tracking,responding to, and recovering from incidents and errors are followed and documented.
Action ID

ActionRisks

MG-4.3-001

Conduct after-action assessments for GAI system incidents to verify incident
response and recovery processes are followed and effective.

MG-4.3-002

Establish and maintain change management records and procedures for GAI
systems, including the reasons for each change, how the change could impact
each intended context of use, and step-by-step details of how changes wereplanned, tested, and deployed.

MG-4.3-003

Establish and maintain policies and procedures to record and track GAI system
reported errors, near-misses, incidents, and negative impacts.

Confabulation, Information
Integrity

MG-4.3-004

Establish processes and procedures for regular sharing of information about
errors, incidents, and negative impacts for each and across contexts, sectors,
and AI actors, including the date reported, the context of use, the number ofreports for each issue, and assessments of impact and severity.

Confabulation, Human AI
Configuration, Information
Integrity

AI Actors: AI Deployment, Affected Individuals and Communities, Domain Experts, End-Users, Human Factors, Operation andMonitoring

2

61

1

Appendix A. Primary GAI Considerations

2
3
4
5
6

The following primary considerations were derived as overarching themes from the GAI PWG
consultation process. These considerations (Governance, Pre-Deployment Testing, Content Provenance,and Incident Disclosure) are relevant to any organization designing, developing, and using GAI and also
inform the Actions to Manage GAI risks. Information included about the primary considerations is not
exhaustive, but highlights the most relevant topics derived from the GAI PWG.

7
8
9

Acknowledgments: These considerations could not have been surfaced without the helpful analysis andcontributions from the community and NIST staff GAI PWG leads: George Awad, Luca Belli, Mat Heyman,
Yooyoung Lee, Reva Schwartz, and Kyra Yee.

10

Governance11

A.1.1. Overview

12
13
14
15
16
17

Like any other technology system, governance principles and techniques can be used to manage risks
related to generative AI models, capabilities, and applications. Organizations may choose to apply their
existing risk tiering to GAI systems, or they may opt to revise or update AI system risk levels to addressthese unique GAI risks. This section describes how organizational governance regimes may be reevaluated and adjusted for GAI contexts. It also addresses third-party considerations for governing acrossthe AI value chain.

18

A.1.2. Organizational Governance

19
20
21
22
23

GAI opportunities, risks and long-term performance characteristics are typically less well-understoodthan non-generative AI tools. and may be perceived and acted upon by humans in ways that vary greatly.
Accordingly, GAI may call for different levels of oversight from AI actors or different human-AI
configurations in order to manage their risks effectively. Organizations’ use of GAI systems may also
warrant additional human review, tracking and documentation, and greater management oversight.

24
25
26
27
28
29

AI technology can produce varied outputs in multiple modalities and present many classes of userinterfaces. This leads to a broader set of AI actors interacting with GAI systems for widely differing
applications and contexts of use. These can include data labeling and preparation, development of GAImodels, content moderation, code generation and review, text generation and editing, image and video
generation, summarization, search, and chat. These activities can take place within organizationalsettings or in the public domain.

30
31
32

Organizations can restrict AI applications that cause harm, exceed stated risk tolerances, or that conflictwith their tolerances or values. Governance tools and protocols that are applied to other types of AI
systems can be applied to GAI systems. These plans and actions include:

33
34
35
36

• Accessibility and reasonableaccommodations• AI actor credentials and qualifications• Alignment to organizational values

37
38
39
40
62

•
•
•
•

Auditing and assessment
Change-management controls
Commercial use
Data provenance

41
42
43
44
45
46
47
48
49
50

•
•
•
•
•
•
•
•
•
•

51
52
53
54
55
56
57
58
59
60

Data protection
Data retention
Consistency in use of defining key terms
Decommissioning
Discouraging anonymous use
Education
Impact assessments
Incident response
Monitoring
Opt-outs

•
•
•
•
•
•

Risk-based controls
Risk mapping and measurement
Science-backed TEVV practices
Secure software development practices
Stakeholder engagement
Synthetic content detection andlabeling tools and techniques• Whistleblower protections• Workforce diversity andinterdisciplinary teams

61
62
63
64
65

Establishing acceptable use policies and guidance for the use of GAI in formal human-AI teaming settings
as well as different levels of human-AI configurations can help to decrease risks arising from misuse,abuse, inappropriate repurpose, and misalignment between systems and users. These practices are justone example of adapting existing governance protocols for GAI contexts.

66

A.1.3. Third-Party Considerations

67
68
69
70
71
72

Organizations may seek to acquire, embed, incorporate, or use open source or proprietary third-party
GAI models, systems, or generated data for various applications across an enterprise. Use of these GAI
tools and inputs has implications for all functions of the organization – including but not limited toacquisition, human resources, legal, compliance, and IT services – regardless of whether they are carriedout by employees or third parties. Many of the actions cited above are relevant and options foraddressing third-party considerations.

73
74
75
76
77
78
79
80
81
82

Third party GAI integrations may give rise to increased intellectual property, data privacy, or information
security risks, pointing to the need for clear guidelines for transparency and risk management regarding
the collection and use of third-party data for model inputs. Organizations may consider varying risk
controls for foundation models, fine-tuned models, and embedded tools, enhanced processes forinteracting with external GAI technologies or service providers. Organizations can apply standard or
existing risk controls and processes to proprietary or open-source GAI technologies, data, and third-party
service providers, including acquisition and procurement due diligence, requests for software bills ofmaterials (SBOMs), application of service level agreements (SLAs), and statement on standards forattestation engagement (SSAE) reports to help with third-party transparency and risk management for
GAI systems.

83

A.1.4. Pre-Deployment Testing

84

Appendix B. Overview

85
86
87

The diverse ways and contexts in which GAI systems may be developed, used, and repurposedcomplicates risk mapping and pre-deployment measurement efforts. Robust test, evaluation, validation,and verification (TEVV) processes can be iteratively applied – and documented – in early stages of the AI
63

88
89
90
91
92

lifecycle and informed by representative AI actors (see Figure 3 of the AI RMF). Until new and rigorous
early lifecycle TEVV approaches are developed and matured for GAI, organizations may use
recommended “pre-deployment testing” practices to measure performance, capabilities, limits, risks,
and impacts. This section describes risk measurement and estimation as part of pre-deployment TEVV,and examines the state of play for pre-deployment testing methodologies.

93

Appendix C. Limitations of Current Pre-deployment Test Approaches

94
95
96
97
98
99

Currently available pre-deployment TEVV processes used for GAI applications may be inadequate, nonsystematically applied, or fail to reflect or mismatched to deployment contexts. For example, theanecdotal testing of GAI system capabilities through video games or standardized tests designed forhumans (e.g., intelligence tests, professional licensing exams) does not guarantee GAI system validity or
reliability in those domains. Similarly, jailbreaking or prompt-engineering tests may not systematicallyassess validity or reliability risks.

100
101
102
103
104
105
106

Measurement gaps can arise from mismatches between laboratory and real-world settings. Current
testing approaches often remain focused on laboratory conditions or restricted to benchmark test
datasets and in silico techniques that may not extrapolate well to—or directly assess GAI impacts in –real world conditions. For example, current measurement gaps for GAI make it difficult to precisely
estimate its potential ecosystem-level or longitudinal risks and related political, social, and economic
impacts. Gaps between benchmarks and real-world use of GAI systems may likely be exacerbated due toprompt sensitivity and broad heterogeneity of contexts of use.

107

A.1.5. Structured Public Feedback

108
109
110

Structured public feedback can be used to evaluate whether GAI systems are performing as intended
and to calibrate and verify traditional measurement methods. Examples of structured feedback include,
but are not limited to:

111
112

•

Participatory Engagement Methods: Methods used to solicit feedback from civil society groups,
affected communities, and users, including focus groups, small user studies, and surveys.

113
114
115

•

Field Testing: Methods used to determine how people interact with, consume, use, and makesense of AI-generated information, and subsequent actions and effects, including UX, usability,
and other structured, randomized experiments.

116
117
118

•

AI Red-teaming: A structured testing exercise used to probe an AI system to find flaws andvulnerabilities such as inaccurate, harmful, or discriminatory outputs, often in a controlledenvironment and in collaboration with system developers.

119
120
121
122
123
124

Information gathered from structured public feedback can inform design, implementation, deployment
approval, maintenance, or decommissioning decisions. Results and insights gleaned from these exercises
can serve multiple purposes, including improving data quality and preprocessing, bolstering governancedecision making, and enhancing system documentation and debugging practices. When implementing
feedback activities, organizations should follow human subjects research requirements and bestpractices such as informed consent and subject compensation.

64

125

C.1.1.1. Participatory Engagement Methods

126
127
128
129
130
131

On an ad hoc or more structured basis, organizations can design and use a variety of channels to engage
external stakeholders in product development or review. Focus groups with select experts can provide
feedback on a range of issues. Small user studies can provide feedback from representative groups or
populations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory
engagement methods are often less structured than field testing or red teaming, and are morecommonly used in early stages of AI or product development.

132

Appendix D. Field Testing

133
134
135
136
137

Field testing involves structured settings to evaluate risks and impacts and to simulate the conditions
under which the GAI system will be deployed. Field style tests can be adapted from a focus on user
preferences and experiences towards AI risks and impacts – both negative and positive. When carriedout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts
in real world interactions.

138
139
140
141
142

Organizations may also collect feedback on outcomes, harms, and user experience directly from users in
the production environment after a model has been released, in accordance with human subject
standards such as informed consent and compensation. Organizations should follow applicable humansubjects research requirements, and best practices such as informed consent and subject compensation,
when implementing feedback activities.

143

Appendix E. AI Red-teaming

144
145
146
147

AI red-teaming exercises are often conducted in a controlled environment and in collaboration with AI
developers building AI models. AI red-teaming can be performed before or after AI models or systems
are made available to the broader public; this section focuses on red-teaming in pre-deployment
contexts.

148
149
150
151
152
153

The quality of AI red-teaming outputs is related to the background and expertise of the AI red-team
itself. Demographically and interdisciplinarily diverse AI red-teams can be used to identify flaws in thevarying contexts where GAI will be used. For best results, AI red-teams should demonstrate domain
expertise, and awareness of socio-cultural aspects within the deployment context. AI red-teaming results
should be given additional analysis before they are incorporated into organizational governance anddecision making, policy and procedural updates, and AI risk management efforts.

154

Various types of AI red-teaming may be appropriate, depending on the use case:

155
156
157
158
159

•

General Public: Performed by general users (not necessarily AI or technical experts) who are
expected to use the model or interact with its outputs, and who bring their own lived
experiences and perspectives to the task of AI red-teaming. These individuals may have been
provided instructions and material to complete tasks which may elicit harmful model behaviors.
This type of exercise can be more effective with large groups of AI-teamers.

160
161

•

Expert: Performed by specialists with expertise in the domain or specific AI red-teaming context
of use (e.g., medicine, biotech, cybersecurity).

65

162
163
164
165
166

•

Combination: In scenarios when it is difficult to identify and recruit specialists with sufficientdomain and contextual expertise, AI red-teaming exercises may leverage both expert andgeneral public participants. For example, expert AI red-teamers could modify or verify theprompts written by general public AI red-teamers. These approaches may also expand coverageof the AI risk attack surface.

167
168
169

•

Human / AI: Performed by GAI in combination with specialist or non-specialist human teams.
GAI-led red-teaming can be more cost effective than human red teamers alone. Human or GAIled AI red-teaming may be better suited for eliciting different types of harms.

170

A.1.6. Content Provenance171

Appendix F. Overview

172
173
174
175
176
177
178
179
180
181

GAI technologies can be leveraged for many applications such as content generation and synthetic data.
Some aspects of GAI output, such as the production of deepfake content, can challenge our ability to
distinguish human-generated content from AI-generated content. To help manage and mitigate theserisks, digital transparency mechanisms like provenance data tracking can trace the origin and history ofcontent. Provenance data tracking and synthetic content detection can help provide greater information
about both authentic and synthetic content to users, enabling trustworthiness in AI systems. When
combined with other organizational accountability mechanisms, digital content transparency can enable
processes to trace negative outcomes back to their source, improve information integrity, and uphold
public trust. Provenance data tracking and synthetic content detection mechanisms provide information
about the origin of content and its history to assist in GAI risk management efforts.

182
183
184
185
186
187

Provenance data can include information about generated content’s creators, date/time of creation,
location, modifications, and sources, including metadata information. Metadata can be tracked for text,
images, videos, audio, and underlying datasets. Provenance data tracking employs various methods andmetrics to assess the authenticity, integrity, credibility, intellectual property rights, and potential
manipulations in GAI output. Some well-known techniques for provenance data tracking include
watermarking, metadata tracking, digital fingerprinting, and human authentication, among others.

188

Appendix G. Provenance Data Tracking Approaches

189
190
191
192
193
194
195
196
197
198
199

Provenance data tracking techniques for GAI systems can be used to track the lineage and integrity ofdata inputs, metadata, and AI-generated content. Provenance data tracking records the origin andhistory for digital content, allowing its authenticity to be determined. It consists of techniques to record
metadata as well as perceptible and imperceptible digital watermarks on digital content. Data
provenance refers to tracking the origin and history of input data through metadata and digitalwatermarking techniques. Provenance data tracking processes can include and assist AI actors across the
lifecycle who may not have full visibility or control over the various trade-offs and cascading impacts ofearly-stage model decisions on downstream performance and synthetic outputs. For example, byselecting a given model to prioritize computational efficiency over accuracy, an AI actor may
inadvertently affect provenance tracking reliability. Organizational risk management efforts forenhancing content provenance include:
66

200

•

Tracking provenance of training data and metadata for GAI systems;

201•

Documenting provenance data limitations within GAI systems;

202•

Monitoring system capabilities and limitations in deployment through rigorous TEVV processes;

203
204
205

•

Evaluating how humans engage, interact with, or adapt to GAI content (especially in decision
making tasks informed by GAI content), and how they react to applied provenance techniquessuch as perceptible disclosures.

206
207
208
209
210

Organizations can document and delineate GAI system objectives and limitations to identify gaps where
provenance data may be most useful. For instance, GAI systems used for content creation may requirewatermarking techniques to identify the source of content or metadata management to trace content
origins and modifications. Further narrowing of GAI task definitions to include provenance data canenable organizations to maximize the utility of provenance data and risk management efforts.

211

A.1.7. Enhancing Content Provenance through Structured Public Feedback

212
213
214
215

While indirect feedback methods such as automated error collection systems are useful, they often lackthe context and depth that direct input from end users can provide. Organizations can leverage feedback
approaches described in the Pre-Deployment Testing section to capture input from external sources suchas through AI red-teaming.

216
217
218
219
220
221
222
223

Integrating pre- and post-deployment external feedback into the monitoring process of applications
involving AI-generated content can help enhance awareness of performance changes and mitigate
potential risks and harms. There are many ways to capture and make use of user feedback – before andafter GAI systems are deployed – to gain insights about authentication efficacy and vulnerabilities,
impacts of adversarial threats, unintended consequences resulting from the utilization of content
provenance approaches, and other unanticipated behavior associated with content manipulation.
Organizations can track and document the provenance of datasets to identify instances in which AIgenerated data is a potential root cause of performance issues with the GAI system.

224

A.1.8. Incident Disclosure

225

Appendix H. Overview

226
227
228
229
230
231
232

AI incidents can be defined as an event, circumstance, or series of events in which the development, use,
or malfunction of one or more AI systems directly or indirectly contributes to identified harms. These
harms include injury or damage to the health of an individual or group of people; disruption of the
management and operation of critical infrastructure; violations of human rights or a breach ofobligations under applicable law intended to protect legal and labor rights; or damage to property,
communities, or the environment. AI incidents can occur in the aggregate (i.e., for systemic
discrimination) or acutely (i.e., for one individual).

67

233

Appendix I. State of AI Incident Tracking and Disclosure

234
235
236
237

Formal channels do not currently exist to report and document AI incidents. However, a number ofpublicly-available databases have been created to document their occurrence. These reporting channels
make decisions on an ad hoc basis about what kinds of incidents to track. Some, for example, track byamount of media coverage.

238
239
240
241

Documenting, reporting, and sharing information about GAI incidents can help mitigate and prevent
harmful outcomes by assisting relevant AI actors in tracing impacts to their source. Greater awarenessand standardization of GAI incident reporting could promote this transparency and improve GAI risk
management across the AI ecosystem.

242

Appendix J. Documentation and Involvement of AI Actors

243
244
245
246
247
248
249
250

AI actors should be aware of their roles in reporting AI incidents. To better understand previous incidents
and implement measures to prevent similar ones in the future, organizations could consider developing
guidelines for publicly available incident reporting which include information about AI actor
responsibilities. These guidelines would help AI system operators identify GAI incidents across the AI
lifecycle and with AI actors regardless of role. Documentation and review of third party inputs andplugins for GAI systems is especially important for AI actors in the context of incident disclosure; LLM
inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient
access control.

251
252
253
254

Documentation practices including logging, recording, and analyzing GAI incidents can facilitatesmoother sharing of information with relevant AI actors. Regular information sharing, change
management records, version history and metadata can also empower AI actors responding to andmanaging AI incidents.

68

255

Appendix K. References

256
257
258

AI Risks and Trustworthiness, NIST Trustworthy & Responsible AI Resource Center. National Institute ofStandards and Technology.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics.

259
260

AI RMF Playbook. National Institute of Standards and Technology.
https://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook.

261
262

AI RMF Profiles. National Institute of Standards and Technology.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile.

263

AI Incident Database. https://incidentdatabase.ai/.

264
265

AI Risk Management Framework. National Institute of Standards and Technology.
https://www.nist.gov/itl/ai-risk-management-framework.

266
267
268
269
270

AI Risk Management Framework. National Institute of Standards and Technology. Appendix A:
Descriptions of AI Actor Tasks, NIST Trustworthy & Responsible AI Resource Center. National Institute ofStandards and Technology.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_A#:~:text=AI%20actors%
20in%20this%20category,data%20providers%2C%20system%20funders%2C%20product.

271
272
273

AI Risk Management Framework. National Institute of Standards and Technology. Appendix B: How AI
Risks Differ from Traditional Software Risks. National Institute of Standards and Technology.
https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B.

274
275
276

Alba, D., (2023) How Fake AI Photo of a Pentagon Blast Went Viral and Briefly Spooked Stocks.
Bloomberg. https://www.bloomberg.com/news/articles/2023-05-22/fake-ai-photo-of-pentagon-blastgoes-viral-trips-stocks-briefly.

277
278

Atherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses.
AI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/.

279
280

Authenticating AI-Generated Content (2024). Information Technology Industry Council.
https://www.itic.org/policy/ITI_AIContentAuthorizationPolicy_122123.pdf.

281

Badyal, N. et al., (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611.

282
283

Bing Chat: Data Exfiltration Exploit Explained. Embrace The Red.
https://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/.

284
285

Bommasani, R. et al., (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to
Outcome Homogenization? arXiv. https://arxiv.org/pdf/2211.13972.

286
287

Boyarskaya, M. et al., (2020) Overcoming Failures of Imagination in AI Infused System Development and
Deployment. arXiv. https://arxiv.org/pdf/2011.13416.

288
289

Browne, D. et al., (2023) Securing the AI Pipeline. Mandiant.
https://www.mandiant.com/resources/blog/securing-ai-pipeline.

69

290
291
292

Building a Glossary for Synthetic Media Transparency Methods, Part 1: Indirect Disclosure (2023)
Partnership on AI. https://partnershiponai.org/glossary-for-synthetic-media-transparency-methods-part1-indirect-disclosure/.

293
294

Burgess, M., (2024) Generative AI’s Biggest Security Flaw Is Not Easy to Fix. WIRED.
https://www.wired.com/story/generative-ai-prompt-injection-hacking/.

295
296
297

Burtell, M. et al., (2024) The Surprising Power of Next Word Prediction: Large Language Models
Explained, Part 1. Georgetown CSET. https://cset.georgetown.edu/article/the-surprising-power-of-nextword-prediction-large-language-models-explained-part-1/.

298
299

Carlini, N., et al., (2021) Extracting Training Data from Large Language Models. Usenix.
https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting.

300
301

Carlini, N. et al., (2023) Quantifying Memorization Across Neural Language Models. ICLR 2023.
https://arxiv.org/pdf/2202.07646.

302
303

Carlini, N. et al., (2024) Stealing Part of a Production Language Model. arXiv.
https://arxiv.org/abs/2403.06634.

304
305
306

Chandra, B. et al., (2023) Dismantling the Disinformation Business of Chinese Influence Operations.
RAND. https://www.rand.org/pubs/commentary/2023/10/dismantling-the-disinformation-business-ofchinese.html.

307
308

Dahl, M. et al., (2024) Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. arXiv.
https://arxiv.org/abs/2401.01301.

309
310

De Angelo, D., (2024) Short, Mid and Long-Term Impacts of AI in Cybersecurity. Palo Alto Networks.
https://www.paloaltonetworks.com/blog/2024/02/impacts-of-ai-in-cybersecurity/.

311
312
313

De Freitas, J., et al. (2023) Chatbots and Mental Health: Insights into the Safety of Generative AI. Harvard
Business School. https://www.hbs.edu/ris/Publication%20Files/23-011_c1bdd417-f717-47b6-bccb5438c6e65c1a_f6fd9798-3c2d-4932-b222-056231fe69d7.pdf.

314
315
316

Dietvorst, B. et al., (2014) Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them
Err. Journal of Experimental Psychology. https://marketing.wharton.upenn.edu/wpcontent/uploads/2016/10/Dietvorst-Simmons-Massey-2014.pdf.

317
318

Duhigg, C., (2012) How Companies Learn Your Secrets. New York Times.
https://www.nytimes.com/2012/02/19/magazine/shopping-habits.html.

319
320
321

Elsayed, G. et al., (2024) Images altered to trick machine vision can influence humans too. Google
DeepMind. https://deepmind.google/discover/blog/images-altered-to-trick-machine-vision-caninfluence-humans-too/.

322
323

Epstein, Z. et al., (2023). Art and the science of generative AI. Science.
https://www.science.org/doi/10.1126/science.adh4451.

324
325
326
327

Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence(2023) The White House.https://www.whitehouse.gov/briefing-room/presidentialactions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-ofartificial-intelligence/.

70

328

Fair Information Practice Principles (FIPPs). FPC. https://www.fpc.gov/resources/fipps/.

329
330

Generative artificial intelligence (AI) - ITSAP.00.041. (2023) Canadian Centre for Cyber Security.
https://www.cyber.gc.ca/en/guidance/generative-artificial-intelligence-ai-itsap00041.

331

GPT-4 System Card (2023) OpenAI. https://cdn.openai.com/papers/gpt-4-system-card.pdf.

332

GPT-4 Technical Report (2024) OpenAI. https://arxiv.org/pdf/2303.08774.

333
334

Greshake, K. et al., (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated
Applications with Indirect Prompt Injection. arXiv. https://arxiv.org/abs/2302.12173.

335
336

Feffer, M. et al., (2024). Red-Teaming for Generative AI: Silver Bullet or Security Theater? arXiv.
https://arxiv.org/pdf/2401.15897.

337
338

Haran, R., (2023). Securing LLM Systems Against Prompt Injection. NVIDIA.
https://developer.nvidia.com/blog/securing-llm-systems-against-prompt-injection/.

339
340
341

Harwell, D., (2023) AI-generated child sex images spawn new nightmare for the web. Washington Post.
https://www.washingtonpost.com/technology/2023/06/19/artificial-intelligence-child-sex-abuseimages/.

342
343

Hubinger, E. et al, (2024) “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training”,
arXiv e-prints. https://arxiv.org/abs/2401.05566.

344
345

Jain, S. et al., (2023) Algorithmic Pluralism: A Structural Approach To Equal Opportunity. arXiv.
https://arxiv.org/pdf/2305.08157.

346
347

Ji, Z. et al (2023) Survey of Hallucination in Natural Language Generation. ACM Comput. Surv. 55, 12,
Article 248. https://doi.org/10.1145/3571730

348
349

Jussupow, E. et al., (2020) Why Are We Averse Towards Algorithms? A Comprehensive Literature Review
on Algorithm Aversion. ECIS 2020. https://aisel.aisnet.org/ecis2020_rp/168/.

350
351

Katzman, J., et al., (2023) Taxonomizing and measuring representational harms: a look at image tagging.
AAAI. https://dl.acm.org/doi/10.1609/aaai.v37i12.26670.

352
353

Kirchenbauer, J. et al., (2023) A Watermark for Large Language Models. OpenReview.
https://openreview.net/forum?id=aX8ig9X2a7.

354
355

Kleinberg, J. et al., (May 2021) Algorithmic monoculture and social welfare. PNAS.
https://www.pnas.org/doi/10.1073/pnas.2018340118.

356

Lakatos, S., (2023) A Revealing Picture. Graphika. https://graphika.com/reports/a-revealing-picture.

357
358

Lenaerts-Bergmans, B., (2024) Data Poisoning: The Exploitation of Generative AI. Crowdstrike.
https://www.crowdstrike.com/cybersecurity-101/cyberattacks/data-poisoning/.

359
360

Liang, W. et al., (2023) GPT detectors are biased against non-native English writers. arXiv.
https://arxiv.org/abs/2304.02819.

361
362

Luccioni, A. et al., (2023) Power Hungry Processing: Watts Driving the Cost of AI Deployment? arXiv.
https://arxiv.org/pdf/2311.16863.

71

363
364

Mouton, C. et al., (2024) The Operational Risks of AI in Large-Scale Biological Attacks. RAND.
https://www.rand.org/pubs/research_reports/RRA2977-2.html.

365
366

Nicoletti, L. et al., (2023) Humans Are Biased. Generative Ai Is Even Worse. Bloomberg.
https://www.bloomberg.com/graphics/2023-generative-ai-bias/.

367
368

Northcutt, C. et al., (2021) Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks.
arXiv. https://arxiv.org/pdf/2103.14749.

369
370
371

OECD (2023), "Advancing accountability in AI: Governing and managing risks throughout the lifecycle fortrustworthy AI", OECD Digital Economy Papers, No. 349, OECD Publishing, Paris,
https://doi.org/10.1787/2448f04b-en.

372OECD AI Incidents Monitor. OECD.AI Policy Observatory. https://oecd.ai/en/incidents-methodology.

373
374

Padmakumar, V. et al., (2024) Does writing with language models reduce content diversity? ICLR.
https://arxiv.org/pdf/2309.05196.

375
376

Paresh, D., (2023) ChatGPT Is Cutting Non-English Languages Out of the AI Revolution. WIRED.
https://www.wired.com/story/chatgpt-non-english-languages-ai-revolution/.

377
378

Qu, Y. et al., (2023) Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From TextTo-Image Models. arXiv. https://arxiv.org/pdf/2305.13873.

379
380

Rafat, K. et al., (2023) Mitigating carbon footprint for knowledge distillation based deep learning modelcompression. PLOS One. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0285668.

381
382
383

Roadmap for Researchers on Priorities Related to Information Integrity Research and Development(2022) The White House. https://www.whitehouse.gov/wp-content/uploads/2022/12/RoadmapInformation-Integrity-RD-2022.pdf?.

384
385

Sandbrink, J., (2023) Artificial intelligence and biological misuse: Differentiating risks of language models
and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952.

386
387

Satariano, A. et al., (2023) The People Onscreen Are Fake. The Disinformation Is Real. New York Times.
https://www.nytimes.com/2023/02/07/technology/artificial-intelligence-training-deepfake.html.

388
389

Schaul, K. et al., (2024) Inside the secret list of websites that make AI like ChatGPT sound smart.
Washington Post. https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/.

390
391

Shelby, R. et al., (2023) Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm
Reduction. arXiv. https://arxiv.org/pdf/2210.05791.

392

Shevlane, T. et al., (2023) Model evaluation for extreme risks. arXiv. https://arxiv.org/pdf/2305.15324.

393
394

Shumailov, I. et al., (2023) The curse of recursion: training on generated data makes models forget. arXiv.
https://arxiv.org/pdf/2305.17493v2.

395
396

Skaug Sætra, H. et al., (2022). Psychological interference, liberty and technology. Technology in Society.
https://www.sciencedirect.com/science/article/pii/S0160791X22001142.

397
398
399

Smith, A. et al., (2023) Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language
Models. PLOS Digital Health.
https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000388.

72

400
401

Soice, E. et al., (2023) Can large language models democratize access to dual-use biotechnology? arXiv.
https://arxiv.org/abs/2306.03809.

402
403

Staab, R. et al., (2023) Beyond Memorization: Violating Privacy via Inference With Large Language
Models. arXiv. https://arxiv.org/pdf/2310.07298

404
405

Stanford, S. et al., (2023) Whose Opinions Do Language Models Reflect? arXiv.
https://arxiv.org/pdf/2303.17548.

406
407

Strubell, E. et al., (2019) Energy and Policy Considerations for Deep Learning in NLP. arXiv.
https://arxiv.org/pdf/1906.02243.

408
409
410

Thiel, D. (2023) Investigation Finds AI Image Generation Models Trained on Child Abuse. Stanford Cyber
Policy Center. https://cyber.fsi.stanford.edu/news/investigation-finds-ai-image-generation-modelstrained-child-abuse.

411

The Toxicity Issue. Jigsaw, Google. https://current.withgoogle.com/the-current/toxicity/.

412
413

Tufekci, Z. (2015) Algorithmic Harms Beyond Facebook and Google: Emergent Challenges ofComputational Agency. https://ctlj.colorado.edu/wp-content/uploads/2015/08/Tufekci-final.pdf

414
415
416

Turri, V. et al., (2023) Why We Need to Know More: Exploring the State of AI Incident Documentation
Practices. AAAI/ACM Conference on AI, Ethics, and Society.
https://dl.acm.org/doi/fullHtml/10.1145/3600211.3604700.

417
418

Urbina, F. et al., (2022) Dual use of artificial-intelligence-powered drug discovery. Nature MachineIntelligence. https://www.nature.com/articles/s42256-022-00465-9.

419
420

Wang, Y. et al., (2023) Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMs. arXiv.
https://arxiv.org/pdf/2308.13387.

421
422

Wang, X. et al., (2023) Energy and Carbon Considerations of Fine-Tuning BERT. ACL Anthology.
https://aclanthology.org/2023.findings-emnlp.607.pdf.

423
424
425

Wardle, C. et al., (2017) Information Disorder: Toward an interdisciplinary framework for research andpolicy making. Council of Europe. https://rm.coe.int/information-disorder-toward-an-interdisciplinaryframework-for-researc/168076277c.

426
427

Weatherbed, J., (2024) Trolls have flooded X with graphic Taylor Swift AI fakes. The Verge.
https://www.theverge.com/2024/1/25/24050334/x-twitter-taylor-swift-ai-fake-images-trending.

428
429

Weidinger, L. et al., (2021) Ethical and social risks of harm from Language Models. arXiv.
https://arxiv.org/pdf/2112.04359.

430
431

Weidinger, L. et al. (2023) Sociotechnical Safety Evaluation of Generative AI Systems. arXiv.
https://arxiv.org/pdf/2310.11986.

432
433

Weidinger, L. et al., (2022) Taxonomy of Risks posed by Language Models. FAccT ’22.
https://dl.acm.org/doi/pdf/10.1145/3531146.3533088.

434
435

Wu, K. et al., (2024) How well do LLMs cite relevant medical references? An evaluation framework andanalyses. arXiv. https://arxiv.org/pdf/2402.02008.

73

436
437

Yin, L. et al., (2024) OpenAI’s GPT Is A Recruiter’s Dream Tool. Tests Show There’s Racial Bias. Bloomberg.
https://www.bloomberg.com/graphics/2024-openai-gpt-hiring-racial-discrimination/.

438
439

Yu, Z. et al., (March 2024) Don’t Listen To Me: Understanding and Exploring Jailbreak Prompts of Large
Language Models. arXiv. https://arxiv.org/html/2403.17336v1

440
441
442
443
444
445

Zhang, Y. et al., (2023) Human favoritism, not AI aversion: People’s perceptions (and bias) toward
generative AI, human experts, and human–GAI collaboration in persuasive content generation. Judgment
and Decision Making. https://www.cambridge.org/core/journals/judgment-and-decisionmaking/article/human-favoritism-not-ai-aversion-peoples-perceptions-and-bias-toward-generative-aihuman-experts-and-humangai-collaboration-in-persuasive-contentgeneration/419C4BD9CE82673EAF1D8F6C350C4FA8.

446
447

Zhang, Y. et al., (2023) Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.
arXiv. https://arxiv.org/pdf/2309.01219.

448
449
450

Zhao, X. et al., (2023) Provable Robust Watermarking for AI-Generated Text. Semantic Scholar.
https://www.semanticscholar.org/paper/Provable-Robust-Watermarking-for-AI-Generated-Text-ZhaoAnanth/75b68d0903af9d9f6e47ce3cf7e1a7d27ec811dc.

74