Model: danicafisher/dfisher-fine-tuned-sentence-transformer
Author: danicafisher
Pipeline: sentence-similarity
Library: sentence-transformers
Tags: sentence-transformers, safetensors, bert, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:128, loss:MultipleNegativesRankingLoss, arxiv:1908.10084, arxiv:1705.00652, base_model:sentence-transformers/all-MiniLM-L6-v2, base_model:finetune:sentence-transformers/all-MiniLM-L6-v2, text-embeddings-inference, endpoints_compatible, region:us
Base models (finetune): id: sentence-transformers/all-MiniLM-L6-v2
Card metadata: base_model: sentence-transformers/all-MiniLM-L6-v2
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags: sentence-transformers, sentence-similarity, feature-extraction, generated_from_trainer, dataset_size:128, loss:MultipleNegativesRankingLoss
widget: source_sentence: What are the implications of large language models potentially deceiving their users under pressure, as discussed in the technical report by Scheurer et al (2023)?
sentences: 48 
• Data protection 
• Data retention  
• Consistency in use of deﬁning key terms 
• Decommissioning 
• Discouraging anonymous use 
• Education  
• Impact assessments  
• Incident response 
• Monitoring 
• Opt-outs  
• Risk-based controls 
• Risk mapping and measurement 
• Science-backed TEVV practices 
• Secure software development practices 
• Stakeholder engagement 
• Synthetic content detection and 
labeling tools and techniques 
• Whistleblower protections 
• Workforce diversity and 
interdisciplinary teams
Establishing acceptable use policies and guidance for the use of GAI in formal human-AI teaming settings 
as well as diﬀerent levels of human-AI conﬁgurations can help to decrease risks arising from misuse, 
abuse, inappropriate repurpose, and misalignment between systems and users. These practices are just 
one example of adapting existing governance protocols for GAI contexts.  
A.1.3. Third-Party Considerations 
Organizations may seek to acquire, embed, incorporate, or use open-source or proprietary third-party 
GAI models, systems, or generated data for various applications across an enterprise. Use of these GAI 
tools and inputs has implications for all functions of the organization – including but not limited to 
acquisition, human resources, legal, compliance, and IT services – regardless of whether they are carried 
out by employees or third parties. Many of the actions cited above are relevant and options for 
addressing third-party considerations. 
Third party GAI integrations may give rise to increased intellectual property, data privacy, or information 
security risks, pointing to the need for clear guidelines for transparency and risk management regarding 
the collection and use of third-party data for model inputs. Organizations may consider varying risk 
controls for foundation models, ﬁne-tuned models, and embedded tools, enhanced processes for 
interacting with external GAI technologies or service providers. Organizations can apply standard or 
existing risk controls and processes to proprietary or open-source GAI technologies, data, and third-party 
service providers, including acquisition and procurement due diligence, requests for software bills of 
materials (SBOMs), application of service level agreements (SLAs), and statement on standards for 
attestation engagement (SSAE) reports to help with third-party transparency and risk management for 
GAI systems. 
A.1.4. Pre-Deployment Testing 
Overview 
The diverse ways and contexts in which GAI systems may be developed, used, and repurposed 
complicates risk mapping and pre-deployment measurement eﬀorts. Robust test, evaluation, validation, 
and veriﬁcation (TEVV) processes can be iteratively applied – and documented – in early stages of the AI 
lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous, 21 
GV-6.1-005 
Implement a use-cased based supplier risk assessment framework to evaluate and 
monitor third-party entities’ performance and adherence to content provenance 
standards and technologies to detect anomalies and unauthorized changes; 
services acquisition and value chain risk management; and legal compliance. 
Data Privacy; Information 
Integrity; Information Security; 
Intellectual Property; Value Chain 
and Component Integration 
GV-6.1-006 Include clauses in contracts which allow an organization to evaluate third-party 
GAI processes and standards.  
Information Integrity 
GV-6.1-007 Inventory all third-party entities with access to organizational content and 
establish approved GAI technology and service provider lists. 
Value Chain and Component 
Integration 
GV-6.1-008 Maintain records of changes to content made by third parties to promote content 
provenance, including sources, timestamps, metadata. 
Information Integrity; Value Chain 
and Component Integration; 
Intellectual Property 
GV-6.1-009 
Update and integrate due diligence processes for GAI acquisition and 
procurement vendor assessments to include intellectual property, data privacy, 
security, and other risks. For example, update processes to: Address solutions that 
may rely on embedded GAI technologies; Address ongoing monitoring, 
assessments, and alerting, dynamic risk assessments, and real-time reporting 
tools for monitoring third-party GAI risks; Consider policy adjustments across GAI 
modeling libraries, tools and APIs, ﬁne-tuned models, and embedded tools; 
Assess GAI vendors, open-source or proprietary GAI tools, or GAI service 
providers against incident or vulnerability databases. 
Data Privacy; Human-AI 
Conﬁguration; Information 
Security; Intellectual Property; 
Value Chain and Component 
Integration; Harmful Bias and 
Homogenization 
GV-6.1-010 
Update GAI acceptable use policies to address proprietary and open-source GAI 
technologies and data, and contractors, consultants, and other third-party 
personnel. 
Intellectual Property; Value Chain 
and Component Integration 
AI Actor Tasks: Operation and Monitoring, Procurement, Third-party entities 
 
GOVERN 6.2: Contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be 
high-risk. 
Action ID 
Suggested Action 
GAI Risks 
GV-6.2-001 
Document GAI risks associated with system value chain to identify over-reliance 
on third-party data and to identify fallbacks. 
Value Chain and Component 
Integration 
GV-6.2-002 
Document incidents involving third-party GAI data and systems, including open-
data and open-source software. 
Intellectual Property; Value Chain 
and Component Integration, 58 
Satariano, A. et al. (2023) The People Onscreen Are Fake. The Disinformation Is Real. New York Times. 
https://www.nytimes.com/2023/02/07/technology/artiﬁcial-intelligence-training-deepfake.html 
Schaul, K. et al. (2024) Inside the secret list of websites that make AI like ChatGPT sound smart. 
Washington Post. https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ 
Scheurer, J. et al. (2023) Technical report: Large language models can strategically deceive their users 
when put under pressure. arXiv. https://arxiv.org/abs/2311.07590 
Shelby, R. et al. (2023) Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm 
Reduction. arXiv. https://arxiv.org/pdf/2210.05791 
Shevlane, T. et al. (2023) Model evaluation for extreme risks. arXiv. https://arxiv.org/pdf/2305.15324 
Shumailov, I. et al. (2023) The curse of recursion: training on generated data makes models forget. arXiv. 
https://arxiv.org/pdf/2305.17493v2 
Smith, A. et al. (2023) Hallucination or Confabulation? Neuroanatomy as metaphor in Large Language 
Models. PLOS Digital Health. 
https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000388 
Soice, E. et al. (2023) Can large language models democratize access to dual-use biotechnology? arXiv. 
https://arxiv.org/abs/2306.03809 
Solaiman, I. et al. (2023) The Gradient of Generative AI Release: Methods and Considerations. arXiv. 
https://arxiv.org/abs/2302.04844 
Staab, R. et al. (2023) Beyond Memorization: Violating Privacy via Inference With Large Language 
Models. arXiv. https://arxiv.org/pdf/2310.07298 
Stanford, S. et al. (2023) Whose Opinions Do Language Models Reﬂect? arXiv. 
https://arxiv.org/pdf/2303.17548 
Strubell, E. et al. (2019) Energy and Policy Considerations for Deep Learning in NLP. arXiv. 
https://arxiv.org/pdf/1906.02243 
The White House (2016) Circular No. A-130, Managing Information as a Strategic Resource. 
https://www.whitehouse.gov/wp-
content/uploads/legacy_drupal_ﬁles/omb/circulars/A130/a130revised.pdf 
The White House (2023) Executive Order on the Safe, Secure, and Trustworthy Development and Use of 
Artiﬁcial Intelligence. https://www.whitehouse.gov/brieﬁng-room/presidential-
actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-
artiﬁcial-intelligence/ 
The White House (2022) Roadmap for Researchers on Priorities Related to Information Integrity 
Research and Development. https://www.whitehouse.gov/wp-content/uploads/2022/12/Roadmap-
Information-Integrity-RD-2022.pdf? 
Thiel, D. (2023) Investigation Finds AI Image Generation Models Trained on Child Abuse. Stanford Cyber 
Policy Center. https://cyber.fsi.stanford.edu/news/investigation-ﬁnds-ai-image-generation-models-
trained-child-abuse, source_sentence: How should human subjects be informed about their options to withdraw participation or revoke consent in GAI applications?
sentences: 39 
MS-3.3-004 
Provide input for training materials about the capabilities and limitations of GAI 
systems related to digital content transparency for AI Actors, other 
professionals, and the public about the societal impacts of AI and the role of 
diverse and inclusive content generation. 
Human-AI Conﬁguration; 
Information Integrity; Harmful Bias 
and Homogenization 
MS-3.3-005 
Record and integrate structured feedback about content provenance from 
operators, users, and potentially impacted communities through the use of 
methods such as user research studies, focus groups, or community forums. 
Actively seek feedback on generated content quality and potential biases. 
Assess the general awareness among end users and impacted communities 
about the availability of these feedback channels. 
Human-AI Conﬁguration; 
Information Integrity; Harmful Bias 
and Homogenization 
AI Actor Tasks: AI Deployment, Aﬀected Individuals and Communities, End-Users, Operation and Monitoring, TEVV 
 
MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are 
informed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as 
intended. Results are documented. 
Action ID 
Suggested Action 
GAI Risks 
MS-4.2-001 
Conduct adversarial testing at a regular cadence to map and measure GAI risks, 
including tests to address attempts to deceive or manipulate the application of 
provenance techniques or other misuses. Identify vulnerabilities and 
understand potential misuse scenarios and unintended outputs. 
Information Integrity; Information 
Security 
MS-4.2-002 
Evaluate GAI system performance in real-world scenarios to observe its 
behavior in practical environments and reveal issues that might not surface in 
controlled and optimized testing environments. 
Human-AI Conﬁguration; 
Confabulation; Information 
Security 
MS-4.2-003 
Implement interpretability and explainability methods to evaluate GAI system 
decisions and verify alignment with intended purpose. 
Information Integrity; Harmful Bias 
and Homogenization 
MS-4.2-004 
Monitor and document instances where human operators or other systems 
override the GAI's decisions. Evaluate these cases to understand if the overrides 
are linked to issues related to content provenance. 
Information Integrity 
MS-4.2-005 
Verify and document the incorporation of results of structured public feedback 
exercises into design, implementation, deployment approval (“go”/“no-go” 
decisions), monitoring, and decommission decisions. 
Human-AI Conﬁguration; 
Information Security 
AI Actor Tasks: AI Deployment, Domain Experts, End-Users, Operation and Monitoring, TEVV, 30 
MEASURE 2.2: Evaluations involving human subjects meet applicable requirements (including human subject protection) and are 
representative of the relevant population. 
Action ID 
Suggested Action 
GAI Risks 
MS-2.2-001 Assess and manage statistical biases related to GAI content provenance through 
techniques such as re-sampling, re-weighting, or adversarial training. 
Information Integrity; Information 
Security; Harmful Bias and 
Homogenization 
MS-2.2-002 
Document how content provenance data is tracked and how that data interacts 
with privacy and security. Consider: Anonymizing data to protect the privacy of 
human subjects; Leveraging privacy output ﬁlters; Removing any personally 
identiﬁable information (PII) to prevent potential harm or misuse. 
Data Privacy; Human AI 
Conﬁguration; Information 
Integrity; Information Security; 
Dangerous, Violent, or Hateful 
Content 
MS-2.2-003 Provide human subjects with options to withdraw participation or revoke their 
consent for present or future use of their data in GAI applications.  
Data Privacy; Human-AI 
Conﬁguration; Information 
Integrity 
MS-2.2-004 
Use techniques such as anonymization, diﬀerential privacy or other privacy-
enhancing technologies to minimize the risks associated with linking AI-generated 
content back to individual human subjects. 
Data Privacy; Human-AI 
Conﬁguration 
AI Actor Tasks: AI Development, Human Factors, TEVV 
 
MEASURE 2.3: AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for 
conditions similar to deployment setting(s). Measures are documented. 
Action ID 
Suggested Action 
GAI Risks 
MS-2.3-001 Consider baseline model performance on suites of benchmarks when selecting a 
model for ﬁne tuning or enhancement with retrieval-augmented generation. 
Information Security; 
Confabulation 
MS-2.3-002 Evaluate claims of model capabilities using empirically validated methods. 
Confabulation; Information 
Security 
MS-2.3-003 Share results of pre-deployment testing with relevant GAI Actors, such as those 
with system release approval authority. 
Human-AI Conﬁguration, 36 
MEASURE 2.11: Fairness and bias – as identiﬁed in the MAP function – are evaluated and results are documented. 
Action ID 
Suggested Action 
GAI Risks 
MS-2.11-001 
Apply use-case appropriate benchmarks (e.g., Bias Benchmark Questions, Real 
Hateful or Harmful Prompts, Winogender Schemas15) to quantify systemic bias, 
stereotyping, denigration, and hateful content in GAI system outputs; 
Document assumptions and limitations of benchmarks, including any actual or 
possible training/test data cross contamination, relative to in-context 
deployment environment. 
Harmful Bias and Homogenization 
MS-2.11-002 
Conduct fairness assessments to measure systemic bias. Measure GAI system 
performance across demographic groups and subgroups, addressing both 
quality of service and any allocation of services and resources. Quantify harms 
using: ﬁeld testing with sub-group populations to determine likelihood of 
exposure to generated content exhibiting harmful bias, AI red-teaming with 
counterfactual and low-context (e.g., “leader,” “bad guys”) prompts. For ML 
pipelines or business processes with categorical or numeric outcomes that rely 
on GAI, apply general fairness metrics (e.g., demographic parity, equalized odds, 
equal opportunity, statistical hypothesis tests), to the pipeline or business 
outcome where appropriate; Custom, context-speciﬁc metrics developed in 
collaboration with domain experts and aﬀected communities; Measurements of 
the prevalence of denigration in generated content in deployment (e.g., sub-
sampling a fraction of traﬃc and manually annotating denigrating content). 
Harmful Bias and Homogenization; 
Dangerous, Violent, or Hateful 
Content 
MS-2.11-003 
Identify the classes of individuals, groups, or environmental ecosystems which 
might be impacted by GAI systems through direct engagement with potentially 
impacted communities. 
Environmental; Harmful Bias and 
Homogenization 
MS-2.11-004 
Review, document, and measure sources of bias in GAI training and TEVV data: 
Diﬀerences in distributions of outcomes across and within groups, including 
intersecting groups; Completeness, representativeness, and balance of data 
sources; demographic group and subgroup coverage in GAI system training 
data; Forms of latent systemic bias in images, text, audio, embeddings, or other 
complex or unstructured data; Input data features that may serve as proxies for 
demographic group membership (i.e., image metadata, language dialect) or 
otherwise give rise to emergent bias within GAI systems; The extent to which 
the digital divide may negatively impact representativeness in GAI system 
training and TEVV data; Filtering of hate speech or content in GAI system 
training data; Prevalence of GAI-generated data in GAI system training data. 
Harmful Bias and Homogenization 
 
 
15 Winogender Schemas is a sample set of paired sentences which diﬀer only by gender of the pronouns used, 
which can be used to evaluate gender bias in natural language processing coreference resolution systems., source_sentence: What is the title of the NIST publication related to Artificial Intelligence Risk Management?
sentences: 53 
Documenting, reporting, and sharing information about GAI incidents can help mitigate and prevent 
harmful outcomes by assisting relevant AI Actors in tracing impacts to their source. Greater awareness 
and standardization of GAI incident reporting could promote this transparency and improve GAI risk 
management across the AI ecosystem.  
Documentation and Involvement of AI Actors 
AI Actors should be aware of their roles in reporting AI incidents. To better understand previous incidents 
and implement measures to prevent similar ones in the future, organizations could consider developing 
guidelines for publicly available incident reporting which include information about AI actor 
responsibilities. These guidelines would help AI system operators identify GAI incidents across the AI 
lifecycle and with AI Actors regardless of role. Documentation and review of third-party inputs and 
plugins for GAI systems is especially important for AI Actors in the context of incident disclosure; LLM 
inputs and content delivered through these plugins is often distributed, with inconsistent or insuﬃcient 
access control. 
Documentation practices including logging, recording, and analyzing GAI incidents can facilitate 
smoother sharing of information with relevant AI Actors. Regular information sharing, change 
management records, version history and metadata can also empower AI Actors responding to and 
managing AI incidents., 23 
MP-1.1-002 
Determine and document the expected and acceptable GAI system context of 
use in collaboration with socio-cultural and other domain experts, by assessing: 
Assumptions and limitations; Direct value to the organization; Intended 
operational environment and observed usage patterns; Potential positive and 
negative impacts to individuals, public safety, groups, communities, 
organizations, democratic institutions, and the physical environment; Social 
norms and expectations. 
Harmful Bias and Homogenization 
MP-1.1-003 
Document risk measurement plans to address identiﬁed risks. Plans may 
include, as applicable: Individual and group cognitive biases (e.g., conﬁrmation 
bias, funding bias, groupthink) for AI Actors involved in the design, 
implementation, and use of GAI systems; Known past GAI system incidents and 
failure modes; In-context use and foreseeable misuse, abuse, and oﬀ-label use; 
Over reliance on quantitative metrics and methodologies without suﬃcient 
awareness of their limitations in the context(s) of use; Standard measurement 
and structured human feedback approaches; Anticipated human-AI 
conﬁgurations. 
Human-AI Conﬁguration; Harmful 
Bias and Homogenization; 
Dangerous, Violent, or Hateful 
Content 
MP-1.1-004 
Identify and document foreseeable illegal uses or applications of the GAI system 
that surpass organizational risk tolerances. 
CBRN Information or Capabilities; 
Dangerous, Violent, or Hateful 
Content; Obscene, Degrading, 
and/or Abusive Content 
AI Actor Tasks: AI Deployment 
 
MAP 1.2: Interdisciplinary AI Actors, competencies, skills, and capacities for establishing context reﬂect demographic diversity and 
broad domain and user experience expertise, and their participation is documented. Opportunities for interdisciplinary 
collaboration are prioritized. 
Action ID 
Suggested Action 
GAI Risks 
MP-1.2-001 
Establish and empower interdisciplinary teams that reﬂect a wide range of 
capabilities, competencies, demographic groups, domain expertise, educational 
backgrounds, lived experiences, professions, and skills across the enterprise to 
inform and conduct risk measurement and management functions. 
Human-AI Conﬁguration; Harmful 
Bias and Homogenization 
MP-1.2-002 
Verify that data or benchmarks used in risk measurement, and users, 
participants, or subjects involved in structured GAI public feedback exercises 
are representative of diverse in-context user populations. 
Human-AI Conﬁguration; Harmful 
Bias and Homogenization 
AI Actor Tasks: AI Deployment, NIST Trustworthy and Responsible AI  
NIST AI 600-1 
Artificial Intelligence Risk Management 
Framework: Generative Artificial 
Intelligence Profile 
 
 
 
This publication is available free of charge from: 
https://doi.org/10.6028/NIST.AI.600-1, source_sentence: What is the purpose of the AI Risk Management Framework (AI RMF) for Generative AI as outlined in the document?
sentences: Table of Contents 
1. 
Introduction ..............................................................................................................................................1 
2. 
Overview of Risks Unique to or Exacerbated by GAI .....................................................................2 
3. 
Suggested Actions to Manage GAI Risks ......................................................................................... 12 
Appendix A. Primary GAI Considerations ............................................................................................... 47 
Appendix B. References ................................................................................................................................ 54, 13 
• 
Not every suggested action applies to every AI Actor14 or is relevant to every AI Actor Task. For 
example, suggested actions relevant to GAI developers may not be relevant to GAI deployers. 
The applicability of suggested actions to relevant AI actors should be determined based on 
organizational considerations and their unique uses of GAI systems. 
Each table of suggested actions includes: 
• 
Action ID: Each Action ID corresponds to the relevant AI RMF function and subcategory (e.g., GV-
1.1-001 corresponds to the ﬁrst suggested action for Govern 1.1, GV-1.1-002 corresponds to the 
second suggested action for Govern 1.1). AI RMF functions are tagged as follows: GV = Govern; 
MP = Map; MS = Measure; MG = Manage. 
• 
Suggested Action: Steps an organization or AI actor can take to manage GAI risks.  
• 
GAI Risks: Tags linking suggested actions with relevant GAI risks.  
• 
AI Actor Tasks: Pertinent AI Actor Tasks for each subcategory. Not every AI Actor Task listed will 
apply to every suggested action in the subcategory (i.e., some apply to AI development and 
others apply to AI deployment).  
The tables below begin with the AI RMF subcategory, shaded in blue, followed by suggested actions.  
 
GOVERN 1.1: Legal and regulatory requirements involving AI are understood, managed, and documented.  
Action ID 
Suggested Action 
GAI Risks 
GV-1.1-001 Align GAI development and use with applicable laws and regulations, including 
those related to data privacy, copyright and intellectual property law. 
Data Privacy; Harmful Bias and 
Homogenization; Intellectual 
Property 
AI Actor Tasks: Governance and Oversight 
 
 
 
14 AI Actors are deﬁned by the OECD as “those who play an active role in the AI system lifecycle, including 
organizations and individuals that deploy or operate AI.” See Appendix A of the AI RMF for additional descriptions 
of AI Actors and AI Actor Tasks., 1 
1. 
Introduction 
This document is a cross-sectoral proﬁle of and companion resource for the AI Risk Management 
Framework (AI RMF 1.0) for Generative AI,1 pursuant to President Biden’s Executive Order (EO) 14110 on 
Safe, Secure, and Trustworthy Artiﬁcial Intelligence.2 The AI RMF was released in January 2023, and is 
intended for voluntary use and to improve the ability of organizations to incorporate trustworthiness 
considerations into the design, development, use, and evaluation of AI products, services, and systems.  
A proﬁle is an implementation of the AI RMF functions, categories, and subcategories for a speciﬁc 
setting, application, or technology – in this case, Generative AI (GAI) – based on the requirements, risk 
tolerance, and resources of the Framework user. AI RMF proﬁles assist organizations in deciding how to 
best manage AI risks in a manner that is well-aligned with their goals, considers legal/regulatory 
requirements and best practices, and reﬂects risk management priorities. Consistent with other AI RMF 
proﬁles, this proﬁle oﬀers insights into how risk can be managed across various stages of the AI lifecycle 
and for GAI as a technology.  
As GAI covers risks of models or applications that can be used across use cases or sectors, this document 
is an AI RMF cross-sectoral proﬁle. Cross-sectoral proﬁles can be used to govern, map, measure, and 
manage risks associated with activities or business processes common across sectors, such as the use of 
large language models (LLMs), cloud-based services, or acquisition. 
This document deﬁnes risks that are novel to or exacerbated by the use of GAI. After introducing and 
describing these risks, the document provides a set of suggested actions to help organizations govern, 
map, measure, and manage these risks. 
 
 
1 EO 14110 deﬁnes Generative AI as “the class of AI models that emulate the structure and characteristics of input 
data in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital 
content.” While not all GAI is derived from foundation models, for purposes of this document, GAI generally refers 
to generative foundation models. The foundation model subcategory of “dual-use foundation models” is deﬁned by 
EO 14110 as “an AI model that is trained on broad data; generally uses self-supervision; contains at least tens of 
billions of parameters; is applicable across a wide range of contexts.”  
2 This proﬁle was developed per Section 4.1(a)(i)(A) of EO 14110, which directs the Secretary of Commerce, acting 
through the Director of the National Institute of Standards and Technology (NIST), to develop a companion 
resource to the AI RMF, NIST AI 100–1, for generative AI., source_sentence: What are the primary information security risks associated with GAI-based systems in the context of cybersecurity?
sentences: 7 
unethical behavior. Text-to-image models also make it easy to create images that could be used to 
promote dangerous or violent messages. Similar concerns are present for other GAI media, including 
video and audio. GAI may also produce content that recommends self-harm or criminal/illegal activities.  
Many current systems restrict model outputs to limit certain content or in response to certain prompts, 
but this approach may still produce harmful recommendations in response to other less-explicit, novel 
prompts (also relevant to CBRN Information or Capabilities, Data Privacy, Information Security, and 
Obscene, Degrading and/or Abusive Content). Crafting such prompts deliberately is known as 
“jailbreaking,” or, manipulating prompts to circumvent output controls. Limitations of GAI systems can be 
harmful or dangerous in certain contexts. Studies have observed that users may disclose mental health 
issues in conversations with chatbots – and that users exhibit negative reactions to unhelpful responses 
from these chatbots during situations of distress. 
This risk encompasses diﬃculty controlling creation of and public exposure to oﬀensive or hateful 
language, and denigrating or stereotypical content generated by AI. This kind of speech may contribute 
to downstream harm such as fueling dangerous or violent behaviors. The spread of denigrating or 
stereotypical content can also further exacerbate representational harms (see Harmful Bias and 
Homogenization below).  
Trustworthy AI Characteristics: Safe, Secure and Resilient 
2.4. Data Privacy 
GAI systems raise several risks to privacy. GAI system training requires large volumes of data, which in 
some cases may include personal data. The use of personal data for GAI training raises risks to widely 
accepted privacy principles, including to transparency, individual participation (including consent), and 
purpose speciﬁcation. For example, most model developers do not disclose speciﬁc data sources on 
which models were trained, limiting user awareness of whether personally identiﬁably information (PII) 
was trained on and, if so, how it was collected.  
Models may leak, generate, or correctly infer sensitive information about individuals. For example, 
during adversarial attacks, LLMs have revealed sensitive information (from the public domain) that was 
included in their training data. This problem has been referred to as data memorization, and may pose 
exacerbated privacy risks even for data present only in a small number of training samples.  
In addition to revealing sensitive information in GAI training data, GAI models may be able to correctly 
infer PII or sensitive data that was not in their training data nor disclosed by the user by stitching 
together information from disparate sources. These inferences can have negative impact on an individual 
even if the inferences are not accurate (e.g., confabulations), and especially if they reveal information 
that the individual considers sensitive or that is used to disadvantage or harm them. 
Beyond harms from information exposure (such as extortion or dignitary harm), wrong or inappropriate 
inferences of PII can contribute to downstream or secondary harmful impacts. For example, predictive 
inferences made by GAI models based on PII or protected attributes can contribute to adverse decisions, 
leading to representational or allocative harms to individuals or groups (see Harmful Bias and 
Homogenization below)., 10 
GAI systems can ease the unintentional production or dissemination of false, inaccurate, or misleading 
content (misinformation) at scale, particularly if the content stems from confabulations.  
GAI systems can also ease the deliberate production or dissemination of false or misleading information 
(disinformation) at scale, where an actor has the explicit intent to deceive or cause harm to others. Even 
very subtle changes to text or images can manipulate human and machine perception. 
Similarly, GAI systems could enable a higher degree of sophistication for malicious actors to produce 
disinformation that is targeted towards speciﬁc demographics. Current and emerging multimodal models 
make it possible to generate both text-based disinformation and highly realistic “deepfakes” – that is, 
synthetic audiovisual content and photorealistic images.12 Additional disinformation threats could be 
enabled by future GAI models trained on new data modalities. 
Disinformation and misinformation – both of which may be facilitated by GAI – may erode public trust in 
true or valid evidence and information, with downstream eﬀects. For example, a synthetic image of a 
Pentagon blast went viral and brieﬂy caused a drop in the stock market. Generative AI models can also 
assist malicious actors in creating compelling imagery and propaganda to support disinformation 
campaigns, which may not be photorealistic, but could enable these campaigns to gain more reach and 
engagement on social media platforms. Additionally, generative AI models can assist malicious actors in 
creating fraudulent content intended to impersonate others. 
Trustworthy AI Characteristics: Accountable and Transparent, Safe, Valid and Reliable, Interpretable and 
Explainable 
2.9. Information Security 
Information security for computer systems and data is a mature ﬁeld with widely accepted and 
standardized practices for oﬀensive and defensive cyber capabilities. GAI-based systems present two 
primary information security risks: GAI could potentially discover or enable new cybersecurity risks by 
lowering the barriers for or easing automated exercise of oﬀensive capabilities; simultaneously, it 
expands the available attack surface, as GAI itself is vulnerable to attacks like prompt injection or data 
poisoning.  
Oﬀensive cyber capabilities advanced by GAI systems may augment cybersecurity attacks such as 
hacking, malware, and phishing. Reports have indicated that LLMs are already able to discover some 
vulnerabilities in systems (hardware, software, data) and write code to exploit them. Sophisticated threat 
actors might further these risks by developing GAI-powered security co-pilots for use in several parts of 
the attack chain, including informing attackers on how to proactively evade threat detection and escalate 
privileges after gaining system access. 
Information security for GAI models and systems also includes maintaining availability of the GAI system 
and the integrity and (when applicable) the conﬁdentiality of the GAI code, training data, and model 
weights. To identify and secure potential attack points in AI systems or speciﬁc components of the AI 
 
 
12 See also https://doi.org/10.6028/NIST.AI.100-4, to be published., 16 
GOVERN 1.5: Ongoing monitoring and periodic review of the risk management process and its outcomes are planned, and 
organizational roles and responsibilities are clearly deﬁned, including determining the frequency of periodic review. 
Action ID 
Suggested Action 
GAI Risks 
GV-1.5-001 Deﬁne organizational responsibilities for periodic review of content provenance 
and incident monitoring for GAI systems. 
Information Integrity 
GV-1.5-002 
Establish organizational policies and procedures for after action reviews of GAI 
system incident response and incident disclosures, to identify gaps; Update 
incident response and incident disclosure processes as required. 
Human-AI Conﬁguration; 
Information Security 
GV-1.5-003 
Maintain a document retention policy to keep history for test, evaluation, 
validation, and veriﬁcation (TEVV), and digital content transparency methods for 
GAI. 
Information Integrity; Intellectual 
Property 
AI Actor Tasks: Governance and Oversight, Operation and Monitoring 
 
GOVERN 1.6: Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities. 
Action ID 
Suggested Action 
GAI Risks 
GV-1.6-001 Enumerate organizational GAI systems for incorporation into AI system inventory 
and adjust AI system inventory requirements to account for GAI risks. 
Information Security 
GV-1.6-002 Deﬁne any inventory exemptions in organizational policies for GAI systems 
embedded into application software. 
Value Chain and Component 
Integration 
GV-1.6-003 
In addition to general model, governance, and risk information, consider the 
following items in GAI system inventory entries: Data provenance information 
(e.g., source, signatures, versioning, watermarks); Known issues reported from 
internal bug tracking or external information sharing resources (e.g., AI incident 
database, AVID, CVE, NVD, or OECD AI incident monitor); Human oversight roles 
and responsibilities; Special rights and considerations for intellectual property, 
licensed works, or personal, privileged, proprietary or sensitive data; Underlying 
foundation models, versions of underlying models, and access modes. 
Data Privacy; Human-AI 
Conﬁguration; Information 
Integrity; Intellectual Property; 
Value Chain and Component 
Integration 
AI Actor Tasks: Governance and Oversight
Likes: 0 | All-time downloads: 8
Created: 2024-09-20 16:14:22