LLM AI Cybersecurity &Governance Checklist
From the OWASP Top 10
for LLM Applications Team

Version: 1.0
Published: February 19, 2024

Revision History
Revision
0.1
0.5
0.9
1.0

Date
2023-11-01
2023-12-06
2023-02-15
2024-02-19

Author(s)Sandy Dunn
SD, Team
SD, Team
SD, Team

Description
initial draft
public draft
pre-release draft
public release v 1.0

The information provided in this document does not, and is not intended to, constitute legal advice.
All information is for general informational purposes only.
This document contains links to other third-party websites. Such links are only for convenienceand OWASP does not recommend or endorse the contents of the third-party sites.

1

Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.1

Responsible and Trustworthy Artiﬁcial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.2

Who is This For? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.3

Why a Checklist? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.4

Not Comprehensive . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.5

Large Language Model Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.6

LLM Threat Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.7

Artiﬁcial Intelligence Security and Privacy Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.8

Incorporate LLM Security and governance with Existing, Established Practices and Controls9

1.9

Fundamental Security Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.10

Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.11

Vulnerability and Mitigation Taxonomy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

2

Determining LLM Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2.1

Deployment Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3

Checklist . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.1

Adversarial Risk . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.2

Threat Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.3

AI Asset Inventory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

3.4

AI Security and Privacy Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.5

Establish Business Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

3.6

Governance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.7

Legal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3.8

Regulatory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

3.9

Using or Implementing Large Language Model Solutions . . . . . . . . . . . . . . . . . . . . . . . . 19

3.10

Testing, Evaluation, Veriﬁcation, and Validation (TEVV) . . . . . . . . . . . . . . . . . . . . . . . . . 19

3.11

Model Cards and Risk Cards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

3.12

RAG: Large Language Model Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.13

AI Red Teaming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

4Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

ATeam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

Overview
Every internet user and company should prepare for the upcoming wave of powerful generative
artiﬁcial intelligence (GenAI) applications. GenAI has enormous promise for innovation, efﬁciency,
and commercial success across a variety of industries. Still, like any powerful early stage technology,
it brings its own set of obvious and unexpected challenges.
Artiﬁcial intelligence has advanced greatly over the last 50 years, inconspicuously supporting avariety of corporate processes until ChatGPT’s public appearance drove the development and use of
Large Language Models (LLMs) among both individuals and enterprises. Initially, these technologies
were limited to academic study or the execution of certain, but vital, activities within corporations,visible only to a select few. However, recent advances in data availability, computer power, GenAI
capabilities, and the release of tools such as Llama 2, ElevenLabs, and Midjourney have raised AI
from a niche to general widespread acceptance. These improvements have not only made GenAI
technologies more accessible, but they have also highlighted the critical need for enterprises todevelop solid strategies for integrating and exploiting AI in their operations, representing a huge
step forward in how we use technology.

• Artiﬁcial intelligence (AI) is a broad term that encompasses all ﬁelds of computer sciencethat enable machines to accomplish tasks that would normally require human intelligence.
Machine learning and generative AI are two subcategories of AI.

• Machine learning is a subset of AI that focuses on creating algorithms that can learn from

data. Machine learning algorithms are trained on a set of data, and then they can use that data
to make predictions or decisions about new data.

• Generative AI is a type of machine learning that focuses on creating new data.
• A large language model (LLM) is a type of AI model that processes and generates human-like

text. In the context of artiﬁcial intelligence a "model" refers to a system that is trained to makepredictions based on input data. LLMs are speciﬁcally trained on large data sets of natural
language and the name large language models.

Organizations are entering uncharted territory in securing and overseeing GenAI solutions. The
rapid advancement of GenAI also opens doors for adversaries to enhance their attack strategies,
introducing a dual challenge of defense and threat escalation.

Businesses use artiﬁcial intelligence in many areas, including HR for recruiting, email spam screening,
SIEM for behavioral analytics, and managed detection and response applications. However, thisdocument’s primary focus is on Large Language Model applications and their function in creating
generated content.

Responsible and Trustworthy Artiﬁcial Intelligence
As challenges and beneﬁts of Artiﬁcial Intelligence emerge - and regulations and laws are passed the principles and pillars of responsible and trustworthy AI usage are evolving from idealistic objects
and concerns to established standards. The OWASP AI Exchange Working Group is monitoringthese changes and addressing the broader and more challenging considerations for all aspects ofartiﬁcial intelligence.

Figure 1.1: Image depicting the pillars of trustworthy artiﬁcial intelligenceWho is This For?
The OWASP Top 10 for LLM Applications Cybersecurity and Governance Checklist is for leadersacross executive, tech, cybersecurity, privacy, compliance, and legal areas, DevSecOps, MLSecOps,
and Cybersecurity teams and defenders. It is intended for people who are striving to stay ahead in
the fast-moving AI world, aiming not just to leverage AI for corporate success but also to protect
against the risks of hasty or insecure AI implementations. These leaders and teams must create
tactics to grab opportunities, combat challenges, and mitigate risks.
This checklist is intended to help these technology and business leaders quickly understand therisks and beneﬁts of using LLM, allowing them to focus on developing a comprehensive list of criticalareas and tasks needed to defend and protect the organization as they develop a Large Language
Model strategy.
It is the hope of the OWASP Top 10 for the LLM Applications team that this list will help organizations
improve their existing defensive techniques and develop techniques to address the new threats thatcome from using this exciting technology.

Why a Checklist?
Checklists used to formulate strategies improve accuracy, deﬁne objectives, preserve uniformity,
and promote focused deliberate work, reducing oversights and missed details. Following a checklist not only increases trust in a safe adoption journey, but also encourages future organizations
innovations by providing a simple and effective strategy for continuous improvement.

Not Comprehensive
Although this document intends to support organizations in developing an initial LLM strategy ina rapidly changing technical, legal, and regulatory environment, it is not exhaustive and does notcover every use case or obligation. While using this document is Organizations should extend
assessments and practices beyond the scope of the provided checklist as required for their use
case or jurisdiction.

Large Language Model Challenges
Large Language models face several serious and unique issues. One of the most important isthat while working with LLMs, the control and data planes cannot be strictly isolated or separable.
Another signiﬁcant challenge is that LLMs are nondeterministic by design, yielding a different
outcome when prompted or requested. LLMs employ semantic search rather than keyword search.
The key distinction between the two is that the model’s algorithm prioritizes the terms in its response.
This is a signiﬁcant departure from how consumers have previously used technology, and it has an
impact on the consistency and reliability of the ﬁndings. Hallucinations, emerging from the gaps
and training ﬂaws in the data the model is trained on, are the result of this method.
There are methods to improve reliability and reduce the attack surface for jailbreaking, model
tricking, and hallucinations, but there is a trade-off between restrictions and utility in both cost andfunctionality.
LLM use and LLM applications increase an organization’s attack surface. Some risks associatedwith LLMs are unique, but many are familiar issues, such as the known software bill of materials(SBoM), supply chain, data loss protection (DLP), and authorized access. There are also increased
risks not directly related to GenAI, but GenAI increases the efﬁciency, capability, and effectivenessof attackers who attack and threaten organizations.
Adversaries are increasingly harnessing LLM and Generative AI tools to reﬁne and expedite traditional
methods of attacking organizations, individuals, and government systems. LLM facilitates theirability to enhance techniques allowing them to effortlessly craft new malware, potentially embeddedwith novel zero-day vulnerabilities or designed to evade detection. They can also generate sophisticated,unique, or tailored phishing schemes. The creation of convincing deep fakes, whether video or audio,
further promotes their social engineering ploys. Additionally, these tools enable them to execute
intrusions and develop innovative hacking capabilities. In the future, more “tailored” and compound
use of AI technology by criminal actors will demand speciﬁc responses and dedicated solutions foran organization’s appropriate defense and resilience capabilities.
Organizations also face the threat of NOT utilizing the capabilities of LLMs such as a competitive
disadvantage, market perception by customers and partners of being outdated, inability to scalepersonalized communications, innovation stagnation, operational inefﬁciencies, the higher risk ofhuman error in processes, and inefﬁcient allocation of human resources.
Understanding the different kinds of threats and integrating them with the business strategy will
help weigh both the pros and cons of using Large Language Models (LLMs) against not using them,making sure they accelerate rather than hinder the business’s meeting business objectives.

LLM Threat Categories

Figure 1.2: Image depicting the types of AI threatsArtiﬁcial Intelligence Security and Privacy Training
Employees throughout organizations beneﬁt from training to understand artiﬁcial intelligence,
generative artiﬁcial intelligence, and the future potential consequences of building, buying, or utilizing
LLMs. Training for permissible use and security awareness should target all employees as well as
be more specialized for certain positions such as human resources, legal, developers, data teams,
and security teams.
Fair use policies and healthy interaction are key aspects that, if incorporated from the very start,
will be a cornerstone to the success of future AI cybersecurity awareness campaigns. This will
necessarily provide users with knowledge of the basic rules for interaction as well as the ability toseparate good behavior from bad or unethical behavior.

Incorporate LLM Security and governance with Existing, Established Practices
and Controls
While AI and generated AI add a new dimension to cybersecurity, resilience, privacy, and meeting
legal and regulatory requirements, the best practices that have been around for a long time are still
the best way to identify issues, ﬁnd vulnerabilities, ﬁx them, and mitigate potential security issues.

• Conﬁrm the management of artiﬁcial intelligence systems is integrated with existing organizational
practices.

• Conﬁrm AIML systems follow existing privacy, governance, and security practices, with AI
speciﬁc privacy, governance, and security practices implemented when required.

Fundamental Security Principles
LLM capabilities introduce a different type of attack and attack surface. LLMs are vulnerable
to complex business logic bugs, such as prompt injection, insecure plugin design, and remotecode execution. Existing best practices are the best way to solve these issues. An internal productsecurity team that understands secure software review, architecture, data governance, and third-party
assessments The cybersecurity team should also check how strong the current controls are to ﬁnd
problems that could be made worse by LLM, such as voice cloning, impersonation, or bypassing
captchas. Given recent advancements in machine learning, NLP (Natural Language Processing),
NLU (Natural Language Understanding), Deep Learning, and more recently, LLMs (Large Language
Models) and Generative AI, it is recommended to include professionals proﬁcient in these areas
alongside cybersecurity and devops teams. Their expertise will not only aid in adopting thesetechnologies but also in developing innovative analyses and responses to emerging challenges.

Risk
Reference to risk uses the ISO 31000 deﬁnition: Risk = "effect of uncertainty on objectives." LLM
risks included in the checklist includes a targeted list of LLM risks that address adversarial, safety,
legal, regulatory, reputation, ﬁnancial, and competitive risks.

Vulnerability and Mitigation Taxonomy
Current systems for classifying vulnerabilities and sharing threat information, like OVAL, STIX, CVE,
and CWE, are still developing the ability to monitor and alert defenders about vulnerabilities andthreats speciﬁc to Large Language Models (LLMs) and Predictive Models. It is expected thatorganizations will lean on these established and recognized standards, such as CVE for vulnerability
classiﬁcation and STIX for the exchange of cyber threat intelligence (CTI), when vulnerabilities orthreats to AI/ML systems and their supply chains are identiﬁed.

Determining LLM Strategy
The rapid expansion of Large Language Model (LLM) applications has heightened the attention
and examination of all AI/ML systems used in business operations, encompassing both Generative
AI and long-established Predictive AI/ML systems. This increased focus exposes potential risks,such as attackers targeting systems that were previously overlooked and governance or legal
challenges that may have been disregarded in terms of legal, privacy, liability, or warranty issues.
For any organization leveraging AI/ML systems in its operations, it’s critical to assess and establish
comprehensive policies, governance, security protocols, privacy measures, and accountability
standards to ensure these technologies align with business processes securely and ethically.
Attackers, or adversaries, provide the most immediate and harmful threat to enterprises, people,
and government agencies. Their goals, which range from ﬁnancial gain to espionage, push them tosteal critical information, disrupt operations, and damage conﬁdence. Furthermore, their ability toharness new technologies such as AI and machine learning increases the speed and sophisticationof attacks, making it difﬁcult for defenses to stay ahead of attacks.
The most pressing non-adversary LLM threat for many organizations stem from "Shadow AI":
employees using unapproved online AI tools, unsafe browser plugins, and third-party applications
that introduce LLM features via updates or upgrades, circumventing standard software approval
processes.

Figure 2.1: Image of options for deployment strategy

Deployment Strategy
The scopes range from leveraging public consumer applications to training proprietary models on
private data. Factors like use case sensitivity, capabilities needed, and resources available help
determine the right balance of convenience vs. control. However, understanding these ﬁve modeltypes provides a framework for evaluating options.

Figure 2.2: Image of options for deployment types

Checklist

Adversarial Risk
Adversarial Risk includes competitors and attackers.
□ Scrutinize how competitors are investing in artiﬁcial intelligence. Although there are risks in AI
adoption, there are also business beneﬁts that may impact future market positions.
□ Investigate the impact of current controls, such as password resets, which use voicerecognition which may no longer provide the appropriate defensive security from new GenAI
enhanced attacks.
□ Update the Incident Response Plan and playbooks for GenAI enhanced attacks and AIML
speciﬁc incidents.

Threat ModelingThreat modeling is highly recommended to identify threats and examine processes and security
defenses. Threat modeling is a set of systematic, repeatable processes that enable making
reasonable security decisions for applications, software, and systems. Threat modeling for GenAI
accelerated attacks and before deploying LLMs is the most cost effective way to Identify and mitigate
risks, protect data, protect privacy, and ensure a secure, compliant integration within the business.
□ How will attackers accelerate exploit attacks against the organization, employees, executives,
or users? Organizations should anticipate "hyper-personalized" attacks at scale using
Generative AI. LLM-assisted Spear Phishing attacks are now exponentially more effective,
targeted, and weaponized for an attack.
□ How could GenAI be used for attacks on the business’s customers or clients through spooﬁng
or GenAI generated content?
□ Can the business detect and neutralize harmful or malicious inputs or queries to LLM solutions?
□ Can the business safeguard connections with existing systems and databases with secure
integrations at all LLM trust boundaries?
□ Does the business have insider threat mitigation to prevent misuse by authorized users?
□ Can the business prevent unauthorized access to proprietary models or data to protect
Intellectual Property?
□ Can the business prevent the generation of harmful or inappropriate content with automatedcontent ﬁltering?

AI Asset Inventory
An AI asset inventory should apply to both internally developed and external or third-party solutions.

□ Catalog existing AI services, tools, and owners. Designate a tag in asset management for
speciﬁc inventory.
□ Include AI components in the Software Bill of Material (SBOM), a comprehensive list of all thesoftware components, dependencies, and metadata associated with applications.
□ Catalog AI data sources and the sensitivity of the data (protected, conﬁdential, public)□ Establish if pen testing or red teaming of deployed AI solutions is required to determine thecurrent attack surface risk.
□ Create an AI solution onboarding process.
□ Ensure skilled IT admin staff is available either internally or externally, following SBoM
requirements.

AI Security and Privacy Training□ Actively engage with employees to understand and address concerns with planned LLM
initiatives.
□ Establish a culture of open, and transparent communication on the organization’s use ofpredictive or generative AI within the organization process, systems, employee management
and support, and customer engagements and how its use is governed, managed, and risks
addressed.
□ Train all users on ethics, responsibility, and legal issues such as warranty, license, and copyright.
□ Update security awareness training to include GenAI related threats. Voice cloning and image
cloning, as well as in anticipation of increased spear phishing attacks□ Any adopted GenAI solutions should include training for both DevOps and cybersecurity forthe deployment pipeline to ensure AI safety and security assurances.

Establish Business Cases
Solid business cases are essential to determining the business value of any proposed AI solution,balancing risk and beneﬁts, and evaluating and testing return on investment. There are an enormous
number of potential use cases; a few examples are provided.
□ Enhance customer experience□ Better operational efﬁciency□ Better knowledge management□ Enhanced innovation□ Market Research and Competitor Analysis□ Document creation, translation, summarization, and analysis

GovernanceCorporate governance in LLM is needed to provide organizations with transparency and accountability.
Identifying AI platform or process owners who are potentially familiar with the technology or theselected use cases for the business is not only advised but also necessary to ensure adequate
reaction speed that prevents collateral damages to well established enterprise digital processes.
□ Establish the organization’s AI RACI chart (who is responsible, who is accountable, who should
be consulted, and who should be informed)□ Document and assign AI risk, risk assessments, and governance responsibility within theorganization.
□ Establish data management policies, including technical enforcement, regarding dataclassiﬁcation and usage limitations. Models should only leverage data classiﬁed for theminimum access level of any user of the system. For example, update the data protection
policy to emphasize not to input protected or conﬁdential data into nonbusiness-managed
tools.
□ Create an AI Policy supported by established policy (e.g., standard of good conduct, dataprotection, software use)
□ Publish an acceptable use matrix for various generative AI tools for employees to use.
□ Document the sources and management of any data that the organization uses from thegenerative LLM models.

LegalMany of the legal implications of AI are undeﬁned and potentially very costly. An IT, security, andlegal partnership is critical to identifying gaps and addressing obscure decisions.
□ Conﬁrm product warranties are clear in the product development stream to assign who is
responsible for product warranties with AI.
□ Review and update existing terms and conditions for any GenAI considerations.
□ Review AI EULA agreements. End-user license agreements for GenAI platforms are very
different in how they handle user prompts, output rights and ownership, data privacy,
compliance, liability, privacy, and limits on how output can be used.
□ Organizations EULA for customers, Modify end-user agreements to prevent the organizationfrom incurring liabilities related to plagiarism, bias propagation, or intellectual property
infringement through AI-generated content.
□ Review existing AI-assisted tools used for code development. A chatbot’s ability to write code
can threaten a company’s ownership rights to its product if a chatbot is used to generate
code for the product. For example, it could call into question the status and protection of thegenerated content and who holds the right to use the generated content.
□ Review any risks to intellectual property. Intellectual property generated by a chatbot could
be in jeopardy if improperly obtained data was used during the generative process, which is
subject to copyright, trademark, or patent protection. If AI products use infringing material, it
creates a risk for the outputs of the AI, which may result in intellectual property infringement.
□ Review any contracts with indemniﬁcation provisions. Indemniﬁcation clauses try to put the
responsibility for an event that leads to liability on the person who was more at fault for it or
who had the best chance of stopping it. Establish guardrails to determine whether the providerof the AI or its user caused the event, giving rise to liability.
□ Review liability for potential injury and property damage caused by AI systems.
□ Review insurance coverage. Traditional (D&O) liability and commercial general liability
insurance policies are likely insufﬁcient to fully protect AI use.
□ Identify any copyright issues. Human authorship is required for copyright. An organization
may also be liable for plagiarism, propagation of bias, or intellectual property infringement if
LLM tools are misused.
□ Ensure agreements are in place for contractors and appropriate use of AI for any development
or provided services.
□ Restrict or prohibit the use of generative AI tools for employees or contractors where
enforceable rights may be an issue or where there are IP infringement concerns.
□ Assess and AI solutions used for employee management or hiring could result in disparate
treatment claims or disparate impact claims.
□ Make sure the AI solutions do not collect or share sensitive information without proper consent
or authorization.

Regulatory
The EU AI Act is anticipated to be the ﬁrst comprehensive AI law but will apply in 2025 at theearliest. The EUś General Data Protection Regulation (GDPR) does not speciﬁcally address AI but
includes rules for data collection, data security, fairness and transparency, accuracy and reliability,
and accountability, which can impact GenAI use. In the United States, AI regulation is included within
broader consumer privacy laws. Ten US states have passed laws or have laws that will go into effectby the end of 2023.
Federal organizations such as the US Equal Employment Opportunity Commission (EEOC), theConsumer Financial Protection Bureau (CFPB), the Federal Trade Commission (FTC), and the US
Department of Justiceś Civil Rights Division (DOJ) are closely monitoring hiring fairness.
□ Determine Country, State, or other Government speciﬁc AI compliance requirements.
□ Determine compliance requirements for restricting electronic monitoring of employees andemployment-related automated decision systems (Vermont, California, Maryland, New York,New Jersey)□ Determine compliance requirements for consent for facial recognition and the AI video analysis
required (Illinois, Maryland, Washington, Vermont)□ Review any AI tools in use or being considered for employee hiring or management.
□ Conﬁrm the vendorś compliance with applicable AI laws and best practices.
□ Ask and document any products using AI during the hiring process. Ask how the model wastrained, and how it is monitored, and track any corrections made to avoid discrimination andbias.
□ Ask and document what accommodation options are included.
□ Ask and document whether the vendor collects conﬁdential data.
□ Ask how the vendor or tool stores and deletes data and regulates the use of facial recognitionand video analysis tools during pre-employment.
□ Review other organization-speciﬁc regulatory requirements with AI that may raise complianceissues. The Employee Retirement Income Security Act of 1974, for instance, has ﬁduciary duty
requirements for retirement plans that a chatbot might not be able to meet.

Using or Implementing Large Language Model Solutions□ Threat Model LLM components and architecture trust boundaries.
□ Data Security, verify how data is classiﬁed and protected based on sensitivity, including
personal and proprietary business data. (How are user permissions managed, and what
safeguards are in place?)□ Access Control, implement least privilege access controls and implement defense-in-depth
measures□ Training Pipeline Security, require rigorous control around training data governance, pipelines,
models, and algorithms.
□ Input and Output Security, evaluate input validation methods, as well as how outputs are
ﬁltered, sanitized, and approved.
□ Monitoring and Response, map workﬂows, monitoring, and responses to understandautomation, logging, and auditing. Conﬁrm audit records are secure.
□ Include application testing, source code review, vulnerability assessments, and red teaming in
the production release process.
□ Check for existing vulnerabilities in the LLM model or supply chain.
□ Look into the effects of threats and attacks on LLM solutions, such as prompt injection, therelease of sensitive information, and process manipulation.
□ Investigate the impact of attacks and threats to LLM models, including model poisoning,
improper data handling, supply chain attacks, and model theft.
□ Supply Chain Security, request third-party audits, penetration testing, and code reviews forthird-party providers. (both initially and on an ongoing basis)□ Infrastructure Security, ask how often a vendor performs resilience testing? What are their
SLAs in terms of availability, scalability, and performance?
□ Update incident response playbooks and include an LLM incident in tabletop exercises.
□ Identify or expand metrics to benchmark generative cybersecurity AI against other approaches
to measure expected productivity improvements.

Testing, Evaluation, Veriﬁcation, and Validation (TEVV)NIST AI Framework recommends a continuous TEVV process throughout the AI lifecycle which
includes the AI system operators, domain experts, AI designers, users, product developers, evaluators,
and auditors. TEVV includes a range of tasks such as system validation, integration, testing,recalibration, and ongoing monitoring for periodic updates to navigate the risks and changes of the
AI system.
□ Establish continuous testing, evaluation, veriﬁcation, and validation throughout the AI model
lifecycle.
□ Provide regular executive metrics and updates on AI Model functionality, security, reliability,
and robustness.

Model Cards and Risk Cards
Model cards and risk cards are foundational elements for increasing the transparency, accountability,and ethical deployment of Large Language Models (LLMs). Model cards help users understandand trust AI systems by providing standardized documentation on their design, capabilities, andconstraints, leading them to make educated and safe applications. Risk cards supplement this by
openly addressing potential negative consequences, such as biases, privacy problems, and security
vulnerabilities, which encourages a proactive approach to harm prevention. These documents are
critical for developers, users, regulators, and ethicists equally since they establish a collaborative
atmosphere in which AIś social implications are carefully addressed and handled. These cards,
developed and maintained by the organizations that created the models, play an important role inensuring that AI technologies fulﬁll ethical standards and legal requirements, allowing for responsible
research and deployment in the AI ecosystem.
Model cards include key attributes associated with the ML model:• Model details: Basic information about the model, i.e., name, version, and type (neural network,
decision tree, etc.), and the intended use case.

• Model architecture: Includes a description of the structure of the model, such as the number
and type of layers, activation functions, and other key architectural choices.

• Training data and methodology: Information about the data used to train the model, suchas the size of the dataset, the data sources, and any preprocessing or data augmentation
techniques used. It also includes details about the training methodology, such as the optimizer
used, the loss function, and any hyperparameters that were tuned.

• Performance metrics: Information about the model’s performance on various metrics, suchas accuracy, precision, recall, and F1 score. It may also include information about how themodel performs on different subsets of the data.

• Potential biases and limitations: Lists potential biases or limitations of the model, such asimbalanced training data, overﬁtting, or biases in the model’s predictions. It may also include
information about the modelś limitations, such as its ability to generalize to new data or its
suitability for certain use cases.

• Responsible AI considerations: Any ethical or responsible AI considerations related to themodel, such as privacy concerns, fairness, and transparency, or potential societal impactsof the model’s use. It may also include recommendations for further testing, validation, ormonitoring of the model.

The precise features contained in a model card may differ based on the model’s context and intended
usage, but the purpose is to give openness and accountability in the creation and deployment ofmachine learning models.
□ Review a models model card
□ Review risk card if available□ Establish a process to track and maintain model cards for any deployed model including
models used through a third party.

RAG: Large Language Model Optimization
Fine tuning, the traditional method for optimizing a pre-trained model, involved retraining an existingmodel on new, and domain-speciﬁc data, modifying it for performance on a task or application.
Fine-tuning is expensive but essential to improve performance.
Retrieval-Augmented Generation (RAG) has evolved as a more effective way of optimizing andaugmenting the capabilities of large language models by retrieving pertinent data from up to dateavailable knowledge sources. RAG can be customized for speciﬁc domains, optimizing the retrievalof domain-speciﬁc information and tailoring the generation process to the nuances of specializedﬁelds. RAG is seen as a more efﬁcient and transparent method for LLM optimization, particularly for
problems where labeled data is limited or expensive to collect. One of the primary advantages of
RAG is its support for continuous learning since new information can be continually updated at the
retrieval stage.
The RAG implementation involves several key steps starting from embedding model deployment,
indexing the knowledge library, to retrieving the most relevant documents for query processing.
Efﬁcient retrieval of the relevant context is made based on vector databases which are used forstorage and querying of document embeddings.
RAG Reference□ Retrieval Augmented Generation (RAG) & LLM: Examples□ 12 RAG Pain Points and Proposed Solutions

AI Red Teaming
AI Red Teaming is an adversarial attack test simulation of the AI System to validate there arent́ any
existing vulnerabilities which can be exploited by an attacker. It is a recommended practice by many
regulatory and AI governing bodies including the Biden administration. Red-teaming alone is not
a comprehensive solution to validate all real-world harms associated with AI systems and should
be included with other forms of testing, evaluation, veriﬁcation, and validation such as algorithmic
impact assessments and external audits.
□ Incorporate Red Team testing as a standard practice for AI Models and applications.

Resources
OWASP Top 10 for Large Language Model Applications

Figure 4.1: Image of OWASP Top 10 for Large Language Model Applications

OWASP Top 10 for Large Language Model Applications Visualized

Figure 4.2: Image of OWASP Top 10 for Large Language Model Applications VisualizedOWASP Resources Using LLM solutions expands an organization’s attack surface and presents newchallenges, requiring special tactics and defenses. It also poses problems that are similar to known
issues, and there are already established cybersecurity procedures and mitigations. Integrating LLM
cybersecurity with an organization’s established cybersecurity controls, processes, and procedures
allows an organization to reduce its vulnerability to threats. How they integrate with each other isavailable at the OWASP Integration Standards.
OWASP Resource
OWASP SAMMOWASP AI Security andPrivacy GuideOWASP AI Exchange

Description

Why It Is Recommended & WhereTo Use It
SoftwareAssurance Provides an effective andMaturity Modelmeasurable way to analyze andimprove an organization’s securedevelopment lifecycle.
SAMM
supports the complete software
lifecycle.
It is interative andrisk-driven, enabling organizations
to identify and prioritize gaps insecure software development
so resources for improving
the process can be dedicatedwhere efforts have the greatest
improvement impact.
OWASP Project with a The OWASP AI Security and Privacy
goal of connecting Guide is a comprehensive list ofworldwide
foran the most important AI security andexchange on AI security, privacy considerations. It is meant
fosteringstandards to be a comprehensive resource foralignment, and driving developers, security researchers,
collaboration.
and security consultants to verify
the security and privacy of AI
systems.
OWASP AI Exchange is The AI Exchange is the primary
the intake method for the intake method used by OWASP to
OWASP AI Security and drive the direction of the OWASP AI
Privacy Guide.
Security and Privacy Guide.

OWASP Resource

Description

OWASP
Learning
Top 10OWASPMachineLearning
Security
Top 10 security issuesof machine learning
systems.

MachineSecurity

OpenCREOWASP Threat Modeling

OWASP CycloneDXOpenCRE(CommonRequirement
Enumeration)istheinteractive
content-linking platform
for uniting security
standards and guidelines
into one overview.
A structured, formal
processforthreatmodelingofanapplication
OWASP
CycloneDX
is a full-stack Billof Materials (BOM)
standard that provides
advanced supply chain
capabilities for cyber risk
reduction.

Why It Is Recommended & WhereTo Use ItThe
OWASP
MachineLearning Security Top 10 is acommunity-driven effort to collectand present the most important
security issues of machine learning
systems in a format that is easy
to understand by both a security
expert and a data scientist. This
project includes the ML Top 10
and is a live working documentthat provides clear and actionableinsights on designing, creating,testing, and procuring secure andprivacy-preserving AI systems. It
is the best OWASP resource for
AI global regulatory and privacy
information.
Use this site to search forstandards. You can search bystandard name or by control type.

Learn everything about Threat
Modeling which is a structured
representation
of
all
the
information that affects the
security of an application.
Modern software is assembled
using third-party and open sourcecomponents.
They are glued
together in complex and uniqueways and integrated with originalcode to achieve the desired
functionality. An SBOM provides
an accurate inventory of allcomponents
whichenablesorganizations to identify risk,
allows for greater transparency,
and enables rapid impact analysis.
EO 14028 provided minimum
requirements for SBOM for federal
systems.

OWASP Resource

Description

OWASP
Software
Component Veriﬁcation
Standard (SCVS)

A
community-driven
effort to establish aframework for identifying
activities, controls, andbest practices can help in
identifying and reducing
risk in a software supply
chain.
API Security focusesonstrategies
andsolutions to understandandmitigatetheunique vulnerabilities
andsecurityrisksofApplication
Programming Interfaces(APIs)
Application
Security
Veriﬁcation
Standard(ASVS) Project provides
a basis for testing web
application
technical
security
controls
andalso
providesdevelopers with a list ofrequirements for securedevelopment.
An action oriented viewto safeguard and enable
the business

OWASP
Project

API

Security

OWASP
Application
Security
Veriﬁcation
Standard ASVSOWASP Threat andSafeguard
Matrix(TaSM)

Defect Dojo

Why It Is Recommended & WhereTo Use It
Use SCVS to develop a common
set of activities, controls, andbest-practices that can reducerisk in a software supply chain
and identify a baseline and pathto mature software supply chain
vigilance.
APIs are a foundational elementof connecting applications, andmitigating misconﬁgurations orvulnerabilities is mandatory toprotect users and organizations.
Use for security testing and red
teaming the build and production
environments.
Cookbook for web application
security requirements, security
testing, and metrics.
Use toestablish security user stories andsecurity use case release testing.

This matrix allows a company tooverlay its major threats with the
NIST Cyber Security Framework
Functions (Identify, Protect, Detect,
Respond, & Recover) to build arobust security plan. Use it as adashboard to track and report on
security across the organization.
Use Defect Dojo to reduce thetime for logging vulnerabilities
with templates for vulnerabilities,
imports for common vulnerability
scanners, report generation, andmetrics.

An
opensourcevulnerability
management tool thatstreamlines the testingprocess by offering
templating,reportgeneration, metrics, andbaseline
self-service
tools.
Table 4.1: OWASP Resources

MITRE Resources The increased frequency of LLM threats emphasizes the value of a resilience-ﬁrst
approach to defending an organization’s attack surface. Existing TTPS are combined with new
attack surfaces and capabilities in LLM Adversary threats and mitigations. MITRE maintains awell-established and widely accepted mechanism for coordinating opponent tactics and procedures
based on real-world observations.
Coordination and mapping of an organization’s LLM Security Strategy to MITRE ATT&CK and MITREATLAS allows an organization to determine where LLM Security is covered by current processes
such as API Security Standards or where security holes exists.
MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) is a framework, collectionof data matrices, and assessment tool that was made by the MITRE Corporation to help organizations
ﬁgure out how well their cybersecurity works across their entire digital attack surface and ﬁnd holesthat had not been found before. It is a knowledge repository that is used all over the world. The
MITRE ATT&CK matrix contains a collection of strategies used by adversaries to achieve a certain
goal. In the ATT&CK Matrix, these objectives are classiﬁed as tactics. The objectives are outlined inattack order, beginning with reconnaissance and progressing to the eventual goal of exﬁltration or
impact.
MITRE ATLAS, which stands for "Adversarial Threat Landscape for Artiﬁcial Intelligence Systems," isa knowledge base that is based on real-life examples of attacks on machine learning (ML) systems
by bad actors. ATLAS is based on the MITRE ATT&CK architecture, and its tactics and procedures
complement those found in ATT&CK.
MITRE Resource

Description

MITRE ATT&CK

Knowledge base ofadversary tactics andtechniques based onreal-world observations

MITRE
Workbench

AT&CK

Create or extend ATT&CK
data in a local knowledge
base

Why It Is Recommended & WhereTo Use It
The ATT&CK knowledge baseis used as a foundation for thedevelopment of speciﬁc threat
modelsandmethodologies.
Map existing controls within theorganization to adversary tactics
and techniques to identify gaps or
areas to test.
Host and manage a customized
copy of the ATT&CK knowledge
base. This local copy of the
ATT&CK knowledge base can beextended with new or updated
techniques, tactics, mitigationgroups, and software that is
speciﬁc to your organization.

MITRE Resource
MITRE ATLAS

Description

Why It Is Recommended & WhereTo Use It
Use it to map known ML
vulnerabilities and map checks
and controls for proposed projects
or existing systems.

MITRE
ATLAS
(Adversarial
ThreatLandscapefor
Artiﬁcial-Intelligence
Systems) is a knowledge
baseofadversary
tactics,
techniques,
and case studies formachine learning (ML)systemsbasedonreal-world observations,demonstrations from ML
red teams and security
groups, and the stateof the possible fromacademic research
MITRE ATT&CK Powered ATT&CK Powered Suit is Add to your browser to quickly
Suita browser extension that search for tactics, techniques,
puts the MITRE ATT&CK and more without disrupting yourknowledge base at your workﬂow.
ﬁngertips.
The
ThreatReport Automates
TTP Mapping TTPs found in CTI reports
ATT&CK Mapper (TRAM) Identiﬁcation in CTI to MITRE ATT&CK is difﬁcult,
Reports
error prone, and time-consuming.
TRAM uses LLMs to automate thisprocess for the 50 most common
techniques.
Supports Juypter
notebooks.
Attack Flow v2.1.0
Attack Flow is a Attack Flow helps visualize how
language for describing an attacker uses a technique, so
how cyber adversaries defenders and leaders understandcombine and sequence how adversaries operate andvariousoffensive improve their own defensive
techniques to achieve posture.
their goals.
MITRE Caldera
A cyber security platform Plugins are available for Caldera(framework) designed that help to expand the coreto easily automate capabilities of the framework andadversaryemulation, provide additional functionality,
assist manual red-teams, including
agents,reporting,and automate incident collections of TTPs and others
response.
CALDERAplugin: A plugin developed for This plugin provides TTPs deﬁned
Arsenaladversary emulation of in MITRE ATLAS to interface with
AI-enabled systems.
CALDERA.

MITRE Resource
Atomic Red Team

MITRE CTI Blueprints

Description

Why It Is Recommended & WhereTo Use It
Library of tests mapped Use to validate and test controls
to the MITRE ATT&CK in an environment. Security teams
framework.
can use Atomic Red Team toquickly, portably, and reproducibly
test their environments. You can
execute atomic tests directly fromthe command line; no installationis required.
Automates Cyber Threat CTI Blueprints helps Cyber ThreatIntelligence reporting.
Intelligence (CTI) analysts createhigh-quality, actionable reports
more consistently and efﬁciently.
Table 4.2: MITRE Resources

AI Vulnerability RepositoriesName
AI Incident Database

Description
A repository of articles about different times AI has
failed in real-world applications and is maintained by acollege research group and crowds sourced.
OECD AI Incidents Monitor (AIM)
Offers an accessible starting point for comprehending
the landscape of AI-related challenges.
Three of the leading companies tracking AI Model vulnerabilities
Huntr Bug Bounty : ProtectAI
Bug bounty platform for AI/ML
AI Vulnerability Database (AVID) : Garak
Database of model vulnerabilities
AI Risk Database: Robust Intelligence
Database of model vulnerabilities
Table 4.3: AI Vulnerability Repositories

AI Procurement Guidance
NameWorld Economic Forum: Adopting AI
Responsibly: Guidelines for Procurement of
AI Solutions by the Private Sector: Insight
Report June 2023

Description
The standard benchmarks and assessment criteria for
procuring Artiﬁcial systems are in early development.
The procurement guidelines provide organizations
with a baseline of considerations for the end-to-end
procurement process.
Use this guidance to augment an organization’s existing
Third Party Risk Supplier and Vendor procurement
process.
Table 4.4: AI Procurement Guidance

Team
Thank you to the OWASP Top 10 for LLM Applications Cybersecurity and Governance Checklist
Contributors.
Checklist Contributors
Sandy Dunn
Heather Linn
John Sotiropoulos
Steve Wilson
Fabrizio Cilli
Aubrey King
Bob Simonoff
David Rowe
Rob Vanderveer
Emmanual Guilherme JuniorAndrea Succi
Jason Ross
Talesh Seeparsan
Anthony Glynn
Julie Tao
Table A.1: OWASP LLM AI Security & Governance Checklist
Team

This project is licensed under the terms of the Creative Commons Attribution-ShareAlike 4.0
International License