ezesecops/rmf-ato-core huggingface dataset:
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
Section 4.1.a of Executive Order (EO) 14110, Safe, Secure, and Trustworthy Development and

Section 4.1.a of Executive Order (EO) 14110, Safe, Secure, and Trustworthy Development and

Use of Artificial Intelligence [1], tasked NIST with "developing a companion resource to the
Secure Software Development Framework to incorporate secure development practices for
generative AI and for dual-use foundation models." This document is that companion resource.
The software development and use of AI models and AI systems inherit much of the same risk
as any other digital system. A unique challenge for this community is the blurring of traditional
boundaries between system code and system data, as well as the use of plain human language
as the means of interaction with the systems. AI models and systems, their configuration
parameters (e.g., model weights), and the data they interact with (e.g., training data, user
queries, etc.) can form closed loops that can be manipulated for unintended functionality.
AI model and system development is still much more of an art than an exact science, requiring
developers to interact with model code, training data, and other parameters over multiple
iterations. Training datasets may be acquired from unknown, untrusted sources. Model weights
and other training parameters can be susceptible to malicious tampering. Some models may be
complex to the point that they cannot easily be thoroughly inspected, potentially allowing for
undetectable execution of arbitrary code. User queries can be crafted to produce undesirable
or objectionable output and - if not sanitized properly - can be leveraged for injection-style
attacks. The goal of this document is to identify the practices and tasks needed to address these
novel risks.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.1 > PO.1.1

PO.1.1: Identify and document all security requirements for the organization's software development infrastructures and processes, and maintain the requirements over time. High R1: Include AI model development in the security requirements for software development infrastructure and processes. R2: Identify and select appropriate AI model architectures and training techniques in accordance with recommended practices for cybersecurity, privacy, and reproducibility. AI RMF: Map 1.3, 1.5, 1.6
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.1 > PO.1.2

PO.1.2: Identify and document all security requirements for organization-developed software to meet, and maintain the requirements over time. High R1: Organizational policies should support all current requirements specific to AI model development security for organization-developed software. These requirements should include the areas of AI model development, AI model operations, and data science. Requirements may come from many sources, including laws, regulations, contracts, and standards. C1: Consider reusing or expanding the organization's existing data classification policy and processes. N1: Possible forms of AI model documentation include data, model, and system cards. AI RMF: Govern 1.1, 1.2, 3.2, 4.1, 5.1, 6.1; Map 1.1
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.1 > PO.1.3

PO.1.3: Communicate requirements to all third parties who will provide commercial software components to the organization for use by the organization's own software. [Modified from SSDF 1.1] Medium R1: Include AI model development security in the requirements being communicated for third-party software components. AI RMF: Map 4.1, 4.2 OWASP: LLM05-1 Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References Implement Roles and Responsibilities (PO.2): Ensure that everyone inside and outside of the organization involved in the SDLC is prepared to perform their SDLC-related roles and responsibilities throughout the SDLC.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.2 > PO.2.1

PO.2.1: Create new roles and alter responsibilities for existing roles as needed to encompass all parts of the SDLC. Periodically review and maintain the defined roles and responsibilities, updating them as needed. High R1: Include AI model development security in SDLC-related roles and responsibilities throughout the SDLC. The roles and responsibilities should include, but are not limited to, AI model development, AI model operations, and data science. N1: Roles and responsibilities involving AI system producers, AI model producers, and other third-party providers can be documented in agreements. AI RMF: Govern 2.1
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.2 > PO.2.2

PO.2.2: Provide role-based training for all personnel with responsibilities that contribute to secure development. Periodically review personnel proficiency and role-based training, and update the training as needed. High R1: Role-based training should include understanding cybersecurity vulnerabilities and threats to AI models and their possible mitigations. AI RMF: Govern 2.2 OWASP: LLM04-7
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.2 > PO.2.3

PO.2.3: Obtain upper management or authorizing official commitment to secure development, and convey that commitment to all with developmentrelated roles and responsibilities. Medium R1: Leadership should commit to secure development practices involving AI models. AI RMF: Govern 2.3 Implement Supporting Toolchains (PO.3): Use automation to reduce human effort and improve the accuracy, reproducibility, usability, and comprehensiveness of security practices throughout the SDLC, as well as provide a way to document and demonstrate the use of these practices. Toolchains and tools may be used at different levels of the organization, such as organization-wide or project-specific, and may address a particular part of the SDLC, like a build pipeline.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.3 > PO.3.1

PO.3.1: Specify which tools or tool types must or should be included in each toolchain to mitigate identified risks, as well as how the toolchain components are to be integrated with each other. High R1: Plan to develop and implement automated toolchains that secure AI model development and reduce human effort, especially at the scale often used by AI models. N1: Ideally, automated toolchains will perform the vast majority of the work related to securing AI model development. N2: See PO.4, PO.5, PS, and PW for information on tool types. AI RMF: Measure 2.1 OWASP: LLM08
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.3 > PO.3.2

PO.3.2: Follow recommended security practices to deploy, operate, and maintain tools and toolchains. High R1: Execute the plan to develop and implement automated toolchains that secure AI model development and reduce human effort, especially at the scale often used by AI models. R2: Verify the security of toolchains at a frequency commensurate with risk. AI RMF: Measure 2.1 OWASP: LLM05-3, LLM05-9, LLM08, LLM09 Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.3 > PO.3.3

PO.3.3: Configure tools to generate artifacts of their support of secure software development practices as defined by the organization. Medium N1: An artifact is "a piece of evidence" [16]. Evidence is "grounds for belief or disbelief; data on which to base proof or to establish truth or falsehood" [17]. Artifacts provide records of secure software development practices. Examples of artifacts specific to AI model development include attestations of the integrity and provenance of training datasets. AI RMF: Measure 2.1 Define and Use Criteria for Software Security Checks (PO.4): Help ensure that the software resulting from the SDLC meets the organization's expectations by defining and using criteria for checking the software's security during development.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.4 > PO.4.1

PO.4.1: Define criteria for software security checks and track throughout the SDLC. Medium R1: Implement guardrails and other controls throughout the AI development life cycle, extending beyond the traditional SDLC. C1: Consider requiring review and approval from a human-in-the-loop for software security checks beyond risk-based thresholds. AI RMF: Measure 2.3, 2.7; Manage 1.1 OWASP: LLM01-2
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.4 > PO.4.2

PO.4.2: Implement processes, mechanisms, etc. to gather and safeguard the necessary information in support of the criteria. Low No additions to SSDF 1.1 AI RMF: Measure 2.3, 2.7; Manage 1.1 OWASP: LLM01-2 Implement and Maintain Secure Environments for Software Development (PO.5): Ensure that all components of the environments for software development are strongly protected from internal and external threats to prevent compromises of the environments or the software being developed or maintained within them. Examples of environments for software development include development, AI model training, build, test, and distribution environments. [Modified from SSDF 1.1]
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.5 > PO.5.1

PO.5.1: Separate and protect each environment involved in software development. High C1: Consider separating execution environments from each other to the extent feasible, such as through isolation, segmentation, containment, access via APIs, or other means. R1: Monitor, track, and limit resource usage and rates for AI model users during model development. R2: Only store sensitive data used during AI model development, including production data, within organization-approved environments and locations within those environments. R3: Protect all training pipelines, model registries, and other components within the environments according to the principle of least privilege. OWASP: LLM01-1, LLM01-4, LLM04, LLM08, Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References R4: Continuously monitor training-related activity in pipelines and model modifications in the model registry. R5: Follow recommended practices for securely configuring each environment. R6: Continuously monitor each environment for plaintext secrets.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.5 > PO.5.2

PO.5.2: Secure and harden development endpoints (endpoints for software designers, developers, testers, builders, etc.) to perform development tasks using a risk-based approach. Medium No additions to SSDF 1.1 OWASP: LLM01-1, LLM05-3, LLM05-9,
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PO > PO.5 > PO.5.3

PO.5.3: Continuously monitor software execution performance and behavior in software development environments to identify potential suspicious activity and other issues. [Not part of SSDF 1.1] High R1: Perform continuous security monitoring for all development environment components that host an AI model or related resources (e.g., model APIs, weights, configuration parameters, training datasets). R2: Continuous monitoring and analysis tools should generate alerts when detected activity involving an AI model passes a risk threshold or otherwise merits additional investigation. AI RMF: Measure 2.4 OWASP: LLM03-7, LLM04, LLM05- 8, LLM09, Protect Software (PS) Protect All Forms of Code and Data from Unauthorized Access and Tampering (PS.1): Help prevent unauthorized changes to code and data, both inadvertent and intentional, which could circumvent or negate the intended security characteristics of the software. For code and data that are not intended to be publicly accessible, this helps prevent theft of the software and may make it more difficult or time-consuming for attackers to find vulnerabilities in the software. [Modified from SSDF 1.1]
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.1 > PS.1.1

PS.1.1: Store all forms of code - including source code, executable code, and configuration-as-code - based on the principle of least privilege so that only authorized personnel, tools, services, etc. have access. High R1: Secure code storage should include AI models, model weights, pipelines, reward models, and any other AI model elements that need their confidentiality, integrity, and/or availability protected. These elements do not all have to be stored in the same place or through the same type of mechanism. R2: Follow the principle of least privilege to minimize direct access to AI models and model elements regardless of where they are stored or executed. R3: Store reward models separately from AI models and data. OWASP: LLM10 Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References R4: Permit indirect access only to model weights. C1: Consider preventing all human access to model weights. C2: Consider requiring all AI model development to be performed within organization-approved environments only.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.1 > PS.1.2

PS.1.2: Protect all training, testing, fine-tuning, and aligning data from unauthorized access and modification. [Not part of SSDF 1.1] High R1: Continuously monitor the confidentiality (for non-public data only) and integrity of training, testing, fine-tuning, and aligning data. C1: Consider securely storing training, testing, fine-tuning, and aligning data for future use and reference if feasible. OWASP: LLM03, LLM06,
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.1 > PS.1.3

PS.1.3: Protect all model weights and configuration parameter data from unauthorized access and modification. [Not part of SSDF 1.1] High R1: Keep model weights and configuration parameters separate from training, testing, fine-tuning, and aligning data. R2: Continuously monitor the confidentiality (for closed models only) and integrity of model weights and configuration parameters. R3: Follow the principle of least privilege to restrict access to AI model weights, configuration parameters, and services during development. R4: Specify and implement additional riskproportionate cybersecurity practices around model weights, such as encryption, cryptographic hashes, digital signatures, multiparty authorization, and air-gapped environments. OWASP: LLM10 Provide a Mechanism for Verifying Software Release Integrity (PS.2): Help software acquirers ensure that the software they acquire is legitimate and has not been tampered with.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.2 > PS.2.1

PS.2.1: Make software integrity verification information available to software acquirers. Medium R1: Generate and provide cryptographic hashes or digital signatures for an AI model and its components, artifacts, and documentation. R2: Provide digital signatures for AI model changes. OWASP: LLM05-6 Archive and Protect Each Software Release (PS.3): Preserve software releases in order to
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.3 > PS.3.1

PS.3.1: Securely archive the necessary files and supporting data (e.g., integrity Low R1: Perform versioning and tracking for infrastructure tools (e.g., pre-processing, OWASP: LLM10 Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References help identify, analyze, and eliminate vulnerabilities discovered in the software after release. verification information, provenance data) to be retained for each software release. transforms, collection) that support dataset creation and model training. R2: Include documentation of the justification for AI model selection in the retained information. R3: Include documentation of the entire training process, such as data preprocessing and model architecture. N1: AI models and their components may need to be added at this time to an organization's asset inventories.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PS > PS.3 > PS.3.2

PS.3.2: Collect, safeguard, maintain, and share provenance data for all components of each software release (e.g., in a software bill of materials [SBOM], through Supply-chain Levels for Software Artifacts [SLSA]). [Modified from SSDF 1.1] Medium R1: Track the provenance of an AI model and its components and derivatives, including the training libraries, frameworks, and pipelines used to build the model. R2: Track AI models that were trained on sensitive data (e.g., payment card data, protected health information, other types of personally identifiable information), and determine if access to the models should be restricted to individuals who already have access to the sensitive data used for training. C1: Consider disclosing the provenance of the training, testing, fine-tuning, and aligning data used for an AI model. OWASP: LLM03-1, LLM05-4, LLM05-5, Produce Well-Secured Software (PW) Design Software to Meet Security Requirements and Mitigate Security Risks (PW.1): Identify and evaluate the security requirements for the software; determine what security risks the software is likely to face during operation and how the software's design and architecture should mitigate those risks; and justify any cases where risk-based analysis indicates that security requirements should be relaxed or waived. Addressing
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.1 > PW.1.1

PW.1.1: Use forms of risk modeling - such as threat modeling, attack modeling, or attack surface mapping - to help assess the security risk for the software. High R1: Incorporate relevant AI model-specific vulnerability and threat types in risk modeling. Examples of these vulnerability and threat types include poisoning of training data, malicious code or other unwanted content in inputs and outputs, denial-of-service conditions arising from adversarial prompts, supply chain attacks, unauthorized information disclosure, theft of AI model weights, and misconfiguration of data pipelines. [3] AI RMF: Govern 4.1, 4.2; Map 5.1; Measure 1.1; Manage 1.2, 1.3 OWASP: LLM01, LLM02, LLM03, LLM04, LLM05, LLM06, Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References security requirements and risks during software design (secure by design) is key for improving software security and also helps improve development efficiency. C1: Consider periodic risk modeling updates for future AI model versions and derivatives after AI model release. C2: During risk modeling, consider checking that the AI model is not in a critical path to make significant security decisions without a human in the loop. LLM07, LLM08, LLM09, LLM10
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.1 > PW.1.2

PW.1.2: Track and maintain the software's security requirements, risks, and design decisions. Medium No additions to SSDF 1.1 AI RMF: Govern 4.1, 4.2; Map 2.1, 2.2, 2.3, 3.2, 3.3, 4.1, 4.2, 5.2; Manage 1.2, 1.3, 1.4
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.1 > PW.1.3

PW.1.3: Where appropriate, build in support for using standardized security features and services (e.g., enabling software to integrate with existing log management, identity management, access control, and vulnerability management systems) instead of creating proprietary implementations of security features and services. Medium No additions to SSDF 1.1 Review the Software Design to Verify Compliance with Security Requirements and Risk Information (PW.2): Help ensure that the software will meet the security requirements and satisfactorily address the identified risk information.
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.2 > PW.2.1

PW.2.1: Have 1) a qualified person (or people) who were not involved with the design and 2) automated processes instantiated in the toolchain review the software design to confirm and enforce that it meets all of the security requirements and satisfactorily addresses the identified risk information. [Modified from SSDF 1.1] High No additions to SSDF 1.1 AI RMF: Measure 2.7; Manage 1.1 Confirm the Integrity of Training, Testing, Fine-Tuning, and Aligning Data Before Use (PW.3): Prevent data that is likely to negatively impact the cybersecurity of the AI model from
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.3 > PW.3.1

PW.3.1: Analyze data for signs of data poisoning, bias, homogeneity, and tampering before using it for AI model training, testing, fine-tuning, or aligning High R1: Verify the provenance (when known) and integrity of training, testing, fine-tuning, and aligning data before use. AI RMF: Measure 2.1; Manage 1.2, 1.3 Practice Task Priority Recommendations [R], Considerations [C], and Notes [N] Specific to AI Model Development Informative References being consumed as part of AI model training, testing, fine-tuning, and aligning. [Not part of SSDF 1.1] purposes, and mitigate the risks as necessary. [Not part of SSDF 1.1] R2: Select and apply appropriate methods for analyzing and altering the training, testing, fine-tuning, and aligning data for an AI model. Examples of methods include anomaly detection, bias detection, data cleaning, data curation, data filtering, data sanitization, factchecking, and noise reduction. C1: Consider using a human-in-the-loop to examine data, such as with exploratory data analysis techniques [18]. OWASP: LLM03, LLM06
Secure Software Development Practices for Generative AI and Dual-Use Foundation Models: An SSDF Community Profile
SSDF Practices > PW > PW.3 > PW.3.2

PW.3.2: Track the provenance, when known, of all training, testing, fine-tuning, and aligning data used for an AI model, and document which data do not have known provenance. [Not part of SSDF 1.1] Medium N1: Provenance verification is not possible in all cases because provenance is not always known. However, it is still beneficial for security purposes to track and verify provenance whenever possible, and to track when provenance is unknown. AI RMF: Measure 2.1 OWASP: LLM03-1 Adv ML