1NIST AI 800-1

2

Initial Public Draft

4

Managing Misuse Risk for Dual-Use
Foundation Models

5
6

U.S. AI Safety Institute

3

7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

This publication is available free of charge from:https://doi.org/10.6028/NIST.AI.800-1.ipd

1NIST AI 800-1

2

Initial Public Draft

Managing Misuse Risk for Dual-Use
Foundation Models

3
4
5
6
7
8
9
10

U.S. AI Safety InstituteThis publication is available free of charge from:https://doi.org/10.6028/NIST.AI.800-1.ipd

July 2024

11
12
13
14
15
16
17
18
19

20
21
22
23

U.S. Department of Commerce
Gina M. Raimondo, Secretary24
25

National Institute of Standards and Technology
Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology

26
27
28
29

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

The U.S. AI Safety Institute (AISI) at NIST is releasing this document as an Initial Public
Draft for public comment.

3
4
5
6
7

Comments on NIST AI 800-1 may be sent electronically to NISTAI800-1@nist.gov with “NIST
AI 800-1, Managing Misuse Risk for Dual-Use Foundation Models” in the subject line.
Electronic submissions may be sent as an attachment in any of the following unlocked
formats: HTML; ASCII; Word; RTF; or PDF.

89All comments are subject to release under the Freedom of Information Act (FOIA).

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

Table of Contents2

1. INTRODUCTION ................................................................................................................................ 1

3

2. SCOPE .............................................................................................................................................. 1

4

3. KEY CHALLENGES IN MAPPING AND MEASURING MISUSE RISKS ...................................................... 2

5

4. OBJECTIVES AND PRACTICES TO MANAGE MISUSE RISKS ................................................................. 4

6
7
8
9
10
11
12
13

Appendix A. Glossary ......................................................................................................................... 18

14

Appendix B. Example Safeguards Against the Misuse of Foundation Models ...................................... 19

15
16
17
18
19
20
21

Disclaimer: Certain equipment, instruments, software, or materials, commercial or non-commercial, areidentified in this paper in order to specify the experimental procedure adequately. Such identification
does not imply recommendation or endorsement of any product or service by NIST, nor does it imply thatthe materials or equipment identified are necessarily the best available for the purpose.

1

1. INTRODUCTION

2
3
4
5
6
7
8
9

This document provides guidelines for improving the safety, security, and trustworthiness ofdual-use foundation models (hereafter referred to as “foundation models”)i consistent with the
National AI Initiative Actii and Executive Order 14110.iii Specifically, it focuses on managing therisk that such models will be deliberately misused to cause harm. The ways that foundation
models can be misused continue to evolve, but they include the risks that models will facilitate
the development of chemical, biological, radiological, or nuclear weapons; enable offensive
cyber attacks; aid deception and obfuscation; and generate child sexual abuse material (CSAM)
and non-consensual intimate imagery (NCII) of real individuals.

10
11
12
13
14
15

The rapid development of foundation models poses significant challenges to understanding
their capabilities and misuse risks,1 and this document provides a basis to identify, measure,and reduce these risks across the AI lifecycle. Misuse risks are not a product of a model alone—they result in part from malicious actors’ motivations, resources, and constraints, as well associety’s defensive measures against that harm.2 As a result, the guidelines provided here
address both technical and social aspects of these risks.

16
17
18
19
20

Building on the AI Risk Management Framework,3 this document identifies best practices tomap, measure, manage, and govern misuse risks from foundation models, as well howorganizations can provide transparency into how they are managing these risks. This document
focuses particularly on foundation models’ initial developers, but actors across the lifecycle of amodel all play a role in managing misuse risks.4iExecutive Order 14110 defines a “dual-use foundation model” as “an AI model that is trained on broad data;
generally uses self-supervision; contains at least tens of billions of parameters; is applicable across a wide range ofcontexts; and that exhibits, or could be easily modified to exhibit, high levels of performance at tasks that pose aserious risk to security, national economic security, national public health or safety, or any combination of thosematters, such as by: (i) substantially lowering the barrier of entry for non-experts to design, synthesize, acquire, oruse chemical, biological, radiological, or nuclear (CBRN) weapons; (ii) enabling powerful offensive cyber operations
through automated vulnerability discovery and exploitation against a wide range of potential targets of cyberattacks; or (iii) permitting the evasion of human control or oversight through means of deception or obfuscation.”
This definition is provided in a glossary at the end of this report, along with definitions of other key terms.
ii
As codified in 14 U.S.C. § 278h-1.
iii
Section 4.1(a)(ii) of Executive Order 14110 directs the Secretary of Commerce to “Establish appropriate
guidelines (except for AI used as a component of a national security system), including appropriate procedures andprocesses, to enable developers of AI, especially of dual-use foundation models, to conduct AI red-teaming tests toenable deployment of safe, secure, and trustworthy systems. These efforts shall include: (A) coordinating ordeveloping guidelines related to assessing and managing the safety, security, and trustworthiness of dual-use
foundation models”.

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

2. SCOPE

2
3
4
5
6
7
8
9

Risks. This document focuses on misuse risk from dual-use foundation models. Consistent with
Section 3(k) of Executive Order 14110,5 this includes foundation models that exhibit, or could
be easily modified to exhibit, high levels of performance at tasks that can pose a serious risk tosecurity, economic security, public health or safety, or any combination of those matters(hereafter referred to jointly as “public safety”). This document addresses both emerging
misuse risks, such as a foundation model facilitating the development of a novel biologicalweapon, as well as current harms from misuse, such as a foundation model generating CSAM or
NCII.

10
11
12
13
14
15

This document does not address other important risks from foundation models, such as bias,
discrimination, and hallucination, nor does it address all risks to public safety, including thosethat may arise from other types of AI models and systems.6 Actors across the AI lifecycle should
manage these risks as well, consistent with relevant guidelines, such as those provided in theBlueprint for an AI Bill of Rights,7 as well as the NIST AI Risk Management Framework and its
Generative AI Profile.8

16
17
18
19
20
21
22

Actors. The practices in this document are principally focused on the central role thatfoundation models’ initial developers have in the supply chain for their models.9 Thesedevelopers contribute most to determining how their models are made available, the models’capabilities, and safeguards against their misuse. In some cases, model developers may shareinfluence over these factors with an external partner, such as a cloud service provider that
leads deployment of the model, and in such cases these partners also have expanded
opportunities and responsibilities to manage the risks that a model may be misused.10

23
24
25
26
27
28

Other parties also play important roles in managing misuse risks, but they are not the focus of
this document. They include downstream developers and deployers, third-party evaluators andauditors, civil society organizations, and government agencies.11 Relevant stakeholders
throughout the AI supply chain are encouraged to share information and collaborate tounderstand and mitigate misuse risks, including to integrate appropriate risk mitigations into
downstream systems that rely on foundation models.12

1

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

3. KEY CHALLENGES IN MAPPING AND MEASURING MISUSE RISKS

2
3
4
5
6

Accurately identifying and measuring foundation models’ misuse risks helps to build confidence
in which potential harms are realistic and which practices are justified to mitigate them.
However, methodological and scientific challenges can limit understanding and measurementof these risks. Organizations should take appropriate steps to address these challenges andstrive to build an empirical basis to evaluate and mitigate risks. These challenges include:

7
8
9
10
11

1. Foundation models are broadly applicable. Models trained on a broad data distribution
can often be applied across many different domains, including domains not explicitly
considered by the developer. This makes it challenging to anticipate potential ways inwhich a model might be misused and complicates measuring or monitoring for misuserisk broadly.

12
13
14
15
16
17

2. Capabilities do not clearly translate across domains. A foundation model’s
performance on one task may not provide reliable evidence of its performance onanother, even when the two tasks appear related.13 For instance, initial benchmarksthat are cheaper and easier to carry out may suggest a model has dangerous
capabilities, but this concern may not be substantiated when the model is tested in
more rigorous and realistic conditions.

18
19
20
21
22

3. It is difficult to predict how scale will affect performance. In many instances, afoundation model’s performance can be improved by increasing the amount or qualityof its training data, the quantity of compute used to build the model, or the number ofparameters of the model.14 But while these factors may be useful heuristics in someinstances, their precision is limited and their relationship to any given risk is uncertain.15

23
24
25
26
27
28
29
30
31

4. The relationship between a measured capability and its potential risk of harm is often
unclear. It remains challenging to determine the likelihood or severity of real-world
harm through isolated testing. For many harms to public safety, domain knowledge andcomputer automation are not the only the limiting factors in carrying out an attack,which may also require physical infrastructure, distribution mechanisms, or complex
interactions in the physical world.16 Bad actors may also already have access to existing
tools that serve their needs better than foundation models, and existing methods toprevent harm—such as controls on physical dual-use materials—may be substantiallymore determinative of real-world risks.17

32
33
34
35

5. Methods to evaluate safeguards are nascent. Foundation model developers implement
safeguards to protect their models from misuse, such as those outlined in Appendix B,
but there is a lack of effective techniques for evaluating the adequacy of thosesafeguards under real-world conditions.

36
37
38
39

6. Evaluating risk may depend on scarce domain expertise. Information about some risks,like the potential for a foundation model to enable a malicious actor to develop achemical or biological weapon, may be closely guarded. In addition, organizations
developing foundation models may not currently have the professional experts or staff,

2

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1
2

nor the developed means of accessing external expertise, necessary to appropriately
assess some misuse risks.

3
4
5
6
7
8
9
10
11

7. It is difficult to accurately emulate threat actors and misuse. Assessing misuse risk
requires realistically profiling threats. This assessment includes understanding how anactor might exploit a model’s capabilities to cause harm, or how they might circumvent
safeguards that have been deployed to protect a model from misuse. This practice canbe challenging if malicious actors are willing to spend more time than evaluators or ifthey have access to infrastructure or niche expertise not available during evaluations.
Realistic emulation of malicious actors may also be prohibitively dangerous or unethical,
such as attempting to develop a volatile substance, generate non-consensual intimate
imagery, or harm real people.

3

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

4. OBJECTIVES AND PRACTICES TO MANAGE MISUSE RISKS

2
3
4
5

This section outlines seven objectives, as well as associated practices that can help achievethem, for organizations to map, measure, manage, and govern the risk that their foundation
models will be misused to deliberately harm public safety, consistent with the NIST AI Risk
Management Framework. These objectives are:

6
7
8
9
10
11
12
13

1.
2.
3.
4.
5.
6.
7.

Anticipate potential misuse risk
Establish plans for managing misuse risk
Manage the risks of model theft
Measure misuse risk
Ensure that misuse risk is managed before deploying foundation models
Collect and respond to information about misuse after deployment
Provide appropriate transparency about misuse risk

14
15
16
17
18
19
20
21

Organizations should consider all seven objectives holistically to manage foundation models’
misuse risks. This document provides practices for each objective as non-exhaustive examples
of how organizations can meet the objectives. Managing risk is an iterative process, andorganizations should assess which practices are most relevant at each point in the lifecycle of afoundation model. Implementing these practices also cannot guarantee that actors will not
misuse a foundation model, and organizations should adopt additional risk management
measures where appropriate.

22

The specific elements of each objective below include:

23
24

1. Objective: an aspirational outcome that, when achieved, will help organizations manage
a foundation model’s misuse risks.

25
26
27
28
29

2. Practice: a suggested practice that can help organizations achieve an objective. This
document proposes a particular set of practices that can be collectively used to achieve
the indicated objectives. However, an individual practice may not be appropriate for allcontexts, alternative practices may also be able to achieve these objectives, and newpractices are likely to be developed over time.

30
31

3. Recommendation: an implementation characteristic or consideration that is often
important for a practice to be effective at achieving an objective.

32
33
34
35
36
37
38
39

4. Documentation: information that could be shared with either the public or with selectparties to help demonstrate whether and how a practice was implemented, as well as
potential evidence of its effectiveness and comprehensiveness. This documentation canalso help to improve collaboration with third parties and to develop shared bestpractices for managing misuse risks. If an organization relies on alternative practices toachieve an objective, a similar level of documentation about those practices can be used
to provide transparency, along with a rationale for why the alternative practices matchor exceed the efficacy of those provided here.
4

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

Objective 1: Anticipate potential misuse risk

2
3
4
5
6

Assess the risks that a foundation model could be misused if it became available to malicious
actors, including assessing these risks before the model is developed based on an estimate of its
capabilities. Identify the most significant expected risks so that they can be measured andmanaged, as necessary.

7
8

Practice 1.1: Identify and maintain a list of threat profiles that covers significant ways inwhich malicious actors might misuse the model.

9

Recommendations to effectively implement Practice 1.1:

10
11
12

1. For each threat profile, specify the relevant capabilities of concern, a threat actor or
actors who might use them to cause harm, and the malicious tasks that the threat actor
might accomplish using the model.

13
14
15

2. Consider known misuse risks, including, as relevant, the risk that a model might be used
to facilitate the development of chemical, biological, radiological, or nuclear weapons,
to automate offensive cyber operations, or to generate CSAM or NCII.

16
17
18
19
20
21

3. Consider consulting external experts with relevant expertise and responsibilities to help
identify gaps in threat profiles, including potentially granting them model access foropen-ended experimentation to identify other ways that a model could be misused tocause harm. The consultation of external experts may be particularly relevant to helping
ensure that the threat profiles cover the most significant expected ways in which amodel could be misused to cause harm.

22
23

The following documentation can help provide transparency about how Practice 1.1 isimplemented:

24

a. The list of key threat profiles.18

25
26

b. A description of how these threat profiles were selected and judged to have adequate
coverage.

27
28

c. A description of any internal and external experts involved and their role in developing
profiles.

29
30
31

Practice 1.2: Assess the impact of each identified threat profile if the malicious actor
successfully misused the model.

32

Recommendations to effectively implement Practice 1.2:

33
34
35

1. For each threat profile, assess the impact to public safety if a threat actor successfully
used the model to carry out the malicious task, including estimates of how much it
would help an actor increase the scale, decrease the cost, or improve the effectiveness

5

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1
2of their malicious activity compared to alternative resources that are available to that
actor, such as other machine learning models and digital tools.19

34

2. For each threat profile, estimate the scale, frequency, or probability with which threat
actors might attempt to misuse the model.

5
6
7
8

3. Consider how potential harms may be prevented or managed outside of the context ofthe model itself, such as mechanisms that will effectively stop harm from a given
malicious activity regardless of how inexpensive or easy the model makes that malicious
activity.

910
11

The following documentation can help provide transparency about how Practice 1.2 isimplemented:
a. An impact assessment for each identified threat profile.

12
13
14

Practice 1.3: Estimate the model’s capabilities of concern before it is developed by comparing
it to existing models.

15

Recommendations to effectively implement Practice 1.3:

16

1. Identify similar models for which capabilities of concern have already been measured.

17
18

2. Based on comparing the characteristics of the upcoming model with existing models,estimate how its performance will compare to those models.

19
20
21

3. Assess the degree of uncertainty in these estimates based on how different the models
are, how those differences are expected to affect their capabilities, and the reliability
and completeness of the evaluations available for the similar models.

22
23
24
25

4. If capability forecasts are uncertain and include the possibility that the model may posesignificant risk, consider development strategies that may reduce reliance on
forecasting, such as taking steps to periodically measure capabilities throughout thedevelopment process.

26
27

The following documentation can help provide transparency about how Practice 1.3 isimplemented:

28

a. Capability estimates that were made prior to developing the model.

29

b. Relevant capability measurements performed after the model was developed.

30
31

c. Estimated comparisons of the model’s misuse risks compared to common benchmarks,
where available.

32
33
34
35
6

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety InstituteObjective 2: Establish plans for managing misuse risk

1
2
3
4
5

Determine acceptable levels of misuse risk, considering legal or regulatory obligations, potential
benefits that may trade off against risk, and other factors. Align development and deployment
plans with the resources, time, and operational constraints that may be required to manage
potential misuse risk.

6

Practice 2.1. Identify a level of misuse risk which the organization considers unacceptable.

7

Recommendations to effectively implement Practice 2.1:89

1. Define a risk threshold for each identified threat profile based on the impact
assessments carried out in Practice 1.2 and other relevant information.20

10
11
12

2. Account for how much the model may reduce the cost of accomplishing the malicious
task, increase the number of actors who could accomplish that task, or increase thescale or quality at which a given actor can accomplish the task.

13

3. When determining risk thresholds, weigh potential benefits against the misuse risks.

14
15

4. Refine thresholds continuously based on adjustments to identified threat profiles orchanges in factors that affect risk tolerance (such as increases in expected benefits).

16
17

5. Consider how external factors such as legal and regulatory obligations may inform whatrisk thresholds are acceptable.

18
19

The following documentation can help provide transparency about how Practice 2.1 isimplemented:

20
21

a. The thresholds for the misuse risks of a model that the organization considers
unacceptable.

22
23
24

Practice 2.2: Establish a roadmap to manage misuse risks for the development of planned
foundation models and future versions.

25

Recommendations to effectively implement Practice 2.2:26
27

1. Define how the organization expects to manage misuse risk and achieve the objectives
established in this document.

28
29

2. Plan to implement security practices to protect models from model theft if necessary to
manage misuse risk. Define security goals and a timeline for achieving those goals.

30
31
32

3. Plan to implement appropriate safeguards to protect deployed models from misuse if
necessary to manage misuse risk. Define safety goals and a timeline for achieving thosegoals.

33
34

4. Plan to adjust deployment or development strategies if misuse risks rise to unacceptablelevels before adequate security and safeguards are available to manage risk.

35
7

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2
3
4
5

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

The following documentation can help provide transparency about how Practice 2.2 isimplemented:
a. A roadmap documenting what risk mitigations are expected to be feasible over time
given technical and resource constraints, and what steps will be taken if measurementssuggest those mitigations are not adequate to manage misuse risk.

6
7
8
9
10
11

Take steps to prevent the theft of information and assets that would allow a malicious actor torecreate the foundation model, where doing so is necessary to manage misuse risk. Only
develop a model that relies on confidentiality to manage misuse risk when the risk of model
theft is sufficiently mitigated.

12

Practice 3.1: Assess the risk of model theft from relevant threat actors.

13

Recommendations to effectively implement Practice 3.1:

Objective 3: Manage the risks of model theft

14
15

1. Assess the risk of model theft by each identified actor or threat profile, based on therelevant actors’ respective resources, motivations, tactics, and sophistication.21

16
17

2. Consider the organization’s compliance with applicable cybersecurity best practices as
an input into assessing the risk of model theft.

18
19
20

3. Consider the risk posed by an insider threat, such as an individual involved in developingor deploying the model who may behave maliciously or collaborate with an external
attacker.

21
22

4. Consider using cybersecurity red teams and penetration testing to assess how difficult it
would be for an actor to circumvent security measures.

23
24
25
26

The following documentation can help provide transparency about how Practice 3.1 isimplemented:
a. A summary of evidence collected to evaluate the risk of model theft, including theresults of any red team exercises and penetration testing.

27
28
29

Practice 3.2: Compare predicted misuse risk to the organization’s risk tolerance prior todeveloping models with increased capabilities of concern.

30

Recommendations to effectively implement Practice 3.2:

31
32

1. Prior to developing a model, assess overall misuse risk from model theft by combining
the estimated capabilities of concern with an estimate of the probability of model theft.

33
34

2. Adjust or halt further development until the risk of model theft is adequately managed
and within the organization’s defined risk thresholds.

8

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2
3
4
5
6
7

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

3. Periodically revisit estimates of misuse risk stemming from model theft, including prior
to individual development decisions that may significantly increase capabilities ofconcern.
The following documentation can help provide transparency about how Practice 3.2 isimplemented:
a. The risk assessments that were performed, either periodically or prior to significant
development and deployment decisions, related to model theft.

8
9

Practice 3.3: Maintain security practices sufficient to prevent model theft.

10

Recommendations to effectively implement Practice 3.3:

11

1. Implement security practices designed to protect against model theft.

12
13
14

2. Where applicable, apply security practices specifically tailored to the context offoundation models, such as protections against exfiltrating large amounts of data (e.g.,model weights) and protections against vulnerabilities like extraction attacks.22

15
16

3. Increase the level of security proportionate to the level of misuse risk posed by themodel and the importance of preventing model theft to managing that risk.

17
18

4. Apply appropriate protections against insider threats, such as limiting access to model
weights within the organization.

19
20

5. Consider adopting existing security standards for sensitive data and applications, such as
U.S. government cybersecurity guidance and applicable international standards.

21
22

6. Re-assess the risk of model theft prior to development with new security practices inplace (if previous assessments found unacceptable risks from model theft).

23
24
25
26

The following documentation can help provide transparency about how Practice 3.3 isimplemented:
a. A summary of security measures that have been implemented to reduce the risk ofmodel theft.

27

Objective 4: Measure the risk of misuse

28
29
30
31
32
33

Where there is a reasonable assessment that a foundation model could be misused, measure
the predicted risks to provide evidence for the model’s actual misuse risk in a real-world context.
Rely on methods that incorporate both technical and social factors, as well as those that provide
accurate evidence and higher confidence, while avoiding any harm that could be caused bymeasuring dangerous activities.

34

Practice 4.1: Measure model capabilities relevant to assessing misuse risk.

35

Recommendations to effectively implement Practice 4.1:
9

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

12
3

1. Directly measure capabilities of concern or identify alternative capability measurementsthat allow for sufficiently reliable inferences about them, such as pursuingrecommendations 2, 3, and 4 below when they are sufficiently reliable.

4
5
6
7
8
9

2. Consider comparing the performance of a new model to an existing model with known
misuse risks (the “proxy model”). If the models are similar across a broad range ofevaluations, results from the proxy model’s capability evaluations may be suitable toassess risk from the new model. If this comparison is inconclusive or suggests that the
new model may pose significant risk, then the new model’s capabilities of concern
should be separately evaluated.

10
11

3. Consider measuring model performance on proxy tasks that are safe and tractable while
being similar enough to allow reliable inferences about the capability of concern.

12
13
14
15
16
17

4. If the most accurate capability measurement would be expensive and it is appropriate
to do so, consider using cheaper tests (such as automated evaluations on simpler tasks
that would be easier for the model to perform than the true capabilities of concern)23
that may indicate that a model lacks dangerous capabilities. If these tests areinconclusive or indicate a potential risk, then rely on more costly and precise
measurements.

18
19
20
21

5. Account for the possibility that the model's capabilities of concern may exceed any
evaluated proxy capabilities and adjust the interpretation of evaluation results
accordingly. This consideration is especially relevant when the proxy capability being
evaluated differs significantly from the actual capability of concern.

22
23
24
25
26
27

6. Assess what a threat actor could achieve given access to the weights of a model and theability to integrate it with other tools by testing AI systems which are reasonably
optimized for performance on the evaluation task. If there is a gap between the effort
applied to optimize system performance during testing and the effort that could beapplied by a threat actor, evaluations should explicitly account for the uncertainty
introduced by that gap.

28
29

7. Avoid overlap between data used to train the model and data used in capability and risk
evaluations and measure the extent of any overlap.

30
31

The following documentation can help provide transparency about how Practice 4.1 isimplemented:

32
33

a. A list of evaluation tasks that were used to evaluate each threat profile, and a
representative subset of datasets used for each evaluation.

34

b. A methodological description for each evaluation in enough detail to reproduce it.

35
36

c. An analysis of the relationship between evaluation tasks and capabilities of concern,addressing the possibility that evaluation tasks are more challenging.

37
38

d. If a model was evaluated via a comparison with a proxy model: the proxy model used
for comparison, evaluations used to establish comparability, the risk assessment for the
10

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Instituteproxy model, and other ways in which the proxy may be meaningfully different from themodel being assessed.

345

Practice 4.2: Use red teams to assess whether threat actors could bypass model and systemsafeguards and misuse any capabilities of concern.

6

Recommendations to effectively implement Practice 4.2:

7
8
9
10
11

1. To justify an assessment that a threat actor is unlikely to be able to misuse a model,
verify that an adequately resourced red team is unable to misuse the model oraccomplish a related proxy task in a realistic deployment context. Apply practices 4.1.1,
4.1.3, and 4.1.4 to the selection of proxy tasks for red team exercises as well ascapability evaluations.

12
13
14

2. Clearly specify what goal the red team is trying to achieve in advance, provide incentives
and accountability for achieving those goals, and select red teams based on their ability
to achieve those goals.

15
16
17

3. Rely on red teams comprised of external experts that are meaningfully independent
from the model developer and who do not have incentives that conflict with their redteaming goal.

18
19
20

4. Compare the red team’s expertise, resources, and time available to those of a relevant
threat actor. To the extent that there are gaps, explicitly account for those gaps wheninterpreting the result of a red team exercise.

21
22
23
24

5. Consider providing additional resources to the red team or making the red team’s task
easier in other ways (such as providing additional access,24 reducing task difficulty, or
dividing complex tasks into constituent pieces) to compensate for any remaining gaps
between the red team and threat actors.

25
26
27

6. Consider providing red teams with available legal protections for their tasks, such as
waiving terms of service and indemnifying them for legal liability for their interactions
with the model.

28
29
30

7. Provide the red team with at least as much information about the system as would beavailable to an attacker and make explicit any respects in which the red team lacks full
information about the design of system safeguards.

31
32
33
34

8. Consider each level of access to a model that a threat actor might have, ranging fromlimited access through an API to direct access to the code and parameters that define
the model, and determine the minimum level of access (if any) that allows the red teamto accomplish its goal.

35
36
37
38

9. Explicitly define the time period over which the red teaming results are intended toapply and consider the fact that a threat actor will have access to any information about
how to use the model or circumvent safeguards that becomes public over that time
period.
11

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

The following documentation can help provide transparency about how Practice 4.2 isimplemented:

3a. A description of the red team’s goal.

4b. A description of the red team’s composition, expertise, resources, and timelines.

5c. The results of the red team exercise given varying levels of access to the model.

6d. A summary of the strategies pursued by the red team.

7

Objective 5: Ensure that misuse risk is managed before deploying foundation
models

8
9
10
11
12

Take actions to increase access to the model (e.g., deploying a model via API or releasing its
weights) only when misuse risks are adequately managed, including that they are at minimum
within the organization’s risk tolerance.

13

Practice 5.1: Assess the effect of a potential deployment on the model’s misuse risk.

14

Recommendations to effectively implement Practice 5.1:

15
16
17

1. Consider potential deployments and the levels of access they would grant actors (e.g.
whether they would increase the number or type of actors who can access a model orwhether they would grant actors greater access to a model’s features).

18
19
20
21
22
23

2. Identify the level of access that a malicious actor could obtain under each proposed
deployment (e.g., would it grant them access to API inference, access to a fine-tuning
API, access to model weights) and consider how the deployment may affect misuse risk.
For example, allowing fine-tuning via API can significantly limit options to prevent
jailbreaking and sharing the model’s weights can significantly limit options to monitor
for misuse (Practice 6.1) and respond to instances of misuse (Practice 6.2).

24
25
26
27
28

The following documentation can help provide transparency about how Practice 5.1 isimplemented:
a. For each deployment: the level of access provided, the assessed misuse risks associatedwith the deployment, and the set of safeguards anticipated to mitigate these misuserisks.

29
30

Practice 5.2: Implement safeguards proportionate to the model’s misuse risk.

31

Recommendations to effectively implement Practice 5.2:

32
33

1. Implement safeguards designed to protect the model from misuse. Appendix A outlines
a non-exhaustive list of possible safeguards.

34
35

2. Establish reliable evidence of safeguards’ effectiveness before relying on them toprevent misuse of meaningful capabilities of concern.
12

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2
3
4
5

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

3. If additional safeguards are added in response to identified risks, consider re-assessing
misuse risk prior to deployment by carrying out red-teaming exercises with theadditional safeguards in place.
The following documentation can help provide transparency about how Practice 5.2 isimplemented:

6a. The list of safeguards that have been implemented.

7b. A summary of the result of safeguard evaluations via red teams and other testing.

8
9

Practice 5.3: Only pursue deployments where misuse risk is adequately managed.

10

Recommendations to effectively implement Practice 5.3:

11
12
13
14
15
16
17

1. For each deployment, establish a process to determine whether the deployment should
proceed based on the assessed misuse risk and a consideration of any safeguards.
Otherwise, determine whether the deployment should be modified, delayed, or
canceled. For instance, consider whether further safety improvements are feasible prior
to deployment, whether additional time could be used to carry out a more reliable
estimate of risk, or whether a more limited deployment may be more appropriate given
the level of assessed risk.

18
19
20
21
22

2. Consider leaving a margin of safety between the estimated level of risk at the point of
deployment and the organization’s risk tolerance. Consider how threat actors may
continue to acquire new knowledge about how to misuse or augment the model after it
is deployed25 and how to circumvent its safeguards.26 Consider a larger margin of safety
to manage risks that are more severe or less certain.

23
24

The following documentation can help provide transparency about how Practice 5.3 isimplemented:

25
26
27

a. The basis for determining that the risk of misuse was adequately managed for a
deployment decision, including that the deployment’s risks were within theorganization’s risk thresholds.

28

Objective 6: Collect and respond to information about misuse after deployment

29
30
31
32
33

Collect information about deployed systems that improves understanding of their misuse risk toadjust deployments and improve future risk management. Engage with and encourage findings
from the public, relevant civil society organizations, external researchers, and the foundation
model’s third-party distribution partners.

34

Practice 6.1: Where possible, monitor distribution channels for evidence of misuse.

35

Recommendations to effectively implement Practice 6.1:

13

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1
2

1. Monitor APIs, websites, and other distribution channels for misuse while maintaining
privacy of users.

3

2. Build or procure systems to enable automated detection of misuse.

4
5
6
7

3. Continually assess the effectiveness of systems that are intended to detect misuse tohelp provide ground truth for their effectiveness and evidence that misuse is not going
undetected. Red teaming can be particularly helpful to assess whether malicious actors
may be able to deliberately avoid detection.

89
10

4. Request that third-party distribution channels for the model monitor those channels for
misuse and that they regularly share information with the developer regarding theirmonitoring.

11
12
13
14

5. Consider using tiered methods of detection when doing so helps prioritize limited
resources, improve privacy, and increase coverage. This can include filtering for misuse
first by less costly automated methods which are then, as appropriate, validated bymethods requiring direct human intervention.

15
16
17
18

6. When malicious actors can avoid direct monitoring by operating the model
independently, such as when its weights are widely available, consider mechanisms
other than those identified in recommendations 6.1.1, 6.1.2, and 6.1.3 that could be
used to monitor for patterns of malicious behavior enabled by the model.

19
20

The following documentation can help provide transparency about how Practice 6.1 isimplemented:

21
22
23

a. A summary of the mechanisms used to monitor each distribution channel for afoundation model and the methods for determining the effectiveness of thosemechanisms.27

24
25

Practice 6.2: Maintain a process to respond to incidents of model misuse.

26

Recommendations to effectively implement Practice 6.2:

27

1. Establish clear organizational responsibilities for responding to incidents of misuse.

28
29

2. Preemptively develop plans for plausible novel scenarios of misuse and how to respond,such as restricting access to the model or strengthening safeguards.28

30
31
32

3. When it may not be possible to effectively reduce access to a model, such as when themodel’s weights are widely available, plan for how identified instances of misuse will
inform future development and deployment decisions.

33
34

4. Consider carrying out drills to practice responding to time-sensitive and safety-criticalscenarios of misuse.

35
36

The following documentation can help provide transparency about how Practice 6.2 isimplemented:
14

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institutea. A summary of the incident response process and the organizational roles andresponsibilities in the process.

3
4

Practice 6.3: Establish protections for internal reporting of misuse issues.

5

Recommendations to effectively implement Practice 6.3:

67

1. Adopt policies that protect and reward individuals who report model issues related tomisuse risk.

89

2. Establish formal processes to adjudicate concerns raised by employees and contractors
in a timely fashion.

10
11
12

The following documentation can help provide transparency about how Practice 6.3 isimplemented:
a. A summary of the organization’s policies with respect to internal safety reporting.

13
14

Practice 6.4: Provide safe harbors for third-party safety research.

15

Recommendations to effectively implement Practice 6.4:

16
17
18

1. Publish a clear vulnerability disclosure policy for model safety issues that outlines how
such vulnerabilities should be shared with the developer and the public, and how theorganization will respond to reported vulnerabilities.

19
20
21

2. Publish a safe harbor policy that commits to not pursuing legal action against or
restricting the accounts of external safety researchers that act in good faith and comply
with the vulnerability disclosure policy.29

22
23
24

3. Consider providing support and accommodations for vetted external researchers’interactions with the model, such as providing researchers with access to models withfewer safeguards to conduct post-deployment red-teaming exercises.

25
26

The following documentation can help provide transparency about how Practice 6.4 isimplemented:

27
28

a. A summary of the organization’s commitment to not pursue legal action against thirdparty researchers acting in good faith.

29
30

b. A description of the organization’s process for providing vetted researchers with accessto models or systems with fewer safeguards.30

31
32

Practice 6.5: Create bounties for issues related to the misuse risk.

33

Recommendations to effectively implement Practice 6.5:

15

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

1
2

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute

1. Establish a program to incentivize researchers for finding vulnerabilities and disclosing
them according to the vulnerability disclosure policy.31

3
4

The following documentation can help provide transparency about how Practice 6.5 isimplemented:

5
6

a. The terms of the bounties and details regarding the process for submitting
vulnerabilities.

7

Objective 7: Provide appropriate transparency about misuse risk

8
9
10
11
12

Provide transparency to the public and relevant entities regarding the organization’s processes
for developing and deploying foundation models that are related to misuse risk to facilitate
understanding, accountability, collaboration, and scientific development related to model
misuse.

13
14

Practice 7.1: Publish regular transparency reports that include key details regarding misuserisks and how those risks are managed.

15

Recommendations to effectively implement Practice 7.1:

16
17
18

1. Share the methodology and results of pre-deployment evaluations of model capabilities,risks, and mitigations, including as much detail about the data and evaluation
methodology as can be disclosed without introducing risks to public safety.32

19
20
21

2. Share details regarding the safeguards in place for the model, including how they areapplied across different distribution channels, with as much as can be shared without
rendering the safeguards ineffective.33

22
23

3. Share information about data used to build the model that is relevant to assessing themisuse risk, such as data sources and criteria for data filtering and selection.34

24
25

4. Share steps that downstream developers and deployers of AI systems that integrate thefoundation model should take to manage misuse risk.

26
27
28

5. Share details regarding the organization’s internal structure and internal governanceprocess with respect to determining how to deploy foundation models and map,
measure, and manage misuse risk.

29
30
31

6. Make the transparency reports publicly available, update them on a regular basis (e.g.,with each new major version of the model), and include key information related tomisuse risk.35

32
33
34

Practice 7.2: Disclose information about risk management practices to promoteaccountability.

35

Recommendations to effectively implement Practice 7.2:

16

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute12

1. Regularly share information about risk management practices through a process
structured to provide meaningful accountability.

345

2. Share information covering the practices used to achieve the objectives listed in thisdocument, including at least as much detail as described in the documentation sections
for each practice listed here.

67
8

3. Ensure recipients of information are effectively independent from the development anddeployment process and that they are empowered to provide accountability (such as bydisclosing risk management deficiencies publicly).

9
10
11
12

The following documentation can help provide transparency about how Practice 7.2 isimplemented:
a. A summary of the list of stakeholders that receive information about risk management
practices, and the types of information disclosed to them.

13
14
15

Practice 7.3: Report incidents and hazards related to the foundation model to AI incident
databases.

16

Recommendations to effectively implement Practice 7.3:

17
18

1. Based on existing best practices and adequate review of the benefits and risks ofdisclosing certain information, define the category of misuse events to report.36

19

2. Collate verified reports of misuse in a commonly used and machine-readable format.

20
21

3. Share verified reports of misuse with relevant third parties, such as AI incident
databases.37

22
23

The following documentation can help provide transparency about how Practice 7.3 isimplemented:

24
25

a. A description of the incident reporting process, including an example of a type ofincident that might be reported and to whom it would be reported.

17

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

Appendix A. Glossary

2
3
4
5
6
7

Artificial Intelligence (AI)A machine-based system that can, for a given set of human-defined objectives, make predictions,recommendations, or decisions influencing real or virtual environments. Artificial intelligence systems use
machine- and human-based inputs to perceive real and virtual environments; abstract such perceptions intomodels through analysis in an automated manner; and use model inference to formulate options for informationor action.38

8
9
10
11
12
13

AI Red-Teaming
A structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment andin collaboration with developers of AI. Artificial Intelligence red-teaming is most often performed by dedicated“red teams” that adopt adversarial methods to identify flaws and vulnerabilities, such as harmful or discriminatory
outputs from an AI system, unforeseen or undesirable system behaviors, limitations, or potential risks associatedwith the misuse of the system.39

14
15
16

Capability of Concern
A capability of an AI model that poses a risk of enabling malicious behavior that would cause serious harm to public
safety.

17
18
19
20

Distribution Channel
The various ways in which a model could be distributed, including, but not limited to, public release of the model
weights, access to the model via an API supplied by a cloud service provider, or access to a fine-tuned or otherwise
augmented version of the model from a third-party deployer.

21
22
23
24
25
26
27
28
29
30

Dual-Use Foundation Model
An AI model that is trained on broad data; generally uses self-supervision; contains at least tens of billions ofparameters; is applicable across a wide range of contexts; and that exhibits, or could be easily modified to exhibit,high levels of performance at tasks that pose a serious risk to security, national economic security, national publichealth or safety, or any combination of those matters, such as by: substantially lowering the barrier of entry for
non-experts to design, synthesize, acquire, or use chemical, biological, radiological, or nuclear weapons; enabling
powerful offensive cyber operations through automated vulnerability discovery and exploitation against a wide
range of potential targets of cyber attacks; or permitting the evasion of human control or oversight through meansof deception or obfuscation. Models meet this definition even if they are provided to end users with technicalsafeguards that attempt to prevent users from taking advantage of the relevant unsafe capabilities.40

31
32
33

Fine-Tuning
An approach in which the parameters of an already trained model are adjusted by training on new data. Finetuning is often used to adapt a model to a particular task, or to mildly modify a model’s behavior.

34
35
36

Foundation Model
An AI model that is trained on broad data; generally uses self-supervision; contains at least tens of billions ofparameters; and is applicable across a wide range of contexts.41

37
38

Jailbreaking
Methods that cause an AI model to act in a manner contrary to how its designer intended for it to behave.

39
40

Misuse Risk
A risk that an AI model will be deliberately misused to cause harm.

41
42
43
44

Model Theft
Unauthorized access to information that substantially aids an actor in recreating an AI model or its capabilities ofconcern, such as the code or weights that define the model or bypassing security measures designed to prevent
access to the model itself such that an actor can utilize AI model capabilities in an unauthorized manner.
18

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute1

Appendix B. Example Safeguards Against the Misuse of Foundation Models

2
3
4
5
6
7
8

The available safeguards against the misuse of foundation models continue to evolve andexpand. Current literature suggests that safeguards’ effectiveness can vary widely, and relianceon them to prevent misuse should be based on evidence of the safeguards’ efficacy, particularly
as some may incur tradeoffs in areas like cost, transparency, research, and user privacy. The
following table provides a non-exhaustive survey of several categories of safeguards thatorganizations can consider:

Safeguard
Improve themodel’s training.

Detect and blockattempted
misuse.

Limit access to themodel’s
capabilities.

Ensure the level ofaccess to themodel’s weights isappropriate.48

Stop development
if a model displays
significant misuse
risk.

Table 1. Example Safeguards.
Possible Implementation Methods• Filter training data to exclude examples that could result in capabilities that increase the likelihoodof misuse, such as biological sequence data, or known CSAM/NCII images.42• Employ training techniques that reduce the model’s propensity to produce harmful output such as
by reducing its knowledge of harmful information, reducing its capabilities of concern, or causing
the model to refuse harmful requests.43• Train models with approaches that make it more difficult for subsequent fine-tuning to remove
safeguards.44• Add additional infrastructure around the base model to monitor for and detect behavior which
may constitute misuse—such as with algorithmic classifiers for misuse. This infrastructure should
be supported, as appropriate, by human review.45• Once detected, block, modify, or otherwise limit unsafe queries and responses, and place
limitations on users and organizations engaging in misuse or attempting to circumvent safeguards.
• Monitor the model’s risk for misuse with a limited audience and gradually expand access to themodel to wider audiences over time.46• Limit the ability to interact with the model to contexts and users where the misuse risks arelower.47• Reduce access to the model reactively when misuse is detected to limit further misuse, such as byrolling back a model to a previous version, or discontinuing availability of a model, if a model
displays significant misuse risk while it is in production.
• Apply limitations on using the model’s capabilities broadly to all interactions with the model ortarget limitations to particular capabilities and features at particular misuse risks, such ascapabilities of concern that also have narrow beneficial purposes.
• For models that pose more severe risk, consider restricting the model only to those directly
involved in its development and evaluation. Internal protections can reduce the risk that anorganization’s own employees or contractors are able to misuse models; these protections can
include logging employee interactions with models, restricting model access to a subset of
employees, requiring multiple employees to access a model together, and ensuring that all
employee interactions with the model are accompanied by the model’s other relevant safeguards.
• Consider when it is appropriate to make the model’s weights widely available, such as available fordownload by the public. Once a model’s weights are made widely available, options to roll back or
prevent its further sharing and modification are severely limited.49• Consider when it is appropriate to allow the foundation model to be fine-tuned via API, which also
can reduce the availability of safeguards, such as by letting users train the model on data related todangerous tasks or reduce how often the model refuses dangerous requests.50• Consider limiting internal access to the models’ weights within an organization.
• If other practically available safeguards are not sufficient to protect a model from misuse–including
considering the risk of theft or internal abuse–then it may be appropriate to make significant
changes to the development plan, or else not develop the model at all.

19

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute11

Anwar, U. et al., (2024) Foundational Challenges in Assuring Alignment and Safety of Large Language Models.
arXiv. https://arxiv.org/pdf/2404.09932. For information on the rate of progress of foundation models, see Maslej,
N. et al., (2024) The AI Index 2024 Annual Report. AI Index Steering Committee, Institute for Human-Centered AI,
Stanford University. https://aiindex.stanford.edu/wp-content/uploads/2024/05/HAI_AI-Index-Report-2024.pdf;
Ho, A. et al., (2024) Algorithmic progress in language models. arXiv. https://arxiv.org/abs/2403.05812; and Sevilla,
J. et al., (2024) Training Compute of Frontier AI Models Grows by 4-5x per Year. Epoch AI.
https://epochai.org/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year.
2
Vogel, M. et al., (2024) Findings & Recommendations: AI Safety. National AI Advisory Committee.
https://ai.gov/wp-content/uploads/2024/06/FINDINGS-RECOMMENDATIONS_AI-Safety.pdf.
3
AI Risk Management Framework. National Institute of Standards and Technology. https://www.nist.gov/itl/airisk-management-framework.
4
Srikumar, M. et al., (2024) Risk Mitigation Strategies for the Open Foundation Model Value Chain. Partnership on
AI. https://partnershiponai.org/resource/risk-mitigation-strategies-for-the-open-foundation-model-value-chain/.
5
Executive Order 14110 on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence (2023)
The White House. https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-orderon-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/.
6
For instance, many smaller, domain-specific models, such as some generative image models, can also be misusedin harmful ways. This document also does not cover risks from accidental AI harms to public safety.
7
Blueprint for an AI Bill of Rights (2022) The White House. https://www.whitehouse.gov/ostp/ai-bill-of-rights/.
8
AI Risk Management Framework: Generative Artificial Intelligence Profile. National Institute of Standards andTechnology. https://airc.nist.gov/docs/NIST.AI.600-1.GenAI-Profile.ipd.pdf.
9Bommasani, R. et al., (2021) On the Opportunities and Risks of Foundation Models. arXiv.
https://arxiv.org/abs/2108.07258.
10
Heim, L. et al., (2024) Governing Through the Cloud: The Intermediary Role of Compute Providers in AI
Regulation. arXiv. https://arxiv.org/pdf/2403.08501
11
Cen, S. et al., (2024) AI Supply Chains. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4789403;
and Constanza-Chock, S. et al., (2022) Who Audits the Auditors? Recommendations from a field scan of algorithmic
auditing ecosystem. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency.
https://dl.acm.org/doi/abs/10.1145/3531146.3533213.
12
Mulani, N. et al., (2023) Proposing a Foundation Model Information-Sharing Regime for the UK. Centre for theGovernance of AI. https://www.governance.ai/post/proposing-a-foundation-model-information-sharing-regimefor-the-uk; and Bommasani, R. et al. (2023) The Foundation Model Transparency Index. arXiv.
https://arxiv.org/abs/2310.12941.
13
Anwar, U. et al., (2024) Foundational Challenges in Assuring Alignment and Safety of Large Language Models.
arXiv. https://arxiv.org/pdf/2404.09932.
14
Hoffmann, J. et al., (2022) Training Compute-Optimal Large Language Models. Proceedings of the 2022
Conference on Neural Information Processing Systems. https://dl.acm.org/doi/10.5555/3600270.3602446.
15
Besiroglu, T. et al., (2024) Chinchilla Scaling: A replication attempt. arXiv. https://arxiv.org/pdf/2404.10102.
16
Terwilliger T. et al., (2023) AlphaFold predictions are valuable hypotheses and accelerate but do not replace
experimental structure determination. Nature. https://www.nature.com/articles/s41592-023-02087-4.
17, S. et al., (2024) Position: On the Societal Impact of Open Foundation Models. Proceedings of the 2024
International Conference on Machine Learning. https://openreview.net/pdf?id=jRX6yCxFhx.
18
Anthropic (2023) Responsible Scaling Policy. Anthropic. https://wwwcdn.anthropic.com/1adf000c8f675958c2ee23805d91aaade1cd4613/responsible-scaling-policy.pdf; OpenAI (2023)
Preparedness Framework (Beta). OpenAI. https://cdn.openai.com/openai-preparedness-framework-beta.pdf;
DeepMind (2024) Frontier Safety Framework. DeepMind. https://storage.googleapis.com/deepmind-

20

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institutemedia/DeepMind.com/Blog/introducing-the-frontier-safety-framework/fsf-technical-report.pdf. Magic (2024) AGI
Readiness Policy. Magic. https://magic.dev/agi-readiness-policy.
19
Kapoor, S. et al., (2024) Position: On the Societal Impact of Open Foundation Models. Proceedings of the 2024
International Conference on Machine Learning. https://openreview.net/pdf?id=jRX6yCxFhx; and Mouton, C. et al.,(2024) The Operational Risks of AI in Large-Scale Biological Attacks. RAND.
https://www.rand.org/pubs/research_reports/RRA2977-2.html.
20
Koessler, L. et al., (2024) Risk thresholds for frontier AI. arXiv. https://arxiv.org/pdf/2406.14713.
21
Nevo, S., et al., (2024) Securing AI Model Weights. RAND.
https://www.rand.org/pubs/research_reports/RRA2849-1.html.
22
Carlini, N. et al., (2024) Stealing Part of a Production Language Model. arXiv, https://arxiv.org/pdf/2403.06634.
23
Barrett, A. et al, (2024) Benchmark Early and Red Team Often: A Framework for Assessing and Managing DualUse Hazards of AI Foundation Models. Center for Long-Term Cybersecurity, UC Berkeley.
https://cltc.berkeley.edu/wp-content/uploads/2024/05/Dual-Use-Benchmark-Early-Red-Team-Often.pdf; Shao, M.
et al., (2024) NYU CTF Dataset: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive
Security. arXiv. https://arxiv.org/pdf/2406.05590v1; and Li, N. et al., (2024) The WMDP Benchmark: Measuring andReducing Malicious Use With Unlearning. arXiv. https://arxiv.org/pdf/2403.03218.
24
Casper, S. et al., (2024) Black-Box Access is Insufficient for Rigorous AI Audits. arXiv.
https://arxiv.org/pdf/2401.14446.
25
Bran, A. et al., (2024) Augmenting large language models with chemistry tools. Nature.
https://www.nature.com/articles/s42256-024-00832-8; Davidson, T. et al., (2023) AI capabilities can besignificantly improved without expensive retraining. arXiv. https://arxiv.org/pdf/2312.07413; and Measuring theimpact of post-training enhancements. METR. https://metr.github.io/autonomy-evals-guide/elicitation-gap/.
26
Jin, H. et al., (2024) JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and VisionLanguage Models. arXiv. https://arxiv.org/pdf/2407.01599.
27
Overview of Meta AI safety policies prepared for the UK AI Safety Summit. Meta.
https://transparency.meta.com/en-gb/policies/ai-safety-policies-for-safety-summit; Microsoft’s AI Safety Policies.
Microsoft. https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RW1dObQ; AI Safety Summit: An update
on our approach to safety and responsibility. Google DeepMind. https://deepmind.google/public-policy/ai-summitpolicies/; AI Safety Summit – Enhancing Frontier AI Safety. AWS Amazon. https://aws.amazon.com/uki/cloudservices/uk-gov-ai-safety-summit/; Our policy on frontier safety. Inflection. https://inflection.ai/frontier-safety;
and OpenAI. https://openai.com/index/openai-safety-update/.
28
O’Brien, J. et al., Deployment Corrections: An incident response framework for frontier AI models. Institute for AI
Policy and Strategy. https://arxiv.org/pdf/2310.00328.
29
Longpre, S. et al., (2024) A Safe Harbor for AI Evaluation and Red-Teaming. arXiv.
https://arxiv.org/pdf/2403.04893.
30
Bucknall, B. et al., (2023). Structured Access for Third-Party Research on Frontier AI Models. Center for theGovernance of AI. https://www.governance.ai/research-paper/structured-access-for-third-party-research-onfrontier-ai-models.
31
Wan, S. et al., (2024) Bridging the Gap: A Study of AI-based Vulnerability Management between Industry andAcademia. arXiv. https://arxiv.org/abs/2405.02435.
32
Phuong, M. et al., (2024) Evaluating Frontier Models for Dangerous Capabilities. arXiv.
https://arxiv.org/pdf/2403.13793; Anthropic (2024) Claude 3.5 Sonnet Model Card Addendum. Anthropic.
https://wwwcdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf; andOpenAI (2023) GPT-4 System Card. OpenAI. https://cdn.openai.com/papers/gpt-4-system-card.pdf
33
Bommasani, R. et al., (2024) The Foundation Model Transparency Index v1.1: May 2024. arXiv.
https://arxiv.org/pdf/2407.12929.
34
Longpre, S. et al., (2023). The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution
in AI. arXiv. https://arxiv.org/abs/2310.16787.

21

NIST AI 800-1 ipd (Initial Public Draft)
July 2024

Managing Misuse Risk for Dual-Use Foundation Models
U.S. AI Safety Institute35

Voluntary AI Commitments. White House. https://www.whitehouse.gov/wpcontent/uploads/2023/09/Voluntary-AI-Commitments-September-2023.pdf; Bommasani, R. et al., (2024) The
Foundation Model Transparency Index v1.1: May 2024. arXiv. https://arxiv.org/pdf/2407.12929.
36
McGregor, S. (2020) Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident
Database. arXiv. https://arxiv.org/pdf/2011.08512.
37
Turri, V. et al., (2023) Why We Need to Know More: Exploring the State of AI Incident Documentation Practices.
AAAI/ACM Conference on AI Ethics and Society. https://doi.org/10.1145/3600211.3604700.
38
The term “artificial intelligence” or “AI” has the meaning set forth in 15 U.S.C. 9401(3).
39
Executive Order 14110.
40
Executive Order 14110.
41
Executive Order 14110.
42
Thiel, D., et al., (2023) Identifying and Eliminating CSAM in Generative ML Training Data and Models. Stanford
Digital Repository. https://purl.stanford.edu/kh752sm9123.
43
Lynch, A. et al., (2024) Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv.
https://arxiv.org/pdf/2402.16835.
44
Henderson, P. et al., (2022) Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation
Models. arXiv. https://arxiv.org/pdf/2211.14946.
45
Bommasani, R. (2023) Ecosystem graphs: The social footprint of foundation models. arXiv, abs/2303.15772.
46
Solaiman, I. (2023) The Gradient of Generative AI Release: Methods and Considerations. arXiv.
https://arxiv.org/pdf/2302.04844.
47
Seger, E. et al., (2024) Open-Sourcing Highly Capable Foundation Models. https://cdn.governance.ai/OpenSourcing_Highly_Capable_Foundation_Models_2023_GovAI.pdf
48
When a model’s weights are available to a threat actor, a range of other safeguards become less effective,
particularly those that are implemented at the application level, such as limiting who can use the model anddetecting when it is misused. Limiting access to the model weights should be weighed against the potential
benefits of access, such as for innovation and research, including research into safety. Limiting access to model
weights is also only effective if organizations can prevent model theft.
49
Kapoor, S. et al., (2024) Position: On the Societal Impact of Open Foundation Models. Proceedings of the 2024
International Conference on Machine Learning. https://openreview.net/pdf?id=jRX6yCxFhx.
50
Qi, X. et al., (2024) Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend
To! arXiv. https://arxiv.org/pdf/2310.03693.

22