Withdrawn Draft
Warning Notice
The attached draft document has been withdrawn and is provided solely for historical purposes.
It has been followed by the document identified below.
Withdrawal Date January 4, 2024
Original Release Date March 8, 2023

The attached draft document is followed by:
Status Final
Series/Number NIST AI 100-2e2023
Title Adversarial Machine Learning: A Taxonomy and Terminology ofAttacks and Mitigations
Publication Date January 2024
DOI https://doi.org/10.6028/NIST.AI.100-2e2023
CSRC URL https://csrc.nist.gov/pubs/ai/100/2/e2023/final
Additional Information

2

3

4

5
6

7
8

9

NIST AI 100-2e2023 ipd

Adversarial Machine Learning
A Taxonomy and Terminology of Attacks and Mitigations

Alina Oprea
Apostol Vassilev

This publication is available free of charge from:https://doi.org/10.6028/NIST.AI.100-2e2023.ipd

10

11

12

13
14

15
16
17

NIST AI 100-2e2023 ipd

Adversarial Machine Learning
A Taxonomy and Terminology of Attacks and Mitigations
Alina Oprea
Northeastern University
Apostol Vassilev
Computer Security Division
Information Technology Laboratory

19

This publication is available free of charge from:https://doi.org/10.6028/NIST.AI.100-2e2023.ipd

20

March 2023

18

21

22
23

24
25

U.S. Department of Commerce
Gina M. Raimondo, Secretary
National Institute of Standards and Technology
Laurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology

26
27
28
29

30
31
32

33
34

35
36
37
38
39

40
41
42

43
44

Certain commercial equipment, instruments, software, or materials, commercial or non-commercial, areidentiﬁed in this paper in order to specify the experimental procedure adequately. Such identiﬁcation does
not imply recommendation or endorsement of any product or service by NIST, nor does it imply that thematerials or equipment identiﬁed are necessarily the best available for the purpose.

NIST Technical Series Policies
Copyright, Use, and Licensing Statements
NIST Technical Series Publication Identiﬁer Syntax

Publication History
Supersedes Draft NIST IR 8269 (October 2019) DOI https://doi.org/10.6028/NIST.IR.8269-draft

How to cite this NIST Technical Series Publication:
Oprea A, Vassilev A, (2023) Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and
Mitigations. (National Institute of Standards and Technology, , Gaithersburg, MD) NIST Artiﬁcial
Intelligence (AI) NIST AI 100-2e2023 ipd.
https://doi.org/10.6028/NIST.AI.100-2e2023.ipd

NIST Author ORCID iDs
Alina Oprea: 0000-0002-4979-5292
Apostol Vassilev: 0000-0002-9081-3042

Public Comment Period
March 08, 2023 - September 30, 2023

46

Submit Comments
ai-100-2@nist.gov

47

All comments are subject to release under the Freedom of Information Act (FOIA).

45

48

Abstract

59

This NIST NIST AI report develops a taxonomy of concepts and deﬁnes terminology in the ﬁeld of
adversarial machine learning (AML). The taxonomy is built on survey of the AML literature and isarranged in a conceptual hierarchy that includes key types of ML methods and lifecycle stage of attack,attacker goals and objectives, and attacker capabilities and knowledge of the learning process. The
report also provides corresponding methods for mitigating and managing the consequences of attacks
and points out relevant open challenges to take into account in the lifecycle of AI systems. The
terminology used in the report is consistent with the literature on AML and is complemented by aglossary that deﬁnes key terms associated with the security of AI systems and is intended to assist
non-expert readers. Taken together, the taxonomy and terminology are meant to inform otherstandards and future practice guides for assessing and managing the security of AI systems, by
establishing a common language and understanding of the rapidly developing AML landscape.

60

Keywords

49
50
51
52
53
54
55
56
57
58

62

artiﬁcial intelligence; machine learning; attack taxonomy; evasion; data poisoning; privacy breach;attack mitigation; data modality; trojan attack, backdoor attack; chatbot.

63

NIST AI Reports (NIST AI)

61

64
65
66
67
68

The National Institute of Standards and Technology (NIST) promotes U.S. innovation and industrial
competitiveness by advancing measurement science, standards, and technology in ways that enhance
economic security and improve our quality of life. Among its broad range of activities, NIST contributes
to the research, standards, evaluations, and data required to advance the development, use, andassurance of trustworthy artiﬁcial intelligence (AI).

i

69
70

71

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

Table of Contents

72

Audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv

73Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv

74

Trademark Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iv

75

How to read this document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

v

76

Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

177

1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

378

2. Attack Classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6

79

2.1. Stages of Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

780

2.2. Attacker Goals and Objectives . . . . . . . . . . . . . . . . . . . . . . . . . .

8

81

2.3. Attacker Capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

982

2.4. Attacker Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

10

83

2.5. Data Modality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1184

3. Evasion Attacks and Mitigations . . . . . . . . . . . . . . . . . . . . . . . . . . .

1385

3.1. White-Box Evasion Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . .

14

86

3.2. Black-Box Evasion Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

87

3.3. Transferability of Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

88

3.4. Mitigations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1789

4. Poisoning Attacks and Mitigations . . . . . . . . . . . . . . . . . . . . . . . . . .

20

90

4.1. Availability Poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

91

4.2. Targeted Poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

92

4.3. Backdoor Poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

93

4.4. Model Poisoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

26

94

5. Privacy Attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

95

5.1. Data Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

96

5.2. Memorization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

97

5.3. Membership Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

29

98

5.4. Model Extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

99

5.5. Property Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

100

5.6. Mitigations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

32

101

6. Discussion and Remaining Challenges . . . . . . . . . . . . . . . . . . . . . . . .

33

ii

102

6.1. Trade-O↵s Between the Attributes of Trustworthy AI . . . . . . . . . . . .

35

103

6.2. Multimodal Models: Are They More Robust? . . . . . . . . . . . . . . . . .

35

104

6.3. Beyond Models and Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

36

105

A. Appendix: Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

62

List of Figures

106

107

Fig. 1.

Taxonomy of attacks on AI systems. . . . . . . . . . . . . . . . . . . .

iii

6

108

Audience110

The intended primary audience for this document includes individuals and groups who areresponsible for designing, developing, deploying, evaluating, and governing AI systems.

111

Background

109

114

This document is a result of an extensive literature review, conversations with experts fromthe area of adversarial machine learning, and research performed by the authors in adversarial machine learning.

115

Trademark Information

116

All trademarks and registered trademarks belong to their respective organizations.

112
113

117
118
119
120
121

122
123
124

125
126

The Information Technology Laboratory (ITL) at NIST develops tests, test methods, reference data, proof of concept implementations, and technical analyses to advance the development and productive use of information technology. ITL’s responsibilities include thedevelopment of management, administrative, technical, and physical standards and guidelines.
This NIST NIST AI report focuses on identifying, addressing, and managing risks associated with adversarial machine learning. While practical guidance1 published by NIST may
serve as an informative reference, this guidance remains voluntary.
The content of this document reﬂects recommended practices. This document is not intended to serve as or supersede existing regulations, laws, or other mandatory guidance.

1 The term ’practice guide,’ ’guide,’ ’guidance’ or the like, in the context of this paper, is a consensus-created,

informative reference intended for voluntary use; it should not be interpreted as equal to the use of the term
’guidance’ in a legal or regulatory context. This document does not establish any legal standard or any otherlegal requirement or defense under any law, nor have the force or effect of law.

iv

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

127

How to read this document

133

This document uses terms such as AI technology, AI system, and AI applications interchangeably. Terms related to the machine learning pipeline, such as ML model or algorithm, are also used interchangeably in this document. Depending on context, the term
“system” may refer to the broader organizational and/or social ecosystem within which thetechnology was designed, developed, deployed, and used instead of the more traditionaluse related to computational hardware or software.

134

Important reading notes:

128
129
130
131
132

135
136

137
138
139
140
141

142

• The document includes a series of blue callout boxes that highlight interesting nuances and important takeaways.
• Terms that are used but not deﬁned/explained in the text are listed and deﬁned in
the G LOSSARY. They are displayed in small caps in the text. Clicking on a word
shown in small caps (e.g., ADVERSARIAL EXAMPLES) takes the reader directly tothe deﬁnition of that term in the Glossary. From there, one may click on the page
number shown at the end of the deﬁnition to return.
Acknowledgments

151

The authors wish to thank colleagues from the U.S. Department of Homeland Security(DHS), National Security Agency (NSA), Federal Bureau of Investigations (FBI), Ofﬁce of
the Director of National Inteligence (ODNI)), the Federal Ofﬁce for Information Security,
Germany (BSI), academia (MIT, Georgia Tech), and industry (Google, Software Engineering Institute) who responded to our call and submitted comments to the draft version ofthis paper. The received comments and suggested references were essential to improving
the paper and the future direction of this work. We also want to thank the many people
who assisted in updating the document, including our NIST colleagues who took the timeto provide their constructive feedback.

152

Author Contributions

153

Authors contributed equally and are listed in alphabetical order.

143
144
145
146
147
148
149
150

v

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

154

155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174

175
176
177
178
179
180
181
182
183
184
185
186
187

188
189
190
191
192
193

Executive Summary
This NIST AI report is intended to be a step toward developing a taxonomy and terminology of adversarial machine learning (AML), which in turn may aid in securing applicationsof artiﬁcial intelligence (AI) against adversarial manipulations of AI systems. The components of an AI system include – at a minimum – the data, model, and processes for training,
testing, and deploying the machine learning (ML) models and the infrastructure required
for using them. The data-driven approach of ML introduces additional security and privacy
challenges in different phases of ML operations besides the classical security and privacy
threats faced by most operational systems. These security and privacy challenges include
the potential for adversarial manipulation of training data, adversarial exploitation of model
vulnerabilities to adversely affect the performance of ML classiﬁcation and regression, andeven malicious manipulations, modiﬁcations or mere interaction with models to exﬁltratesensitive information about people represented in the data or about the model itself. Suchattacks have been demonstrated under real-world conditions, and their sophistication andpotential impact have been increasing steadily. AML is concerned with studying the capabilities of attackers and their goals, as well as the design of attack methods that exploit thevulnerabilities of ML during the development, training, and deployment phase of the ML
life cycle. AML is also concerned with the design of ML algorithms that can withstandthese security and privacy challenges. When attacks are launched with malevolent intent,
the robustness of ML refers to mitigations intended to manage the consequences of suchattacks.
This report adopts the notions of security, resilience, and robustness of ML systems fromthe NIST AI Risk Management Framework [170]. Security, resilience, and robustness aregauged by risk, which is a measure of the extent to which an entity (e.g., a system) is threatened by a potential circumstance or event (e.g., an attack) and the severity of the outcome
should such an event occur. However, this report does not make recommendations on risk
tolerance (the level of risk that is acceptable to organizations or society) because it is highly
contextual and application/use-case speciﬁc. This general notion of risk offers a useful approach for assessing and managing the security, resilience, and robustness of AI systemcomponents. Quantifying these likelihoods is beyond the scope of this document. Correspondingly, the taxonomy of AML is deﬁned with respect to the following four dimensionsof AML risk assessment: (i) learning method and stage of the ML life cycle process when
the attack is mounted, (ii) attacker goals and objectives, (iii) attacker capabilities, (iv) andattacker knowledge of the learning process and beyond.
The spectrum of effective attacks against ML is wide, rapidly evolving, and covers allphases of the ML life cycle – from design and implementation to training, testing, and ﬁnally, to deployment in the real world. The nature and power of these attacks are different
and can exploit not just vulnerabilities of the ML models but also weaknesses of the infrastructure in which the AI systems are deployed. Although AI system components may
also be adversely affected by various unintentional factors, such as design and implemen1

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

194
195
196

197
198
199
200
201
202
203
204
205
206
207
208
209
210

tation ﬂaws and data or algorithm biases, these factors are not intentional attacks. Even
though these factors might be exploited by an adversary, they are not within the scope ofthe literature on AML or this report.
This document deﬁnes a taxonomy of attacks and introduces terminology in the ﬁeld of
AML. The taxonomy is built on a survey of the AML literature and is arranged in a conceptual hierarchy that includes key types of ML methods and life cycle stages of attack,attacker goals and objectives, and attacker capabilities and knowledge of the learning process. The report also provides corresponding methods for mitigating and managing theconsequences of attacks and points out relevant open challenges to take into account in thelife cycle of AI systems. The terminology used in the report is consistent with the literature on AML and is complemented by a glossary that deﬁnes key terms associated withthe security of AI systems in order to assist non-expert readers. Taken together, the taxonomy and terminology are meant to inform other standards and future practice guides forassessing and managing the security of AI systems by establishing a common language andunderstanding for the rapidly developing AML landscape. Like the taxonomy, the terminology and deﬁnitions are not intended to be exhaustive but rather to aid in understanding
key concepts that have emerged in AML literature.

2

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

211

212
213
214
215
216
217
218
219
220

221
222
223
224
225
226
227
228
229
230
231
232
233
234

235
236
237
238
239
240
241
242
243
244
245
246

247
248
249
250

1.

Introduction

Artiﬁcial intelligence (AI) systems [165] are on a global multi-year accelerating expansion
trajectory. These systems are being developed by and widely deployed into the economiesof numerous countries, leading to the emergence of AI-based services for people to use
in many spheres of their lives, both real and virtual [57]. Advances in the generative capabilities of AI in text and images are directly impacting society at unprecedented levels.
As these systems permeate the digital economy and become inextricably essential parts ofdaily life, the need for their secure, robust, and resilient operation grows. These operational attributes are critical elements of Trustworthy AI in the NIST AI Risk Management
Framework [170] and in the taxonomy of AI Trustworthiness [167].
However, despite the signiﬁcant progress that AI and machine learning (ML) have made ina number of different application domains, these technologies are also vulnerable to attacks
that can cause spectacular failures with dire consequences. For example, in computer visionapplications to image classiﬁcation, well-known cases of adversarial perturbations of input
images have caused autonomous vehicles to swerve into the opposite direction lane andthe misclassiﬁcation of stop signs as speed limit signs, the disappearance of critical objects
from images, and even the misidentiﬁcation of people wearing glasses in high-security
settings [76, 116, 194, 207]. Similarly, in the medical ﬁeld where more and more ML
models are being deployed to assist doctors, there is the potential for medical record leaks
from ML models that can expose deeply personal information [8, 103]. Attackers can also
manipulate the training data of ML algorithms, thus making the AI system trained on it
vulnerable to attacks [191]. Scraping of training data from the Internet also opens up the
possibility of hackers poisoning the data to create vulnerabilities that allow for security
breaches down the pipeline.
Large language models (LLMs) [27, 50, 62, 155, 206, 257] are also becoming an integral
part of the Internet infrastructure. LLMs are being used to create more powerful online
search, help software developers write code, and even power chatbots that help with customer service. With the exception of BLOOM [155], most of the companies developing
such models do not release detailed information about the data sets that have been used
to build their language models, but these data sets inevitably include some sensitive personal information, such as addresses, phone numbers, and email addresses. This creates
serious risks for user privacy online. The more often a piece of information appears in adata set, the more likely a model is to leak it in response to random or speciﬁcally designed
queries or prompts. This could perpetuate wrong and harmful associations with damaging consequences for the people involved and bring additional security and safety concerns[34, 148].
As ML models continue to grow in size, many organizations rely on pre-trained models
that could either be used directly for prediction or be ﬁne-tuned with new datasets to enable different predictive tasks. This creates opportunities for malicious modiﬁcations of
pre-trained models by inserting TROJANS to enable attackers to compromise the model
3

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

251

availability, force incorrect processing, or leak the data when instructed [91].

252This report offers guidance for the development of:

253
254

255
256

257
258

259
260

• Standardized terminology in AML to be used by the ML and cybersecurity communities;• A taxonomy of the most widely studied and effective attacks in AML, including
evasion, poisoning, and privacy attacks; and• A discussion of potential mitigations in AML that have withstood the test of time andlimitations of some of the existing mitigations.
As AML is a fast evolving ﬁeld, we envision the need to update the report regularly as new
developments emerge on both the attack and mitigation fronts.
The goal of this report is not to provide an exhaustive survey of all literature on
AML. In fact, this by itself is an almost impossible task as a search on arXiv for
AML articles in 2021 and 2022 yielded more than 5000 references. Rather, thisreport provides a categorization of attacks and their mitigations, starting with thethree main types of attacks: 1) evasion, 2) data and model poisoning, and 3) data
and model privacy.

261

262
263
264
265
266
267
268
269
270
271
272

273
274
275
276
277
278
279
280
281
282

Historically, modality-speciﬁc ML modeling technology has emerged for each input modality (e.g., text, images, speech, tabular data), each of which is susceptible to domain-speciﬁc
attacks. For example, the attack approaches for image classiﬁcation tasks do not directlytranslate to attacks against natural language processing (NLP) models. Recently, the transformer technology from NLP has entered the computer vision domain [68]. In addition,
multimodal ML has made exciting progress in many tasks, and there have been attempts to
use multimodal learning as a potential mitigation of single-modality attacks [245]. However, powerful simultaneous attacks against all modalities in a multimodal model have also
emerged [44, 195, 243]. The report discusses attacks against all viable learning methods(e.g., supervised, unsupervised, semi-supervised, federated learning, reinforcement learning) across multiple data modalities.
Fundamentally, the machine learning methodology used in modern AI systems is susceptible to attacks through the public APIs that the model provides and against the platforms
on which they are deployed. This report focuses on the former and considers the latter tobe out of scope. Attackers can breach the conﬁdentiality and privacy protections of thedata and model by simply exercising the public interfaces of the model and supplying data
inputs that are within the acceptable range. In this sense, the challenges facing AML are
similar to those facing cryptography. Modern cryptography relies on algorithms that are
secure in an information-theoretic sense. Thus, people need to focus only on implementing
them robustly and securely, which is no small task by itself. Unlike cryptography, there are
no information-theoretic security proofs for the widely used machine learning algorithms.
4

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

283
284

285
286
287
288
289
290
291
292

As a result, many of the advances in developing mitigations against different classes ofattacks tend to be empirical in nature.
This report is organized as follows. Section 2 introduces the taxonomy of attacks. The
taxonomy is organized by ﬁrst deﬁning the broad categories of attacker objectives/goals.
Based on that, we deﬁne the categories of capabilities the adversary must be able to leverageto achieve the corresponding objectives. Then, we introduce speciﬁc attack classes foreach type of capability. Sections 3, 4, and 5 discuss the major classes of attacks: evasion,poisoning, and privacy, respectively. A corresponding set of mitigations for each class of
attacks is provided in the attack class sections. Section 6 discusses the remaining challenges
in the ﬁeld.

5

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

293

294
295
296
297
298
299
300
301
302

2.

Attack Classiﬁcation

Figure 1 introduces a taxonomy of attacks in adversarial machine learning. The attacker’s
objectives are shown as disjointed circles with the attacker’s goal at the center of eachcircle: Availability breakdown, Integrity violations, and Privacy compromise. The capabilities that an adversary must leverage to achieve their objectives are shown in the outerlayer of the objective circles. Attack classes are shown as callouts connected to the capabilities required to mount each attack. Multiple attack classes that requiring same capabilities
for reaching the same objective are shown in a single callout. Related attack classes that
require different capabilities for reaching the same objective are connected with dottedlines.
Clean-Label
Poisoning

Model Poisoning

Energy-Latency

Availability

Clean-Label
Poisoning

Evasion

Data Poisoning

Clean-Label
Backdoor
Model
Poisoning

Black-Box
EvasionIntegrity

Backdoor
Poisoning

Targeted
Poisoning

Model Extraction
Model

Privacy
Data

Reconstruction;
Memorization;
Membership Inference;
Property Inference;

Fig. 1. Taxonomy of attacks on AI systems.

6

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

308

These attacks are classiﬁed according to the following dimensions: 1) learning method andstage of the learning process when the attack is mounted, 2) attacker goals and objectives, 3)
attacker capabilities, and 4) attacker knowledge of the learning process. Several adversarial
attack classiﬁcation frameworks have been introduced in prior works [23, 212], and the goal
here is to create a standard terminology for adversarial attacks on ML that uniﬁes existing
work.

309

2.1.

303
304
305
306
307

310
311
312
313
314
315
316
317

318
319
320
321
322

323
324
325
326
327
328
329
330

331
332
333
334
335
336
337
338

339
340

Stages of Learning

Machine learning involves a TRAINING STAGE, in which a model is learned, and a DEPLOYMENT STAGE , in which the model is deployed on new, unlabeled data samples to generatepredictions. In the case of SUPERVISED LEARNING in the training stage labeled training
data is given as input to a training algorithm and the ML model is optimized to minimize a
speciﬁc loss function. Validation and testing of the ML model is usually performed before
the model is deployed in the real world. Common supervised learning techniques include
CLASSIFICATION , in which the predicted labels or classes are discrete, and LOGISTIC RE GRESSION , in which the predicted labels or response variables are continuous.
ML models may be GENERATIVE (i.e., learn the distribution of training data and generate similar examples, such as generative adversarial metworks [GAN] and large language
models [LLM]) or DISCRIMINATIVE (i.e., learn only a decision boundary, such as LO GISTIC REGRESSION , S UPPORT V ECTOR M ACHINES , and CONVOLUTIONAL NEURALNETWORKS ).
Other learning paradigms in the ML literature are UNSUPERVISED LEARNING, which trains
models using unlabeled data at training time; SEMI - SUPERVISED LEARNING, in which asmall set of examples have labels, while the majority of samples are unlabeled; REIN FORCEMENT LEARNING , in which an agent interacts with an environment and learns an
optimal policy to maximize its reward; FEDERATED LEARNING, in which a set of clients
jointly train an ML model by communicating with a server, which performs an aggregationof model updates; ENSEMBLE LEARNING which is an approach in machine learning thatseeks better predictive performance by combining the predictions from multiple models.
Adversarial machine learning literature predominantly considers adversarial attacks against
AI systems that could occur at either the training stage or the ML deployment stage. During
the ML training stage, the attacker might control part of the training data, their labels, themodel parameters, or the code of ML algorithms, resulting in different types of poisoning
attacks. During the ML deployment stage, the ML model is already trained, and the adversary could mount evasion attacks to create integrity violations and change the ML model’s
predictions, as well as privacy attacks to infer sensitive information about the training dataor the ML model.
Training-time attacks. Attacks during the ML training stage are called POISONING ATTACKS [21]. In a DATA POISONING attack [21, 94], an adversary controls a subset of the7

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

341
342
343
344
345
346

training data by either inserting or modifying training samples. In a MODEL POISONING attack [138], the adversary controls the model and its parameters. Data poisoning attacks areapplicable to all learning paradigms, while model poisoning attacks are most prevalent infederated learning [118], where clients send local model updates to the aggregating server,and in supply-chain attacks where malicious code may be added to the model by suppliersof model technology.

353

Deployment-time attacks. Two different types of attacks can be mounted at testing/deployment
time. First, evasion attacks modify testing samples to create ADVERSARIAL EXAMPLES [19,93, 216], which are similar to the original sample (according to certain distance metrics)
but alter the model predictions to the attacker’s choices. Second, privacy attacks, such as
membership inference [200] and data reconstruction [67], are typically mounted by attackers with query access to an ML model. They could be further divided into data privacy
attacks and model privacy attacks.

354

2.2.

347
348
349
350
351
352

355
356
357
358
359

360
361
362
363
364
365
366
367
368
369

370
371
372
373
374
375
376
377
378
379

Attacker Goals and Objectives

The attacker’s objectives are classiﬁed along three dimensions according to the three maintypes of security violations considered when analyzing the security of a system (i.e., availability, integrity, conﬁdentiality): availability breakdown, integrity violations, and privacy
compromise. Figure 1 separates attacks into three disjointed circles according to their objective, and the attacker’s objective is shown at the center of each circle.
Availability Breakdown. An AVAILABILITY ATTACK is an indiscriminate attack against
ML in which the attacker attempts to break down the performance of the model at testing/deployment time. Availability attacks can be mounted via data poisoning, when theattacker controls a fraction of the training set; via model poisoning, when the attacker controls the model parameters; or as energy-latency attacks via query access. Data poisoning
availability attacks have been proposed for S UPPORT V ECTOR M ACHINES [21], linear regression [110], and even neural networks [141, 161], while model poisoning attacks havebeen designed for neural networks [138] and federated learning [6]. Recently, ENERGYLATENCY ATTACKS that require only black-box access to the model have been developed
for neural networks across many different tasks in computer vision and NLP [203].
Integrity Violations. An INTEGRITY ATTACK targets the integrity of an ML model’s output, resulting in incorrect predictions performed by an ML model. An attacker can cause an
integrity violation by mounting an evasion attack at testing/deployment time or a poisoning
attack at training time. Evasion attacks require the modiﬁcation of testing samples to create
adversarial examples that are mis-classiﬁed by the model to a different class, while remaining stealthy and imperceptible to humans [19, 93, 216]. Integrity attacks via poisoning
can be classiﬁed as TARGETED POISONING ATTACKS [89, 193], BACKDOOR POISONINGATTACKS [94], and MODEL POISONING [6, 17, 78]. Targeted poisoning tries to violate theintegrity of a few targeted samples and assumes that the attacker has training data control
to insert the poisoned samples. Backdoor poisoning attacks require the generation of a8

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

380
381
382
383
384
385

BACKDOOR PATTERN , which is added to both the poisoned samples and the testing sam-ples to cause misclassiﬁcation. Backdoor attacks are the only attacks in the literature that
require both training and testing data control. Model poisoning attacks could result in either targeted or backdoor attacks, and the attacker modiﬁes model parameters to cause an
integrity violation. They have been designed for centralized learning [138] and federated
learning [6, 17].

394

Privacy Compromise. Attackers might be interested in learning information about thetraining data (resulting in DATA PRIVACY attacks) or about the ML model (resulting in
MODEL PRIVACY attacks). The attacker could have different objectives for compromising the privacy of training data, such as DATA RECONSTRUCTION [67] (inferring content
or features of training data), MEMBERSHIP - INFERENCE ATTACKS [99, 201] (inferring thepresence of data in the training set), data MEMORIZATION [33, 34] (ability to extract training data from generative models), and PROPERTY INFERENCE [86] (inferring properties
about the training data distribution). MODEL EXTRACTION is a model privacy attack inwhich attackers aim to extract information about the model [108].

395

2.3.

386
387
388
389
390
391
392
393

396
397

398
399
400

401
402
403

404
405
406

407
408
409
410
411

412
413
414

415
416

Attacker Capabilities

An adversary might leverage six types of capabilities to achieve their objectives, as shownin the outer layer of the objective circles in Figure 1:• TRAINING DATA CONTROL: The attacker might take control of a subset of the training data by inserting or modifying training samples. This capability is used in datapoisoning attacks (e.g., availability poisoning, targeted or backdoor poisoning).
• MODEL CONTROL: The attacker might take control of the model parameters by either
generating a Trojan trigger and inserting it in the model or by sending malicious local
model updates in federated learning.
• TESTING DATA CONTROL: The attacker may utilize this to add perturbations to testing samples at model deployment time, as performed in evasion attacks to generate
adversarial examples or in backdoor poisoning attacks.
• LABEL LIMIT: This capability is relevant to restrict the adversarial control over thelabels of training samples in supervised learning. Clean-label poisoning attacks assume that the attacker does not control the label of the poisoned samples – a realistic
poisoning scenario, while regular poisoning attacks assume label control over the
poisoned samples.
• SOURCE CODE CONTROL: The attacker might modify the source code of the ML
algorithm, such as the random number generator or any third-party libraries, which
are often open source.
• QUERY ACCESS: When the ML model is managed by a cloud provider (using Machine Learning as a Service – MLaaS), the attacker might submit queries to the model
9

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

and receive predictions (either labels or model conﬁdences). This capability is usedby black-box evasion attacks, energy-latency attacks, and all privacy attacks.

417
418

419
420
421

Note that even if an attacker does not have the ability to modify training/testing data, source
code, or model parameters, access to these are still crucial for mounting white-box attacks.
See Section 2.4 for more details on attacker knowledge.

427

Figure 1 connects each attack class with the capabilities required to mount the attack. For
instance, backdoor attacks that cause integrity violations require control of training data andtesting data to insert the backdoor pattern. Backdoor attacks can also be mounted via sourcecode control, particularly when training is outsourced to a more powerful entity. Cleanlabel backdoor attacks do not allow label control on the poisoned samples, in addition to
the capabilities needed for backdoor attacks.

428

2.4.

422
423
424
425
426

429
430

431
432
433
434
435
436
437

438
439
440
441
442

443
444
445
446
447
448
449
450

Attacker Knowledge

Another dimension for attack classiﬁcation is how much knowledge the attacker has about
the ML system. There are three main types of attacks: white-box, black-box, and gray-box.
White-box attacks. These assume that the attacker operates with full knowledge about theML system, including the training data, model architecture, and model hyper-parameters.
While these attacks operate under very strong assumptions, the main reason for analyzing
them is to test the vulnerability of a system against worst-case adversaries and to evaluate
potential mitigations. Note that this deﬁnition is more general and encompasses the notionof adaptive attacks where the knowledge of the mitigations applied to the model or thesystem is explicitly tracked.
Black-box attacks. These attacks assume minimal knowledge about the ML system. An
adversary might get query access to the model, but they have no other information about
how the model is trained. These attacks are the most practical since they assume that theattacker has no knowledge of the AI system and utilize system interfaces readily available
for normal use.
Gray-box attacks. There are a range of gray-box attacks that capture adversarial knowledge between black-box and white-box attacks. Suciu et al. [212] introduced a framework
to classify gray-box attacks. An attacker might know the model architecture but not its parameters, or the attacker might know the model and its parameters but not the training data.
Other common assumptions for gray-box attacks are that the attacker has access to data
distributed identically to the training data and knows the feature representation. The latterassumption is important in applications where feature extraction is used before training an
ML model, such as cybersecurity, ﬁnance, and healthcare.

10

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

451

2.5.

Data Modality

455

Adversarial attacks against ML have been discovered in a range of data modalities used inmany application domains. Until recently, most attacks and defenses have operated undera single modality, but a new ML trend is to use multimodal data. The taxonomy of attacks
deﬁned in Figure 1 is independent of the modality of the data in speciﬁc applications.

456

The most common data modalities in the adversarial ML literature include:

452
453
454

457
458
459
460

461
462
463
464

465
466
467
468
469
470
471
472
473
474

475
476
477
478
479
480
481
482
483
484
485
486
487

1. Image: Adversarial examples of image data modality [93, 216] have the advantage
of a continuous domain, and gradient-based methods can be applied directly for optimization. Backdoor poisoning attacks were ﬁrst invented for images [94], and many
privacy attacks are run on image datasets (e.g., [200]).
2. Text: Natural language processing (NLP) is a popular modality, and all classes ofattacks have been proposed for NLP applications, including evasion [96], poisoning [48, 132], and privacy [252]. Audio systems and text generated from audio signals have also been attacked [37].
3. Cybersecurity2 : The ﬁrst poisoning attacks were discovered in cybersecurity forworm signature generation (2006) [177] and spam email classiﬁcation (2008) [166].
Since then, poisoning attacks have been shown for malware classiﬁcation, malicious
PDF detection, and Android malicious app classiﬁcation [192]. Evasion attacks
against the same data modalities have been proposed as well: malware classiﬁcation [63, 211], PDF malware classiﬁcation [209, 242], and Android malicious app
detection [179]. Clements et al. [58] developed a mechanism for effective generationof evasion attacks on small, weak routers in network intrusion detection. Poisoning unsupervised learning models has been shown for clustering used in malware
classiﬁcation [22] and network trafﬁc anomaly detection [185].
Industrial Control Systems (ICS) and Supervisory Control and Data Acquisition(SCADA) systems are part of modern Critical Infrastructure (CI) such as power grids,
power plants (nuclear, fossil fuel, renewable energy), water treatment plants, oil reﬁneries, etc. ICS are an attractive target for adversaries because of the potential forhighly consequential disruptions of CI [38, 128]. The existence of targeted stealthattacks has led to the development of defense-in-depth mechanisms for their detection and mitigation. Anomaly detection based on data-centric approaches allows
automated feature learning through ML algorithms. However, the application of ML
to such problems comes with speciﬁc challenges related to the need for a very low
false negative and low false positive rates, ability to catch zero-day attacks, accountfor plant operational drift, etc. This challenge is compounded by the fact that trying to accommodate all these together makes ML models susceptible to adversarial
attacks [123, 180, 264].
2 Strictly speaking, cybersecurity data may not include a single modality, but rather multiple modalities suchas network-level, host-level, or program-level data.

11

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

488
489
490
491
492

493
494
495
496
497

4. Tabular data: Numerous attacks against ML models working on tabular data in ﬁnance, business, and healthcare applications have been demonstrated. For example,
poisoning availability attacks have been shown against healthcare and business applications [110]; privacy attacks have been shown against healthcare data [249]; andevasion attacks have been shown against ﬁnancial applications [90].
Recently, the use of ML models trained on multimodal data has gained traction, particularly the combination of image and text data modalities. Several papers have shown thatmultimodal models may provide some resilience against attacks [245], but other papers
show that multimodal models themselves could be vulnerable to attacks mounted on allmodalities at the same time [44, 195, 243]. See Section 6.2 for additional discussion.
An interesting open challenge is to test and characterize the resilience of a variety
of multimodal ML against evasion, poisoning, and privacy attacks.

498

12

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

499

500
501
502
503
504
505
506
507
508
509
510
511
512
513
514

515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531

532
533
534
535
536
537
538

3.

Evasion Attacks and Mitigations

The discovery of evasion attacks against machine learning models has generated increased
interest in adversarial machine learning, leading to signiﬁcant growth in this research spaceover the last decade. In an evasion attack, the adversary’s goal is to generate adversarial examples, which are deﬁned as testing samples whose classiﬁcation can be changed at
deployment time to an arbitrary class of the attacker’s choice with only minimal perturbation [216]. Early known instances of evasion attacks date back to 1988 with the workof Kearns and Li [120], and to 2004, when Dalvi et al. [61], and Lowd and Meek [140]demonstrated the existence of adversarial examples for linear classiﬁers used in spam ﬁlters. Adversarial examples became even more intriguing to the research community when
Szedegy et al. [216] showed that deep neural networks used for image classiﬁcation canbe easily manipulated, and adversarial examples were visualized. In the context of image
classiﬁcation, the perturbation of the original sample must be small so that a human cannot
observe the transformation of the input. Therefore, while the ML model can be tricked toclassify the adversarial example in the target class selected by the attacker, humans stillrecognize it as part of the original class.
In 2013, Szedegy et al. [216] and Biggio et al. [19] independently discovered an effective
method for generating adversarial examples against linear models and neural networks byapplying gradient optimization to an adversarial objective function. Both of these techniques require white-box access to the model and were improved by subsequent methods
that generated adversarial examples with even smaller perturbations [5, 36, 144]. Adversarial examples are also applicable in more realistic black-box settings in which attackers only
obtain query access capabilities to the trained model. Even in the more challenging blackbox setting in which attackers obtain the model’s predicted labels or conﬁdence scores,
deep neural networks are still vulnerable to adversarial examples. Methods for creating
adversarial examples in black-box settings include zeroth-order optimization [47], discrete
optimization [156], and Bayesian optimization [202], as well as transferability, which involves the white-box generation of adversarial examples on a different model architecture
before transferring them to the target model [173, 174, 223]. Cybersecurity and image
classiﬁcations were the ﬁrst application domains that showcased evasion attacks. However,with the increasing interest in adversarial machine learning, ML technology used in manyother application domains went under scrutiny, including speech recognition [37], natural
language processing [115], and video classiﬁcation [134, 236].
Mitigating adversarial examples is a well-known challenge in the community and deserves
additional research and investigation. The ﬁeld has a history of publishing defenses evaluated under relatively weak adversarial models that are subsequently broken by more powerful attacks, a process that appears to iterate in perpetuity. Mitigations need to be evaluatedagainst strong adaptive attacks, and guidelines for the rigorous evaluation of newly proposed mitigation techniques have been established [60, 221]. The most promising directions for mitigating the critical threat of evasion attacks are adversarial training [93, 144]
13

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

539
540
541
542
543
544
545
546

(iteratively generating and inserting adversarial examples with their correct labels at training time); certiﬁed techniques, such as randomized smoothing [59] (evaluating ML prediction under noise); and formal veriﬁcation techniques [88, 119] (applying formal method
techniques to verify the model’s output). Nevertheless, these methods come with different
limitations, such as decreased accuracy for adversarial training and randomized smoothing,and computational complexity for formal methods. There is an inherent trade-off between
robustness and accuracy [220, 225, 255]. Similarly, there are trade-offs between a model’srobustness and fairness guarantees [41].

548This section discusses white-box and black-box evasion attack techniques, attack transferability, and the potential mitigation of adversarial examples in more detail.

549

3.1.

547

550
551
552
553
554

555
556
557
558
559
560

561
562
563
564
565
566
567

568
569
570
571
572
573
574

575
576

White-Box Evasion Attacks

There are several optimization-based methods for designing evasion attacks that generate
adversarial examples at small distances from the original testing samples. There are also
several choices for distance metrics, universal evasion attacks, and physically realizableattacks, as well as examples of evasion attacks developed for multiple data modalities,
including NLP, audio, video, and cybersecurity domains.
Optimization-based methods. Szedegy et al. [216] and Biggio et al. [19] independently
proposed the use of optimization techniques to generate adversarial examples. In theirthreat models, the adversary is allowed to inspect the entirety of the ML model and compute gradients relative to the model’s loss function. These attacks can be targeted, in which
the adversarial example’s class is selected by the attacker, or untargeted, in which the adversarial examples are misclassiﬁed to any other incorrect class.
Szedegy et al. [216] coined the widely used term adversarial examples. They considered
an objective that minimized the `2 norm of the perturbation, subject to the model prediction changing to the target class. The optimization is solved using the Limited-memory
Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) method. Biggio et al. [19] considered thesetting of a binary classiﬁer with malicious and benign classes with continuous and differentiable discriminant function. The objective of the optimization is to minimize thediscriminant function in order to generate adversarial examples of maximum conﬁdence.
While Biggio et al. [19] apply their method to linear classiﬁers, kernel SVM, and multilayer perceptrons, Szedegy et al. [216] show the existence of adversarial examples on deeplearning models used for image classiﬁcation. Goodfellow et al. [93] introduced an efﬁcient method for generating adversarial examples for deep learning: the Fast GradientSign Method (FGSM), which performs a single iteration of gradient descent for solving the
optimization. This method has been extended to an iterative FGSM attack by Kurakin etal. [125].
Subsequent work on generating adversarial examples have proposed new objectives andmethods for optimizing the generation of adversarial examples with the goals of minimizing
14

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

577

578
579

580
581
582
583
584

585
586
587
588

589
590
591
592
593
594
595

596
597
598
599
600
601
602
603
604

605
606
607
608
609
610
611
612

613
614
615

the perturbations and supporting multiple distance metrics. Some notable attacks include:
1. DeepFool is an untargeted evasion attack for `2 norms, which uses a linear approximation of the neural network to construct the adversarial examples [158].
2. The Carlini-Wagner attack uses multiple objectives that minimize the loss or logits
on the target class and the distance between the adversarial example and original
sample. The attack is optimized via the penalty method [36] and considers three
distance metrics to measure the perturbations of adversarial examples: `0 , `2 , and `• .
The attack has been effective against the defensive distillation defense [175].
3. The Projected Gradient Descent (PGD) attack [144] minimizes the loss function andprojects the adversarial examples to the space of allowed perturbations at each iteration of gradient descent. PGD can be applied to the `2 and `• distance metrics formeasuring the perturbation of adversarial examples.
Universal evasion attacks. Moosavi-Dezfooli et al. [157] showed how to construct small
universal perturbations (with respect to some norm), which can be added to most images
and induce a misclassiﬁcation. Their technique relies on successive optimization of the universal perturbation using a set of points sampled from the data distribution. An interestingobservation is that the universal perturbations generalize across deep network architectures,
suggesting similarity in the decision boundaries trained by different models for the sametask.
Physically realizable attacks. These are attacks against machine learning systems that
become feasible in the physical world. One of the ﬁrst physically realizable attacks in theliterature is the attack on facial recognition systems by Sharif et al. [194]. The attack canbe realized by printing a pair of eyeglass frames, which misleads facial recognition systems
to either evade detection or impersonate another individual. Eykholt et al. [77] proposed anattack to generate robust perturbations under different conditions, resulting in adversarial
examples that can evade vision classiﬁers in various physical environments. The attack isapplied to evade a road sign detection classiﬁer by physically applying black and white
stickers to the road signs.
Other data modalities. In computer vision applications, adversarial examples must be
imperceptible to humans. Therefore, the perturbations introduced by attackers need to beso small that a human correctly recognizes the images, while the ML classiﬁer is tricked
into changing its prediction. The concept of adversarial examples has been extended toother domains, such as audio, video, natural language processing (NLP), and cybersecurity.
In some of these settings, there are additional constraints that need to be respected byadversarial examples, such as text semantics in NLP and the application constraints incybersecurity. Several representative works are discussed below:• Audio: Carlini and Wagner [37] showed a targeted attack on models that generate
text from speech. They can generate an audio waveform that is very similar to an
existing one but that can be transcribed to any text of the attacker’s choice.
15

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

616
617
618
619

620
621
622
623
624
625
626
627

628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643

644

645
646
647
648
649
650
651

652
653
654

• Video: Adversarial evasion attacks against video classiﬁcation models can be split
into sparse attacks that perturb a small number of video frames [236] and dense
attacks that perturb all of the frames in a video [134]. The goal of the attacker is tochange the classiﬁcation label of the video.
• NLP: Jia and Liang [115] developed a methodology for generating adversarial NLP
examples. This pioneering work was followed by many advances in developing adversarial attacks on NLP models (see a comprehensive survey on the topic [259]).
Recently, La Malfa and Kwiatkowska [126] proposed a method for formalizing perturbation deﬁnitions in NLP by introducing the concept of semantic robustness. The
main challenges in NLP are that the domain is discrete rather than continuous (e.g.,
image, audio, and video classiﬁcation), and adversarial examples need to respect text
semantics.
• Cybersecurity: In cybersecurity applications, adversarial examples must respect theconstraints imposed by the application semantics and feature representation of cyberdata, such as network trafﬁc or program binaries. FENCE is a general framework forcrafting white-box evasion attacks using gradient optimization in discrete domains
and supports a range of linear and statistical feature dependencies [53]. FENCE
has been applied to two network security applications: malicious domain detection
and malicious network trafﬁc classiﬁcation. Sheatsley et al. [196] propose a methodthat learns the constraints in feature space using formal logic and crafts adversarial examples by projecting them onto a constraint-compliant space. They apply thetechnique to network intrusion detection and phishing classiﬁers. Both papers observe that attacks from continuous domains cannot be readily applied in constrained
environments, as they result in infeasible adversarial examples. Pierazzi et al. [179]discuss the difﬁculty of mounting feasible evasion attacks in cyber security due toconstraints in feature space and the challenge of mapping attacks from feature space
to problem space. They formalize evasion attacks in problem space and constructfeasible adversarial examples for Android malware.
3.2.

Black-Box Evasion Attacks

Black-box evasion attacks are designed under a realistic adversarial model, in which theattacker has no prior knowledge of the model architecture or training data. Instead, theadversary can interact with a trained ML model by querying it on various data samples andobtaining the model’s predictions. Similar APIs are provided by machine learning as a service (MLaaS) offered by public cloud providers, in which users can obtain the model’s predictions on selected queries without information about how the model was trained. There
are two main classes of black-box evasion attacks in the literature:• Score-based attacks: In this setting, attackers obtain the model’s conﬁdence scoresor logits and can use various optimization techniques to create the adversarial examples. A popular method is zeroth-order optimization, which estimates the model’s
16

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

gradients without explicitly computing derivatives [47, 105]. Other optimization
techniques include discrete optimization [156], natural evolution strategies [104],
and random walks [162].

655
656
657

658
659
660
661
662
663
664
665

• Decision-based attacks: In this more restrictive setting, attackers obtain only theﬁnal predicted labels of the model. The ﬁrst method for generating evasion attacks
was the Boundary Attack based on random walks along the decision boundary andrejection sampling [25], which was extended with an improved gradient estimation to
reduce the number of queries in the HopSkipJumpAttack [46]. More recently, several
optimization methods search for the direction of the nearest decision boundary (theOPT attack [51]), use sign SGD instead of binary searches (the Sign-OPT attack[52]), or use Bayesian optimization [202].
The main challenge in creating adversarial examples in black-box settings is reducing the number of queries to the ML models. Recent techniques can successfully evade the ML classiﬁers with a relatively small number of queries, typicallyless than 1000 [202].

666

667

668
669
670
671
672
673
674

3.3.

Transferability of Attacks

Another method for generating adversarial attacks under restrictive threat models is via
transferability of an attack crafted on a different ML model. Typically, an attacker trainsa substitute ML model, generates white-box adversarial attacks on the substitute model,
and transfers the attacks to the target model. Various methods differ in how the substitutemodels are trained. For example, Papernot et al. [173, 174] train the substitute model withscore-based queries to the target model, while several papers train an ensemble of models
without explicitly querying the target model [136, 223, 235].

681

Attack transferability is an intriguing phenomenon, and existing literature attempts to understand the fundamental reasons why adversarial examples transfer across models. Several
papers have observed that different models learn intersecting decision boundaries in bothbenign and adversarial dimensions, which leads to better transferability [93, 157, 223].
Demontis et al. [64] identiﬁed two main factors that contribute to attack transferability forboth evasion and poisoning: the intrinsic adversarial vulnerability of the target model andthe complexity of the surrogate model used to optimize the attack.

682

3.4.

675
676
677
678
679
680

683
684
685
686
687

Mitigations

Mitigating evasion attacks is challenging because adversarial examples are widespread ina variety of ML model architectures and application domains, as discussed above. Possible explanations for the existence of adversarial examples are that ML models rely on
non-robust features that are not aligned with human perception in the computer vision domain [106]. In the last few years, many of the proposed mitigations against adversarial
17

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

688
689

690
691
692
693
694

695
696
697
698
699

700
701
702
703

704
705
706

707
708
709
710
711
712
713
714
715

716
717
718
719
720
721
722
723

724
725
726

examples have been ineffective against stronger attacks. Furthermore, several papers have
performed extensive evaluations and defeated a large number of proposed mitigations:• Carlini and Wagner showed how to bypass 10 methods for detecting adversarial examples and described several guidelines for evaluating defenses [35]. Recent workshows that detecting adversarial examples is as difﬁcult as building a defense [219].
Therefore, this direction for mitigating adversarial examples is similarly challenging
when designing defenses.
• The Obfuscated Gradients attack [5] was speciﬁcally designed to defeat several proposed defenses that mask the gradients using the `0 and `• distance metrics. It relies
on a new technique, Backward Pass Differentiable Approximation, which approximates the gradient during the backward pass of backpropagation. It bypasses sevenproposed defenses.
• Tramèr et al. [221] described a methodology for designing adaptive attacks against
proposed defenses and circumvented 13 existing defenses. They advocate designing adaptive attacks to test newly proposed defenses rather than merely testing thedefenses against well-known attacks.
From the wide range of proposed defenses against adversarial evasion attacks, three mainclasses have proved resilient and have the potential to provide mitigation against evasionattacks:
1. Adversarial training: Introduced by Goodfellow et al. [93] and further developed by
Madry et al. [144], adversarial training is a general method that augments the training
data with adversarial examples generated iteratively during training using their correct labels. The stronger the adversarial attacks for generating adversarial examples
are, the more resilient the trained model becomes. Interestingly, adversarial training
results in models with more semantic meaning than standard models [225], but thisbeneﬁt usually comes at the cost of decreased model accuracy on clean data. Additionally, adversarial training is expensive due to the iterative generation of adversarial
examples during training.
2. Randomized smoothing: Proposed by Lecuyer et al. [129] and further improved by
Cohen et al. [59], randomized smoothing is a method that transforms any classiﬁer
into a certiﬁable robust smooth classiﬁer by producing the most likely predictions
under Gaussian noise perturbations. This method results in provable robustness for `2
evasion attacks, even for classiﬁers trained on large-scale datasets, such as ImageNet.
Randomized smoothing typically provides certiﬁed prediction to a subset of testing
samples (the exact number depends on the radius of the `2 ball and the characteristicsof the training data and model).
3. Formal veriﬁcation: Another method for certifying the adversarial robustness ofa neural network is based on techniques from FORMAL METHODS. Reluplex uses
satisﬁability modulo theories (SMT) solvers to verify the robustness of small feed18

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

727
728
729
730
731
732
733

734
735
736

forward neural networks [119]. AI2 is the ﬁrst veriﬁcation method applicable toconvolutional neural networks using abstract interpretation techniques [88]. Thesemethods have been extended and scaled up to larger networks in follow-up veriﬁcation systems, such as DeepPoly [204], ReluVal [233], and Fast Geometric Projections(FGP) [85]. Formal veriﬁcation techniques have signiﬁcant potential for certifying
neural network robustness, but their main limitations are their lack of scalability,
computational cost, and restriction in the type of supported operations.
All of these proposed mitigations exhibit inherent trade-offs between robustness and accuracy, and they come with additional computational costs during training. Therefore, designing ML models that resist evasion while maintaining accuracy remains an open problem.

19

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

737

738
739
740
741
742
743
744
745
746
747
748

4.

Poisoning Attacks and Mitigations

Another relevant threat against machine learning systems is the risk of adversaries mounting poisoning attacks, which are broadly deﬁned as adversarial attacks during the training
stage of the ML algorithm. Poisoning attacks have a long history in cybersecurity, as theﬁrst known poisoning attack was developed for worm signature generation in 2006 [177].
Since then, poisoning attacks have been studied extensively in several application domains:computer security (for spam detection [166]), network intrusion detection [227], vulnerability prediction [187], malware classiﬁcation [192, 240]), computer vision [89, 94, 193],
natural language processing [48, 132, 229], and tabular data in healthcare and ﬁnancial
domains [110]. Recently, poisoning attacks have gained more attention in industrial applications as well. A Microsoft report revealed that they are considered to be the most criticalvulnerability of machine learning systems deployed in production [124].

762

Poisoning attacks are very powerful and can cause either an availability violation or an
integrity violation. In particular, availability poisoning attacks cause indiscriminate degradation of the machine learning model on all samples, while targeted and backdoor poisoning attacks are stealthier and induce integrity violations on a small set of target samples.
Poisoning attacks leverage a wide range of adversarial capabilities, such as data poisoning,model poisoning, label control, source code control, and test data control, resulting in several subcategories of poisoning attacks. They have been developed in white-box adversarial
scenarios [21, 110, 240], gray-box settings [110], and black-box models [20]. This section
discusses the threat of availability poisoning, targeted poisoning, backdoor poisoning, andmodel poisoning attacks classiﬁed according to their adversarial objective. For each poisoning attack category, techniques for mounting the attacks as well as existing mitigations
and their limitations are also discussed. Our classiﬁcation of poisoning attacks is inspiredby the framework developed by Cinà et al. [56], which includes additional references topoisoning attacks and mitigations.

763

4.1.

749
750
751
752
753
754
755
756
757
758
759
760
761

764
765
766
767
768
769
770
771
772
773
774
775

Availability Poisoning

The ﬁrst poisoning attacks discovered in cybersecurity applications were availability attacks against worm signature generation and spam classiﬁers, which indiscriminately impact the entire machine learning model and, in essence, cause a denial-of-service attackon users of the AI system. Perdisci et al. [177] generated suspicious ﬂows with fake invariants that mislead the worm signature generation algorithm in Polygraph [168]. Nelson
et al. [166] designed poisoning attacks against Bayes-based spam classiﬁers, which generate spam emails that contain long sequences of words appearing in legitimate emails to
induce the misclassiﬁcation of spam emails. Both of these attacks were conducted under
the white-box setting in which adversaries are aware of the ML training algorithm, feature
representations, training datasets, and ML models. ML-based methods have been proposed
for the detection of cybersecurity attacks targeting ICS. Such detectors are often retrainedusing data collected during system operation to account for plant operational drift of the20

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

776
777
778

779
780
781
782
783
784
785
786
787
788
789
790
791

792
793
794
795
796
797
798
799
800

monitored signals. This retraining procedure creates opportunities for an attacker to mimic
the signals of corrupted sensors at training time and poison the learning process of thedetector such that attacks remain undetected at deployment time [123].
A simple black-box poisoning attack strategy is LABEL FLIPPING, which generates training examples with a victim label selected by the adversary [20]. This method requires alarge percentage of poisoning samples for mounting an availability attack, and it has been
improved via optimization-based poisoning attacks introduced for the ﬁrst time against
S UPPORT V ECTOR M ACHINES (SVM) [21]. In this approach, the attacker solves a bilevel
optimization problem to determine the optimal poisoning samples that will achieve theadversarial objective (i.e., maximize the hinge loss for SVM [21] or maximize the meansquare error [MSE] for regression [110]). These optimization-based poisoning attacks havebeen subsequently designed against linear regression [110] and neural networks [161], andthey require white-box access to the model and training data. In gray-box adversarial settings, the most popular method for generating availability poisoning attacks is transferability, in which poisoning samples are generated for a surrogate model and transferred to thetarget model [64, 212].
A realistic threat model for supervised learning is that of clean-label poisoning attacks inwhich adversaries can only control the training examples but not their labels. This casemodels scenarios in which the labeling process is external to the training algorithm, as
in malware classiﬁcation where binary ﬁles can be submitted by attackers to threat intelligence platforms, and labeling is performed using anti-virus signatures or other external
methods. Clean-label availability attacks have been introduced for neural network classiﬁers by training a generative model and adding noise to training samples to maximize theadversarial objective [82]. A different approach for clean-label poisoning is to use gradientalignment and minimally modify the training data [83].

805

Availability poisoning attacks have also been designed for unsupervised learning against
centroid-based anomaly detection [121] and behavioral clustering for malware [22]. In
federated learning, an adversary can mount a model poisoning attack to induce availability
violations in the globally trained model [78, 197, 198]. More details on model poisoning
attacks are provided in Section 4.4.

806

Mitigations.

801
802
803
804

813

Availability poisoning attacks are usually detectable by monitoring the standard performance metrics of ML models – such as precision, recall, accuracy, F1 scores, and area
under the curve – as they cause a large degradation in the classiﬁer metrics. Nevertheless,detecting these attacks during the testing or deployment stages of ML is less desirable, andexisting mitigations aim to proactively prevent these attacks during the training stage togenerate robust ML models. Among the existing mitigations, some generally promisingtechniques include:814• Training data sanitization: These methods leverage the insight that poisoned sam-

807
808
809
810
811
812

21

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

ples are typically different than regular training samples not controlled by adversaries. As such, data sanitization techniques are designed to clean the training set
and remove the poisoned samples before the machine learning training is performed.
Nelson et al. [166] propose the Region of Non-Interest (RONI) method, which examines each sample and excludes it from training if the accuracy of the model decreases when the sample is added. Subsequently proposed sanitization methods improved upon this early approach by reducing its computational complexity. Paudice
et al. [176] introduced a method for label cleaning that was speciﬁcally designed
for label ﬂipping attacks. Steinhardt et al. [210] propose the use of outlier detection
methods for identifying poisoned samples. Clustering methods have also been used
for detecting poisoned samples [127, 217]. In the context of network intrusion detection, computing the variance of predictions made by an ensemble of multiple ML
models has proven to be an effective data sanitization method [227]. Once sanitized,
the datasets should be protected by cybersecurity mechanisms for dataset origin andintegrity attestation [165].

815
816
817
818
819
820
821
822
823
824
825
826
827
828
829

830
831
832
833
834
835
836

837

838
839
840
841
842
843

844
845
846
847
848
849
850
851
852
853

• Robust training: An alternative approach to mitigating availability poisoning attacks is to modify the ML training algorithm and perform robust training instead of
regular training. The defender can train an ensemble of multiple models and generatepredictions via model voting [18, 131, 234]. Several papers apply techniques fromrobust optimization, such as using a trimmed loss function [66, 110]. Rosenfeld etal. [184] proposed the use of randomized smoothing for adding noise during training
and obtaining certiﬁcation against label ﬂipping attacks.
4.2.

Targeted Poisoning

In contrast to availability attacks, targeted poisoning attacks induce a change in the MLmodel’s prediction on a small number of targeted samples. If the adversary can control thelabeling function of the training data, then label ﬂipping is an effective targeted poisoning
attack. The adversary simply inserts several poisoned samples with the target label, and themodel will learn the wrong label. Therefore, targeted poisoning attacks are mostly studied
in the clean-label setting in which the attacker does not have access to the labeling function.
Several techniques for mounting clean-label targeted attacks have been proposed. Koh andLiang [122] showed how inﬂuence functions – a statistical method that determines the mostinﬂuential training samples for a prediction – can be leveraged for creating poisoned samples in the ﬁne-tuning setting in which a pre-trained model is ﬁne-tuned on new data. Suciu
et al. [212] designed StingRay, a targeted poisoning attack that modiﬁes samples in feature
space and adds poisoned samples to each mini batch of training. An optimization procedure based on feature collision was crafted by Shafahi et al. [193] to generate clean-label
targeted poisoning for ﬁne-tuning and end-to-end learning. ConvexPolytope [263] andBullseyePolytope [2] optimized the poisoning samples against ensemble models, which
offers better advantages for attack transferability. MetaPoison [101] uses a meta-learning
22

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

854
855

856
857
858
859
860
861
862
863
864
865

866
867
868
869

algorithm to optimize the poisoned samples, while Witches’ Brew [89] performs optimization by gradient alignment, resulting in a state-of-the-art targeted poisoning attack.
All of the above attacks impact a small set of targeted samples that are selected by theattacker during training, and they have only been tested for continuous image datasets(with the exception of StingRay, which requires adversarial control of a large fraction of the
training set). Subpopulation poisoning attacks [111] were designed to poison samples froman entire subpopulation, deﬁned by matching on a subset of features or creating clusters
in representation space. Poisoned samples are generated using label ﬂipping (for NLP
and tabular modalities) or a ﬁrst-order optimization method (for continuous data, such asimages). The attack generalizes to all samples in a subpopulation and requires minimal
knowledge about the ML model and a small number of poisoned samples (proportional to
the subpopulation size).
Targeted poisoning attacks have also been introduced for semi-supervised learning algorithms [29], such as MixMatch [15], FixMatch [205], and Unsupervised Data Augmentation (UDA) [241] in which the adversary poisons a small fraction of the unlabeled training
dataset to change the prediction on targeted samples at deployment time.

882

Mitigations. Targeted poisoning attacks are notoriously challenging to defend against.
Jagielski et al. [111] showed an impossibility result for subpopulation poisoning attacks.
To mitigate some of the risks associated with such attacks, cybersecurity mechanisms fordataset origin and integrity attestation [165] should be used judiciously. Ma et al. [142]proposed the use of differential privacy (DP) as a defense (which follows directly from thedeﬁnition of differential privacy), but it is well known that differentially private ML models
have lower accuracy than standard models. The trade-off between robustness and accuracy
needs to be considered in each application. If the application has strong data privacy requirements, and differentially private training is used for privacy, then an additional beneﬁt
is protection against targeted poisoning attacks. However, the robustness offered by DP
starts to fade once the targeted attack requires multiple poisoning samples (as in subpopulation poisoning attacks) because the group privacy bound will not provide meaningful
guarantees for large poisoned sets.

883

4.3.

870
871
872
873
874
875
876
877
878
879
880
881

884
885
886
887
888
889
890
891
892

Backdoor Poisoning

In 2017, Gu et al. [94] proposed BadNets, the ﬁrst backdoor poisoning attack. They observed that image classiﬁers can be poisoned by adding a small patch trigger in a subset ofimages at training time and changing their label to a target class. The classiﬁer learns to
associate the trigger with the target class, and any image – including the trigger or backdoor pattern – will be misclassiﬁed to the target class at testing time. Concurrently, Chen etal. [49] introduced backdoor attacks in which the trigger is blended into the training data.
Follow-up work introduced the concept of clean-label backdoor attacks [226] in which
the adversary is restricted in preserving the label of the poisoned examples. Clean-label
attacks typically require more poisoning samples to be effective, but the attack model is
23

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

893

894
895
896
897
898
899
900
901
902
903

904
905
906

907
908
909

910
911
912
913
914
915
916
917
918

919
920
921
922
923
924
925
926
927
928

929
930
931

more realistic.
In the last few years, backdoor attacks have become more sophisticated and stealthy, making them harder to detect and mitigate. Latent backdoor attacks were designed to survive
even upon model ﬁne-tuning of the last few layers using clean data [247]. Backdoor Generating Network (BaN) [189] is a dynamic backdoor attack in which the location of the trigger
changes in the poisoned samples so that the model learns the trigger in a location-invariant
manner. Functional triggers are embedded throughout the image or change according to
the input. For instance, Li et al. [133] used steganography algorithms to hide the trigger inthe training data. Liu et al. [139] introduced a clean-label attack that uses natural reﬂection
on images as a backdoor trigger. Wenger et al. [237] poisoned facial recognition systems
by using physical objects as triggers, such as sunglasses and earrings.
Other data modalities. While the majority of backdoor poisoning attacks are designed
for computer vision applications, this attack vector has been effective in other application
domains with different data modalities, such as audio, NLP, and cybersecurity settings.
• Audio: In audio domains, Shi et al. [199] showed how an adversary can inject anunnoticeable audio trigger into live speech, which is jointly optimized with the target
model during training.
• NLP: In natural language processing, the construction of meaningful poisoning samples is more challenging as the text data is discrete, and the semantic meaning ofsentences would ideally be preserved for the attack to remain unnoticeable. Recent
work has shown that backdoor attacks in NLP domains are becoming feasible. For
instance, Chen et al. [48] introduced semantic-preserving backdoors at the character, word, and sentence level for sentiment analysis and neural machine translation
applications. Li et al. [132] generated hidden backdoors against transformer models using generative language models in three NLP tasks: toxic comment detection,neural machine translation, and question answering.
• Cybersecurity: Early poisoning attacks in cybersecurity were designed against worm
signature generation in 2006 [177] and spam detectors in 2008 [166], well before
rising interest in adversarial machine learning. More recently, Severi et al. [192]showed how AI explainability techniques can be leveraged to generate clean-label
poisoning attacks with small triggers against malware classiﬁers. They attacked multiple models (i.e., neural networks, gradient boosting, random forests, and SVMs),
using three malware datasets: Ember for Windows PE ﬁle classiﬁcation, Contagiofor PDF ﬁle classiﬁcation, and DREBIN for Android app classiﬁcation. Jigsaw Puzzle [246] designed a backdoor poisoning attack for Android malware classiﬁers thatuses realizable software triggers harvested from benign code.
Mitigations. The literature on backdoor attack mitigation is vast compared to other poisoning attacks. Below we discuss several classes of defenses, including data sanitization,
trigger reconstruction, model inspection and sanitization, and also their limitations.
24

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

932
933
934
935
936
937
938
939
940

941
942
943
944
945
946
947
948
949

950
951
952
953
954
955
956
957
958
959
960
961

962
963
964
965
966
967
968
969
970
971
972

• Training Data Sanitization: Similar to poisoning availability attacks, training data
sanitization can be applied to detecting backdoor poisoning attacks. For instance,outlier detection in the latent feature space [98, 178, 224] has been effective for convolutional neural networks used for computer vision applications. Activation Clustering [43] performs clustering of training data in representation space with the goalof isolating the backdoored samples in a separate cluster. Data sanitization achieves
better results when the poisoning attack controls a relatively large fraction of training
data, but is not that effective against stealthy poisoning attacks. Overall, this leads toa trade-off between attack success and detectability of malicious samples.
• Trigger reconstruction: This class of mitigations aims to reconstruct the backdoor
trigger, assuming that it is at a ﬁxed location in the poisoned training samples. NeuralCleanse by Wang et al. [230] developed the ﬁrst trigger reconstruction approachand used optimization to determine the most likely backdoor pattern that reliably
misclassiﬁes the test samples. The initial technique has been improved to reduce
performance time on several classes and simultaneously support multiple triggers inserted into the model [100, 239]. A representative system in this class is Artiﬁcial
Brain Simulation (ABS) by Liu et al. [137], which stimulates multiple neurons andmeasures the activations to reconstruct the trigger patterns.
• Model inspection and sanitization: Model inspection analyzes the trained ML
model before its deployment to determine whether it was poisoned. An early work inthis space is NeuronInspect [102], which is based on explainability methods to determine different features between clean and backdoored models that are subsequently
used for outlier detection. DeepInspect [45] uses a conditional generative model tolearn the probability distribution of trigger patterns and performs model patching
to remove the trigger. Xu et al. [244] proposed the Meta Neural Trojan Detection(MNTD) framework, which trains a meta-classiﬁer to predict whether a given ML
model is backdoored (or Trojaned, in the authors’ terminology). This technique isgeneral and can be applied to multiple data modalities, such as vision, speech, tabular
data, and NLP. Once a backdoor is detected, model sanitization can be performed via
pruning [238], retraining [253], or ﬁne-tuning [135] to restore the model’s accuracy.
Most of these mitigations have been designed against computer vision classiﬁers basedon convolutional neural networks using backdoors with ﬁxed trigger patterns. Severi etal. [192] showed that some of the data sanitization techniques (e.g., spectral signatures [224]and Activation Clustering [43]) are ineffective against clean-label backdoor poisoning on
malware classiﬁers. Most recent semantic and functional backdoor triggers would alsopose challenges to approaches based on trigger reconstruction or model inspection, which
generally assume ﬁxed backdoor patterns. The limitation of using meta classiﬁers for predicting a Trojaned model [244] is the high computational complexity of the training stageof the meta classiﬁer, which requires training thousands of SHADOW MODELS. Additional
research is required to design strong backdoor mitigation strategies that can protect MLmodels against this important attack vector without suffering from these limitations.
25

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

978

In cybersecurity, Rubinstein et al. [185] proposed a principal component analysis (PCA)based approach to mitigate poisoning attacks against PCA subspace anomaly detection
method in backbone networks. It maximized Median Absolute Deviation (MAD) insteadof variance to compute principal components, and used a threshold value based on Laplace
distribution instead of Gaussian. Madani and Vlajic [143] built an autoencoder-based intrusion detection system, assuming malicious poisoning attack instances were under 2%.

979

4.4.

973
974
975
976
977

980
981
982
983
984
985
986
987

988
989
990

991
992
993
994

995
996
997
998
999

1000
1001

1002
1003
1004
1005
1006
1007
1008
1009

Model Poisoning

Model poisoning attacks attempt to directly modify the trained ML model to inject malicious functionality into the model. In centralized learning, TrojNN [138] reverse engineers
the trigger from a trained neural network and then retrains the model by embedding thetrigger in external data to poison it. Most model poisoning attacks have been designed in
the federated learning setting in which clients send local model updates to a server that
aggregates them into a global model. Compromised clients can send malicious updates topoison the global model. Model poisoning attacks can cause both availability and integrity
violation in federated models:• Poisoning availability attacks that degrade the global model’s accuracy have beeneffective, but they usually require a large percentage of clients to be under the controlof the adversary [78, 197].
• Targeted model poisoning attacks induce integrity violations on a small set of samples at testing time. They can be mounted by a model replacement or model boosting
attack in which the compromised client replaces the local model update according to
the targeted objective [7, 16, 214].
• Backdoor model poisoning attacks introduce a trigger via malicious client updates
to induce the misclassiﬁcation of all samples with the trigger at testing time [7, 16,
214, 232]. Most of these backdoors are forgotten if the compromised clients do not
regularly participate in training, but the backdoor becomes more durable if injectedin the lowest utilized model parameters [260].
Model poisoning attacks are also possible in supply-chain scenarios where models or components of the model provided by suppliers are poisoned with malicious code.
Mitigations. To defend federated learning from model poisoning attacks, a variety of
Byzantine-resilient aggregation rules have been designed and evaluated. Most of them attempt to identify and exclude the malicious updates when performing the aggregation at theserver [3, 24, 28, 95, 149–151, 213, 250]. However, motivated adversaries can bypass these
defenses by adding constraints in the attack generation optimization problem [7, 78, 197].
Gradient clipping and differential privacy have the potential to mitigate model poisoning
attacks to some extent [7, 169, 214], but they usually decrease accuracy and do not provide
complete mitigation.

26

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

Designing federated learning models that are fully robust against model poisoning
attacks remains an open research problem in the community.
1010

27

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1011

1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028

5.

Privacy Attacks

Although privacy issues have long been a concern, privacy attacks against aggregate statistical information collected from user records started with the seminal work of Dinur andNissim [67] on reconstruction attacks. The goal of reconstruction attacks is to reverse
engineer private information about an individual user record or sensitive critical infrastructure data from access to aggregate statistical information. More recently, memorization
attacks that reconstruct or regenerate the training data have been shown in the context oflarge generative language models, such as GPT-2 [34]. A less devastating privacy attackis that of membership inference in which an adversary can determine whether a particular
record was included in the dataset used for computing statistical information or training amachine learning model. Membership inference attacks were ﬁrst introduced by Homeret al. [99] for genomic data. Recent literature focuses on membership attacks against ML
models in mostly black-box settings in which adversaries have query access to a trained ML
model [30, 200, 249]. Another privacy violation for MLaaS is model extraction attacks,
which are designed to extract information about an ML model such as its architecture ormodel parameters [32, 40, 108, 222]. Property inference attacks [4, 42, 86, 145, 215, 258]
aim to extract global information about a training dataset, such as the fraction of training
examples with a certain sensitive attribute.

1032

This section discusses privacy attacks related to data reconstruction, the memorization oftraining data, membership inference, model extraction, and property inference, as well as
mitigations for some of these attacks and open problems in designing general mitigation
strategies.

1033

5.1.

1029
1030
1031

1034
1035
1036
1037
1038
1039
1040
1041
1042

1043
1044
1045
1046
1047
1048
1049

Data Reconstruction

Data reconstruction attacks are the most concerning privacy attacks as they have the ability
to recover an individual’s data from released aggregate statistical information. Dinur andNissim [67] were the ﬁrst to introduce reconstruction attacks that recover user data fromlinear statistics. Their original attack requires an exponential number of queries for reconstruction, but subsequent work has shown how to perform reconstruction with a polynomial
number of queries [74]. A survey of privacy attacks, including reconstruction attacks, isgiven by Dwork et al. [72]. More recently, the U.S. Census Bureau performed a large-scale
study on the risk of data reconstruction attacks on census data [87], which motivated theuse of differential privacy in the decennial release of the U.S. Census in 2020.
In the context of ML classiﬁers, Fredrickson et al. [84] introduced model inversion attacks
that reconstruct class representatives from the training data of an ML model. While model
inversion generates semantically similar images with those in the training set, it cannot
directly reconstruct the training data of the model. Recently, Balle et al. [9] trained a reconstructor network that can recover a data sample from a neural network model, assuming
a powerful adversary with information about all other training samples. Haim et al. [97]showed how the training data of a neural network can be reconstructed from access to the28

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1052

model parameters by leveraging theoretical insights about implicit bias in neural networks.
Another relevant privacy attack is attribute inference, in which the attacker extracts a sensitive attribute of the training set [114].

1053

5.2.

1050
1051

1054
1055
1056
1057
1058
1059
1060

Memorization

Memorization attacks are a powerful class of techniques that allow an adversary to extract
training data from generative ML models, such as language models. Carlini et al. [33] werethe ﬁrst to practically demonstrate memorization attacks in language models. By inserting
synthetic canaries in the training data, they developed a methodology for extracting thecanaries and introduced a metric called exposure to measure memorization. Subsequent
work demonstrated the risk of memorization in large language models, such as GPT-2 [34],and showed that models with a larger capacity tend to memorize more [31].

1068

An orthogonal line of work is analyzing the connection between memorization and generalization in ML models. Zhang et al. [254] discussed how neural networks can memorize
randomly selected datasets. Feldman [80] showed that the memorization of training labels is necessary to achieving almost optimal generalization error in ML. Brown et al. [26]constructed two learning tasks based on next-symbol prediction and cluster labeling inwhich memorization is required for high-accuracy learning. Feldman and Zhang empirically evaluated the beneﬁt of memorization for generalization using an inﬂuence estimation
method [81].

1069

5.3.

1061
1062
1063
1064
1065
1066
1067

1070
1071
1072
1073
1074
1075

1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087

Membership Inference

Membership inference attacks generally expose less private information about an individual
than reconstruction or memorization attacks but are still of great concern when releasing
aggregate statistical information or ML models trained on user data. In certain situations,determining that an individual is part of the training set already has privacy implications,such as in a medical study of patients with a rare disease. Moreover, membership inference
can be used as a building block for mounting extraction attacks [33, 34].
In membership inference, the attacker’s goal is to determine whether a particular record
or data sample was part of the training dataset used for the statistical or ML algorithm.
These attacks were introduced by Homer et al. [99] for statistical computations on genomic
data under the name tracing attacks. Robust tracing attacks have been analyzed when an
adversary gains access to noisy statistical information about the dataset [73]. In the last ﬁve
years, the literature has used the terminology membership inference for attacks against ML
models. Most of the attacks in the literature are performed against deep neural networks
used for classiﬁcation [30, 54, 130, 200, 248, 249]. Similar to other attacks in adversarialmachine learning, membership inference can be performed in white-box settings [130, 163,
186] in which attackers have knowledge of the model’s architecture and parameters, but
most of the attacks have been developed for black-box settings in which the adversary
generates queries to the trained ML model [30, 54, 200, 248, 249].
29

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103

1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114

The attacker’s success in membership inference has been formally deﬁned using a cryptographically inspired privacy game in which the attacker interacts with a challenger andneeds to determine whether a target sample was used in training the queried ML model [113,188, 249]. In terms of techniques for mounting membership inference attacks, the lossbased attack by Yeom et al. [249] is one of the most efﬁcient and widely used method.
Using the knowledge that the ML model minimizes the loss on training samples, the attack
determines that a target sample is part of training if its loss is lower than a ﬁxed threshold(selected as the average loss of training examples). Sablayrolles et al. [186] reﬁned the lossbased attack by scaling the loss using a per-example threshold. Another popular techniqueintroduced by Shokri et al. [200] is that of shadow models, which trains a meta-classiﬁer
on examples in and out of the training set obtained from training thousands of shadow ML
models on the same task as the original model. This technique is generally expensive, andwhile it might improve upon the simple loss-based attack, its computational cost is high andrequires access to many samples from the distribution to train the shadow models. These
two techniques are at opposite ends of the spectrum in terms of their complexity, but they
perform similarly in terms of precision at low false positive rates [30].
An intermediary method that is currently attaining state-of-the-art performance in terms of
the A REA U NDER THE C URVE (AUC) metric is the LiRA attack by Carlini et al. [30],which trains a smaller number of shadow models to learn the distribution of model logits on examples in and out of the training set. Using the assumption that the model logit
distributions are Gaussian, LiRA performs a hypothesis test for membership inference by
estimating the mean and standard deviation of the Gaussian distributions. Ye et al. [248] designed a similar attack that performs a one-sided hypothesis test, which does not make any
assumptions on the loss distribution but achieves slightly lower performance than LiRA.
Membership inference attacks have also been designed under the stricter label-only threat
model in which the adversary only has access to the predicted labels of the queried samples [54].

1116

There are several public privacy libraries that offer implementations of membership inference attacks: the TensorFlow Privacy library [208] and the ML Privacy Meter [160].

1117

5.4.

1115

1118
1119
1120
1121
1122
1123
1124
1125
1126

Model Extraction

In MLaaS scenarios, cloud providers typically train large ML models using proprietary data
and would like to keep the model architecture and parameters conﬁdential. The goal of an
attacker performing a model extraction attack is to extract information about the modelarchitecture and parameters by submitting queries to the ML model trained by an MLaaS
provider. The ﬁrst model stealing attacks were shown by Tramer at al. [222] on several
online ML services for different ML models, including logistic regression, decision trees,
and neural networks. However, Jagielski et al. [108] have shown the exact extraction of
ML models to be impossible. Instead, a functionally equivalent model can be reconstructedthat is different than the original model but achieves similar performance at the prediction
30

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1127
1128

task. Jagielski et al. [108] have shown that even the weaker task of extracting functionallyequivalent models is NP-hard.

1139

Several techniques for mounting model extraction attacks have been introduced in the literature. The ﬁrst method is that of direct extraction based on the mathematical formulationof the operations performed in deep neural networks, which allows the adversary to compute model weights algebraically [32, 108, 222]. A second technique explored in a seriesof papers is to use learning methods for extraction. For instance, active learning [40] canguide the queries to the ML model for more efﬁcient extraction of model weights, and reinforcement learning can train an adaptive strategy that reduces the number of queries [172].
A third technique is the use of SIDE CHANNEL information for model extraction. Batina
et al. [12] used electromagnetic side channels to recover simple neural network models,
while Rakin et al. [182] recently showed how ROWHAMMER ATTACKS can be used formodel extraction of more complex convolutional neural network architectures.

1140

5.5.

1129
1130
1131
1132
1133
1134
1135
1136
1137
1138

1141
1142
1143
1144
1145

1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158

1159
1160
1161
1162
1163
1164

Property Inference

In property inference attacks, the attacker tries to learn global information about the training
data distribution by interacting with an ML model. For instance, an attacker can determine
the fraction of the training set with a certain sensitive attribute, such as demographic information, that might reveal potentially conﬁdential information about the training set that is
not intended to be released.
Property inference attacks were introduced by Ateniese et al. [4] and formalized as a distinguishing game between the attacker and the challenger training two models with different
fractions of the sensitive data [215]. Property inference attacks were designed in white-box
settings in which the attacker has access to the full ML model [4, 86, 215] and black-box
settings in which the attacker issues queries to the model and learns either the predictedlabels [145] or the class probabilities [42, 258]. These attacks have been demonstrated for
HIDDEN M ARKOV MODELS , S UPPORT V ECTOR M ACHINES [4], FEED - FORWARD NEU RAL NETWORKS [86, 145, 258], CONVOLUTIONAL NEURAL NETWORKS [215], FEDER ATED LEARNING MODELS [147], GENERATIVE ADVERSARIAL NETWORKS [262], and
GRAPH NEURAL NETWORKS [261]. Mahloujifar et al. [145] and Chaudhauri et al. [42]showed that poisoning the property of interest can help design a more effective distinguishing test for property inference. Moreover, Chaudhauri et al. [42] designed an efﬁcient
property size estimation attack that recovers the exact fraction of the population of interest.
Several papers have reported negative results on various mitigation strategies against these
attacks, including differential privacy which was designed to reveal aggregate statistics
about a dataset [42, 145]. It seems inherent that a high accuracy ML model will reveal
some aggregate information about its training dataset. While property inference might
not be easy to mitigate, an open problem is understanding whether these attacks pose real
privacy risk to users who contribute their data to ML training.

31

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1165

1166
1167
1168
1169
1170
1171
1172
1173
1174

1175
1176
1177
1178
1179
1180
1181
1182

1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198

1199
1200
1201
1202
1203
1204

5.6.

Mitigations

The discovery of reconstruction attacks against aggregate statistical information motivated
the rigorous deﬁnition of differential privacy (DP) [70, 71]. Differential privacy is an extremely strong deﬁnition of privacy that guarantees a bound on how much an attacker withaccess to the algorithm output can learn about each individual record in the dataset. The
original pure deﬁnition of DP has a privacy parameter e (i.e., privacy budget), which bounds
the probability that the attacker with access to the algorithm’s output can determine whether
a particular record was included in the dataset. DP has been extended to the notions of approximate DP, which includes a second parameter d that is interpreted as the probability of
information accidentally being leaked in addition to e and Rènyi DP [154].
DP has been widely adopted due to several useful properties: group privacy (i.e., the extension of the deﬁnition to two datasets differing in k records), post-processing (i.e., privacy
is preserved even after processing the output), and composition (i.e., privacy is composed
if multiple computations that are performed on the dataset). DP mechanisms for statistical computations include the Gaussian mechanism [71], the Laplace mechanism [71], andthe Exponential mechanism [146]. The most widely used DP algorithm for training ML
models is DP-SGD [1], with recent improvements such as DP-FTRL [117] and DP matrix
factorization [65].
By deﬁnition, DP provides mitigation against reconstruction attacks, the memorization oftraining data, and membership inference attacks. In fact, the deﬁnition of DP immediately
implies an upper bound on the success of a membership inference attack. Tight bounds
on the success of membership inference have been derived by Thudi et al. [218]. However, DP does not provide guarantees against model extraction or property inference attacks [42, 145]. One of the main challenges of using DP in practice is setting up the privacy
parameters to achieve a trade-off between privacy and utility, which is typically measured
in terms of accuracy for ML models. Analysis of privacy-preserving algorithms, such as
DP-SGD, is often worst case, and selecting privacy parameters based purely on theoreticalanalysis results in utility loss. Therefore, large privacy parameters are often used in practice (e.g., the 2020 U.S. Census release used e = 19.61), and the exact privacy obtainedin practice is difﬁcult to estimate. Recently, a promising line of work is that of privacy
auditing introduced by Jagielski et al. [112] with the goal of empirically measuring the actual privacy guarantees of an algorithm and determining privacy lower bounds by mounting
privacy attacks. Auditing can be performed with membership inference attacks [113], butpoisoning attacks are much more effective for empirical privacy auditing [112, 164].
Other mitigation techniques against model extraction, such as limiting user queries to themodel, detecting suspicious queries to the model, or creating more robust architectures toprevent side channel attacks exist in the literature. However, these techniques can be circumvented by motivated and well-resourced attackers and should be used with caution.
We refer the reader to available practice guides for securing machine learning deployments [39, 170].
32

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1205

1206
1207
1208
1209
1210
1211
1212
1213

1214
1215
1216
1217
1218
1219

1220
1221
1222
1223
1224

1225
1226
1227
1228
1229
1230
1231

1232
1233
1234
1235
1236

1237
1238
1239
1240
1241
1242
1243

6.

Discussion and Remaining Challenges

The literature on AML shows a trend of designing new attacks with higher power andstealthier behavior. The attacks considered above and those discussed in Section 6.2 illustrate this well. Moreover, Goldwasser et al. [91] recently introduced a new class of attacks:
information-theoretically undetectable Trojans that can be planted in ML models. Suchattacks can only be prevented or detected and mitigated by procedures that restrict andcontrol who in the organization has access to the model throughout the life cycle and by
thoroughly vetting third-party components coming through the supply chain. The NIST AI
Risk Management Framework [170] offers more information on this.
One of the ongoing challenges facing the AML ﬁeld is the ability to detect when the modelis under attack. Knowing this would provide an opportunity to counter the attack before
any information is lost or an adverse behaviour is triggered in the model. Tramèr [219]has shown that designing techniques to detect adversarial examples is equivalent to robust
classiﬁcation, which is inherently hard to construct, up to computational complexity and afactor of 2 in the robustness radius.
Adversarial examples may be from the same data distribution on which the model is trained
and to which it expects the inputs to belong or may be OUT- OF - DISTRIBUTION (OOD) inputs. Thus, the ability to detect OOD inputs is also an important challenge in AML. Fang etal. [79] established useful theoretical bounds on detectability, particularly an impossibility
result when there is an overlap between the in-distribution and OOD data.
Given the onslaught of powerful attacks, designing appropriate mitigations is a challengethat needs to be addressed before deploying AI systems in critical domains. This challenge
is exacerbated by the lack of information-theoretically secure machine learning algorithms
for many tasks in the ﬁeld, as we discussed in Section 1. This implies that presently designing mitigations is an inherently ad hoc and fallible process. We refer the readers toavailable practice guides for securing machine learning deployments [39, 170], as well as
existing guidelines for mitigating AML attacks [75].
The data and model sanitization techniques discussed in Section 4 reduce the impact of arange of poisoning attacks and should be widely used. However, they should be combinedwith cryptographic techniques for origin and integrity attestation to provide assurances
downstream, as recommended in the ﬁnal report of the National Security Commission on
AI [165].
The robust training techniques discussed in Section 4 offer different approaches to providing theoretically certiﬁed defenses against data poisoning attacks with the intention ofproviding much-needed information-theoretic guarantees for security. The results are encouraging, but more research is needed to extend this methodology to more general assumptions about the data distributions, the ability to handle OOD inputs, more complex
models, and multiple data modalities. Another challenge is applying these techniques tovery large models like LLMs and generative models, which are quickly becoming targets
33

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1244

1245
1246
1247
1248
1249
1250

1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263

1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275

1276
1277

1278
1279
1280

1281
1282

of attacks [55].
Another general problem of AML mitigations for both evasion and poisoning attacks is
the lack of reliable benchmarks which causes results from AML papers to be routinely
incomparable, as they do not rely on the same assumptions and methods. While there
have been some promising developments into this direction [60, 191], more research andencouragement is needed to foster the creation of standardized benchmarks to allow gaining
reliable insights into the actual performance of proposed mitigations.
Formal methods veriﬁcation has a long history in other ﬁelds where high assurance is required, such as avionics and cryptography. The lessons learned there teach us that although
the results from applying this methodology are excellent in terms of security and safetyassurances, they come at a very high cost, which has prevented formal methods from being
widely adopted. Currently, formal methods in these ﬁelds are primarily used in applications
mandated by regulations. Applying formal methods to neural networks has signiﬁcant potential to provide much-needed security guarantees, especially in high-risk applications.
However, the viability of this technology will be determined by a combination of technical and business criteria – namely, the ability to handle today’s complex machine learningmodels of interest at acceptable costs. More research is needed to extend this technology
to all algebraic operations used in machine learning algorithms, to scale it up to the large
models used today, and to accommodate rapid changes in the code of AI systems while
limiting the costs of applying formal veriﬁcation.
There is an imbalance between the large number of privacy attacks listed in Section 5(i.e., memorization, membership inference, model extraction, and property inference) andavailable reliable mitigation techniques. In some sense, this is a normal state of affairs: a
rapidly evolving technology gaining widespread adoption – even “hype” – which attracts
the attention of adversaries, who try to expose and exploit its weaknesses before the technology has matured enough for society to assess and manage it effectively. To be sure, not
all adversaries have malevolent intent. Some simply want to warn the public of potential
breakdowns that can cause harm and erode trust in the technology. Additionally, not all
attacks are as practical as they need to be to pose real threats to AI system deployments
of interest. Yet the race between developers and adversaries has begun, and both sides
are making great progress. This poses many difﬁcult questions for the AI community of
stakeholders, such as:• What is the best way to mitigate the potential exploits of memorized data from Section 5.2 as models grow and ingest larger amounts of data?
• What is the best way to prevent attackers from inferring membership in the training
set or other properties of the training data using the attacks listed in Sections 5.3 and
5.5?
• How can developers protect their ML models and associated intellectual property
from the emerging threats of algebraic methods that utilize the public API of the ML
34

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

model to query and exploit its secret weights or the side-channel leakage attacks from
Section 5.4? The known mechanisms of preventing large numbers of queries throughthe API are ineffective in conﬁgurations with anonymous or unauthenticated accessto the model.

1283
1284
1285
1286

1289

As answers to these questions become available, it is important for the community of stakeholders to develop speciﬁc guidelines to complement the NIST AI RMF [170] for use cases
where privacy is of utmost importance.

1290

6.1.

1287
1288

1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301

Trade-O↵s Between the Attributes of Trustworthy AI

The trustworthiness of an AI system depends on all of the attributes that characterize
it [170]. For example, an AI system that is accurate but easily susceptible to adversarial
exploits is unlikely to be trusted. Similarly, an AI system that produces harmfully biased
or unfair outcomes is unlikely to be trusted even if it is robust. There are also trade-offs
between explainability and adversarial robustness [107, 153]. In cases where fairness isimportant and privacy is necessary to maintain, the trade-off between privacy and fairness
needs to be considered [109]. Unfortunately, it is not possible to simultaneously maximize
the performance of the AI system with respect to these attributes. For instance, AI systems optimized for accuracy alone tend to underperform in terms of adversarial robustness
and fairness [41, 69, 181, 225, 255]. Conversely, an AI system optimized for adversarialrobustness may exhibit lower accuracy and deteriorated fairness outcomes [14, 231, 255].
The full characterization of the trade-offs between the different attributes of trustworthy AI is still an open research problem that is gaining increasing importancewith the adoption of AI technology in many areas of modern life.

1302

1306

In most cases, organizations will need to accept trade-offs between these properties anddecide which of them to prioritize depending on the AI system, the use case, and potentiallymany other considerations about the economic, environmental, social, cultural, political,
and global implications of the AI technology [170].

1307

6.2.

1303
1304
1305

1308
1309
1310
1311
1312
1313
1314
1315
1316

Multimodal Models: Are They More Robust?

MULTIMODAL MODELS have shown great potential for achieving high performance on
many machine learning tasks [10, 13, 159, 183, 256]. It is natural to assume that because
there is redundancy of information across the different modalities, the model should be
more robust against adversarial perturbations of a single modality. However, emerging evidence from practice shows that this is not necessarily the case. Combining modalities andtraining the model on clean data alone does not seem to improve adversarial robustness.
In addition, one of the most effective defenses against evasion attacks based on adversarial
training, which is widely used in single modality applications, is prohibitively expensive
in practical applications of multimodal learning. Additional effort is required to beneﬁt

35

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1317
1318
1319
1320
1321
1322
1323

from the redundant information in order to improve robustness against single modality
attacks [245]. Without such an effort, single modality attacks can be effective and compromise multimodal models across a wide range of multimodal tasks despite the information
contained in the remaining unperturbed modalities [245, 251]. Moreover, researchers havedevised efﬁcient mechanisms for constructing simultaneous attacks on multiple modalities, which suggests that multimodal models might not be more robust against adversarial
attacks despite improved performance [44, 195, 243].
The existence of simultaneous attacks on multimodal models suggests that mitigation techniques that only rely on single modality perturbations are not likely to
be robust. Attackers in real life do not constrain themselves to attacks within agiven security model but employ any attack that is available to them.

1324

1325

1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336

1337
1338
1339
1340
1341
1342

6.3.

Beyond Models and Data

As pointed out in the Introduction, chatbots [50, 62, 152, 171] enabled by recent advances
in deep learning have emerged as a powerful technology with great potential for numerous
business applications, from entertainment to more critical ﬁelds. AI-enabled chatbots use
NLP to process and respond to human input, but these chatbots have more complicatedarchitectures than just a language model. For example, a critical element of a conversational
chatbot is the dialog component whose task is to determine the purpose of the user input
and identify relevant intents (i.e., establish the context for the conversation). Only then is
the chatbot able to determine an appropriate response and return it to the user. Traditional
attacks on chatbots have focused on overwhelming the chatbot with toxic input in order
to alter its behaviour [190]. Recently, speciﬁc attacks using ”PROMPT INJECTIONS” haveemerged as effective ways to trigger bad behaviour in the bot [228].
However, the design of AI systems that can communicate with humans is not just a technical problem but a deeply socio-technical challenge. In addition, the potential for attacks
that could compromise the function of the dialog component and maliciously change thesubject of the conversation for the unsuspecting user can lead to the chatbot offering misleading or even harmful advice. The potential harms in this case go beyond the traditionalharms considered by AML and deﬁned in Section 2.
Despite progress in the ability of chatbots to perform well on certain tasks [171],
this technology is still limited and should not be deployed in applications that
require a high degree of trust in the information they generate.

1343

1344
1345
1346
1347

As the development of AI-enabled chatbots continues and their deployment becomes moreprevalent online, these concerns will come to the forefront and be pursued by adversaries
to discover and exploit vulnerabilities and by companies developing the technology to improve their design and implementation to protect against such attacks.

36

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1348
1349
1350
1351
1352
1353
1354
1355

Realistic risk management throughout the entire life cycle of the technology is criticallyimportant to identify risks and plan early corresponding mitigation approaches [170]. For
example, incorporating human adversarial input in the process of training the system (i.e.,
red teaming) or employing reinforcement learning from human feedback appear to offer
beneﬁts in terms of making the chatbot more resilient against toxic input or prompt injections [62]. Barrett et al. [11] have developed detailed risk proﬁles for cutting-edge generative AI systems that map well to the NIST AI RMF [57] and should be used for assessing
and mitigating potentially catastrophic risks to society that may arise from this technology.

37

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1356

1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396

References[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov,
Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In ACM Conference on Computer and Communications Security, CCS ’16, pages 308–318, 2016.
https://arxiv.org/abs/1607.00133.
[2] Hojjat Aghakhani, Dongyu Meng, Yu-Xiang Wang, Christopher Kruegel, and Giovanni Vigna. Bullseye polytope: A scalable clean-label poisoning attack with improved transferability. In IEEE European Symposium on Security and Privacy, 2021,
Vienna, Austria, September 6-10, 2021, pages 159–178. IEEE, 2021.
[3] Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine Stochastic Gradient Descent. In NeurIPS, 2018.
[4] Giuseppe Ateniese, Luigi V. Mancini, Angelo Spognardi, Antonio Villani,
Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones:
How to extract meaningful data from machine learning classiﬁers. Int. J. Secur.
Netw., 10(3):137–150, September 2015.
[5] Anish Athalye, Nicholas Carlini, and David A. Wagner. Obfuscated gradients givea false sense of security: Circumventing defenses to adversarial examples. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden,
July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages
274–283. PMLR, 2018.
[6] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly
Shmatikov. How to backdoor federated learning. In Silvia Chiappa and Roberto
Calandra, editors, Proceedings of the Twenty Third International Conference on Artiﬁcial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning
Research, pages 2938–2948. PMLR, 26–28 Aug 2020.
[7] Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly
Shmatikov. How to backdoor federated learning. In AISTATS. PMLR, 2020.
[8] Marieke Bak, Vince Istvan Madai, Marie-Christine Fritzsche, Michaela Th.
Mayrhofer, and Stuart McLennan. You can’t have ai both ways: Balancing healthdata privacy and access fairly. Frontiers in Genetics, 13, 2022. https://www.frontiersin.org/articles/10.3389/fgene.2022.929453.
[9] Borja Balle, Giovanni Cherubin, and Jamie Hayes. Reconstructing training data withinformed adversaries. In NeurIPS 2021 Workshop on Privacy in Machine Learning(PRIML), 2021.
ˇ
[10] Tadas Baltrusaitis,
Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy, 2017.
[11] Anthony M. Barrett, Dan Hendrycks, Jessica Newman, and Brandie Nonnecke. Actionable Guidance for High-Consequence AI Risk Management: Towards Standards
Addressing AI Catastrophic Risks. https://arxiv.org/abs/2206.08966, 2022.
[12] Lejla Batina, Shivam Bhasin, Dirmanto Jap, and Stjepan Picek. CSI NN: Reverse
38

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438

engineering of neural network architectures through electromagnetic side channel.
In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19,
page 515–532, USA, 2019. USENIX Association.
[13] Khaled Bayoudh, Raja Knani, Fayçal Hamdaoui, and Abdellatif Mtibaa. A survey
on deep multimodal learning for computer vision: Advances, trends, applications,and datasets. Vis. Comput., 38(8):2939–2970, August 2022.
[14] Philipp Benz, Chaoning Zhang, Soomin Ham, Gyusang Karjauv, Adil Cho, andIn So Kweon. The triangular trade-off between accuracy, robustness, and fairness.
Workshop on Adversarial Machine Learning in Real-World Computer Vision Systems and Online Challenges (AML-CV) at CVPR, 2021.
[15] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver,and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett,
editors, Advances in Neural Information Processing Systems 32, pages 5050–5060.
Curran Associates, Inc., 2019.
[16] Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo.
Model Poisoning Attacks in Federated Learning. In NeurIPS SECML, 2018.
[17] Arjun Nitin Bhagoji, Supriyo Chakraborty, Prateek Mittal, and Seraphin Calo. Analyzing federated learning through an adversarial lens. In Kamalika Chaudhuri and
Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on
Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages
634–643. PMLR, 09–15 Jun 2019.
[18] Battista Biggio, Igino Corona, Giorgio Fumera, Giorgio Giacinto, and Fabio Roli.
Bagging classiﬁers for ﬁghting poisoning attacks in adversarial classiﬁcation tasks.
In Proceedings of the 10th International Conference on Multiple Classiﬁer Systems,
MCS’11, page 350–359, Berlin, Heidelberg, 2011. Springer-Verlag.
ˇ[19] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndić,Pavel
Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learningat test time. In Joint European conference on machine learning and knowledge
discovery in databases, pages 387–402. Springer, 2013.
[20] Battista Biggio, Blaine Nelson, and Pavel Laskov. Support vector machines underadversarial label noise. In Chun-Nan Hsu and Wee Sun Lee, editors, Proceedings of
the Asian Conference on Machine Learning, volume 20 of Proceedings of Machine
Learning Research, pages 97–112, South Garden Hotels and Resorts, Taoyuan, Taiwain, 14–15 Nov 2011. PMLR.
[21] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support
vector machines. In Proceedings of the 29th International Coference on International Conference on Machine Learning, ICML, 2012.
[22] Battista Biggio, Konrad Rieck, Davide Ariu, Christian Wressnegger, Igino Corona,
Giorgio Giacinto, and Fabio Roli. Poisoning behavioral malware clustering. In
Proceedings of the 2014 Workshop on Artiﬁcial Intelligent and Security Workshop,
AISec ’14, page 27–36, New York, NY, USA, 2014. Association for Computing
39

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480

Machinery.
[23] Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarialmachine learning. Pattern Recognition, 84:317–331, December 2018.
[24] Peva Blanchard, El Mahdi El Mhamdi, Rachid Guerraoui, and Julien Stainer. Machine Learning with Adversaries: Byzantine Tolerant Gradient Descent. In NeurIPS,
2017.
[25] Wieland Brendel, Jonas Rauber, and Matthias Bethge. Decision-based adversarial
attacks: Reliable attacks against black-box machine learning models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC,
Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net,
2018.
[26] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. Whenis memorization of irrelevant training data necessary for high-accuracy learning? In
Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing,
STOC 2021, page 123–132, New York, NY, USA, 2021. Association for Computing
Machinery.
[27] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell,
Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon
Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, andDario Amodei. Language models are few-shot learners. CoRR, abs/2005.14165,
2020.
[28] Xiaoyu Cao, Minghong Fang, Jia Liu, and Neil Zhenqiang Gong. FLTrust:
Byzantine-robust federated learning via trust bootstrapping. In NDSS, 2021.
[29] Nicholas Carlini. Poisoning the unlabeled dataset of Semi-Supervised learning.
In 30th USENIX Security Symposium (USENIX Security 21), pages 1577–1592.
USENIX Association, August 2021.
[30] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from ﬁrst principles. In 2022 IEEE
Symposium on Security and Privacy (S&P), pages 1519–1519, Los Alamitos, CA,
USA, May 2022. IEEE Computer Society.
[31] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian
Tramer, and Chiyuan Zhang. Quantifying memorization across neural language
models. https://arxiv.org/abs/2202.07646, 2022.
[32] Nicholas Carlini, Matthew Jagielski, and Ilya Mironov. Cryptanalytic extractionof neural network models. In Daniele Micciancio and Thomas Ristenpart, editors,
Advances in Cryptology – CRYPTO 2020, pages 189–218, Cham, 2020. Springer
International Publishing.
´[33] Nicholas Carlini, Chang Liu, Ulfar
Erlingsson, Jernej Kos, and Dawn Song. The
Secret Sharer: Evaluating and testing unintended memorization in neural networks.
40

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522

In USENIX Security Symposium, USENIX ’19), pages 267–284, 2019. https://arxiv.
org/abs/1802.08232.
` Eric Wallace, Matthew Jagielski, Ariel Herbert[34] Nicholas Carlini, Florian Tramer,´Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar
Erlingsson,
Alina Oprea, and Colin Raffel. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650.
USENIX Association, August 2021.
[35] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected:
Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on
Artiﬁcial Intelligence and Security, AISec ’17, page 3–14, New York, NY, USA,2017. Association for Computing Machinery.
[36] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neuralnetworks. In Proc. IEEE Security and Privacy Symposium, 2017.
[37] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks
on speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW), pages 1–7.
IEEE, 2018.
[38] Defense Use Case. Analysis of the cyber attack on the Ukrainian power grid. Electricity Information Sharing and Analysis Center (E-ISAC), 388:1–29, 2016.
[39] National Cyber Security Center. Introducing our new machine learning security
principles, retrieved February 2023 from https://www.ncsc.gov.uk/blog-post/introd
ucing-our-new-machine-learning-security-principles.
[40] Varun Chandrasekaran, Kamalika Chaudhuri, Irene Giacomelli, Somesh Jha, andSongbai Yan. Exploring connections between active learning and model extraction.
In Proceedings of the 29th USENIX Conference on Security Symposium, SEC’20,
USA, 2020. USENIX Association.
[41] Hong Chang, Ta Duy Nguyen, Sasi Kumar Murakonda, Ehsan Kazemi, and
R. Shokri. On adversarial bias and the robustness of fair machine learning. https://arxiv.org/abs/2006.08669, 2020.
`[42] Harsh Chaudhari, John Abascal, Alina Oprea, Matthew Jagielski, Florian Tramer,
and Jonathan Ullman. SNAP: Efﬁcient extraction of private properties with poisoning. In 2023 IEEE Symposium on Security and Privacy (S&P), 2023.
[43] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks
on deep neural networks by activation clustering. https://arxiv.org/abs/1811.03728,
2018.
[44] Hongge Chen, Huan Zhang, Pin-Yu Chen, Jinfeng Yi, and Cho-Jui Hsieh. Attacking
visual language grounding with adversarial examples: A case study on neural image
captioning. https://arxiv.org/abs/1712.02051, 2017.
[45] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. DeepInspect: A blackbox trojan detection and mitigation framework for deep neural networks. In Proceedings of the Twenty-Eighth International Joint Conference on Artiﬁcial Intelligence,
IJCAI-19, pages 4658–4664. International Joint Conferences on Artiﬁcial Intelli41

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564

gence Organization, 7 2019.
[46] Jianbo Chen, Michael I. Jordan, and Martin J. Wainwright. HopSkipJumpAttack:
A query-efﬁcient decision-based attack. In 2020 IEEE Symposium on Security andPrivacy, SP 2020, San Francisco, CA, USA, May 18-21, 2020, pages 1277–1294.
IEEE, 2020.
[47] Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without
training substitute models. In Proceedings of the 10th ACM Workshop on Artiﬁcial Intelligence and Security, AISec ’17, page 15–26, New York, NY, USA, 2017.
Association for Computing Machinery.
[48] Xiaoyi Chen, Ahmed Salem, Dingfan Chen, Michael Backes, Shiqing Ma, Qingni
Shen, Zhonghai Wu, and Yang Zhang. Badnl: Backdoor attacks against nlp models
with semantic-preserving improvements. In Annual Computer Security Applications
Conference, ACSAC ’21, page 554–569, New York, NY, USA, 2021. Association for
Computing Machinery.
[49] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted
backdoor attacks on deep learning systems using data poisoning. arXiv preprint
arXiv:1712.05526, 2017.
[50] Heng-Tze Cheng and Romal Thoppilan. LaMDA: Towards Safe, Grounded, andHigh-Quality Dialog Models for Everything. https://ai.googleblog.com/2022/01/la
mda-towards-safe-grounded-and-high.html, 2022. Google Brain.
[51] Minhao Cheng, Thong Le, Pin-Yu Chen, Huan Zhang, Jinfeng Yi, and Cho-Jui
Hsieh. Query-efﬁcient hard-label black-box attack: An optimization-based approach. In 7th International Conference on Learning Representations, ICLR 2019,
New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[52] Minhao Cheng, Simranjit Singh, Patrick H. Chen, Pin-Yu Chen, Sijia Liu, and ChoJui Hsieh. Sign-opt: A query-efﬁcient hard-label adversarial attack. In International
Conference on Learning Representations, 2020.
[53] Alesia Chernikova and Alina Oprea. FENCE: Feasible evasion attacks on neuralnetworks in constrained environments. ACM Transactions on Privacy and Security(TOPS) Journal, 2022.
[54] Christopher A. Choquette-Choo, Florian Tramer, Nicholas Carlini, and Nicolas Papernot. Label-only membership inference attacks. In Marina Meila and Tong Zhang,
editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1964–1974. PMLR,
18–24 Jul 2021.
[55] Sheng-Yen Chou, Pin-Yu Chen, and Tsung-Yi Ho. How to backdoor diffusion models? https://arxiv.org/abs/2212.05400, 2022.
[56] Antonio Emanuele Cinà, Kathrin Grosse, Ambra Demontis, Sebastiano Vascon,
Werner Zellinger, Bernhard A. Moser, Alina Oprea, Battista Biggio, MarcelloPelillo, and Fabio Roli. Wild patterns reloaded: A survey of machine learning security against training data poisoning. ACM Computing Surveys, March 2023.
42

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606

[57] Jack Clark and Raymond Perrault. 2022 AI index report. https://aiindex.stanford.e
du/wp-content/uploads/2022/03/2022-AI-Index-Report Master.pdf, 2022. Human
Centered AI, Stanford University.
[58] Joseph Clements, Yuzhe Yang, Ankur Sharma, Hongxin Hu, and Yingjie Lao. Rallying adversarial techniques against deep learning for network security, 2019.
[59] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certiﬁed adversarial robustness via
randomized smoothing. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors,
Proceedings of the 36th International Conference on Machine Learning, volume 97
of Proceedings of Machine Learning Research, pages 1310–1320. PMLR, 09–15
Jun 2019.
[60] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti,
Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-ﬁfth Conference
on Neural Information Processing Systems Datasets and Benchmarks Track (Round2), 2021.
[61] Nilesh Dalvi, Pedro Domingos, Mausam, Sumit Sanghai, and Deepak Verma. Adversarial classiﬁcation. In Proceedings of the Tenth ACM SIGKDD International
Conference on Knowledge Discovery and Data Mining, KDD ’04, page 99–108,
New York, NY, USA, 2004. Association for Computing Machinery.
[62] DeepMind. Building safer dialogue agents. https://www.deepmind.com/blog/buildi
ng-safer-dialogue-agents, 2022. Online.
[63] Luca Demetrio, Battista Biggio, Giovanni Lagorio, Fabio Roli, and Alessandro Armando. Functionality-preserving black-box optimization of adversarial windows
malware. IEEE Transactions on Information Forensics and Security, 16:3469–3478,
2021.
[64] Ambra Demontis, Marco Melis, Maura Pintor, Matthew Jagielski, Battista Biggio,
Alina Oprea, Cristina Nita-Rotaru, and Fabio Roli. Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In 28th USENIX
Security Symposium (USENIX Security 19), pages 321–338. USENIX Association,
2019.
[65] Serguei Denissov, Hugh Brendan McMahan, J Keith Rush, Adam Smith, andAbhradeep Guha Thakurta. Improved differential privacy for SGD via optimal private linear operators on adaptive streams. In Alice H. Oh, Alekh Agarwal, Danielle
Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing
Systems, 2022.
[66] Ilias Diakonikolas, Gautam Kamath, Daniel Kane, Jerry Li, Jacob Steinhardt, andAlistair Stewart. Sever: A robust meta-algorithm for stochastic optimization. In
International Conference on Machine Learning, pages 1596–1606. PMLR, 2019.
[67] Irit Dinur and Kobbi Nissim. Revealing information while preserving privacy. In
Proceedings of the 22nd ACM Symposium on Principles of Database Systems, PODS
’03, pages 202–210. ACM, 2003.
[68] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xi43

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648

aohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth
16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929,
2021.
[69] Sanghamitra Dutta, Dennis Wei, Hazar Yueksel, Pin-Yu Chen, Sijia Liu, and Kush R.
Varshney. Is there a trade-off between fairness and accuracy? A perspective using
mismatched hypothesis testing. In Proceedings of the 37th International Conference
on Machine Learning, ICML’20. JMLR.org, 2020.
[70] Cynthia Dwork. Differential privacy. In Automata, Languages and Programming,
33rd International Colloquium, ICALP 2006, Venice, Italy, July 10-14, 2006, Proceedings, Part II, pages 1–12, 2006.
[71] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise
to sensitivity in private data analysis. In Conference on Theory of Cryptography,
TCC ’06, pages 265–284, New York, NY, USA, 2006.
[72] Cynthia Dwork, Adam Smith, Thomas Steinke, and Jonathan Ullman. Exposed! A
survey of attacks on private data. Annual Review of Statistics and Its Application,
4:61–84, 2017.
[73] Cynthia Dwork, Adam Smith, Thomas Steinke, Jonathan Ullman, and Salil Vadhan.
Robust traceability from trace amounts. In IEEE Symposium on Foundations ofComputer Science, FOCS ’15, 2015.
[74] Cynthia Dwork and Sergey Yekhanin. New efﬁcient attacks on statistical disclosure
control mechanisms. In Annual International Cryptology Conference, pages 469–
480. Springer, 2008.
[75] ETSI Group Report SAI 005. Securing artiﬁcial intelligence (SAI); mitigation strategy report, retrieved February 2023 from https://www.etsi.org/deliver/etsi gr/SAI/
001 099/005/01.01.01 60/gr SAI005v010101p.pdf.
[76] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei
Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world
attacks on deep learning visual classiﬁcation. In 2018 IEEE/CVF Conference on
Computer Vision and Pattern Recognition, pages 1625–1634, 2018.
[77] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei
Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classiﬁcation. In 2018 IEEE Conference on Computer
Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22,
2018, pages 1625–1634. Computer Vision Foundation / IEEE Computer Society,2018.
[78] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Zhenqiang Gong. Local Model
Poisoning Attacks to Byzantine-Robust Federated Learning. In USENIX Security,
2020.
[79] Zhen Fang, Yixuan Li, Jie Lu, Jiahua Dong, Bo Han, and Feng Liu. Is out-ofdistribution detection learnable? In Proceedings of the 36th Conference on Neural
Information Processing Systems (NeurIPS 2022). online: https://arxiv.org/abs/2210
44

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690

.14707, 2022.
[80] Vitaly Feldman. Does learning require memorization? A short tale about a longtail. In ACM Symposium on Theory of Computing, STOC ’20, pages 954–959, 2020.
https://arxiv.org/abs/1906.05271.
[81] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why:
Discovering the long tail via inﬂuence estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red
Hook, NY, USA, 2020. Curran Associates Inc.
[82] Ji Feng, Qi-Zhi Cai, and Zhi-Hua Zhou. Learning to confuse: Generating training
time adversarial data with auto-encoder. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[83] Liam Fowl, Ping-yeh Chiang, Micah Goldblum, Jonas Geiping, Arpit Bansal, Wojtek Czaja, and Tom Goldstein. Preventing unauthorized use of proprietary data:
Poisoning for secure dataset release, 2021.
[84] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that
exploit conﬁdence information and basic countermeasures. In Proceedings of the
22nd ACM SIGSAC Conference on Computer and Communications Security, CCS’15, page 1322–1333, New York, NY, USA, 2015. Association for Computing Machinery.
[85] Aymeric Fromherz, Klas Leino, Matt Fredrikson, Bryan Parno, and Corina Pasareanu. Fast geometric projections for local robustness certiﬁcation. In International
Conference on Learning Representations, 2021.
[86] Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. Property
inference attacks on fully connected neural networks using permutation invariant
representations. In Proceedings of the 2018 ACM SIGSAC Conference on Computerand Communications Security, CCS ’18, page 619–633, New York, NY, USA, 2018.
Association for Computing Machinery.
[87] Simson Garﬁnkel, John Abowd, and Christian Martindale. Understanding databasereconstruction attacks on public data. Communications of the ACM, 62:46–53, 02
2019.
[88] Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat
Chaudhuri, and Martin Vechev. AI2: Safety and robustness certiﬁcation of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security andPrivacy (S&P), pages 3–18, 2018.
[89] Jonas Geiping, Liam H Fowl, W. Ronny Huang, Wojciech Czaja, Gavin Taylor,Michael Moeller, and Tom Goldstein. Witches’ brew: Industrial scale data poisoning
via gradient matching. In International Conference on Learning Representations,2021.
[90] Micah Goldblum, Avi Schwarzschild, Ankit Patel, and Tom Goldstein. Adversarial
attacks on machine learning systems for high-frequency trading. In Proceedings of
the Second ACM International Conference on AI in Finance, ICAIF ’21, New York,45

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732

NY, USA, 2021. Association for Computing Machinery.
[91] Shaﬁ Goldwasser, Michael P. Kim, Vinod Vaikuntanathan, and Or Zamir. Planting
undetectable backdoors in machine learning models. https://arxiv.org/abs/2204.069
74, 2022. arXiv.
[92] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press,2016. http://www.deeplearningbook.org.
[93] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing
adversarial examples. In International Conference on Learning Representations,
2015.
[94] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230–47244,
2019.
´ emy
´
[95] Rachid Guerraoui, Arsany Guirguis, Jer
Plassmann, Anton Ragot, and S´ebastien
Rouault. Garﬁeld: System support for byzantine machine learning (regular paper).
In DSN. IEEE, 2021.
[96] Chuan Guo, Alexandre Sablayrolles, Herv´e J´egou, and Douwe Kiela. Gradientbased adversarial attacks against text transformers. In Proceedings of the 2021
Conference on Empirical Methods in Natural Language Processing, pages 5747–
5757, Online and Punta Cana, Dominican Republic, November 2021. Association
for Computational Linguistics.
[97] Niv Haim, Gal Vardi, Gilad Yehudai, michal Irani, and Ohad Shamir. Reconstructing
training data from trained neural networks. In Alice H. Oh, Alekh Agarwal, Danielle
Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing
Systems, 2022.
[98] Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. SPECTRE:
Defending against backdoor attacks using robust statistics. In Marina Meila andTong Zhang, editors, Proceedings of the 38th International Conference on Machine
Learning, volume 139 of Proceedings of Machine Learning Research, pages 4129–
4139. PMLR, 18–24 Jul 2021.
[99] Nils Homer, Szabolcs Szelinger, Margot Redman, David Duggan, Waibhav Tembe,
Jill Muehling, John V Pearson, Dietrich A Stephan, Stanley F Nelson, and David W
Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS genetics,
4(8):e1000167, 2008.
[100] Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao, Susmit Jha, and Chao Chen.
Trigger hunting with a topological prior for trojan detection. In International Conference on Learning Representations, 2022.
[101] W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein.
Metapoison: Practical general-purpose clean-label data poisoning. In H. Larochelle,
M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 12080–12091. Curran Associates,
Inc., 2020.
46

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774

[102] Xijie Huang, Moustafa Alzantot, and Mani Srivastava. NeuronInspect: Detecting
backdoors in neural networks via output explanations, 2019.
[103] W. Nicholson Price II. Risks and remedies for artiﬁcial intelligence in health care.
https://www.brookings.edu/research/risks-and-remedies-for-artiﬁcial-intelligence-i
n-health-care/, 2019. Brookings Report.
[104] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15,
2018, volume 80 of Proceedings of Machine Learning Research, pages 2142–2151.
PMLR, 2018.
[105] Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Blackbox adversarial attacks with bandits and priors. In International Conference on
Learning Representations, 2019.
[106] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran,and Aleksander Madry. Adversarial examples are not bugs, they are features. In
H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett,
editors, Advances in Neural Information Processing Systems, volume 32. Curran
Associates, Inc., 2019.
[107] Shahin Jabbari, Han-Ching Ou, Himabindu Lakkaraju, and Milind Tambe. An empirical study of the trade-offs between interpretability and fairness. In ICML Workshop on Human Interpretability in Machine Learning, International Conference on
Machine Learning (ICML), 2020.
[108] Matthew Jagielski, Nicholas Carlini, David Berthelot, Alex Kurakin, and Nicolas
Papernot. High accuracy and high ﬁdelity extraction of neural networks. In Proceedings of the 29th USENIX Conference on Security Symposium, SEC’20, USA,
2020. USENIX Association.
[109] Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth,
Saeed Shariﬁ Malvajerdi, and Jonathan Ullman. Differentially private fair learning.
In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th
International Conference on Machine Learning, Proceedings of Machine Learning
Research, pages 3000–3008. PMLR, 2019.
[110] Matthew Jagielski, Alina Oprea, Battista Biggio, Chang Liu, Cristina Nita-Rotaru,
and Bo Li. Manipulating machine learning: Poisoning attacks and countermeasures
for regression learning. In 2018 IEEE Symposium on Security and Privacy (S&P),
pages 19–35, 2018.
[111] Matthew Jagielski, Giorgio Severi, Niklas Pousette Harger, and Alina Oprea. Subpopulation data poisoning attacks. In Proceedings of the ACM Conference on Computer and Communications Security, CCS, 2021.
[112] Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private SGD? In Advances in Neural Information Processing Systems, volume 33, pages 22205–22216, 2020.
47

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816

[113] Bargav Jayaraman and David Evans. Evaluating differentially private machine learning in practice. In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19, page 1895–1912, USA, 2019. USENIX Association.
[114] Bargav Jayaraman and David Evans. Are attribute inference attacks just imputation?
In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS ’22, page 1569–1582, New York, NY, USA, 2022. Association for Computing Machinery.
[115] Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in
Natural Language Processing, pages 2021–2031, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[116] Pengfei Jing, Qiyi Tang, Yuefeng Du, Lei Xue, Xiapu Luo, Ting Wang, Sen Nie, andShi Wu. Too good to be safe: Tricking lane detection in autonomous driving withcrafted perturbations. In 30th USENIX Security Symposium (USENIX Security 21),
pages 3237–3254. USENIX Association, August 2021.
[117] Peter Kairouz, Brendan Mcmahan, Shuang Song, Om Thakkar, Abhradeep
Thakurta, and Zheng Xu. Practical and private (deep) learning without sampling or
shufﬂing. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine
Learning Research, pages 5213–5225. PMLR, 18–24 Jul 2021.
[118] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode,
Rachel Cummings, Rafael G. L. D’Oliveira, Hubert Eichner, Salim El Rouayheb,
David Evans, Josh Gardner, Zachary Garrett, Adri`a Gasc´on, Badih Ghazi, Phillip B.
Gibbons, Marco Gruteser, Zaid Harchaoui, Chaoyang He, Lie He, Zhouyuan Huo,
Ben Hutchinson, Justin Hsu, Martin Jaggi, Tara Javidi, Gauri Joshi, Mikhail Khodak,ˇ y, Aleksandra Korolova, Farinaz Koushanfar, Sanmi Koyejo, Tancr`ede
Jakub Konecn´¨
Lepoint, Yang Liu, Prateek Mittal, Mehryar Mohri, Richard Nock, Ayfer Ozgür,
Rasmus Pagh, Mariana Raykova, Hang Qi, Daniel Ramage, Ramesh Raskar, Dawn
Song, Weikang Song, Sebastian U. Stich, Ziteng Sun, Ananda Theertha Suresh, Florian Tramèr, Praneeth Vepakomma, Jianyu Wang, Li Xiong, Zheng Xu, Qiang Yang,
Felix X. Yu, Han Yu, and Sen Zhao. Advances and open problems in federated
learning, 2019.
[119] Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer.
Reluplex: An efﬁcient SMT solver for verifying deep neural networks. In Rupakˇ
Majumdar and Viktor Kuncak,
editors, Computer Aided Veriﬁcation, pages 97–117,
Cham, 2017. Springer International Publishing.
[120] Michael Kearns and Ming Li. Learning in the presence of malicious errors. In
Proceedings of the Twentieth Annual ACM Symposium on Theory of Computing,
STOC ’88, page 267–280, New York, NY, USA, 1988. Association for Computing
Machinery.
[121] Marius Kloft and Pavel Laskov. Security analysis of online centroid anomaly detec48

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858

tion. Journal of Machine Learning Research, 13(118):3681–3724, 2012.
[122] Pang Wei Koh and Percy Liang. Understanding black-box predictions via inﬂuence functions. In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 1885–1894. JMLR. org, 2017.
[123] Moshe Kravchik, Battista Biggio, and Asaf Shabtai. Poisoning attacks on cyberattack detectors for industrial control systems. In Proceedings of the 36th Annual
ACM Symposium on Applied Computing, SAC ’21, page 116–125, New York, NY,
USA, 2021. Association for Computing Machinery.
[124] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall,
Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning – industry perspectives. https://arxiv.org/abs/2002.05646, 2020.
[125] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in thephysical world. https://arxiv.org/abs/1607.02533, 2016.
[126] E. La Malfa and M. Kwiatkowska. The king is naked: On the notion of robustness fornatural language processing. In Proceedings of the Thirty-Sixth AAAI Conference on
Artiﬁcial Intelligence, volume 10, page 11047–57. Association for the Advancementof Artiﬁcial Intelligence, 2022.
[127] Ricky Laishram and Vir Virander Phoha. Curie: A method for protecting SVM
classiﬁer from poisoning attack. CoRR, abs/1606.01584, 2016.
[128] Ralph Langner. Stuxnet: Dissecting a cyberwarfare weapon. IEEE Security & Privacy, 9(3):49–51, 2011.
[129] Mathias Lécuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman
Jana. Certiﬁed robustness to adversarial examples with differential privacy. In 2019
IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May
19-23, 2019, pages 656–672. IEEE, 2019.
[130] Klas Leino and Matt Fredrikson. Stolen memories: Leveraging model memorization
for calibrated white-box membership inference. In Proceedings of the 29th USENIX
Conference on Security Symposium, SEC’20, USA, 2020. USENIX Association.
[131] Alexander Levine and Soheil Feizi. Deep partition aggregation: Provable defenses
against general poisoning attacks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net,
2021.
[132] Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu,
and Jialiang Lu. Hidden backdoors in human-centric language models. In Yongdae Kim, Jong Kim, Giovanni Vigna, and Elaine Shi, editors, CCS ’21: 2021 ACM
SIGSAC Conference on Computer and Communications Security, Virtual Event, Republic of Korea, November 15 - 19, 2021, pages 3123–3140. ACM, 2021.
[133] Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang.
Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Transactions on Dependable and Secure Computing, 18:2088–2105,
2021.
[134] Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song, Srikanth V. Krishnamurthy,
49

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900

Amit K. Roy-Chowdhury, and Ananthram Swami. Adversarial perturbations against
real-time video classiﬁcation systems. CoRR, abs/1807.00458, 2018.
[135] Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. Fine-pruning: Defending
against backdooring attacks on deep neural networks. In Michael Bailey, Sotiris
Ioannidis, Manolis Stamatogiannakis, and Thorsten Holz, editors, Research in Attacks, Intrusions, and Defenses - 21st International Symposium, RAID 2018, Proceedings, Lecture Notes in Computer Science, pages 273–294. Springer Verlag,
2018.
[136] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable adversarial examples and black-box attacks. In International Conference on Learning
Representations, 2017.
[137] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. ABS: Scanning neural networks for back-doors by artiﬁcial brain
stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computerand Communications Security, CCS ’19, page 1265–1282, New York, NY, USA,2019. Association for Computing Machinery.
[138] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang,and Xiangyu Zhang. Trojaning attack on neural networks. In NDSS. The InternetSociety, 2018.
[139] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reﬂection backdoor: A natural
backdoor attack on deep neural networks. In Andrea Vedaldi, Horst Bischof, Thomas
Brox, and Jan-Michael Frahm, editors, Computer Vision – ECCV 2020, pages 182–
199, Cham, 2020. Springer International Publishing.
[140] Daniel Lowd and Christopher Meek. Adversarial learning. In Proceedings of theEleventh ACM SIGKDD International Conference on Knowledge Discovery in Data
Mining, KDD ’05, page 641–647, New York, NY, USA, 2005. Association for Computing Machinery.
[141] Yiwei Lu, Gautam Kamath, and Yaoliang Yu. Indiscriminate data poisoning attacks
on neural networks. https://arxiv.org/abs/2204.09092, 2022.
[142] Yuzhe Ma, Xiaojin Zhu, and Justin Hsu. Data poisoning against differentially-private
learners: Attacks and defenses. In Proceedings of the 28th International Joint Conference on Artiﬁcial Intelligence (IJCAI), 2019.
[143] Pooria Madani and Natalija Vlajic. Robustness of deep autoencoder in intrusion
detection under adversarial contamination. In HoTSoS ’18: Proceedings of the 5th
Annual Symposium and Bootcamp on Hot Topics in the Science of Security, pages
1–8, 04 2018.
[144] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, andAdrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6thInternational Conference on Learning Representations, ICLR 2018, Vancouver, BC,
Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net,
2018.
[145] Saeed Mahloujifar, Esha Ghosh, and Melissa Chase. Property inference from poi50

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942

soning. In 2022 IEEE Symposium on Security and Privacy (S&P), pages 1120–1137,
2022.
[146] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In
IEEE Symposium on Foundations of Computer Science, FOCS ’07, pages 94–103,
Las Vegas, NV, USA, 2007.
[147] Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov. Exploiting unintended feature leakage in collaborative learning. In 2019 IEEE Symposium on Security and Privacy, SP 2019, San Francisco, CA, USA, May 19-23, 2019,
pages 691–706. IEEE, 2019.
[148] Melissa Heikkilä. What does GPT-3 “know” about me? https://www.technologyre
view.com/2022/08/31/1058800/what-does-gpt-3-know-about-me/, August 2022.
MIT Technology Review.
[149] El Mahdi El Mhamdi, Sadegh Farhadkhani, Rachid Guerraoui, Arsany Guirguis,Lˆe-Nguyˆen Hoang, and S´ebastien Rouault. Collaborative learning in the jungle (decentralized, byzantine, heterogeneous, asynchronous and nonconvex learning). In
NeurIPS, 2021.
[150] El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. The Hidden Vulnerability of Distributed Learning in Byzantium. In ICML, 2018.
[151] El Mahdi El Mhamdi, Rachid Guerraoui, and Sébastien Rouault. Distributed momentum for byzantine-resilient stochastic gradient descent. In ICLR, 2021.
[152] Microsoft. Power virtual agents. https://powervirtualagents.microsoft.com/en-us/a
i-chatbot/, 2022. Online.
[153] Dang Minh, H. Xiang Wang, Y. Fen Li, and Tan N. Nguyen. You can’t have AI bothways: Balancing health data privacy and access fairly. Artiﬁcial Intelligence Review
volume, 55:3503–3568, 2022. https://doi.org/10.1007/s10462-021-10088-y.
[154] Ilya Mironov, Kunal Talwar, and Li Zhang. R\’enyi differential privacy of the sampled gaussian mechanism. arXiv preprint arXiv:1908.10530, 2019.
[155] Margaret Mitchell, Giada Pistilli, Yacine Jernite, Ezinwanne Ozoani, Marissa Gerchick, Nazneen Rajani, Sasha Luccioni, Irene Solaiman, Maraim Masoud, Somaieh Nikpoor, Carlos Muñoz Ferrandis, Stas Bekman, Christopher Akiki, Danish
Contractor, David Lansky, Angelina McMillan-Major, Tristan Thrush, Suzana Ilić,Gérard Dupont, Shayne Longpre, Manan Dey, Stella Biderman, Douwe Kiela, Emi
Baylor, Teven Le Scao, Aaron Gokaslan, Julien Launay, and Niklas Muennighoff.
BigScience Large Open-science Open-access Multilingual Language Model. https://huggingface.co/bigscience/bloom, 2022. Hugging Face.
[156] Seungyong Moon, Gaon An, and Hyun Oh Song. Parsimonious black-box adversarial attacks via efﬁcient combinatorial optimization. In International Conference on
Machine Learning (ICML), 2019.
[157] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal
Frossard. Universal adversarial perturbations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[158] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deep51

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984

Fool: A simple and accurate method to fool deep neural networks. https://arxiv.org/
abs/1511.04599, 2015.
[159] Ghulam Muhammad, Fatima Alshehri, Fakhri Karray, Abdulmotaleb El Saddik,
Mansour Alsulaiman, and Tiago H. Falk. A comprehensive survey on multimodal
medical signals fusion for smart healthcare systems. Information Fusion, 76:355–
375, 2021.
[160] Sasi Kumar Murakonda and Reza Shokri. ML Privacy Meter: Aiding regulatorycompliance by quantifying the privacy risks of machine learning, 2020.
[161] Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin
Wongrassamee, Emil C. Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM
Workshop on Artiﬁcial Intelligence and Security, AISec ’17, 2017.
[162] Nina Narodytska and Shiva Kasiviswanathan. Simple black-box adversarial attacks
on deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern
Recognition Workshops (CVPRW), pages 1310–1318, 2017.
[163] Milad Nasr, Reza Shokri, and Amir Houmansadr. Comprehensive privacy analysis
of deep learning: Passive and active white-box inference attacks against centralized
and federated learning. In IEEE Symposium on Security and Privacy, pages 739–
753. IEEE, 2019.
[164] Milad Nasr, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Nicholas
Carlini. Adversary instantiation: Lower bounds for differentially private machinelearning. In IEEE Symposium on Security & Privacy, IEEE S&P ’21, 2021. https://arxiv.org/abs/2101.04535.
[165] National Security Commission on Artiﬁcial Intelligence. Final report. https://www.
nscai.gov/2021-final-report/, 2021.
[166] Blaine Nelson, Marco Barreno, Fuching Jack Chi, Anthony D. Joseph, Benjamin I.P.
Rubinstein, Udam Saini, Charles Sutton, and Kai Xia. Exploiting machine learning
to subvert your spam ﬁlter. In First USENIX Workshop on Large-Scale Exploits andEmergent Threats (LEET 08), San Francisco, CA, 2008. USENIX Association.
[167] Jessica Newman. A Taxonomy of Trustworthiness for Artiﬁcial Intelligence: Connecting Properties of Trustworthiness with Risk Management and the AI Lifecycle. Technical report, Center for Long Term Cybersecurity, University of California,
Berkeley, 2023. Online: https://cltc.berkeley.edu/wp-content/uploads/2023/01/Tax
onomy of AI Trustworthiness.pdf.
[168] J. Newsome, B. Karp, and D. Song. Polygraph: Automatically generating signatures
for polymorphic worms. In 2005 IEEE Symposium on Security and Privacy (S&P),
pages 226–241, 2005.
[169] Thien Duc Nguyen, Phillip Rieger, Huili Chen, Hossein Yalame, Helen Möllering,
Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Shaza
Zeitouni, Farinaz Koushanfar, Ahmad-Reza Sadeghi, and Thomas Schneider.
FLAME: Taming backdoors in federated learning. In 31st USENIX Security Symposium (USENIX Security 22), pages 1415–1432, Boston, MA, August 2022. USENIX
52

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026

Association.
[170] National Institute of Standards and Technology. Artiﬁcail Intelligence Risk Management Framework (AI RMF 1.0). https://doi.org/10.6028/NIST.AI.100-1, 2023.
Online.
[171] OpenAI. ChatGPT: Optimizing Language Models for Dialogue. https://openai.com/blog/chatgpt/, 2022. Online.
[172] Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Knockoff nets: Stealing
functionality of black-box models. In 2019 IEEE/CVF Conference on Computer
Vision and Pattern Recognition (CVPR), pages 4949–4958, 2019.
[173] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machinelearning: from phenomena to black-box attacks using adversarial samples. https://arxiv.org/abs/1605.07277, 2016.
[174] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z. Berkay Celik,and Ananthram Swami. Practical black-box attacks against machine learning. In
Proceedings of the 2017 ACM on Asia Conference on Computer and Communications Security, ASIA CCS ’17, page 506–519, New York, NY, USA, 2017. Association for Computing Machinery.
[175] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.
Distillation as a defense to adversarial perturbations against deep neural networks.
In 2016 IEEE Symposium on Security and Privacy (S&P), pages 582–597, 2016.
[176] Andrea Paudice, Luis Mu˜noz-Gonz´alez, and Emil C. Lupu. Label sanitization against label ﬂipping poisoning attacks. In Carlos Alzate, Anna Monreale, Haytham Assem, Albert Bifet, Teodora Sandra Buda, Bora Caglayan, Brett` Stefan Kramer, Niklas Lavesson,
Drury, Eva Garcı́a-Martı́n, Ricard Gavalda,
Michael Madden, Ian Molloy, Maria-Irina Nicolae, and Mathieu Sinn, editors,
Nemesis/UrbReas/SoGood/IWAISe/GDM@PKDD/ECML, volume 11329 of Lecture
Notes in Computer Science, pages 5–15. Springer, 2018.
[177] R. Perdisci, D. Dagon, Wenke Lee, P. Fogla, and M. Sharif. Misleading worm signature generators using deliberate noise injection. In 2006 IEEE Symposium on
Security and Privacy (S&P’06), Berkeley/Oakland, CA, 2006. IEEE.
[178] Neehar Peri, Neal Gupta, W. Ronny Huang, Liam Fowl, Chen Zhu, Soheil Feizi,
Tom Goldstein, and John P. Dickerson. Deep k-nn defense against clean-label datapoisoning attacks. In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision– ECCV 2020 Workshops, pages 55–70, Cham, 2020. Springer International Publishing.
[179] Fabio Pierazzi, Feargus Pendlebury, Jacopo Cortellazzi, and Lorenzo Cavallaro. Intriguing properties of adversarial ML attacks in the problem space. In 2020 IEEE
Symposium on Security and Privacy (S&P), pages 1308–1325. IEEE Computer Society, 2020.
[180] Gauthama Raman M. R., Chuadhry Mujeeb Ahmed, and Aditya Mathur. Machinelearning for intrusion detection in industrial control systems: Challenges and lessons
from experimental evaluation. Cybersecurity, 4(27), 2021.
53

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068

[181] Aida Rahmattalabi, Shahin Jabbari, Himabindu Lakkaraju, Phebe Vayanos, Max
Izenberg, Ryan Brown, Eric Rice, and Milind Tambe. Fair inﬂuence maximization:
A welfare optimization approach. In Proceedings of the AAAI Conference on Artiﬁcial Intelligence 35th, 2021.
[182] Adnan Siraj Rakin, Md Haﬁzul Islam Chowdhuryy, Fan Yao, and Deliang Fan.
DeepSteal: Advanced model extractions leveraging efﬁcient weight stealing in memories. In 2022 IEEE Symposium on Security and Privacy (S&P), pages 1157–1174,
2022.
[183] Dhanesh Ramachandram and Graham W. Taylor. Deep multimodal learning: A
survey on recent advances and trends. IEEE Signal Processing Magazine, 34(6):96–
108, 2017.
[184] Elan Rosenfeld, Ezra Winston, Pradeep Ravikumar, and Zico Kolter. Certiﬁed robustness to label-ﬂipping attacks via randomized smoothing. In International Conference on Machine Learning, pages 8230–8241. PMLR, 2020.
[185] Benjamin IP Rubinstein, Blaine Nelson, Ling Huang, Anthony D Joseph, Shinghon Lau, Satish Rao, Nina Taft, and J Doug Tygar. Antidote: understanding anddefending against poisoning of anomaly detectors. In Proceedings of the 9th ACM
SIGCOMM conference on Internet measurement, pages 1–14, 2009.
[186] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and HervéJégou. White-box vs black-box: Bayes optimal strategies for membership inference.
In ICML, volume 97 of Proceedings of Machine Learning Research, pages 5558–
5567. PMLR, 2019.
[187] Carl Sabottke, Octavian Suciu, and Tudor Dumitras. Vulnerability disclosure in theage of social media: Exploiting Twitter for predicting real-world exploits. In 24th
USENIX Security Symposium (USENIX Security 15), pages 1041–1056, Washington, D.C., August 2015. USENIX Association.
[188] Ahmed Salem, Giovanni Cherubin, David Evans, Boris Köpf, Andrew Paverd, Anshuman Suri, Shruti Tople, and Santiago Zanella-Béguelin. SoK: Let the privacy
games begin! A uniﬁed treatment of data inference privacy in machine learning.
https://arxiv.org/abs/2212.10986, 2022.
[189] Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic
backdoor attacks against machine learning models. https://arxiv.org/abs/2003.036
75, 2020.
[190] Oscar Schwartz. In 2016, Microsoft’s racist chatbot revealed the dangers of online
conversation: The bot learned language from people on Twitter—but it also learned
values. https://spectrum.ieee.org/in-2016-microsofts-racist-chatbot-revealed-the-dangers-of-online-conversation, 2019. IEEE Spectrum.
[191] Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom
Goldstein. Just how toxic is data poisoning? A uniﬁed benchmark for backdoor anddata poisoning attacks. https://arxiv.org/abs/2006.12557, 2020. arXiv.
[192] Giorgio Severi, Jim Meyer, Scott Coull, and Alina Oprea. Explanation-guided backdoor poisoning attacks against malware classiﬁers. In 30th USENIX Security Sym54

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110

posium (USENIX Security 2021), 2021.
[193] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer,
Tudor Dumitras, and Tom Goldstein. Poison frogs! Targeted clean-label poisoning
attacks on neural networks. In Advances in Neural Information Processing Systems,
pages 6103–6113, 2018.
[194] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K. Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In
Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security, October 2016.
[195] Vasu Sharma, Ankita Kalra, Vaibhav, Simral Chaudhary, Labhesh Patel, andLP Morency. Attend and attack: Attention guided adversarial attacks on visual question answering models. https://nips2018vigil.github.io/static/papers/accepted/33.pd
f, 2018.
[196] Ryan Sheatsley, Blaine Hoak, Eric Pauley, Yohan Beugin, Michael J. Weisman, andPatrick McDaniel. On the robustness of domain constraints. In Proceedings ofthe 2021 ACM SIGSAC Conference on Computer and Communications Security,
CCS ’21, page 495–515, New York, NY, USA, 2021. Association for Computing
Machinery.
[197] Virat Shejwalkar and Amir Houmansadr. Manipulating the byzantine: Optimizing
model poisoning attacks and defenses for federated learning. In NDSS, 2021.
[198] Virat Shejwalkar, Amir Houmansadr, Peter Kairouz, and Daniel Ramage. Back to
the drawing board: A critical evaluation of poisoning attacks on production federated
learning. In 43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco,
CA, USA, May 22-26, 2022, pages 1354–1371. IEEE, 2022.
[199] Cong Shi, Tianfang Zhang, Zhuohang Li, Huy Phan, Tianming Zhao, Yan Wang,
Jian Liu, Bo Yuan, and Yingying Chen. Audio-domain position-independent backdoor attack via unnoticeable triggers. In Proceedings of the 28th Annual International Conference on Mobile Computing And Networking, MobiCom ’22, page583–595, New York, NY, USA, 2022. Association for Computing Machinery.
[200] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership
inference attacks against machine learning models. In 2017 IEEE Symposium on
Security and Privacy (SP), pages 3–18. IEEE, 2017.
[201] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership
inference attacks against machine learning models. In IEEE Symposium on Security
and Privacy (S&P), Oakland, 2017.
[202] Satya Narayan Shukla, Anit Kumar Sahu, Devin Willmott, and Zico Kolter. Simple
and efﬁcient hard label black-box adversarial attacks in low query budget regimes.
In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery &Data Mining, KDD ’21, page 1461–1469, New York, NY, USA, 2021. Association
for Computing Machinery.
[203] Ilia Shumailov, Yiren Zhao, Daniel Bates, Nicolas Papernot, Robert Mullins, andRoss Anderson. Sponge examples: Energy-latency attacks on neural networks. http55

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152

s://arxiv.org/abs/2006.03463, 2020.
[204] Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. An abstract
domain for certifying neural networks. Proc. ACM Program. Lang., 3, January 2019.
[205] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini,
Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. FixMatch: Simplifying
semi-supervised learning with consistency and conﬁdence. In Proceedings of the34th International Conference on Neural Information Processing Systems, NIPS’20,
Red Hook, NY, USA, 2020. Curran Associates Inc.
[206] Saleh Soltan, Shankar Ananthakrishnan, Jack FitzGerald, Rahul Gupta, Wael
Hamza, Haidar Khan, Charith Peris, Stephen Rawls, Andy Rosenbaum, Anna
Rumshisky, Chandana Satya Prakash, Mukund Sridhar, Fabian Triefenbach, Apurv
Verma, Gokhan Tur, and Prem Natarajan. AlexaTM 20B: Few-shot learning using alarge-scale multilingual seq2seq model. https://www.amazon.science/publications/alexatm-20b-few-shot-learning-using-a-large-scale-multilingual-seq2seq-model,
2022. Amazon.
[207] Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramèr, Atul Prakash, and Tadayoshi Kohno. Physical adversarial examples for object detectors. In 12th USENIX Workshop on Offensive Technologies(WOOT 18), Baltimore, MD, August 2018. USENIX Association.
[208] Shuang Song and David Marn. Introducing a new privacy testing library in TensorFlow, 2020.
[209] N. Srndic and P. Laskov. Practical evasion of a learning-based classiﬁer: A casestudy. In Proc. IEEE Security and Privacy Symposium, 2014.
[210] Jacob Steinhardt, Pang Wei W Koh, and Percy S Liang. Certiﬁed defenses for datapoisoning attacks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[211] Octavian Suciu, Scott E Coull, and Jeffrey Johns. Exploring adversarial examples
in malware detection. In 2019 IEEE Security and Privacy Workshops (SPW), pages
8–14. IEEE, 2019.
[212] Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning FAIL? generalized transferability for evasion
and poisoning attacks. In 27th USENIX Security Symposium (USENIX Security 18),
pages 1299–1316, 2018.
[213] Jingwei Sun, Ang Li, Louis DiValentin, Amin Hassanzadeh, Yiran Chen, and Hai
Li. FL-WBC: Enhancing robustness against model poisoning attacks in federated
learning from a client perspective. In NeurIPS, 2021.
[214] Ziteng Sun, Peter Kairouz, Ananda Theertha Suresh, and H Brendan McMahan. Can
you really backdoor federated learning? arXiv:1911.07963, 2019.
[215] Anshuman Suri and David Evans. Formalizing and estimating distribution inference
risks. Proceedings on Privacy Enhancing Technologies, 2022.
[216] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan,
56

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
2187
2188
2189
2190
2191
2192
2193
2194

Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
[217] Rahim Taheri, Reza Javidan, Mohammad Shojafar, Zahra Pooranian, Ali Miri, andMauro Conti. On defending against label ﬂipping attacks on malware detection
systems. CoRR, abs/1908.04473, 2019.
[218] Anvith Thudi, Ilia Shumailov, Franziska Boenisch, and Nicolas Papernot. Bounding
membership inference. https://arxiv.org/abs/2202.12232, 2022.
[219] Florian Tramer. Detecting adversarial examples is (Nearly) as hard as classifying
them. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang
Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference
on Machine Learning, volume 162 of Proceedings of Machine Learning Research,
pages 21692–21702. PMLR, 17–23 Jul 2022.
[220] Florian Tramer, Jens Behrmann, Nicholas Carlini, Nicolas Papernot, and JoernHenrik Jacobsen. Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations. In Hal Daumé III and Aarti Singh, editors, Proceedings of the37th International Conference on Machine Learning, volume 119 of Proceedings of
Machine Learning Research, pages 9561–9571. PMLR, 13–18 Jul 2020.
[221] Florian Tramèr, Nicholas Carlini, Wieland Brendel, and Aleksander Ma̧dry. On
adaptive attacks to adversarial example defenses. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red
Hook, NY, USA, 2020. Curran Associates Inc.
` Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart.
[222] Florian Tramer,
Stealing machine learning models via prediction APIs. In USENIX Security, 2016.
` Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc[223] Florian Tramer,
Daniel. The space of transferable adversarial examples. https://arxiv.org/abs/1704.0
3453, 2017.
[224] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor
attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and
R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018.
[225] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In International Conference on Learning Representations, 2019.
[226] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor
attacks. In ICLR, 2019.
[227] Sridhar Venkatesan, Harshvardhan Sikka, Rauf Izmailov, Ritu Chadha, Alina Oprea,and Michael J. De Lucia. Poisoning attacks and data sanitization mitigations for machine learning models in network intrusion detection systems. In MILCOM, pages
874–879. IEEE, 2021.
[228] Brandon Vigliarolo. GPT-3 ’prompt injection’ attack causes bad bot manners. https://www.theregister.com/2022/09/19/in brief security/, 2022. The Register, Online.
[229] Eric Wallace, Tony Z. Zhao, Shi Feng, and Sameer Singh. Concealed data poisoning
57

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2195
2196
2197
2198
2199
2200
2201
2202
2203
2204
2205
2206
2207
2208
2209
2210
2211
2212
2213
2214
2215
2216
2217
2218
2219
2220
2221
2222
2223
2224
2225
2226
2227
2228
2229
2230
2231
2232
2233
2234
2235
2236

attacks on NLP models. In NAACL, 2021.
[230] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao
Zheng, and Ben Y. Zhao. Neural Cleanse: Identifying and Mitigating Backdoor
Attacks in Neural Networks. In 2019 IEEE Symposium on Security and Privacy(SP), pages 707–723, San Francisco, CA, USA, May 2019. IEEE.
[231] Haotao Wang, Tianlong Chen, Shupeng Gui, Ting-Kuei Hu, Ji Liu, and Zhangyang
Wang. Once-for-All Adversarial Training: In-Situ Tradeoff between Robustness andAccuracy for Free. In Proceedings of the 34th Conference on Neural Information
Processing Systems (NeurIPS 2020), Vancouver, Canada, 2020.
[232] Hongyi Wang, Kartik Sreenivasan, Shashank Rajput, Harit Vishwakarma, Saurabh
Agarwal, Jy-yong Sohn, Kangwook Lee, and Dimitris Papailiopoulos. Attack of theTails: Yes, You Really Can Backdoor Federated Learning. In NeurIPS, 2020.
[233] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Formal
security analysis of neural networks using symbolic intervals. In 27th USENIX Security Symposium (USENIX Security 18), pages 1599–1614, Baltimore, MD, August
2018. USENIX Association.
[234] Wenxiao Wang, Alexander Levine, and Soheil Feizi. Improved certiﬁed defenses
against data poisoning with (deterministic) ﬁnite aggregation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato,
editors, International Conference on Machine Learning, ICML 2022, 17-23 July
2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning
Research, pages 22769–22783. PMLR, 2022.
[235] Xiaosen Wang and Kun He. Enhancing the transferability of adversarial attacks
through variance tuning. In IEEE Conference on Computer Vision and Pattern
Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 1924–1933. Computer
Vision Foundation / IEEE, 2021.
[236] Xingxing Wei, Jun Zhu, Sha Yuan, and Hang Su. Sparse adversarial perturbations
for videos. In Proceedings of the Thirty-Third AAAI Conference on Artiﬁcial Intelligence and Thirty-First Innovative Applications of Artiﬁcial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artiﬁcial Intelligence,
AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.
[237] Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao
Zheng, and Ben Y. Zhao. Backdoor attacks against deep learning systems in thephysical world. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6202–6211, 2020.
[238] Dongxian Wu and Yisen Wang. Adversarial neuron pruning puriﬁes backdooreddeep models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 16913–16925. Curran Associates, Inc., 2021.
[239] Zhen Xiang, David J. Miller, and George Kesidis. Post-training detection of backdoor attacks for two-class and multi-attack scenarios. In The Tenth International
Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29,
58

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2237
2238
2239
2240
2241
2242
2243
2244
2245
2246
2247
2248
2249
2250
2251
2252
2253
2254
2255
2256
2257
2258
2259
2260
2261
2262
2263
2264
2265
2266
2267
2268
2269
2270
2271
2272
2273
2274
2275
2276
2277
2278

2022. OpenReview.net, 2022.
[240] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert, andFabio Roli. Is feature selection secure against training data poisoning? In International Conference on Machine Learning, pages 1689–1698, 2015.
[241] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised
data augmentation for consistency training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing
Systems, volume 33, pages 6256–6268. Curran Associates, Inc., 2020.
[242] Weilin Xu, Yanjun Qi, and David Evans. Automatically evading classiﬁers. In
Proceedings of the 2016 Network and Distributed Systems Symposium, pages 21–
24, 2016.
[243] Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darrell, and Dawn
Song. Fooling vision and language models despite localization and attention mechanism. https://arxiv.org/abs/1709.08693, 2017.
[244] Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A. Gunter, and Bo Li. Detecting AI trojans using meta neural analysis. In IEEE Symposium on Security andPrivacy, S&P 2021, pages 103–120, United States, May 2021.
[245] Karren Yang, Wan-Yi Lin, Manash Barman, Filipe Condessa, and Zico Kolter.
Defending multimodal fusion models against single-source adversaries. In 2021
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE
Xplore, 2022.
[246] Limin Yang, Zhi Chen, Jacopo Cortellazzi, Feargus Pendlebury, Kevin Tu, Fabio
Pierazzi, Lorenzo Cavallaro, and Gang Wang. Jigsaw puzzle: Selective backdoor
attack to subvert malware classiﬁers. CoRR, abs/2202.05470, 2022.
[247] Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Latent backdoor attacks
on deep neural networks. In Proceedings of the 2019 ACM SIGSAC Conference on
Computer and Communications Security, CCS ’19, page 2041–2055, New York,NY, USA, 2019. Association for Computing Machinery.
[248] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, andReza Shokri. Enhanced membership inference attacks against machine learningmodels. In Proceedings of the 2022 ACM SIGSAC Conference on Computer andCommunications Security, CCS ’22, page 3093–3106, New York, NY, USA, 2022.
Association for Computing Machinery.
[249] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk
in machine learning: Analyzing the connection to overﬁtting. In IEEE Computer
Security Foundations Symposium, CSF ’18, pages 268–282, 2018. https://arxiv.org/
abs/1709.01604.
[250] Dong Yin, Yudong Chen, Ramchandran Kannan, and Peter Bartlett. ByzantineRobust Distributed Learning: Towards Optimal Statistical Rates. In ICML, 2018.
[251] Youngjoon Yu, Hong Joo Lee, Byeong Cheon Kim, Jung Uk Kim, and Yong Man
Ro. Investigating vulnerability to adversarial examples on multimodal data fusion in
deep learning. https://arxiv.org/abs/2005.10987, 2020. Online.
59

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2279
2280
2281
2282
2283
2284
2285
2286
2287
2288
2289
2290
2291
2292
2293
2294
2295
2296
2297
2298
2299
2300
2301
2302
2303
2304
2305
2306
2307
2308
2309
2310
2311
2312
2313
2314
2315
2316
2317
2318
2319
2320

[252] Santiago Zanella-B´eguelin, Lukas Wutschitz, Shruti Tople, Victor R¨uhle, Andrew
Paverd, Olga Ohrimenko, Boris Köpf, and Marc Brockschmidt. Analyzing information leakage of updates to natural language models. In ACM Conference on Computer and Communications Security, page 363–375, New York, NY, USA, 2020.
Association for Computing Machinery.
[253] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia. Adversarial
unlearning of backdoors via implicit hypergradient. In International Conference on
Learning Representations, 2022.
[254] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning (still) requires rethinking generalization. Commun.
ACM, 64(3):107–115, feb 2021.
[255] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, andMichael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the36th International Conference on Machine Learning, volume 97 of Proceedings of
Machine Learning Research, pages 7472–7482. PMLR, 09–15 Jun 2019.
[256] Su-Fang Zhang, Jun-Hai Zhai, Bo-Jun Xie, Yan Zhan, and Xin Wang. Multimodal representation learning: Advances, trends and challenges. In 2019 International Conference on Machine Learning and Cybernetics (ICMLC), pages 1–6.
IEEE, 2019.
[257] Susan Zhang, Mona Diab, and Luke Zettlemoyer. Democratizing access to largescale language models with OPT-175B. https://ai.facebook.com/blog/democratizi
ng-access-to-large-scale-language-models-with-opt-175b/, 2022. Meta AI.
[258] Wanrong Zhang, Shruti Tople, and Olga Ohrimenko. Leakage of dataset properties
in Multi-Party machine learning. In 30th USENIX Security Symposium (USENIX
Security 21), pages 2687–2704. USENIX Association, August 2021.
[259] Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. Adversarial
attacks on deep-learning models in natural language processing: A survey. ACM
Trans. Intell. Syst. Technol., 11(3), apr 2020.
[260] Zhengming Zhang, Ashwinee Panda, Linyue Song, Yaoqing Yang, Michael Mahoney, Prateek Mittal, Ramchandran Kannan, and Joseph Gonzalez. Neurotoxin:
Durable backdoors in federated learning. In Kamalika Chaudhuri, Stefanie Jegelka,
Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the39th International Conference on Machine Learning, volume 162 of Proceedings of
Machine Learning Research, pages 26429–26446. PMLR, 17–23 Jul 2022.
[261] Zhikun Zhang, Min Chen, Michael Backes, Yun Shen, and Yang Zhang. Inference attacks against graph neural networks. In 31st USENIX Security Symposium(USENIX Security 22), 2022.
[262] Junhao Zhou, Yufei Chen, Chao Shen, and Yang Zhang. Property inference attacks
against GANs. In Proceedings of Network and Distributed System Security, NDSS,
2022.
[263] Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom
60

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2321
2322
2323
2324
2325
2326
2327
2328

Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine
Learning Research, pages 7614–7623. PMLR, 09–15 Jun 2019.
[264] Giulio Zizzo, Chris Hankin, Sergio Maffeis, and Kevin Jones. Adversarial machinelearning beyond the image domain. In Proceedings of the 56th Annual Design Automation Conference 2019, DAC ’19, New York, NY, USA, 2019. Association for
Computing Machinery.

61

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2330

Note: one may click on the page number shown at the end of the deﬁnition of each glossary
entry to go to the page where the term is used.

2331

A.

2329

2332
2333

2334
2335
2336
2337
2338

2339
2340

2341
2342
2343
2344
2345
2346

Appendix: Glossary

adversarial examples Modiﬁed testing samples which induce mis-classiﬁcation of a machine learning model at deployment time. v, 8
Area Under the Curve In ML the Area Under the Curve (AUC) is a measure of the ability of a classiﬁer to distinguish between classes. The higher the AUC, the better the
performance of the model at distinguishing between the two classes. AUC measuresthe entire two-dimensional area underneath the R ECEIVER O PERATING C HARAC TERISTICS (ROC) curve. 30
availability attack Adversarial attacks against machine learning which degrade the overall model performance. 8
backdoor pattern A trigger pattern inserted into a data sample to induce mis-classiﬁcationof a poisoned model. For example, in computer vision it may be constructed from a
set of neighboring pixels, e.g., a white square, and added to a speciﬁc target label. To
mount a backdoor attack, the adversary ﬁrst poisons the data by adding the trigger toa subset of the clean data and changing their corresponding labels to the target label.
92348

backdoor poisoning attacks Poisoning attacks against machine learning which change
the prediction on samples including a backdoor pattern. 82349

classiﬁcation Type of supervised learning in which data labels are discrete. 7

2347

2350
2351
2352
2353
2354

2355
2356

2357
2358

convolutional neural networks A Convolutional Neural Network (CNN) is a class of artiﬁcial neural networks whose architecture connects neurons from one layer to thenext layer and includes at least one layer performing convolution operations. CNNs
are typically applied to image analysis and classiﬁcation. See [92] for further details.
7, 31
data poisoning Poisoning attacks in which a part of the training data is under the controlof the adversary. 7
data privacy Attacks against machine learning models to extract sensitive information
about training data. 92360

data reconstruction Data privacy attacks which reconstruct sensitive information about
training data records. 92361

deployment stage Stage of ML pipeline in which the model is deployed on new data. 7

2359

62

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2362
2363

2364
2365
2366
2367

2368
2369

2370
2371

2372
2373
2374
2375
2376
2377
2378
2379
2380
2381

2382
2383
2384

2385
2386

2387
2388

2389
2390
2391
2392
2393

2394
2395
2396
2397

discriminative Type of machine learning methods which learn to discriminate between
classes. 7
energy-latency attacks Attacks that exploit the performance dependency on hardware andmodel optimizations to negate the effects of hardware optimizations, increase computation latency, increase hardware temperature and massively increase the amountof energy consumed. 8
ensemble learning Type of a meta machine learning approach that combines the predictions of several models to improve the performance of the combination. 7
federated learning Type of collaborative machine learning, in which multiple users train
jointly a machine learning model. 7
federated learning models Federated learning is a methodology to train a decentralizedmachine learning model (e.g., deep neural networks or a pre-trained large language
model) across multiple end-devices without sharing the data residing on each device.
Thus, the end-devices collaboratively train a global model by exchanging model updates with a server that aggregates the updates. Compared to traditional centralized
learning where the data are pooled, federated learning has advantages in terms of data
privacy and security but these may come as tradeoffs to the capabilities of the models learned through federated data. Other potential problems one needs to contend
with here concern the trustworthiness of the end-devices and the impact of malicious
actors on the learned model. 31
feed-forward neural networks A Feed Forward Neural Network is an artiﬁcial neuralnetwork in which the connections between nodes is from one layer to the next anddo not form a cycle. See [92] for further details. 31
formal methods Formal methods are mathematically rigorous techniques for the speciﬁcation, development, and veriﬁcation of software systems. 18
generative Type of machine learning methods which learn the data distribution and cangenerate new examples from distribution. 7
generative adversarial networks A generative adversarial network (GAN) is a class ofmachine learning frameworks in which two neural networks contest with each otherin the form of a zero-sum game, where one agent’s gain is another agent’s loss.
GAN’s learn to generate new data with the same statistics as the training set. See [92]for further details. 31
graph neural networks A Graph Neural Network (GNN) is an optimizable transformation on all attributes of the graph (nodes, edges, global-context) that preserves thegraph symmetries (permutation invariances). GNNs utilize a “graph-in, graph-out”
architecture that takes an input graph with information loaded into its nodes, edges
63

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2398
2399

2400
2401
2402
2403
2404
2405
2406

2407
2408

2409
2410

2411
2412

2413
2414

2415
2416

2417
2418

and global-context, and progressively transform these embeddings into an output
graph with the same connectivity as that of the input graph. 31
hidden Markov models A hidden Markov model (HMM) is a statistical Markov model inwhich the system being modeled is assumed to be a Markov process with unobservable states. In addition, the model provides an observable process whose outcomesare ”inﬂuenced” by the outcomes of Markov model in a known way. HMM can beused to describe the evolution of observable events that depend on internal factors,
which are not directly observable. In machine learning it is assumed that the internal
state of a model is hidden but not the hyperparameters. 31
integrity attack Adversarial attacks against machine learning which change the output
prediction of the machine learning model. 8label ﬂipping a type of data poisoning attack where the adversary is restricted to changing
the training labels. 21
label limit Capability in which the attacker in some scenarios does not control the labelsof training samples in supervised learning. 9
logistic regression Type of linear classiﬁer that predicts the probability of an observation
to be part of a class.. 7
membership-inference attacks Data privacy attacks to determine if a data sample waspart of the training set of a machine learning model. 9
memorization The ability of a machine learning model to encode, remember, and potentially emit the training data. 9

2420

model control Capability in which the attacker has control over machine learning model
parameters. 9

2421

model extraction Type of privacy attack to extract model architecture and parameters. 9

2419

2422
2423

2424
2425

2426
2427
2428

model poisoning Poisoning attacks in which the model parameters are under the controlof the adversary. 8
model privacy Attacks against machine learning models to extract sensitive information
about the model. 9
multimodal models Modality is associated with the sensory modalities which represent
primary human channels of communication and sensation, such as vision or touch.
Multimodal models process and relate information from multiple modalities. 35

64

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2431

out-of-distribution This term refers to data that was collected at a different time, and possibly under different conditions or in a different environment, than the data collected
to train the model. 33

2432

poisoning attacks Adversarial attacks against machine learning at training time. 7

2429
2430

2433
2434
2435

2436
2437

2438
2439

2440
2441
2442

2443
2444

2445
2446
2447
2448
2449

2450
2451

2452
2453
2454
2455

2456
2457
2458
2459
2460
2461

prompt injections Malicious plain text instructions to a generative AI system that uses
textual instructions (a “prompt”) to accomplish a task causing the AI system to generate text on a topic prohibited by the designers of the system. 36
property inference Data privacy attacks which infer global property about the training
data of a machine learning model. 9
query access Capability in which the attacker can issue queries to a trained machine learning model and obtain predictions. 9
Receiver Operating Characteristics (ROC) In ML the Receiver Operating Characteristics (ROC) curve plots true positive rate versus false positive rate for a classiﬁer.
62
reinforcement learning Type of machine learning in which an agent interacts with theenvironment and learns to take actions which optimize a reward function. 7
rowhammer attacks Rowhammer is a software-based fault-injection attack that exploits
DRAM disturbance errors via user-space applications and allows the attacker to infer
information about certain victim secrets stored in memory cells. Mounting this attack
requires attacker’s control of a user-space unprivileged process that runs on the samemachine as the victim’s ML model. 31
semi-supervised learning Type of machine learning in which a small number of training
samples are labeled, while the majority are unlabeled. 7
shadow models Shadow models imitate the behavior of the target model. The training
datasets and thus the ground truth about membership in these datasets are known forthese models. Typically, the attack model is trained on the labeled inputs and outputsof the shadow models. 25
side channel side channels allow an attacker to infer information about a secret by observing nonfunctional characteristics of a program, such as execution time or memory or
by measuring or exploiting indirect coincidental effects of the system or its hardware,like power consumption variation, electromagnetic emanations, while the program is
executing. Most commonly, such attacks aim to exﬁltrate sensitive information, including cryptographic keys. 31

65

NIST AI 100-2e2023 ipd (Initial Public Draft)
March 2023

2463

source code control Capability in which the attacker has control over the source code of
the machine learning algorithm. 92464

supervised learning Type of machine learning methods based on labeled data. 7

2462

2465
2466
2467
2468

2469
2470

2471
2472

2473
2474

2475
2476

Support Vector Machines A Support Vector Machine implements a decision function in
the form of a hyperplane that serves to separate (i.e., classify) observations belonging
to one class from another based on patterns of information about those observations(i.e., features). . 7, 8, 21, 31
targeted poisoning attacks Poisoning attacks against machine learning which change theprediction on a small number of targeted samples. 8
testing data control Capability in which the attacker has control over the testing data input
to the machine learning model. 9
training data control Capability in which the attacker has control over a part of the training data of a machine learning model. 9
training stage Stage of machine learning pipeline in which the model is trained using
training data. 7

2481

trojans A malicious code/logic inserted into the code of a software or hardware system,
typically without the knowledge and consent of the organization that owns/develops
the system, that is difﬁcult to detect and may appear harmless, but can alter theintended function of the system upon a signal from an attacker to cause a malicious
behavior desired by the attacker. 3

2482

unsupervised learning Type of machine learning methods based on unlabeled data. 7

2477
2478
2479
2480

66