Withdrawn Draft
Warning Notice
The attached draft document has been withdrawn and is provided solely for historical purposes.
It has been superseded by the document identified below.
Withdrawal Date March 8, 2023
Original Release Date October 30, 2019

Superseding Document
Status Initial Public Draft (ipd)Series/Number NIST AI 100-2e2023 ipd
Title Adversarial Machine Learning: A Taxonomy and Terminology ofAttacks and Mitigations
Publication Date March 8, 2023
DOI https://doi.org/10.6028/NIST.AI.100-2e2023.ipd
CSRC URL https://csrc.nist.gov/publications/detail/whitepaper/2023/03/08/adversarial-machine-learning-taxonomy-andterminology/draft
Additional Information

1

Draft NISTIR 8269

2
3
4

A Taxonomy and Terminology ofAdversarial Machine Learning

5
6
7
8
9
10
11

Elham Tabassi
Kevin J. Burns
Michael Hadjimichael
Andres D. Molina-Markham
Julian T. Sexton

12
13
14
15
16
17

This publication is available free of charge from:https://doi.org/10.6028/NIST.IR.8269-draft

18

Draft NISTIR 8269

19
20
21

A Taxonomy and Terminology ofAdversarial Machine Learning

22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

42
43
44
45
46
47
48
49

Elham Tabassi
National Institute of Standards and Technology
Information Technology Laboratory
Kevin J. Burns
Michael Hadjimichael
Andres D. Molina-Markham
Julian T. Sexton
National Cybersecurity Center of Excellence
The MITRE Corporation

This publication is available free of charge from:https://doi.org/10.6028/NIST.IR.8269-draft

October 2019

U.S. Department of Commerce
Wilbur L. Ross, Jr., Secretary
National Institute of Standards and Technology
Walter Copan, NIST Director and Under Secretary of Commerce for Standards and Technology

50
51

National Institute of Standards and Technology Interagency or Internal Report 8269

52
53This publication is available free of charge from:https://doi.org/10.6028/NIST.IR.8269-draft

54
55
56
57

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an
experimental procedure or concept adequately. Such identification is not intended to imply recommendation or
endorsement by NIST, nor is it intended to imply that the entities, materials, or equipment are necessarily the best
available for the purpose.

58
59
60
61
62
63
64
65
66

There may be references in this publication to other publications currently under development by NIST in accordancewith its assigned statutory responsibilities. The information in this publication, including concepts and methodologies,
may be used by federal agencies even before the completion of such companion publications. Thus, until eachpublication is completed, current requirements, guidelines, and procedures, where they exist, remain operative. For
planning and transition purposes, federal agencies may wish to closely follow the development of these new
publications by NIST.

67
68
69
70
71

Public comment period: October 30, 2019 through December 16, 201972

All comments are subject to release under the Freedom of Information Act (FOIA).

73

35 pages (October 2019)

Organizations are encouraged to review all draft publications during public comment periods and provide feedback to
NIST. Many NIST cybersecurity publications, other than the ones noted above, are available athttps://csrc.nist.gov/publications.

National Institute of Standards and Technology
Attn: National Cybersecurity Center of Excellence (NCCoE)
100 Bureau Drive (Mail Stop 2002) Gaithersburg, Maryland 20899-2000
Email: ai-nccoe@nist.gov

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

74

Reports on Computer Systems Technology

75
76
77
78
79
80
81
82

The Information Technology Laboratory (ITL) at the National Institute of Standards andTechnology (NIST) promotes the U.S. economy and public welfare by providing technicalleadership for the Nation’s measurement and standards infrastructure. ITL develops tests, test
methods, reference data, proof of concept implementations, and technical analyses to advance thedevelopment and productive use of information technology. ITL’s responsibilities include thedevelopment of management, administrative, technical, and physical standards and guidelines forthe cost-effective security and privacy of other than national security-related information in federal
information systems.

83

Abstract

84
85
86
87
88
89
90
91

This NIST Interagency/Internal Report (NISTIR) is intended as a step toward securing
applications of Artificial Intelligence (AI), especially against adversarial manipulations of
Machine Learning (ML), by developing a taxonomy and terminology of Adversarial Machine
Learning (AML). Although AI also includes various knowledge-based systems, the data-driven
approach of ML introduces additional security challenges in training and testing (inference)phases of system operations. AML is concerned with the design of ML algorithms that can resist
security challenges, the study of the capabilities of attackers, and the understanding of attack
consequences.

92
93
94
95
96
97
98
99

This document develops a taxonomy of concepts and defines terminology in the field of AML.
The taxonomy, built on and integrating previous AML survey works, is arranged in a conceptual
hierarchy that includes key types of attacks, defenses, and consequences. The terminology,
arranged in an alphabetical glossary, defines key terms associated with the security of ML
components of an AI system. Taken together, the terminology and taxonomy are intended to
inform future standards and best practices for assessing and managing the security of ML
components, by establishing a common language and understanding of the rapidly developing
AML landscape.

100

Keywords

101
102

adversarial; artificial intelligence; attack; cybersecurity; defense; evasion; information
technology; machine learning; oracle; poisoning.

ii

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

103

Acknowledgments

104
105
106
107
108
109
110

The authors wish to thank the many people who assisted with the development of this document,
including our NIST colleague Tim McBride. We would also like to thank the technical review
team from The MITRE Corporation for their support on this effort: Dan Aiello, Lashon Booker,
Ron Ferguson, Chuck Howell, Keith Manville, Joseph Mikhail, Scott Musman, Colin SheaBlymyer, Anne Townsend, and Michael Zoracki. Also, we would like to thank the technicalreview from our academic team: Edward Colbert and Laura Freeman from Virginia Tech, andTim Oates from University of Maryland, Baltimore County.

111

Audience

112
113
114
115
116
117

The main audience for this document is researchers and practitioners in the field of machinelearning (artificial intelligence). Researchers and practitioners in adversarial machine learning
will find this useful for choosing the correct and standardized terminology to be used in theirown reports. Machine learning researchers may also benefit by understanding the relationships
between adversarial attacks, defenses, and consequences, and by referencing the definitions ofstandardized terminology.

118

Trademark Information

119

All trademarks and registered trademarks belong to their respective organizations.

iii

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

120

Call for Patent Claims

121
122
123
124
125
126

This public review includes a call for information on essential patent claims (claims whose use
would be required for compliance with the guidance or requirements in this Information
Technology Laboratory (ITL) draft publication). Such guidance and/or requirements may bedirectly stated in this ITL Publication or by reference to another publication. This call also
includes disclosure, where known, of the existence of pending U.S. or foreign patent applications
relating to this ITL draft publication and of any relevant unexpired U.S. or foreign patents.

127
128

ITL may require from the patent holder, or a party authorized to make assurances on its behalf,in written or electronic form, either:

129
130

a) assurance in the form of a general disclaimer to the effect that such party does not holdand does not currently intend holding any essential patent claim(s); or

131
132
133

b) assurance that a license to such essential patent claim(s) will be made available toapplicants desiring to utilize the license for the purpose of complying with the guidanceor requirements in this ITL draft publication either:

134
135
136
137

i.
ii.

under reasonable terms and conditions that are demonstrably free of any unfair
discrimination; orwithout compensation and under reasonable terms and conditions that aredemonstrably free of any unfair discrimination.

138
139
140
141
142

Such assurance shall indicate that the patent holder (or third party authorized to make assuranceson its behalf) will include in any documents transferring ownership of patents subject to theassurance, provisions sufficient to ensure that the commitments in the assurance are binding on
the transferee, and that the transferee will similarly include appropriate provisions in the event of
future transfers with the goal of binding each successor-in-interest.

143
144

The assurance shall also indicate that it is intended to be binding on successors-in-interest
regardless of whether such provisions are included in the relevant transfer documents.

145

Such statements should be addressed to: ai-nccoe@nist.gov

iv

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

Table of Contents

146
147

1

Introduction ............................................................................................................ 1

148

2

Taxonomy ............................................................................................................... 2149

2.1 Attacks ............................................................................................................ 6150

2.1.1 Targets ................................................................................................. 6151

2.1.2 Techniques ........................................................................................... 6

152

2.1.3 Knowledge............................................................................................ 8153

2.2 Defenses ......................................................................................................... 8154

2.3 Consequences .............................................................................................. 10

155

3

Terminology.......................................................................................................... 11

156

References ................................................................................................................... 27

157
158

List of Figures

159
160

Figure 1. An illustration of example Attacks and Defenses in the Machine Learning
Pipeline. ................................................................................................................... 3

161
162

Figure 2. Taxonomy of Attacks, Defenses, and Consequences in Adversarial Machine
Learning ................................................................................................................... 4163

Figure 3. Example of adversarial perturbation used to evade classifiers [14].................. 8

164
165

Figure 4. An example of Feature Squeezing, which smooths inputs to remove
adversarial inputs [16]. ............................................................................................. 9

166
167

List of Tables

168

Table 1. Terminology..................................................................................................... 11v

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

169

1Introduction

170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188

This NIST Interagency/Internal Report (NISTIR) is intended as a step toward securing
applications of Artificial Intelligence (AI), especially against adversarial manipulations of
Machine Learning (ML), by developing a taxonomy and terminology of Adversarial Machine
Learning (AML). AI refers to computer systems able to perform tasks that normally require
human intelligence, such as image classification and speech recognition. ML refers to thecomponents of AI systems that learn from data to perform such tasks. The ML components of an
AI system include the data, model, and processes for training, testing, and validation. Although
AI also includes various knowledge-based approaches, the data-driven approach of ML
introduces additional security challenges in training and testing (inference) phases of ML
operations. These security challenges include the potential for adversarial manipulation oftraining data, and adversarial exploitation of model sensitivities to adversely affect theperformance of ML classification and regression. AML is concerned with the design of ML
algorithms that can resist security challenges, the study of the capabilities of attackers, and theunderstanding of attack consequences [1]. Attacks are launched by adversaries with malevolent
intentions, and security of ML refers to defenses intended to prevent or mitigate theconsequences of such attacks. Although ML components may also be adversely affected byvarious unintentional factors, such as design flaws or data biases, these factors are not intentional
adversarial attacks, and they are not within the scope of security addressed by the literature on
AML.

189
190
191
192
193
194
195
196

This document presents a taxonomy of concepts and defines terminology in the field of AML.
The taxonomy, built on and integrating previous AML survey works, is arranged in a conceptual
hierarchy that includes key types of attacks, defenses, and consequences. The terminology,
arranged in an alphabetical glossary, defines key terms associated with the security of the ML
components of an AI system. Taken together, the terminology and taxonomy are intended to
inform future standards and best practices for assessing and managing the security of ML
components, by establishing a common language and understanding of the rapidly developing
AML landscape.

197
198
199
200
201
202
203
204
205

The literature on AML uses various terms to characterize security and assurance, including
robustness and resilience. In cybersecurity more generally (NIST Glossary of Key Information
Security Terms, NISTIR 7298, Revision 2), robustness refers to reliable operation of a system
across a range of conditions (including attacks), and resilience refers to adaptable operations andrecovery from disruptions (including attacks). Also, in cybersecurity more generally (NIST
Glossary of Key Information Security Terms, NISTIR 7298, Revision 2), both robustness andresilience are gauged by risk, which is a measure of the extent to which an entity (e.g., system) isthreatened by a potential circumstance or event (e.g., attack). Therefore, this general notion ofrisk offers a useful approach for assessing and managing the security of ML components.

206

As introduced in the NIST Guide for Conducting Risk Assessments (NIST 800-30, Revision 1):

207
208
209

Risk assessment is one of the fundamental components of an organizational risk
management process... The purpose of risk assessments is to inform decision makers andsupport risk responses by identifying: (i) relevant threats to organizations or threats
1

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

210
211
212
213

directed through organizations against other organizations; (ii) vulnerabilities bothinternal and external to organizations; (iii) impact (i.e., harm) to organizations that may
occur given the potential for threats exploiting vulnerabilities; and (iv) likelihood that
harm will occur.

214
215
216
217
218
219
220

On that basis, a risk-based approach would begin by identifying relevant threats, vulnerabilities,
and impacts. In the case of AML, threats are defined by the types of attacks and adversarial
contexts in which attacks may occur; vulnerabilities are defined by the types of defenses, or lackthereof, for preventing or mitigating attacks; and impacts are defined by the consequences that
result from attacks and associated defenses against those attacks. Therefore, the taxonomy of
AML here is aligned with these three dimensions of AML risk assessment, namely: attacks,
defenses, and consequences.

221
222
223
224
225
226
227
228

The taxonomy is presented below, by discussing key concepts in each dimension based onreviews of other taxonomies and surveys of the AML literature. In the discussion, concepts
appearing in the taxonomy are written in title case italics. Because of rapid growth of concepts
and methods in this field, the intent is not to be exhaustive but rather to aid readers inunderstanding relevant concepts pertaining to AML attacks, defenses, and consequences. Also,
while the taxonomy identifies attacks, defenses, and consequences from a risk-based perspective,
no attempt is made here to quantify the likelihoods and consequences that may arise from AML
attacks and defenses.

229
230
231
232
233
234

The taxonomy is followed by a glossary of terminology, including a stand-alone definition foreach individual term. This terminology was also extracted from existing literature and is intended
to complement the taxonomy by defining additional descriptive terms that do not appear
explicitly as headings in the taxonomy. Like the taxonomy, the terminology and definitions areintended not to be exhaustive but rather to aid in understanding key concepts as discussed invarious other authors’ reviews of the AML literature.

235

2

236
237
238
239
240
241
242
243
244

The taxonomy is based on recently published papers that survey the AML literature and offer
taxonomies of attacks and defenses. More than a dozen such papers, identified via keyword
searches, were reviewed with the aim of identifying those themes and terms that appeared to be
most prevalent among authors. Special attention was paid to papers that provided lucid
explanations and recent compilations reflecting common if not consensus views across a numberof authors. The primary sources used here include: Akhtar (2018) [2], Biggio (2018) [3],Chakraborty (2018) [4], Liu (2018) [5], and Papernot (2018) [6]. Additional sources used here
include: Kuznetsov (2019) [7], Goodfellow (2018) [8], Yuan (2019) [9], Papernot (2017) [10],Papernot (2016) [11], Huang (2011) [1], Barreno (2010) [12], and Barreno (2006) [13].

245
246
247
248
249

The primary sources noted above treat topics in AML from different perspectives, with varying
degrees of breadth and depth. For example, Akhtar [2], concerned with computer visionapplications, addresses attacks and defenses in that domain with greater depth than the otherauthors noted above. Biggio [3] offers more of a historical perspective, tracing the evolution of
AML with a broader focus on computer vision and cybersecurity tasks. Charkraborty [4], Liu

Taxonomy

2

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

250
251
252

[5], and Papernot (2018) [6] are all concerned with cataloging attacks and defenses with an evenbroader focus independent of the specific area of application. Much overlap exists in these
papers, with authors often citing the same sources for the topics and terms they discuss.

253
254
255
256
257
258
259
260
261

This NISTIR is intended to capture common aspects of these previous papers surveying the fieldof AML, in an integrated taxonomy adopting a risk-based perspective (see NIST Guide for
Conducting Risk Assessments, NIST 800-30, Revision 1) that applies across areas of application.
The highest levels of the resulting taxonomy include various aspects of Attacks and Defenses, as
illustrated by Figure 1 in the context of Training and Testing (Inference) phases of the machinelearning pipeline. Figure 2 organizes these and lower levels of the taxonomy in a hierarchicalfashion along the three dimensions of Attacks, Defenses, and Consequences. The third
dimension, Consequences, does not appear in the other taxonomies noted above and instead has
been addressed by other authors as an aspect of Attacks dealing with the adversary’s intent.

262
263
264
265
266
267
268
269
270

A contribution here is to address Consequences as a separate dimension of risk, because
Consequences will depend on Defenses as well as Attacks, and because the actual or potential
Consequences of Attacks and Defenses may or may not be consistent with the adversary’s intent.
As noted earlier, while we identify aspects of Consequences as well as Attacks and Defenses, we
do not attempt to quantify these individual dimensions of risk or overall risk. Indeed, we expect
risk will depend highly on the specific application context in which an ML component isdeployed. Nevertheless, our intent is to introduce a taxonomy (and associated terminology) of
AML in a manner that may support future efforts to assess and manage operational risks inpractical applications of ML.

271

Details of each dimension in the taxonomy are summarized in sections below.

Data Access
Poisoning

Attacks

Training
Data Encryption
Data Sanitization
Robust Statistics

EvasionOracleTesting
Defenses

Robustness Improvements
Differential Privacy
Homomorphic Encryption

272
273

Figure 1. An illustration of example Attacks and Defenses in the Machine Learning Pipeline.

3

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

274

275
276

Figure 2. Taxonomy of Attacks, Defenses, and Consequences in Adversarial Machine Learning

4

NISTIR 8269 (DRAFT)

277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

1. Attacks
a. Targets
i. Physical Domain (of input sensors or output actions)ii. Digital Representationiii. Machine Learning Model
1. Supervised Learning
2. Unsupervised Learning
3. Reinforcement Learningb. Techniques
i. Training
1. Data Access
2. Poisoninga. Indirect Poisoning
b. Direct Poisoning
i. Data Injection
ii. Data Manipulation
1. Label Manipulation
2. Input Manipulationiii. Logic Corruption
ii. Testing (Inference)
1. Evasiona. Gradient-basedi. Single Step
ii. Iterativeb. Gradient-freec. Oracle
i. Extraction
ii. Inversion
iii. Membership Inferencec. Knowledge
i. Black Box
1. Samples
2. Oracle
ii. Gray Box
1. Model Architecture
2. Parameters Values
3. Training Method (Loss Function)
4. Training Dataiii. White Box
2. Defenses
a. Defenses Against Training Attacks
i. Data Encryption
ii. Data Sanitization (Reject on Negative Impact)iii. Robust Statistics
b. Defenses Against Testing (Inference) Attacks
5

NISTIR 8269 (DRAFT)

322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

i. Robustness Improvements
1. Adversarial Training
2. Gradient Masking
3. Defensive Distillation
4. Ensemble Method
5. Feature Squeezing
6. Reformers/Autoencoders
ii. Differential Privacy
iii. Homomorphic Encryption
3. Consequencesa. Integrity Violationi. Confidence Reduction
ii. Misclassificationiii. Targeted Misclassification
iv. Source-Target Misclassification
b. Availability Violationc. Confidentiality Violationi. Privacy Violation

2.1

Attacks

ML components may be Targets of Attacks by adversaries using various Techniques andKnowledge about the systems.

345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363

2.1.1

364
365

2.1.2

Targets

The Targets of Attacks are defined by stages in the ML pipeline, including the Physical Domain
of input sensors, the Digital Representation for pre-processing, the Machine Learning Model
itself, or the Physical Domain of output actions. The types of methods generating a Machine
Learning Model include Supervised Learning, Unsupervised Learning, and Reinforcement
Learning. In Supervised Learning, training data are provided in the form of inputs labeled withcorresponding outputs, and the model learns a mapping between inputs and outputs. The learning
task is referred to as classification when the outputs take on categorical values, and regression
when the outputs take on numerical values. In Unsupervised Learning, training data are
unlabeled inputs, and the model learns an underlying structure of the data. For example, themodel may perform clustering of inputs according to some similarity metric, or dimensionality
reduction to project data into lower dimensional subspaces. In Reinforcement Learning, areward-based policy for acting in an environment is learned from training data represented as
sequences of actions, observations, and rewards. In some applications, Reinforcement Learning
may be combined with Supervised Learning and Unsupervised Learning. Although all three
types of systems may be Targets of Attacks, most research in AML has focused on Supervised
Learning systems, typically as applied to image classification tasks. However, algorithms
developed to craft adversarial examples for classification are equally applicable to reinforcement
learning [6].
Techniques

Adversarial Techniques used for launching Attacks against Targets may apply to the Training or
6

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382

Testing (Inference) phases of system operation. Attacks in the Training phase attempt to acquire
or influence the training data or model itself. In Data Access Attacks, some or all of the training
data is accessed and can be used to create a substitute model. This substitute model can then be
used to test the effectiveness of potential inputs before submitting them as Attacks in the Testing(Inference) phase of operation. In Poisoning, also known as Causative Attacks, the data or model
are altered indirectly or directly. In Indirect Poisoning, adversaries without access to preprocessed data used by the target model must instead poison the data before pre-processing. In
direct poisoning, the data are altered by Data Injection or Data Manipulation, or the model isaltered directly by Logic Corruption. In Data Injection, adversarial inputs are inserted into theoriginal training data, thereby changing the underlying data distribution without changing thefeatures or labels of the original training data. Injected adversarial samples can be optimized bylinear programming methods that shift the decision boundary of a centroid model (inUnsupervised Learning), or by gradient ascent on the test error of the model to degradeclassification accuracy (in Supervised Learning). Data Manipulation involves adversarial
modification of output labels (Label Manipulation) and input data (Input Manipulation) of theoriginal training data. Logic Corruption is accomplished by an adversary who can tamper with
the ML algorithm and thereby alter the learning process and model itself.

383
384
385
386

Attacks in the Testing (Inference) phase, also known as Exploratory Attacks, do not tamper withthe target model or the data used in training. Instead these Attacks generate adversarial examples
as inputs that are able to evade proper output classification by the model, in Evasion Attacks, orcollect and infer information about the model or training data, in Oracle Attacks.

387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403

In Evasion Attacks, the adversary solves a constrained optimization problem to find a small input
perturbation that causes a large change in the loss function and results in output
misclassification. This typically involves Gradient-based search algorithms such as Limitedmemory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS), Fast Gradient Sign Method (FGSM),
or Jacobian-based Saliency Map Attack (JSMA). L-BFGS was the first algorithm used togenerate misclassifications by a computer vision system model using input perturbations thatwere imperceptible to human observers. FGSM improves the computational efficiency ofgradient ascent, in a Single Step approach that eliminates iterations required to obtain aperturbation that will cause a large change in the loss function. Compared to FGSM, JSMA is an
Iterative Algorithm that provides more fine-grained control of perturbed features and thereby cangenerate more convincing adversarial examples, albeit at increased computational cost. Theseand other algorithms for Evasion Attacks require knowledge of the model, or a substitute model,
in order to compute gradients in the loss function across input-output pairings. Besides L-BFGS,
FGSM, and JSMA, many other techniques with similar operating principles have been developed
to generate adversarial examples [2] [4] [9], one of which is depicted in Figure 3 [14]. Gradientfree attacks [15] have also been developed, but they typically require access to model confidence
values in order to be effective.

7

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

404
405

Figure 3. Example of adversarial perturbation used to evade classifiers [14].

406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421

In Oracle Attacks, an adversary uses an Application Programming Interface to present the modelwith inputs and to observe the model’s outputs. Even when the adversary has no direct
knowledge of the model itself, the input-output pairings obtained from Oracle Attacks can beused to train a substitute model that operates much like the target model, due to thetransferability property exhibited by many model architectures. This substitute model, in turn,can then be used to generate adversarial examples for use in Evasion Attacks against the target
model. Oracle Attacks include Extraction Attacks, Inversion Attacks, and Membership Inference
Attacks. These attacks collect information such as output and confidence values, to inferparameters or characteristics of the model or data. In Extraction Attacks, an adversary extracts
the parameters or structure of the model from observations of the model’s predictions, typically
including probabilities returned for each class. In the case of Inversion Attacks, the inferredcharacteristics may allow the adversary to reconstruct data used to train the model, including
personal information that violates the privacy of an individual. In a Membership Inference
Attack, the adversary uses returns from queries of the target model to determine whether specificdata points belong to the same distribution as the training dataset, by exploiting differences in themodel’s confidence on points that were or were not seen during training.

422
423
424
425
426
427
428
429
430
431
432
433
434

2.1.3

Knowledge

Besides Techniques used to launch Attacks against Targets, threats to ML components also
depend on the adversary’s Knowledge about the target model. In Black Box Attacks, theadversary has no knowledge about the model except input-output Samples of training data or
input-output pairings obtained using the target model as an Oracle. In Gray Box Attacks, theadversary has partial information about the model, which may include the Model Architecture,
Parameter Values, Training Method (Loss Function), or Training Data. In White Box Attacks,
the adversary has complete knowledge of the model including architecture, parameters, methods,
and data. Even when an adversary does not have the complete knowledge needed for a WhiteBox Attack, Data Access or Oracle Attacks that produce input-output pairings can be used totrain a substitute model, which operates much like the actual model due to the transferability
property exhibited by many model architectures. This substitute model can then be used as aWhite Box to generate adversarial examples for use in Evasion Attacks.

435
436
437
438

2.2

439

Defenses Against Training Attacks involving Data Access include traditional access control
8

Defenses

Defenses can be characterized by whether they apply to Attacks launched against the Training or
Testing (Inference) phases of system operation. In both cases, defensive methods often can incur
performance overhead as well as have a detrimental effect on model accuracy [4].

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

440
441
442
443
444
445
446

measures such as Data Encryption. Defenses against Poisoning Attacks include Data Sanitization
and Robust Statistics. In Data Sanitization, adversarial examples are identified by testing theimpacts of examples on classification performance. Examples that cause high error rates inclassification are then removed from the training set, in an approach known as Reject on
Negative Impact. Rather than attempting to detect poisoned data, Robust Statistics use constraints
and regularization techniques to reduce potential distortions of the learning model caused bypoisoned data.

447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462

Defenses Against Testing (Inference) Attacks include various model Robustness Improvements,
including Adversarial Training, Gradient Masking, Defensive Distillation, Ensemble Methods,
Feature Squeezing, and Reformers/Autoencoders. Although used as Defenses against Attacks
made in the Testing (Inference) phase, these Defenses are deployed by the defender in theTraining phase that precedes Testing (Inference). In Adversarial Training, inputs containing
adversarial perturbations but with correct output labels are injected into the training data in orderto minimize classification errors caused by adversarial examples. Gradient Masking reduces themodel’s sensitivity to small perturbations in inputs by computing first order derivatives of themodel with respect to its inputs and minimizing these derivatives during the learning phase. A
similar idea motivates Defensive Distillation, where a target model is used to train a smaller
model that exhibits a smoother output surface, and Ensemble Methods, where multiple classifiers
are trained together and combined to improve robustness. Similarly, Feature Squeezing, shownin Figure 4, uses smoothing transformations of input features in an attempt to undo adversarial
perturbations [16]. Reformers take a given input and push it toward the closest example in the
training set, typically using neural networks called Autoencoders, to counter adversarial
perturbations.

463
464

Figure 4. An example of Feature Squeezing, which smooths inputs to remove adversarial inputs [16].

465
466
467
468
469
470
471

It is important to acknowledge that the adversary may defeat various Robustness Improvement
Defenses by launching Data Access or Oracle Attacks to obtain input-output pairings. Thesepairings can be subsequently used to train a substitute model that does not mask gradients orsmooth outputs like the target model. The substitute model can then be used as a White Box tocraft adversarial examples, by exploiting the transferability property of ML-trained models, so it
can be difficult to defend against Evasion Attacks by an adversary capable of creating asubstitute model.

472
473
474

Besides the Robustness Improvements noted above, Defenses Against Testing (Inference) Attacks
also include randomization mechanisms applied to training data or model outputs to provide
Differential Privacy guarantees. Differential Privacy formulates privacy as a property satisfied
9

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

475
476
477
478
479
480
481
482

by a randomization mechanism on pairs of adjacent datasets. Ultimately, the Differential Privacy
property ensures that model outputs do not reveal any additional information about an individual
record included in the training data. However, there is an inherent performance tradeoff because
a model’s prediction accuracy is degraded by the randomization mechanisms used to achieve
Differential Privacy. An alternative approach is Homomorphic Encryption, which encrypts data
in a form that a neural network can process without decrypting the data. This protects the privacyof each individual input but introduces computational performance overhead and limits the set ofarithmetic operations to those supported by Homomorphic Encryption.

483
484
485
486
487
488

2.3

489
490
491
492
493
494
495

In Integrity Violations, the inference process is undermined, resulting in Confidence Reduction or
Misclassification to any class different from the original class. More specific misclassifications
include Targeted Misclassification of inputs to a specific target output class and Source-Target
Misclassification of a specific input to a specific target output class. In Unsupervised Learning,an Integrity Violation may produce a meaningless representation of the input in an unsupervised
feature extractor. In Reinforcement Learning, an Integrity Violation may cause the learning agent
to act unintelligently or with degraded performance in its environment.

496
497
498
499
500

Availability Violations induce reductions in quality (such as inference speed) or access (denial ofservice) to the point of rendering the ML component unavailable to users. Although Availability
Violations may involve Confidence Reductions or Misclassifications similar to those of Integrity
Violations, the difference is that Availability Violations result in behaviors such as unacceptablespeed or denial of access that render a model’s output or action unusable.

501
502
503
504
505
506
507
508

Confidentiality Violations occur when an adversary extracts or infers usable information about
the model and data. Attacks on confidential information about the model include an Extraction
Attack that reveals model architecture or parameters, or an Oracle Attack that enables theadversary to construct a substitute model. Attacks that reveal confidential information about thedata include an Inversion Attack whereby an adversary exploits the target model to recover
missing data using partially known inputs, or a Membership Inference Attack whereby anadversary performs a membership test to determine if an individual was included in the dataset
used to train the target model.

509
510
511
512

Privacy Violations are a specific class of Confidentiality Violation in which the adversary obtains
personal information about one or more individual and legitimate model inputs, either includedin the training data or not. An example would be when an adversary acquires or extracts an
individual’s medical records in violation of privacy policies.

Consequences

The Consequences of Attacks against Targets depend on implemented Defenses. For a given
combination of Attack (including Target, Technique, and Knowledge) and Defense(s), theConsequences can be characterized categorically as Violations of Integrity, Availability,
Confidentiality, or Privacy. Within each category, varying levels of severity may also be used tomeasure the violation of security.

10

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

513

3

Terminology

514
515

As a complement to the taxonomy discussed above, this section presents a glossary ofterminology with a stand-alone definition for each term.

516
517
518
519
520
521
522
523

Similar to the taxonomy, the terminology is based on recently published papers that survey the
AML literature as well as papers that address recent advances in the field. These papers werereviewed with the aim of identifying those themes and terms that appeared to be most prevalent
among authors. The primary sources used here include: Akhtar (2018) [2], Biggio (2018) [3],Chakraborty (2018) [4], Liu (2018) [5], and Papernot (2018) [6]. Additional sources used here
include: Kuznetsov (2019) [7], Goodfellow (2018) [8], Yuan (2019) [9], Papernot (2017) [10],Papernot (2016) [11], Huang (2011) [1], Barreno (2010) [12], and Barreno (2006) [13].
Terminology definitions were constructed from the identified themes and terms.

524
525
526
527
528
529
530
531
532
533

The field of AI Security is currently heavily centered around AML, and much of the terminology
draws from the fields of ML. The goal and contribution of this NISTIR terminology is toaggregate those terms that are in common usage in AML and use the sources to compile
common, standardized definitions. The guidelines for selecting terms for inclusion here are thatthe terms are not general ML (e.g., deep learning) terms that are likely already defined in that
more general fields. Also excluded are terms that are specifically named and publishedalgorithms. In case of varying definitions, definitions were prioritized based on recency,
generality, and most common usage in source surveys. The references provided indicate one ormore possible sources of relevant information or the stated definition. They are not intended toindicate specific endorsement or to assign originator credit.

534
535

Table 1. Terminology. This table lists terms, synonyms for these terms, definitions, and references for thesedefinitions.

ID

Term

1

Synonym

Assigned Definition

Reference

Adversarial
capabilities

The various actions, information,techniques or attack vectors
available to an attacker on athreat surface.

[6]

2

Activation
maximization

The synthetization of inputs that
activate specific neurons in aneural network to produce
synthetic inputs that are humaninterpretable.

[6]3

Adversarial
exampletransferability

The property that adversarial
examples crafted to be
misclassified by a model are
likely to be misclassified by adifferent model.

[6]

11

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

4Adversarial
exampleML input sample formed by[3], [2]applying a small but intentionallyworst-case perturbation (seeadversarial perturbation) to aclean example, such that the
perturbed input causes a learned
model to output an incorrect
answer.

5

Adversarial
perturbation

The noise added to an input
sample to make it an adversarial
example.

[2]

6

Adversarial
training

Defensive method to increase
model robustness by injecting
adversarial examples into thetraining set.

[4]

7

Adversary

The agent who conducts or
intends to conduct detrimental
activities, perhaps by creating anadversarial example.

[2], [17]8AttackAction targeting a learning
system to cause malfunction.

[13]

9

Attack detection

The action of differentiating
between anomalous and normal
behavior, or between an
adversarial example and a benign
example.

[6]

10

Attack detector

A mechanism to (only) detect if a [2]sample is an adversarial.

11

Autoencoder
attackA perturbation attack on
autoencoders that leads theautoencoder to reconstruct acompletely different image.

[2]

12

Auxiliary model

An attacker's model trained toapproximate the decisionboundary of the target model.
Useful for testing attacks offline.

[2], [4], [6]Substitute or
Surrogate
model

12

NISTIR 8269 (DRAFT)

13

Availability
violation

14Black-box attack

15

Causative attack

16

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

A compromise of the normal
system functionalities available
to legitimate users, such as
accuracy, quality, or access,resulting in inaccessible orunusable model output.

[3], [6]Zeroknowledge
attackAttack that assumes noknowledge about the model
under attack. The adversary may
use context or historicalinformation to infer model
vulnerability. The attacker may
probe the system to informsystem vulnerabilities.

[2], [6], [4]Poisoning
attackSee “Poisoning Attack.”[13]

Confidence
reduction

Reducing the confidence ofprediction for the target model.
For example, a legitimate imageof a ‘stop’ sign can be predictedwith a lower confidence having a
lesser probability of classmembership.

[4]

17

Confidentiality
attackAn attack in which theadversarial goal is to reveal
evidence of a model'scharacteristics or information
about its training data.

[6], [3]18

Data sanitization

Defensive method that identifies
and treats manipulated samples
as outliers in the training data, tobe detected and removed.

[18], [3]19

Datasetmodification

Altering the training data
directly, in contrast to injection.

[6]

20

Deep Contractive
Network

An ML technique in which, for[2], [6]defensive purposes, a smoothness
penalty is applied to reduce
susceptibility to adversarial
examples. It penalizes output
13

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

variation with respect to input
variation to increase the variationneeded to produce adversarial
examples.
21

Defensive
distillation

Distillation

A procedure to train deep neural [19], [4]network (DNN)-based classifiermodels that are more robust to
perturbations. Distillation
extracts additional knowledge
about training points asprobability vectors produced by a
DNN, which is fed back into thetraining regimen. Distillation
generates smoother classifiermodels by reducing their
sensitivity to input perturbations.
These smoother DNN classifiers
are found to be more resilient toadversarial samples and have
improved class generalizability
properties. A type of gradientmasking.

22

Dense evasionattack

L2-norm
attackEvasion (L2-norm) attack where
the cost of modifying features is
proportional to the distance
between the original andmodified sample in Euclidean
space. The attacker will prefer tomake small changes to many or
all features.

[20]

23

Differential
privacy

A mathematical formulation thatdefines the privacy provided by
an ML model as the property thata learning algorithm's output will
not differ statistically by thechange of a single training
example. This formulation isleveraged by multiple defenses
that aim to protect data privacy.

[6], [5]14NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

24

Disinformation
technique

Altering data seen by theadversary with the goal ofconfusing the adversary’s
estimate of the learner’s state.

[13]

25

Distinguishability
measureA measurement of classifier
robustness that describes thedifference between classes of adataset. Distinguishability is the
distance between the means of
two classes for linear classifiers
and the distance between thematrices of second ordermoments for non-linear
classifiers.

[2]

26

Distribution drift

A situation in which the training
and test input distributions differ.

[6]

27

Enchanting attack

An attack on deep reinforcement
learning in which the adversarylures the attacked system to a
designated target state byintegrating a generative modeland a planning algorithm. The
generative model is used forpredicting the future states of theagent, whereas the planningalgorithm generates the actions
for luring it.

[2]

28

Ensemble learning
or method

A classification method using
multiple classifiers to enhancerobustness including against
evasion attacks.

[3], [21]29

Error specificity

Describes the misclassification
goal of an attacker: if the attacker
aims to have a sample
misclassified as a specific class,specificity is specific (targeted
attack); if the attacker aims forany misclassification, specificity
is generic (non-targeted attack).

[3]

15

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

30Error-genericevasion attack

The attacker is interested incausing a misclassification of atest sample, regardless of theoutput class predicted by theclassifier.

[3]

31

Error-genericpoisoning attack

The attacker, using training set
poisoning, aims to cause a denialof service, by inducing as many
misclassifications as possible(regardless of the classes inwhich they occur).

[3]

32

Error-specificevasion attack

The attacker aims to misleadclassification of a test sample,
such that the adversarial samples
are misclassified as a specificclass.

[3]

33

Error-specificpoisoning attack

The attacker, using training set
poisoning, aims to cause specifictypes of misclassifications.

[3]

34

Evasion attack

The attacker manipulates input
samples to evade (cause amisclassification) a trainedclassifier at test time.

[3]

35

Explainability

The ability to provide a humaninterpretable explanation for an
ML prediction and produce
insights about the causes ofdecisions, potentially to line up
with human reasoning.

[22]

36

Exploratory
attackThe attacker manipulates only
test data. Aims to cause
misclassification with respect toadversarial samples (evasion) or
to uncover sensitive informationfrom training data and learningmodels (oracle).

[3], [5]37Fast GradientSign Method

An efficient method forcomputing an adversarial image
perturbation, using the gradient

[2]

16

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING(FGSM)of the cost function. The image isperturbed to increase the loss ofthe classifier on the resulting
image.

38

Fast-flipping
attributetechnique (attack)An attack on facial recognition
which imperceptibly modifies asingle attribute to cause the face
to be wrongly classified.
Adversarial images are generated
by flipping the binary decision ofa deep neural network.

[2], [23]39

Foveation Based
DefenseAn ML technique in which
neural networks are applied tosegments of images to improverobustness to adversarial patterns
in the images.

[2]

40

Generative
adversarial
network

An ML technique which
increases the effectiveness of amodel generator by training it in
the presence of an adversary—adiscriminator which seeks todifferentiate between real dataand generated data. The
effectiveness of the generator ismeasured by the error rate of thediscriminator. Used in thegeneration of training data in an
autoencoder attack or as adefense to train a more robust
classifier.

[2]

41

Generative model

An ML model trained with thegoal of generating new data
points. The model takes a
training set, consisting ofsamples drawn from a
distribution, and learns torepresent an estimate of that
distribution. As an attack, thegenerative model is trained togenerate candidate adversarial
samples.

[2], [8], [5]17NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

42

Generic specificity

Describes the goal of an attack as
misclassifying a sample as any of
the classes different from its trueclass.

[3]

43

Gradient ascent

An iterative algorithm used tofind a minimum of a function.
Identifies the optimal adversarial
inputs corresponding to local
maxima in the test error of themodel. Operates by calculating
the gradient of objectivefunctions that measureeffectiveness.

[6], [4], [5]44Gradient masking

An ML technique in which
gradients are minimized toreduce the model's sensitivity toadversarial examples. Hides thegradient direction used to craft
adversarial examples.

[6], [3]45Gray-box attack(grey-box attack)Attack which assumes partial
knowledge about the model
under attack (e.g., type offeatures, or type of training data).

[2], [3]46

Homomorphic
encryption

A technique in which encrypted
data can be processed by a neuralnetwork without decryption,
allowing for data protection andimproving data privacy whenprocessed by an ML algorithm.

[6], [5]47

Image
perturbation

A change or transformation to an
image, often to cause amisclassification.

[2]

48

Indiscriminate
attackAn attack that aims to cause
misclassification of any sample
to target any system user orprotected service.

[3], [5]49

Inference

The stage of ML in which amodel is applied to a task. For
example, a classifier model[6]

Limitedknowledge
attack18

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

produces the classification of atest sample.
50

Injection (datainjection) attack

The insertion of adversarial
inputs into the existing training
data.

[6]

51

Input
manipulation
attackA threat model that assumes theadversary can corrupt the input
features of training samples ortraining sample labels.

[6]

52

Integrity violation

To induce a particular output or
behavior of the adversary's
choosing. Compare against
Confidentiality and Availability
violations.

[6]

53

Jacobian-based
Saliency Map
Attack (JSMA)An attack that makes optimal
miniscule changes to input data
until the classifier is fooled or a
maximum number of changes ismet.

[2], [6]54

L2-norm attack

55

Label
manipulation
attackAn attack in which the adversary
corrupts the labels of trainingdata.

[6], [4]56Label smoothing
defenseA defense mechanism in whichlabels are changed from classes
to real numbers, allowing forclassification outside of the strict
class labels.

[6], [4]57

Limitedknowledge attack

58

Linearity
hypothesis

Dense evasionattackGray-box, or
semi-blackbox

See “Dense evasion attack.”

See “Gray-box attack.”

The hypothesis that designs of
DNNs that intentionallyencourage linear behavior forcomputation efficiency, makethem susceptible to cheaper
19[2]

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

adversarial perturbations.

59

Logic corruption
attackAn attack on an ML model inwhich the learning algorithm orlogic itself is tampered with.

[6], [4]60Membership
attackAn attack that targets theinformation of whether or not agiven data point was part of the
training dataset or part of thesame distribution as the training
dataset.

[6], [4]61Misclassification
attackAttack to alter the output[4]classification of an input example
to any class different from itstrue class. For example, a
legitimate image of a ‘stop’ sign
will be predicted as any otherclass different from the class ofstop sign.

62

Model extraction
attackAn exploratory attack that aims
to discover the structure orparameters of the model byobserving its predictions.

[6], [4]63Model inversion
attackAn oracle attack that aims todiscover training data and othersensitive data through knowledgeof the model and auxiliary data.

[6]

64

Non-targeted
attackUntargeted
attackAn attack that causes any[2], [6]misclassification as opposed tocausing classification into a
specific (incorrect) class. The
predicted label of the adversarial
example is irrelevant, as long as
it is not the correct label. See also
“Error specificity.”

20

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

65

Obfuscation
attackAn attack against a targeted[5]cluster of samples that attempts
to generate a blend of adversarial
samples and normal ones fromother clusters without altering theclustering results of these normal
samples, resulting in a set ofstealthy adversarial samples.

66

Obfuscation
defenseA defense mechanism in which
details of the model or training
data are kept secret.

[6]

67

One Pixel Attack

An (evasion) attack that alters asingle pixel in an image to causea misclassification.

[2]

68

One-shot/one-stepmethodGenerates an adversarial[2]
perturbation by performing asingle step computation, e.g.
computing gradient of model loss
once. The opposite are iterative
methods that perform the same
computation multiple times to geta single perturbation. The latterare often computationallyexpensive.

69

Oracle attack

An attack in which an adversary
is able to craft inputs and receive
outputs to the attacked model, in
an attempt to learn information
about the model and craft betterattacks.

[6]70

Outputrandomization

A defense randomizing theclassifier’s output to give
imperfect feedback to theattacker.

[3]

71

OutsidersExternal users or adversaries that
may be able to influence asystem, not including enterpriseusers (consumers).

[6]

21

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

72

Perceptual
distance73

Perfect-knowledge
attack74

Poisoning attack

Aims to increase the number of[4], [5]misclassified samples at test timeby injecting a small fraction ofcarefully designed adversarial
samples into the training data.
Indirect poisoning manipulates
data before any preprocessing,while direct poisoning the data
are altered by Data Injection or
Data Manipulation, or the modelis altered directly by Logic
Corruption. Also known as acontamination of the training
data. Alternately, also includes
tampering with the ML algorithm
itself, to compromise the wholelearning process.

75

Privacypreserving model

A model that does not reveal
personal details that may be
included in its training data.

[6]

76

Privacy violation

Revealing personal information
about an individual included in
the training data.

[6]

77

Quantitative input
influence

A measurement of the influenceof certain inputs on model
output.

[6]

78

Quasiimperceptible
perturbation

Perturbation that impairs images
very slightly for humanperception.

[2]

White-box
attackMeasures how similar two
images are in a way thatcoincides with human judgment.

[24]

Attack that exploits modelinternal information. It assumes
complete knowledge of thetargeted model, including its
parameter values, architecture,training method, and in somecases its training data as well.

[3]

22

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

79

Randomization
defenseA defense mechanism that adds[3], [6]random noise to the training data,
the model training cost function,
the learned parameters, or model
output to preserve privacy.

80

Reactive defenses

Defenses that aim to counter pastattacks, for example, by analysisof the target classifier, by timely
detection of novel attacks, by
frequent classifier retraining, orby verification of consistency ofclassifier decisions.

[3], [5]81Real-world
attacks

Attacks successfully executed on
existing systems.

[2]

82

Regularization

A mechanism at training to
improve generalizability of themodel. It reduces model
sensitivity or complexity, with
the intent of limiting
exploitability.

[6], [2]83Resilience“The ability to prepare for andadapt to changing conditions andwithstand and recover rapidly
from disruptions. Resilience
includes the ability to withstandand recover from deliberate
attacks, accidents, or naturally
occurring threats or incidents.”
The ability of a system to adapt
to and recover from adverse
conditions.

[25]

84

Robust learning

Learning algorithms based on
robust statistics that are
intrinsically less sensitive tooutlying training samples.

[3]

85

Robustoptimization

Formulates adversarial learning
as a mini- max problem in which
the inner problem maximizes thetraining loss by manipulating thetraining points under worst-case,

[3]

23

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

bounded perturbations, while theouter problem trains the learningalgorithm to minimize thecorresponding worst-casetraining loss.
86

Robustness

The ability of an ML
model/algorithm to maintaincorrect and reliable performance
under different conditions (e.g.,
unseen, noisy, or adversariallymanipulated data)

87

Sample rejection
defenseDefensive mechanism detecting[3]
and rejecting samples that are
sufficiently far (as measured by adistance metric) from the training
data in feature space.

88

Security
evaluation curve

Shows the extent to which theperformance of a learningalgorithm drops gracefully underattacks of increasing strength.

[3]

89

Source-target
misclassification
attackAn adversarial attempt to force
the output of classification for a
specific input to be a particulartarget class. For example, theinput image of ‘stop’ sign will be
predicted as a ‘Speed Limit’ sign
by the classification model.

[4]

90

Sparse evasionattackAttack (using L1-norm) wherecost depends on the number ofmodified features, and attacker
aims to minimize the number ofmodified features.

[20]

91

Specific error

Describes the goal of an attack as
misclassifying a sample as a
specific class.

[3]

92

Strategicallytimed attack

An attack on reinforcement
learning in which the adversary
attacks the model in a small
subset of time steps to affect the

[2]

24

[2], [25]NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

model's behavior without
detection.
93

Substitute modelor network

Surrogate or
Auxiliarymodel

See “Auxiliary model.”94

Surrogate model

Substitute or
Auxiliarymodel

See “Auxiliary model.”95

Targeted
misclassification
attackThe adversary tries to produce
inputs that force the output of theclassification model to be a
specific target class. For
example, any input image to theclassification model will be
predicted as a class of images
having a ‘Speed Limit’ sign. See“Error specificity.”

[4]

96

Threat model

Adversarial goals, knowledge,
and capabilities that a system isdesigned to defend against.

[6], [3], [4]97Training data
extraction attack

An attack in which the goal is to
discover parts or all of the
training data.

[6]

98

Transferability ofexampleThe ability of an adversarial
example to remain effective evenfor the models other than the oneused to generate it.

[2], [6]99Transparency

Understanding the working logicof the model.

[26]

100

Trust model

A description of the level of trust
assigned to various actors in asystem deployment. Actors
include data owners, systemproviders, service consumers,
and outsiders who access or
influence the system.

[6]

25

NISTIR 8269 (DRAFT)A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

101

Universal(Adversarial)
perturbation

Perturbation able to fool a given
model on ‘any’ image with highprobability. Note that,
universality refers to the property
of a perturbation being ‘imageagnostic’ as opposed to having
good transferability.

102

Untargeted attack

Non-targeted
attackSee “Non-targeted attack.”

103

White-box attack

Perfectknowledge
attackSee “Perfect knowledge attack.”[2], [6], [4]104Zero-knowledge
attackBlack-box
attackSee “Black-box attack”.

[3]

536

26

[2]

NISTIR 8269 (DRAFT)

537

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

References
[1]

[2]
[3]
[4]
[5]

[6]

[7]

[8]
[9]
[10]

[11]

[12]
[13]

[14]

[15]

L. Huang, A. D. Joseph, B. Nelson, B. I. P. Rubinstein and J. D. Tygar, "Adversarial
Machine Learning," in Proceedings of the 4th ACM Workshop on Security and Artificial
Intelligence, New York, NY, USA, 2011.
N. Akhtar and A. Mian, "Threat of adversarial attacks on deep learning in computer
vision: A survey," IEEE Access, vol. 6, pp. 14410-14430, 2018.
B. Biggio and F. Roli, "Wild patterns: Ten years after the rise of adversarial machinelearning," Pattern Recognition, vol. 84, pp. 317-331, 2018.
A. Chakraborty, M. Alam, V. Dey, A. Chattopadhyay and D. Mukhopadhyay,"Adversarial Attacks and Defences: A Survey," 28 9 2018.
Q. Liu, P. Li, W. Zhao, W. Cai, S. Yu and V. C. M. Leung, "A survey on security
threats and defensive techniques of machine learning: A data driven view," IEEE
access, vol. 6, pp. 12103-12117, 2018.
N. Papernot, P. McDaniel, A. Sinha and M. P. Wellman, "SoK: Security and privacy inmachine learning," in 2018 IEEE European Symposium on Security and Privacy(EuroS&P), 2018.
P. Kuznetsov, R. Edmunds, T. Xiao, H. Iqbal, R. Puri, N. Golmant and S. Shih,"Adversarial Machine Learning," in Artificial Intelligence Safety and Security,
Chapman and Hall/CRC, 2018, pp. 235-248.
I. Goodfellow, P. McDaniel and N. Papernot, "Making machine learning robust against
adversarial inputs," Communications of the ACM, vol. 61, pp. 56-66, 2018.
X. Yuan, P. He, Q. Zhu and X. Li, "Adversarial examples: Attacks and defenses fordeep learning," IEEE transactions on neural networks and learning systems, 2019.
N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik and A. Swami, "Practical
black-box attacks against machine learning," in Proceedings of the 2017 ACM on Asia
Conference on Computer and Communications Security, 2017.
N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik and A. Swami, "Thelimitations of deep learning in adversarial settings," in Security and Privacy (EuroS&P),
2016 IEEE European Symposium on, 2016.
M. Barreno, B. Nelson, A. D. Joseph and J. D. Tygar, "The security of machinelearning," Machine Learning, vol. 81, pp. 121-148, 2010.
M. Barreno, B. Nelson, R. Sears, A. D. Joseph and J. D. Tygar, "Can machine learning
be secure?," in Proceedings of the 2006 ACM Symposium on Information, computer andcommunications security, 2006.
A. Mądry and L. Schmidt, "A Brief Introduction to Adversarial Examples," Gradient
Science, [Online]. Available: http://gradientscience.org/intro_adversarial/. [Accessed 19
July 2019].
N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow,
A. Madry and A. Kurakin, "On Evaluating Adversarial Robustness," 18 2 2019.

27

NISTIR 8269 (DRAFT)

[16]

[17]

[18]
[19]

[20]

[21]

[22]

[23]

[24]
[25]
[26]

[27]

[28]

[29]

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

W. Xu, D. Evans and Y. Qi, "Is Robust Machine Learning Possible?," EvadeML(University of Virginia), [Online]. Available: https://evademl.org/. [Accessed 19 July2019].
G. Stoneburner, A. Y. Goguen and A. Feringa, "SP 800-30 Rev.1 Guide for Conducting
Risk Assessments," National Institute of Standards & Technology, Gaithersburg, MD,
United States, 2012.
J. Steinhardt, P. W. W. Koh and P. S. Liang, "Certified defenses for data poisoning
attacks," in Advances in neural information processing systems, 2017.
N. Papernot, P. McDaniel, X. Wu, S. Jha and A. Swami, "Distillation as a defense to
adversarial perturbations against deep neural networks," in 2016 IEEE Symposium on
Security and Privacy (SP), 2016.
A. Demontis, P. Russu, B. Biggio, G. Fumera and F. Roli, "On security and sparsity oflinear classifiers for adversarial settings," in Joint IAPR International Workshops on
Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic
Pattern Recognition (SSPR), 2016.
B. Biggio, G. Fumera and F. Roli, "Multiple classifier systems for robust classifier
design in adversarial environments," International Journal of Machine Learning andCybernetics, vol. 1, pp. 27-41, 2010.
F. Doshi-Velez and M. Kortz, "Accountability of AI Under the Law:," 21 November
2017. [Online]. Available: https://arxiv.org/pdf/1711.01134.pdf. [Accessed 17
September 2019].
A. Rozsa, M. Gunther and T. E. Boult, "Towards robust deep neural networks with
BANG," in IEEE Winter Conference on Applications of Computer Vision (WACV),2018.
H. Zhang, I. Goodfellow, D. Metaxas and A. Odena, "Self-attention generative
adversarial networks," arXiv preprint arXiv:1805.08318, 2018.
C. National Security Systems Glossary Working Group, "Committee on National
Security Systems (CNSS) Glossary," Gaithersburg, 2010.
F. K. Dosilovic, M. Brcic and N. Hlupic, "Explainable artificial intelligence: A survey,"in 41st International Convention on Information and Communication Technology,
Electronics and Microelectronics (MIPRO), Opatija, Croatia, 2018.
C. Xie, J. Wang, Z. Zhang, Y. Zhou, L. Xie and A. Yuille, "Adversarial examples forsemantic segmentation and object detection," in Proceedings of the IEEE International
Conference on Computer Vision, 2017.
H. Stern, J. Mason and M. Shepherd, "A linguistics-based attack on personalised
statistical e-mail classifiers," See http://www. cs. dal. ca/research/techreports/2004/CS2004-06. shtml, 2004.
A. S. Ross and F. Doshi-Velez, "Improving the adversarial robustness andinterpretability of deep neural networks by regularizing their input gradients," in Thirtysecond AAAI conference on artificial intelligence, 2018.

28

NISTIR 8269 (DRAFT)[30]

A TAXONOMY AND TERMINOLOGY OF
ADVERSARIAL MACHINE LEARNING

L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter and L. Kagal, "Explaining
Explanations: An Overview of Interpretability of Machine Learning," in 2018 IEEE 5thInternational Conference on Data Science and Advanced Analytics (DSAA), 2018.

29