博士論文

Automated Detection System for Adversarial Examples
on Images with Image Transformation and Filters
画像のフィルタリングと変換を用いた敵対例攻撃の自動検出システム

情報セキュリティ大学院大学
情報セキュリティ研究科
情報セキュリティ専攻
5674101 DANG DUY THANG

指導教員

松井俊浩

2020 年 3 月

Automated Detection System for Adversarial
Examples on Images with Image Transformation andFiltering

画像のフィルタリングと変換を用いた敵対例攻撃の自動検出システムbyDang Duy Thang
Submitted to the Institute of Information Security
in fulfillment of the requirements for the degree of
DOCTOR OF PHILOSOPHY
at
THE INSTITUTE OF INFORMATION SECURITY
March 2020c The Institute of Information Security, Yokohama, Japan○

Author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Institute of Information Security
March, 2020
Certified by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Toshihiro Matsui
Professor
Thesis Supervisor
Accepted by . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Atsuhiro Goto
Chairman, The Institute of Information Security

2Doctoral Dissertation
Automated Detection System for Adversarial
Examples on Images with Image Transformation andFiltering

画像のフィルタリングと変換を用いた敵対例攻撃の自動検出システムbyDang Duy Thang
Submitted to the Institute of Information Security
on March, 2020, in fulfillment of therequirements for the degree of
DOCTOR OF PHILOSOPHY

Abstract
Deep neural networks has been applied in many task with encouraging results andreached human-level performance. These models that make predictions by learningfrom the training data. In this context, accuracy prediction denotes the fraction oftest inputs that a model processes correctly the proportion of images that an object
recognition algorithm recognizes as belonging to the correct class, and the proportionof executables that a malware detector correctly designates as benign or malicious.
The estimate of a model’s accuracy varies greatly with the choice of the dataset used
to compute the estimate. The model’s accuracy is generally evaluated on test inputs
that were not used during the training process. The accuracy is usually higher if thetest inputs resemble the training images more closely. For example, a image classification system trained carefully on images may obtain high accuracy when tested onother images in the same distribution. Machine learning has traditionally been developed following the assumption that the environment is benign during both training
and evaluation of the model. Specifically, the inputs are usually assumed to all bedrawn independently from the same probability distribution at both training and test
time. This means that while test inputs are new and previously unseen during thetraining process, they at least have the same statistical properties as the inputs used
for training. Such assumptions have been useful for designing effective machine learn3

ing algorithms but implicitly rule out the possibility that an adversary could alter the
distribution at either training time or test time. However, deep neural networks havebeen recently found vulnerable to well-designed input samples that called adversarial
examples. Such an issue causes deep neural networks to misclassify adversarial examples that are imperceptible to humans. Because of these major security issues ofadversarial examples, our research is focusing on how to make the AI system more secure, able to identify and distinguish between adversarial and legitimate images. Our
research mainly investigates building detection and distinguishing between adversarial examples and legitimate images in image classification tasks. In this research, we
propose the-state-of-the-art detection systems for automatically detecting adversarial
examples on deep neural networks by using affine transformation and image filtering. Our proposed systems can perfectly distinguish adversarial samples and benign
images in an end-to-end manner without human intervention. We exploited the important role of image transformation and filters in adversarial samples and proposing
state-of-the-art methods for detecting malicious samples based on our observations.
We evaluated our methods on a variety of standard benchmark datasets including
MNIST and ImageNet. Our methods reached out to detection rates in a range from
93.2% to 100% in many settings. Besides, we proposed a new idea to evaluate therobustness of the AI system against adversarial examples. Evaluating the robustnessof AI systems is very important because of raising adversarial attacks. This idea isalso discussed in this thesis.
Thesis Supervisor: Toshihiro Matsui
Title: Professor

4

Acknowledgments
This thesis would have been impossible without the support and mentoring of myadvisor, Professor Toshihiro Matsui. Even after three years of working with him, I am
constantly surprised by his amazing intelligence, infinite energy, boundless optimism,
and genuine friendliness. I wish I could incorporate more of his qualities. Professor
Matsui highlighted fully security on machine learning as an interesting open problem
and especially adversarial examples is a subject undergoing intense study around theworld at the first our meeting three years ago. I am inspired by his explanation anddiscussion very much. Afterward, I received that I am really interested in this researcharea. He also frequently pushed and corrected my English. I have also received a
lot of fruitful comments and supports from my labmates Jun Fukuyama, Takumi
Iwaki, Hidehito Kobayashi, Masatoshi Ogawa, Taisei Kondō and Kazuyuki Ishii in
the Matsui’s laboratory. I really appreciated Professor Akira Otsuka for his helpfulcomments in every presentation. I would not have the chance to study and settled
down in Japan without enthusiasm support from Professor Atsuhiro Goto, Professor
Hidehiko Tanaka, Professor Harumichi Yuasa, and Ms. Miwako Yamaguchi. I also
would like to express my gratitude to my parents Dang Duy Quyet and Tran Thi
Kim Thanh, my wife Nguyen Thi Quynh Trang, my son Dang Duy Nhat Minh andanother my beloved family members. I can not imagine what will happen without
their help and support. This work was supported by the Iwasaki Tomomi scholarship.

5

6

Contents
1 Introduction and Background

17

1.1

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

17

1.2

Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

1.2.1

Image Classification . . . . . . . . . . . . . . . . . . . . . . . .

19

1.2.2

Linear Classification . . . . . . . . . . . . . . . . . . . . . . .

20

1.2.3

Optimization Algorithms . . . . . . . . . . . . . . . . . . . . .

21

1.2.4

Convolutional Neural Networks . . . . . . . . . . . . . . . . .

24

2 Literature Review on Adversarial Examples
2.1

2.2

29

A taxonomy of Adversarial Attacks . . . . . . . . . . . . . . . . . . .

29

2.1.1

Targeted/Non-targeted Attacks . . . . . . . . . . . . . . . . .

30

2.1.2

White-box/Black-box Attacks . . . . . . . . . . . . . . . . . .

30

2.1.3

Poisoning Attack vs Evasion Attack . . . . . . . . . . . . . . .

34

2.1.4

Physical Attacks . . . . . . . . . . . . . . . . . . . . . . . . .

35

2.1.5

Classification-based Attacks . . . . . . . . . . . . . . . . . . .

37

2.1.6

Miscellaneous Attacks on other Domain . . . . . . . . . . . . .

39

2.1.7

Summary on Adversarial Attacks . . . . . . . . . . . . . . . .

42

Defense Strategies to Adversarial Examples . . . . . . . . . . . . . . .

43

2.2.1

Adversarial Training . . . . . . . . . . . . . . . . . . . . . . .

43

2.2.2

Gradient Masking . . . . . . . . . . . . . . . . . . . . . . . . .

43

2.2.3

Image Compression . . . . . . . . . . . . . . . . . . . . . . . .

44

2.2.4

Image Denoising . . . . . . . . . . . . . . . . . . . . . . . . .

45

2.2.5

Image Transformation . . . . . . . . . . . . . . . . . . . . . .

45

7

2.2.6

Miscellaneous countermeasures. . . . . . . . . . . . . . . . .

46

2.2.7

Summary on Adversarial Defense . . . . . . . . . . . . . . . .

48

3 Proposed Automatic Detection System for Adversarial Examples 51
3.1

3.2

3.3

3.4

Image Transformation for Detecting Adversarial Examples . . . . . .

51

3.1.1

Crafting Adversarial Examples Phase . . . . . . . . . . . . . .

52

3.1.2

Coordinate Transformation . . . . . . . . . . . . . . . . . . . .

53

3.1.3

Geometric Translation and Combination . . . . . . . . . . . .

58

3.1.4

Proposed Defending Method . . . . . . . . . . . . . . . . . . .

60

3.1.5

Experimental setup . . . . . . . . . . . . . . . . . . . . . . . .

61

3.1.6

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

64

3.1.7

Summary for Image Transformation for Detecting Adversarial
Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

69

Image Filters for Detecting Adversarial Examples . . . . . . . . . . .

71

3.2.1

Detection phase . . . . . . . . . . . . . . . . . . . . . . . . . .

72

3.2.2

Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75

3.2.3

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . .

76

3.2.4

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

77

3.2.5

Summary for Denoising Detection System . . . . . . . . . . .

80

Geometric Transformation and Denoiser for identifying Adversarial Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

81

3.3.1

Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . .

85

3.3.2

Implementation and Settings . . . . . . . . . . . . . . . . . . .

85

3.3.3

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

87

3.3.4

Summary for combination of Image Transformation and Filters

96

Evaluating the robustness of adversarial perturbation against Image
Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

98

3.4.1

Search Space on Attacking Phase . . . . . . . . . . . . . . . .

98

3.4.2

Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . 100

3.4.3

Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
8

3.4.4

Summary for Evaluating the robustness of adversarial perturbation against Image Filters . . . . . . . . . . . . . . . . . . . 105

4 Conclusion and Future Works

111

4.1

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

4.2

Future Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

A Figures

115

9

10

List of Figures
1-1 Inception V3 [108] - one of deep convolutional neural networks . . . .

25

2-1 Example of the adversarial attack on mobile phone cameras: A clean
image (a) was taken and used to generate different adversarial images.
The images were printed and the TensorFlow Camera Demo app wasused to classify them. A clean image (b) is recognized correctly asa “washer” when perceived through the camera, whereas adversarial
images (c) and (d) are misclassified. The images also show network
confidence in the range [0,1] for each image. . . . . . . . . . . . . . .

36

2-2 Example of road sign attack [24]: The success rate of fooling LISACNN classifier on all the shown images is 100%. The distance andangle to the camera are also shown. The classifier is trained using
LISA dataset for road signs [78]. . . . . . . . . . . . . . . . . . . . . .

37

3-1 Coordinate Transformation with 20 degree counterclockwise rotation
around the center . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

54

3-2 Geometric Translation with 20 degree from top-lelf corner . . . . . . .

59

3-3 Combination of Coordinate Transformation and Geometric Translation.First, we shift image from top-left to bottom-right with 20 degree.
Then, we rotate image with 20 degree counterclock wise from center ofimage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

60

3-4 Our automated detection system for adversarial examples by using
image filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

72

3-5 Attack Phase Samples. . . . . . . . . . . . . . . . . . . . . . . . . . .

77

11

3-6 Adversarial ostrich image (true class: Oscilloscope) suffers to our sieve
process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3-7 Adversarial Examples Identification System. . . . . . . . . . . . . .

78
83

3-8 An example: Attack phase by using FGSM, PGD, CW_L2 and EAD
methods with targeted 𝑜𝑠𝑡𝑟𝑖𝑐ℎ label . . . . . . . . . . . . . . . . . . .

87

3-9 Our proposed Model 1 defeats the adversarial perturbation’s affect . .

89

3-10 Observation on the probabilities of the ground truth label with GT&R
method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

90

3-11 Observation on the probabilities of the targeted label with GT&R method 91
3-12 Our proposed Model 1 regain the true recognition . . . . . . . . . . .

92

3-13 Adversarial examples suffers to our first proposed model (LIMG +LIMM) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

93

3-14 Our observation on original 𝑜𝑠𝑐𝑖𝑙𝑙𝑜𝑠𝑐𝑜𝑝𝑒 image and adversarial 𝑜𝑠𝑡𝑟𝑖𝑐ℎimages that created by FGSM [31], PGD [70], CW_𝐿2 [14], and EAD [16]methods and when we use LIMG and LIMM method. First row, percentage values illustrate classification rates for the ground truth label.
Other rows, percentage values are classification rates for the targetedlabel. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

94

3-15 Classification Results on Oscilloscope Image with observation on theprobabilities of Original Label . . . . . . . . . . . . . . . . . . . . . . 102
3-16 Classification Results on Oscilloscope Image with observation on theprobabilities of Adversarial Label . . . . . . . . . . . . . . . . . . . . 102
3-17 Classification Results on Vending Machine Image with observation on
the probabilities of Original Label . . . . . . . . . . . . . . . . . . . . 103
3-18 Classification Results on Vending Machine Image with observation on
the probabilities of Adversarial Label . . . . . . . . . . . . . . . . . . 103
3-19 Classification Results on Keyboard and Computer Mouse Images . . . 104
3-20 Adversarial Oscilloscope (targeted class: Ostrich Sign). . . . . . . . 105

3-21 Original Oscilloscope with Image Filters . . . . . . . . . . . . . . . . 106
12

3-22 Adversarial FGSM L1 Oscilloscope (targeted class: Ostrich) with Image Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
3-23 Adversarial FGSM L2 Oscilloscope (targeted class: Ostrich) with Image Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
3-24 Adversarial FGSM L∞ Oscilloscope (targeted class: Ostrich) with Image Filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
A-1 Coordinate transformation - Geometric translation on the Adversarial
images (“Keyboard”, “C-Mouse”, “V-Machine”). . . . . . . . . . . . . . 115
A-2 MNIST - Coordinate Transformation with 𝛼 ∈ [0, 30]. Index numbers identify the index of images in MNIST training dataset. The figure illustrates the adversarial perturbation is defeated by Coordinate
Transformation in all experiments.

13. . . . . . . . . . . . . . . . . . . 116

14

List of Tables
3.1

Google Inception Architecture [108] . . . . . . . . . . . . . . . . . . .

63

3.2

LeNet architecture [56] . . . . . . . . . . . . . . . . . . . . . . . . . .

64

3.3

Geometric Translation results on our own images . . . . . . . . . . .

65

3.4

Coordinate Transformation results on our own images . . . . . . . . .

65

3.5

Combination results on our own images . . . . . . . . . . . . . . . . .

67

3.6

Summary performance on our own images . . . . . . . . . . . . . . .

68

3.7

Summary performance on ImageNet dataset . . . . . . . . . . . . . .

68

3.8

Summary performance on MNIST . . . . . . . . . . . . . . . . . . . .

69

3.9

Detection Rate on MNIST dataset. . . . . . . . . . . . . . . . . . .

79

3.10 Detection Rate on ImageNet dataset . . . . . . . . . . . . . . . . . .

79

3.11 Detection Rate on MNIST dataset. . . . . . . . . . . . . . . . . . .

95

3.12 Detection Rate on ImageNet dataset . . . . . . . . . . . . . . . . . .

96

3.13 Implementation Results. . . . . . . . . . . . . . . . . . . . . . . . . 107

15

16

Chapter 1
Introduction and Background
1.1

Introduction

The rapid increase in large amounts of data processing and automation has been
partly satisfied by the outstanding development of deep learning models or sometimes mentioned as deep neural networks (DNNs). Deep learning has been applied
to many challenges and achieve very impressive results in big data [121], computer
vision [115], natural language processing [8, 40], speech recognition [39], and playing games [103]. Because of its high performance, deep learning is also being widely
applied to systems requiring high safety such as self-driving cars [46], surveillancesystems based on facial recognition [41], gestures [117], and fingerprints [116]. Thesesystems are being used very widely and continue to grow strongly in the following
years. However, in recent studies, many researchers [109, 31, 10] have found the security risks associated with deep learning systems. One of the typical examples thatcan be considered is the safety of the self-driving car system based on automatic image processing features. What if the camera system on self-driving cars misidentified
the "STOP" sign to the "SPEED UP" sign. Another example, the security system
misidentifies a stranger’s face as a company employee, and the fact that strangers cangain access to a successful agency through a camera system has been deceived. This is
one of the security issues that are very challenging for systems based on deep learning.
Typical of these risks is adversarial examples. This is a concept that is considered17

to be first mentioned in the study of Szegedy et al. [109]. Adversarial examples canbe interpreted as a very elaborate form of data that makes it impossible for humans
to recognize the difference from the original data, but it causes deep learning models
to make a completely wrong recognition. Adversarial examples not only affect thescope of image classification [31], but can also impact other fields such as natural
language processing [60], speech recognition [15], malware detection [34] and object
detection [112]. In the field of natural language processing, many recent studies onadversarial examples have been also proposed. Bin Liang et al [60] proposed a new
attack method based on the Hot Training Phrases (HTP) to deceive a deep learning
system. Ji Gao et al. [27] used two main steps for evading a target deep learningclassifier. The first step is to determine the most important tokens to modify, andthe second step is to change slightly the important words before fooling a target deeplearning system. In the speech recognition field, Nicholas Carnili et al. [15] introduced
an audio adversarial attack method that targeted to DeepSpeech [37], a state-of-theart speech-to-text transcription system. Rohan Taori et al. [111] combined genetic
algorithm and gradient method to create a black-box audio adversarial example that
attacked to an automatic speech recognition (ASR) system. In Malware detection,
Kathrin Grosse et al. [34] adopted the adversarial attack method based on the jacobian matrix to craft adversarial samples by using the DREBIN Android malware
dataset [6] to fool a malware classifier. In image classification, adversarial examples
garnered the most attention with tremendous research [3] on offensive and defensive
methods.
To deal with the rising risks of adversarial examples, there are many studies havebeen proposed with the aim to detect or defeat adversarial examples and protect a
deep learning system. Papernot et al. [87] invented a defensive method called defensive distillation to remove the effectiveness of adversarial samples on deep neuralnetworks. Buckman et al. [12] proposed a new modification strategy to neural networkarchitectures that make it more robustness against adversarial examples. Guneet etal. [20] introduced the stochastic activation pruning strategy for adversarial defense,
this method is based on the game theory to minimize the zero-sum game between
18

the adversary and the model. However, there is still no perfect defensive method foradversarial examples and previous defensive methods demand many fine-tuning andintervention from a human. For solving these disadvantages of previous defensivealgorithms, we proposed a state-of-the-art detection system that can automaticallydistinguish adversarial and benign images. Our defense strategies for defeating adversarial examples are based on image transformation and image filters that our research
focus on this direction so we will review and discuss more related research works in
the Chapter 3.

1.2

Background

Adversarial examples are affected not only in image processing, but also in manyother research fields such as natural language processing, audio/speech processing,
malware detection. But in our research direction, we only focus on image processing,
especially image classification with adversarial examples. So we concentrate on image
classification and it will be as default task throughout this thesis.

1.2.1 Image Classification
In this section we will introduce an Image Classification problem, which is a taskof assigning an input image one label from a fixed set of categories. This is one of
the core problems in Computer Vision that, despite its simplicity, has a large variety
of practical applications. Moreover, as we will see later in the section, many otherseemingly distinct Computer Vision tasks (such as object detection, segmentation)
can be reduced to image classification. Neural networks consist of elementary computing units named neurons organized in interconnected layers. Each neuron applies
an activation function to its input to produce an output. Let’s start with the input
of a machine learning model, in which each network layer produces an output used
as input by the next layer. Networks with a single intermediate hidden layer are
qualified as shallow neural networks, whereas models with multiple hidden layers are
deep neural networks. Using of multiple hidden layers is interpreted as hierarchically19extracting representations from the input, eventually producing a representation relevant to solve the machine learning task and output a prediction. A neural networkmodel F can be formalized as the composition of multidimensional and parametrizedfunctions 𝑓𝑖 each corresponding to a layer of the network architecture and a representation of the input, where each vector 𝑖 parametrizes layer i of the network F andincludes weights for the links connecting layer i to layer 𝑖1 . A set of model parameters = 𝑖 is learned during training. For instance, in a supervised learning manner,
parameter values are learned by computing prediction errors 𝑓 (𝑥) − 𝑦 on a collectionof known input output pairs (𝑥, 𝑦).

1.2.2 Linear Classification
We are now going to develop a more powerful approach to image classification that wewill eventually naturally extend to entire Neural Networks and Convolutional Neural
Networks. The approach will have two major components: a score function thatmaps the raw data to class scores, and a loss function that quantifies the agreement
between the predicted scores and the ground truth labels. We will then cast this as
an optimization problem in which we will minimize the loss function with respect tothe parameters of the score function.

Score function
The first component of this approach is to define the score function that maps the pixel
values of an image to confidence scores for each class. We will develop the approachwith a concrete example. Let’s assume a training dataset of images 𝑥𝑖 ∈ 𝑅𝐷 , eachassociated with a label 𝑦𝑖 . Here 𝑖=1...N and 𝑦𝑖 ∈ 1..𝐾 . That is, we haveN examples

D) and K distinct categories. For examples, in MNIST
dataset, we have a training set of N=50,000 images, each with D=28×28×1=784
pixels, and K=10, since there are 10 distinct classes (digit 0, 1, ..., 9). We will now(each with a dimensionality

define the score function 𝑓 : 𝑅𝐷 ↦→ 𝑅𝐾 that maps the raw image pixels to class
scores. Let consider the simplest possible linear classifier function, a linear mapping:20

𝑓 (𝑥𝑖 , 𝑊, 𝑏) = 𝑊 𝑥𝑖 + 𝑏. We assume that the image 𝑥𝑖 has all of its pixels flattenedout to a single column vector of shape [𝐷 × 1]. The matrix 𝑊 (of size[K × D]), andb (of size [K × 1]) are the parameters of the function. The parameters in
W are often called the weights, and b is called the bias vector because it influences

the vector

the output scores, but without interacting with the actual data 𝑥𝑖 . However, you willoften hear people use the terms weights and papameters interchaneably.

Loss function
In the context of an optimization algorithm, the function used to evaluate a candidate
solution (i.e. a set of weights) is referred to as the objective function. We may seek
to maximize or minimize the objective function, meaning that we are searching for a
candidate solution that has the highest or lowest score respectively. Typically, withneural networks, we seek to minimize the error. As such, the objective function isoften referred to as a cost function or a loss function and the value calculated by theloss function is referred to as simply “loss”.

1.2.3 Optimization Algorithms
Optimization is the process of finding the set of parameters

W that minimize the loss

function.

Random Search
Since it is so simple to check how good a given set of parameters

W is, the first ideathat may come to mind is to simply try out many different random weights and keep
track of what works best. The core idea is that finding the best set of weights
a very difficult or even impossible problem (especillay once

W is

W contains weights for

entire complex neural networks), but the problem of refining a specific set of weights

W to be slightly better is significantly less difficult. In other words, our approach
will be to start with a random W and then iteratively refine it, making it slightly
better each time.
21Gradient
We can compute the best direction along which we should change our weight vector
that is mathematically guaranteed to be the direction of the steepest descend (atleast in the limit as the step size goes towards zero). This direction will be related to
the gradient of the loss function. In our hiking analogy, this approach roughtly corresponds to feeling the slope of the hill below our feet and stepping down the directionthat feels steepest. In one-dimensional functions, the slope is the instantaneous rateof change of the function at any point you might be interested in. The gradient isa generalization of slope for functions that don’t take a single number but a vectorof numbers. Additionally, the gradient is just a vector of slopes (more commonlyreferred to as derivatives) for each dimension in the input space. The mathematicalexpression for the derivative of a 1-D function with respect its input is:

𝑑𝑓 (𝑥)𝑓 (𝑥 + ℎ) − 𝑓 )𝑥
= lim
ℎ→0
𝑑𝑥
ℎ

(1.1)

When the function of interest take a vector of numbers instead of a single number,we call the derivatives

partial derivatives, and the gradient is simple the vector ofpartial derivatives in each dimension. There are two ways to compute the gradient:
A slow, approximate but easy way (numerical gradient), and a fast, exact but moreerror-prone way that requires calculus (analytic gradient). The numerical gradient isvery simple to compute using the finite difference approximation, but the downside isthat it is approximate (since we have to pick a small value of h, while the true gradientis defined as the limit as h goes to zero), and that it is very computationally expensive
to compute. The second way to compute the gradient is analytically using Calculus,
which allows us to derive a direct formula for the gradient (no approximations) thatis also very fast to compute. However, unlike the numerical gradient it can be moreerror prone to implement, which is why in practice it is very common to compute theanalytic gradient and compare it to the numerical gradient to check the correctnessof your implementation. This is called a gradient check. Lets use the example of the∑︀
SVM loss function for a single datapoint: 𝐿𝑖 = 𝑗̸=𝑦𝑖 [max(0, 𝑤𝑗𝑇 𝑥𝑖 − 𝑤𝑦𝑇𝑖 𝑥𝑖 + ∆)]. We
22

can differentiate the function with respect to the weights. For example, taking the gra(︁∑︀)︁𝑇𝑇dient with respect to 𝑤𝑦𝑖 we obtain:▽𝑤𝑦𝑖 𝐿𝑖 = −
(𝑤
𝑥
−
𝑤
𝑥
+
∆
>
0)
𝑥𝑖 ,
𝑖
𝑖
𝑗
𝑦𝑖
𝑗̸=𝑦𝑖

1

where

1 is the indicator function that is one if the condition inside is true or zerootherwise. While the expression may look scary when it is written out, when you’re implementing this in code you would simply count the number of classes that did
not meet the desired margin (and hence contributed to the loss function) and thenthe data vector 𝑥𝑖 scaled by this number is the gradient. Notice that this is thegradient only with respect to the row of 𝑊 that corresponds to the correct class. For1

the other rows where 𝑗 ̸= 𝑦𝑖 the gradient is: ▽𝑤𝑦𝑖 𝐿𝑖 = (𝑤𝑗𝑇 𝑥𝑖 − 𝑤𝑦𝑇𝑖 𝑥𝑖 + ∆ > 0)𝑥𝑖 .
Once you derive the expression for the gradient it is straight-forward to implement
the expressions and use them to perform the gradient update.

Gradient Descent Now that we can compute the gradient of the loss function,
the procedure of repeatedly evaluating the gradient and then performing a parameter
update is called Gradient Descent.

Mini-batch gradient descent, in large-scale

applications (such as the ILSVRC challenge), the training data can have on orderof millions of examples. Hence, it seems wasteful to compute the full loss function
over the entire training set in order to perform only a single parameter update. A
very common approach to addressing this challenge is to compute the gradient overbatches of the training data. For example, in current state of the art Convolutional
Neural Network, a typical batch contains 256 examples from the entire training set of
1.2 million. The extreme case of this is a setting where the mini-batch contains only

Stochastic Gradient Descent (SGD) (oralso sometimes on-line gradient descent). This is relatively less common to seea single example. This process is called

because in practice due to vectorized code optimizations it can be computationallymuch more efficient to evaluate the gradient for 100 examples, than the gradient for
one example 100 times. Even though SGD technically refers to using a single example
at a time to evaluate the gradient, you will hear people use the term SGD even whenreferring to mini-batch gradient descent (i.e. mentions of MGD for “

Minibatch

Gradient Descent”, or BGD for “Batch gradient descent” are rare to see),where it is usually assumed that mini-batches are used. The size of the mini-batch is a23

hyperparameter but it is not very common to cross-validate it. It is usually based on
memory constraints (if any), or set to some value, e.g. 32, 64 or 128. We use powersof 2 in practice because many vectorized operation implementations work faster when
their inputs are sized in powers of 2.

1.2.4 Convolutional Neural Networks

Architecture Overview
Convolutional Neural Networks (CNNs) take advantage of the fact that the input
consists of images and they constrain the architecture in a more sensible way. In
particular, unlike a regular Neural Network, the layers of a Convolutional Neural
Network have neurons arranged in 3 dimensions: width, height, depth (note thatthe word depth here refers to the third dimension of an activation volume, not tothe depth of a full Neural Network, which can refer to the total number of layers
in a network.) Neural Networks receive an input (a single vector), and transform it
through a series of hidden layers. Each hidden layer is made up of a set of neurons,
where each neuron is fully connected to all neurons in the previous layer, and where
neurons in a single layer function completely independently and do not share any
connections. The last fully-connected layer is called the “output layer” and inclassification settings it represents the class scores.

Layers in CNNs. A simple CNN is a sequence of layers, and every layer of a CNN

transforms one volume of activations to another through a differentiable function.
We use three main types of layers to build CNN architectures: 𝐶𝑜𝑛𝑣𝑜𝑙𝑢𝑡𝑖𝑜𝑛𝑎𝑙𝐿𝑎𝑦𝑒𝑟,

𝑃 𝑜𝑜𝑙𝑖𝑛𝑔𝐿𝑎𝑦𝑒𝑟, and 𝐹 𝑢𝑙𝑙𝑦 − 𝐶𝑜𝑛𝑛𝑒𝑐𝑡𝑒𝑑𝐿𝑎𝑦𝑒𝑟 (exactly as seen in regular Neural Networks). We will stack these layers to form a full CNN architecture.

Convolutional Layer
The Convolutional Neural layer is the core building block of a Convolutional Neural Network that does most of the computational heavy lifting. Fig. 1-1 shows anarchitecture of Inception V3 [108], one of deep convolutional neural networks. Lets
24

Figure 1-1: Inception V3 [108] - one of deep convolutional neural networks

first discuss what the CNN layer computes without brain/neuron analogies. The CNN
layer’s parameters consist of a set of learnable filters. Every filter is small space(along
width and height), but extends through the full depth of the input volume. For example, a typical filter on a first layer of a CNN might have size 5×5×3 (i.e. 5 pixels
width and height, and 3 because images have depth 3, the color channels). During
the forward pass, we slide (more precisely, convolve) each filter across the width andheight of the input volume and compute dot products between the entries of the filter
and the input at any position. As we slide the filter over the width and height of the
input volume we will produce a 2-dimensional activation map that gives the responsesof that filter at every spatial position. Intuitively, the network will learn filters that
activate when they see some type of visual feature such as an edge of some orientation or a blotch of some color on the first layer, or eventually entire honeycomb or
wheel-like patterns on higher layers of the network. Now, we will have an entire setof filters in each CNN layer (e.g. 12 filters), and each of them will produce a separate
2-dimensional activation map. We will stack these activation maps along the depth
dimension and produce the output volume.

Local Connectivity. When dealing with high-dimensional inputs such as images, as we saw above it is impractical to connect neurons to all neurons in theprevious volume. Instead, we will connect each neuron to only a local region of the
input volume. The spatial extent of this connectivity is a hyper parameter called thereceptive field of the neuron (equivalently this is the filter size). The extent of the25connectivity along the depth axis is always equal to the depth of the input volume.
It is important to emphasize again this asymmetry in how we treat the spatial dimensions (width and height) and the depth dimension: The connections are local inspace (along width and height), but always full along the entire depth of the input
volume.

Spatial arrangement. We have explained the connectivity of each neuron in

the Convolutional Neural Layer to the input volume, but we haven’t yet discussed
how many neurons there are in the output volume or how they are arranged. Three
hyperparameters control the size of the output volume: the depth, stride and zeropadding. We discuss these as bellow:∙

depth. The depth of the output volume is a hyper parameter: it corresponds tothe number of filters we would like to use, each learning to look for something
different in the input. For example, if the first

Convolutional Layer takes

as input the raw image, then different neurons along the depth dimension may
activate in presence of various oriented edges, or blobs of color. We will refer toa set of neurons that are all looking at the same region of the input as acolumn.
∙depth

stride. We must specify the stride with which we slide the filter. When thestride is 1 then we move the filters one pixel at a time. When the stride is 2 (oruncommonly 3 or more, though this is rare in practice) then the filters jump
2 pixels at a time as we slide them around. This will produce smaller output
volumes spatially.

∙

zero-padding. Sometimes it will be convenient to pad the input volume withzeros around the border. The size of this zero-padding is a hyper parameter.
The nice feature of zero padding is that it will allow us to control the spatial
size of the output volumes (most commonly as we will see soon we will use it
to exactly preserve the spatial size of the input volume so the input and output
width and height are the same).
26

Pooling Layer
It is common to periodically insert a Pooling layer in-between successive Convolutional
Neural layers in a Convolutional Neural Network architecture. Its function is toprogressively reduce the spatial size of the representation to reduce the amount ofparameters and computation in the network, and hence to also control overfitting.
The Pooling Layer operates independently on every depth slice of the input and resizes
it spatially, using the MAX operation. The most common form is a pooling layer withfilters of size 2 × 2 applied with a stride of 2 downsamples every depth slice in the
input by 2 along both width and height, discarding 75% of the activations. Every
MAX operation would in this case be taking a max over 4 numbers (little 2 × 2 region
in some depth slice). The depth dimension remains unchanged. More generally, thepooling layer:∙ Accepts a volume of size W1 × H1 × D1
∙ Requires two hyper parameters: their spatial extent

F and the stride S∙ Produces a volume of size W2 × H2 × D2 where: W2 = (W1 − F/)S + 1, H2 =(H1 − F/)S + 1 and D2 = D1
∙ Creates zero parameters since it computes a fixed function of the input∙ For Pooling layers, it is not common to pad the input using zero-padding
It is worth noting that there are only two commonly seen variations of the max
pooling layer found in practice: a pooling layer with F=3, S=2 (also called overlapping
pooling), and more commonly F=2, S=2. Pooling sizes with larger receptive fields
are too destructive.

Normalization Layer
Many types of normalization layers have been proposed for use in CNN architectures,
sometimes with the intentions of implementing inhibition schemes observed in the biological brain. However, these layers have since fallen out of favor because in practicetheir contribution has been shown to be minimal, if any.
27

Fully-connected layer
Neurons in a fully connected (FC) layer have full connections to all activations in
the previous layer, as seen in regular Neural Networks. Their activations can hence
be computed with a matrix multiplication followed by a bias offset. It is worth
noting that the only difference between FC and CNN layers is that the neurons in the
convolutional layer are connected only to a local region in the input, and that manyof the neurons in a convolutional volume share parameters. However, the neurons inboth layers still compute dot products, so their functional form is identical. Therefore,
it turns out that it’s possible to convert between FC and convolutional layers:∙ For any convolutional layer there is an FC layer that implements the same
forward function. The weight matrix would be a large matrix that is mostly
zero except for at certain blocks (due to local connectivity) where the weights
in many of the blocks are equal (due to parameter sharing).

∙ Conversely, any FC layer can be converted to a convolutional layer. For example,
an FC layer with K=4,096 that is looking at some input volume of size 7×7×512
can be equivalently expressed as a convolutional layer with F=7, P=0, S=1,
K=4,096. In other words, we are setting the filter size to be exactly the size ofthe input volume, and hence the output will simply be 1 × 1 × 4, 096 since onlya single depth column “fits” across the input volume, giving identical result
as the initial FC layer.
Of these two conversions, the ability to convert an FC layer to a convolutional layer isparticularly useful in practice. Consider a CNN architecture that takes a 299×299×3
image, and then uses a series of convolutional layers and POOL layers to reduce theimage to an activations volume of size 7 × 7 × 512.

28

Chapter 2
Literature Review on Adversarial
Examples2.1

A taxonomy of Adversarial Attacks

In this section, we will provide a qualitative taxonomy on different terms and keywords
related to adversarial attacks and categorize the threat models. Adversarial examples
are considered as one variant of adversarial machine learning that has been studied
for decades [19]. Adversarial machine learning covers many privacy and security
problems on a machine learning system, such as biometric authentication [54], spam
detection [92], and fraud detection [26]. Adversarial examples [109] is a new type ofadversarial machine learning that is mainly focused on deceiving a machine learningby slightly modifying legitimate input but unrecognized by a human. A machinelearning system has normally been designed with an assumption that both training
and test datasets are benign during training and evaluating phases. In the training
process, the data distribution is squeezed and only the weights of the model are finetuned by using gradient descent and backpropagation of the gradients to find thebest weights for the input-output pairs (𝑥, 𝑦). The inputs 𝑥 are usually considered tobe sampled independently from the unchanged distribution in the training process.
These assumptions are useful to concentrate on adopting the weights of the machinelearning model. The creation process of adversarial examples is performed in the29

opposite way. The attackers squeeze the weights of the victim machine learningmodel and slightly modify the input data to cause the classifier to misclassify. The
new input data is called adversarial examples. There are many ways to categorize
the adversarial examples attacks and they depend on the adversary purposes or theamount of knowledge that attackers are able to learn from the victim models.

2.1.1 Targeted/Non-targeted Attacks
Firstly, the adversarial attacks can be categorized by the type of adversary goals.

∙

Non-targeted Attack. In this the case adversary’s goal is to cause the classifier to predict any inccorect label. The specific incorrect label does not matter.
There are several not-targeted attacks [52, 106, 81, 80] that were considered andinvestigated.

∙

Targeted Attack. In this case the adversary aims to change the classifier’s prediction to some specific target class. This type of adversarial attackis attracted by many researchers and there are many proposed targeted attacks [109, 31, 85, 14].

2.1.2 White-box/Black-box Attacks
Second, the adversarial attacks can be categorized by the amount of knowledge theattackers have about the victim models.

∙White-box Attacks. In the white box scenario [109, 31, 85, 14, 81, 80, 69, 14],
the adversary has full knowledge of the model including model type, modelarchitecture and values of all parameters an trainable weights.

∙

Black-box Attacks. There are two type of black-box attacks [106, 84, 64, 43,
1, 14] where attackers have limitation to access into victim models for creating
adversarial examples.

Black box with querying. In this scenario, the adversary does not know
very much about the model, but can query the model, i.e. feed some inputs and30

observe outputs. There are many variants of this scenario—the adversary may
know the architecture but not the parameters or the adversary may not evenknow the architecture, the adversary may be able to observe output probabilities
for each class or the adversary may only be to observe the choice of the most
likely class.

Black box without querying. In the black box without querying scenario,
the adversary has limited or no knowledge about the model under attack and is
not allowed to query the model while constructing adversarial examples. In thiscase, the attacker must construct adversarial examples that fool most machinelearning models.
We describe several common adversarial attack methods as below. Note that someof them can be applied to both White-box and Black-box manners.

L-BFGS. Szegedy et al. [109] used a method name L-BFGS (Limited-memory

Broyden-Fletcher-Goldfarb-Shanno) to create targeted adversarial examples. Thismethod minimize the weighted sum of perturbation size 𝜀 and loss function 𝐿(𝑥* , 𝑦𝑡𝑎𝑟𝑔𝑒𝑡 )while constraining the elements of 𝑥* to be normal pixel value.

FGSM. Goodfellow et al. [31] assumed that adversarial examples can be caused

by cumulative effects of high dimensional model weights. They proposed a simple
attack method but very effective, called Fast Gradient Sign Method (FGSM):(2.1)

𝑥* = 𝑥 + 𝜀 · 𝑠𝑖𝑔𝑛(O𝑥 𝐿(𝑥, 𝑦))where 𝜀 denotes the perturbation size for crafting adversarial example 𝑥* from original
input 𝑥. Given a clean image 𝑥, this method tries to create a similar image 𝑥* in

𝐿∞ neighborhood of 𝑥 that fools the target classifier. This leads to maximize loss
function 𝐿(𝑥, 𝑦) which is the cost of classifying image 𝑥 as the target label 𝑦 . The fast
gradient sign method solves this problem by performing a one-step gradient updatefrom 𝑥 in the input space with a small size of perturbation 𝜀. Increasing 𝜀 will leadto a higher and faster attack success rate however it may also make your adversarial
sample to be more different from the original input. FGSM computes the gradients31

for once, so it is much more efficient than L-BFGS. This method is very simple but
it is fast and powerful for creating the adversarial examples.

PGD. Madry et al. [69] proposed Projected Gradient Descent (PGD) attack:𝑥* = 𝑥 + 𝛿 · (▽𝐿 (𝑥, 𝑦)) respect to project(𝑥,𝜖) (𝑥* )(2.2)

Where project(𝑥,𝜖) (𝑥* ) defines a projection operator with parameter 𝑥* on the circle
area around 𝑥 with radius 𝜖, 𝛿 is a clip value that is searched in a box (𝑥, 𝜖). Thismethod based on another method named Basic Iterative Method (BIM) [52] that is
extended from the FGSM by applying it multiple times with a small step size andclipping values of intermediate results after each step to ensure that they are in an 𝜖neighborhood of the original input. The adversarial examples and their counterparts
are defined as indistinguishable by humans. Because it is hard to model humanperception, researchers use three popular distance metrics to approximate human’s
perception based on the 𝐿𝑝 norm:||𝑥||𝑝 =

(︂ ∑︁
𝑛

𝑝

)︂ 𝑝1

(2.3)

|𝑥𝑖 |

𝑖=1

C&W. Carnili et al. [14] proposed a very powerful attack method by using
𝐿0 , 𝐿2 , 𝐿∞ metrics to control adversarial perturbation. 𝐿0 helps to calculate thenumber of pixels with different values at corresponding positions in two images. It
indicates how many pixels between the two images are modified. 𝐿2 is applied formeasuring the Euclidean distance between two images. And 𝐿∞ will help to measure
the maximum difference for all pixels at corresponding positions in two images. So far
it is unclear which one is the best distance metric for crafting adversarial examples
because it depends on the proposed algorithms.

EAD:Elastic-Net Attack. EAD adversarial attack was invented by Pin-Yu
Chen et al. [16] inspired from [14]. This paper uses elastic-net regularization technique [124] that is widely used in solving high-dimensional feature selection problems
to invent new attack method by extending from C&W method.
32

Deep Fool. Moosavi-Dezfooli et al. [81] proposed an attack method to computea minimal norm adversarial perturbation for a given image in an iterative manner.
Their algorithm, i.e. DeepFool initializes with the clean image that is assumed toreside in a region confined by the decision boundaries of the classifier. This region
decides the class-label of the image. At each iteration, the algorithm perturbs theimage by a small vector that is computed to take the resulting image to the boundaryof the polyhedron that is obtained by linearizing the boundaries of the region within
which the image resides. The perturbations added to the image in each iteration are
accumulated to compute the final perturbation once the perturbed image changes
its label according to the original decision boundaries of the networks. The authors
show that the DeepFool algorithm is able to compute perturbations that are smaller
than the perturbations computed by FGSM [31] in terms of their norm while having
similar fooling ratios.

One-pixel Attack. [106] proposed a new method when only one pixel in theimage is changed to fool the classifier. The method successfully deceives three different
network models on 70.97% of the tested images by changing just on a pixel per image.
They also reported that the average confidence of the networks on the wrong labels
was found to be 97.47%. For a clean image, they first created a set of 400 vectors in

𝑅5 such that each vector contained xy-coordinates and RGB values for an arbitrarycandidate pixel. They, they randomly modified the elements of the vectors to createchildren such that a child competes with its parent for fitness in the next iteration,
while the probabilistic predicted label of the network is used as the fitness criterion.
The last surviving feature is used to alter the pixel in the image.

Universal Attack. Whereas the methods like FGSM [31], ILCM [52], DeepFool [81] etc. compute perturbations to fool a network on a single image, the ’universal’ adversarial perturbations computed by Moosavi-Dezfooli et al. [80] are able to foola network on ’any’ image with high probability. These image gnostic perturbations
also remain quasi-imperceptible for the human vision system. The authors computed
the universal perturbations by restricting their 𝑙2 norm as well as 𝑙∞ norm and showedthat the perturbations with their norms already achieved significant fooling ratios of33

around 0.8 or more for state-of-the-art image classifiers. Their iterative approach tocompute a perturbation is related to the DeepFool [81] strategy of gradually pushinga data point to the decision boundary for its class. However, in that case, ’all’ thetraining data points are sequentially pushed to the respective decision boundaries
and the perturbations computed over all the images are gradually accumulated byback-projecting the accumulator to the desired 𝑙𝑝 ball of radius 𝜖 every time.

Spatially Transformed Attack. Traditional adversarial attack algorithms di-

rectly modify the pixel value of an image, which changes the image’s color intensity.
Xiao et al. [118] proposed a new method called Spatially Transformed Attack. They
perturb the image by doing slight spatial transformation: they translate, rotate anddistort the local image features slightly. The perturbation is small enough to evade
human inspection but it can fool the classifiers.

Jacobian-Based Saliency Map Attack. Papernot et al. [85] proposed anattack method based on Jacobian call Jacobian-Based Saliency Map Attack. Thatmethod calculates the Jacobian matrix of the score function 𝐹 . It can be viewed
as a greedy attack algorithm by iterative manipulating the pixel which is the mostinfluential to the model output.

2.1.3 Poisoning Attack vs Evasion Attack
Third, the adversarial attacks can be classified by the adversary’s goal. Adversaries
with no access to the pre-processed data must instead poison the model’s training
data before its pre-processing. For instance, Perdisci et al. [82] prevented Polygraph,a worm signature generation tool, from learning meaningful signatures by inserting
perturbations in worm traffic flows [88]. Polygraph combines a flow tokenizer together with a classifier that determines whether a flow should be in the signature.
Polymorphic worms are crafted with noisy traffic flows such that (1) their tokenizedrepresentations will share tokens, not representative of the worm’s traffic flow, and(2) they modify the classifier’s threshold for using a signature to flag worms. This
attack forces Polygraph to generate signatures with tokens that do not correspondto invariants of the worm’s behavior. Later, Xiao et al. [119] adapted the gradient34

ascent strategy introduced in feature selection algorithms like LASSO.

∙

Poisoning Attacks. The attacking algorithms [11, 48, 98] that allow an attacker to insert/modify several fake samples into the training dataset of a deeplearning system. These fake samples can cause failures of the trained classifier.
They can result in poor accuracy or wrong prediction [125] on some given test
samples. This type of attacks frequently appears in the situation where theadversary has access to the training database. For example, web-based repositories and "honeypots" [105] often collect malware examples [35] for training,which provides an opportunity for adversaries to poison the data.

∙

Evasion Attacks. The classifiers are fixed and usually have a good performance on benign testing samples [10]. The adversaries do not have the authority to change the classifier or its parameters, but they craft some fake samples
that the classifier cannot recognize. In other words, the adversaries generatesome fraudulent examples to evade detection by the classifier. For example,in [24], in autonomous driving vehicles, sticking a few pieces of tapes on thestop signs can confuse the vehicles’s road sign recognizer.

2.1.4 Physical Attacks
In the case of an attack in the physical world, the adversary does not have directaccess to the digital representation provided to the model. Instead, the model is fed
input obtained by sensors such as a camera or microphone. The adversary is able
to place objects in the physical environment seen by the camera or produce sounds
heard by the microphone. The exact digital representation obtained by the sensors
will change based on factors like the camera angle, the distance to the microphone,ambient light or sound in the environment, etc. This means the attacker has lessprecise control over the input provided to the machine learning model.

Attack via Camera Phone. Kurakin et al. [52] first demonstrated that threatsof adversarial attacks also exist in the physical world. To illustrate this (see Fig. 2-1),
they printed adversarial images and took snapshots from a cell-phone camera. These35

images were fed to the Tensor-Flow Camera Demo app that uses Google’s Inception
model [108] for object classification. It was shown that a large fraction of images weremisclassified even when perceived through the camera. This work studies FGSM [31]and ILCM [52] attack methods for attacks in the physical world.

Figure 2-1: Example of the adversarial attack on mobile phone cameras: A clean
image (a) was taken and used to generate different adversarial images. The images
were printed and the TensorFlow Camera Demo app was used to classify them. A
clean image (b) is recognized correctly as a “washer” when perceived through thecamera, whereas adversarial images (c) and (d) are misclassified. The images also
show network confidence in the range [0,1] for each image.

Attack to Road Sign. Eykholt Kevin et al. [24] built on the attacks proposed in [14] and [64] to design robust perturbations for the physical world. They
demonstrated the possibility of attacks that are robust to physical conditions, such as
variation in view angles, distance, and resolution. The proposed algorithm, termed

𝑅𝑃2 for Robust Physical Perturbations, was used to generate adversarial examples
for road sign recognition systems that achieved high fooling ratios in practical driveby settings. Two attack classes were introduced in this work for the physical roadsigns, (a) poster-printing: where the attacker prints a perturbed road sign poster andplaces it over the real sign (see Fig. 2-2), (b) sticker perturbation: where the printing is done on a paper and the paper is stuck over the real sign. For (b) two typesof perturbations were studied, (b1) subtle perturbations: that occupied the entire
36

sign and (b2) camouflage perturbations: that took the form of graffiti sticker on thesign. As such, all these perturbations require access to a color printer and no otherspecial hardware. Successful generation of perturbations for both (a) and (b) suchthat the perturbations remained robust to natural variations in the physical world
demonstrate the threat of adversarial example in the real world.

Figure 2-2: Example of road sign attack [24]: The success rate of fooling LISA-CNN
classifier on all the shown images is 100%. The distance and angle to the camera arealso shown. The classifier is trained using LISA dataset for road signs [78].

2.1.5 Classification-based Attacks
Adversarial attacks are also categorized by the targeted classification/recognition
tasks.

Attack on Recurrent neural networks
Papernot et al. [86] successfully generated adversarial input sequences for Recurrent
Neural Networks (RNNs). RNNs are deep learning models that are particularly suitable for learning mappings between sequential inputs and outputs. Papernot et al.
demonstrated that the algorithms proposed to compute adversarial examples for the
feed-forward neural networks can also be adapted for fooling RNNs. In particular,
the authors demonstrated successful fooling of the popular Long short-term memory(LSTM) [40] RNN architecture. It is concluded that the cyclic neural network model
like RNNs are also not immune to the adversarial perturbations that were originally
uncovered in the context of acyclic neural networks.
37

Attack on Deep Reinforcement Learning
Lin et al. [63] proposed two different adversarial attacks for the agents trained by deep
reinforcement learning [77]. In the first attack, called ’strategically-timed attack’, theadversary minimizes the reward of the agent by attacking it at a small subset oftime steps in an episode. A method is proposed to determine when an adversarial
example should be crafted and applied, which enables the attack to go undetected. In
the second attack, referred as ’enchanting attack’, the adversary lures the agent to a
designated target state by integrating a generative model and a planning algorithm.
The generative model is used for predicting the future states of the agent, whereas
the planning algorithm generates the actions for alluring it.

Attack on Autoencoders and generative models
Tabacof et al. [110] proposed an adversarial attack for autoencoders [89], that misleads
the autoencoder to reconstruct a completely different image. Their approach attacks
the internal representation of a neural network such that the representation for theadversarial image becomes similar to that of the target image. However, [110] reported
that autoencoders seem to be much more robust to adversarial attacks than the typicalclassifier networks. Kos et al. [50] also explored methods for computing adversarial
examples for deep generative models, e.g. variational autoencoder (VAE) and VAEGenerative Adversarial Networks (VAE-GANs). GANs, such as [123] are becoming
exceedingly popular nowadays in Computer Vision applications due to their ability
to learn data distributions and generate realistic images using those distributions.
The authors introduced three different classes of attacks for VAE and VAE-GANs.
Owing to the success of these attacks it is concluded that the deep generative models
are also vulnerable to adversaries that can convince them to turn inputs into very
different outputs. This work adds further support to the hypothesis that ’adversarial
examples’ are a general phenomenon for current neural network architectures’.
38Attack on Semantic Segmentation and Object Detection
Semantic image segmentation and object detection are among the mainstream problems in Computer Vision. Inspired by Moosavi-Dezfooli et al. [80] and Metzen etal. [73] showed the existence of image-agnostic quasi-imperceptible perturbations thatcan fool a deep neural network into significantly corrupting the predicted segmentation of the images. Moreover, they also showed that it is possible to computenoise vectors that can remove a specific class from the segmented classes while keeping most of the image segmentation unchanged (e.g removing pedestrians from road
scenes). Although it is argued that the ’space of the adversarial perturbations for thesemantic image segmentation is presumably smaller than image classification’, theperturbations have been shown to generalize well for unseen validation images withhigh probability. Arnab et al. [5] also evaluated FGSM [31] based adversarial attacks
for semantic segmentation and noted that many observations about these attacks forclassification do not directly transfer to segmentation task.

2.1.6 Miscellaneous Attacks on other Domain
The adversarial attacks discussed above are either the popular ones in the recent
literature or they are representative of the research directions that are fast becoming
popular.

Graph Adversarial Examples
Adversarial examples also exist in graph-structured data [125, 18]. Attackers usuallyslightly modify the graph structure and node features, in an effort to cause the graphneural networks [79, 47] to give the wrong prediction for node classification or graphclassification tasks. These adversarial attacks, therefore, raise concerns on the securityof applying graph neural networks. For example, a bank needs to build a reliablecredit evaluation system where their model should not be easily attacked by malicious
manipulations.
39

Adversarial Examples in Audio
The work in [15] studies how to attack state-of-the-art speech-to-text transcription
networks, such as DeepSpeech [37]. In their settings, when given any speech waveform𝑥, they can add an inaudible sound perturbation 𝜃 that makes the synthesized speech𝑥 + 𝜃 be recognized as any targeted desired phrase. In their attacking work, they
limited the maximum decibels (dB) at any time of the added perturbation noise, sothat the audio distortion is unnoticeable.

Adversarial Examples in Video
Most works concentrate on attacking static image classification models. However,success on image attacks cannot guarantee that there exist adversarial examples onvideos and video classification systems. There are some works [104, 112] aim to deceive
object detection systems such as YOLOv2 [90], Faster R-CNN [91].The work [112]generates adversarial patches to targets with lots of intra-class variety, namely persons. The goal is to generate a patch that is able to successfully hide a person fromthe state-of-the-art object detector YOLOv2 [90].

Adversarial Examples in Text
Text classification is one of the main tasks in natural language processing. In textclassification, the model is devised to understand a sentence and correctly label thesentence. For example, text classification models can be applied to the IMDB dataset
for characterizing user’s opinion (positive or negative) on the movies, based on theprovided reviews. Recent works of adversarial attacks have demonstrated that textclassifiers are easily misguided by slightly modifying the text’s spelling, words or
structures. There are many researches work on the adversarial example in text [76, 23,
27, 59]. The work [76] considers to add perturbation on the word embedding [74], so as
to fool a LSTM [40] classifier. However, this attack only considers perturbing the wordembedding, instead of the original input sentence. The work HotFlip [23] considers
replacing a letter in a sentence in order to mislead a character level text classifier (each40letter is encoded to a vector). For example, changing a single letter in a sentence alters
the model’s prediction on its topic. The attack algorithm manages to achieve thisby finding the most influential letter replacement via gradient information. These
adversarial perturbations can be noticed by human readers, but they don’t change
the content of the text as a whole, nor do they affect human judgments. The workin [60] considers manipulating the victim sentence on word and phrase levels. They
try adding, removing or modifying the words and phrases in the sentences.

Adversarial Examples on Face recognition
Face attributes are among the emerging soft biometrics for modern security systems.
Although face attribute recognition can also be categorized as a classification problem, we separately review some interesting attacks in this direction because facerecognition itself is treated as a mainstream problem in Computer Vision. Rozsa etal.

[95, 94] explored the stability of multiple deep learning approaches using the

CelebA benchmark [65] by generating adversarial examples to alter the results offacial attribute recognition. By attacking the deep network classifiers with their socalled ‘Fast Flipping Attribute’ technique, they found that robustness of deep neuralnetworks against the adversarial attacks varies highly between facial attributes. It
is claimed that adversarial attacks are very effective in changing the label of a target attribute to a correlated attribute. Mirjalili and Ross [75] proposed a techniquethat modifies a face image such that its gender (for a gender classifier) is modified,
whereas its biometric utility for a face matching system remains intact. Similarly,
Shen et al. [101] proposed two different techniques to generate adversarial examples
for faces that can have high ‘attractiveness scores’ but low ‘subjective scores’ for theface attractiveness evaluation using deep neural network. We refer to [100] for further
attacks related to the task of face recognition.

Adversarial Examples on Malware Detection
The existence of adversarial examples in safety-critical tasks, such as malware detection, should be paid much attention. The work [34] built a DNN model on the
41

DREBIN dataset by [51], which contains 120,000 Android application samples, where
over 5,000 are malware samples. The trained model has 97% accuracy, but malware samples can evade the classifier if attackers add fake features to them. Anotherwork [4] considers using GANs to generate adversarial malware.

Adversarial Examples on Fingerprint Recognition
Fingerprint recognition systems are also one of the most safety-critical fields wheremachine learning models are adopted. While there are adversarial attacks undermining the reliability of these models. For example, fingerprint spoof attacks copy
an authorized person’s fingerprint and replicate it on some special materials suchas liquid latex or gelatin. Traditional fingerprint recognition techniques especially
minutiae-based models fail to distinguish the fingerprint images generated from different materials. The works [49, 17] design a modified CNN to effectively detect thefingerprint spoof attack.

2.1.7 Summary on Adversarial Attacks
In summary, the proliferation of AI in many fields means an increase in many security risks for those AI systems themselves. Adversarial attacks have been and willcontinue to receive a lot of attention both in research and reality. The adversarial
attacks methods are proposed in various fields such as image classification [14], object
detection [80], speech recognition [15], face recognition [101], malware [4], fingerprint
recognition [49, 17], and text classification [23]. It is obvious that adversarial attacks
are a very challenging issue because the characteristics of it are difficult to detect byhumans but it is possible to deceive the latest AI system. The adversarial attacks
will certainly continue to be studied more in the future.
42

2.2

Defense Strategies to Adversarial Examples

2.2.1 Adversarial Training
Since the discovery of adversarial examples for the deep neural networks [109], there
has been a general consensus in the related literature that the robustness of neural networks against these examples improves with adversarial training. Therefore,
most of the contributions introducing new adversarial attacks, e.g. [109, 31, 81] simultaneously propose adversarial training as the first line of defense against those
attacks. Although adversarial training improves the robustness of a network, it is a
non-adaptive strategy that requires training to be performed using strong attacks andthe architecture of the network is sufficiently expressive. Since adversarial training
necessitates increased training data size, we refer to it as a “brute-force” strategy. It isalso commonly observed in the literature that brute force adversarial training results
in regularizing the network (e.g. see [31, 97]) to reduce overfitting, which in turn improves the robustness of the networks against the adversarial attacks. Inspired by thisobservation, Miyato et al. [76] proposed a “Virtual Adversarial Training” approach tosmooth the output distributions of the neural networks. A related “stability training”
method is also proposed by Zheng et al. [122] to improve the robustness of neuralnetworks against small distortions to input images. It is noteworthy that whereas
adversarial training is known to improve the robustness of neural networks, MoosaviDezfooli et al. [80] showed that effective adversarial examples can again be computed
for already adversarially trained networks.

2.2.2 Gradient Masking
Ross and Doshi-Velez [93] studied input gradient regularization [21] as a method foradversarial robustness. Their method trains differentiable models (e.g. deep neuralnetworks) while penalizing the degree of variation resulting in the output with respect
to change in the input. Implying, a small adversarial perturbation becomes unlikely
to change the output of the trained model drastically. It is shown that this method,43

when combined with brute-force adversarial training, can result in very good robustness against attacks like FGSM [31] and JSMA [85]. However, each of these methods
almost doubles the training complexity of a network, which is already prohibitive inmany cases. Previously, Lyu et al. [68] also used the notion of penalizing the gradientof loss function of network models with respect to the inputs to incorporate robustness in the networks against L-BFGS [109] and FGSM [31] based attacks. Similarly,
Shaham et al. [99] attempted to improve the local stability of neural networks by minimizing the loss of a model over adversarial examples at each parameter update. They
minimized the loss of their model over worst-case adversarial examples instead of theoriginal data. In related work, Nguyen and Sinha [83] introduced a masking based
defense against C&W attack [14] by adding noise to the logit outputs of networks.

2.2.3 Image Compression
Dziugaite et al. [22] noted that most of the popular image classification datasetscomprise JPG images. Motivated by this observation, they studied the effects of
JPG compression on the perturbations computed by FGSM [31]. It is reported that
JPG compression can actually reverse the drop in classification accuracy to a large
extent for the FGSM perturbations. Nevertheless, it is concluded that compression
alone is far from an effective defense. JPEG compression was also studied by Guo
et al. [36] for mitigating the effectiveness of adversarial images. Moreover, Shin andSong [102] have demonstrated the existence of adversarial examples that can survive
JPEG compression. Compression under Discrete Cosine Transform (DCT) was also
found inadequate as a defense against the universal perturbations [80] in a previous
work [2]. One major limitation of compression based defense is that larger compressions also result in loss of classification accuracy on clean images, whereas smaller
compressions often do not adequately remove the adversarial perturbations. In another related approach, Bhagoji et al. [9] proposed to compress input data using
Principal Component Analysis for adversarial robustness.
44

2.2.4 Image Denoising
There are some previous research works that tried to remove adversarial noises andregained the right recognition of classifiers. Liao et al. [62] proposed a method named
High-level representation Guided Denoiser (HGD) as a defense for image classification
systems. This paper argued that many defense models can not remove all adversarial
perturbations so non-removed adversarial noises will are amplified to enormities in the
top layers of the target model and this will lead to a wrong prediction. To thoroughly
overcome this problem, they proposed a system in which the denoiser is trained by ahigh-level representation guided denoiser (HGD) loss function. However, this paper
only implemented on ImageNet includes color images and not applied to a gray-scale
dataset such as MNIST. This is not very important but it may not work well on grayimages when the method based on the high-level representation in a very deep neuralnetwork meanwhile a simple neural network can be used and works well for MNIST.
Xu et al. [120] used a strategy to reduce the degress of freedom available to an adversary by squeezing out unnecessary input features that they call “Feature Squeezing".
The main idea in this paper is that they use two different denoisers to input forsqueezing unnecessary features, afterward, they compare the prediction results fromthe target model and decide that input is adversarial or legitimate. They applied two
denoising methods: (1) squeezing color bit depth and (2) spatial smoothing method.
They evaluated their proposed method on variety of adversarial attacks methods,
however, there is unclear how they decided variety specified thresholds on different
benchmark datasets so those settings will make operators be exhausted when they
have to decide the thresholds for their systems and it is obviously ineffective when
this system has to cope with new and unknown dataset.

2.2.5 Image Transformation
There are several proposed defense methods based on image transformation. Lu
et al. [67] proposed the method to defeat adversarial examples based on changing
distance scaling. In this work, the authors evaluated the recognition capabilities of45

the AI system when a distance from a road sign to a camera is 0.5 meters and 1.5
meters. This method is regarded as a scale-based method since the authors tried totest the AI system’s recognition capabilities with a range of sizes of different object
sizes in images. Athalye et al. [7] proposed a new method to craft adversarial examples
on a variety of synthesized images by applying distortion and affine transformation.
This paper proposed the attack method, not a defense in an adversarial manner.
Flowchart of this proposed method can be described as original images -> modified
images (by distortion and/or affine transformation) -> create adversarial examples
from modified images. Tian et al. [113] used rotation techniques to train detector
neural networks for MNIST and CIFAR-10 datasets. They trained four detector
networks on different training sets. This research belongs to a training classifier-based
method where authors use both benign and adversarial images to train a classifier. By
using this strategy, a classifier that was trained with benign and adversarial examples
learned more important features and discriminated features than other networks thatwere trained only with benign images. After training, the author re-test detector
networks by using rotated benign and adversarial images to evaluate the AI system’srecognition capabilities. However, this paper only evaluated their method on smallscale datasets such as MNIST and CIFAR-10 without a large-scaled dataset like
ImageNet.

2.2.6 Miscellaneous countermeasures
Liang et al. [61] treated perturbations to images as noise and used scalar quantizationand spatial smoothing filter to separately detect such perturbations. In a related approach, Feinman et al. [25] proposed to detect adversarial perturbations by harnessing
uncertainty estimates (of dropout neural networks) and performing density estimation in the feature space of neural networks. Eventually, separate binary classifiers
are trained as adversarial example detectors using the proposed features. Gebhart
and Schrater [28] viewed neural network computation as information flow in graphs
and proposed a method to detect adversarial perturbations by applying persistenthomology to the induced graphs.
46

Detector subnetwork
Metzen et al. [72] proposed to augment a targeted network with a subnetwork thatis trained for a binary classification task of detecting adversarial perturbations in
inputs. It is shown that appending such a network to the internal layers of a modeland using adversarial training can help in detecting perturbations generated using
FGSM [31], BIM [52] and DeepFool [81] methods.

Detection method based on activation function
Lu et al. [66] hypothesized that adversarial examples produce different patterns of
ReLU activations in (the late stages of) networks than what is produced by clean
images. Based on this hypothesis, they proposed to append a Radial Basis Function
SVM classifier to the targeted models such that the SVM uses discrete codes computedby the late-stage ReLUs of the network. To detect perturbation in a test image, its
code is compared against those of training samples using the SVM. Effective detectionof adversarial examples generated by [31, 52, 81] is demonstrated by their framework,
named SafetyNet.

Data augmentation
Grosse et al. [33] proposed to augment the potentially targeted neural network modelwith an additional class in which the model is trained to classify all the adversarial
examples. Hosseini et al. [42] also employed a similar strategy to detect black-box
attacks.

Defense against universal perturbations
Akhtar et al. [2] proposed a defense framework against the adversarial attacks generated using universal perturbations [80]. The framework appends extra ‘pre-input’
layers to the targeted network and trains them to rectify a perturbed image so thatthe classifier’s prediction becomes the same as its prediction on the clean version of
the same image. The pre-input layers are termed Perturbation Rectifying Network
47

(PRN), and they are trained without updating the parameters of the targeted network. A separate detector is trained by extracting features from the input-output
differences of PRN for the training images. A test image is first passed through the
PRN and then its features are used to detect perturbations. If adversarial perturbations are detected, the output of PRN is used to classify the test image.

Defense based on Generative Adversarial Networks
Lee et al. [58] used the popular framework of Generative Adversarial Networks [30]to train a network that is robust to FGSM like attacks. The authors proposed todirectly train the network along with a generator network that attempts to generate
perturbation for that network. During its training, the classifier keeps trying tocorrectly classify both the clean and perturbed images. We categorize this technique
as an ‘add-on’ approach because the authors propose to always train any network inthis fashion. In another GAN-based defense, Shen et al. [45] use the generator partof the network to rectify a perturbed image.

MagNet method
Meng and Chen [71] proposed a framework that uses one or more external detectors
to classify an input image as adversarial or clean. During training, the frameworkaims at learning the manifold of clean images. In the testing phase, the images thatare found far from the manifold are treated as adversarial and are rejected. The
images that are close to the manifold (but not exactly on it) are always reformed tolie on the manifold and the classifier is fed with the reformed images. The notionof attracting nearby images to the manifold of clean images and dropping the far-off
images also inspires the name of the framework, i.e. MagNet.

2.2.7 Summary on Adversarial Defense
Since adversarial perturbations generated by many methods look like high-frequency
noise to a human observer1 multiple authors have suggested using image preprocess48

ing and denoising as a potential defense against adversarial examples. There is alarge variation in the proposed preprocessing techniques, like doing JPEG compression [22] or applying median filtering and reducing the precision of input data [120].
While such defenses may work well against certain attacks, defenses in this category
have been shown to fail in the white box case, where the attacker is aware of the defense [38]. Many defenses, intentionally or unintentionally, fall into a category called“gradient masking”. Most white-box attacks operate by computing gradients of themodel and thus fail if it is impossible to compute useful gradients. Gradient masking
consists of making the gradient useless, either by changing the model in some waythat makes it non-differentiable or makes it have zero gradients in most places or makethe gradients point away from the decision boundary. Essentially, gradient masking
means breaking the optimizer without actually moving the class decision boundaries
substantially. Because the class decision boundaries are more or less the same, defenses based on gradient masking are highly vulnerable to black-box transfer [84].
Some defense strategies (like replacing smooth sigmoid units with hard thresholdunits) are intentionally designed to perform gradient masking. Many defenses are
based on detecting adversarial examples and refusing to classify the input if there aresigns of tampering [72]. This approach works long as the attacker is unaware of thedetector or the attack is not strong enough. Otherwise, the attacker can construct
an attack that simultaneously fools the detector into thinking an adversarial input isa legitimate input and fools the classifier into making the wrong classification [13].
The most popular defense in current research papers is probably adversarial training [31, 109, 81]. The idea is to inject adversarial examples into the training process
and train the model either on adversarial examples or on a mix of clean and adversarial examples. The approach was successfully applied to large datasets [53], and canbe made more effective by using discrete vector code representations rather than real
number representations of the input [12]. One key drawback of adversarial trainingis that it tends to overfit to the specific attack used at training time. This has been
overcome, at least on small datasets, by adding noise prior to starting the optimizer
for the attack [70]. Another key drawback of adversarial training is that it tends to49

inadvertently learn to do gradient masking rather than to actually move the decisionboundary. This can be largely overcome by training on adversarial examples drawn
from an ensemble of several models [114]. A remaining key drawback of adversarial
training is that it tends to overfit to specific constraint region used to generate theadversarial examples (models trained to resist adversarial examples in a max-normball may not resist adversarial examples based on large modifications to backgroundpixels [29] even if the new adversarial examples do not appear particularly challenging
to a human observer).

50

Chapter 3
Proposed Automatic Detection
System for Adversarial Examples
3.1

Image Transformation for Detecting Adversarial
ExamplesIn this section, we introduce some common methods for creating adversarial examples and defense approaches. We mainly focus on adversarial examples for image
classification. We have to note that adversarial examples are effective to other tasks
such as face recognition, one of image classification [32], natural language processing [27] and malware detection [35]. However those tasks are out of our researchscope. In this work, we explored the robustness of deep neural networks through thevery simple techniques but very effective coordinate transformation. Firstly, we craft
the adversarial examples by using the PGD algorithm [69] on the original training
images. Afterward, we apply our proposed method to those adversarial examples andevaluate our method through by observing the recognition performances. The results
show that our method is very effective for making deep neural networks more robust
against adversarial examples.

Our contributions. Our research shows the following contributions:∙ We investigated and analyzed attack approaches for crafting adversarial exam51

ples. We showed attack approaches along their different strategies to provide
an intuitive overview of these attack methods.

∙ We investigated the modern defense approaches and their variants in adversarial
settings. Base on our investigation, we showed that the defense of a deep neuralnetwork is still challenging and is not yet completely satisfactory. It remains a
rapidly evolving research area and that is also our motivation for this research.

∙ Our proposed approach is successfully applied two types of common datasetsthat are a small-scale dataset (MNIST) and a large-scale dataset (ImageNet).
Our defense method worked well to defeat adversarial examples and in somecases it recovered the deep learning classifier’s performance with high accuracy
rates.

3.1.1 Crafting Adversarial Examples Phase
We consider the white-box targeted attack settings, where the attacker can fully
access into the model type, model architecture, all trainable parameters and theadversary aims to change the classifier’s prediction to some specific target class. The
attackers use available information to identify the feature space where the model isvulnerable or try to find the victim decision boundaries. Then the victim model is
exploited by altering a clean input by using adversarial example methods. To create
adversarial samples that are misclassified by machine learning model, an adversary
with knowledge of the model’s classifier 𝑓 and its trainable parameters. In this work,
we use Projected Gradient Descent [70] method for crafting adversarial examples. We[︁]︁𝑛define classifier function 𝑓 : R → 1...𝑘 that maps image pixel value vectors to a[︁]︁
particular label. Then we assume that function 𝑓 has a loss function 𝐿 : R𝑛 × 1...𝑘 →[︁]︁𝑛
R. For an input image 𝑥 ∈ R and target label 𝑦 ∈ 1...𝑘 , our system aims to solve[︁ ]︁𝑛the following optimization problem: 𝛿 + 𝐿(𝑥, 𝑦) subject to 𝑥 + 𝛿 ∈ 0, 1 , where 𝛿 isa perturbation noise that we add to the original image 𝑥. Whenever finding the best
adversarial candidate, we have to project adversarial to constrain area (𝑥, 𝜖) by using
a projection operator 𝑝𝑟𝑜𝑗𝑒𝑐𝑡(𝑥,𝜖) (). We have to note that this function method would
52

yield the solution for 𝑓 (𝑥) in the case of convex losses, however the neural networks
are non-convex so we end up with an approximation in this case. In this case, we
use the output of the second-to-last layer logits for calculating the gradient insteadof using the output of the Softmax. So our attack method is denoted as algorithm
1. In attacking phase, we set learning rate for crafting adversarial examples is 0.01
and iterative process is 500 times. From the clean images we will create the targeted
output images.

Algorithm 1: Crafting Adversarial Examples Algorithm
input : 𝑥, 𝑦 , 𝑦 , 𝑓 , 𝜖, 𝛼output : 𝑥
parameter: learning rate = 0.01, epochs = 500
𝑡𝑟𝑢𝑒

*

*

1 𝑥 ← 𝑥* // initial adversarial sample
2 𝛿𝑥 ← ⃗0 // initial perturbation factor
3 𝑖𝑡𝑒𝑟 ← 1 // initial iteration counter4||𝛿𝑥 ||∞ < 𝜖 and 𝑓 (𝑥* ) ̸= 𝑦 * and 𝑖𝑡𝑒𝑟 <= 𝑒𝑝𝑜𝑐ℎ𝑠5𝑥* ← 𝑥 + 𝛿 · 𝑠𝑖𝑔𝑛(▽𝐿(𝑦 * |𝑥* ))6
𝑥* ← project(𝑥,𝜖) (𝑥* )7maximize 𝐿(𝑦 * |𝑥* ) respect to ||𝛿𝑥 ||∞8𝛿 ← 𝑐𝑙𝑖𝑝(𝑥* , 𝑥 − 𝜖, 𝑥 + 𝜖)9𝑖𝑡𝑒𝑟 ← 𝑖𝑡𝑒𝑟 + 1

while

10
11

end
return 𝑥do*

3.1.2 Coordinate Transformation
Affine transformations have been widely used in computer vision [44]. Now we define
the range of defense that we want to apply into the adversarial examples for protecting
a deep neural network. In this section, we describe the Coordinate Transformationmanner that is one of an affine transformations, we find the parameter (𝑖* , 𝑗 * , 𝛼)that rotates the adversarial image with degree around the center (see Fig. 3-1) will
make classifier remove the adversarial noises. Formally, the pixel at position (𝑖, 𝑗)
is rotated counterclockwise by multiplying a rotation matrix that has cos and sin
53elements corresponding to the angle 𝛼:

⎡ ⎤ ⎡
⎤ ⎡ ⎤
*
𝑖
cos 𝛼 − sin 𝛼
𝑖
⎣ ⎦=⎣
⎦·⎣ ⎦
𝑗*
sin 𝛼 cos 𝛼
𝑗

(3.1)

⎡ ⎤
⎡ ⎤
𝑖*
𝑖
⎦ have the same magnitude and they are separated by
⎦
⎣
⎣
and
So the vectors
*
𝑗
𝑗
[︁
]︁
an given angle 𝛼. In our research, we set angle 𝛼 ∈ 1, 30 because it is obviously when
we increase angle degree too much the classification capabilities of machine learning
system will be effected and decrease accuracy rate for original images. So the range[︁]︁
1, 30 is suitable for defeating adversarial effectiveness and regain the accuracy on
original images. Our experiments show that we can find the best angle that it defeats
completely the adversarial noise and re-recognize the correct image.

(a) Original Oscilloscope Image(b) Coordinate Transformation with 20 degree
counterclockwise rotation

Figure 3-1: Coordinate Transformation with 20 degree counterclockwise rotation
around the center
When we apply this technique on the adversarial examples that are generated, we
observe that adversarial examples failed to misclassify to coordinate transformation.
For more intuitive understanding, supposing we have original data (𝑥, 𝑦) and model𝑓 . So our prediction label is 𝑦𝑝𝑟𝑒𝑑 . And the loss function 𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) shows us how far
54

𝑦𝑝𝑟𝑒𝑑 is away from 𝑦 . When we create adversarial examples, the purpose is to increase
the loss function 𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) by adding a small adversarial noise to original input 𝑥.
Recall the Projected Gradient Descent algorithm, equation (2.2) will be rewritten as
the equation as bellow:(︂*

𝑥 =𝑥+𝛿·𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 )𝜕𝑥)︂respect to project(𝑥,𝜖) (𝑥* )(3.2)

We aim to solve equation (3.2) by maximizing loss function 𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) insteadof 𝐿(𝑥, 𝑦). The logits (vector of raw prediction) is the output of the deep neuralnetwork before we feed them into the Softmax activation function for normalizing, it
is described as:
logits = 𝑓 (𝑥)(3.3)

𝑦𝑝𝑟𝑒𝑑 = softmax(logits)(3.4)

Softmax function takes an N-dimensional vector logits and transfers it into a vectorof real number in range (0,1) which sum of them is equal to 1:

𝑒logits
softmax(logits) = ∑︀ logits
𝑒

(3.5)

𝑔(𝑥)
And in the fundamental derivative rule, from a given 𝑓 (𝑥) = ℎ(𝑥), we have 𝑓 ′ (𝑥) =∑︀ logits𝑔 ′ (𝑥)ℎ(𝑥)−ℎ′ (𝑥)𝑔(𝑥). So in equation (3.5), 𝑔(𝑥) = 𝑒logits = 𝑒𝑓 (𝑥) and ℎ(𝑥) =
𝑒
=
ℎ(𝑥)2
∑︀ 𝑓 (𝑥)𝑒 . In general, we need to calculate the partial derivate of softmax by pass itback to the previous layer during backpropagation and it will be defined in generalform as:𝑒 𝑥𝑖

𝜕 ∑︀𝑁 𝑒𝑥𝑘
𝜕 softmax(𝑓𝑖 (𝑥)))
𝜕𝑠𝑖
𝑘=1
=
=
𝜕𝑥𝑗
𝜕𝑥𝑗
𝜕𝑥𝑗

(3.6)

Where 𝑠 shorts for softmax function. In this case, we have 𝑔(𝑥) = 𝑒𝑥𝑖 and ℎ(𝑥) =
55

∑︀𝑁

𝑘=1 𝑒

𝑥𝑘

. It is obviously that derivative of exponential 𝑔 ′ (𝑥) = 𝑒′𝑥𝑖 = 𝑒𝑥𝑖 . Meanwhile,ℎ′ (𝑥) = 𝑒𝑥𝑖 when 𝑖 = 𝑗 and ℎ′ (𝑥) = 0 when 𝑖 ̸= 𝑗 .
Let consider case when 𝑖 = 𝑗 :𝑥𝜕 ∑︀𝑁𝑒 𝑖 𝑒𝑥𝑘
𝑘=1

𝜕𝑥𝑗

∑︀𝑁

𝑒𝑥𝑘 − 𝑒𝑥𝑗 𝑒𝑥𝑖
)︁2
𝑁
𝑥
𝑘
𝑘=1 𝑒
(︁∑︀
)︁
𝑁
𝑥𝑘
𝑥𝑗
𝑒 𝑥𝑖
𝑒
−
𝑒
𝑘=1
=
(︁∑︀
)︁2
𝑁
𝑥
𝑘
𝑘=1 𝑒
∑︀
𝑁
𝑥𝑘
− 𝑒𝑥𝑗
𝑒𝑥𝑖
𝑘=1 𝑒
= ∑︀𝑁
∑︀𝑁 𝑥
𝑥𝑘
𝑘
𝑘=1 𝑒
𝑘=1 𝑒
𝑒𝑥𝑖

=

(3.7)

𝑘=1
(︁∑︀

(3.8)

(3.9)

𝑠𝑖 (1 − 𝑠𝑗 )(3.10)

0 − 𝑒𝑥𝑗 𝑒𝑥𝑖
)︁2
(︁∑︀
𝑁
𝑥𝑘
𝑒
𝑘=1

(3.11)

−𝑒𝑥𝑗
𝑒𝑥𝑖
= ∑︀𝑁
× ∑︀𝑁
𝑥𝑘
𝑥𝑘
𝑘=1 𝑒
𝑘=1 𝑒

(3.12)−𝑠𝑗 𝑠𝑖(3.13)=For 𝑖 ̸= 𝑗 , we have:𝑥𝜕 ∑︀𝑁𝑒 𝑖 𝑒𝑥𝑘
𝑘=1

𝜕𝑥𝑗

=

=

We will calculate loss between 𝑦 and 𝑦𝑝𝑟𝑒𝑑 by using cross entropy function thatindicates the distance between what the model believes the output distribution should
be and what the original distribuation really is.

𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) = CrossEntropy(𝑦, 𝑦𝑝𝑟𝑒𝑑 )(3.14)For two discrete probability distributions 𝑦 and 𝑦𝑝𝑟𝑒𝑑 , the cross-entropy function isdefined as:
CrossEntropy(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) = −∑︁𝑦𝑖 log(𝑦𝑝𝑟𝑒𝑑 )𝑖56(3.15)

By calculating partial derivative of function (3.14), we have:𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 )𝜕𝑥𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) 𝜕𝑦𝑝𝑟𝑒𝑑𝜕𝑦𝑝𝑟𝑒𝑑𝜕𝑥𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) 𝜕𝑓 (𝑥)=𝜕𝑓 (𝑥)𝜕𝑥∑︀𝜕(− 𝑖 𝑦𝑖 log(𝑦𝑝𝑟𝑒𝑑 )) 𝜕𝑦𝑝𝑟𝑒𝑑=𝜕𝑦𝑝𝑟𝑒𝑑𝜕𝑥∑︁ 𝜕 log(𝑦𝑝𝑟𝑒𝑑 ) 𝜕𝑦𝑝𝑟𝑒𝑑=−𝑦𝑖𝜕𝑦𝑝𝑟𝑒𝑑𝜕𝑥𝑖∑︁
1 𝜕𝑦𝑝𝑟𝑒𝑑=−𝑦𝑖𝑦𝑝𝑟𝑒𝑑 𝜕𝑥𝑖∑︁1 𝜕 softmax(𝑓 (𝑥))
= −
𝑦𝑖
𝑦𝑝𝑟𝑒𝑑
𝜕𝑥
𝑖

(3.16)=(3.17)(3.18)(3.19)(3.20)(3.21)

From derivative of softmax equations (3.6), (3.7) and (3.11), we have:∑︁𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 )1= −𝑦𝑖 (1 − 𝑦𝑝𝑟𝑒𝑑(𝑖) ) −𝑦𝑘(−𝑦𝑝𝑟𝑒𝑑(𝑘) .𝑦𝑝𝑟𝑒𝑑(𝑖) )𝜕𝑥𝑦𝑝𝑟𝑒𝑑(𝑘)𝑘̸=𝑖∑︁=−𝑦𝑖 (1 − 𝑦𝑝𝑟𝑒𝑑(𝑖) ) +𝑦𝑘 .𝑦𝑝𝑟𝑒𝑑(𝑖)(3.22)(3.23)𝑘̸=𝑖=−𝑦𝑖 + 𝑦𝑖 𝑦𝑝𝑟𝑒𝑑(𝑖) +∑︁(3.24)

𝑦𝑘 .𝑦𝑝𝑟𝑒𝑑(𝑖)

𝑘̸=𝑖

(︃
=

−𝑦𝑝𝑟𝑒𝑑(𝑖) 𝑦𝑖 +

)︃
∑︁

𝑦𝑘

(3.25)− 𝑦𝑖𝑘̸=𝑖Because y is a one hot encoded vector for the true labels, so∑︀𝑘̸=𝑖 𝑦𝑘 = 1. So we have:𝜕𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 )= (𝑦𝑝𝑟𝑒𝑑 − 𝑦)
𝜕𝑥

∑︀

𝑘 𝑦𝑘

= 1 and 𝑦𝑖 +(3.26)

From equations (3.6) and (3.26), it is clear that 𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) is influenced by productof trainable weights and activations when we calculate partial derivative of softmax
by passing it back during backpropagation to find the best adversarial perturbation57𝛿 . For examples, when we have two images with the same label, their activationsin any fixed networks are similar and the weights of the network are unchanged.
Consequently, 𝐿(𝑦, 𝑦𝑝𝑟𝑒𝑑 ) is a constant for any given image 𝑥 with the same class. Thismeans the gradient is highly correlated with true label 𝑦 . Because of this property,
when attacker added a adversarial noise, the classifying 𝑥* becomes a simpler problem
than the original problem of classifying 𝑥, as 𝑥* contains extra information i.e theadded noise. However, with a small change in input data by rotating adversarial
images with a particular angle the system makes a different decision. And in thiscase, we show that deep neural network recognizes the rotated adversarial images as
the true label instead of the targeted label.

3.1.3 Geometric Translation and Combination
In this section, we describe about the Geometric Translation and combination method.
Geometric Translation is one of method of Geometric Transformation that is interpreted as the addition of a constant vector to every point or shifting the origin of
the coordinates. In our work, we applied Geometric Translation on the adversarial
examples by shifting them from the top-left to bottom-right direction (see Fig. 3-2).
We define the shift vector 𝑉 in (𝑣𝑖 ,𝑣𝑗 ) direction:⎡⎤
1 0 𝑣𝑖⎦𝑉 =⎣0 1 𝑣𝑗(3.27)

Before applying this transformation, we flatten adversarial example matrix to column
vector, then our adversarial example 𝑥* can be multiplied by this geometric translation
matrix as:

⎡ ⎤
⎡
⎤ 𝑥𝑖
⎡
⎤
⎢
⎥
1 0 𝑣𝑖 ⎢ ⎥𝑣 + 𝑥𝑖⎦ · ⎢𝑥𝑗 ⎥ = ⎣ 𝑖⎦ = 𝑥**
𝑉 · 𝑥* = ⎣⎣⎦0 1 𝑣𝑗𝑣𝑗 + 𝑥𝑗
158(3.28)

𝑥** is created by geometric translation and we feed it into the deep learning classifier.
Moreover, geometric translation shifts whole image into a particular direction and thiswill leave blank spaces, we will fill in those blank spaces by black pixels and we callthat is the black canvas. After applying the geometric translation into the adversarial
examples, we use the target machine learning classifier to classify new images andobserve the worst case and best case and comparing them with the classification rateon the clean inputs. Combination phase is that we will combine rotation (coordinate(a) Original Oscilloscope Image(b)

Geometric

Translation

with

20

degreefrom top-left corner

Figure 3-2: Geometric Translation with 20 degree from top-lelf cornertransformation) and translation (geometric translation) into end-to-end system for
defeating adversarial examples. Firstly, we applied the geometric translation to theadversarial examples with (𝑣𝑖 , 𝑣𝑗 ) : {0, 1, 2, 3}. By shifting the adversarial examples
in very small distance, we expect to keep the key features of the images in the visible
window and to avoid losing performance of our system. And by using the black
canvas settings, we will ensure that there is no information of input is lost during ourprocess. Secondly, we apply the use the coordinate transformation on the translated
images and feed them into the machine learning classifier to verify our assumption.
We keep the same coordinate transformation setting in this phase with 𝛼 ∈ [0, 30]
in the clockwise direction. The final step, we evaluate the target machine learningclassifier’s performance by using new combination patterns (see Fig. 3-3).
59

(a) Original Oscilloscope Image(b) Combination of Coordinate Transformation and Geometric Translation with 20 degree

Figure 3-3: Combination of Coordinate Transformation and Geometric Translation.First, we shift image from top-left to bottom-right with 20 degree. Then, we
rotate image with 20 degree counterclock wise from center of image

3.1.4 Proposed Defending Method
In this section, we describe about our proposed defending method for Deep Neural
Networks. As we mentioned earlier, we define three main functions include TRNS,
ROT and TR that stand for Geometric Translation, Coordination Transformation
and Combination, respectively. We set a threshold equal to 0.9 for detecting adversarial examples. Our proposed defending method can be illustrated as Algorithm 2.
Recall the loss function that we used for crafting adversarial examples in the previous
section, which is used to measure the inconsistency between predicted label 𝑦 * andactual label 𝑦 . It is a non-negative value, where the robustness of model increases
along with the decrease of the value of loss function. And in our deep learning model,
we use the Softmax classifier and cross-entropy, one of the popular loss function forevaluating softmax classifier. The Softmax classifier that has a probabilistic interpretation that we can get normalized class probabilities. Softmax function is defined as∑︀𝑓𝑖 (𝑥) = 𝑒𝑥𝑖 / 𝑘 𝑒𝑥𝑘 ; where 𝑓𝑖 defines the 𝑖-th element of the vector of class function 𝑓 ,𝑥𝑘 defines the 𝑘 -th element of input samples. It can be interpreted as the normalized60

probability assigned to the correct label 𝑦𝑖 given the image 𝑥𝑖 . Exponentiating thesequantities therefore gives the probabilities, and the division performs the normalization so that the probabilities sum to one. In the probabilistic interpretation, we are
therefore minimizing the negative log likelihood of the correct class, which can beinterpreted as performing Maximum Likelihood Estimation (MLE). A nice featureof this view is that we can now also interpret the regularization term 𝑅(𝑊 ) in thefull loss function as coming from a Gaussian prior over the weight matrix 𝑊 , where
instead of MLE we are performing the Maximum a posteriori (MAP) estimation.
The full details of this derivation are beyond the scope of this work so we mention
these interpretations to make it more clear about Softmax classifier and the output
of Softmax classifier is probabilities of all labels. In our proposed methods, we use
Softmax function to compute the probabilities of primary input and modified input
and compare them with threshold to make a decision which one is benign or not. In
algorithm 2, we define 𝑏𝑎𝑠𝑒𝑃 𝑟𝑜𝑏𝑠 is probability of primary input 𝑥, after applying
our methods, we get the modified input and measure its probability 𝑝𝑟𝑜𝑏. If |𝑝𝑟𝑜𝑏 -𝑏𝑎𝑠𝑒𝑃 𝑟𝑜𝑏| ≥ 0.9, it means probability of input dramatically drops so we can point
out it is adversarial example, whereas it is benign input.

3.1.5 Experimental setup

Datasets. We consider the ImageNet dataset [96], which is a very large database
designed for use in visual object recognition research and MNIST dataset [57], which
is very common dataset for evaluating research works related to neural networks. The
original ImageNet includes more than 14 millions images in 20,000 categories with atypical category, such as “computer mouse” or “vending machine”, consisting of several
hundred images. MNIST is a database of handwritten digits that is commonly usedin various image processing systems. It includes 60,000 training images and 10,000
testing images that fit into a 28 × 28 pixel and grayscale levels.

Deep learning models. The machine learning models that we use are Google

Inception V3 [108] and LeNet [55]. Google Inception V3 is a widely-used image
classification model that has been shown to attain very high accuracy on the ImageNet
61

Algorithm 2: Proposed Defending Algorithm
input : 𝑥, 𝑦, 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑, 𝑅𝑂𝑇 , 𝑇 𝑅𝐴𝑁 𝑆 , 𝑇 𝑅
output : 1: adversarial input, 0: clean input
1 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 ← 0.9; 𝑏𝑎𝑠𝑒𝑃 𝑟𝑜𝑏 ← 𝑃 (𝑦𝑖 |𝑥𝑖 ) = ∑︀𝑒
2
3
4
5
6
7

if ROT then𝑠𝑡𝑎𝑟𝑡𝑃 𝑜𝑖𝑛𝑡 ← 0 ; 𝑒𝑛𝑑𝑃 𝑜𝑖𝑛𝑡 ← 30
for 𝑖 in range(𝑠𝑡𝑎𝑟𝑡𝑃 𝑜𝑖𝑛𝑡,𝑒𝑛𝑑𝑃 𝑜𝑖𝑛𝑡) do

𝑓𝑦𝑖

𝑓𝑗
𝑗𝑒

𝑎𝑛𝑔𝑙𝑒 ← i * 𝜋/180
𝑥𝑅𝑂𝑇 ← 𝑅𝑂𝑇 (𝑥, 𝑎𝑛𝑔𝑙𝑒)

𝑓𝑦

𝑝𝑟𝑜𝑏 ← 𝑃 (𝑦𝑖 |𝑥𝑅𝑂𝑇 ) = ∑︀𝑒 𝑒𝑖𝑓𝑖

end
9 else if TRNS then
10
𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡 ← 0; 𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡 ← 3
11for 𝑖 in range(𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡,𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡) do
12for 𝑗 in range(𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡,𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡) do
𝑖

8

13
14
15

𝑣𝑖 ← 𝑖𝑣𝑗 ← 𝑗𝑥𝑇 𝑅𝑁 𝑆 ← 𝑇 𝑅𝑁 𝑆(𝑥, 𝑣𝑖 , 𝑣𝑗 )16

𝑝𝑟𝑜𝑏 ← 𝑃 (𝑦𝑖 |𝑥𝑇 𝑅𝑁 𝑆 ) = ∑︀𝑒 𝑒𝑖𝑓𝑖

𝑓𝑦

end
18
end
19 else20
𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡 ← 0; 𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡 ← 3
21
for 𝑖 in range(𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡,𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡) do
22
for 𝑗 in range(𝑚𝑖𝑛𝑃 𝑜𝑖𝑛𝑡,𝑚𝑎𝑥𝑃 𝑜𝑖𝑛𝑡) do
𝑖

17

23
24
25
26
27
28
29
30

𝑣𝑖 ← 𝑖𝑣𝑗 ← 𝑗𝑠𝑡𝑎𝑟𝑡𝑃 𝑜𝑖𝑛𝑡 ← 0 ; 𝑒𝑛𝑑𝑃 𝑜𝑖𝑛𝑡 ← 30
𝑥𝑇 𝑅𝑁 𝑆 ← 𝑇 𝑅𝑁 𝑆(𝑥, 𝑣𝑖 , 𝑣𝑗 )𝑞 in range(𝑠𝑡𝑎𝑟𝑡𝑃 𝑜𝑖𝑛𝑡,𝑒𝑛𝑑𝑃 𝑜𝑖𝑛𝑡)
𝑎𝑛𝑔𝑙𝑒 ← 𝑞 * 𝜋/180
𝑥𝑇 𝑅 ← 𝑅𝑂𝑇 (𝑥𝑇 𝑅𝑁 𝑆 , 𝑎𝑛𝑔𝑙𝑒)

for

𝑓𝑦

𝑝𝑟𝑜𝑏 ← 𝑃 (𝑦𝑖 |𝑥𝑇 𝑅 ) = ∑︀𝑒 𝑒𝑖𝑓𝑖

end
32
end
33
end
34 end35 if |𝑝𝑟𝑜𝑏 - 𝑏𝑎𝑠𝑒𝑃 𝑟𝑜𝑏| ≥ 0.9 then36
return 1
37 else38
return 0
39 end

𝑖

31

62

do

dataset. The model itself is made up of symmetric and asymmetric building blocks,
including convolutional layers, average and max pooling, concats, dropouts and fully
connected layers. Batchnorm is used extensively throughout the model and applied
to activation inputs while loss is computed by Softmax layer. For this model, Google
customized the ImageNet dataset that includes 1,331,167 images which are divided
into training and evaluation sets containing 1,281,167 and 50,000 images, respectively.
Its architecture that we use for crafting adversarial examples is as in table 3.1. Weemphasize that we use the logits value in the last adjacency layer for compute theloss function 𝐿(𝑥, 𝑦).
Table 3.1: Google Inception Architecture [108]
type
conv
conv
conv padded
pool
conv
conv
conv
3 × Inception
5 × Inception
2 × Inception
poollinearsoftmax

patch
size/stride
or remarks
3 × 3/2
3 × 3/1
3 × 3/1
3 × 3/2
3 × 3/1
3 × 3/2
3 × 3/1
Inceptionfilters
Inceptionfilters
Inceptionfilters
8×8
logitsclassifier

input size
299 × 299 × 3
149 × 149 × 32
147 × 147 × 32
147 × 147 × 64
73 × 73 × 64
71 × 71 × 80
35 × 35 × 192
35 × 35 × 288
17 × 17 × 768
8 × 8 × 1280
8 × 8 × 2048
1 × 1 × 2048
1 × 1 × 1000

LeNet [55] includes 7 deep layers not counting input layer that are convolutional
layers and pooling operation. LeNet architecture that we use for crafting adversarial
examples is as in table 3.2. This original model worked on the 32 × 32 input size,
however we customized this model in order to accept to MNIST dataset of 28 × 28
input sizes.
63

Table 3.2: LeNet architecture [56]
type
conv
pool
conv
pool
conv
full connected
softmax

kernelsize/featuremaps
5 × 5/6
2 × 2/6
5 × 5/16
2 × 2/16
5 × 5/120
logitsclassifier

input size
32 × 32 × 1
28 × 28 × 6
14 × 14 × 6
10 × 10 × 16
5 × 5 × 16
1 × 1 × 84
1 × 1 × 10

3.1.6 Results
For evaluating our method, we consider 30 images from three categories such as Computer Mouse (C-Mouse), Computer Keyboard (Keyboard) and Vending Machine (VMachine) in ImageNet with 10 images per each category. By considering both original
images and adversarial images, the total number of samples is 60 images. Because ofimage copyright issues, for visualization we used three images of Keyboard, C-Mouse
and V-Machine that were captured by ourself. Meanwhile, for MNIST dataset, we
randomly select ten digit-0 images for crafting adversarial examples with targetedclass is digit-4. The first, we use the PGD method for creating adversarial images
in the white-box setting. Next, we apply the coordinate transformation, geometrictranslation and combination of these two transformations for those adversarial images and feed them into again the machine learning classifiers for classification. The
results show that our proposed method is very effective for removing the adversarial
noise and recovering the acceptable classification accuracy rate.
For attacking Google Inception model, we use 30 images from ImageNet and three
our own images to craft the adversarial examples toward to ostrich label. After applying geometric translations, coordinate transformations and combination steps, weobserve the effects of adversarial examples on the classifier and recognize that those
adversarial perturbations are mostly removed and in some particular cases our method
recover the performance of classifier with right answers. The geometric translation64

settings are 𝑣𝑖 ∈ [0, 3] and 𝑣𝑗 ∈ [0, 3]. That means we test 15 difference transformations for each image. In table 3.3 we only record the worst and the best case.
In this table, the column “Primary” shows the classification’s results on the original
images (Clean) and adversarial images (Adv) before applying our method. The next
two columns “Original class” and “Targeted class” show the results after applying our
method. “Original class” column shows the worst-case and the best-case that classifierclassify them as “Original” and the same meaning to “Targeted class” column. From
this table, we illustrate that geometric translation mostly defeat adversarial effects
when it makes confidence rate dramatically decrease. For example, in “Keyboard”
case, the confidence recognition rate for “ostrich” drops from 99.9% to 5.07%.
Table 3.3: Geometric Translation results on our own images
Image
Keyboard
C-Mouse
V-Machine

Primary
Clean Adv
72.4% 99.9%
86.1% 100%
77.9% 99.8%

Original class
MinMax
15.69% 62.01%
3.01%
15.43%
3.74%
66.71%

Targeted class
Min
Max
0.02% 5.07%
0.1%
10.4%
0.18% 17.2%

The performance of the coordinate transformation is very impressive that is shownin table 3.4. We observe that deep learning classifier are correctly recognize theoriginal class at the best-case is 92.67%, it is even better than when our method has
not been applied and the maximum probability for targeted recognition is only 3.67%.

Table 3.4: Coordinate Transformation results on our own images
Image
Keyboard
C-Mouse
V-Machine

Primary
Clean Adv
72.4% 99.9%
86.1% 100%
77.9% 99.8%

Original class
Min
Max
28.27% 56.5%
5.81%
92.67%
2.62%
55.04%

Targeted class
Min
Max
0.01% 0.22%
0.0%
2.16%
0.01% 3.67%

In combination setting, firstly, we perform the geometric translation on the adversarial images, after that we apply coordinate transformation in the range [0,30].
In geometric translation step, we also consider 15 cases for each image 𝑣𝑖 ∈ [0, 3] and65

𝑣𝑗 ∈ [0, 3]. It is obvious that it excludes (𝑣𝑖 , 𝑣𝑗 ) = (0, 0). The result is shown intable 3.5. For combination setting, we observe that it works well on all settings andthe worst record with “V-Machine” image is only 2.14%. In “Keyboard”, the minimumof classification rate of targeted class all is 0.01%, and the worst case is 0.23%. Meanwhile, the best case for original label is very potential at 65.82%. For “C-Mouse”, we
observe that the best classification rate for original label reaches 93.86%, and classification accuracy rate for adversarial label is less than 1.83. The results confirm thatthe combination method gets the best effect in our experiment.
In table 3.6 summaries performance on three our own images by using TRNS, ROT
and TR with Google Inception. We observe that combination method overwhelms
and wins in most categories, there is only one exceptional case that TRNS gets thehighest performance with clean vending machine image at 66.71%.
In table 3.7 shows the results on 60 images (includes 30 benign and 30 adversarial
images) from the ImageNet dataset. We observe min, max, median and variance of
probability for recognizing true label and adversarial label with original and adversarial inputs. In Geometric Transformation (TRNS), we analyze 60 × 15 = 900 different
cases. When using TRNS, with original inputs, the median of recognition probabilities for true label that is gave by machine learning system reaches to 84.43%. Whenthe inputs are adversarial, TRNS answered with mean of recognition probabilities
for adversarial label is only 0.6% confidence. In Coordinate Transformation (ROT),
we analyze 60 × 30 = 1, 800 different cases. ROT method also performed perfectly
on both original and adversarial samples. And in Combination (TR), we work with60 × 15 × 30 = 27, 000 different cases. We observed that TR method outperformed
almost cases to defeat adversarial noises and regain the true label. The median ofrecognition probabilities for adversarial label when using TR method is only 0.04%
with adversarial inputs. Meanwhile, with original inputs the median of probabilities for true label is 74.65%. It is obviously that our system can completely defeat
adversarial perturbation and regain true classification with acceptable rate.
Intuitively, Fig A-1 shows the coordinate transformation and geometric translationon the adversarial perturbation for three our Keyboard, C-Mouse and V-Machine66

Table 3.5: Combination results on our own images
Image

Keyboard

C-Mouse

V-Machine

Translation
𝑣𝑖 ,𝑣𝑗
(0,1)
(0,2)
(0,3)
(1,0)
(1,1)
(1,2)
(1,3)
(2,0)
(2,1)
(2,2)
(2,3)
(3,0)
(3,1)
(3,2)
(3,3)
(0,1)
(0,2)
(0,3)
(1,0)
(1,1)
(1,2)
(1,3)
(2,0)
(2,1)
(2,2)
(2,3)
(3,0)
(3,1)
(3,2)
(3,3)
(0,1)
(0,2)
(0,3)
(1,0)
(1,1)
(1,2)
(1,3)
(2,0)
(2,1)
(2,2)
(2,3)
(3,0)
(3,1)
(3,2)
(3,3)

Original class
Min
Max
26.36% 58.44%
26.76% 65.82%
29.26% 61.13%
27.78% 60.26%
26.16% 57.33%
28.92% 64.2%
28.53% 61.69%
23.85% 57.82%
26.19% 62.14%
27.3%
64.44%
30.79% 61.67%
28.2%
61.11%
26.78% 58.1%
28.29% 55.5%
29.6%
59.83%
6.41%
90.95%
5.98%
87.82%
6.32%
69.83%
7.92%
88.63%
6.55%
90.91%
6.23%
93.86%
4.95%
77.05%
6.39%
86.09%
6.95%
90.56%
5.35%
83.72%
4.15%
89.0%
11.56% 82.31%
3.43%
79.85%
3.85%
80.86%
3.96%
90.01%
2.55%
51.44%
1.53%
48.13%
4.53%
47.44%
2.99%
55.62%
1.74%
61.7%
2.3%
62.24%
6.48%
61.21%
1.9%
53.28%
1.16%
55.63%
1.22%
54.3%
1.42%
57.37%
0.97%
50.23%
0.69%
67 49.96%
1.36%
48.41%
0.84%
63.48%

Targeted class
Min
Max
0.01% 0.19%
0.01% 0.07%
0.01% 0.05%
0.01% 0.22%
0.01% 0.16%
0.01% 0.06%
0.01% 0.04%
0.01% 0.25%
0.01% 0.18%
0.01% 0.07%
0.01% 0.05%
0.01% 0.23%
0.01% 0.16%
0.01% 0.08%
0.01% 0.06%
0.0%
0.65%
0.0%
0.47%
0.0%
0.21%
0.0%
1.83%
0.0%
0.42%
0.0%
0.41%
0.0%
0.15%
0.0%
0.7%
0.0%
0.29%
0.0%
0.16%
0.0%
0.12%
0.0%
0.36%
0.0%
0.22%
0.0%
0.12%
0.0%
0.12%
0.01% 2.14%
0.01% 0.08%
0.01% 0.44%
0.01% 1.09%
0.01% 1.21%
0.01% 0.78%
0.0%
0.39%
0.01% 1.66%
0.01% 1.2%
0.01% 0.63%
0.01% 0.33%
0.0%
0.75%
0.01% 0.54%
0.01% 0.25%
0.01% 0.33%

Table 3.6: Summary performance on our own images
Image
Keyboard
C-Mouse
V-Machine

TRNS
Clean
Adv
62.01% 5.07%
15.43% 10.4%
66.71% 17.2%

ROT
Clean
Adv
56.5%
0.22%
92.67% 2.16%
55.04% 3.67%

Combination
Clean
Adv
65.82% 0.07%
93.86% 0.41%
63.48% 0.33%

Table 3.7: Summary performance on ImageNet dataset
Method

Cases

TRNS

900

ROT
TR

Input

Org
Adv
Org
1,800
Adv
Org
27,000Adv

Probability for True LabelMin
Max
Median Var
7.98% 99.51% 84.43% 0.13%
0.22% 95.69% 38.58% 2.21%
1.56% 99.72% 75.71% 9.60%
0.71% 99.69% 53.15% 3.83%
0.38% 99.76% 74.65% 8.76%
0.15% 99.84% 56.27% 5.84%

Probability for Adv LabelMin
Max
Median Var
0.0% 0.06% 0.01% 0.0%
0.01% 88.15% 0.60% 3.95%
0.0% 0.07% 0.01% 0.0%
0.0% 28.2% 0.05% 0.0%
0.0% 0.11% 0.01% 0.0%
0.0% 50.14% 0.04% 0.0%

images. In Fig A-1, the first column shows the results when we use the coordinatetransformation and second one is for geometric translation. In the figures in the firstcolumn, we define that the red line illustrates the probabilities that machine learning
recognizes the input as adversarial ostrich and the green line shows for the trueclass. We observe that with a very small coordinate transformation, the adversarial
ostrich is completely defeated. In the second column of the figures, the adversarial
ostrich is also defeated by geometric translation, meanwhile this method attains highprobabilities of true classification.
For MNIST dataset, we randomly extract ten digit-0 from the test set for creating
adversarial examples with the target of digit-4. The adversarial digits completely
make the LeNe’s classifier misclassify digit-0 as digit-4. With the same experimental settings as ImageNet, we also apply TRNS, ROT and combination for defeating
adversarial digits and recovering the clean digits.
We iteratively use the coordinate transformation on the adversarial examples with𝛼 ∈ [0, 30] and recognize that the perturbations are defeated by this method with
only small value of 𝛼.
For MNIST dataset, we also recognize that the combination method is more pow68

erful and effective for defeating adversarial perturbation than single use of TRNS or
ROT. The combination results are shown in table 3.8.
Table 3.8: Summary performance on MNIST
Index
4457
5584
1438
1558
7031
9962
297
5259
4070
4515

TRNS
Clean
Adv
99.96% 0.71%
99.13% 3.22%
96.65% 4.01%
89.63% 9.74%
85.22% 1.15%
99.36% 6.93%
98.95% 14.22%
99.97% 0.23%
99.96% 0.40%
99.99% 0.53%

ROT
Clean
Adv
93.69% 0.33%
96.1%
0.01%
83.52% 0.27%
49.57% 4.96%
94.05% 0.12%
97.47% 0.09%
96.38% 0.09%
99.24% 0.05%
99.09% 0.01%
94.65% 1.04%

Combination
Clean
Adv
99.96% 0.00%
99.13% 0.14%
97.00% 0.13%
89.63% 1.74%
85.22% 0.06%
99.36% 0.05%
99.56% 0.09%
99.97% 0.00%
99.96% 0.00%
99.99% 0.00%

Fig A-2 shows the effectiveness of coordinate transformation (ROT) on adversarial
examples. The sub figure besides of each picture shows the change in the probability oftrue label (digit 0) with green curve line and adversarial label (digit 4) with red curveline. The best case is in digit 0 with index 5584. In this best case, the probabilityof true label linearly increases with angle of rotation, meanwhile the probability ofadversarial label decreases dramatically and reaches to 0 when angle of rotation isincreased to 5 degree. The worst case is in digit 0 with index 1438. In this worst case,although the probability of adversarial label dramatically drops from nearly 100% tounder 10% but it still remains and fluctuates around 10%.

3.1.7 Summary for Image Transformation for Detecting Adversarial Examples
Adversarial attack so far is a very serious problem for security and privacy on machinelearning systems. This research work provides evidence that the deep neural networks
can be made more robust to adversarial attacks by giving TRNS, ROT and their
combination transformations to input images. As our theory and experiments, we now
can develop powerful defense methods for adversarial problem. Our experiments on
69

ImageNet and MNIST datasets have not reached the best on all of cases. However, ourresults already show that our approach lead to significant increase in the robustnessof deep neural network. Our best performance case (in table. 3.7) on the ImageNet
images shows that it can reduce the classification rate for adversarial example from
99.9% to 0.00%. And we also believe that our findings will be further explored in our
future work.

70

3.2

Image Filters for Detecting Adversarial Examples

In this section, we introduce new techniques to overcome adversarial examples byusing Image Filters. Our proposed system can automatically detect and classify
adversarial samples and legitimate samples. We assume that most of adversarial perturbations are created in high frequencies of image because adversarial examples are
created only respect to loss(𝑥𝑎𝑑𝑣 ,𝑦 𝑎𝑑𝑣 ), so there are many properly adversarial noises
exist in high frequencies. Base on this assumption, we focus on the way how to reducehigh frequency adversarial noises and still keep other high frequency benign features.
For proving our hypothesis right, first we put a low pass filter layer between adversarial example and target classifier. We observed that the probability of targeted classfrom classifier dropped significantly to near zero, meanwhile we regained the recognition results for the primary class. We will also demonstrate the correctness of these
implementations into theory proof in Sec 3.2.1. Based on the previous observation,
we proposed the new end-to-end system for automatically detecting adversarial bydesign a sieve layer between input and deep neural network to stick suspicious noises.
In parallel with that process, we feed the un-sieved input to classifier and mark thehighest confidence class as an anchor. We compare the probability of anchor andsieved input from classifier based on a specified fixed threshold to make a decisionthat one is adversarial or benign. The our key idea is depicted in Fig. 3-4.
The main contributions of this research are as follows:∙ We investigated and analyzed attack approaches for crafting adversarial examples. We showed attack approaches along their different strategies to provide
an intuitive overview of these attack methods.

∙ We investigated the modern defense approaches and their variants in adversarial
settings. We assume most of adversarial perturbations are created in high frequencies. After implementing many experiments respect to theory framework,
we confidently confirm our hypothesis right.
71

Figure 3-4: Our automated detection system for adversarial examples by using image
filters∙ We created new automated detection method for adversarial examples based ona thorough analysis of observations obtained from many experimentation andtheoretical framework. Our approach is different from other previous researchworks that they usually only have experimental steps based on the original
hypothesis. Our proposed approach is successfully applied two types of common datasets that are a small-scale dataset (MNIST) and a large-scale dataset(ImageNet). Our defense method worked well to classify adversarial examples
and legitimate samples. Moreover, in some cases it recovered the deep learningmodel’s classification with high accuracy rates.

3.2.1 Detection phaseWe create new benchmark dataset for our detection system by combination of benign
images and adversarial images that we created in attack phase. We assume thatadversarial noises are high frequencies feature on images, so we will consider the way
to remove them from high frequency domain on images while still keep all features in
low frequency area. There are some common algorithms that are applied to reduce
the noises from images for further processing such as classification. In this work, we72

investigate two most known filters in image denoising field are linear and non-linear
filters. Let consider an example, by constructing a new array that has the same sizeas the specified image. Fill each location of this new array with a weighted sumof the pixel values from the locations surrounding the corresponding location in theimage, using the same set of weights each time. The result of this procedure is shiftinvariant — meaning that the value of the output depends on the pattern in an image
neighbourhood, rather than the position of the neighbourhood and linear meaningthat the output for the sum of two images is the same as the sum of the outputs
obtained for the images separately. The procedure itself is known as linear filtering.
This process helps us to smooth noises in images. And one of famous linear filter is
Gaussian filter that is defined as equation:𝐺𝜎 (𝑖, 𝑗) =1 − 𝑖2 +𝑗22
𝑒 2𝜎
2𝜋𝜎 2

(3.29)

Where 𝑖, 𝑗 is denoted as input’s coordinate signal and 𝜎 is standard deviation of theGaussian distribution. And an alternative approach to remove noises is to think of afilter as a statistical estimator. In particular, the goal here is to estimate the actual
image value at a pixel, in the presence of noisy measurements. This view leads us toa class of filters that are hard to analyze, but can be extremely useful. Smoothing animage with a symmetric Gaussian kernel replaces a pixel with some weighted averageof its neighbours. If an image has been corrupted with stationary additive zeromean Gaussian noise, then this weighted average gives a reasonable estimate of theoriginal value of the pixel. The expected noise response is zero, and the estimate has
better behaviour in terms of spatial frequency than a simple average. However, if theimage noise is not stationary additive Gaussian noise, difficulties arise. In particular,consider a region of the image which has a constant dark value and there is a single
bright pixel due to noise — smoothing with a Gaussian will leave a smooth, Gaussianlike, bright bump centered on this pixel. The problem here is that a weighted averagecan be arbitrarily badly affected by very large noise values. Thus, we can make thebright bump arbitrarily bright by making the bright pixel arbitrarily bright perhaps as73

result of a transient error in reading a memory element. Estimators that do not havethis most undesirable property are often known as robust estimates. The best known
robust estimator involves estimating the mean of a set of values using its median.
A median filter is specified by giving some form of neighbourhood shape (which cansignificantly affect the behaviour of the filter). This neighbourhood is passed over theimage as in convolution, but instead of taking a weighted sum of elements within theneighbourhood, we take the median. If we write the neighbourhood centered at (𝑖,𝑗)so the filter can be described by:

𝑥𝑖𝑗 = 𝑚𝑒𝑑𝑖𝑎𝑛(𝑋𝑢𝑣 |𝑋𝑢𝑣 ∈ N𝑖𝑗 )(3.30)

Where 𝑋𝑢𝑣 is denoted as the neighbourhood points of 𝑥𝑖𝑗 . By smoothing pixels inimage, we can leverage adversarial noises if they are exist. In case without adversarial
noises, smoothing pixels does not affect the input image quality too much so target’sclassifier still recognizes the right label. We name this process is sieve process (notedby green arrow in Fig. 3-4).

Our proposed detection system has two parallel processes include sieve process andanchor process. Sieve process will leverage the high frequencies in input processing
while anchor process will transfer input directly to machine learning model. The
probability of the highest confidence class from machine learning model will be sticked
as anchor, and we use this anchor class to tracking the oscillation of the class similar
to anchor class on the sieve process. If the differentiation of probability 𝑝 of anchor
and sieve is greater than a fixed Θ threshold, our system will confidently point out
it as adversarial and vice versa. Our system is defined in algorithm 3. Where: 𝜅
is denoted for kernel size, 𝑓 is machine learning function to produce probabilities ofpredicted class, 𝑠 is sieve function. We denote the sieve function created based on
Gaussian filter is Detection System based on Gaussian - DSG and another one is
Detection System based on Median - DSM.
74

Algorithm 3: Automated Detection System of Adversarial Examples withHigh Frequency Sieve
input : 𝑋, Θ, 𝑠, 𝑓
output : 0, 1// 0: benign; 1: adversarial
parameter: 𝜅 = [(3 × 3); (5 × 5)]
1 for 𝑥 in 𝑋 do

2
3
4
5
6
7
8
9
10
11

𝑎𝑛𝑐ℎ𝑜𝑟𝑥 ← 𝑥𝑠𝑖𝑒𝑣𝑒𝑥 ← 𝑥𝑠𝑖𝑒𝑣𝑒𝑥 ← 𝑠(𝑠𝑖𝑒𝑣𝑒𝑥 , 𝜅, 𝜎)𝑝(𝑎𝑛𝑐ℎ𝑜𝑟𝑦 ) ← 𝑓 (𝑎𝑛𝑐ℎ𝑜𝑟𝑥 )𝑝(𝑠𝑖𝑒𝑣𝑒𝑦 ) ← 𝑓 (𝑠𝑖𝑒𝑣𝑒𝑥 |𝑎𝑛𝑐ℎ𝑜𝑟𝑦 )diff(𝑝(𝑎𝑛𝑐ℎ𝑜𝑟𝑦 ), 𝑝(𝑠𝑖𝑒𝑣𝑒𝑦 )) > Θ1

if
returnelse
return 0end
12 endthen

3.2.2 Datasets
In this work, we consider two common benchmark datasets for classification task
include MNIST and ImageNet.

Setup for MNISTMNIST dataset [57] includes 70,000 gray images of hand-written digits from 0 to 9.
It is separated into two parts including 60,000 training images and 10,000 testing
images. Each image in MNIST is 28 × 28 pixels with each pixel is encoded by 8-bit
grayscale. We randomly extract 200 images of digit “0" from 10,000 testing images.
From each of 200 images of digit “0", we create nine different adversarial images target
to the rest digit (from 1 to 9). Finally we create new benchmark dataset include 2,000
images (200 benign images and 1,800 adversarial images).

Setup for ImageNet
We consider the ImageNet dataset [96] that is a very large database designed foruse in visual object recognition research. The original ImageNet includes more than75

14 millions images in 20,000 categories with a typical category, such as “computer
mouse” or “vending machine”, consisting of several hundred images. The machinelearning model that we use is Google Inception V3 [108] that was trained with 1,000
common categories ImageNet. We randomly select 1,000 testing images exclude imageof “ostrich”. This selection of targeted class does not compromise the generality of
our system. By applying FGSM and PGD method to craft adversarial images target
to “ostrich”, we generate new 2,000 adversarial images. We combine them together toform new benchmark repository including 3,000 images for our experiment.

3.2.3 Implementation
Adversarial examples recently attracted a lot of interest from researchers however
there is still no public benchmark dataset for evaluating the robustness of defense
system. So the purpose of attack phase is to create a new benchmark dataset that
evaluates the detection capabilities of our detection system.
In MNIST dataset, from 200 random images of digit “0”, we use FGSM method
to craft adversarial images with targeted class from 1 to 9. The number of iteration(epochs) in FGSM is 1,000. Afterward, we combinate them to form a benchmark
dataset for evaluating our detection system. The proposed detection system knows
true label of input and we only provide input after that our system automaticallyprocesses and returns decision that the input is adversarial or benign.
In ImageNet dataset, from 360 random testing images without “ostrich”, we use
FGSM, PGD, CW and EAD to craft adversarial images with targeted class is “ostrich”.
The number of iteration (epochs) in FGSM, PGD, CW and EAD is 500 times. In
Fig. 3-5, we show five samples from 390 random picked images that we use for crafting
adversarial examples. In the first row, there are original images (or we usually callas benign images) includes dish, hammerhead, mosque, oscilloscope and parachute.
The probability row shows the highest probability of class in class name in brackets.
We finally create 1,800 images including 360 original images and 1,440 adversarial
images.
In sieve process, we set kernel sizes for Gaussian and Median filter are (3 × 3) and76

(5 × 5). We observe the change of probability of anchor before and after applying
filters to make a decision base on a given threshold.

Figure 3-5: Attack Phase Samples.

3.2.4 Results
We compare our result with Xu et al. [120]. Our system is more convenient that Xu’ssystem in either detection accuracy and easy-to-setup, our system consistently uses a
fixed threshold while Xu’s system has to adapt a variety of threshold values. We also
report our system’s performance by using F1-scores metric. Our detection system
based on Gaussian named as DSG and another one is Detection System based onMedian named as DSM.
We observe and analyze a typical case with image of oscilloscope. From benign
oscilloscope image with probability is around 99.7%, we created two adversarial ostrich by using FGSM and PGD method (Fig. 3-5). Afterward, we used DSG and
77

DSM function for sieving adversarial ostrich noises and regain oscilloscope features.
Fig. 3-6 has expressed probabilities change dramatically on targeted ostrich and legitimate oscilloscope when processed by DSG function. This observation confirms ourassumption that adversarial noises are high frequencies and by adapting low pass filterin our model, we can proposed a powerful detection model for adversarial examples.

(a) Observation on probabilities of Oscillo-(b) Observation on probabilities of Oscillo-

scope label

scope label

Figure 3-6: Adversarial ostrich image (true class: Oscilloscope) suffers to our sieve
process
Adversarial Detection results on MNIST dataset are summated in Table 3.9. In
Table 3.9, “-” means we do not have information from the other research work. Although the number of images in comparison is the same, the way we create test set
is more challenge than Xu et al. [120]. Xu kept a balanced dataset by creating 1,000
legitimate images and 1,000 adversarial examples while we created 1,800 adversarial
images from 200 legitimate inputs so our model had to cope with imbalanced [107]dataset in our experiment. Our detection rates are very competitive and performed
better than Xu’s work. Besides that, our automatic detection system used a fixed
threshold for all settings while Xu’s work used a variety of threshold respect to settings. In the implementation results on MNIST dataset, the DSG method achieves
the best performance on detection rate at 99.9% while Xu’s the best result is only at
98.2%.
In ImageNet dataset, we observed our detection rates reached the highest score forcomparing to Xu’s work. In Table 3.10 also shows that the number of files we used inthis implementation is larger than Xu et al. and we still keep our benchmark dataset
is imbalance for evaluating our proposed model. Our detection rate achieved around78

Table 3.9: Detection Rate on MNIST dataset

No. Files
Threshold
True Positive
True Negative
False Positive
False Negative
Accuracy
Precision
RecallF1 score

Our Method
DSG DSM2,000 2,0000.1
0.11799 1796
198
195
2
5
1
4
0.999 0.996
0.999 0.997
0.999 0.998
0.999 0.998

Bit-Depth
2,000
0.00050.903-

Xu et al. [120]Smoothing Best-Joint
2,000
2,000
0.00290.0029
0.868
0.982
-

Table 3.10: Detection Rate on ImageNet dataset

No. Files
Threshold
True Positive
True Negative
False Positive
False Negative
Accuracy
Precision
RecallF1 score

Our Method
DSG DSM1,800 1,800
0.9
0.9
1,380 1,433
356
350
4
10
60
7
0.964 0.991
0.997 0.993
0.958 0.995
0.997 0.994

Xu et al. [120]Bit-Depth Smoothing Best-Joint
1,800
1,800
1,800
1.4417
1.1472
1.2128
0.751
0.816
0.859
-

95.8% with DSG and 99.5% with DSM. DSM worked perfectly to detect all adversarial
examples while Xu’s the best performance is only 85.9% detection accuracy rate. We
use a fix threshold at 0.9 in both DSM and DSG methods. We recognize that the
DSM works better than DSG to detect adversarial image but worse to recognize theoriginal images. From these observation, we find out that the DSM is very strong
method to remove the features on the high-frequency and in some case it also remove
the importance features that makes false positive is higher than DSG method.
79

3.2.5 Summary for Denoising Detection System

In this section, we investigated the high frequencies in the adversarial examples. We
assume that adversarial noises are high frequencies feature on images, so we willconsider the way to remove them from high frequency domain on images while still
keep all features in low frequency area. There are some common algorithms thatare applied to reduce the noises from images for further processing such as classification. In this work, we investigate two most known filters in image denoising field
are linear (Gaussian filter) and non-linear filter (Median filter). Based on assumption
and theory framework, we demonstrated the effectiveness of low pass filter in remove
high frequency adversarial noises. From this observation we proposed a automateddetection system for adversarial examples.
On MNIST dataset, we use FGSM to create adversarial examples and our detection system achieved detection accuracy rate up to 99.9% with DSG method and99.8% with DSM method.
On ImageNet dataset, we use FGSM, PGD, CW and EAD methods to create
adversarial examples. Our detection system also reached to 99.5% detection accuracy
rate with DSM method and 95.8% with DSG method.
From our experimental results, we recognized that Median filter is better thanGaussian filter for defeating adversarial noises although Median filter will take moretime than Gaussian in computation. In evaluation of our system, we setup the new
benchmark datasets are more challenge than [120, 62] when they used images fromtraining set for evaluating rather than testing set, while we use testing images for
implementation. We also challenged our model by keeping imbalanced datasets andour detection system still reached state-of-the-art performance. Another important
contribution in this work is that we not only defeated adversarial noises, but also oursystems regained the legitimate class from effectiveness of adversarial examples.
80

3.3

Geometric Transformation and Denoiser for identifying Adversarial Examples

In this section, we describe the state-of-the-art automatic detection system that detects and distinguishes between adversarial and legitimate samples from unknown
inputs. We assume that most of the adversarial noises are crafted with only respect
to deceive a classifier and imperceptible to perceive by a human. By adopting slightly
the pixel values of input image toward to new targeted label, adversarial attack methods are inattentive to spatial constraints from the original image, and it leads to drop
adversarial noises on the high-frequency domain. Moreover, most of the adversarial
attack methods based on gradient descent to create malicious patterns and optimize
loss function between legitimate input with a targeted label, it tries to find the best
candidate with the targeted classifier. This leads to a variant of overfitting whenfine-tuning the important features of the original image. Besides, the image classification systems based on the deep learning are very sensitive to transformation so we
deeply investigate how to use transformation to break the effectiveness of adversarial
examples on the classification systems.
To prove our assumption, we first set up a transform layer and a denoiser layer
between the adversarial image and a classifier. We observe the probabilities of outputs
and analyze the changing of probabilities of the original label and adversarial label
before and after using layers. From those observations, we propose a state-of-theart automatic adversarial identification system with three parallel layers between
input images and a classifier. Three parallel layers include a forward layer, transformlayer and denoiser layer. The forward layer’s role is forwarding the unknown input
to the classifier. The transform layer and denoiser layer perform on the unknown
input to deform and remove the adversarial features in the high-frequency band. The
observed outputs from the classifier are the labels with the highest probabilities toform into a vector. We set up a decision system to evaluate the vector to decide which
unknown input is adversarial or benign. Our proposed model is described in Fig. 3-7
81

Algorithm 4: Automatic Identification System with Image Transformation
Layersinput : 𝑥, 𝑓 , 𝐿, 𝐺𝑅, 𝐺𝑇 , 𝐺𝑇 &𝑅
output : 1: adversarial input, 0: clean input

1 𝑙𝑚𝑎𝑟𝑘𝑒𝑑 ← 𝑓 (𝑥) // generating a marked label

if GR then3
for 𝑖 in range(5,20) do

2

4𝑎𝑛𝑔𝑙𝑒 ← i * 𝜋/180

5𝑥𝐺𝑅𝑖 ← 𝐺𝑅(𝑥, 𝑎𝑛𝑔𝑙𝑒) ; 𝑙𝑖 ← 𝑓 (𝑥𝐺𝑅𝑖 ); 𝐿 ← 𝑙𝑖

end
7 else if GT then8
for 𝑖 in range(5,20) do
9for 𝑗 in range(5,20) do
610𝑣𝑖 ← 𝑖; 𝑣𝑗 ← 𝑗 ; 𝑥𝐺𝑇 𝑖 ← 𝐺𝑇 (𝑥, 𝑣𝑖 , 𝑣𝑗 )11𝑙𝑖 ← 𝑓 (𝑥𝐺𝑇 𝑖 ); 𝐿 ← 𝑙𝑖

end
13
end
14 else15
for 𝑖 in range(5,20) do
16
for 𝑗 in range(5,20) do
12

17𝑣𝑖 ← 𝑖; 𝑣𝑗 ← 𝑗

18

𝑥𝐺𝑇 𝑖𝑗 ← 𝐺𝑇 (𝑥, 𝑣𝑖 , 𝑣𝑗 )19

for 𝑞 in range(5,20) do

20

𝑎𝑛𝑔𝑙𝑒 ← 𝑞 * 𝜋/180

21𝑥𝐺𝑅𝑞 ← 𝐺𝑅(𝑥𝐺𝑇 𝑖𝑗 , 𝑎𝑛𝑔𝑙𝑒)

22

𝑙𝑖 ← 𝑓 (𝑥𝐺𝑅𝑞 ); 𝐿 ← 𝑙𝑖

end
24
end
25
end
26 end23

82

FORWARD

X0

X1

TRANSFORM

DENOISER

X2

Xn-1

Xn

ln-1

ln

DNNs

l0

l1

l2

ADVERSARIAL

BENIGN

Figure 3-7: Adversarial Examples Identification System

In this model, we set up three layers simultaneously between the input and theclassifier. The forward layer is passed directly into the classifier. Transform layer
83using image transformation algorithm (geometric rotation, geometric translation, andcombination). The Denoiser layer uses algorithms to eliminate noise in the highfrequency domain. Afterward, the data has been processed through these three layers,
from each input data we have a set of data 𝑋 = {𝑋0 , 𝑋1 , ..., 𝑋𝑛−1 , 𝑋𝑛 }. These data
will be the input of the classifier. The outputs of the classifier are labels with thehighest probability 𝑙 = {𝑙0 , 𝑙1 , ..., 𝑙𝑛−1 , 𝑙𝑛 }. Based on the set of labels 𝑙, we use asimilarity function to compare the similarities of the elements in the set 𝑙 to decide
whether the input data is adversarial or benign. Our system performs to analyze
adversarial data automatically and there is no need to set a threshold value to makea final decision. The filter layer places the high-frequency components in the input
processing while the forward layer transfers the input directly to the target classifier.
The highest-confidence class from the classifier is assigned as the marked label. The
filter process then tracks the labels similar to the marked label. If the marked label
is equal to all filtered labels, our system confidently determines the input as benign,
but in the case, where are more than two different marked labels that are different
from the marked label, the detection system points out it as an adversarial example.
Our system proceeds by algorithm 3, where x defines the input image, 𝐿 is filteredlabels, 𝑙𝑚𝑎𝑟𝑘𝑒𝑑 is marked label, 𝜅 denotes the kernel sizes, 𝑓 is a machine learning
function that computes the predicted label with the highest probability, and 𝑠 is thefilter function. The filter function based on the Gaussian filter is called the labelbased identification model based on Gaussian (LIMG); the other filter function is ouridentification model based on Median (LIMM). We set the kernel size in range [3×3,
5×5].In the transform layer (algorithm 4), we define three main functions include
GR, GT, and GT&R that stand for geometric rotation, geometric translation anda combination of them, respectively. For GR method, we chose the rotation angle
in range [5,20]. In GT method, we select the translation in range[5,20]. For GT&R
method, the first we apply the GT, after that we use GR on the image input. The
composite model that combines transform and denoiser layers we call CMTD.

84

3.3.1 Evaluation Metrics
We use the some commonn metrics for evaluating our implementation as below.

∙

True Positive (TP) is an outcome where the model correctly predicts the
positive class.

∙

True Negative (TN) is an outcome where the model correctly predicts thenegative class.

∙

False Positive (FP) is an outcome where the model incorrectly predicts the
positive class.

∙

False Negative (FN) is an outcome where the model incorrectly predicts thenegative class.

∙Precision or positive predictive value is the proportion of positive and negative
results and equal to TP/(TP+FP).

∙

Recall or sensitive or true positive rate measures the proportion of actual positives that are correctly identified and it is defined as TP/(TP+FN).

∙

Accuracy is the closeness of the measurements to a specific value, and it isdefined as (TP+TN)/(TP+TN+FP+FN).

∙

F1 score is a measure of a test’s accuracy, and it equals to 2*(Precision *Recall)/(Precision + Recall).

3.3.2 Implementation and Settings
In this section, we describe the setup and settings of the proposed automatic identification models. In general, we use two benchmark data sets: small scale MNIST andlarge scale ImageNet to build synthesized data sets. In this study, we use the stateof-the-art deep neural network is Google Inception V3 [108] as a victim classifier tocreate adversarial images as well as evaluating the identification results because the85problem we focus on is the white-box manner. In the creation of adversarial examples, we use four attack methods: FGSM, PDG, CW_L2 , and EAD. In the transformlayer, we use three methods of GT, GR, and GTR. In the denoiser layer, we use two
methods of LIMG, LIMM. The proposed composite model uses the transform anddenoiser layers. The test was performed on Intel (R) Core (TM) i9-9900X 20 CPUs
3.50 GHz, 64 GB Memory, NVIDIA GeForce RTX 2080Ti 11GB.

Setup of MNISTThe MNIST dataset [57] includes 70,000 gray images of hand-written digits ranging
from 0 to 9. It is separated into 60,000 training images and 10,000 testing images.
A single MNIST image is composed of 28 × 28 pixels, and is encoded by an 8-bit
grayscale. We randomly extract 200 images of the digit “0” from the 10,000 testing
images. From each of these 200 images, we create nine adversarial images targeting
the remaining digits (1-9). Finally, we create a new synthesized benchmark datasetof 2,000 images (200 benign images and 1,800 adversarial images).

Setup of ImageNet
We consider the ImageNet dataset [96] that is a very large database created forusing in many tasks such as image classification, and object detection. The original
ImageNet includes more than 14 million images in 20,000 categories with a typicalcategory, such as “oscilloscope” or “ostrich”, consisting of several hundred images. The
machine learning model that we use is Google Inception V3 [108] that was trainedwith 1,000 common categories ImageNet. We randomly select 360 testing images. By
applying FGSM, PGD, CW_L2 and EAD methods to craft adversarial images target
to randomly chosen labels, we generate new 1,440 adversarial images. We combine
them together to form a new benchmark repository including 1,800 images for our
experiment. Although adversarial examples have recently attracted much interest
from researchers, a public benchmark dataset for evaluating the performance andquality of defense systems remains lacking. In the attack phase of our system, we
thus created a new benchmark dataset for evaluating the detection capabilities.
863.3.3 Results(a) FGSM Adversarial Image, 99.99% as os-(b) PGD Adversarial Image, 99.90% as ostrich

trich label

label(c) CW_𝐿2 Adversarial Image, 99.85% as os-(d) EAD Adversarial Image, 99.89% as ostrich

trich label

label

87
(e) Original Image,99.35% as the groundWe compare our results with Xu et al. [120]. Our system is more convenient than
Xu’s system in either detection accuracy and easy-to-setup, our system does not use
a threshold for making the decision while Xu’s system has to carefully choose a setof threshold values. Our system only setup the paramters for the transform layerand denoiser layer such as kernel sizes, rotation angles, and translation angles. All ofthese parameter’s values are selected based on our observation in the implementation
results. We also report our system’s performance by using the F1-scores metric. We
observe and analyze a typical case with an image from the ImageNet dataset. In Figure 3-8, from a random image in the ImageNet dataset, the classifier system produces
the highest probability of 99.7% of the 𝑜𝑠𝑐𝑖𝑙𝑙𝑜𝑠𝑐𝑜𝑝𝑒 label. We use four different methods FGSM, PGD, CW_L2 and EAD to create adversarial images with targeted labels
as a 𝑜𝑠𝑡𝑟𝑖𝑐ℎ label by randomly selected from 1,000 labels in the ImageNet dataset.

88

(a) Observation on probability of targeted la-(b) Observation on probability of targeted la-bel when using GT method

bel when using GR method(c) Observation on probability of targeted label when using GT&R method

Figure 3-9: Our proposed Model 1 defeats the adversarial perturbation’s affectFigure 3-9 shows the changing of probability classification for the targeted label
before and after using the transform layer. We observe that the probabilities of thetargeted labels suddenly drop after using the transform layer. In figure 3-9.a, when
the angle value is zero (meaning GT is not used), the probability of the targeted label
is close to 1.0 for adversarial images, but for blue lines (the observation on the benign
image), the probability is 0.0. However, when the angle value increases to 5 degrees,
we can see that the probability of targeted labels on adversarial images decreases
to approximately 0.0 (exclude PGD adversarial image in green line). And this is
maintained when increasing the value of the angle up to 10 degrees for adversarial
examples created using the FGSM, PGD and CW_L2 methods. The probabilities of89

the targeted label tend to increase in the range [10,20] when using the GT method inthis case. In figure 3-9.b, the GR method works more effectively than the GT method
when the angle value increases from 5 to 20 degrees, the probability of targetedlabel decreases and remains at 0.0 for all adversarial images created by four attack
methods. In figure 3-9.c, the GT&R method also works well when the effect of theadversarial perturbation on the classifier system has been eliminated. This is a very
important observation that we decide on the range of angles of transformation in ouridentification system in range [1, 5] in the percentage of an input image.

(a) Input: FGSM Adversarial Image(b) Input: PGD Adversarial Image(c) Input: CW_L2 Adversarial Image(d) Input: EAD Adversarial Image(e) Input: Original Image

Figure 3-10: Observation on the probabilities of the ground truth label with GT&R
method90

Figure 3-10 shows the observations on the probabilities of the ground truth labels
when we use the GT&R method. We can observe that in some cases (such as Figure 310.c and Figure 3-10.e), the probabilities of the ground truth label are even better
than without GT&R.

(a) Input: FGSM Adversarial Image(b) Input: PGD Adversarial Image(c) Input: CW_L2 Adversarial Image(d) Input: EAD Adversarial Image(e) Input: Original Image

Figure 3-11: Observation on the probabilities of the targeted label with GT&R
methodFigure 3-11 shows observations of the probabilities of a targeted label change
when the GT&R method is applied. It is obvious to see that in many cases the
GT&R method suppresses the probability of targeted labels to around 0.0.
91(a) Observation on probability of ground truth(b) Observation on probability of ground truthlabel when using GT method

label when using GR method(c) Observation on probability of ground truthlabel when using GT&R method

Figure 3-12: Our proposed Model 1 regain the true recognition
Figure 3-12 shows the probability classification results for the ground truth label
are stable when input is the original image after and before using the transform layer.
On the other hand, probability classification for the ground truth label increased
dramatically when inputs are adversarial images after using transform. Figure 3-12.ashows the probability of ground truth label when the input images are adversarial.
These probabilities are recovered when the angle value increases to 5 degrees. But
when the angle value increases more, the probabilities tend to go down, excluding the
CW_L2 adversarial image. In Figure 3-12.b, we can observe that the GR method
works more effectively than the GT method when the probabilities of the groundtruth label increase steadily with the increment in angle value. In Figure 3-12.c, the92

GT&R method achieved slightly better results when the probabilities of the groundtruth labels increased close to the baseline. This observation confirms our assumptionthat adversarial perturbations are weak to our proposed model and by adapting thetransform layer in our model, we can propose a powerful automatic identification
model for adversarial examples.

(a) Probabilities of ground truth label with(b) Probabilities of targeted label with LIMG

LIMG method

method(c) Probabilities of ground truth label with(d) Probabilities of targeted label with LIMM

LIMM method

methodFigure 3-13: Adversarial examples suffers to our first proposed model (LIMG +LIMM)

Next, this is followed by empirical results using the LIMG and LIMM methods.
Figure 3-13 shows the observations on the probabilities of the ground truth andthe targeted label. In Figure 3-13.a and 3-13.b, both methods help the classifier toincrease the probability of ground truth when we increase the kernel size from (3 × 3)
to (5 × 5). In contrast, the probability of targeted labels also decreases as the kernel
size increases as are depicted in Figure 3-13.b and 3-13.d.
93

Figure 3-14: Our observation on original 𝑜𝑠𝑐𝑖𝑙𝑙𝑜𝑠𝑐𝑜𝑝𝑒 image and adversarial 𝑜𝑠𝑡𝑟𝑖𝑐ℎimages that created by FGSM [31], PGD [70], CW_𝐿2 [14], and EAD [16] methods
and when we use LIMG and LIMM method. First row, percentage values illustrateclassification rates for the ground truth label. Other rows, percentage values areclassification rates for the targeted label.

Figure 3-14 depicts the experimental results with the original image of 𝑜𝑠𝑐𝑖𝑙𝑙𝑜𝑠𝑐𝑜𝑝𝑒
and the targeted label of 𝑜𝑠𝑡𝑟𝑖𝑐ℎ when the LIMG and LIMM methods are applied.
In the first row, the percent values represent the probabilities of the oscilloscopelabel. In the other rows, the percent values show the probabilities of the ostrich label.
Obviously, when the probability of the 𝑜𝑠𝑐𝑖𝑙𝑙𝑜𝑠𝑐𝑜𝑝𝑒 label decreases, the probabilityof the 𝑜𝑠𝑡𝑟𝑖𝑐ℎ label increases and vice versa. Figure 3-14 shows both the LIMG and
LIMM methods perfectly to remove the adversarial noises with any kernel sizes inGaussian and Median filters.
94

Table 3.11: Detection Rate on MNIST dataset
Our Method

Xu et al. [120]

GT

GR

GT&R

LIMG

LIMM

CMTD

Bit-D

Best-Joint

No. Files

2,000

2,000

2,000

2,000

2,000

2,000

2,000

2,000

Threshold

NA

NA

NA

NA

NA

NA

0.00050.0029

TP

1,767

1,786

1,775

1,786

1,776

1,799

-

-

TN

174

199

171

199

182

156

-

-

FP

26

1

29

1

18

44

-

-

FN

33

14

25

14

24

1

-

-

0.978

-

-

0.976

-

-

Precision

0.993
0.985 0.999

Recall

0.982

0.992

F1 score

0.984

0.996

Accuracy

0.971

0.984

0.993 0.979
0.999 0.999

0.986

0.992

0.973

0.996 0.996

0.987

0.999

0.903

0.982

0.988

0.988

-

-

Table 3.11 and 3.12 summarize the empirical results of our models with Xu etal [120]. On the MNIST dataset (Table 3.11), the CMTD method achieves the best
detection accuracy rate at 99.9%. However, in general, GT&T and LIMG methods
have the better results on F1 score at 99.6% to compare with 98.8% from CMTD
method. On the ImageNet dataset (Table 3.12), the CMTD method also has thebest performance in detection accuracy rate at 99.9%. But with F1 score, the GT&T
method is slightly better than the CMTD method at 97.8%. In general, it is obvious
to see that our proposed CMTD method (figure 3-7) has the best result in identifying
adversarial images with the rate of 99.9% in both MNIST and ImageNet datasets.
The ’-’ notation in the table means there is no information from Xu et al. The
methods we propose do not need to set threshold parameters like Xu’s work [120], so
the installation and deployment are much faster and more efficient. In addition, we
also evaluate our models with other indicators such as accuracy, precision, recall andF1 score that also reached the highest value of 99.6% on MNIST dataset and 97.8%
on ImageNet dataset.
95

Table 3.12: Detection Rate on ImageNet dataset
Our Method

Xu et al. [120]

GT

GR

GT&R

LIMG

LIMM

CMTD

Bit-D

Best-Joint

No. Files

1,800

1,800

1,800

1,800

1,800

1,800

1,800

1,800

Threshold

NA

NA

NA

NA

NA

NA

1.44171.2128

TP

1,362

1,390

1,419

1,395

1,434

1,439

-

-

TN

343

337

318

319

294

289

-

-

FP

17

23

42

41

66

71

-

-

FN

78

50

21

6

1

-

-

Accuracy

0.947

0.959

0.995

45
0.952

0.960

0.960

-

-

0.971

0.971

0.956

0.953

-

0.985

0.969

0.996

0.999

0.751

0.859

0.970

0.976

0.976

-

-

Precision

0.988 0.984

Recall

0.946

0.965

F1 score

0.966

0.974

0.978

3.3.4 Summary for combination of Image Transformation andFilters
In this section, we investigated the defensive strategies for the deep neural networks
to the adversarial examples by combining image transform and filters. Based on theassumption and theory framework, we demonstrated the effectiveness of geometrictransform and frequency domain in removing adversarial noises. From this observation, we proposed an automated detection system for adversarial examples. We
obtained up to 99.9% detection rate on both MNNIST and ImageNet datasets. Our
approach is different from related works that we observe the effectiveness of geometric transform and frequency changing on adversarial examples first. Afterward, basedon our assumption, theory framework and observation we invented a powerful automatic identification system for distinguishing adversarial examples and legitimate
images without threshold setup. Moreover, we explored the best parameters in imagetransformation and filters for removing the adversarial features. Another important96

contribution of our work is that our proposed methods not only defeated adversarial
noises but also in some cases our systems regained the ground truth label recognition.

97

3.4

Evaluating the robustness of adversarial perturbation against Image Filters

The superiority of deep learning performance is threatened by safety issues for itself.
Recent findings have shown that deep learning systems are very weak to adversarial
examples, an attack form that was altered by the attacker’s intent to deceive thedeep learning system. There are many proposed defensive methods to protect deeplearning systems against adversarial examples. However, there is still lack of principal
strategies to deceive those defensive methods. Any time a particular countermeasure
is proposed, a new powerful adversarial attack will be invented to deceive that countermeasure. In this study, we focus on investigating the ability to create adversarial
patterns in search space against defensive methods that use image filters. Experimental results conducted on the ImageNet dataset with image classification tasks showed
the correlation between the search space of adversarial perturbation and filters. Thesefindings open a new direction for building stronger offensive methods towards deeplearning systems.

3.4.1 Search Space on Attacking Phase
We consider the white-box targeted attack settings, where the attacker can fully
access the model type, model architecture, all trainable parameters, etc., and theadversary aims to change the classifier’s prediction to some specific target class. The
attackers use available information to identify the feature space where the model isvulnerable or try to find the victim decision boundaries. Then the victim modelis exploited by altering a clean input by using adversarial example methods. To
create adversarial samples that are misclassified by the machine learning model, an
adversary with knowledge of the model’s classifier 𝑓 and its trainable parameters. In
this work, we use FGSM [31] method for crafting adversarial examples. We define[︁]︁𝑛classifier function 𝑓 : R → 1...𝑘 that maps image pixel value vectors to a particular[︁]︁label. Then we assume that function 𝑓 has a loss function 𝐿 : R𝑛 × 1...𝑘 → R. For
98

[︁
]︁
an input image 𝑥 ∈ R𝑛 and target label 𝑦 ∈ 1...𝑘 , our system aims to solve the[︁ ]︁𝑛following optimization problem: 𝛿 + 𝐿(𝑥 + 𝛿, 𝑦) subject to 𝑥 + 𝛿 ∈ 0, 1 , where𝛿 is an adversarial perturbation that we aim to add it to the original image 𝑥. We
have to note that this function method would yield the solution for 𝑓 (𝑥) in the caseof convex losses, however, the neural networks are non-convex so we end up with anapproximation in this case. In this case, we use the output of the second-to-last layer
logits for calculating the gradient instead of using the output of the softmax function.
Our main purpose to decide the size of adversarial perturbation, it means the search
space of adversarial perturbation. We consider the norm operation to determine thesize of the adversarial noises. Mathematically, a norm is a total size or length of allvectors in a vector space or matrices. For simplicity, we can say that the higher thenorm is, the bigger tha value matrix or vector. Formally, the 𝑙𝑝 -norm of vector x is
√︀∑︀
𝑝
𝑡ℎ
defined as: ‖𝑥‖𝑝 = 𝑝
𝑖 |𝑥𝑖 | , where 𝑝 ∈ R. This is a 𝑝 -root of a summation of allelements to the 𝑝𝑡ℎ power is what we call a norm. The interesting point is even though
every 𝑙𝑝 -norm is all look very similar to each other, their mathematically propertiesare very different and thus their application are dramatically different when we use
to create the adversarial examples. In this work, we consider three common norm
methods: 𝑙1 -norm, 𝑙2 -norm, and 𝑙∞ -norm for evaluating the size of the search spaceof adversarial perturbation.

l1-norm. From the definition of 𝑙 -norm, 𝑙 -norm of x is defined as:𝑝1‖𝑥‖1 =∑︁(3.31)|𝑥𝑖 |𝑖This norm is quite common among the norm family. It has many name and many
forms among various fields, namely Manhattan norm.

l2-norm. The most popular of all norm is the 𝑙 -norm. It is used in almost everyy
2

field of engineering and science as a whole. 𝑙2 -norm is defined as:‖𝑥‖2 =

√︃
∑︁
𝑖

99

𝑥2𝑖

(3.32)

𝑙2 -norm is well known as a Euclidean norm, which is used as a standard quantity
for measuring a vector difference. If the Euclidean norm is computed for a vector
difference, it is known as a Euclidean distance:√︃∑︁‖𝑥1 − 𝑥2 ‖2 =(𝑥𝑖1 − 𝑥𝑖2 )2(3.33)

𝑖l∞-norm. The 𝑙 -norm is defined as:
∞

√︃
∑︁

‖𝑥‖∞ = ∞𝑥∞𝑖(3.34)𝑖

Let consider the vector 𝑋 , if 𝑥𝑖 is the highest element in vector 𝑋 , by the property∑︀ ∞∞∞of the infinity itself, we have: 𝑥∞
𝑖 ≈ 𝑥𝑘 ∀𝑖 ̸= 𝑘 , then𝑖 𝑥𝑖 = 𝑥𝑘 . And we have√︀∑︀ ∞√ ∞∞‖𝑥‖∞ = ∞𝑥𝑘 = |𝑥𝑘 |. Now we have simple definition of 𝑙∞ -norm as:𝑖 𝑥𝑖 =‖𝑥‖∞ = 𝑚𝑎𝑥(|𝑥𝑖 |).
So our attack phase is denoted as Algorithm 5 by using FGSM. Where x defines
the original input, 𝑦𝑡𝑟𝑢𝑒 defines the ground truth label, 𝑦 * is an adversarial label, f
is the activation function of machine learning model, 𝜖 is the maximum adversarial
value, 𝑙𝑖 defines the norm. In the attacking phase, we set the learning rate for crafting
adversarial examples to 0.01 to keeps adversarial noises are as small as possible andthe iterative process is repeated 500 times. From the clean images, we will create thetargeted output images.

3.4.2 Implementation
The classification task was evaluated on the ImageNet benchmark dataset. We consider the ImageNet dataset [96] that is a very large database designed for use in visualobject recognition research. The original ImageNet includes more than 14 million images in 20,000 categories with a typical category, such as “vending machine” or “streetsign”, consisting of several hundred images. The machine learning model that we use
is Google Inception V3 [108] that was trained with 1,000 common categories ImageNet. We randomly select some testing images. By applying FGSM with 𝑙1 -norm,
100

Algorithm 5: Crafting Adversarial Examples with 𝑙-norm optimization
input : 𝑥, 𝑦 , 𝑦 , 𝑓 , 𝜖, 𝑙output : 𝑥
parameter: learning rate = 0.01, epochs = 500
𝑡𝑟𝑢𝑒

*

𝑖

*

1 𝑥 ← 𝑥* // initial adversarial sample
2 𝛿𝑥 ← ⃗0 // initial perturbation factor
3 𝑖𝑡𝑒𝑟 ← 1 // initial iteration counter4𝛿𝑥 < 𝜖 and 𝑓 (𝑥* ) ̸= 𝑦 * and 𝑖𝑡𝑒𝑟 <= 𝑒𝑝𝑜𝑐ℎ𝑠5𝑥* ← 𝑥 + 𝛿 · 𝑠𝑖𝑔𝑛(▽𝐿(𝑦 * |𝑥* ))6
𝛿𝑥 ← norm(𝑙𝑖 )7maximize 𝐿𝑜𝑠𝑠(𝑦 * |𝑥* ) respect to 𝛿𝑥8𝛿 ← 𝑐𝑙𝑖𝑝(𝑥* , 𝑥 − 𝜖, 𝑥 + 𝜖)9𝑖𝑡𝑒𝑟 ← 𝑖𝑡𝑒𝑟 + 1

while

10
11

end
return 𝑥

do*𝑙2 -norm, and 𝑙∞ -norm to craft adversarial images target to randomly targeted labels.

3.4.3 Results
Intuitively, we use our own images (include pictures of vending machine, computer
mouse and keyboard) and an image of oscilloscope from ImageNet dataset for analysis.
We randomly selected targeted labels for the creation of adversarial images. By using
the FGSM method in combination with 𝑙1 -norm, 𝑙2 -norm, and 𝑙∞ -norm, from eachoriginal image we create three different adversarial images.
Figure 3-15 shows the results of creating adversarial images from the original
image of the oscilloscope. We find that the deep learning system is easily fooledwith adversarial images. We observe the changing of probabilities of the originallabel before and after using Gaussian and Median filters. In addition, we intuitively
observe that adversarial images created with 𝑙1 -norm and 𝑙2 -norm are easier to defeat
than 𝑙∞ -norm. With Gaussian kernel size 3×3, the probability of original label is stillequal to 0 with adversarial 𝑙∞ -norm. It means that Gaussian kernel size 3×3 can notremove the adversarial noises in this case.
Figure 3-16 shows the implementation results with the observation on the probabilities of adversarial label. The results demonstrate the probabilities of adversarial
101

Figure 3-15: Classification Results on Oscilloscope Image with observation on theprobabilities of Original Labellabel drop from nearly 100% to 0% when we use both Gaussian and Median filter
exclude Gaussian kernel size 3×3 with adversarial 𝑙∞ -norm.

Figure 3-16: Classification Results on Oscilloscope Image with observation on theprobabilities of Adversarial Label
Figure 3-17 and Figure 3-18 show the implementation results with image of vending machine. The adversarial vending machine 𝑙∞ -norm is the strongest adversarial
when both Gaussian and Median can not regain the probabilities of original label.
Figure 3-19 shows the implementation results with images of computer mouse andcomputer keyboard.
Figure 3-20 shows the results of creating adversarial images from the original102

Figure 3-17: Classification Results on Vending Machine Image with observation on
the probabilities of Original Label

Figure 3-18: Classification Results on Vending Machine Image with observation on
the probabilities of Adversarial Label

image of the oscilloscope. We find that the deep learning system is easily fooled withadversarial images. In addition, we intuitively observe that adversarial images createdwith 𝑙1 -norm and 𝑙2 -norm are harder to detect than 𝑙∞ -norm.
Figure 3-21 shows the experimental results when we use the image filters methodon the original image of the oscilloscope. We find that the Gaussian filter reducesclassification accuracy more than the median filter. Especially in the case of themedian with size filter 3×3 and 5×5, the classification results are better than theoriginal image.
103

(a) Observation on the probabilities of Origi-(b) Observation on the probabilities of Adver-

nal Keyboard Labelsarial Label(c) Observation on the probabilities of Origi-(d) Observation on the probabilities of Adver-

nal Computer Mouse Labelsarial Label

Figure 3-19: Classification Results on Keyboard and Computer Mouse Images
Similar to the original image, we also apply image filter methods to adversarial
images. Figure 3-22 shows classification results on adversarial images created by the
FGSM method in combination with 𝑙1 -norm. Figure 3-23 illustrates classification
results on adversarial images created by the FGSM method in combination with𝑙2 -norm. We observed that Gaussian kernel size 3x3 could not restore identity toground truth label on adversarial image with 𝑙2 -norm. The probability for vending
machine label is only 14.8%. Meanwhile, the median filter still works effectively inremoving adversarial noises. Figure 3-24 shows classification results on adversarial
images created by the FGSM method in combination with 𝑙∞ -norm. We observedthat Gaussian kernel size 3×3 could not eliminate the effect of adversarial noise with𝑙∞ -norm on deep learning system classification. Gaussian 5×5 gives better results,
but the label with the highest probability of identification is “tabacco shop”. The
Median filter removes adversarial noises but cannot help the deep learning systemcorrectly identify ground truth labels.
Table 3.13 shows experimental results on oscilloscope, vending machine (v-machine),104

(a) Orginal Oscilloscope(b) Adversarial FGSM_L1 Vending Machine(c) Adversarial FGSM_L2 Vending Machine(d) Adversarial FGSM_L∞ Vending Machine

Figure 3-20: Adversarial Oscilloscope (targeted class: Ostrich Sign)
computer mouse (c-mouse) and keyboard sets. This result shows us a large correlation between norm operations in search space of adversarial examples. It is clearthat for the 𝑙∞ -norm, the Gaussian (3×3, 5×5) and the Median (3×3) methods are
more difficult to completely eliminate adversarial noises based on the 𝑙1 and 𝑙2 norm.
Median (5×5) still proved superior in removing adversarial noises in all settings.

3.4.4 Summary for Evaluating the robustness of adversarial
perturbation against Image Filters
In this section, we focus on investigating the connection between the search space ofadversarial examples and the defense based on the frequency domain. Our empiricalresults demonstrate that the FGSM method in combination with 𝑙∞ -norm produces
the strongest adversarial examples. In this case, both the Gaussian and the Medianfilters are unable to restore identification to the ground truth label. However, whenusing 𝑙∞ -norm to create adversarial examples, we also significantly reduce the qualityof the original image compared to using 𝑙1 and 𝑙2 norm. In terms of similarities with105

(a) Orginal Oscilloscope with Gaussian(b) Orginal Oscilloscope with Gaussian(3 × 3)(5 × 5)(c) Orginal Oscilloscope with Median (3 × 3)(d) Orginal Oscilloscope with Median (5 × 5)

Figure 3-21: Original Oscilloscope with Image Filters
the original image, 𝑙1 and 𝑙2 norm produce much better adversarial examples than 𝑙∞
norm.

106

(a) Adversarial Oscilloscope with Gaussian(b) Adversarial Oscilloscope with Gaussian(3 × 3)(5 × 5)(c) Adversarial Oscilloscope with Median(d) Adversarial Oscilloscope with Median(3 × 3)(5 × 5)Figure 3-22: Adversarial FGSM L1 Oscilloscope (targeted class: Ostrich) with
Image Filters
Table 3.13: Implementation Results
Input
Org oscilloscope
adv 𝑙1 oscilloscope
adv 𝑙2 oscilloscope
adv 𝑙∞ oscilloscope
Org v-machine
adv 𝑙1 v-machine
adv 𝑙2 v-machine
adv 𝑙∞ v-machine
Org keyboard
adv 𝑙1 keyboard
adv 𝑙2 keyboard
adv 𝑙∞ keyboard
Org c-mouse
adv 𝑙1 c-mouse
adv 𝑙2 c-mouse
adv 𝑙∞ c-mouse

No filter
OLAL
0.997 001
01
01
0.779 0
0
0.999
0
1
0
1
0.894 0
0
0.999
0
0.999
0
1
0.724 0
0
0.999
0
1
0
0.999

Gau 3x3
OLAL
0.997 00.997 00.995 000.558 00.589 0.004
0.148 0.015
0.031
0.767 00.529 0.002
0.596 0.002
00.93
00.518 0.004
0.116 0.045
0

0.995
0.157
0.977
0.995

107

Med 3x3
OLAL
0.996 00.993 00.992 00.945
0.881 00.805 0.001
0.543
0.053 0.005
0.761 00.665 00.567 0.001
0.139
0.937 00.911 00.168 0.005
0

0.001
0.006
0.107
0.9

Gau 5x5
OLAL
0.995 00.996 00.994 00.937 00.517 00.604 0.001
0.407 0.001
0.066
0.661 00.556 0
0.57
0
0.132
0.964 00.914 00.757 0
0.014

0.002
0.01

0.013

Med 5x5
OLAL
0.983 00.982 00.985 00.997 00.857 00.827 00.804 00.064 00.4
00.451 00.383 00.267 00.924 00.884 00.793 00.218 0(a) Adversarial Oscilloscope with Gaussian(b) Adversarial Oscilloscope with Gaussian(3 × 3)(5 × 5)(c) Adversarial Oscilloscope with Median(d) Adversarial Oscilloscope with Median(3 × 3)(5 × 5)Figure 3-23: Adversarial FGSM L2 Oscilloscope (targeted class: Ostrich) with
Image Filters

108

(a) Adversarial Oscilloscope with Gaussian(b) Adversarial Oscilloscope with Gaussian(3 × 3)(5 × 5)(c) Adversarial Oscilloscope with Median(d) Adversarial Oscilloscope with Median(3 × 3)(5 × 5)Figure 3-24: Adversarial FGSM L∞ Oscilloscope (targeted class: Ostrich) with
Image Filters

109

110

Chapter 4
Conclusion and Future Works
4.1

Conclusion

Adversarial examples (AE) is emerging as one of the key topics in the field of security
for AI-based systems. With very little modification of the input data, AE can deceiveand completely change the prediction of the output from the AI system while thehuman is not able to realize the difference between AE and the original data. AE is
not only exploited on image classification systems but also appears in natural language
processing systems, speech recognition or object detection systems. Because of thesemajor security issues of AE, I focus on how to make the AI system more secure, able
to identify and distinguish between adversarial and legitimate patterns. My researchmainly focuses on building detection and distinguishing systems between AE andlegitimate images in image classification tasks. My main research directions include:(1) AE detection system based on the spatial domain; (2) AE detection system basedon the frequency domain; (3) Evaluation of the robustness of AE against image filters.
The first is the spatial domain. Image processing systems based on deep learning such as image classification, image segmentation or object recognition are often
trained to overcome changes in spatial structure by using data augmentation techniques. In addition, those deep learning systems are designed with multiple layers withmultiple operations and different settings to help the system learn more importantfeatures. These properties help deep learning systems can produce very impressive111

results in their own tasks. However, most of the training process in deep learning systems assumes that the data distribution is constant and it only focuses on fine-tuning
the system trainable parameters and weights. This makes it possible for attackers totake advantage of AEs based on reusing or re-simulating parameters and deep learning structure and then editing them toward to a desired target. It is important thatthe attack methods are based on the loss function optimization and are not concernedwith the spatial structure of the image. However deep learning systems have been
trained to work with spatial variation. Based on this analysis, I take advantage of thespatial weaknesses of AE to build an AE identification system based on affine transformations. Our detection system based on affine transformation (see Table. 3.8 on
MNIST dataset, Table. 3.7 on ImageNet dataset) mostly defeat many state-of-the-art
AE attack methods.
The second one is the frequency domain. During the creation of the AE, the attack
methods unintentionally created pixel values that far different from the neighboringareas. Therefore, AEs have been unintentionally created on the high digital currency
in the image. Based on this observation and analysis, we design and build automateddetection systems (see Table 3.9 for MNIST dataset and 3.10 for ImageNet dataset)for identifying AE and legitimate images based on the frequency domain. And theexperimental results have proved my system to produce very satisfactory results on
popular data sets such as MNIST and ImageNet datasets. Moreover, the combinationof transformation and filter achieve the best detection results to compare with othermethods (Table 3.11 for MNIST dataset and 3.12 for ImageNet dataset). From theimplementation results, we found that the combination of transformation and filters(the CMTD method - see Figure 3-7) achieves the best detection accuracy rate inboth MNIST and ImageNet datasets.
The third one is how to evaluate the robustness of AE against image filters. In
the context of adversarial inputs at test time, we observed several effective attack
algorithms but few strong countermeasures. Any time a particular countermeasure isinvented, a new strong and powerful attack will be proposed to devastate that countermeasure. We can explain adversarial examples in current machine learning models
112

as the result of unreasonably linear extrapolation but do not know what will happen
when we fix this particular problem; it may simply be replaced by another equallyannoying category of vulnerabilities. The vastness of the set of all possible inputs toa machine learning model seems to be cause for pessimism. These questions may be
addressed empirically, by actually playing out the competition between new attacks
and new countermeasures are developed. So there is a very important question thatis how to measure the robustness of a machine learning algorithm. If we can answer
this question, we confidence to determine a particular machine learning algorithm isstrong or weak to AE. We evaluated the robustness of AE to a defensive system basedon the frequency domain. We investigated the FGSM method in combination with 𝑙1 ,𝑙2 and 𝑙∞ -norm to create AE. After that, we evaluate the robustness of the AE against
image filters. The implementation results (Table 3.13) show the AE with 𝑙∞ -norm
is the strongest AE but image filters (Gaussian and Median) still defeat most of AE
noises in many settings.

4.2

Future Works

Adversarial machine learning is at a turning point. In the context of adversarial
inputs at test time, we observed several effective attack algorithms but few strong
countermeasures. Any time a particular countermeasure is invented, a new strong
and powerful attack will be proposed to devastate that countermeasure. Can we
expect this situation to continue indefinitely? Or can we expect the defender toeventually gain a fundamental advantage? We can explain adversarial examples incurrent machine learning models as the result of unreasonably linear extrapolation but
do not know what will happen when we fix this particular problem; it may simply be
replaced by another equally annoying category of vulnerabilities. The vastness of theset of all possible inputs to a machine learning model seems to be cause for pessimism.
On the other hand, one may hope that as classifiers become more robust, it could
become impractical for an attacker to find input points that are reliably misclassified
by the target model, particularly in the black-box setting. These questions may be113addressed empirically, by actually playing out the competition between new attacks
and new countermeasures are developed. From our proposed countermeasure by using
image transformation and filtering, there are still unclear that will not have new
attacks to overcome them. So far, almost adversarial attacks and defense strategies
were proposed based heavily on heuristic and lack of concrete theory. Besides, thegap between machine learning capability and the human brain still remains verylarge. An arbitrary picture that humans may be easily recognized and give the rightanswer but it still keeps challenges for a recognition system from different viewpoints,
illustrations, luminance, and noises. So there is a very important question that is how
to measure the robustness of a machine learning algorithm. If you can answer thisquestion, you can confidence to determine a particular machine learning algorithm isstrong or weak to adversarial example. Let consider a multi-classification classifier
with 𝐾 classes and 𝑑 features where one has a classification function 𝑓 : R𝑑 → R𝐾
and a sample x is classified via 𝑓 (𝑥). We call a classifier robust at sample 𝑥 if withsmall changes of the original input do not alter the final decision. Suppose that theclassifier outputs class 𝑐 for input 𝑥, that is 𝑓𝑥 (𝑥) > 𝑓𝑗 (𝑥) for 𝑗 ̸= 𝑐. The problem ofgenerating an adversarial sample 𝑥 + 𝛿 such that the classifier decision changes, canbe formulated as:

𝑚𝑖𝑛, s.th. 𝑚𝑎𝑥𝑓𝑙 (𝑥 + 𝛿) ≥ 𝑓𝑐 (𝑥 + 𝛿) and 𝑥 + 𝛿 ∈ 𝐶 ,
𝛿∈R𝑑

𝑙̸=𝑐

where 𝐶 is a constraint set specifying certain requirements on the adversarial
sample 𝑥+𝛿 . However, in computer vision, with small unintended changes 𝑥+𝜃 should
not change the output classification. So the idea here to evaluate the robustness ofthe machine learning system is that how to calculate the probability Ψ that the value𝛿∈/ 𝜃. If a machine learning system has a large probability Ψ that means it is a strong
system to adversarial examples and vice versa.

114

Appendix A
Figures(a) Coordinate Transformation on Adversarial(b)“Keyboard” Image (targeted class is ostrich)tion on Adversarial “Key-

Geometric

board”

Image

Transla(targetedclass is ostrich)(c) Coordinate Transformation on Adversarial(d)Geometric“C-Mouse” Image (targeted class is ostrich)

tion

on

Mouse”

Transla-Adversarial
Image

“C-

(targetedclass is ostrich)(e) Coordinate Transformation on Adversarial(f )Geometric“V-Machine” Image (targeted class is ostrich)

tion

on

Transla-Adversarial“V-

Machine” Image (targetedclass is ostrich)Figure A-1: Coordinate transformation - Geometric translation on the Adversarial
images (“Keyboard”, “C-Mouse”, “V-Machine”).
115(a) digit 0 - index: 4457(b) digit 0 - index: 5584

(c) digit 0 - index: 1438(d) digit 0 - index: 1558(e) digit 0 - index: 7031(f ) digit 0 - index: 9962

(g) digit 0 - index: 297(h) digit 0 - index: 5259

(i) digit 0 - index: 4070(j) digit 0 - index: 4515

Figure A-2: MNIST - Coordinate Transformation with 𝛼 ∈ [0, 30]. Index numbers
identify the index of images in MNIST training dataset. The figure illustrates theadversarial perturbation is defeated by Coordinate Transformation in all experiments.
116

Bibliography[1] Wieland Brendel *, Jonas Rauber *, and Matthias Bethge. Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.
In International Conference on Learning Representations ICLR, 2018.
[2] Naveed Akhtar, Jian Liu, and Ajmal Mian. Defense against universal adversarial
perturbations. In Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition, pages 3389–3398, 2018.
[3] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learningin computer vision: A survey. IEEE Access, 6:14410–14430, 2018.
[4] Hyrum S Anderson, Jonathan Woodbridge, and Bobby Filar. Deepdga:
Adversarially-tuned domain generation and detection. In Proceedings of the2016 ACM Workshop on Artificial Intelligence and Security, pages 13–21. ACM,
2016.
[5] Anurag Arnab, Ondrej Miksik, and Philip HS Torr. On the robustness of semantic segmentation models to adversarial attacks. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition CVPR, pages 888–
897, 2018.
[6] Daniel Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, Konrad
Rieck, and CERT Siemens. Drebin: Effective and explainable detection ofandroid malware in your pocket. In Ndss, volume 14, pages 23–26, 2014.
[7] Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok. Synthesizing
robust adversarial examples. In International Conference on Machine Learning,
pages 284–293, 2018.
[8] Yonatan Belinkov and James Glass. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72, 2019.
[9] Arjun Nitin Bhagoji, Daniel Cullina, Chawin Sitawarin, and Prateek Mittal.
Enhancing robustness of machine learning systems via data transformations. In
2018 52nd Annual Conference on Information Sciences and Systems (CISS),
pages 1–5. IEEE, 2018.
117

[10] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šrndić,
Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European conference on machine learningand knowledge discovery in databases, pages 387–402. Springer, 2013.
[11] Battista Biggio, B Nelson, and P Laskov. Poisoning attacks against support
vector machines. In 29th International Conference on Machine Learning ICML,
pages 1807–1814. ArXiv e-prints, 2012.
[12] Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometerencoding: One hot way to resist adversarial examples. In International Conference on Learning Representations ICLR, 2018.
[13] Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM
Workshop on Artificial Intelligence and Security, pages 3–14, 2017.
[14] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (S&P 2017),
pages 39–57. IEEE, 2017.
[15] Nicholas Carlini and David Wagner. Audio adversarial examples: Targeted attacks on speech-to-text. In 2018 IEEE Security and Privacy Workshops (SPW),
pages 1–7. IEEE, 2018.
[16] Pin-Yu Chen, Yash Sharma, Huan Zhang, Jinfeng Yi, and Cho-Jui Hsieh. Ead:elastic-net attacks to deep neural networks via adversarial examples. In Thirtysecond AAAI conference on artificial intelligence, 2018.
[17] Tarang Chugh, Kai Cao, and Anil K Jain. Fingerprint spoof buster: Use ofminutiae-centered patches. IEEE Transactions on Information Forensics andSecurity, 13(9):2190–2202, 2018.
[18] Hanjun Dai, Hui Li, Tian Tian, Xin Huang, Lin Wang, Jun Zhu, and Le Song.
Adversarial attack on graph structured data. In International Conference on
Machine Learning (ICML), 2018.
[19] Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, Deepak Verma, et al. Adversarial classification. In Proceedings of the tenth ACM SIGKDD international
conference on Knowledge discovery and data mining, pages 99–108. ACM, 2004.
[20] Guneet S. Dhillon, Kamyar Azizzadenesheli, Jeremy D. Bernstein, Jean Kossaifi, Aran Khanna, Zachary C. Lipton, and Animashree Anandkumar. Stochastic activation pruning for robust adversarial defense. In International Conference on Learning Representations ICLR, 2018.
[21] Harris Drucker and Yann Le Cun. Improving generalization performance using
double backpropagation. IEEE Transactions on Neural Networks, 3(6):991–997,
1992.
118

[22] Gintare Karolina Dziugaite, Zoubin Ghahramani, and Daniel M Roy. A studyof the effect of jpg compression on adversarial images. In the International
Society for Bayesian Analysis (ISBA 2016) World Meeting, 2016.
[23] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box
adversarial examples for text classification. In Proceedings of the 56th Annual

Meeting of the Association for Computational Linguistics (Volume 2: Short
Papers), pages 31–36, 2018.

[24] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati,
Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust
physical-world attacks on deep learning models. In Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition CVPR, 2018.
[25] Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner.
Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410,
2017.
[26] Ugo Fiore, Alfredo De Santis, Francesca Perla, Paolo Zanetti, and Francesco
Palmieri. Using networks for improving classification effectiveness in credit cardfraud detection. Information Sciences, 2017.
[27] Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generationof adversarial text sequences to evade deep learning classifiers. In 2018 IEEE
Security and Privacy Workshops (SPW), pages 50–56. IEEE, 2018.
[28] Thomas Gebhart and Paul Schrater. Adversary detection in neural networks
via persistent homology. arXiv preprint arXiv:1711.10056, 2017.
[29] Justin Gilmer, Luke Metz, Fartash Faghri, Sam Schoenholz, Maithra Raghu,
Martin Wattenberg, and Ian Goodfellow. Adversarial spheres. In International
Conference on Learning Representations ICLR, 2018.
[30] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David WardeFarley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages
2672–2680, 2014.
[31] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations ICLR, 2015.
[32] Gaurav Goswami, Nalini Ratha, Akshay Agarwal, Richa Singh, and Mayank
Vatsa. Unravelling robustness of deep learning based face recognition against
adversarial attacks. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
119

[33] Kathrin Grosse, Praveen Manoharan, Nicolas Papernot, Michael Backes, andPatrick McDaniel. On the (statistical) detection of adversarial examples. arXiv
preprint arXiv:1702.06280, 2017.
[34] Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, andPatrick McDaniel. Adversarial perturbations against deep neural networks formalware classification. arXiv preprint arXiv:1606.04435, 2016.
[35] Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, andPatrick McDaniel. Adversarial examples for malware detection. In European
Symposium on Research in Computer Security, pages 62–79. Springer, 2017.
[36] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten.
Countering adversarial images using input transformations. In International
Conference on Learning Representations ICLR, 2018.
[37] Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos,
Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates,
et al. Deep speech: Scaling up end-to-end speech recognition. arXiv preprint
arXiv:1412.5567, 2014.
[38] Warren He, James Wei, Xinyun Chen, Nicholas Carlini, and Dawn Song. Adversarial example defense: Ensembles of weak defenses are not strong. In 11th
{USENIX} Workshop on Offensive Technologies ({WOOT} 17), 2017.
[39] Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed,
Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian
Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal processing magazine, 29, 2012.
[40] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neuralcomputation, 9(8):1735–1780, 1997.
[41] Sungeun Hong, Woobin Im, Jongbin Ryu, and Hyun S Yang. Sspp-dan: Deep
domain adaptation network for face recognition with single sample per person.
In 2017 IEEE International Conference on Image Processing (ICIP), pages 825–
829. IEEE, 2017.
[42] Hossein Hosseini, Yize Chen, Sreeram Kannan, Baosen Zhang, and Radha
Poovendran. Blocking transferability of adversarial examples in black-box learning systems. arXiv preprint arXiv:1703.04318, 2017.
[43] Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International Conference
on Machine Learning ICML, pages 2142–2151, 2018.
[44] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer
networks. In Advances in neural information processing systems NeurIPS 2015,
pages 2017–2025, 2015.
120

[45] Guoqing Jin, Shiwei Shen, Dongming Zhang, Feng Dai, and Yongdong Zhang.
Ape-gan: Adversarial perturbation elimination with gan. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 3842–3846. IEEE, 2019.

[46] Jinkyu Kim and John Canny. Interpretable learning for self-driving cars byvisualizing causal attention. In Proceedings of the IEEE international conferenceon computer vision ICCV, pages 2942–2950, 2017.
[47] Thomas N Kipf and Max Welling. Semi-supervised classification with graphconvolutional networks. In International Conference on Learning Representations (ICLR), 2017.
[48] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine
Learning-Volume 70, pages 1885–1894. JMLR. org, 2017.
[49] Jascha Kolberg, Marta Gomez-Barrero, Sushma Venkatesh, Raghavendra Ramachandra, and Christoph Busch. Presentation attack detection for finger
recognition. In Handbook of Vascular Biometrics, pages 435–463. Springer,
2020.
[50] Jernej Kos, Ian Fischer, and Dawn Song. Adversarial examples for generativemodels. In 2018 IEEE Security and Privacy Workshops (SPW), pages 36–42.
IEEE, 2018.
[51] Rajesh Kumar, Zhang Xiaosong, Riaz Ullah Khan, Jay Kumar, and Ijaz Ahad.
Effective and explainable detection of android malware based on machine learning algorithms. In Proceedings of the 2018 International Conference on Computing and Artificial Intelligence, pages 35–40. ACM, 2018.
[52] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in
the physical world. In International Conference on Learning Representations
ICLR, 2017.
[53] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations ICLR,
2017.
[54] Ruggero Donida Labati, Enrique Muñoz, Vincenzo Piuri, Roberto Sassi, andFabio Scotti. Deep-ecg: Convolutional neural networks for ecg biometric recognition. Pattern Recognition Letters, 126:78–85, 2019.
[55] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.
521(7553):436, 2015.

nature,[56] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradientbased learning applied to document recognition. Proceedings of the IEEE,
86(11):2278–2324, 1998.
121

[57] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit
database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist,
2, 2010.
[58] Hyeungill Lee, Sungyeob Han, and Jungwoo Lee. Generative adversarial trainer: Defense to adversarial perturbations with gan. arXiv preprint
arXiv:1705.03387, 2017.
[59] J Li, S Ji, T Du, B Li, and T Wang. Textbugger: Generating adversarial text
against real-world applications. In 26th Annual Network and Distributed System
Security Symposium, 2019.
[60] Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang
Shi. Deep text classification can be fooled. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4208–4215. AAAI
Press, 2018.
[61] Bin Liang, Hongcheng Li, Miaoqiang Su, Xirong Li, Wenchang Shi, and Xiaofeng Wang. Detecting adversarial image examples in deep networks withadaptive noise reduction. arXiv preprint arXiv:1705.08378, 2017.
[62] Fangzhou Liao, Ming Liang, Yinpeng Dong, Tianyu Pang, Xiaolin Hu, and
Jun Zhu. Defense against adversarial attacks using high-level representation
guided denoiser. In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition CVPR, pages 1778–1787, 2018.
[63] Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu,and Min Sun. Tactics of adversarial attack on deep reinforcement learning
agents. In Proceedings of the 26th International Joint Conference on Artificial
Intelligence, pages 3756–3762. AAAI Press, 2017.
[64] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song. Delving into transferable
adversarial examples and black-box attacks. In International Conference on
Learning Representations ICLR, 2017.
[65] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face
attributes in the wild. In Proceedings of the IEEE international conference on
computer vision CVPR, pages 3730–3738, 2015.
[66] Jiajun Lu, Theerasit Issaranon, and David Forsyth. Safetynet: Detecting andrejecting adversarial examples robustly. In Proceedings of the IEEE International Conference on Computer Vision, pages 446–454, 2017.
[67] Jiajun Lu, Hussein Sibai, Evan Fabry, and David Forsyth. No need to worry
about adversarial examples in object detection in autonomous vehicles. In Pro-

ceedings of the IEEE Conference on Computer Vision and Pattern Recognition
CVPR, 2018.
122

[68] Chunchuan Lyu, Kaizhu Huang, and Hai-Ning Liang. A unified gradient regularization family for adversarial examples. In 2015 IEEE International Conference on Data Mining, pages 301–309. IEEE, 2015.
[69] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, andAdrian Vladu. Towards deep learning models resistant to adversarial attacks.
arXiv preprint arXiv:1706.06083, 2017.
[70] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, andAdrian Vladu. Towards deep learning models resistant to adversarial attacks.
In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings,
2018.
[71] Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 2017 ACM SIGSAC Conference on
Computer and Communications Security, pages 135–147. ACM, 2017.
[72] Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On
detecting adversarial perturbations. In International Conference on Learning
Representations (ICLR), 2017.
[73] Jan Hendrik Metzen, Mummadi Chaithanya Kumar, Thomas Brox, and Volker
Fischer. Universal adversarial perturbations against semantic image segmentation. In Proceedings of the IEEE International Conference on Computer Vision
ICCV, pages 2774–2783. IEEE, 2017.
[74] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.
Distributed representations of words and phrases and their compositionality. In
Advances in neural information processing systems, pages 3111–3119, 2013.
[75] Vahid Mirjalili and Arun Ross. Soft biometric privacy: Retaining biometricutility of face images while perturbing gender. In 2017 IEEE International
joint conference on biometrics (IJCB), pages 564–573. IEEE, 2017.
[76] Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial training
methods for semi-supervised text classification. In International Conference
on Learning Representations ICLR, 2017.
[77] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland,
Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
[78] Andreas Mogelmose, Mohan Manubhai Trivedi, and Thomas B Moeslund.
Vision-based traffic sign detection and analysis for intelligent driver assistancesystems: Perspectives and survey. IEEE Transactions on Intelligent Transportation Systems, 13(4):1484–1497, 2012.
123

[79] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition (CVPR), pages 5115–5124, 2017.
[80] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal
Frossard. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition CVPR, pages 1765–1773,
2017.
[81] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedingsof the IEEE conference on computer vision and pattern recognition CVPR, pages
2574–2582, 2016.
[82] James Newsome, Brad Karp, and Dawn Song. Polygraph: Automatically generating signatures for polymorphic worms. In 2005 IEEE Symposium on Security
and Privacy (S&P’05), pages 226–241. IEEE, 2005.
[83] Linh Nguyen, Sky Wang, and Arunesh Sinha. A learning and masking approachto secure learning. In International Conference on Decision and Game Theory
for Security, pages 453–464. Springer, 2018.
[84] Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay
Celik, and Ananthram Swami. Practical black-box attacks against machinelearning. In Proceedings of the 2017 ACM on Asia Conference on Computerand Communications Security, pages 506–519. ACM, 2017.
[85] Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay
Celik, and Ananthram Swami. The limitations of deep learning in adversarial
settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&P), pages 372–387. IEEE, 2016.
[86] Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang.
Crafting adversarial input sequences for recurrent neural networks. In MILCOM
2016-2016 IEEE Military Communications Conference, pages 49–54. IEEE,
2016.
[87] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram
Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (S&P 2016),
pages 582–597. IEEE, 2016.
[88] Roberto Perdisci, David Dagon, Wenke Lee, Prahlad Fogla, and Monirul Sharif.
Misleading worm signature generators using deliberate noise injection. In 2006
IEEE Symposium on Security and Privacy (S&P’06), pages 15–pp. IEEE, 2006.
124

[89] Yunchen Pu, Zhe Gan, Ricardo Henao, Xin Yuan, Chunyuan Li, Andrew
Stevens, and Lawrence Carin. Variational autoencoder for deep learning ofimages, labels and captions. In Advances in neural information processing systems, pages 2352–2360, 2016.
[90] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Pro-

ceedings of the IEEE conference on computer vision and pattern recognition(CVPR), pages 7263–7271, 2017.

[91] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards
real-time object detection with region proposal networks. In Advances in neural
information processing systems NeurIPS 2015, pages 91–99, 2015.
[92] Yafeng Ren and Donghong Ji. Neural networks for deceptive opinion spam
detection: An empirical study. Information Sciences, 385:213–224, 2017.
[93] Andrew Slavin Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input
gradients. In Thirty-second AAAI conference on artificial intelligence, 2018.
[94] Andras Rozsa, Manuel Günther, Ethan M Rudd, and Terrance E Boult. Are
facial attributes adversarially robust? In 2016 23rd International Conference
on Pattern Recognition (ICPR), pages 3121–3127. IEEE, 2016.
[95] Andras Rozsa, Manuel Günther, Ethan M Rudd, and Terrance E Boult. Facial
attributes: Accuracy and adversarial robustness. Pattern Recognition Letters,
2017.
[96] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean
Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.
Imagenet large scale visual recognition challenge. International Journal of Computer Vision IJCV, 115(3):211–252, 2015.
[97] Swami Sankaranarayanan, Arpit Jain, Rama Chellappa, and Ser Nam Lim.
Regularizing deep networks using efficient layerwise adversarial training. In
Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[98] Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, ChristophStuder, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted cleanlabel poisoning attacks on neural networks. In Advances in Neural Information
Processing Systems, pages 6103–6113, 2018.
[99] Uri Shaham, Yutaro Yamada, and Sahand Negahban. Understanding adversarial training: Increasing local stability of supervised models through robust
optimization. Neurocomputing, 307:195–204, 2018.
[100] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition.
125

In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 1528–1540. ACM, 2016.
[101] Sijie Shen, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. Fooling neural networks in face attractiveness evaluation: Adversarial examples withhigh attractiveness score but low subjective score. In 2017 IEEE Third International Conference on Multimedia Big Data (BigMM), pages 66–69. IEEE,
2017.
[102] Richard Shin and Dawn Song. Jpeg-resistant adversarial images. In NeurIPS
2017 Workshop on Machine Learning and Computer Security, 2017.
[103] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George
Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neuralnetworks and tree search. nature, 529(7587):484, 2016.
[104] Dawn Song, Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir
Rahmati, Florian Tramer, Atul Prakash, and Tadayoshi Kohno. Physical adversarial examples for object detectors. In 12th {USENIX} Workshop on Offensive
Technologies ({WOOT} 18), 2018.
[105] Lance Spitzner. Honeypots: Catching the insider threat. In 19th Annual
Computer Security Applications Conference, 2003. Proceedings., pages 170–179.
IEEE, 2003.
[106] Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack
for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 2019.
[107] Yanmin Sun, Andrew KC Wong, and Mohamed S Kamel. Classification of
imbalanced data: A review. International Journal of Pattern Recognition and
Artificial Intelligence, 23(04):687–719, 2009.
[108] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew
Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2818–2826, 2016.
[109] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks.
In International Conference on Learning Representations ICLR, 2014.
[110] Pedro Tabacof, Julia Tavares, and Eduardo Valle. Adversarial images for variational autoencoders. In Advances in neural information processing systems
NeurIPS Workshop 2016, 2016.
126

[111] Rohan Taori, Amog Kamsetty, Brenton Chu, and Nikita Vemuri. Targeted
adversarial examples for black box audio systems. In 2019 IEEE Security andPrivacy Workshops (SPW), pages 15–20. IEEE, 2019.
[112] Simen Thys, Wiebe Van Ranst, and Toon Goedemé. Fooling automated surveillance cameras: adversarial patches to attack person detection. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Workshops, 2019.

[113] Shixin Tian, Guolei Yang, and Ying Cai. Detecting adversarial examples
through image transformation. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[114] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan
Boneh, and Patrick McDaniel. Ensemble adversarial training: Attacks anddefenses. In International Conference on Learning Representations ICLR, 2018.
[115] Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018, 2018.
[116] Xuyu Wang, Lingjun Gao, Shiwen Mao, and Santosh Pandey. Deepfi: Deeplearning for indoor fingerprinting using channel state information. In 2015 IEEE
wireless communications and networking conference (WCNC), pages 1666–1671.
IEEE, 2015.
[117] Jonathan Wu, Prakash Ishwar, and Janusz Konrad. Two-stream cnns forgesture-based verification and identification: Learning user style. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 42–50, 2016.

[118] Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song.
Spatially transformed adversarial examples. In International Conference on
Learning Representations (ICLR), 2018.
[119] Huang Xiao, Battista Biggio, Gavin Brown, Giorgio Fumera, Claudia Eckert,
and Fabio Roli. Is feature selection secure against training data poisoning? In
International Conference on Machine Learning ICML, pages 1689–1698, 2015.
[120] Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In 25th Annual Network and Dis-

tributed System Security Symposium, NDSS 2018, San Diego, California, USA,
February 18-21, 2018.

[121] Qingchen Zhang, Laurence T Yang, Zhikui Chen, and Peng Li. A survey on
deep learning for big data. Information Fusion, 42:146–157, 2018.
127

[122] Stephan Zheng, Yang Song, Thomas Leung, and Ian Goodfellow. Improving
the robustness of deep neural networks via stability training. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition CVPR,
pages 4480–4488, 2016.
[123] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired imageto-image translation using cycle-consistent adversarial networks. In Proceedingsof the IEEE International Conference on Computer Vision ICCV, 2017.
[124] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic
net. Journal of the royal statistical society: series B (statistical methodology),67(2):301–320, 2005.
[125] Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM
SIGKDD International Conference on Knowledge Discovery & Data Mining,
pages 2847–2856. ACM, 2018.

128

My Publications

Refereed journal papers:[2] Dang Duy Thang and Toshihiro Matsui, “Adversarial Examples Identificationin an End-to-end System with Image Transformation and Filters”, IEEE ACCESS(SCIE Q1, IF: 4.098). Accepted.
[1] Dang Duy Thang and Toshihiro Matsui, “Search Space of Adversarial Perturbations Against Image Filters” , International Journal of Advanced Computer
Science and Applications (ESCI, IF: 1.324), Vol. 11 No 1, pages 11-19, 2020.

Refereed conference papers:[2] Dang Duy Thang and Toshihiro Matsui, “A Label-based Approach for Automatic Identifying Adversarial Examples with Image Transformation”, In Proceedings of the Seventh International Symposium on Computing and Networking (CANDAR’19), Nagasaki, Japan, November 26-29, pages 112-120. IEEE, 2019 (acceptance
rate 14.9%).
[1] Dang Duy Thang and Toshihiro Matsui, “Automated Detection System for
Adversarial Examples with High-Frequency Noises Sieve” , In Proceedings of the
11th International Symposium on Cyberspace Safety and Security (CSS), Guangzhou,
China, December 1-3, pages 348-362. Springer, 2019 (acceptance rate 26%).

Non-reviewed papers:[3] Dang Duy Thang, Taisei Kondo, Toshihiro Matsui, “A Label-based System
for Detecting Adversarial Examples by Using Low Pass Filters”, Computer Security
Symposium, Nagasaki, Japan, October 21-24, pages 1356-1363, 2019.
[2] Taisei Kondo, Dang Duy Thang, Toshihiro Matsui、“Creation of Adversarial
Example Attack that is Difficult to Protect by LPF”, Computer Security Symposium,
Nagasaki, Japan, October 21-24, pages 1364-1369, 2019.
[1] Dang Duy Thang, Toshihiro Matsui, “White-box attack on Google Machine
Learning system” , Poster Session, the 13th International Workshop on Security,
Tohoku University, Japan, 2018. Best Poster Award

129

130