Year: 2018

Abstract:
Adversarial examples remain an issue for contemporary neural networks. This paper draws on Background Check (Perello-Nieto et al., 2016), a technique in model calibration, to assist two-class neural networks in detecting adversarial examples, using the one dimensional difference between logit values as the underlying measure. This method interestingly tends to achieve the highest average recall on image sets that are generated with large perturbation vectors, which is unlike the existing literature on adversarial attacks (Cubuk et al., 2017). The proposed method does not need knowledge of the attack parameters or methods at training time, unlike a great deal of the literature that uses deep learning based methods to detect adversarial examples, such as Metzen et al. (2017), imbuing the proposed method with additional flexibility.

heading: 1 INTRODUCTION AND RELATED WORK
text: Adversarial examples are specially crafted input instances generated by adversarial attacks. The term was introduced by Szegedy et al. (2013) in the context of image classification. These attacks generate, or manipulate data, to achieve poor performance when classified by neural networks, which poses existential questions about their usage in high stakes security critical applications. Since they were introduced, there have been many papers that have introduced novel attack methods and other papers that attempt to combat those attacks. For instance, Goodfellow et al. (2014) introduces the fast gradient sign method (FGSM), and Papernot et al. (2016c) proposes a method based on modifying the gradient of the softmax function as a defence.
Adversarial attacks can be identified into various classes such as white box and black box, where in the former, the attack has full knowledge of all model parameters. Examples created by these attacks can be false positives or false negatives. In the case of images, they can be nonsensical data (e.g. noise classified as a road sign) or clear cut (e.g. a visually clear cat, classified as a road sign). These attacks can be non-targeted or targeted such that the classifier chooses a specific class for the adversarial example. Various adversarial defences exist, some based on deep learning techniques and others on purely distributional techniques. Similar work on adversarial defences has been done by Grosse et al. (2017), in which the network is trained on specific attack types and parameters with an additional outlier class for adversarial examples. A multi-dimensional statistical test over the maximum mean discrepancy and the energy distance on input features is then used to classify instances as adversarial. Other work has been done by Bradshaw et al. (2017), where Gaussian Processes are placed on top of conventional convolutional neural network architectures, with radial basis kernels, imbuing the neural network with a way of understanding its own perceptual limits. The authors find that the network becomes more resistant to adversarial attack. The work that follows continues in a similar vein to both of these methods. Some methods such as Metzen et al. (2017) use sub-units of deep learning architectures to detect adversarial instances.
Calibration is a technique of converting model scores, normally, through application of a post processing function, to probability estimates. Background Check is a method to yield probability estimates, via a set of explicit assumptions, in regions of space where no data has been observed. In this work, Background Check is useful in producing calibrated probabilities for adversarial data that often exists in regions where no training and test data has been seen. Reliable probability estimates can then be measured by calibration and refinement loss. Various calibrating procedures exist such as binning, logistic regression, isotonic regression and softmax. Jordan et al. (1995) demonstrates
the logistic function is optimal when the class-conditional densities are Gaussians with unit variance. Softmax extends this to multi-variate Gaussian densities with unit variance.
Calibration of neural network models has been performed by Guo et al. (2017), using a method called Temperature Scaling, that modifies the gradient of the softmax function allowing softmax to calibrate densities with non-unit variance. The authors perform this calibration after noticing that calibration loss for neural networks has increased in recent years. When adversarial attacks against neural networks are brought into perspective, a problem arises for existing calibration techniques, which is the question of mapping adversarial logit scores to reliable probability estimates (which should be zero for a successful adversarial attack). In this work, a method is demonstrated that uses Background Check to identify adversarial attacks.

heading: 2 CALIBRATION
text: A classifier is said to be well calibrated, if, as the number of predictions approaches infinity, the proportion of outcomes given probability p, occur p fraction of the time. Denoting x1, x2, ..., xn as data-set instances and y1, y2, ..., yn as their corresponding ground truth class labels, a scoring classifier s = f(xi) has a calibrating function µ applied to it yielding µ(f(xi)). Perfect calibration is defined as the expectation si = f(xi) such that si = E[Y |f(X) = si] where random variables X , Y denote the features and class label of a uniformly randomly drawn instance from the dataset respectively, such that Y = 1, Y = 0 represent an individual positive and negative instance, respectively.
To visualize calibration performance of a classifier, Murphy & Winkler (1977) plots the observed frequency of an event against the predicted frequency yielding calibration curves. Calibration curves plot the observed relative frequency against the predicted probability for all test data. Perfect calibration occurs if the calibration curve exactly fits the identity line. Refinement loss measures the difference between a probability estimate and zero or one.
This can be combined with a frequency distribution, giving an indication of spread, which is useful if there are few events associated with a particular probability. The notion of refinement is also useful when considering calibration. By considering the crude constant classifier, which predicts the probability corresponding to the class distribution for all inputs, it is clear that this calibration estimate is perfectly calibrated. However, an intuitively more valuable calibration estimate, is one which predicts a value closer to either zero or one. For this reason, DeGroot & Fienberg (1981) suggests measuring refinement loss, which measures the distance of the classifiers probability estimates to either zero or one. Together, calibration and refinement loss make up the Brier Score. Kull & Flach (2015) defines calibration and refinement loss.
Calibration loss = E[d(S,C)] is the loss due to the expected difference between the model score S and the proportion of positives among instances (observed relative frequency) with the same score.
Refinement loss = E[d(C, Y )] is the loss due to the presence of instances from multiple classes among instances with the same estimate S. In the worst case, this clearly reduces to the crude constant classifier mentioned above.
An instance of recent work related to calibration is Beta calibration. Beta calibration Kull et al. (2017) is based on the beta distribution which includes functions such as the logit, sigmoid and identity. This allows it to calibrate scores produced by models such as naive bayes, which biases its scores towards extremities when the assumption of feature independence is not met, using the inverse sigmoid or logit function, a function which is not in softmax’s repertoire.

heading: 3 ADVERSARIAL ATTACKS
text: In the context of adversarial attacks, the optimization problem that is often formulated to construct adversarial examples is shown.
argmin δx
||δx||s.t.F (X + δx) = Y ∗
An adversarial example X∗ is constructed by adding a perturbation vector δx, where Y ∗ is the desired adversarial output and X + δx ∈ [0, 1] where [0, 1] is the upper and lower bound of a well
formed example, subject to the constraints above. Intuitively, these constraints coerce the network into mis-classifying each example with a minimal perturbation vector. Example distance metrics that measure the size of the perturbation include Lp metrics, as well as the PASS score Rozsa et al. (2016), a metric designed to better reflect notions of psycho-physical similarity than Lp metrics. Adversarial attack methods include L-BFGS from Szegedy et al. (2013), which uses linear trial and error to find a c for each data instance such that cmultiplied by an arbitrary small perturbation vector mis-classifies the instance. FGSM builds on the L-BFGS method, replacing expensive linear search with gradient descent to find the perturbation vector. This uses the following optimization strategy to create the perturbation vector η, such that x is the input to the model, y is the predicted class, θ are the model parameters and J(θ, x, y) is the cost function. Ox indicates the derivative of the cost function is taken with respect to the input x. The sign function takes the sign of the resulting derivative. The adversarial example is then constructed as x′ = x+ η.
η = sign(Ox, J(θ, x, y))
A momentum term can be added to the gradient descent process to yield the work of Dong et al. (2018). The BIM attack by Kurakin et al. (2016), uses a smaller noise vector produced by FGSM, applied iteratively, before a clip operation is applied to the resulting image, keeping it within the maximum image pixel values after each iteration.
JSMA is a forward derivative approach, by Papernot et al. (2016b), which uses a Jacobian matrix, to produce an adversarial saliency map, which indicate the features that when (positively or negatively) perturbed, most efficiently achieve a desired network output. DeepFool (Moosavi Dezfooli et al., 2016) finds an image that is at a minimal distance to the decision boundary from the proposed example, to another target class which isn’t the source class, treating multi-class classifiers as combinations of binary affine classifiers.

heading: 4 BACKGROUND CHECK
text: Background Check, introduced by Perello-Nieto et al. (2016), is the calibrating procedure that our proposed method uses to defend neural networks from adversarial attacks. Background Check takes as input, the scores from logit vectors and maps them to probabilities, replacing softmax after training. Background Check provides a framework to classify regions where no previous data has been seen as background regions, that in the context of the proposed method will be classified as regions where adversarial data may lie.
Background Check also provides a framework to resolve ambiguity inherent in a single value representing probability. More specifically, Background Check provides two values to represent a single probability value of a data instance. One value represents distance from the data density, and another represents the certainty of a particular class. This approach avoids overloading the meaning of a single number representing probability. More specifically, an uncertainty of 1/n for all classes, could represent an instance very close to the decision boundary or very far away from training data. On the other hand, an output of zero, could represent a point very far away from training data or a classification that the data is definitely not that particular class.

heading: 4.1 BACKGROUND CHECK IN MODELLING SPARSE DATA
text: Background Check introduces an additional outlier background class, b, representing regions of space where data is sparse or non-existent. Then, a foreground class is introduced, f . This represents regions of space where data is plentiful or dense, i.e. data from any class apart from the background class, is abundant. b is introduced as an additional class whilst f is kept as a reference class. Every instance x necessarily belongs to either f or b. P (b|x) = 0 and P (f |x) = 1 refers to absolute certainty that the instance belongs to one of the classes with sufficient training data, where P (C|x) is a conditional probability measure. The ratio of the two conditional measures defines the reliability factor r(x).
r(x) = P (f |x) P (b|x) = P (f1|x) + ...+ P (fn|x) P (b|x) = P (f, x)P (x) P (b, x)P (x) = P (x, f) P (x, b)
If r(x) ≤ 1, the classification that the reliability factor indicates is b, else if r(x) > 1 then the classification is f . P (x, f) and P (x, b) are referred to as the foreground and background densities. Furthermore, the relative foreground and background densities can be defined, qf (x) and qb(x).
qf (x) = P (x, f)
maxx P (x, f) , qb(x) =
P (x, b)
maxx P (x, f)
Intuitively, the relative density outputs the proportion of f or b, at the point in space corresponding to the instance x being evaluated. Simple dividing qf (x) by qb(x) yields r(x).
4.1.1 JUSTIFICATION OF qb(x)
To construct qb(x), four inductive biases, in increasing strength, are given. This work only uses the third inductive bias.
1. Inductive bias 1 : qb(x) is a function of qf (x). This is justified, by the idea that with no other information, there is no reason to assign different background densities to points with the same foreground density. The domain knowledge informs the function used µ : [0, 1] −→ [0,∞).
2. Inductive bias 2 : monotonicity of µ, that is, when moving to a region with higher foreground density the background increases or decreases.
3. Inductive bias 3 : an affine bias, i.e. µ(x) = ax + b or by replacing a and b: µ(x) = (1− x)µ(0) + xµ(1).
4. Inductive bias 4 : constant background ie µ(0), µ(1) = 0.5.

heading: 4.1.2 IMPLEMENTATION OF BACKGROUND CHECK
text: The key notion in Background Check is the implementation of the inductive bias that shapes qb. This implementation is provided in two different different ways.
1. BCD : referred to as the discriminative approach, involves generating artificial background instances around foreground data and then training a binary discriminative classifier to separate them. The instances are generated in a hypercube or a hypersphere, such that the background is half as dense as maxxP (x, f).
2. BCF : referred to as the familiarity approach, this involves fitting a one class model on the foreground data to obtain qf , then using an inductive bias to obtain qb. The data, x, that is being fit must have an underlying measure. The familiarity factor r(x) can be found, allowing, the posterior probabilities P (b|x) and P (f |x) to be computed.
The implementation that the proposed method in this work uses, is the BCF method for its speed in high dimensional spaces. The measure underlying the space was the one-dimensional L1 difference between elements of the logit vector. For instance, given a two-class score vector [−5, 5], then the score difference is 10. This measure represents the distance of a data point to the softmax decision boundary. The one-class model used to fit qf (x) was a gamma function optimized using maximum likelihood estimation. To establish the link from qf (x) to qb(x), the third inductive bias was used with µ(0) = 1 and µ(1) = 0, with domain knowledge informing the use of a power value. This link manifests itself in the equation below.
qb = (1− qf (x))5 × µ(0) + qf (x)× µ(1)

heading: 5 EXPERIMENTS
text: One neural network for each attack type, parameter combination and dataset, is trained with the Adam optimizer (Kingma & Ba (2014)). A batch size of 256 and a learning rate of 0.001 is employed. The biases of the neurons are set to 0 and the weights are sampled uniformly at random
from the interval [0, 1]. Each neural network is tasked with classifying a variety of paired class combinations from the CIFAR10 dataset. These pairs were: dog versus plane (DVP), fish vs ship (FVS) and airplane vs horse (AVH). Regularization techniques such as L2 regularization and dropout, with p = 0.5 were applied to the networks. The training vs test ratio is 5:1, such that each class has exactly 1000 test images and 5000 training images. The images were pre-processed such that all image pixel values were floating point values between zero and one, rather than values between 0 and 255. Seven different adversarial attacks, from the cleverhans API (Papernot et al., 2016a), were tested against the networks with a variety of parameter combinations. The adversarial images were generated from test data. The attacks were all white-box attacks and performed on the network which included a final softmax layer in its structure. The final two-class average recall of each network on the validation set of the network was always above 80% after only 300 iterations over the training data. The attacks had different effects on each image, for identical parameter settings and as such, the parameters were subjectively chosen based on whether the image fell into one of four classes. Some attacks, due to a mixture of constraints and difficulty in searching the adversarial image space, only had images in one or two of the available classes.
1. Large - Image consists entirely of noise.
2. Typical - Moderate noise levels, underlying image recognizable.
3. Small - Recognizable noise, but clear image.
4. Very Small - No noticeable noise, clear image.
For instance, BIM an iterative method had 10 iterations, with : 0.8 and i : 0.05, for a large perturbation vector, yet the method from Miyato et al. (2015), a non-iterative method had : 12.0. The full parameter settings are listed in the appendix.
The average recall, defined in the equation below, rather than accuracy, is evaluated due to the presence of varying class proportions. The 2-class average recall is on the two classes in CIFAR-10.
The 3-class average recall includes the adversarial class.
( C∑ i TPi TPi + FNi )× 1/C

heading: 6 RESULTS
text: The table of results demonstrates that large perturbation vectors, associate with a mean reduction in average recall of 11.6, whereas for very small perturbation vectors, the mean reduction in average recall is 35.7. Typical perturbation vectors have a mean reduction in average recall of 6.9. In all cases except for two, the average recall decreases. It is clear that the adversarial TPR generally increases as the size of the perturbation vector increases. In particular, three out of the five large perturbation vectors achieve a TPR on the adversarial class of 100%. All models achieve higher average recall than the baseline, which was simply a strategy that with uniform randomness guesses the class.
In order to visualize areas where Background Check assigns foreground and background densities, it is helpful to construct histograms. When constructing histograms of the test, training and adversarial L1 logit differences, three categories were established over the space.
1. Adversarial examples in a distinct cluster, closer to the decision boundary, than the training and test data.
2. Adversarial examples scattered amongst the training/test data.
3. Adversarial examples in a distinct cluster, further from the decision boundary, than the training and test data.
The JSMA and DeepFool attacks found logit differences smaller than the test and training logits, yet still high enough to yield a significant confidence level when applied to the softmax function. The Madry, Momentum and BIM attacks produced logit differences far higher than the test and training logit differences. However, some attacks found logit differences within the test and training distributions.

heading: 7 DISCUSSION
text: Background check models the data density over the logit differences. It is clear from the figures in the appendix that the adversarial attacks find examples in regions where the test and training data do not exist. Thus, Background Check improves the discriminative performance of the classifier when dealing with adversarial examples with large perturbation vectors. These resulting images are noisy and hard to allocate a non-ambiguous class, though we argue that these images can occur and be just as damaging in the real world and as such need to be defended against. It would be useful to follow on from this method with an analysis of generative probability estimation and a corresponding measure of calibration and refinement loss.
Promising future research would scale up the logit difference metric underlying Background Check to higher dimensional spaces to deal with a full ten classes to allow for comparability to mainstream literature on adversarial defences. Possible metrics that can underlie Background Check could use the energy distance. In addition, Background Check could be applied to each layer of a neural network. However, this must be setup such that it does not interfere with the ability of the neural network to generalize.
The performance of Background Check can be measured in different ways. For example, Condessa et al. (2017) proposes performance measures for classification systems with the rejection option. These measures consist of metrics such as the non-rejected accuracy, which measures the ability of the classifier to accurately classify non-rejected samples. The classification quality, which measures the correct decision making of the classifier with the rejector and finally, the rejection quality, which measures the ability to concentrate all mis-classified samples onto the set of rejected samples.

heading: 8 CONCLUSION
text: A novel approach to defending neural networks against adversarial attacks has been established. This approach intersects two previously unrelated fields of machine learning, calibration and adversarial defences, using the principles underlying Background Check. This work demonstrates that adversarial attacks, produced as a result of large perturbations of various forms, can be detected and assigned to an adversarial class. The larger the perturbation, the easier it was for the attacks to be detected.

heading: A PARAMETERS
text: is the size of the perturbation, i is the change of the size each iteration and nb is the number of iterations.
B EXTRA DISTRIBUTIONS

Review summaries:
This paper addresses adversarial detection through the absolute-value difference between the two logit vector values of a DNN binary classifier, with one class associated with normal data and the other with adversarial data. Assignment of examples to an "adversarial" class is problematic in that adversarial examples are typically generated in regions for which training data is very sparse. To cope with this, the authors propose use of the Background Check calibration techniques recently proposed by Perello-Nieto et al. (ICDM 2016).