Title: THERMOMETER ENCODING: ONE HOT WAY TO RESIST ADVERSARIAL EXAMPLES

Abstract: It is well known that it is possible to construct "adversarial examples" for neural networks: inputs which are misclassified by the network yet indistinguishable from true data. We propose a simple modification to standard neural network architectures, thermometer encoding, which significantly increases the robustness of the network to adversarial examples. We demonstrate this robustness with experiments on the MNIST, CIFAR-10, CIFAR-100, and SVHN datasets, and show that models with thermometer-encoded inputs consistently have higher accuracy on adversarial examples, without decreasing generalization. State-of-the-art accuracy under the strongest known white-box attack was increased from 93.20% to 94.30% on MNIST and 50.00% to 79.16% on CIFAR-10. We explore the properties of these networks, providing evidence that thermometer encodings help neural networks to find more-non-linear decision boundaries.

1. INTRODUCTION AND RELATED WORK:

Adversarial examples are inputs to machine learning models that are intentionally designed to cause the model to produce an incorrect output. The term was introduced by Szegedy et al. (2014) in the context of neural networks for computer vision. In the context of spam and malware detection, such inputs have been studied earlier under the name evasion attacks (Biggio et al., 2013) . Adversarial examples are interesting from a scientific perspective, because they demonstrate that even machine learning models that have superhuman performance on I.I.D. test sets fail catastrophically on inputs that are modified even slightly by an adversary. Adversarial examples also raise concerns in the emerging field of machine learning security because malicious attackers could use adversarial examples to cause undesired behavior (Papernot et al., 2016) .

Unfortunately, there is not yet any known strong defense against adversarial examples. Adversarial examples that fool one model often fool another model, even if the two models are trained on different training examples (corresponding to the same task) or have different architectures (Szegedy et al., 2014) , so an attacker can fool a model without access to it. Attackers can improve their success rate by sending inputs to a model, observing its output, and fitting their own own copy of the model to the observed input-output pairs (Papernot et al., 2016) . Attackers can also improve their success rate by searching for adversarial examples that fool multiple different models-such adversarial examples are then much more likely to fool the unknown target model (Liu et al., 2016) . Szegedy et al. (2014) proposed to defend the model using adversarial training (training on adversarial examples as well as regular examples) but it was not feasible to generate enough adversarial examples in the inner loop of the training process for the method to be effective at the time. Szegedy et al. (2014) used a large number of iterations of L-BFGS to produce their adversarial examples. Goodfellow et al. (2014) developed the fast gradient sign method (FGSM) of generating adversarial examples and demonstrated that adversarial training is effective for reducing the error rate on adversarial examples. A major difficulty of adversarial training is that it tends to overfit to the method of adversarial example generation used at training time. For example, models trained to resist FGSM adversarial examples usually fail to resist L-BFGS adversarial examples. Kurakin et al. (2016) introduced the basic iterative method (BIM) which lies between FGSM and L-BFGS on a curve trading speed for effectiveness (the BIM consists of running FGSM for a medium number of iterations). Adversarial training using BIM still overfits to the BIM, unfortunately, and different iterative methods can still successfully attack the model. Recently, Madry et al. (2017) showed that adversarial training using adversarial examples created by adding random noise before running BIM results in a model that is highly robust against all known attacks on the MNIST dataset. However, it is less effective on more complex datasets, such as CIFAR. A strategy for training networks which are robust to adversarial attacks across all contexts is still unknown. In this work, we demonstrate that thermometer code discretization and one-hot code discretization of real-valued inputs to a model significantly improves its robustness to adversarial attack, advancing the state of the art in this field.

2. INPUT DISCRETIZATION:

We propose to break the linear extrapolation behavior of machine learning models by preprocessing the input with an extremely nonlinear function. This function must still permit the machine learning model to function successfully on naturally occurring inputs. The recent success of the PixelRNN model (Oord et al., 2016) has demonstrated that one-hot discrete codes for 256 possible values of color pixels are effective representations for input data. Other extremely nonlinear functions may also defend against adversarial examples, but we focused attention on vector-valued discrete encoding as our nonlinear function because of the evidence from PixelRNN that it would support successful machine learning.

Images are often encoded as a 3D tensor of integers in the range [0, 255] . The tensor's three dimensions correspond to the image's height, width, and color channels (e.g. three for RGB, one for greyscale). Each value represents an intensity value for a given color at a given horizontal/vertical position. For classification tasks, these values are typically normalized to floating-point approximations in the range (0, 1). Input discretization refers to the process of separating these continuousvalued pixel inputs into a set of non-overlapping buckets, which are each mapped to a fixed binary vector. Past work, for example depth-color-squeezing (Xu et al., 2017) , has explored what we will refer to as quantization of inputs as a potential defense against adversarial examples. In that approach, each pixel value is mapped to the low-bit version of its original value, which is a fixed scalar. The key novel aspect of our approach is that rather than replacing a real number with a number of low bit depth, we replace each real number with a binary vector. Different values of the real number activate different bits of the input vector. Multiplying the input vector by the network's weights thus performs an operation similar to an embedding lookup in a language model, so different input values actually use different parameters of the network. To avoid confusion, we will consistently refer to scalar-toscalar precision reduction as quantization and scalar-to-vector encoding schemes as discretization throughout this work. A comparison of these techniques can be seen in Table 1 . Note that, unlike depth-color-squeezing, discretization makes a meaningful change to the model, even when it is configured to use enough discretization levels to avoid losing any information from a traditionally formatted computer vision training set; discretizing each pixel to 256 levels will preserve all of the information contained in the original image. Table 1 : Examples mapping from continuous-valued inputs to quantized inputs, one-hot codes, and thermometer codes, with ten evenly-spaced levels.

2.1. DISCRETIZATION AS A DEFENSE:

In Goodfellow et al. (2014) , the authors provide evidence that several network architectures, including LSTMs (Hochreiter & Schmidhuber, 1997) , sigmoid networks (Han & Moraga, 1995) , and maxout networks (Goodfellow et al., 2013) , are vulnerable to adversarial examples due to the empirical fact that, when trained, the loss function of these networks tends to be highly linear with respect to its inputs.

We briefly recall the reasoning of Goodfellow et al. (2014) . Assume that we have a logistic regressor with weight matrix w. Consider an image x ∈ R n which is perturbed into x = x + η by some noise η such that η ∞ ≤ ε for some ε. The probability that the model assigns to the true class is equal to:

L( x) = σ(w x) = σ(w (x + η)) = σ(w x + w η)

If the perturbation η is adversarial, such as in the case where

η i := ε • sign ∂L(x)

∂xi , then the input to the sigmoid is increased by ε • n. If n is large, as is typically the case in images and other highdimensional spaces of interest, this linearity implies that even imperceptibly small values of ε can have a large effect on the model's prediction, making the model vulnerable to adversarial attacks.

Though neural networks in principle have the capacity to represent highly nonlinear functions, networks trained via stochastic gradient descent on real-world datasets tend to converge to mostly-linear solutions. This is illustrated in the empirical studies conducted by Goodfellow et al. (2014) . One hypothesis proposed to explain this phenomenon is that the nonlinearities typically used in networks are either piecewise linear, like ReLUs, or approximately linear in the parts of their domain in which training takes place, like the sigmoid function.

One potential solution to this problem is to use more non-linear activation functions, such as quadratic or RBF units. Indeed, it was shown by Goodfellow et al. (2014) that such units were more resistant to adversarial perturbations of their inputs. However, these units are difficult to train, and the resulting models do not generalize very well (Goodfellow et al., 2014) , sacrificing accuracy on clean examples. As an alternative to introducing highly non-linear activation functions in the network, we propose applying a non-differentiable and non-linear transformation (discretization) to the input, before passing it into the model. A comparison of the input to the model under various regimes can be seen in Figure 1 , highlighting the strong non-linearity of discretization techniques. Figure 1 : Comparison of regular inputs, quantized inputs, and discretized inputs (16 levels, projected to one dimension) on MNIST, adversarially trained with ε = 0.3. The x-axis represents the true pixel value of the image, and the y-axis represents the value that is passed as input to the network after the input transformation has been applied. For real-valued inputs, the inputs to the network are affected linearly by perturbations to the input. Quantized inputs are also affected approximately linearly by perturbations where ε is greater than the bucket width. Discretizing the input, and then using learned weights to project the discretized value back to a single scalar, we see that the model has learned a highly non-linear function to represent the input in a fashion that is effective for resisting the adversarial perturbations it has seen. When starting at the most common pixel-values for MNIST, 0 and 1, any perturbation of the pixels (where ε ≤ 0.3) has barely any effect on the input to the network.

2.2. TYPES OF DISCRETIZATION:

In this work we consider two approaches to constructing discretized representations f (x) of the input image x. Assume for the sake of simplicity that the entries of x take values in the continuous domain [0, 1].

We first describe a quantization function b.

Choose 0 < b 1 < b 2 < • • • < b k = 1 in some fashion.

(In this work, we simply divide the domain evenly, i.e. b i = i k .) For a real number θ ∈ [0, 1] define b(θ) to be the largest index α ∈ {1, . . . , k} such that θ ≤ b α .

2.2.1. ONE-HOT ENCODINGS:

For an index j ∈ {1, . . . , k} let χ(j) ∈ R k be the indicator or one-hot vector of j, i.e.,

χ(j) l = 1 if l = j 0 otherwise.

The discretization function is defined pixel-wise for a pixel i ∈ {1, . . . , n} as:

f onehot (x i ) = χ (b(x i )) .

One-hot encodings are simple to compute and understand, and are often used when it is necessary to represent a categorical variable in a neural network. However, one-hot encodings are not well suited for representing categorical variables with an interpretation of ordering between them. Note that the ordering information between two pixels x i and x j is lost by applying the transformation f onehot ; for a pair of pixels i, j whenever b(x i ) = b(x j )), we see:

χ(b(x i )) 2 = χ(b(x j )) 2 = 1.

In the case of pixel values, this is not a good inductive bias, as there is a clear reason to believe that neighboring buckets are more similar to each other than distant buckets.

2.2.2. THERMOMETER ENCODINGS:

In order to discretize the input image x without losing the relative distance information, we propose thermometer encodings. For an index j ∈ {1, . . . , k}, let τ (j) ∈ R k be the thermometer vector defined as

τ (j) l = 1 if l ≥ j 0 otherwise.

Then the discretization function f is defined pixel-wise for a pixel i ∈ {1, . . . , n} as:

f therm (x) i = τ (b(x i )) = C(f onehot (x i ))

where C is the cumulative sum function,

C(c) l = l j=0 c l .

Note that the thermometer encoding preserves pairwise ordering information, i.e., for pixels i,

j if b(x j ) = b(x k ) and x i < x j then τ (b(x i )) 2 < τ (b(x j )) 2 .

2.3. WHITE-BOX ATTACKS ON DISCRETIZED INPUTS:

Discretizing the input makes it difficult to attack the model with standard white-box attack algorithms, such as FGSM (Goodfellow et al., 2014) and PGD (Madry et al., 2017) , since it is impossible to backpropagate through our discretization function to determine how to adversarially modify the model's input. In this section, we describe two novel iterative attacks which allow us to construct adversarial examples for networks trained on discretized inputs.

Constructing white-box attacks on discretized inputs serves two primary purposes. First, it allows us to more completely evaluate whether the model is robust to all adversarial attacks, as white-box attacks are typically more powerful than their black-box counterparts. Secondly, adversarial training is typically performed in a white-box fashion, and so in order to utilize and properly compare against the adversarial training techniques of Madry et al. (2017) , it is important to have strong white-box attacks.

For ease of presentation, we will describe the attacks assuming that f : R → R k discretizes inputs into thermometer encodings; in order to attack one-hot encodings, simply replace all instances of f therm with f onehot , τ with χ, and C with the identity function I. We represent the adversarial image after t steps of the attack as z t , where the value of the ith pixel is z t i . The first attack, Discrete Gradient Ascent (DGA), follows the direction of the gradient of the loss with respect to f (x), but is constrained at every step to be a discretized vector. If we have discretized the input image into k-dimensional vectors using the one-hot encoding, this corresponds to moving to a vertex of the simplex (∆ k )

n at every step. The second attack, Logit-Space Projected Gradient Ascent (LS-PGA), relaxes this assumption, allowing intermediate iterates to be in the interior of the simplex. The final adversarial image is obtained by projecting the final point back to the nearest vertex of the simplex.

Note that if the number of attack steps is 1, then the two attacks are equivalent; however, for larger numbers of attack steps, LS-PGA is a generalization of DGA.

2.3.1. DISCRETE GRADIENT ASCENT (DGA):

Following PGD (Madry et al., 2017) , we initialize DGA by placing each pixel into a random bucket that is within ε of the pixel's true value. At each step of the attack, we look at all buckets that are within ε of the true value, and select the bucket that is likely to do the most 'harm', as estimated by the gradient of setting that bucket's indicator variable to 1, with respect to the model's loss at the previous step.

z 0 i = f therm (x i + U (-ε, ε)) harm(z t i ) l = (z t i -τ (l)) • ∂L(z t ) ∂z t i if ∃(-ε ≤ η ≤ ε) s.t. b(x i + η) = l 0 otherwise. z t+1 i = τ arg max harm z t i

Because the outcome of this optimization procedure will vary depending on the initial random perturbation, we suggest strengthening the attack by re-running it several times and using the perturbation with the greatest loss. The pseudo-code for the DGA attack is given in Section B of the appendix.

2.3.2. LOGIT-SPACE PROJECTED GRADIENT ASCENT (LS-PGA):

To perform LS-PGA, we soften the discrete encodings into continuous relaxations, and then perform standard Projected Gradient Ascent (PGA) on these relaxed values. We represent the distribution over embeddings as a softmax over logits u, each corresponding to the unnormalized log-weight of a specific bucket's embedding. To improve the attack, we scale the logits with temperature T , allowing us to trade off between how closely our softmax approximates a true one-hot distribution as in the Gumbel-softmax trick (Jang et al., 2016; Maddison et al., 2016) , and how much gradient signal the logits receive. At each step of a multi-step attack, we anneal this value via exponential decay with rate δ.

z t i = C σ u t i T t z f inal i = τ arg max u f inal i T t = T t-1 • δ

We initialize each of the logits randomly with values sampled from a standard normal distribution. At each step, we ensure that the model does not assign any probability to buckets which are not within ε of the true value by fixing the logits to be -∞. The model's loss is a continuous function of the logits, so we can simply utilize attacks designed for continuous-valued inputs, in this case PGA with step-size ξ.

u 0 i = N (0; 1) if ∃(-ε ≤ η ≤ ε) s.t. b(x i + η) = l -∞ otherwise. u t+1 i l = (u t i ) l + ξ • sign ∂L(z t ) ∂u t i l if ∃(-ε ≤ η ≤ ε) s.t. b(x i + η) = l -∞ otherwise.

Because the outcome of this optimization procedure will vary depending on the initial perturbation, we suggest strengthening the attack by re-running it several times and using the perturbation with the greatest loss. The pseudo-code for the LS-PGA attack is given in Section B of the appendix.

3. EXPERIMENTS:

We compare models trained with input discretization to state-of-the-art adversarial defenses on a variety of datasets. We match the experimental setup of the prior literature as closely as possible.

Rows labeled with "Vanilla (Madry)" give the numbers reported in Madry et al. (2017) ; other rows contain results of our own experiments, with "Vanilla" containing a direct replication. For our MNIST experiments, we use a convolutional network; for CIFAR-10, CIFAR-100, and SVHN we use a Wide ResNet (Zagoruyko & Komodakis, 2016) . We use a network of depth 30 for the CIFAR-10 and CIFAR-100 datasets, while for SVHN we use a network of depth 15. The width factor of all the Wide ResNets is set to k = 4.1 Unless otherwise specified, all quantized and discretized models use 16 levels.

We found that in all cases, LS-PGA was strictly more powerful than DGA, so all attacks on discretized models use LS-PGA with ξ = 0.01, δ = 1.2, and 1 random restart. To be consistent with In Madry et al. (2017) (Goodfellow et al., 2014; Cisse et al., 2017) . To be consistent with Madry et al. (2017) , we list experiments on models trained only on adversarial inputs in the main paper; additional experiments on a mix of clean and adversarial inputs can be found in the appendix.

We also run experiments exploring the model's relationship with the number of distinct levels to which we quantize the input before discretizing it, and exploring various settings of hyperparameters for LS-PGA.

4. RESULTS:

Our adversarially-trained baseline models were able to approximately replicate the results of Madry et al. (2017) 

5. DISCUSSION:

In Goodfellow et al. (2014) , the seeming linearity of deep neural networks was shown by visualizing the networks in several different ways. To test our hypothesis that discretization breaks some of this linearity, we replicate these visualizations and contrast them to visualizations of discretized models. See Appendix G for an illustration of these properties.

For non-discretized, clean trained models, test-set examples always yield a linear boundary between correct and incorrect classification; in contrast, non-adversarially-trained models have a more interesting parabolic shape (see Figure 9 ). When discretizing the input, we introduce

C w • C h • C o • c • (k -1

) extra parameters, where c is the number of channels in the image, k is the number of levels of discretization, and C w , C h , C o are the width, height, and output channels of the first convolutional layer. Discretizing using 16 levels introduced 0.03% extra parameters for MNIST, 0.08% for CIFAR-10 and CIFAR-100, and 2.3% for SVHN. This increase is negligible, so it is likely that the robustness comes from the input discretization, and is not merely a byproduct of having a slightly higher-capacity model.

6. CONCLUSION:

Our findings convincingly demonstrate that the use of thermometer encodings, in combination with adversarial training, can reduce the vulnerability of neural network models to adversarial attacks. Our analysis reveals that the resulting networks are significantly less linear with respect to their inputs, supporting the hypothesis of Goodfellow et al. (2014) 

A HYPERPARAMETERS:

In this section, we describe the hyperparameters used in our experiments. For CIFAR-10 and CIFAR-100 we follow the standard data augmenting scheme as in (Lin et al., 2013; He et al., 2016; Huang et al., 2016; Zagoruyko & Komodakis, 2016) : each training image is zero-padded with 4 pixels on each side and randomly cropped to a new 32 × 32 image. The resulting image is randomly flipped with probability 0.5, it's brightness is adjusted with a delta chosen uniformly at random in the interval [-63, 63 ) and it's contrast is adjusted using a random contrast factor in the interval [0.2, 1.8]. For MNIST we use the Adam optimizer with a fixed learning rate of 1e-4 as in Madry et al. (2017) . For CIFAR-10 and CIFAR-100 we use the Momentum optimizer with momentum 0.9, 2 weight decay of λ = 0.0005 and an initial learning rate of 0.1 which is annealed by a factor of 0.2 after epochs 60, 120 and 160 respectively as in Zagoruyko & Komodakis (2016) . For SVHN we use the same optimizer with initial learning rate of 1e-2 which is annealed by a factor of 0.1 after epochs 80 and 120 respectively. We also use a dropout of 0.3 for CIFAR-10, CIFAR-100 and SVHN.

B PSEUDO-CODE:

The DGA attack is described in Algorithm 2 and the LS-PGA attack is described in Algorithm 3. Both these algorithms make use of a getM ask() sub-routine which is described in Algorithm 1.

Input: Image x, parameter ε Output: ε-discretized mask around x mask ← (0) n×k low ← max{0, x -ε} high ← min{1, x + ε} for α ← 0 to 1 by 1 k do mask ← mask + f onehot (α * low + (1 -α) * high) end return mask

Algorithm 1: Sub-routine for getting an ε-discretized mask of an image.

C ADDITIONAL EXPERIMENTS ON MNIST:

In this section we list the additional experiments we performed using discretized models on MNIST.

The main hyperparameters of Algorithm 3 are the step size ξ used to perform the projected gradient ascent, and the annealing rate of δ. We found that the choice of these hyperparameters was not critical to the robustness of the model. In particular, we performed experiments with ξ = 1.0 and ξ = 0.001, and both achieved similar accuracies as in Table 2 and Table 3 . Additionally, we found that without annealing, i.e., δ = 1.0, the performance was only slightly worse than with δ = 1.2.

We also experimented with discretizing by using percentile information per color channel instead of using uniformly distributed buckets. This did not result in any significant changes in robustness or accuracy for the MNIST dataset.

Input: Image x, label y, discretization function f , loss L(θ θ θ, f (x), y), l attack steps, parameter ε Output:

Adversarial input to the network z η ← U (-ε, ε) z 0 ← f (x + η) mask ← getM ask(x, ε) for t ← 1 to l do / * Loop invariant: z t i is discretized for every pixel i * / grad ← ∇ z t-1 L(θ θ θ, z t-1 , y) if f is one-hot then harm l ← (z t-1 -χ(l)) grad else harm l ← (z t-1 -τ (l)) grad end harm ← harm * mask -(1 -mask) * ∞ z t ← f (arg max(harm)) end return z ← z l

Algorithm 2: Discrete Gradient Ascent (DGA)

Input: Image x, label y, discretization function f , loss L(θ θ θ, f (x), y), l attack steps, parameters ε, δ Output: Adversarial input to the network z mask ← getM ask(x, ε) 4 and 5 ). original image) was designed to ensure that any adversarially-perturbed image is still recognizable as its original image by a human. However, this artificial constraint excludes many other potential attacks that also result in human-recognizable images. State-of-the art defenses in the standard setting can still be easily defeated by non-standard attacks; for an example of this, see appendix A of ICLR submission "Adversarial Spheres". A "larger epsilon" attack is just one special case of a "non-standard" attack. If we permit non-standard attacks, a fair comparison would show that all current approaches are easily breakable. There is nothing special about the "larger epsilon" attack that makes a vulnerability to this non-standard attack in particular more problematic than vulnerabilities to other non-standard attacks.

u 0 ← N (0 k ; 1 k ) n * mask -(1 -mask) * ∞ T ← 1 if f is one-hot then F ← I else F ← C end z 0 ← F σ u 0 T for t ← 1 to l do grad ← ∇ u t-1 L(θ θ θ, z t-1 , y) u t ← u t-1 + ξ • sign (grad) z t ← F σ u t T T ← T • δ end return z ← τ (arg max(u l i )) Algorithm 3: Logit-Space Projected Gradient Ascent (LS-PGA) Model ξ δ White-box

Model:

In Figures 9, 10 , 11, 12 , 13 and 14 we plot several examples of church-window plots for MNIST (Goodfellow et al., 2014) . Each plot is crafted by taking several test-set images, calculating the vector corresponding to an adversarial attack on each image, and then choosing an additional random orthogonal direction. In each plot, the clean image is at the center and corresponds to the color white, the x-axis corresponds to the magnitude of a perturbation in the adversarial direction, and the y-axis corresponds to the magnitude of a perturbation in the orthogonal direction. Note that we use the same random seed to generate the test set examples and the adversarial directions across different church-window plots. 

:

A full list of hyperparameters can be found in the appendix. Source code is available at http://anonymized

Figures and Tables:

Madry et al. (2017), we describe attacks in terms of the maximum ∞ -norm of the attack, ε. All MNIST experiments used ε = 0.3 and 40 steps for iterative attacks; experiments on CIFAR used ε = 0.031 and 7 steps for iterative attacks; experiments on SVHN used ε = 0.047 and 10 steps for iterative attacks. These settings were used for adversarial training, white-box attacks, and blackbox attacks. Figure3plots the effectiveness of the iterated PGD/LS-PGA attacks on vanilla and discretized models for MNIST and shows that increasing the number of iterations beyond 40 would have no effect on the performance of the model on ∞ -bounded adversarial examples for MNIST.

Figure 2: Comparison of the convergence rate of various adversarially trained models on the SVHN dataset.

Figure 5: Comparison of the convergence rate of various adversarially trained models on the CIFAR-10 dataset. The discretized models use 16 levels per color channel. (5a) shows the accuracy on clean examples, while (5b) shows the accuracy on white-box PGD/LS-PGA examples, in wall-clock time.

Figure 7: Linear extrapolation plot as in Goodfellow et al. (2014) for MNIST. (7a) shows the behavior of a vanilla model while (7b) for a discretized model using 16 levels and thermometer encoding. The ε-bound is [-1.5, 1.5].

Figure 8: Plot showing the accuracy of various adversarially trained models on MNIST with ε = 0.3 (8a) and on CIFAR-10 with ε = 0.031 (8b), when attacked with increasing values of ε using PGD/LS-PGA.

Figure 9: Church-window plots of clean-trained models on MNIST. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0]. Notice the almost-linear decision boundaries on nondiscretized models.

Figure 10: Church-window plots of adversarially-trained models on MNIST, trained on only adversarial examples. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0].

Figure 11: Church-window plots of adversarially-trained models on MNIST, trained using a mix of clean and adversarial examples. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0]. Notice the almost-linear decision boundaries on non-discretized models.

Figure 12: Church-window plots of clean-trained models on CIFAR-10. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0]. Notice the almost-linear decision boundaries on nondiscretized models.

Figure 13: Church-window plots of adversarially-trained models on CIFAR-10, trained on only adversarial examples. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0].

Figure 14: Church-window plots of adversarially-trained models on CIFAR-10, trained using a mix of clean and adversarial examples. The x-axis of each sub-plot represents the adversarial direction, while the y-axis represents a random orthogonal direction. The correct class is represented by white. Every row in the plot contains a training data point chosen uniformly at random, while each column uses a different random orthogonal vector for the y-axis. The ε bound for both axes is [-1.0, 1.0]. Notice the almost-linear decision boundaries on non-discretized models.

Discretization defends against adversarial examples by changing which parameters of the model are used, and may also discard information if the number of discretization levels is low; quantization can only defend the model by discarding information.

Table content: <table><tr><td colspan="4">Real-valued Quantized Discretized (one-hot) Discretized (thermometer)</td></tr><tr><td>0.13</td><td>0.15</td><td>[0100000000]</td><td>[0111111111]</td></tr><tr><td>0.66</td><td>0.65</td><td>[0000001000]</td><td>[0000001111]</td></tr><tr><td>0.92</td><td>0.95</td><td>[0000000001]</td><td>[0000000001]</td></tr></table>

In Figures2 and 5(located in appendix), we plot the test-set accuracy across training timesteps for various adversarially trained models on the SVHN and CIFAR-10 datasets, and observe that the discretized models become robust against adversarial examples more quickly. Comparison of adversarial robustness to white-box attacks on MNIST .

Table content: <table><tr><td/><td/><td/><td>Model</td><td/><td colspan="4">Clean FGSM PGD/LS-PGA</td></tr><tr><td/><td/><td/><td colspan="3">Vanilla (Madry) 99.20</td><td>6.40</td><td>-</td></tr><tr><td/><td/><td>Clean</td><td colspan="2">Vanilla Quantized One-hot</td><td>99.30 99.19 99.13</td><td>0.19 1.10 0</td><td>0 0 0</td></tr><tr><td/><td/><td/><td colspan="2">Thermometer</td><td>99.20</td><td>0</td><td>0</td></tr><tr><td/><td/><td>Adv. train</td><td colspan="3">Vanilla (Madry) 98.80 Vanilla 98.67 Quantized 98.75 One-hot 98.61 Thermometer 99.03</td><td>95.60 96.17 96.29 96.22 95.84</td><td>93.20 93.30 94.23 94.30 94.02</td></tr><tr><td colspan="2">X X Target X X X</td><td>Source X X X X</td><td colspan="6">Clean Vanilla One-hot Thermometer Vanilla One-hot Thermometer Adv. train</td></tr><tr><td/><td colspan="2">Vanilla</td><td>2.04</td><td>36.02</td><td/><td>24.58</td><td>3.48</td><td>80.44</td><td>57.69</td></tr><tr><td>Clean</td><td colspan="2">Quantized One-hot</td><td>39.22 14.57</td><td>32.39 6.91</td><td/><td>25.63 8.11</td><td>75.02 39.02</td><td>75.92 39.60</td><td>52.32 18.02</td></tr><tr><td/><td colspan="2">Thermometer</td><td>41.12</td><td>14.30</td><td/><td>10.98</td><td>61.84</td><td>59.16</td><td>32.93</td></tr><tr><td>Adv. train</td><td colspan="2">Vanilla (Madry) Quantized Vanilla One-hot Thermometer</td><td>-97.65 97.62 97.78 98.07</td><td>-98.16 98.05 98.48 98.75</td><td/><td>-97.14 97.06 97.87 98.02</td><td>96.0 95.27 95.43 96.87 97.05</td><td>-95.31 95.38 96.60 96.88</td><td>-96.53 96.23 96.87 97.13</td></tr></table>

Comparison of adversarial robustness to black-box attacks on MNIST .

Table content: <table><tr><td/><td>Model</td><td colspan="3">Clean FGSM PGD/LS-PGA</td></tr><tr><td/><td colspan="2">Vanilla (Madry) 95.20</td><td>25.10</td><td>4.10</td></tr><tr><td>Clean</td><td>Vanilla Quantized One-hot</td><td>94.29 93.49 93.26</td><td>46.15 43.89 52.07</td><td>1.66 3.57 53.11</td></tr><tr><td/><td>Thermometer</td><td>94.22</td><td>48.50</td><td>50.50</td></tr><tr><td>Adv. train</td><td>Vanilla (Madry) Vanilla Quantized One-hot Thermometer</td><td>87.3 87.67 85.75 88.67 89.88</td><td>60.3 59.7 53.53 68.76 80.96</td><td>50.0 41.78 42.09 67.83 79.16</td></tr></table>

Comparison of adversarial robustness to white-box attacks on CIFAR-10 .

Table content: <table/>

Comparison of adversarial robustness to black-box attacks on CIFAR-10 .

Table content: <table><tr><td colspan="2">X X Target X X X</td><td>Source X X X X</td><td colspan="6">Clean Vanilla One-hot Thermometer Vanilla One-hot Thermometer Adv. train</td></tr><tr><td/><td colspan="2">Vanilla (Madry)</td><td>0.0</td><td>-</td><td>-</td><td>79.7</td><td>-</td><td>-</td></tr><tr><td>Clean</td><td colspan="2">Vanilla Quantized One-hot</td><td>3.38 70.54 83.00</td><td>60.10 62.46 56.25</td><td>52.60 55.38 63.94</td><td>45.48 51.74 54.59</td><td>37.21 45.37 49.21</td><td>49.91 55.64 57.28</td></tr><tr><td/><td colspan="2">Thermometer</td><td>80.33</td><td>66.22</td><td>53.45</td><td>57.04</td><td>51.03</td><td>60.90</td></tr><tr><td>Adv. train</td><td colspan="2">Vanilla (Madry) Vanilla Quantized One-hot Thermometer</td><td>85.60 85.60 84.56 86.01 88.25</td><td>-74.99 82.43 77.19 81.59</td><td>-73.78 82.22 77.70 80.80</td><td>67.0 67.0 72.52 61.92 67.96</td><td>-50.09 72.29 60.02 67.43</td><td>-71.03 79.43 72.89 77.68</td></tr></table>

that many adversarial examples are caused by over-generalization in networks that are too linear. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. ICLR, abs/1312.6199, 2014. URL http://arxiv.org/abs/1312.6199. Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. arXiv preprint arXiv:1704.01155, 2017.

Table content: <table><tr><td>Sergey Zagoruyko and Nikos Komodakis.</td><td>Wide residual networks.</td><td>arXiv preprint</td></tr><tr><td>arXiv:1605.07146, 2016.</td><td/><td/></tr></table>

Comparison of adversarial robustness to white-box attacks on MNIST using 16 levels and with various choices of the hyperparameters ξ and δ for Algorithm 3. The models are evaluated on white-box attacks and on black-box attacks using a vanilla, clean trained model; both use LS-PGA.Finally, we also trained on a mix of clean and adversarial examples: this resulted in significantly higher accuracy on clean examples, but decreased accuracy on white-box and black-box attacks compared to Tables2 and 3.

Table content: <table><tr><td>Black-box</td></tr></table>

Comparison of adversarial robustness to white-box attacks on MNIST using a mix of clean and adversarial examples.

Table content: <table><tr><td>X Target X X X</td><td>X</td><td>Source X X X X</td><td colspan="5">Clean Vanilla One-hot Thermometer Vanilla One-hot Thermometer Adv. train</td></tr><tr><td colspan="3">Vanilla</td><td>97.88</td><td>97.87</td><td>96.99</td><td>93.07</td><td>90.97</td><td>96.46</td></tr><tr><td colspan="3">One-hot</td><td>98.28</td><td>98.83</td><td>98.08</td><td>95.73</td><td>95.96</td><td>97.25</td></tr><tr><td colspan="3">Thermometer</td><td>98.45</td><td>98.70</td><td>98.08</td><td>96.35</td><td>95.72</td><td>96.97</td></tr></table>

Comparison of adversarial robustness to black-box attacks on MNIST of various models using a mix of clean and adversarial examples.

Table content: <table><tr><td/><td/><td>White-box</td><td>Black-box</td><td/></tr><tr><td>Levels</td><td colspan="4">Clean PGD/LS-PGA Vanilla, Clean Vanilla, PGD</td></tr><tr><td>One-hot (4)</td><td>99.09</td><td>92.95</td><td>98.23</td><td>95.64</td></tr><tr><td>One-hot (8)</td><td>99.10</td><td>92.65</td><td>98.49</td><td>96.37</td></tr><tr><td>One-hot (16)</td><td>99.14</td><td>93.08</td><td>98.39</td><td>96.26</td></tr><tr><td>One-hot (32)</td><td>99.06</td><td>93.51</td><td>98.38</td><td>95.78</td></tr><tr><td>One-hot (64)</td><td>98.89</td><td>93.63</td><td>98.35</td><td>95.74</td></tr><tr><td>Thermometer (4)</td><td>99.11</td><td>92.62</td><td>98.23</td><td>95.67</td></tr><tr><td>Thermometer (8)</td><td>99.08</td><td>93.45</td><td>98.44</td><td>95.93</td></tr><tr><td colspan="2">Thermometer (16) 99.07</td><td>93.88</td><td>98.24</td><td>95.28</td></tr><tr><td colspan="2">Thermometer (32) 99.02</td><td>94.14</td><td>98.24</td><td>95.49</td></tr><tr><td colspan="2">Thermometer (64) 99.00</td><td>94.62</td><td>98.33</td><td>95.71</td></tr></table>

Comparison of adversarial robustness on MNIST as the number of levels of discretization is varied. All models are trained mix of adversarial examples and clean examples.D ADDITIONAL EXPERIMENTS ON CIFAR-10In this section we list the additional experiments we performed on CIFAR-10. Firstly, we trained models on a mix of both clean and adversarial examples. The results for mixed training are listed in Tables10 and 11; as expected it has lower accuracy on adversarial examples, but higher accuracy on clean examples, compared to training on only adversarial examples (Tables

Table content: <table/>

Comparison of adversarial robustness to white-box attacks on CIFAR-10 of various models using a mix of regular and adversarial training.

Table content: <table><tr><td/><td colspan="3">Clean FGSM PGD/LS-PGA</td></tr><tr><td>Vanilla</td><td>87.16</td><td>54.50</td><td>34.71</td></tr><tr><td>One-hot</td><td>92.19</td><td>58.87</td><td>58.96</td></tr><tr><td colspan="2">Thermometer 92.32</td><td>66.60</td><td>65.67</td></tr></table>

Peer review (iclr2018):
 This is a beautiful work that introduces both (1) a novel way of defending against adversarial examples generated in a black-box or white-box setting, and (2) a principled attack to test the robustness of defenses based on discretized input domains. Using a binary encoding of the input to reduce the attack surface is a brilliant idea. Even though the dimensionality of the input space is increased, the intrinsic dimensionality of the data is drastically reduced. The direct relationship between robustness to adversarial examples and intrinsic dimensionality is well known (paper by Fawzi.). This article exploits this property nicely by designing an encoding that preserves pairwise distances by construction. It is well written overall, and the experiments support the claims of the authors. This work has a crucial limitation: scalability. The proposed method scales the input space dimension linearly with the number of discretization steps. Consequently, it has a significant impact on the number of parameters of the model when the dimensionality of the inputs is large. All the experiments in the paper report use relatively small dimensional datasets. For larger input spaces such as Imagenet, the picture could be entirely different: - How would thermometer encoding impact the performance on clean examples for larger dimensionality data (e.g., Imagenet)? - Would the proposed method be significantly different from bit depth reduction in such setting? - What would be the impact of the hyper-parameter k in such configuration? - Would the proposed method still be robust to white box attack? - The DGA and LS-PGA attacks look at all buckets that are within ε of the actual value, at every step. Would this be feasible in a large dimensional setting? More generally, would the resulting adversarial training technique be practically possible? While positive results on Imagenet would make this work a home run, negative results would not affect the beauty of the proposal and would shed critical light on the settings in which thermometer encoding is applicable. I lean on the accept side, and I am willing to increase the score greatly if the above questions are answered.