Universal Learning Approach for Adversarial Defense
Authors: Uriya Pesso
Koby Bibas
Meir Feder
Venue:  (2020)
Keywords: Adversarial examples
Adversarial training
Universal learning
pNML for DNN

Abstract: Adversarial attacks were shown to be very effective in degrading the performance of neural networks. By slightly modifying the input, an almost identical input is misclassified by the network. To address this problem, we adopt the universal learning framework. In particular, we follow the recently suggested Predictive Normalized Maximum Likelihood (pNML) scheme for universal learning, whose goal is to optimally compete with a reference learner that knows the true label of the test sample but is restricted to use a learner from a given hypothesis class. In our case, the reference learner is using his knowledge on the true test label to perform minor refinements to the adversarial input. This reference learner achieves perfect results on any adversarial input. The proposed strategy is designed to be as close as possible to the reference learner in the worst-case scenario.  Specifically, the defense essentially refines the test data according to the different hypotheses, where each hypothesis assumes a different label for the sample. Then by comparing the resulting hypotheses probabilities, we predict the label and detect whether the sample is adversarial or natural. Combining our method with adversarial training we create a robust scheme which can handle adversarial input along with detection of the attack. The resulting scheme is demonstrated empirically.

Under review as a conference paper at ICLR 2020
UNIVERSAL LEARNING APPROACH FOR ADVERSAR- IAL DEFENSE
Anonymous authors Paper under double-blind review
ABSTRACT
Adversarial attacks were shown to be very effective in degrading the performance of neural networks. By slightly modifying the input, an almost identical input is misclassified by the network. To address this problem, we adopt the universal learning framework. In particular, we follow the recently suggested Predictive Normalized Maximum Likelihood (pNML) scheme for universal learning, whose goal is to optimally compete with a reference learner that knows the true label of the test sample but is restricted to use learner from given hypothesis class. our case, the reference learner is using his knowledge on the true test label to per- form minor refinements to the adversarial input. This reference learner achieves perfect results any adversarial input. The proposed strategy is designed to be close as possible to the reference learner in the worst-case scenario. Specifically, the defense essentially refines the test data according the different hypothe- ses, where each hypothesis assumes a different label for the sample. Then by comparing the resulting hypotheses probabilities, we predict the label and detect whether the sample is adversarial or natural. Combining our method with adver- sarial training we create a robust scheme which can handle adversarial input along with detection of the attack. The resulting scheme is demonstrated empirically.
INTRODUCTION
Deep neural networks (DNNs) have shown to have state-of-the-art performance in many machine learning tasks (Goodfellow et al. 2016a). Despite the impressive performance, has been found that DNNs are susceptible to adversarial attacks (Szegedy et al.) [0133: Biggio et al. 2013). These attacks cause the network to under-pertee mnder-perform by adding specially crafted noise to the input, such that the original and modified inputs almost indistinguishabl..
In the case of an image classification task, adversarial attacks can be characterized according to three properties (Carlini et al., 2019): the adversary goal, the adversary capabilities, and the adversary knowledge. (i) The adversary goal may be to simply cause misclassification. This is referred to as an untargeted attack. Alternatively, the goal can be have the model misclassify certain examples from source class into a target class of its choice. This is referred to as targeted attack. (ii) The adversary capabilities relate to the strength of the perturbation the adversary is allowed cause the data, i.c., the distance between the original sample and the adversarial sample under a certain distance metric must be smaller than c. (iii) The adversary knowledge represents what knowledge the adversary has about the model. This can range from fully white-box scenario, where the adversary has complete knowledge about the model and its parameters, to black-box scenario where the adversary does not know the model and only has limited number of queries on it.
Out of the many different adversarial attack algorithms, gradient optimization- based attacks are known to be the most powerful kind of attacks (Carlini al.] (201)). In these attacks, the adversary uses the gradients of a loss function with respect to the input in order to find the perturbation that minimizes the performance of the network. Such attacks include, among others, Fast Gradient Sign Method (FGSM) (Goodfellow et al. (0044). Projected Gradient Descent (PGD) (Madry et al.) 2017) and CW-attack (Carlini& & Wagner 2017).
Adversarial defenses can be separated into two categories according to their goal: to increase robust- ness against adversarial examples (Madry al. 2017) and to detect whether example is natural

---

Under review as a conference paper at ICLR 2020
or adversarial (Li Li 2017 Feinman et al. 2017). While recent years have shown growing inter- est in understanding and defending against adversarial examples, no solution has been found yet to white-box settings, where the adversary has full access to the network.
The universal prediction framework considers both the stochastic setting in which the true relation between the features and labels is given by stochastic function, and the individual setting in which no probabilistic connection between the data and the labels is assumed. The Predictive Normalize Maximum Likelihood (pNML) as the universal learning solution is proposed for the batch learning in the individual settings (Fogel & Feder 2018a Sa Bibas et al. 2019a b). The pNML scheme gives the optimal solution for min-max game where the goal learner, who knows the true label but is restricted to use a learner from given hypothesis class.
as close as possible to a reference
Further elaboration on the PNML. learner is presented in section 3
We propose the Adversarial PNML scheme as a new adversarial defense and detector. Adversarial p.NML improves DNNs robustmess against adversarial attacks and allows the detection of adversarial samples. Based on the pNML, we restrict the genie learner, learner who knows the true test label, to perform minor refinements to the adversarial examples. Our method essentially uses an existing model trained with adversarial training. Then, we compose a hypothesis class. Each member in the class assumes a different label for the input. The member refines the input sample based on the assumed label. Finally, by comparing the resulting hypotheses probabilities, we can predict the true label along with detection that the sample is adversarial.
In this paper, we propose novel scheme that uses simple targeted adversarial attack method of refinement. By performing that simple refinement we create a hypothesis class and use it to pre- dict the test label. We demonstrate our method robustness state-of-the-art adversarial attacks as PGD (Madry et al. (017) for MNIST and CIFAR 10 datasets. We show that by combining adver- sarial sample detection, which an inherent property of our method, we manage to overcome defense-aware adaptive attack. Unlike existing methods that attempt to remove the adversary per- turbation (Samangouei et al., 20188: Song et al. [017)] Guo al., (017). our method is unique the sense it does not remove the perturbation but rather exploits the adversarial subspace properties.
2 RELATED WORK
In this section, we mention related works on common attack methods, various type of defence and the properties of the adversarial subspace.
Attack Methods One of the simplest attacks is FGSM that was introduced by Goodfellow et al. 2014). Let wo be the parameters of the trained model, x be the test data and y its corresponding label, L the loss function of the model, Tody the adversary input and specifies the maximum l distortion || - Tadvlloo S e. First, the sign of the loss function gradients are computed with respect to the image pixels, then after multiplying the signs by c, it is added to the original image create an adversary untargeted attack:
Tede z e sign V LWw,, x. Virue)
It is also possible to improve classification chance for a certain label Yiarger (targeted attack):
Fado = x - e sign V L (Wo, x, Morget))
A multi-step variant of FGSM that was used by Madry et al. (2017) is called Projected Gradient Descent (PGD). Denote a as the size of the update, for each iteration, an FGSM step is executed
x .+
a'a + a sign(V VV x, Yome), 0 5
The number of iterations is predetermined. For each sample, PGD attack creates multiple different adversarial samples by randomly choosing multiple different starting points s0 = x + where U[-c,e] c). Then the sample with the highest loss is chosen. The strength of the attack is defined by the allowed distance between final adversarial input and the original image ||x - x S e.
Defence Methods Several defense methods have been suggested to increase DNNs robustness.
The most prominent defense is adversarial training, which augments the trainset to include adver- sarial examples (Goodfellow et al. 2014). Other methods include transformation of the input as
2

---

Under review as a conference paper at ICLR 2020
suggested in Guo et al. (0177) where the imput is reconstructed from its lower resolution version. A different approach aims only to detect adversarial examples. Such a method is suggested by /Fein man et al. 2017), where detection is performed by measuring the confidence for given input using bayesian uncertainty estimation, available in dropout neural networks
Adversarial Subspace (Szegedy et al. (2013) states that adversarial examples represent low- probability pockets in the manifold which are hard to find by randomly sampling around the given sample. Later works show that while adversarial subspace low-probability they inhabit large and contiguous regions (Goodfellow et al.) 2014 Madry et al. 2017). Tabacof & Valle (2016) showed that the adversarial subspaces are less stable compared the true data subspaces. Our scheme takes advantage of these characteristics in order to improve the robustness of the model.
3 PRELIMINARIES: THE PREDICTIVE NORMALIZE MAXIMUM LIKELIHOOD
We now describe the universal learning problem and its associated solution: the predictive Normal- ized Maximum Likelihood (pNML). In the case of supervised machine learning, a trainset consisting of N pairs of examples, NN { yt } } is given to learner, where X' is the data and y E y is the label. The goal of the learner is to predict the label given a new test data z. More formally, the learner attempts to minimize a loss function that measures the accuracy of the prediction. In the information-theoree framework, prediction is done by assigning probability distribution q(-|xr) to the unknown label. The performance of the predictor is evaluated using the log-loss function
(4) For the problem to be well-posed we must make further assumptions on the class of possible models or "hypothesis" that is used in order find the relation between and y. Denote O as general index set, this class is a set of conditional probability distributions Pe = {po (yl x.), E θ}.
L(g; x, y)
log ((ylz))
Another assumption, required solve the problem, is on how the data and the labels gen- erated. The most common setting in learning theory is Probably Approximately Correct (PAC), established in [Valiant 1984) where x and y are assumed be generated by some source P u) P())z) P(ylæ) is not necessarily a member of Pe. Another possible setting for the learning problem, recently suggested by (Fogel& & Feder (2018a) and and[Merhas & Feder (1998), is the individual setting, where the data and labels of both the training and test are specific and individual values, In this setting, the goal is to compete with a "genie" or reference learner that knows the desired label value but is restricted to use a model from the given hypotheses class Pe, and that does not know which the samples the test. This learner then chooses:
A(zN, x. 8) = arg max [po II
(5)
The log-loss difference between a universal learner q and the reference the regret:
R(q; x .x = log Po (a, (ular)
(6)
9/0l q,01=N 2N ,x)
As advocated in Fogel & Feder 2018b), the chosen universal learner solves: (a x) = min max R(g: =N y).
(7)
This min-max optimal solution, termed Predictive Normalized Maximum Likelihood (pNML), is obtained using "equalizer" reasoning, following (Shtar' (Shtar'kov) 1987):
max Po (ul 2 .r,y) v)
genan Nowwn(y/a;; [ x
8E0
(8)
Evey
max po (y) NN xx, BEB
Its corresponding regret, independent of the true y, is:
r log Σ max
po(y xr y) 6EE
DEY
The pNML has been derived for several model classes in related works such as 1-D perceptron in Fogel & Feder (018b) and linear regression in Bibas et al. (2019a). They show that the regret can serve as a pointwise measure of learnability for the specific training and data sample. Bibas et al. 2019h) also reported that the PNML. regret can be used to detect out of distribution test samples.
3

---

Under review as a conference paper at ICLR 2020
4 ADVERSARIAL PNML
Intuitively, for each of the possible labels E Y the pNML chooses learner from hypothesis class that maximizes the probability of that label. The probability of the respective label is then normalized by the normalization factor, which is the same for all labels. Denote the unnormalized probability of the true label as the "genie"
Recall the pNML solution in equation g
max (o L. Virue). #ge
(10)
The "genie" is the best learner one can attain when knowing the test label. The main issue is how to select the hypothesis class. "good" hypothesis class should be large enough so that the genie could achieve good performance while avoiding overfitting of the other class members such that the overall regret (the distance from the genie learner as mentioned in equation 9 stays small.
We propose novel hypothesis class by adding refinement stage before pretrained DNN model utp The refinement stage changes the test sample with respect to the model wo and certain label
Drefine (x. y) = x À sign( V (wo: 2. v))
(11)
where À is the refinement strength. The hypothesis class that we consider is therefore
θ (Pwo (ineingeee (v. U.)), Vya E Y).
(12)
Each member in the hypothesis class produces probability assignment by adding perturbation that strengthen one of the possible value of the test label.
Our Adversarial PNML scheme consists the following steps: At first, train a DNN model wp- The training could be udversarial-training (Goodfellow et al.] 2014) or normal training. Then, we produce the hypothesis class: Given test data æ, we refine using the trained DNN wo and one of the possible value of the test label y; E Y as in equation [ We produce a probability assignment for the selected label y; by feeding the refined test data Zrefines the trained DNN wp
Ps Pioo Trefine (I, 3/)).
(13)
The process is repeated for every possible value of the test label. In the end of the process we get a set of predictions (probability distributions), we normalize them and return the Adversarial pNML. probability assignment
I Σ
(3:) Pi:
C=
Pi-
(14)
x=
The corresponding regret is the log normalization factor:
|Y
r T= log C' log
P
(15)
Previous works demonstrated that it is possible detect adversarial examples by measuring the confidence of the predictions (Feinman et al. 2017). As mentioned in section 3, the regret is the distance from the genie learner, who knows the true label of the test sample. Having high regret means being far from the genie and therefore having low confidence in the prediction. Such a situation used to detect adversarial examples and is demonstrated in section 6
We now discuss the reasons behind the choice of the suggested hypothesis class. Let z be strong adversarial example with respect to the label Yhargets i.e., the DNN model has high probability mistakenly classifying x as Muaggtt We now differentiate between three kind of members in our suggested hypothesis class: refinement towards the true label yours refinement towards the adversary target Ytorger and refinement towards other label y {yrre ).arger).
Refinement towards the true label When perturbing towards the true label Youes the refinement moves the adversarial example outside the loss function's local maximum, thus increasing the prob- ability of predicting the true label.
4

---

Under review as a conference paper at ICLR 2020
Table Accuracy rate for various adversary attacks and adversary trained models.
MNIST
CIFARIO
Model AAttack
None
FGSM
PGD
None
FGSM
PGD
Natural
99.3%
0.6%
0.0%
93.6%
6.1%
0%
Natural + Ours
99.2%
1.6%
0.0%
87.2%
5.9%
1.0%
FGSM 97.1%
97.7%
0.3%
83.5%
49,4%
36.2%
FGSM Ours
97.2%
91.4%
23.3%
79.4%
50.1%
48.8%
PGD 98.3%
95.8%
90.5%
84.3%
59.5%
37.4%
PGD + Ours
98.3%
93.9%
95.2%
79.1%
60.8%
60.1%
Refinement towards the adversary target Refinement towards Yurger is essentially taking another step towards the local maximum of the loss. In the case of a strong adversary. the loss was already converged to the local maximum and another step towards Mauger would not dramatically increase the probability of Youger- The loss remains roughly the same and so the probability assignment of the hypothesis. Since the adversarial subspace is relatively small and unstable (see section 2), a weak refinement towards Yurget could even move the ad adversarial example outside the local maximum of loss function which effectively reduces the strength of the attack.
Refinement towards other label This refinement effectively applies a weak targeted attack to- wards a specific label. Therefore, increases the probability to misclassify the test data as y. As mentioned in Section 2 the adversarial examples represent low-probability subspaces in the man- ifold which are hard to find. weak step towards is unlikely to find strong adversarial points.
Since the low-probability and instability of the adversarial subspace crucial for Adversarial PNML,, we combine our method with adversarial training, which is known to decrease the size of the adversarial subspace (Madry al. 2017).
EXPERIMENTS
In this section we present experiments that test our proposed Adversarial pNML scheme de- fence for adversarial attack. We test the performance on MNIST (LeCun et al. 2010 and CIFARIO (Krizhevsky et al., 2014) datasets for PGD attack which represents attack that that efficiently maxi- mizes the loss of an example (Madry et al. 2017). and FGSM attack that illustrates weaker attack.
5.1 MNIST DATASET
We follow the model architecture as described in Madry et al. (017)). We use a network that consists of two convolutional layers with 32 and 64 filters respectively, each followed by x max-pooling, and fully connected layer of size 1024. We use three different trainsets that are used in training the baseline approaches: (i) Natural trainset; (ii) Adversarial trainset that was generated by executing FGSM attack on the natural trainset with value of 0.3; (iii) Adversarial trainset that was produced by PGD based attack on the natural trainset with 40 steps of size 0.01 with a maximal value of 0.3. We train three different models using these three trainsets. The training consists of 100 epochs using stochastic gradient descent (SGD) with a learning rate of 0.01, momentum value 0.9 and weight decay of 0.0001.
We consider the following threat: white-box lac that unaware of the defense, with attack strength that equals to 0.3. We test the accuracy in the following test cases: natural samples, FGSM attack, and PGD attack. For the PGD attack, the attacker performs 40 steps of size 0.01 with 20 restarts.
Table I shows the efficiency of our scheme for the various attacks. First, in case of natural images, when no perturbation added to the original MNIST dataset, our method obtains approximately the same accuracy the basclines. In the face of PGD attack, which is the strongest adversary, model that was trained using FGSM perturbation obtains a significant improvement of 20% in the accuracy rate when using our method, and improvement of 5% model that trained with PGD
5

---

Under review as a conference paper at ICLR 2020
1.0
0.7
0.8
0.6
Accuracy
Accuracy
0.5
0.6
04
0.2
Without
Without With
6.0
0.2
0.02
0.04
0.06
Attack strength
Attack strength
(a) MNIST dataset.
(b) CIFARIO dataset.
Figure/ 1: Comparison between a model without scheme (Without) to the same model with Adversarial pNML scheme (With) for various attack strength e. For each dataset the corresponding model was trained with PGD trainset. (a) MNIST dataset. (b) CIFARIO dataset.
mixed results. When using the FGSM train method we see reduction of 6% in the accuracy rate and a decrease of 1% in the case of PGD based training. As explained in section 4 our scheme works best when the adversary finds a local maximum of the error function. this scenario is not necessarily the case when using FGSM weak attack. Nevertheless, increasing model robustness is first and foremost increasing the robustness in the face of the worst-case attack (Madry et al. 2017). We point out to the fact that for a model trained with PGD trainset, the worst-case attack for our scheme is created by FGSM attack and not PGD attack. This indicates that PGD can fail to find the optimal attack for our scheme. In section 6 we present an adaptive adversary that is explicitly designed to attack our proposed method. We show that we indeed maintain better performance in the worst-case attack.
trainset. On the other hand, in the case of FGSM attack,
as it is regarded
In order to perform a broader evaluation of the adversarial robustness of our scheme, we investigate the robustness to loo bounded attacks for different attack strengths. We use the PGD based trained model and keep during the training the e value constant with a value of 0.3. We compare between scheme. For the attack, we use PGD with 50 iterations of size 0.01 with 20 random restarts. We chose the value to be equal 0.1 for the refinement strength. The results for MNIST dataset appear in figure Tal Our scheme show significant improvement compared to the same model without Adversarial PNML for all values. There is a rapid decrease for e value of 0.3 for the basic model. The decrease can be explained by attacking with that greater than the one it was trained with. Using our scheme, we able to maintain the performance up to value that equals to 0.39.
this model with and without
5.2 CIFARIO DATASET
modified version of Wide-ResNet (Zagoruyko & Komodakis 2016) with each layer being wider by factor of 10. This DNN contains three residual units with 160, 320, and 640 filters respectively, with total of 32 layers. As in MNIST dataset, we use three different trainsets to generate three different models: (i) Natural trainset; (ii) Adversarial trainset that was generated by FGSM attack on the natural trainset with value that equals to 8/255, (iii) Adversarial trainset that was produced by using PGD attack on the natural trainset with 20 steps of size 2/255 with total e of 8 8/255. We train over 200 epochs using SGD optimizer with a learning rate of 0.001, reducing it to 0.0001 and 0.00001 after 100 and 150 epochs respectively. We also use momentum value of 0.9 and weight decay of 0.0002. We evaluate the performance of our method against the performance of the same model without our scheme for all three trainsets.
For CIFARI0
We consider the following threat model: white-box Loo that is unaware of the defense with an attack strength of 8/ 8/255. We test the accuracy in the following test cases: natural samples, FGSM attack, and PGD attack. For the PGD attack, we perform 20 steps of size 2/255 with restart.
We illustrate results in Table I Similar to the MNIST results, the robustness against PGD attack by 23.7% for model that was trained with PGD trainset. Overall, the robustness against the worst-case attack is improved for all models. On the other hand, the
that scheme improves

---

Under review as a conference paper at ICLR 2020
1.0
10
0.8
Natural
Density (symlog scale)
0.6
Accuracy
0.4
0.2
0.0
Natural train, PGD attack
10"
-0.2
Natural train, No attack
PGD PCD
PGD train, No attack
0.03 Refinement strength
0,05
0.0
0.5
1.0
1.5
2.0
Regret
(a) Accuracy rate different attack strength
(b) CIFARIO regret histogram
Figure 2: (a) CIFARIO performance for different attack strength c. Two models are tested, the first trained with adversarial PGD trainset and the second with natural trainset. We compare the performance of those models with and without our proposed scheme. (b) Regret histogram for correctly classified natural samples and PGD adversarial samples which causes misclassification.
accuracy of natural samples is decreased. This is especially true when the model is trained with natural trainset when adversarial pNML is used the accuracy drops by 6.4%.
In figure lb we further explore PGD attacks accuracy as a function of the attack strength c. We use the previous model that was trained with PGD adversarial trainset, during training the remains constant with a value of 8/255. We compare the model robustness against the same model that was integrated with our scheme. Our scheme shows significant improvement, especially around e that equals 0.05 where we gain almost 30% improvement.
Choosing the refinement strength represents a trade-off between increasing the robustness against adversarial attacks and decreasing the accuracy of natural samples. This trade-off is explored in figure 2a The advantage of our scheme is displayed when it is combined with adversarial training (training with a PGD attacked trainset). As the refinement strength increases so is the robustness to PGD attack with almost no loss in accuracy for the natural data. On the other hand, when our method applied to model trained without adversarial training natural data. As elaborated in section[4thu the refinement process is, in fact, a weak targeted adversarial attack toward different labels. Therefore, when natural sample is inserted weakly adversarial. A model that was trained without adversarial training is susceptible even to weak attacks, hence the performance for natural samples drops. To limit the performance degradation for a non-adversarial trained model we chose a small X value of 0.02, while for the adversarially trained model we chose À value of 0.
decrease in accuracy for
scheme it becomes
We demonstrate the ability of the regret to differentiate between correctly classified natural samples and adversarial samples that causes misclassification in figure 2b which shows a clear separation between natural images and adversarial ones.
ADAPTIVE ADVERSARY
Part of the defense evaluation to create and test against adaptive adversaries that are aware of the defense (Athalye et al.) 2018 Carlini et al. 2019). We design an adaptive adversary for our scheme: We create an end-to-end model that calculates all possible hypotheses in the same computational graph. Then by using gradient-basce optimization the new end-to-end model, we create adaptive adversarial attack.
The first obstacle with using gradient-based optimization for that end-to-end model is that sign(-) operator that is used in the refinement stage, which sets the gradient to zero during the backpropa- gation. Athally al. (2018) suggested using Backward Pass Differentiable Approximation (BPDA) technique to overcome that problem. In BPDA, we perform forward-pas as usual, but on the back- ward pass, replace the non-differentiable part with the identity operator.

---

Under review as a conference paper at ICLR 2020
10
1.000
Natural BPDA
Density (symlog scale)
0.975
0.950
Accuracy
0.925
0.900
0.875
0.830
B=0.0
B=1.0
2
0.825
B=0.01
Without
B-0.5
%..1
0.5
0.00
0.15
0.30
Regret
FPR
(a) MNIST regret histogram
(b) Adversarial PNML with regret based detector
Figure 3: (a) Regret histogram for correctly classified natural samples and BPDA adversarial sam- ples which cause misclassification. In (b) the accuracy represents the ratio between adversarial samples correctly classified or detected and the total number of sample for various B. The black star point represents the worst-case attack (PGD) against the same model without defense.
order to overcome the BPDA attack, we use the inhered detection property of our scheme and we define the following goal (as in Meng & Chen (2017))) Correct classification is made when the true label assigned to the sample or when an adversarial sample is detected. This new goal gives rise to scheme which incorporates adversarial detector in addition adversarial pNML.. We utilize the regret from equation [5] to form an adversarial detector. Figure 3a shows the regret histogram for MNIST dataset. We note that adversarial samples generated by BPDA attack have higher regret values compared to natural samples. Using different regret thresholds, one can control the trade-off' between the accuracy of the adversarial testset and the False Positive Ratio (FPR) of the natural testset, i... the number of natural samples that are detected as adversarial samples.
Due to the assumption that the attacker is aware our defense, we propose second new attack that designed to target our detector and name it Adaptive Attack. The Adaptive Attack is designed minimize the regret to avoid detection. This is done by adding regularization term the loss:
L(x, Vtrue) = - log Pieg (ytrue Preefine (x. Virue))
3 log Σ Pt-
(16)
MM Pi
The advantage of our defense is that the attacker have contradictory objectives. On one hand, to avoid detection the attacker needs to minimize the regret log (>=1 Pi). and on the other hand to
increase the classifier loss - logg (Peg (Utrue (Frefine) / //M Ps) it needs to maximize log <M P:))
We demonstrate the results of the adversarial PNML combined with the regret detector in figure 3b over 3000 MNIST samples. We use the same model as described in section 5.1 trained with PGD adversarial trainset. For the refinement strength we chose value of 0.03. For adaptive attack, we examine various 3 values as shown in figure 3b Our proposed defense is successful against adaptive attack. For regret threshold of O. 19, we get 91 01% accuracy with FPR 2% against adaptive attack. This represents 4% improvement compared to model without our scheme.
7 CONCLUSION AND FUTURE WORK
In this paper we presented the Adversarial pNML scheme for defending DNNs from adversarial attacks by increasing robustness and detection. We showed empirically that our defense increases robustness against unaware white-box attacks for MNIST and CIFARI0 datasets. To overcome adaptive attacks, we successfully use the regret, which is an inherent property of our method, as an adversarial detector in order enhance our defense.
This work suggests several potential directions for future work. First, we plan to try our scheme against other attacks with different norms. Second, it is interesting to explore other hypothesis classes, one such class is the entire "model class" where instead refining the sample we refine the model according to different hypotheses.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Cireumventing defenses adversarial examples. arXiv preprint (tl::8800.04444 2018.
Koby Bibas, Yaniv Fogel, and Meir Feder. A new look old problem: A universal learning approach to linear regression. arXiv preprint arXiv:/905.04: 2019a.
Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pnml: Predictive normalized maximum likelihood for deep neural networks. arXiv preprint arXiv: 122866 2019b.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smdić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli. Evasion attacks against machine learning test time. Joint European conference machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017
IEEE Symposium on Security and Privacy (SP). pp. 39-57. IEEE, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, and Aleksander Madry. On evaluating adversarial robustness. arXiv preprint arXiv:1902.067000 2019.
Reuben Feinman, Ryan R Curtin, Saurabh Shintre, and Andrew B Gardner. Detecting adversarial samples from artifacts. arXiv preprint 1rXiv:1703.00410, 2017.
Yaniv Fogel and Meir Feder. Universal batch learning with log-loss. 2018 IEEE International Symposium on Information Theory (ISIT), pp. 21-25, IEEE, 2018a.
Yaniv Fogel and Meir Feder. Universal supervised learning for individual data. arXiv preprint 111.:2281,.09520, 2018b.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016a.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577: 2014.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens Van Der Maaten. Countering adversarial images using input transformations. arXiv preprint 1rXiv:1711.000 2017.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The sifar-10 dataset. online: http://www. cs. toronto. edu/kriz/cifar html, 55, 2014.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [On- line]. Available: http://yann. lecun. com/exdb/mnist, 2:18, 2010.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. Proceedings of the IEEE International Conference on Computer Vision, pp. 5764- 2772,2017,
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Dongyu Meng and Hao Chen. Magnet: a two-pronged defense against adversarial examples. In Proceedings of the 20/7 ACM SIGSAC Conference Computer and Communications Security, pp. 147- ACM, 2017.
Neri Merhav and Meir Feder. Universal prediction. IEEE Transactions on Information Theory, 44 (6):2124-2147 1998.
Pouya Samangouei, Maya Kabkab, and Rama Chellappa. Defense- gan: Protecting classifiers against adversarial attacks using generative models. arXiv preprint arXiv: 18855066055 2018.

---

Under review as a conference paper at ICLR 2020
Yurii Mikhailovich Shtar kov. Universal sequential coding of single messages. Problemy Peredachi Informatii,, 23,3)3317,,198 1987.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. arXiv preprint arXiv: 177,, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Pedro Tabacof and Eduardo Valle. Exploring the space of adversarial images. In 2016 International Joint Conference Neural Networks (IJCNN), pp. 426 433. IEEE, 2016.
Lestie G Valiant. A theory of the learnable. In Proceedings of the sixteenth annual ACM symposium on Theory of computing, pp. 436-445, ACM, 1984.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146,, 2016.
Wide residual networks.
arXiv preprint
10


Reviews:
[{"review_id": "Skxnt89CYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an adversarial pNML scheme for adversarial defence and adversarial example detection. The idea is very intuitive and pNML is adopted from the literature work for the purpose of adversarial defence. \n\nThe authors provided some explanation on why the adversarial pNML should work. The reasoning is quite intuitive, lacking of thorough justification.  The authors may consider using experiments to provide empirical justifications for the explanations. \n\nThe proposed method is heavily dependent on previous works. The section 6 adaptive adversary part is not clear.  How to do the adaptive attack based on Eq.(16)? Maximizing the loss in Eq.(16)?  How to determine the threshold for adversarial example detection?\n\nThe experimental results in Table 1 seems to be very good. However, have the state-of-the-art defence methods been compared?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJlBSAC3FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed the Adversarial predictive normalize maximum likelihood (pNML) scheme to achieve adversarial defense and detection. \nThe proposed method is an application of universal learning, which is compatible with existing adversarial training strategies like FGSM and PGD. \nHowever, the experimental results indicate that the proposed method is more suitable for the models trained under PGD-based attacks. \nAccording to the analysis shown in the paper, the proposed method works best when the adversary finds a local maximum of the error function, which makes it more robust to strong attacks. \nIt seems that the proposed work is a good attempt that applies universal learning to adversarial training, but more experiments are required to support its usefulness and effectiveness, especially for the weak attack like FGSM. Additionally, I would like to see more discussions about the limitations of the proposed method. \n\nMinors:\nIn Figure 2, I would like to see the results related to FGSM.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJx5pd_6Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper focuses on the area of adversarial defense -- both to improve robustness and to detect adversarially perturbed images. They approach the problem from the universal prediction / universal learning framework.\nModivation:\nI had a hard time understanding the motivation of this work -- specifically the connection to the universal learning framework which to be fair I am unfamiliar with. In the absence of the universal learning framework formalism the method proposed here is quite simple and in my opinion clever -- create a new prediction based on performing an adversarial attack to each target class. What value does universal learning bring to this?\nSecond, I do not follow the intuition for the chosen hypothesis class -- why work off of refined images in the first place? Is there some reason to believe this will improve robustness?\nFinally, the view that adversarial defense and attack are important topics to explore is under some debate. I am not considering this as part of my review but I would encourage the authors to look at [1].\nWriting:\nThe writing was clear and typo free.\nExperiments:\nOverall the experiments seemed inconclusive.\nSection 5 shows robustness against the unmodified / unrefined model (the attacks are done on the base model not the refined model). Given that these attacks are performed against the unmodified model then evaluated on the modified model the results seem a bit unfair / harder to interpret. The authors note this, and in Section 6 explore the \"Adaptive Adversary\" setting.\nThe results presented are performed on Mnist and Cifar10. Overall the results were not convincing to me. Table 1 shows mixed performance -- a drop in natural accuracy in all cases, decreases in FGSM. The main increase in performance is in the PGD. This was noted, but understanding in more depth why this method helps here will hopefully lead to improved performance in FGSM as well.\nFigure 2a shows very weak correlations. Figure 2b seems promising but also not necessarily a surprise given that the adversarial examples are generated against the base model and not the refined model.\nFor section 6, one risk is that the BPDA attack doesn't successfully work. Having some more proof that the attacks presented here are strong would greatly improve the work.\nLarger scale experiments would of course be nice and strengthen the paper but more importantly it would be great to see some form of toy example or demonstration of the principle improving robustness as well over just results. Something to probe the mechanism of action for example.\nFinally, having some comparisons to other defense strategies would improve this paper.\nRating:\nGiven the gap between the universal learning framework and the method proposed, as well as the inconclusive experiments at this point I would not recommend the paper for acceptance.\n\n[1] https://arxiv.org/abs/1807.06732", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers attempted to give this paper a fair assessment, but were unanimous in recommending rejection.  The technical quality of motivation was questioned, while the experimental evaluation was not found to be clear or convincing.  Hopefully the feedback provided can help the authors improve their paper.
Decision: Reject